Apply an animation from BVH or data stream to a humanoid character at runtime

We are working on a system which creates visualizations out of motion capture data (e.g. make particle systems according to the movement). The motion capture data can be both file based (offline visualization) or real-time stream (online visualization). We use Unity's Mecanim to apply the animation from the motion capture data to various characters with different meshes and bone structures. The FBX is the format that’s normally used for animation in unity. So far we have worked out all the visualization with animations stored in FBX files. (We rig the FBX files when import, thus we can obtain the animation stored in those FBX files and apply them to the characters we got from the assets store.) However, the huge problem is that our data input is not FBX files, but either BVH file (offline visualization) or data stream (online visualization).

1) Offline visualization: In this case, our input data is BVH file instead of FBX file. We want to load and visualize the motion capture data that we have in the BVH format to the system. The only problem is that we cannot load any BVH file into unity and apply the animation in the BVH file to our character in the scene.

We tried two potential solutions that failed.

a. Parse BVH file into animation clip by library, and then apply the clip to the character We have tried to use the BvhImportorExportor library (https://winterdust.itch.io/bvhimporterexporter ). This library can read BVH files and extract the animation clip from the BVH file. But the problem is that we cannot apply the animation clip to the character. We successfully get the animation clip from the BVH file, and applied that to the character's animation controller (by using animation override controller). But the character won't move as expected, and the clip complains that the animation targets are missing (as screenshot below).

We think this is because that the skeleton structure extracted from the BVH file won't match the underlying humanoid skeleton structure inside unity Mecanim system.

b. Convert BVH file to FBX file We tried to convert BVH files to FBX using Blender but we could not get the new FBX files working properly in Unity. The FBX file can successfully be generated by Blender, but the FBX file works weird in unity. Also, this way is not helpful for the online visualization.

However, even we manage to convert BVH file to FBX file, when we move to the online visualization in the future, the problem still exists.

2) Online visualization: This is down the road once we figure out the first problem. We want to be able to generate new movements and stream the generated data (which will be joint rotations) to Unity to animate them.

We see that to deal with this problem we need more in-depth knowledge about the mechanism of animation controller and unity mecanim, the underlying humanoid structure (how to transfer one structure in an animation clip into the underlying humanoid structure in runtime, or can we change the underlying humanoid structure to meet the structure in the animation clip).

Can anyone share some opinions on our problem?