koobas.hobune.stream
Duplicating a Mesh including blendshapes to send via network
Hello all,
I'm stuck trying to save all information of a mesh (imported from an fbx) to a serializable format so that I can send that information via network. Don't worry, I only have to do this once, so file size is not an issue for now. Copying vertices, normals, tangents, and so on is not an issue and works just as expected.
However, when I want to copy the blendshapes, this happens:
How the original looks with blendshape "jawopen" modified: 
How the copy looks: 
As you can see, while something is modified when altering the blendshape weights, some information is lost here.
Does anyone know what the issue might be here?
For testing, I made a simple script that copies all information from the current mesh to a new one.
public class MeshHolder
{
public string name;
public Vector3[] vertices;
public Vector3[] normals;
public Vector4[] tangents;
public Vector2[] uv;
public int[] triangles;
public BlendShapeHolder[] BlendShapeHolders;
public Matrix4x4[] bindposes;
public BoneWeight[] boneWeights;
public Bounds bounds;
public int blendshapeCount;
}
public class BlendShapeHolder
{
public string name;
public float weight;
public int frameCount;
public int vertexCount;
public Vector3[][] dVertices;
public Vector3[][] dNormals;
public Vector3[][] dTangents;
}
public void SaveMesh(){
Mesh myMesh = CurrentMesh.sharedMesh;
MeshHolder.normals = myMesh.normals;
MeshHolder.vertices = myMesh.vertices;
MeshHolder.uv = myMesh.uv;
MeshHolder.tangents = myMesh.tangents;
MeshHolder.triangles = myMesh.triangles;
MeshHolder.bindposes = myMesh.bindposes;
MeshHolder.boneWeights = myMesh.boneWeights;
MeshHolder.bounds = myMesh.bounds;
MeshHolder.blendshapeCount = myMesh.blendShapeCount;
Vector3[] dVertices = new Vector3[myMesh.vertexCount];
Vector3[] dNormals = new Vector3[myMesh.vertexCount];
Vector3[] dTangents = new Vector3[myMesh.vertexCount];
MeshHolder.BlendShapeHolders = new BlendShapeHolder[myMesh.blendShapeCount];
for (int shape = 0; shape < myMesh.blendShapeCount; shape++)
{
BlendShapeHolder holder = new BlendShapeHolder();
holder.dVertices = new Vector3[myMesh.blendShapeCount][];
holder.dTangents = new Vector3[myMesh.blendShapeCount][];
holder.dNormals = new Vector3[myMesh.blendShapeCount][];
holder.frameCount = myMesh.GetBlendShapeFrameCount(shape);
for (int frame = 0; frame < holder.frameCount; frame++)
{
holder.name = myMesh.GetBlendShapeName(shape);
float frameWeight = myMesh.GetBlendShapeFrameWeight(shape, frame);
holder.weight = frameWeight;
holder.dVertices[frame] = new Vector3[myMesh.vertices.Length];
holder.dTangents[frame] = new Vector3[myMesh.vertices.Length];
holder.dNormals[frame] = new Vector3[myMesh.vertices.Length];
myMesh.GetBlendShapeFrameVertices(shape, frame, dVertices, dNormals, dTangents);
for (int k = 0; k < holder.dVertices[frame].Length; k++)
{
holder.dVertices[frame][k] = dVertices[k];
holder.dNormals[frame][k] = dNormals[k];
holder.dTangents[frame][k] = dTangents[k];
}
}
MeshHolder.BlendShapeHolders[shape] = holder;
}
}
... and transfers it to the new one.
public void LoadMesh(){
newMesh = new Mesh();
newMesh.vertices = MeshHolder.vertices;
newMesh.normals = MeshHolder.normals;
newMesh.uv = MeshHolder.uv;
newMesh.tangents = MeshHolder.tangents;
newMesh.triangles = MeshHolder.triangles;
newMesh.bindposes = MeshHolder.bindposes;
newMesh.boneWeights = MeshHolder.boneWeights;
newMesh.bounds = MeshHolder.bounds;
for (int shape = 0; shape < MeshHolder.blendshapeCount; shape++)
{
int frameCount = MeshHolder.BlendShapeHolders[shape].frameCount;
for (int frame = 0; frame < frameCount; frame++)
{
BlendShapeHolder holder = MeshHolder.BlendShapeHolders[shape];
string shapeName = holder.name;
float frameWeight = holder.weight;
Vector3[] dVertices = holder.dVertices[frame];
Vector3[] dNormals = holder.dNormals[frame];
Vector3[] dTangents = holder.dTangents[frame];
newMesh.AddBlendShapeFrame(shapeName, frameWeight, dVertices, dNormals, dTangents);
}
}
I'm linking you @Bunny83 , as you've posted the mesh serializer that I've tried. Using it, this above seen error occurs.
So, I came up with a solution after all. As a reminder, my main issue was that when copying data from a mesh object to a new mesh object, I seem to lose some information that affects the animation of the blendshapes as seen in the above images.
The solution: Do not use a new mesh at all but use a prefab of a working mesh, instantiate it, whenever you need to create a new object and then switch out all the information from the instantianted mesh with the copied mesh.
As an example: I saved one avatar head with working blendshapes as a prefab. Whenever I send mesh data from another avatar head through the network, I do not use this data to create a new mesh, but I instantiate the prefab and override the prefab's mesh data with the data sent through the network ;)
I'm a bit confused what your question is about. Did you use my MeshSerializer or did you use your code? If you use your code, how do you actually send it over the network? Also what kind of network connection / transport layer do you use? Maybe you have issues with the index format? By default a new mesh uses 16bit indices so you are restricted to 64k vertices. You don't seem to care about submeshes (maybe your mesh doesn't have any?!) Are you sure your mesh is made of triangles? Unity now support different topologies like Quads.
Have you tried serializing and deserializing the mesh on the same machine, right away from the freshly serialized data? Just to ensure it's not a pure transport issue?
Ah, I see, that was a bit misleading. The above example was me trying to exclude everything that might cause errors:
Hence, it is already running serialization and deserialization on the same machine, so no networking involved. Also, I used your code as an example but left out everything to do with byte conversion to exclude another possible source of error.
Lastly, the mesh is made of triangles and has only one mesh.
All meshes are below 64k vertices.
Well, if you have the two meshes side by side, you could use my UVViewer editor window to better analyse the mesh structure. Besides visualizing the UVs of the mesh it can also show the list of vertices and triangles. $$anonymous$$aybe you can figure out what goes wrong. The code you posted doesn't seem to contain any errors. You still haven't mentioned how your "$$anonymous$$eshHolder" is actually serialized. (Json? BinaryFormatter? something else?). Keep in $$anonymous$$d that the Skinned$$anonymous$$eshRenderer itself also contains certain information. Specifically the linkage to the skeleton and the current weight of each blendshape.
I've created a rather simple test cube with only one blendshape that deforms one edge of the cube. Here, one can see quite well what goes wrong:
The cube on the right is the original imported cube, the one on the left after copying all mesh data.
I'm not serializing at all right now. I am simply copying the data into the $$anonymous$$eshHolder class and then use the data stored there to fill a new mesh object. Using your UVViewer, the mesh structure is identical..
