How to transform a gameobject based on 3d 3d registration matrix

I am trying to do a 3d-3d point based registration. I have six points that I used to determine the registration matrix. When i do RegMat.multiplyPoint(point), it gives correct answers. However, when I try to transform a gameobject based on the matrix (code below), it gives me some strange results as shown in the image. What is the correct way to transform a gameobject in Unity?

Note1: The blue points on the virtual object should coincide with green (groundtruth) and red (RegMat.multiplyPoint(point)) sphere positions.

Note2: The object is placed at origin so multiplying the current position with RegMat gives the same result.

Note 3: The object centre position seems to be at the right position but the rotation is totally wrong.

 obj.transform.localPosition = MatrixHelper.GetPosition(RegMat);
 obj.transform.localRotation = MatrixHelper.GetQuaternion(RegMat);
 obj.transform.localScale = MatrixHelper.GetScale(RegMat);
 
     public class MatrixHelper
     {
         // Start is called before the first frame update
         public static Quaternion GetQuaternion(Matrix4x4 matrix)
         {
             Vector3 forward = new Vector3(matrix.m02, matrix.m12, matrix.m22);
             Vector3 upwards = new Vector3(matrix.m01, matrix.m11, matrix.m21);
             return Quaternion.LookRotation(forward, upwards);
         }
 
         public static Vector3 GetPosition(Matrix4x4 matrix)
         {
             return new Vector3(matrix.m03, matrix.m13, matrix.m23);
         }
 
         public static Vector3 GetScale(Matrix4x4 matrix)
         {
             Vector3 scale;
             scale.x = new Vector4(matrix.m00, matrix.m10, matrix.m20, matrix.m30).magnitude;
             scale.y = new Vector4(matrix.m01, matrix.m11, matrix.m21, matrix.m31).magnitude;
             scale.z = new Vector4(matrix.m02, matrix.m12, matrix.m22, matrix.m32).magnitude;
             return scale;
         }
 
     }