Unity Plugin: Mocap Bodies

[Balint-H]

I'd like to track a Unity-driven humanoid animation's kinematic state with MuJoCo, estimating the body segments' linear and angular velocities, for use in imitation learning. As I understand the appropriate way to do this is assigning MjMocapBodies to each tracked body segment, and welding a MjBody to them, the kinematics of which can then be tracked.

I downloaded the Modular Prosthetic Limb environment, which is listed as the recommended example in the documentation, and loaded it in Unity. I was surprised to see that the mocap body that controls the forearm (has mocap="true" listed in the XML) was imported as a MjBody, not as a MjMocapBody. Is this as intended? I found no way to alter the position of the mocap target (either through the transform, or via mocap_pos), and MjScene.Instance.Model->nmocap returned 0.

I replaced the MjBody component with a MjMocapBody, which updates mocap_pos and mocap_quat based on the Unity Transform. MjScene.Instance.Model->nmocap returned 1 now. The position of the Unity transform still didn't affect the mujoco simulation, but, surprisingly, the Unity rotation affected the position in a great deal (along circular paths), with no effect on the rotation. What could be causing this? What's the right way to set up a kinematically controlled body like this?

[Balint-H]

Interestingly, changing

MjEngineTool.SetMjVector3(data->mocap_pos, transform.position, MujocoId);

to

MjEngineTool.SetMjVector3(data->mocap_quat, transform.position, MujocoId);

in MjMocapBody.cs and commenting out the existing line that sets mocap_quat based on the rotation makes the position behaviour work as expected.

[Balint-H]

In case anyone needs this, this seems to do the whole job, position + orientation tracking:

 public override unsafe void OnSyncState(MujocoLib.mjData_* data)
        {
            MjEngineTool.SetMjVector3(data->mocap_quat, transform.position, MujocoId);
            var mjQuat = MjEngineTool.MjQuaternion(transform.rotation);
            var startOffset = 7 * MujocoId;

            data->mocap_quat[startOffset + 3] = mjQuat.w;
            data->mocap_quat[startOffset + 4] = mjQuat.x;
            data->mocap_quat[startOffset + 5] = mjQuat.y;
            data->mocap_quat[startOffset + 6] = mjQuat.z;

        }

It appears that mocap_quat contains the whole 7 values of the transform? What is mocap_pos used for then? Am I missing something?

[erez-tom]

Thanks for sharing! This is a very weird bug, and I failed to recreate it (on V2021.2 on OSX nor on V2018.3 on Linux) -- I see MuJoCo's mocap fields updating according to the Editor transform as expected. Can you share more information on your specific setup (e.g., Unity version, OS)? Does this also happen with a minimal model (e.g., a scene with only one game object with an MjMocapBody component)?

[Balint-H]

Hello, apologies for the late reply.

I'm on Unity 2021.2.5f1, on Windows 10.

I tested it on a simple scene with just a single mocap body + MjBody welded together, and then in one with 3 repeats of the same object. I tried both importing the XML, in which case MjMocapBodies import as just MjBodies so I remove those and add the MjMocapBodies to those gameobjects, and I update the weld constraint references accordingly. Altering the transform of the default MjMocapBody in Unity does not alter the simulation state.

I also tried just building the scene manually inside Unity, with the same results.

I copied the code of MjMocapBody into a new class (I called it MjMocapBody_) and added the changes in the original post. It seems to be working well now even on a full humanoid mocap tracking scene, although I'm aware that it may break at some point.

I attached the XML file of the scene (as a TXT, Github didn't allow uploading with the .xml extension). Thank you for your support, I'm really looking forward to using MuJoCo in Unity! Let me know if any more info would be useful.

mocap_bodies.txt

[erez-tom]

I believe this was fixed in the latest release. Can you please revert your ad-hoc changes, update to the latest version, and let us know what you find?

[Balint-H]

Apologies for the late reply, MjMocapBodies work as expected on Windows in the most recent release! Thanks for your help. As far as I can tell, bodies with mocap=true in the XML are still imported as only MjBodies, but if they are replaced afterwards there are no issues.