mjcf loading consistency with unity plugin and python interface

[spoonbobo]

Intro

Hi!

I am an ai researcher, recently migrating my application with mujoco to unity plugin.

My setup

mujoco version: 3.2.7
mujoco-plugin (unity): 3.2.7

My question

I am trying to use dm_control.Mujoco (or mujoco , since dm_control is wrapping mujoco) to load mjcf model and reset data with Unity plugin's MJCF runtime mjData (qpos, qvel for example).

With this workflow, can I assume both unity and python mujoco are loading mjcf consistently, so I could still use dm_control APIs while having mjData generated in Unity runtime? Thanks!

Confirmations

• I searched the latest documentation thoroughly before posting.
• I searched previous Issues and Discussions, I am certain this has not been raised before.

[Balint-H]

Hello,

The intention is for the data arrays to be ordered the same way. One thing that is guaranteed to change at the moment is the names of the mujoco components - they get a string appended for their names in the underlying mjData (not visible in Unity). So index-based environments are expected to behave similarly, but named access based environments are expected to fail.

Once you imported your MJCF in Unity, check out Asset > Export MJCF. The generated file will contain the fully expanded MJCF that the Unity runtime will use for MuJoCo. If the mjData generated from that MJCF is incompatible with your environment, then the mjData generated in Unity will also be incompatible.

There are also certain conditions in Unity that may alter the order of components which would also break index based referencing, if no additional care is taken.

Can you explain more about your goals and the types of scenes/environments you are investigating?

[spoonbobo]

Hi @Balint-H , thanks for your response

sure, so one of my goals is:

• First, I have a mjcf namely humanoid.xml link
• Then, i imported this in Unity, Asset -> Import MJCF scene (Note: I kept original actuators/control, no joints/actuators are modified in Unity runtime)
• In Unity, I've tagged few Unity boxes (like 3D cubes) to have colliders with Added a matching Mujoco geom.
• In Unity scripting interface, i have following code to prepare qpos and qvel:

    public unsafe List<double> getQpos()
    {
        var mjModel = MjScene.Instance.Model;
        var mjData = MjScene.Instance.Data;
        List<double> qpos = new List<double>();
        for (int i=0; i < mjModel->nq; i++)
        {
            qpos.Add(mjData->qpos[i]);
        }
        return qpos;
    }

    public unsafe List<double> getQvel()
    {
        var mjModel = MjScene.Instance.Model;
        var mjData = MjScene.Instance.Data;
        List<double> qvel = new List<double>();
        for (int i=0; i < mjModel->nv; i++)
        {
            qvel.Add(mjData->qvel[i]);
        }
        return qvel;
    }

• The qpos and qvel arrays will be sent to a python process through zmq or netmq.
• Then in python interface, I initialized physics using from dm_control import mujoco and then physics = mujoco.Physics.from_xml_string("humanoid.xml"). Like you said, I've modified some Unity objects to have Mujoco geoms attached, but again no actuators/ joints are modified. (Fyi I also handled the appended suffix as you mentioned in your reply by removing the suffix and matching the names, this part is not hard.)
• After intialization of physics, I reset physics Mujoco interface by

with physics.reset_context():
  physics.named.data.qpos = NEW_QPOS
  # same for qvel

• Next, I want to reuse IK calculation API by dm_control in here: https://github.com/google-deepmind/dm_control/blob/main/dm_control/utils/inverse_kinematics.py, defined as

def qpos_from_site_pose(physics,
                        site_name,
                        target_pos=None,
                        target_quat=None,
                        joint_names=None,
                        tol=1e-14,
                        rot_weight=1.0,
                        regularization_threshold=0.1,
                        regularization_strength=3e-2,
                        max_update_norm=2.0,
                        progress_thresh=20.0,
                        max_steps=100,
                        inplace=False):

My actual questions now is

• Is my physics.reset_context() valid? can I set data (qpos/ qvel) like this. (I've checked using python-interface / Unity-C# interface, both Mujoco give same length/ ordering of joints, like you mentioned.

I wish to confirm this way of resetting data can result in proper IK calculation given loading humanoid.xml in both Unity Plugin & python interfaces.

[Balint-H]

As far as setting qpos, it is very much possible. Naturally it introduces a discontinuity in your physics, and there will be missing dynamics (unless you do inverse dynamics as well) for the frames you do the "teleportation". It is appropriate and common however for a custom reset function like this to just set the kinematics.

Using reset_context lets you skip calling mj_forward manually, which would be needed to ensure all you global positions and contacts will be correct immediately.
(The context calls forward after exiting it: https://github.com/google-deepmind/dm_control/blob/a4b9b312c4ae82dded5d2a331d0b5904bf33b869/dm_control/mujoco/engine.py#L307)

As long as the values of the new qpos refer to the corresponding joint, it should not matter where the values come from. So far your approach looks good to me.

It is still unclear what the benefit of including the Unity runtime is - is the Unity application the end target? If so, you may consider using Unity's IK tools or reimplementing the dm_control IK function in C# instead of communicating between two MuJoCo versions, if there are no other features used from python. This would likely avoid potential problems in the messaging and give you a performance boost.

[spoonbobo]

@Balint-H Thanks for your prompt response.

for our app, the only reason now to load physics (a mujoco version) in python interface is reusing the IK calculation API available there. In other words, we just wanted the new qpos from its output IKResult. I'd definitely consider to use unity's IK tools/ re-implementation like you said. It seems beneficial and reduce the need of managing IK between versions.

To your question, Unity environment benefits us in scene creations, simulated sensory data, etc. So Unity runtime is more like a visual environment for us to validate our AI framework. The end target leans heavily toward on python side in our case, which our AI research and algorithm take place. So we treat Unity with mujoco more like a simulated environment, and the humanoid inside is kind of controlled by the AI framework.

On python, we are loading a lot of test-time training models with libs like pytorch as some sort of AI backend at the moment, so we foresee that in short term we still need a python backend to perform, but I think I should to try to get rid of 2 Mujoco versions, and only stick 1 Mujoco runtime API usage in Unity C# env.

Btw, thanks a lot! 😄