Simulating with custom sequences

[DavideA]

Hi,
and thank you for the great work!

I am interested in running the simulator on my own recordings (i.e. not in the NuRec dataset).
Is there a tutorial on how to properly format the data? I could only find the expected USDZ structure in this readme, I was wondering whether there was further info or tutorial for it.

[maximilianigl]

Would you bring your own renderer or would you want to use NuRec, but with your own data?

There isn't a tutorial for either case. The first case (i.e. bring your own renderer) is something we would like to support more explicitly and are working on it, but don't have a timeline for it yet. However, this is something you could already try yourself and we're happy to help if you have questions.

For the second case, you'd need to process your data through the NuRec pipeline, which is not yet publicly available. This might be made public in the future, but there also isn't a timeline yet, unfortunately.

[DavideA]

Would you bring your own renderer or would you want to use NuRec, but with your own data?

Good question :D
Ideally both, but I hear that the former is easier. I can get it started, but what confused me is that other services read the usdz too (e.g. eval, physics, runtime).

So it seems that in any case I need to go through the NuRec data pipeline to run them, am I right?

[maximilianigl]

Yeah, the data abstraction here isn't great yet. It's one area we want to improve (which is what the Artifact class is meant for).
You do get some generality because we're using trajdata under the hood for maps, which support multiple formats. But for agent trajectories we're currently still relying on the usdz format.

[maximilianigl]

If you do start working on this, let us know - there is some ongoing work in that direction (both internal and external) so we should make sure we stay in sync in that case. Or you wait a couple of months to see how far we get then :).

[codedetermination]

Would you bring your own renderer or would you want to use NuRec, but with your own data?

There isn't a tutorial for either case. The first case (i.e. bring your own renderer) is something we would like to support more explicitly and are working on it, but don't have a timeline for it yet. However, this is something you could already try yourself and we're happy to help if you have questions.

For the second case, you'd need to process your data through the NuRec pipeline, which is not yet publicly available. This might be made public in the future, but there also isn't a timeline yet, unfortunately.

Thanks for the insights! Just to follow up on the map requirement: Since handles the maps, is a .xodr file strictly required to initialize the simulation, or is it primarily used for evaluation purposes (e.g., calculating off-road scores)?trajdata

If I successfully reconstruct the USDZ assets for the agents and environment, would that be enough to run the core simulation/visualization, or does the runtime logic rely on the underlying .xodr for lane-following or physics constraints?

[codedetermination]

Regarding the USDZ structure, I am specifically looking into using 3DGRUT—another NVIDIA project—to reconstruct 3D environments and trajectories from my own video recordings.

Since 3DGRUT naturally outputs USDZ files, I was wondering if these assets would be compatible with the NuRec simulation loop? Specifically, are there any unique schemas or metadata requirements in the NuRec USDZ format that 3DGRUT might miss, or should they be theoretically interchangeable for the runtime/physics modules?

[DavideA]

Thanks for the insights! Just to follow up on the map requirement: Since handles the maps, is a .xodr file strictly required to initialize the simulation, or is it primarily used for evaluation purposes (e.g., calculating off-road scores)?trajdata

If I successfully reconstruct the USDZ assets for the agents and environment, would that be enough to run the core simulation/visualization, or does the runtime logic rely on the underlying .xodr for lane-following or physics constraints?

I am looking into the same. In my understanding, the runtime does need the map (see here, whereas the physics simulation only needs the ply mesh or the ground mesh.

In my current understanding this is needed by services reading the usdz artifacts:

1. eval service: map.xodr
2. physics service: ground_mesh.ply or mesh.ply
3. Runtime: metadata.yaml, rig_trajectories.json, sequence_tracks.json, map.xodr, ground_mesh.ply or mesh.ply

[maximilianigl]

The primary use case for the map are indeed the metrics.
The runtime is using it for route generation: it projects the GT path onto the center lines, extends those into the future (to minimize information leakage through the length of the route) and uses that as route for the policy.
That being said, one can use the GT directly (set route_generator_type=RECORDED in base_config.yaml), so the map isn't required.

I know that people have run AlpaSim without maps and only a subset of metrics (e.g. collisions). It's possible that the route generator change alone is enough, but it's also possible that there's a couple of additional hickups to work through. But there shouldn't be any large obstacles. @karkuspeter might know more.

Re USDZ structure: Let me find someone from the NuRec team that can hopefully answer this - we're just using their library.

The ground mesh is needed to project the vehicle(s) onto the ground (which is what we call physics). You can disable physics (see physics/disabled.yaml).

rig_trajectories and sequence_tracks are needed, those provide the GT positions for the Ego and other agents (including initial positions)

[DZiYou]

The primary use case for the map are indeed the metrics. The runtime is using it for route generation: it projects the GT path onto the center lines, extends those into the future (to minimize information leakage through the length of the route) and uses that as route for the policy. That being said, one can use the GT directly (set route_generator_type=RECORDED in base_config.yaml), so the map isn't required.

I know that people have run AlpaSim without maps and only a subset of metrics (e.g. collisions). It's possible that the route generator change alone is enough, but it's also possible that there's a couple of additional hickups to work through. But there shouldn't be any large obstacles. @karkuspeter might know more.

Re USDZ structure: Let me find someone from the NuRec team that can hopefully answer this - we're just using their library.

The ground mesh is needed to project the vehicle(s) onto the ground (which is what we call physics). You can disable physics (see physics/disabled.yaml).

rig_trajectories and sequence_tracks are needed, those provide the GT positions for the Ego and other agents (including initial positions)

@maximilianigl Hi,

Thanks again for the help! Just wanted to follow up—have you had a chance to connect with anyone from the NuRec team regarding the USDZ requirements?

I am currently experimenting with NuRec's rendering using my own data. I have generated a .ply file via 3DGS (3D Gaussian Splatting) and converted it to a .usdz asset using the scripts provided in the 3DGRUT repository.

However, I noticed that the standard NuRec scene structure seems to expect additional files that are missing from my 3DGRUT output, specifically parsed_config.yaml and checkpoints. This has left me a bit stuck on how to properly initialize the renderer for a custom scene.

Could you clarify (or ask the NuRec team):

How can I adapt a custom 3DGS-based .ply or .usdz for the NuRec rendering pipeline?

Are parsed_config.yaml and specific checkpoints strictly required for rendering, or is there a way to run a "vanilla" render of the reconstructed geometry?

Thank you so much for your support!

[DZiYou]

Regarding the USDZ structure, I am specifically looking into using 3DGRUT—another NVIDIA project—to reconstruct 3D environments and trajectories from my own video recordings.

Since 3DGRUT naturally outputs USDZ files, I was wondering if these assets would be compatible with the NuRec simulation loop? Specifically, are there any unique schemas or metadata requirements in the NuRec USDZ format that 3DGRUT might miss, or should they be theoretically interchangeable for the runtime/physics modules?
@codedetermination
Hi,

I suspect you might have already tried generating USDZ files using 3DGRUT, but as I've found, the output structure differs significantly from what NuRec expects.

The most prominent gaps are the missing parsed_config.yaml and checkpoints files. Furthermore, I haven't been able to find any detailed documentation regarding the NuRec rendering image (docker/environment), which makes it very difficult to adapt my custom data to their specific rendering pipeline.

If you have made any progress on aligning these two projects or have any insights on the renderer's requirements, I’d love to sync up. Looking forward to hearing from you!

[maximilianigl]

Sorry for not getting back to you sooner! I'm not familiar myself with the internals of Nurec unfortunately, but I just forwarded the questions to someone from their team.

The lack of information about the container's internals is currently intentional (it's an obfuscated version).

[DZiYou]

Thanks for forwarding the questions. I understand the internals are obfuscated for security; I’m just looking to render some general 3DGS scenes. Looking forward to the update.

[habibian]

Sorry for not getting back to you sooner! I'm not familiar myself with the internals of Nurec unfortunately, but I just forwarded the questions to someone from their team.

The lack of information about the container's internals is currently intentional (it's an obfuscated version).

Thanks for the context @maximilianigl . Any insights on why the nurec is being obfuscated? Not being able to run simulations (beyond the provided canned scenes) drastically limits the usability of such a great tool

[maximilianigl]

I can't really comment on NuRec as it's another team, and also has more "customers" than just AlpaSim.
However, we realize that "Bring your own data" would be super useful and it's something we hope to enable in the future.

[shikhar4]

Hi @DZiYou !

However, I noticed that the standard NuRec scene structure seems to expect additional files that are missing from my 3DGRUT output, specifically parsed_config.yaml and checkpoints. This has left me a bit stuck on how to properly initialize the renderer for a custom scene.

The parsed_config.yaml + checkpoint file is required for compatibility with the current nurec distribution. This workflow (using 3dgrut) is incompatible with the artifacts necessary for Alpasim.

But stay tuned though, I confirmed today we will be publishing workflows for users to bring their own data soon.