(feat): dVRK PSM retargeters#769
Conversation
Signed-off-by: Chris von Csefalvay <chris@chrisvoncsefalvay.com>
Signed-off-by: Chris von Csefalvay <chris@chrisvoncsefalvay.com>
|
Important Review skippedAuto incremental reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
📝 WalkthroughWalkthroughAdds dVRK PSM clutch and gripper retargeters with NumPy state machines for absolute tool poses and paired jaw targets. The change exposes public configurations and retargeter classes, integrates the package into builds, adds extensive simulator-free tests, and documents behavior, reference frames, Python wiring, and validation commands. Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant ControllersSource
participant DVRKPSMClutchRetargeter
participant DVRKPSMGripperRetargeter
participant OutputCombiner
participant SimulatorIK
ControllersSource->>DVRKPSMClutchRetargeter: grip pose and squeeze
ControllersSource->>DVRKPSMGripperRetargeter: trigger and squeeze
DVRKPSMClutchRetargeter-->>OutputCombiner: ee_pose
DVRKPSMGripperRetargeter-->>OutputCombiner: jaw_targets
OutputCombiner->>SimulatorIK: combined pose and jaw targets
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
|
📝 Docs preview is not auto-deployed for fork PRs. A maintainer with write access to |
There was a problem hiding this comment.
Pull request overview
Adds simulator-independent XR retargeters for the dVRK Patient Side Manipulator (PSM), providing a clean boundary that outputs a bounded absolute tool pose plus paired jaw targets for downstream IK / articulation control.
Changes:
- Introduces
DVRKPSMClutchRetargeter(7Dee_pose) andDVRKPSMGripperRetargeter(2Djaw_targets) with NumPy-only control kernels. - Wires the new dVRK retargeters into the Python package surface and build/test tooling (mypy path, package staging).
- Adds extensive simulator-free unit tests and new retargeter documentation/reference material.
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| src/retargeters/DVRK/psm_retargeter.py | New engine-facing retargeter wrappers for pose clutch and jaw targets. |
| src/retargeters/DVRK/control.py | New NumPy-only clutch + jaw-intent state machines and math helpers. |
| src/retargeters/DVRK/init.py | Lazy-export surface for the DVRK retargeter subpackage. |
| src/retargeters/CMakeLists.txt | Ensures DVRK __pycache__ is removed during Python package staging. |
| src/retargeters/init.py | Exposes new dVRK retargeters/configs from isaacteleop.retargeters. |
| src/core/retargeting_engine_tests/python/test_dvrk_psm_retargeters.py | New simulator-free test suite for dVRK PSM retargeters and kernels. |
| src/core/retargeting_engine_tests/python/CMakeLists.txt | Adds mypy coverage for the new src/retargeters/DVRK modules. |
| src/core/python/pyproject.toml.in | Includes isaacteleop.retargeters.DVRK in the packaged modules list. |
| docs/source/references/retargeting/index.rst | Adds dVRK entry to the retargeter reference index. |
| docs/source/references/retargeting/dvrk.rst | New documentation page describing dVRK PSM retargeter usage and contracts. |
Signed-off-by: Chris von Csefalvay <chris@chrisvoncsefalvay.com>
|
@jiwenc-nv &al. - this is very closely linked with a PR in Isaac Lab that's contingent on it: #6527, and might make sense to be reviewed together. The latter is still in draft, so when this is merged, I can undraft it. |
|
Hey Chris, thanks for contributing. Will take a look soon. Wondering whether you have more context on the overall effort? Looks interesting, but wanna to know how we can better support your work. |
|
Hi @jiwenc-nv, very fair point, and my apologies for not getting this in sooner. The dVRK (da Vinci Research Kit) is the closest that surgical robotics has to a SO-ARM 101. It is a research tool made available by Intuitive Surgical (ISRG), who make the da Vinci surgical robots. These consist of four arms, of which typically one holds an endoscopic camera (the ECM) and up to three hold what are known as patient-side modules (PSMs), which are the tool arms. They come loaded with instruments such as needle drivers, forceps &c. Unfortunately, you can't just go out and buy a dVRK - there are around 35 or so of them around in the whole world, all of them at universities, and the requirements to get one are fairly stringent. This makes developing policies and testing them on surgical tasks rather difficult. At the same time, exactly because of the academic abundance (relative to all other surgical robots) of the dVRK, as well as the fact that it's effectively open-source and is arguably the closest thing we have to a broadly adopted, purpose-built research embodiment, and because it has an explicit non-clinical research boundary, it is very widely used. Open-H-Embodiment, an NVIDIA-sponsored initiative to collect surgical robotic training data, uses it extensively (around 1/7th of the data comes from dVRK or da Vinci units, which are kinematically identical - the CMR Versius is more abundant but less open), and most of the basic robotic surgical routine primitives (needle passing being one example) are implemented. It is thus an optimal embodiment to offer an entry into surgical robotic teleoperation. Not only is this a way for users without access to a dVRK to collect high quality teleop data, it is also representative of a class of retargeters that is not present here (there are some kinematic quirks to how laparoscopic surgical robots work that are not replicated by the conventional structures) and since it's almost always (at least) bimanual, it's a good veer into that realm as well. I have previously explored dVRK + Isaac in LeOnardo, an Isaac Sim extension for bimanual PSM teleoperation using two SO-ARM 101 leader arms. That was useful proof that the combination works, but it also reinforced that there is little value in maintaining a separate bespoke stack when IsaacTeleop, Isaac Lab, OpenXR/CloudXR and Isaac Sim already form such a strong ecosystem. I would much rather put the reusable control seam upstream here and keep task-specific IK, articulation, contacts, limits and emergency-stop behaviour downstream. An example of an immediate downstream use case is now IsaacLab #6570, which supersedes the earlier #6527. It adds a contact- and load-qualified bimanual needle pass—one of the foundational subtasks within suturing—with paired-controller XR teleoperation. The same primitive already supports SurgiSabre, essentially Beat Saber for surgical robots: a deliberately playful, low-stakes familiarisation environment for clutching, remote-centre motion and workspace limits. Longer term, I would love to see this shared dVRK teleoperation layer become part of the backbone of CERTIS, an Isaac-native collection of composable surgical tasks for teleoperation, demonstration collection, training and reproducible evaluation, and also be a starting point for others developing in surgical robotics. I have deliberately kept #769 small and simulator-independent: controller input in, bounded PSM pose and jaw targets out. I would be grateful for any guidance and support on making sure that my way of making this work aligns with what your longer term architectural ideas are for IsaacTeleop - after all, I'm the guest here, I have tried to align as well as I could with the preexisting codebase but happy to accommodate any changes. |
Very thoughtful and nice writeup! Thanks for posting. I will encourage you to include a version of that explanation into: docs/source/references/retargeting/dvrk.rst as well. Also on dvrk.rst, I'd recommend you include a gif (no larger than 5MB please to show case the teleoperation you have). We are also brining in SO-101 leader arm within Isaac Teleop as well. (also more leader arm to come), would love you to also think about how those hardware can be useful for you as well. @rwiltz can you take a pass on the PR as well? as you are the expert for our retargeting pipeline. |
Brief context of embodiment, ECM/PSM roles and simulation, add GIF of bimanual needle passing teleop run sub 5 megs. Signed-off-by: Chris von Csefalvay <chris@chrisvoncsefalvay.com>
|
@jiwenc-nv added, thanks! |
Description
Adds reusable XR retargeters for da Vinci Research Kit (dVRK) Patient Side Manipulators (PSMs).
The retargeters provide a simulator-independent boundary between an XR controller and a PSM integration: they emit a bounded absolute tool-tip pose and paired jaw-joint targets, while robot-specific differential IK, articulation control, collision handling, and emergency-stop behaviour remain with the downstream consumer.
What is included
DVRKPSMClutchRetargeteremits a float32ee_posein[x, y, z, qx, qy, qz, qw]order.DVRKPSMGripperRetargeteremits float32jaw_targetsfor the two native PSM jaw joints.isaacteleop.retargetersand included in Python package staging.Control and safety semantics
The squeeze input (deadman) gates target updates. Releasing it holds the last commanded target and re-arms controller-origin capture; it does not disable downstream actuation or latch the measured articulation state. Command disable, measured-state hold, joint and collision limits and emergency-stop behaviour remain the responsibility of the simulator or robot integration.
The output is designed to be fail-safe, i.e. tracking loss, inactive sessions, missing controller data, malformed poses, degenerate quaternions and non-finite samples hold the last safe output and require a fresh engagement. Configuration values that will be narrowed to float32 are validated against the finite float32 range.
The jaw retargeter preserves the last jaw target across clutch release and tracking transitions. Its opening dwell uses observed graph time rather than frame count, including protection against repeated or regressed timestamps.
Type of change
Testing
SKIP=check-copyright-year rtk uvx pre-commit run --all-files: passed.git diff --check: passed.Coverage includes clutch state transitions, workspace clipping, no-jump re-engagement, an independent non-commuting orientation-conjugation oracle, paired jaw interpolation, deliberate opening dwell, graph-time regressions, malformed and non-finite samples, float32-range configuration rejection and, hopefully, public wrapper/kernel parity.
Scope
This PR contains the reusable dVRK retargeters, public API and configuration, focused tests, packaging, and retargeter documentation only. For the sake of separation of concerns, the Isaac Sim/Lab example, diff IK and articulation validation will be in the downstream dual PSM integration.
Checklist
SKIP=check-copyright-year pre-commit run --all-filesgit commit -s) per the DCOSummary by CodeRabbit
New Features
Documentation
Tests