Motion

.gitmodules

@@ -13,3 +13,15 @@
 [submodule "third_party/ObjectClear"] # for object clearing [reconstruction]
 	path = third_party/ObjectClear
 	url = [email protected]:PointsCoder/ObjectClear.git
+[submodule "third_party/graspness_unofficial"] # for grasp generation [motion]
+	path = third_party/graspness_unofficial
+	url = [email protected]:PointsCoder/graspness_unofficial.git
+[submodule "third_party/graspnetAPI"] # for grasp evaluation and utilities [motion]
+	path = third_party/graspnetAPI
+	url = [email protected]:PointsCoder/graspnetAPI.git
+[submodule "third_party/MinkowskiEngine"] # for sparse convolution operations [motion]
+	path = third_party/MinkowskiEngine
+	url = https://github.com/NVIDIA/MinkowskiEngine.git
+[submodule "third_party/WiLoR"] # for hand extraction [motion]
+	path = third_party/WiLoR
+	url = [email protected]:DynamoDuan/WiLoR.git

docker/compose.yml

@@ -1,6 +1,6 @@
 services:
   # Real To Sim [Master Container]
-  openreal2sim:
+  openreal2sim-new:
     build:
       context: ..
       dockerfile: docker/real2sim/Dockerfile

docker/real2sim/Dockerfile

@@ -8,7 +8,7 @@ ENV DEBIAN_FRONTEND=noninteractive
 # -------------------------------------------------------------------------
 RUN apt-get update && apt-get install -y --no-install-recommends \
     python3 python3-dev python3-pip python3-venv python-is-python3 \
-    build-essential cmake ninja-build pkg-config \
+    build-essential cmake ninja-build pkg-config libopenblas-dev \
     git curl wget ffmpeg bash libspatialindex-dev \
     && python3 -m pip install --upgrade --no-cache-dir pip wheel setuptools packaging \
     && rm -rf /var/lib/apt/lists/*
@@ -46,6 +46,9 @@ RUN pip install --no-cache-dir torch-scatter \
 COPY docker/real2sim/requirements.docker.txt         /tmp/requirements.docker.txt
 RUN pip install --no-cache-dir -r /tmp/requirements.docker.txt
 
+# Install chumpy separately (setup.py needs pip module, ensure build deps are available)
+RUN pip install --no-build-isolation --no-cache-dir "git+https://github.com/mattloper/chumpy"
+
 
 # -------------------------------------------------------------------------
 # Third-Party compiled Python deps
@@ -82,7 +85,18 @@ RUN cd / && wget https://gitlab.com/libeigen/eigen/-/archive/3.4.0/eigen-3.4.0.t
     cmake .. &&\
     make install
 
+RUN cd /
+COPY third_party/graspnetAPI /app/third_party/graspnetAPI
+RUN cd /app/third_party/graspnetAPI && pip install -e .
+
+COPY third_party/graspness_unofficial /app/third_party/graspness_unofficial
+RUN cd /app/third_party/graspness_unofficial/pointnet2 && python -m pip install --no-cache-dir -v --no-build-isolation .
+RUN cd /app/third_party/graspness_unofficial/knn && python -m pip install --no-cache-dir -v --no-build-isolation .
+
+COPY third_party/MinkowskiEngine /app/third_party/MinkowskiEngine
+RUN cd /app/third_party/MinkowskiEngine && python setup.py install --blas=openblas --force_cuda
+
 # -------------------------------------------------------------------------
 # Runtime env
 # -------------------------------------------------------------------------
-ENV PYTHONPATH=/app
+ENV PYTHONPATH=/app:/app/third_party:/app/third_party/graspness_unofficial/pointnet2:

docker/real2sim/requirements.docker.txt

@@ -20,6 +20,7 @@ scipy==1.15.3
 wandb==0.21.3
 gdown
 
+
 #__________________________________________________________________________________
 # 2) Related deps (torch-scatter is separate)
 ninja==1.11.1
@@ -61,4 +62,27 @@ Rtree
 # 8) Foundation Pose
 transformations==2024.6.1
 ruamel.yaml==0.18.6
-h5py==3.10.0
+h5py==3.10.0
+
+# 9) PromptDA
+pyliblzfse
+lightning==2.1.3
+termcolor
+tyro==0.9.2
+spaces
+trimesh 
+
+# 10) WiLoR
+pyrender
+pytorch-lightning
+scikit-image
+smplx==0.1.28
+yacs
+xtcocotools
+pandas
+hydra-submitit-launcher
+hydra-colorlog
+pyrootutils
+webdataset
+ultralytics==8.1.34
+# Note: chumpy installed separately in Dockerfile due to build isolation issues

openreal2sim/motion/README.md

@@ -0,0 +1,136 @@
+544## Motion & Grasp Preparation
+
+This directory processes human demonstrations into robot-friendly motion goals and grasp priors.  
+It consumes `outputs/<key>/scene/scene.pkl` and `simulation/scene.json` from the reconstruction pipeline and augments them with:
+
+- frame-aligned hand/keypoint observations
+- cleaned trajectories and task metadata
+- contact-based grasp points & approach directions
+- dense grasp proposals for every object
+
+This folder operates on `motion` in `scene.pkl`, and `scene.json`.
+
+The resulting assets are consumed by the simulation agents (`openreal2sim/simulation/*`) for replay, teleoperation, or policy training.
+
+---
+
+## Code Structure
+
+```
+openreal2sim/motion
+├── motion_manager.py 
+├── modules
+│   ├── hand_extraction.py
+│   ├── demo_motion_process.py 
+│   ├── grasp_point_extraction.py
+│   └── grasp_generation.py
+├── utils
+│   ├── grasp_utils.py
+│   └── visualize_scene.py
+└── README.md
+```
+
+Each module can be run independently, but `motion_manager.py` provides a stage-aware wrapper similar to `recon_agent.py`.
+
+---
+
+## Workflow
+
+### 1. Hand Extraction
+`modules/hand_extraction.py`
+
+- Detects hands with YOLO + SAM2, then lifts 3D hand pose using **WiLoR**.
+- Records per-frame keypoints, MANO global orientation, and binary masks under `scene_dict["motion"]`.
+- These signals define contact windows and guide later grasp localization.
+
+### 2. Demonstration Motion Processing
+`modules/demo_motion_process.py`
+
+- Reprojects FD-POSE / simple / hybrid trajectories into the world frame.
+- Picks the manipulated object by total displacement, classifies remaining objects as static, and decides which trajectory family (`traj_key`) best matches the demonstration.
+- Downsample the trajectory, exports a final GLB at the end pose, and writes helper metadata (`manipulated_oid`, `start_frame_idx`, `task_type`, `start/end_related`, `gripper_closed`, etc.) back into `simulation/scene.json`.
+- Whether a static object is start-related or end-related is determined by whether it is interleaved with the manipulated object in the x-y bounding box at corresponding timestep. We consider the start/end-related object as semantcially related with the manipulated object concerning the task.
+- Currently we support three type of tasks: a task is `simple_pick` if the gripper is finally closed, and there is no end-related object; a task is `simple_pick_place` if the manipulated object is final put onto the ground and gripper is opened, and if there is no end-related object; a task is `targetted_pick_place` if there exists end-related objects.
+
+
+### 3. Grasp Point & Direction Extraction
+`modules/grasp_point_extraction.py`
+
+- Aligns the manipulated mesh to the selected keyframe, renders it with PyTorch3D, and fuses depth/hand keypoints to locate the contact patch.
+- Projects the contact into object coordinates to obtain `grasp_point` and estimates a world-frame approach direction from MANO orientation.
+- Stores diagnostics (render overlays) under `outputs/<key>/simulation/debug/`.
+
+### 4. Grasp Proposal Generation
+`modules/grasp_generation.py`
+
+- Samples dense surface points per object, runs **GraspNet** to produce grasp candidates, and optionally applies NMS + score truncation.
+- Saves raw `.npy` proposals & `.ply` visualizations under `outputs/<key>/grasps/`.
+- For the manipulated object, rescores grasps using the contact point/direction hints and writes both `grasps` and `rescored_grasps` paths into `scene.json`.
+- There are three strategies to rescore grasps:
+    - `default`: use the z-align strategy to rescore grasps.
+    - `point_only`: only use the contact point to rescore grasps.
+    - `point_and_direction`: use both the contact point and the approach direction to rescore grasps.
+
+---
+
+## How to Use
+
+1. **Prerequisites**
+   - Run preprocess + reconstruction so that `outputs/<key>/scene/scene.pkl` and `simulation/scene.json` exist.
+   - Ensure third-party weights are in place:
+     - `third_party/WiLoR/pretrained_models` and `third_party/WiLoR/mano_data/MANO_RIGHT.pkl` 
+     - `third_party/Grounded-SAM-2/checkpoints`
+     - GraspNet checkpoints 
+
+2. **All-in-one pipeline**
+
+```bash
+python openreal2sim/motion/motion_manager.py
+```
+
+Optional flags mirror reconstruction:
+
+```bash
+python openreal2sim/motion/motion_manager.py \
+    --stage grasp_generation \
+    --key demo_video
+```
+
+3. **Stage-specific scripts**
+
+```bash
+python openreal2sim/motion/modules/hand_extraction.py
+
+python openreal2sim/motion/modules/demo_motion_process.py
+
+python openreal2sim/motion/modules/grasp_point_extraction.py
+
+
+python openreal2sim/motion/modules/grasp_generation.py \
+    --n_points 120000 --keep 200 --nms True
+```
+
+All scripts read keys from `config/config.yaml`; set `OPENREAL2SIM_KEYS` or pass `--key` through `motion_manager.py` to limit processing.
+
+---
+
+## Outputs
+
+Per key, after running the pipeline you should find:
+
+- `outputs/<key>/scene/scene.pkl`
+  - `motion.hand_kpts`: `[N,21,2]` MANO keypoints (image plane)
+  - `motion.hand_global_orient`: rotation matrices per frame
+  - `motion.hand_masks`: binary hand silhouettes
+- `outputs/<key>/simulation/scene.json`
+  - `manipulated_oid`, `traj_key`, `start_frame_idx`, `task_type`, `start_related`, `end_related`, `gripper_closed`
+  - `objects[oid].final_trajs`, 
+  - `objects[manipulated_oid].grasp_point`, `grasp_direction`, `grasps`, `rescored_grasps`
+- `outputs/<key>/motion/scene_end.glb`: fused background + final manipulated mesh.
+- `outputs/<key>/grasps/*.npy`: raw GraspNet candidates per object.
+- `outputs/<key>/simulation/debug/grasp_point_visualization_*.png`: sanity-check renders.
+
+These artifacts are consumed by the Isaac Lab / ManiSkill / MuJoCo simulation runners during trajectory replay and policy datasets export.
+
+---
+