VLA-Synth-Data

A training-free synthetic-data pipeline for Vision-Language-Action (VLA) model training. Composes MuJoCo physics, classical motion planning, and VLM-based annotation — with zero model fine-tuning — to generate annotated robot trajectory datasets in LeRobot v3.0 format.

⚠️ ALPHA — no sim-to-real validation yet. This repo is a working end-to-end data-factory prototype, not a finished dataset or a validated training pipeline. As of the last tagged state: 200 real-pixel trajectories have been generated on one MacBook, 795 unit tests pass, and zero VLA policies have been trained on this data. Use at your own risk. See Current state and Known limitations below for the full picture.

Why this exists

Most VLA training datasets come from expensive teleoperation rigs or from large closed pipelines at industrial labs. The ones that are public (Open-X-Embodiment, DROID, BridgeData) are slow to collect and hard to extend. VLA-Synth-Data generates annotated trajectories without human teleoperation or model training, leaning on three core pieces:

1. Physics-signal subtask segmentation — contact-force discontinuities and velocity minima drive a training-free subtask segmenter.
2. VLM-based per-segment language annotation — a vision-language model produces captions conditioned on the segment's keyframes, with three prompt styles (simple, compositional, referential) and a grounded per-keyframe schema including referring expressions, RT-H-style motion phrases, and counterfactual reasoning.
3. Native LeRobot v3.0 output — the built dataset is immediately compatible with the lerobot ecosystem, not a custom format.

The sim side is intentionally commodity: MuJoCo 3.x + dm_control + classical IK + waypoint-based motion planner. Existing OSS robot assets (the MuJoCo menagerie for Franka / UR5e / Sawyer / xArm7) are consumed rather than rebuilt.

Install

Requirements: Python 3.10 or 3.11 (tested on 3.11), macOS or Linux, and on Linux a GL backend (libEGL or libOSMesa) if you want real rendered images. macOS uses native CGL and needs no extra GL install.

git clone https://github.com/henrycashe26/vla-synth-data.git
cd vla-synth-data

python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip setuptools wheel
pip install -e .

# One-time: clone mujoco_menagerie into configs/robots/ (shallow clone of HEAD)
bash scripts/install_menagerie.sh

Optional extras (install only what you need):

pip install -e ".[photorealistic]"   # albumentations-based pixel augmentation
pip install -e ".[tracking]"         # wandb experiment tracking
pip install -e ".[publishing]"       # huggingface_hub dataset upload
pip install -e ".[pi0]"              # openpi-client for the pi0 inference backend

Quickstart

Generate ten trajectories on the real Franka, build a tiny LeRobot v3.0 dataset from them, and play back a couple of episodes:

# 1. A single trajectory as a smoke test
python scripts/smoke_test_stage2.py

# 2. Ten more
python scripts/generate_diagnostic_dataset.py --n 10 --continue-on-error

# 3. Quality report + physics rejection gates
python scripts/diagnostic_report.py

# 4. Replay-eval retention under randomization (should be ≥70%)
python scripts/replay_eval.py --n-perturbations 5

# 5. Build as LeRobot v3.0 (add --include-images for real video tracks)
python scripts/build_dataset_from_raw.py --input data/raw --output data/final

# 6. Sanity playback
python scripts/sanity_check_dataset.py --dataset data/final --n 5

No external API key is required for any of the above — the task generation and language annotation stages are opt-in.

Current state

What actually ships in this repo, today:

Passing tests	795 / 795 (full suite green)
Real-pixel trajectories generated	200 (on one M2 MacBook Pro)
State-only trajectories generated	321 (on a headless Linux sandbox, images blank)
Replay-eval retention under randomization	90.6 % (target ≥70 %)
Readiness score	0.80 / 1.00
Built LeRobot v3.0 datasets	1 (local only, not published)
Committed dataset baselines	4 snapshots under docs/diagnostics/
VLA policies trained on this data	0
Sim-to-real transfer results	0
Real-robot deployments	0

See the full auto-generated dataset card at docs/diagnostics/dataset_card_v0.md for the honest scale + quality-gate numbers in one place.

Architecture overview

Six pipeline stages, each independently replaceable via a plugin registry:

Stage 1  task_generator.py      Pydantic task specs (opt. via external LLM)
Stage 2  trajectory_generator.py MuJoCo physics + classical IK + waypoints
Stage 3  annotator.py           Success detection + segmentation + VLM captions
Stage 4  randomizer.py          Domain randomization + photorealistic augmentation
Stage 5  dataset_builder.py     LeRobot v3.0 assembly (parquet/jsonl + MP4)
Stage 6  validator.py           Data quality audit + eval-script harness

Cross-cutting infrastructure:

• base.py + registry.py — abstract base classes and plugin registry
• hydra_config.py — OmegaConf-based config composition
• assets.py — manifest-driven asset manager (robots, objects, scenes)
• robots.py — RobotConfig dataclass + 4 built-ins (Franka, UR5e, Sawyer, xArm7)
• scenes.py — procedural scene generator with Poisson-disk layout
• parallel.py — multiprocessing rollout pool
• replay_eval.py — sim-to-sim robustness harness
• lerobot_reader.py — LeRobot v3.0 → pipeline-dict reader for hybrid co-training
• hooks.py + callbacks.py — Lightning-style observability (W&B / versioning / HF Hub)

A fuller walkthrough of each module with LOC counts and responsibility lines lives in docs/ARCHITECTURE.md.

Known limitations

Honest, not exhaustive. Open an issue if you find more.

• No sim-to-real validation. Nobody has trained a VLA on this data and run it on a real robot (or a real-robot-correlated proxy like SimplerEnv). Every claim about "usable for VLA training" in this README is aspirational until a trained checkpoint ships. Do not cite this project in a paper as a source of validated training data.
• Pickle-based checkpoints — partial mitigation. scripts/run_pipeline.py delegates to pipeline.safe_checkpoint.SafeUnpickler, which whitelists numpy / stdlib / pathlib classes and rejects everything else (downgrading severity from HIGH to MEDIUM). Residual risk: restricted unpickling is not a perfect sandbox; __reduce__ paths on whitelisted classes still have edge cases. Long-term fix is to migrate the format to JSON + safetensors. See SECURITY.md for details.
• Only pick-and-place has been end-to-end tested. The _subtasks_to_waypoints dispatcher has push / stack / sort templates, but the diagnostic generator only emits pick-place specs.
• ~18 % IK failure rate on the randomized diagnostic workspace. The rejection gates catch these post-hoc; a pre-filter on reachable cube positions is future work.
• Language annotation has only been run against mocked responses. The annotation schemas (AnnotationResult, ReferentialPhrasings, GroundedKeyframeAnnotation) are fully tested but never fed a real rendered frame end-to-end.
• No GPU-parallel rollouts. An MJX backend is deferred. Comparable pipelines are 10-100× faster per trajectory.
• RoboCasa scene consumption is not wired into MuJoCoEnvironment._build_scene. The RoboCasaAdapter + 10 manifest entries exist but the scene loader refactor is not implemented.
• Pi0-FAST backend never smoke-tested on a real rollout. The client exists in trajectory_generator.py but all tests use the classical planner.
• Images are blank on Linux hosts without libEGL / libOSMesa. The warning is loud, but you still need to install a GL backend. macOS works out of the box via native CGL.

Documentation

• docs/ARCHITECTURE.md — detailed module walkthrough, plugin-registry flow, callback lifecycle, config layering, and key invariants.
• docs/API.md — public API reference for the ~15 stable classes with stability tiers.
• docs/diagnostics/ — committed baseline snapshots for the readiness, replay-eval, and dataset card numbers.
• CONTRIBUTING.md — how to file issues, submit PRs, and run the test suite.
• SECURITY.md — security policy and known risks.
• CHANGELOG.md — version history.

Running the tests

Install the dev extra (pytest + pytest-cov + ruff), then run:

pip install -e ".[dev]"
pytest tests/ -q --no-cov --ignore=tests/test_e2e_integration.py

Expected: 795 passed in ~2 minutes on a laptop. The test_e2e_integration.py suite is gated on the MENAGERIE_PATH env var and runs full real-physics rollouts — opt in with:

export MENAGERIE_PATH=$(pwd)/configs/robots/mujoco_menagerie
pytest tests/test_e2e_integration.py -q --no-cov

Contributing

PRs welcome. See CONTRIBUTING.md for dev setup, commit style, and testing expectations. A good first contribution is anything under "Known limitations" above — each item is scoped enough to land as a single PR.

Citation

If this project is useful for your work, please cite via the CITATION.cff file (GitHub's "Cite this repository" button surfaces it automatically).

License

Apache License 2.0. See LICENSE for the full text and NOTICE for third-party attributions.