VLA Architecture Investigation: SmolVLA Fine-tuning on Custom Robots

An investigation into the architectural constraints of pretrained Vision-Language-Action (VLA) models when applied to custom robot morphologies. This project documents the challenges encountered when attempting to fine-tune SmolVLA on a humanoid manipulation task in MuJoCo simulation.

Full technical writeup coming soon!

Key Findings

• Hardcoded Sequence Lengths: Pretrained VLAs have fixed attention mask dimensions determined during pretraining
• Normalization Constraints: Frozen normalization statistics require exact input dimensions matching the pretraining setup
• Camera Configuration Dependencies: Models trained on multi-camera setups (e.g., ALOHA) may not transfer to single-camera configurations
• Tokenization Sensitivity: Total sequence length depends on image patches, language tokens, state encoding, and action dimensions

Environment Setup

Prerequisites

• Python 3.12
• mise for version management
• GPU: NVIDIA, Intel Arc or AMD (auto-detected)

Installation

# Install mise and Python 3.12
make setup-mise

# Install all dependencies (auto-detects GPU)
make install

# Setup environment variables
make setup-env

# Verify installation
make verify-setup

Data Collection

The project includes a fully functional data collection system for MuJoCo simulation:

make collect-data

Features:

• Terminal-based keyboard teleoperation
• Contact-aware grasping
• Randomized object spawning
• 224x224 RGB observations + 15-dim proprioceptive state
• Natural language task descriptions

Controls:

• W/A/S/D/Q/E: Arm movement
• G/H: Gripper close/open
• J/K: Torso lift up/down (very minimal impact so basically useless with the current implementation)
• X: Save episode
• Z: Reset episode

Multi-GPU Support

This project automatically detects and configures for multiple GPU vendors:

# Display GPU information
make gpu-info

# Supported GPUs:
# - NVIDIA (CUDA)
# - Intel Arc (XPU)
# - AMD (ROCm) - I haven't tested this yet as I am yet to get an AMD GPU, soon though!

Technical Details

Robot Configuration

• Robot: ROBOTIS FFW-BG2 humanoid torso (source)
• Control: Single left arm (7-DOF) + parallel gripper (1-DOF)
• Observations: RGB camera (224x224) + joint state (15-dim)
• Actions: 8-dimensional continuous control

Data Format

{
    "observation.image": [3, 224, 224],    # RGB image
    "observation.state": [15],              # qpos(7) + qvel(7) + gripper(1)
    "action": [8],                          # 7 arm joints + gripper
    "task": str                             # Natural language instruction
}

Issues Encountered

1. Dimension Mismatch (Solved): Normalization layers expected specific dimensions
2. Sequence Length Mismatch (Unsolved): Attention mechanism has hardcoded 145-token expectation, got 163 tokens
3. Root Cause: Architectural constraints from pretraining on different robot configuration

Lessons Learned

For Practitioners

1. Check pretrained model architecture first before collecting data
2. Match the pretraining setup exactly or train from scratch
3. Consider policy alternatives (ACT, Diffusion Policy) for custom robots
4. Document failures - they're educational for the community

For Researchers

This investigation highlights the need for more architecturally flexible VLA models that can:

• Adapt to variable sequence lengths
• Handle different camera configurations
• Support diverse robot morphologies without retraining

Alternative Approaches

For custom robot setups, consider:

1. Action Chunking Transformers (ACT) - Flexible architecture, works with single cameras
2. Diffusion Policy - Generative approach, adaptable to various configurations
3. Behavior Cloning - Simple and effective for imitation learning
4. Training SmolVLA from scratch - Requires massive compute and data

Citations

@misc{shukor2025smolvlavisionlanguageactionmodelaffordable,
      title={SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Robotics},
      author={Mustafa Shukor and Dana Aubakirova and Francesco Capuano and Pepijn Kooijmans and Steven Palma and Adil Zouitine and Michel Aractingi and Caroline Pascal and Martino Russi and Andres Marafioti and Simon Alibert and Matthieu Cord and Thomas Wolf and Remi Cadene},
      year={2025},
      eprint={2506.01844},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2506.01844},
}

Resources

• Paper: SmolVLA (arXiv:2506.01844)
• Pretrained Model: lerobot/smolvla_base
• Robot Assets: ROBOTIS MuJoCo Menagerie (Apache 2.0)
• Framework: LeRobot

Development

# Format code
make format

# Run linting
make lint

# Run tests
make test

# Clean temporary files
make clean

License

MIT License - See LICENSE file for details

Robot assets from ROBOTIS are licensed under Apache 2.0.

Author

William Obino

• Email: obinowilliam@staka.cc
• Location: Nairobi, Kenya
• GitHub: @andomeder