Skip to content

feat(teleop): show live camera feeds beside the 3D viewer#42

Merged
nicolas-rabault merged 1 commit into
huggingface:mainfrom
cn0303:feat/teleop-camera-feed
Jun 30, 2026
Merged

feat(teleop): show live camera feeds beside the 3D viewer#42
nicolas-rabault merged 1 commit into
huggingface:mainfrom
cn0303:feat/teleop-camera-feed

Conversation

@cn0303

@cn0303 cn0303 commented Jun 19, 2026

Copy link
Copy Markdown
Contributor

What

Adds an optional camera panel to the teleoperation page so you can watch your cameras live while
you drive the arm; handy for verifying framing and catching a camera that came loose.

  • Off by default. Landing on the page never calls getUserMedia; the user opts in with a
    toggle; the same consent pattern as the calibration camera toggle (fix(calibration): don't open cameras until the user turns them on #21).
  • Mirrors the robot's configured cameras. The panel shows one live feed per camera on the
    selected robot record (e.g. wrist_cam, webcam), stacked vertically with their names.
  • Strict mirror. If the robot has no cameras configured, the panel shows nothing but a hint to
    add them during calibration — it never surfaces a device that wasn't deliberately configured.
  • A configured-but-currently-absent camera still appears (name + "preview failed" placeholder) so
    it's obvious it's expected but not detected.

Why this works during teleop

Teleoperation opens no cv2 cameras (SO101FollowerConfig has no cameras), so the browser is
free to stream them directly via getUserMedia while the arm runs. (Recording is different; there
cv2 owns the devices, so this browser-source approach is teleop-only by design.)

Implementation notes

  • CameraFeed is a small presentational component bound to a browser deviceId.
  • TeleopCameraPanel reads the selected robot's cameras via useRobots() and renders a CameraFeed
    per configured camera. A retry button remounts the feeds for a fresh getUserMedia attempt that is
    handy if a camera was unplugged and reconnected. (No backend camera enumeration: the configured
    cameras already carry the browser device_id we stream directly.)
  • Mounted in the teleop VisualizerPanel via a new rightSlot.

Testing

  • Manual: configured cameras show stacked with correct names; zero-config shows the hint only;
    unplugged-but-configured shows the placeholder.
  • npx tsc --noEmit clean; ESLint clean on the changed files (the repo's full npm run lint has
    pre-existing errors/warnings unrelated to this change); npm run build succeeds.
  • Source-only PR — frontend/dist is left to CI's build_frontend.yml to rebuild on merge.

Add an optional camera panel to the teleoperation page, off by default so
landing on the page never calls getUserMedia (same consent pattern as the
calibration camera toggle). Teleoperation opens no cv2 cameras, so the
browser can stream them directly while the arm runs.

The panel mirrors the selected robot's configured cameras: it shows a live
feed for each camera on the robot record (e.g. wrist_cam, webcam),
stacked vertically, and shows nothing when the robot has none -- teleop
never surfaces a device that was not deliberately configured.
@nicolas-rabault
nicolas-rabault force-pushed the feat/teleop-camera-feed branch from 6ee4aa3 to 38ddf12 Compare June 30, 2026 12:06
@nicolas-rabault
nicolas-rabault self-requested a review June 30, 2026 12:06

@nicolas-rabault nicolas-rabault left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice addition.

@nicolas-rabault
nicolas-rabault merged commit 8c622ca into huggingface:main Jun 30, 2026
1 check passed
@cn0303
cn0303 deleted the feat/teleop-camera-feed branch July 3, 2026 18:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants