Skip to content

feat(record): show a live camera preview during recording#66

Open
cn0303 wants to merge 1 commit into
huggingface:mainfrom
cn0303:feat/recording-camera-preview
Open

feat(record): show a live camera preview during recording#66
cn0303 wants to merge 1 commit into
huggingface:mainfrom
cn0303:feat/recording-camera-preview

Conversation

@cn0303

@cn0303 cn0303 commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

The recording page showed only a timer, so you could not see what the cameras
see while recording an episode. This adds a live feed for each configured camera
on the recording page, so you can check framing and catch a camera that drops
mid-episode. It is the feature requested in #31.

A browser getUserMedia preview like the teleop panel cannot work here: during
recording cv2 owns each camera exclusively (that is why the camera config
releases its browser streams on record start), so the browser cannot open the
same device. The feed has to come from the backend, which already has the frames.

image

How it works

  • record.py: a module global current_robot is set to the connected
    SO101Follower after robot.connect() and cleared in the recording finally,
    before the cameras are torn down, so the feed never blocks camera release.
    handle_recording_status reports cameras (the configured camera names) while
    active. A generator camera_feed_frames(cam_key) yields multipart MJPEG JPEG
    frames.
  • server.py: GET /camera-feed/{cam_key} returns a StreamingResponse with
    media_type="multipart/x-mixed-replace; boundary=frame", guarded by
    recording_active.
  • Recording.tsx: renders one <img> MJPEG feed per configured camera.

Design points, grounded against lerobot 0.6.0:

  • Only configured cameras are shown. The backend reports exactly the recording
    config's camera names and the frontend renders feeds for only those.
  • Zero device contention. Frames are read with OpenCVCamera.read_latest(), a
    non-blocking peek of the lock-protected latest_frame buffer the record loop
    already fills. This deliberately does not use async_read(), which consumes
    the new-frame event and could disturb record-loop timing.
  • Color. read_latest() returns RGB, converted to BGR before
    cv2.imencode(".jpg", ...).
  • Cadence. Capped at about 15 fps, below the record loop, so the tap stays a
    passive observer.

Layout

The windows appear from the preparing phase as empty slots (one per configured
camera) and fill with the live feed once recording starts, so the layout is
stable from the moment the session begins.

The page stays within one screen with no scrolling. The header, timer, progress
bar, and controls take their natural height, and the camera area absorbs the rest.
A ResizeObserver measures that leftover space, and the windows are sized to the
largest that fit: for the given camera count it picks the column count that
maximizes video area, and sizes each window to the camera's aspect ratio (read
from its configured resolution, default 4:3) so there is no letterboxing. Videos
scale up on larger monitors and down on smaller ones.

Testing

Hardware-confirmed on Windows 11 (SO-101 with two USB cameras, lerobot 0.6.0):
both feeds render side by side during recording, the placeholder slots show during
preparing, and the whole page fits one screen at different window sizes.

ruff and pre-commit (mypy, bandit, typos) green. Backend imports cleanly and the
record tests pass (9). ESLint, tsc --noEmit, and npm run build pass. Source
only; frontend/dist is left to CI to rebuild on merge.

Notes

<img src> cannot send custom headers, which is fine on localhost; behind a
tunnel the feed URL would need to be publicly reachable. Each open feed holds one
threadpool thread for the stream's duration (sync generator plus a sleep), which
is fine for a handful of cameras.

Until now the recording page showed only a timer. Show each configured camera's
live feed on the page so the user can verify framing and catch a camera that
drops mid-episode.

The session exposes its connected SO101Follower as a module global once the
robot connects, cleared on teardown. A new GET /camera-feed/{name} streams that
camera's frames as multipart MJPEG, read via the non-blocking read_latest() peek
so it shares the record loop's lock-protected frame buffer with zero device
contention (never async_read, which would consume the new-frame event). The
recording status reports the configured camera names and the page renders a feed
for exactly those, so only cameras the user added are shown.

The windows appear from the preparing phase as empty slots, one per configured
camera, and fill with the live feed once recording starts. The layout keeps the
whole page within one screen: a ResizeObserver measures the space left after the
timer and controls, and the windows are sized to the largest that fit, matched to
each camera's aspect ratio and the best column count for the number of cameras.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant