feat(record): show a live camera preview during recording#66
Open
cn0303 wants to merge 1 commit into
Open
Conversation
Until now the recording page showed only a timer. Show each configured camera's
live feed on the page so the user can verify framing and catch a camera that
drops mid-episode.
The session exposes its connected SO101Follower as a module global once the
robot connects, cleared on teardown. A new GET /camera-feed/{name} streams that
camera's frames as multipart MJPEG, read via the non-blocking read_latest() peek
so it shares the record loop's lock-protected frame buffer with zero device
contention (never async_read, which would consume the new-frame event). The
recording status reports the configured camera names and the page renders a feed
for exactly those, so only cameras the user added are shown.
The windows appear from the preparing phase as empty slots, one per configured
camera, and fill with the live feed once recording starts. The layout keeps the
whole page within one screen: a ResizeObserver measures the space left after the
timer and controls, and the windows are sized to the largest that fit, matched to
each camera's aspect ratio and the best column count for the number of cameras.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
The recording page showed only a timer, so you could not see what the cameras
see while recording an episode. This adds a live feed for each configured camera
on the recording page, so you can check framing and catch a camera that drops
mid-episode. It is the feature requested in #31.
A browser
getUserMediapreview like the teleop panel cannot work here: duringrecording cv2 owns each camera exclusively (that is why the camera config
releases its browser streams on record start), so the browser cannot open the
same device. The feed has to come from the backend, which already has the frames.
How it works
record.py: a module globalcurrent_robotis set to the connectedSO101Followerafterrobot.connect()and cleared in the recordingfinally,before the cameras are torn down, so the feed never blocks camera release.
handle_recording_statusreportscameras(the configured camera names) whileactive. A generator
camera_feed_frames(cam_key)yields multipart MJPEG JPEGframes.
server.py:GET /camera-feed/{cam_key}returns a StreamingResponse withmedia_type="multipart/x-mixed-replace; boundary=frame", guarded byrecording_active.Recording.tsx: renders one<img>MJPEG feed per configured camera.Design points, grounded against lerobot 0.6.0:
config's camera names and the frontend renders feeds for only those.
OpenCVCamera.read_latest(), anon-blocking peek of the lock-protected
latest_framebuffer the record loopalready fills. This deliberately does not use
async_read(), which consumesthe new-frame event and could disturb record-loop timing.
read_latest()returns RGB, converted to BGR beforecv2.imencode(".jpg", ...).passive observer.
Layout
The windows appear from the preparing phase as empty slots (one per configured
camera) and fill with the live feed once recording starts, so the layout is
stable from the moment the session begins.
The page stays within one screen with no scrolling. The header, timer, progress
bar, and controls take their natural height, and the camera area absorbs the rest.
A ResizeObserver measures that leftover space, and the windows are sized to the
largest that fit: for the given camera count it picks the column count that
maximizes video area, and sizes each window to the camera's aspect ratio (read
from its configured resolution, default 4:3) so there is no letterboxing. Videos
scale up on larger monitors and down on smaller ones.
Testing
Hardware-confirmed on Windows 11 (SO-101 with two USB cameras, lerobot 0.6.0):
both feeds render side by side during recording, the placeholder slots show during
preparing, and the whole page fits one screen at different window sizes.
ruff and pre-commit (mypy, bandit, typos) green. Backend imports cleanly and the
record tests pass (9). ESLint,
tsc --noEmit, andnpm run buildpass. Sourceonly;
frontend/distis left to CI to rebuild on merge.Notes
<img src>cannot send custom headers, which is fine on localhost; behind atunnel the feed URL would need to be publicly reachable. Each open feed holds one
threadpool thread for the stream's duration (sync generator plus a sleep), which
is fine for a handful of cameras.