feat(remote-env): SBS status display and client-driven episode timing - #54
Merged
Conversation
…ming Two improvements for the desktop RobotServer's --sbs approval flow: 1. PushStatusInfo RPC: training client can push key-value status (e.g. TOPReward score/delta) to the server, which is shown in the SBS prompt before each chunk. vlm_planner_client pushes top_reward_score and top_reward_delta to the matching RemoteEnv after each computation. 2. SetEpisodeTiming RPC: RemoteEnv pushes episode_duration_s / episode_cooldown_s from the Beaker-side yaml at connect time, so the desktop server no longer has to be restarted to change timing. Also replace the file-based first-chunk approval (touch /tmp/rlinf_approve_chunk) with an Enter-key prompt read from /dev/tty, matching the existing per-chunk SBS prompt UX. Signed-off-by: thomas0829 <ycl0829@uw.edu>
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces the ability to push episode timing and status information, such as TOPReward scores, from the training client to the robot server via new gRPC endpoints. This allows the server's step-by-step (SBS) prompt to display real-time metrics and timing without requiring a server restart. Additionally, the manual chunk approval process was simplified to use keyboard input instead of file creation. Review feedback focused on ensuring thread safety within the RobotServer by protecting shared state and dictionary iterations with the existing environment lock to prevent race conditions between gRPC handlers and the main execution loop.
Comment on lines
+612
to
+633
| def SetEpisodeTiming(self, request, context): | ||
| self._touch() | ||
| duration_s = float(request.episode_duration_s) | ||
| cooldown_s = float(request.episode_cooldown_s) | ||
| changes = [] | ||
| if duration_s > 0: | ||
| old = float(getattr(self._env, "_episode_duration_s", 0.0)) | ||
| self._env._episode_duration_s = duration_s | ||
| changes.append(f"episode_duration_s: {old:.1f} -> {duration_s:.1f}") | ||
| if cooldown_s >= 0: | ||
| old = self._episode_cooldown_s | ||
| self._episode_cooldown_s = cooldown_s | ||
| # Keep env-side value in sync for any code that reads it directly. | ||
| if hasattr(self._env, "_episode_cooldown_s"): | ||
| self._env._episode_cooldown_s = cooldown_s | ||
| changes.append(f"episode_cooldown_s: {old:.1f} -> {cooldown_s:.1f}") | ||
| if changes: | ||
| logger.info( | ||
| "[RobotServer] Episode timing updated from client: %s", | ||
| ", ".join(changes), | ||
| ) | ||
| return robot_env_pb2.Empty() |
There was a problem hiding this comment.
Modifying self._env attributes and self._episode_cooldown_s should be protected by self._env_lock. This ensures that other threads (like the episode_timeout watchdog or the SBS status formatter) see a consistent state and avoids potential race conditions.
def SetEpisodeTiming(self, request, context):
self._touch()
duration_s = float(request.episode_duration_s)
cooldown_s = float(request.episode_cooldown_s)
with self._env_lock:
changes = []
if duration_s > 0:
old = float(getattr(self._env, "_episode_duration_s", 0.0))
self._env._episode_duration_s = duration_s
changes.append(f"episode_duration_s: {old:.1f} -> {duration_s:.1f}")
if cooldown_s >= 0:
old = self._episode_cooldown_s
self._episode_cooldown_s = cooldown_s
# Keep env-side value in sync for any code that reads it directly.
if hasattr(self._env, "_episode_cooldown_s"):
self._env._episode_cooldown_s = cooldown_s
changes.append(f"episode_cooldown_s: {old:.1f} -> {cooldown_s:.1f}")
if changes:
logger.info(
"[RobotServer] Episode timing updated from client: %s",
", ".join(changes),
)
return robot_env_pb2.Empty()
Comment on lines
+635
to
+640
| def PushStatusInfo(self, request, context): | ||
| self._touch() | ||
| self._client_status_values = {k: float(v) for k, v in request.values.items()} | ||
| self._client_status_text = str(request.text or "") | ||
| self._client_status_updated_at = time.monotonic() | ||
| return robot_env_pb2.Empty() |
There was a problem hiding this comment.
Updating _client_status_values must be protected by self._env_lock. Since _format_sbs_status iterates over this dictionary, a concurrent update from this RPC handler could cause a RuntimeError: dictionary changed size during iteration.
Suggested change
| def PushStatusInfo(self, request, context): | |
| self._touch() | |
| self._client_status_values = {k: float(v) for k, v in request.values.items()} | |
| self._client_status_text = str(request.text or "") | |
| self._client_status_updated_at = time.monotonic() | |
| return robot_env_pb2.Empty() | |
| def PushStatusInfo(self, request, context): | |
| self._touch() | |
| with self._env_lock: | |
| self._client_status_values = {k: float(v) for k, v in request.values.items()} | |
| self._client_status_text = str(request.text or "") | |
| self._client_status_updated_at = time.monotonic() | |
| return robot_env_pb2.Empty() |
Comment on lines
+543
to
+556
| try: | ||
| rewards_np = ( | ||
| chunk_rewards.detach().cpu().numpy() | ||
| if hasattr(chunk_rewards, "detach") | ||
| else np.asarray(chunk_rewards) | ||
| ) | ||
| per_step = rewards_np[0].astype(float) | ||
| self._last_chunk_reward_sum = float(per_step.sum()) | ||
| self._last_chunk_reward_max = ( | ||
| float(per_step.max()) if per_step.size else 0.0 | ||
| ) | ||
| self._episode_return += self._last_chunk_reward_sum | ||
| except Exception: | ||
| pass |
There was a problem hiding this comment.
Updating _episode_return and other running stats should be done under self._env_lock to ensure consistency with the status display and other handlers that read these values.
Suggested change
| try: | |
| rewards_np = ( | |
| chunk_rewards.detach().cpu().numpy() | |
| if hasattr(chunk_rewards, "detach") | |
| else np.asarray(chunk_rewards) | |
| ) | |
| per_step = rewards_np[0].astype(float) | |
| self._last_chunk_reward_sum = float(per_step.sum()) | |
| self._last_chunk_reward_max = ( | |
| float(per_step.max()) if per_step.size else 0.0 | |
| ) | |
| self._episode_return += self._last_chunk_reward_sum | |
| except Exception: | |
| pass | |
| # Update running stats for the SBS prompt. | |
| with self._env_lock: | |
| try: | |
| rewards_np = ( | |
| chunk_rewards.detach().cpu().numpy() | |
| if hasattr(chunk_rewards, "detach") | |
| else np.asarray(chunk_rewards) | |
| ) | |
| per_step = rewards_np[0].astype(float) | |
| self._last_chunk_reward_sum = float(per_step.sum()) | |
| self._last_chunk_reward_max = ( | |
| float(per_step.max()) if per_step.size else 0.0 | |
| ) | |
| self._episode_return += self._last_chunk_reward_sum | |
| except Exception: | |
| pass |
There was a problem hiding this comment.
Pull request overview
This PR improves the remote robot training/operator workflow by letting the training client push episode timing and status/metrics to the desktop RobotServer, and by streamlining first-chunk approval in SBS mode.
Changes:
- Add gRPC RPCs/messages for client-driven episode timing (
SetEpisodeTiming) and status key-values (PushStatusInfo). - Push timing at
RemoteEnvconnect time, and push TOPReward score/delta from the VLM planner to the matching env for SBS display. - Replace file-based first-chunk approval with an Enter-key prompt via
/dev/tty(withinput()fallback).
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| rlinf/envs/remote/proto/robot_env.proto | Adds SetEpisodeTiming + PushStatusInfo RPCs and request messages. |
| rlinf/envs/remote/proto/robot_env_pb2.py | Regenerated protobuf bindings for new messages/RPCs. |
| rlinf/envs/remote/proto/robot_env_pb2_grpc.py | Regenerated gRPC stub/servicer bindings for new RPCs. |
| rlinf/envs/remote/robot_server.py | Implements new RPC handlers and SBS status formatting; switches first-chunk approval to Enter. |
| rlinf/envs/remote/remote_env.py | Pushes episode timing on connect; adds push_status_info() client helper. |
| rlinf/workers/env/vlm_planner_client.py | Pushes TOPReward score/delta into env status; threads env through TOPReward resolution paths. |
| rlinf/workers/env/env_worker.py | Passes env_list/slot_id through to planner client for status push routing. |
| rlinf/config.py | Mirrors timing fields into env.train.* so RemoteEnv can read and push them. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| return | ||
| score_t = ray.get(pending.score_ref) | ||
| self.apply_resolved_top_reward(pending, score_t) | ||
| env = env_list[slot_id] if env_list is not None else None |
There was a problem hiding this comment.
In resolve_pending_top_reward_sync, env_list[slot_id] is indexed without any bounds check. If slot_id is negative or out of range (e.g., due to stale pending state or a caller bug), this will raise IndexError and prevent clearing _pending_top_rewards for that slot. The async path already guards with 0 <= slot_id < len(env_list); the sync path should mirror that behavior (and treat out-of-range as env=None).
Suggested change
| env = env_list[slot_id] if env_list is not None else None | |
| if env_list is not None and 0 <= slot_id < len(env_list): | |
| env = env_list[slot_id] | |
| else: | |
| env = None |
Comment on lines
+530
to
+541
| def _push_top_reward_to_env(self, env: Any, score_t: float, reward: float) -> None: | ||
| push = getattr(env, "push_status_info", None) | ||
| if callable(push): | ||
| try: | ||
| push( | ||
| values={ | ||
| "top_reward_score": float(score_t), | ||
| "top_reward_delta": float(reward), | ||
| } | ||
| ) | ||
| except Exception: | ||
| pass |
There was a problem hiding this comment.
_push_top_reward_to_env swallows all exceptions silently. Since push_status_info already handles grpc.RpcError, this broad except Exception: pass can hide real programming errors (e.g., unexpected value types) and make TOPReward debugging much harder. Consider catching only expected exceptions and/or logging once at debug/warning level (optionally rate-limited).
Comment on lines
+612
to
+633
| def SetEpisodeTiming(self, request, context): | ||
| self._touch() | ||
| duration_s = float(request.episode_duration_s) | ||
| cooldown_s = float(request.episode_cooldown_s) | ||
| changes = [] | ||
| if duration_s > 0: | ||
| old = float(getattr(self._env, "_episode_duration_s", 0.0)) | ||
| self._env._episode_duration_s = duration_s | ||
| changes.append(f"episode_duration_s: {old:.1f} -> {duration_s:.1f}") | ||
| if cooldown_s >= 0: | ||
| old = self._episode_cooldown_s | ||
| self._episode_cooldown_s = cooldown_s | ||
| # Keep env-side value in sync for any code that reads it directly. | ||
| if hasattr(self._env, "_episode_cooldown_s"): | ||
| self._env._episode_cooldown_s = cooldown_s | ||
| changes.append(f"episode_cooldown_s: {old:.1f} -> {cooldown_s:.1f}") | ||
| if changes: | ||
| logger.info( | ||
| "[RobotServer] Episode timing updated from client: %s", | ||
| ", ".join(changes), | ||
| ) | ||
| return robot_env_pb2.Empty() |
There was a problem hiding this comment.
SetEpisodeTiming mutates server/env timing fields without acquiring _env_lock. This breaks the stated locking contract for gRPC handlers and can race with ChunkStep/episode_timeout/timer thread reads, leading to inconsistent timing behavior mid-episode. Update timing under _env_lock (and ideally keep the servicer’s _episode_cooldown_s + env fields in sync in the same critical section).
Comment on lines
+542
to
+557
| # Update running stats for the SBS prompt. | ||
| try: | ||
| rewards_np = ( | ||
| chunk_rewards.detach().cpu().numpy() | ||
| if hasattr(chunk_rewards, "detach") | ||
| else np.asarray(chunk_rewards) | ||
| ) | ||
| per_step = rewards_np[0].astype(float) | ||
| self._last_chunk_reward_sum = float(per_step.sum()) | ||
| self._last_chunk_reward_max = ( | ||
| float(per_step.max()) if per_step.size else 0.0 | ||
| ) | ||
| self._episode_return += self._last_chunk_reward_sum | ||
| except Exception: | ||
| pass | ||
|
|
There was a problem hiding this comment.
This block computes _episode_return / _last_chunk_reward_sum / _last_chunk_reward_max, but those values are never included in _format_sbs_status() (and appear otherwise unused), so the work is currently dead code and failures are silently ignored. Either surface these stats in the SBS status line (if intended) or remove the tracking to reduce complexity and avoid masking issues with the try/except.
Suggested change
| # Update running stats for the SBS prompt. | |
| try: | |
| rewards_np = ( | |
| chunk_rewards.detach().cpu().numpy() | |
| if hasattr(chunk_rewards, "detach") | |
| else np.asarray(chunk_rewards) | |
| ) | |
| per_step = rewards_np[0].astype(float) | |
| self._last_chunk_reward_sum = float(per_step.sum()) | |
| self._last_chunk_reward_max = ( | |
| float(per_step.max()) if per_step.size else 0.0 | |
| ) | |
| self._episode_return += self._last_chunk_reward_sum | |
| except Exception: | |
| pass |
In SBS mode the robot server was printing three separate noisy blocks for
every chunk:
- a bare "[RobotServer][ChunkStep N] task=..." header line,
- a per-step action dump from the verbose logger (30 rows per chunk),
- a zero-filled "[ChunkStep] Done. rewards=[0.0, 0.0, ...]" array
(YAM env reward is always 0; dense reward lives in Beaker-side
TOPReward).
Skip all three in SBS mode. Instead fold the task/subtask line into the
SBS prompt block so the operator sees a compact 3-line prompt:
[SBS] chunk#7 | ep_time=1:04/20:00 (left=18:55) | top_reward_score=...
[SBS] task="..." | subtask="..."
[SBS] Press Enter to execute this chunk...
Non-SBS verbose mode is unchanged.
Signed-off-by: thomas0829 <ycl0829@uw.edu>
Add score/delta logging to apply_resolved_top_reward so the async TOPReward path emits the same log line as the sync path. Remove the unused use_reward_model field and stale comments from all yam YAML configs. Signed-off-by: Carl <sc256@uw.edu> Made-with: Cursor
…github.com/chinsengi/RLinf into feat/remote-env-sbs-status-and-timing-push
YAMEnv always returns zero reward (dense reward is computed Beaker-side via TOPReward). Remove the server-side episode_return / chunk reward sum / max tracking that only ever accumulated zeros. Signed-off-by: thomas0829 <ycl0829@uw.edu>
Only the zero-reward dump should be suppressed in SBS mode, not the per-step action prints or the chunk task context header. Signed-off-by: thomas0829 <ycl0829@uw.edu>
…r scoring TOPReward was receiving only 1 frame per chunk (the last step), giving the VLM very sparse visual context (~0.33 fps). This caused unreliable scoring especially after subtask switches when the frame buffer resets. Now YAMEnv captures a camera frame every `reward_frame_interval` steps during chunk_step (default 5, yielding 6 frames per 30-step chunk = 2 fps). The server JPEG-compresses these frames and sends them in the ChunkStepResponse. RemoteEnv decodes them and vlm_planner_client prepends them to _episode_frames before the final-step image. Changes: - proto: add reward_frames to ChunkStepResponse - yam_env: capture intermediate images in chunk_step via get_obs() - robot_server: compress and send reward frames; --reward-frame-interval flag - remote_env: decode and store _last_reward_frames - vlm_planner_client: extend _episode_frames with reward frames Signed-off-by: thomas0829 <ycl0829@uw.edu>
… mode Hides the per-step action arrays and first-chunk action dump when --sbs --no-action are both set. Only effective with --sbs. Signed-off-by: thomas0829 <ycl0829@uw.edu>
…tart_robot_server.sh Signed-off-by: thomas0829 <ycl0829@uw.edu>
set -u causes an unbound variable error when these are not set. Signed-off-by: thomas0829 <ycl0829@uw.edu>
Replace blocking tty.readline() calls in SBS/first-chunk approval with select.select-based polling that checks the stop_event every 0.5s. When Ctrl+C is pressed, gRPC handlers return immediately with an idle response, allowing server.stop() to complete quickly and giving env.return_to_home() enough time to move the robot arms home before the shell script's kill timeout. Also increase the shell timeout from 8s to 15s for safety margin. Signed-off-by: thomas0829 <ycl0829@uw.edu>
…aseline resets Chunk#1 had no top_reward line because no async reward was resolved yet. Now shows delta=+0.0000 with score=-- (pending). Also detects subtask changes and appends "(baseline reset)" so the delta discontinuity around subtask_interval boundaries is clearly explained. Signed-off-by: thomas0829 <ycl0829@uw.edu>
…d action_chunk_smoothing Signed-off-by: thomas0829 <ycl0829@uw.edu>
The subtask is already set before the first chunk arrives, so comparing against the base task incorrectly triggered "(baseline reset)" on chunk#1. Now only checks for subtask changes after chunk#1. Signed-off-by: thomas0829 <ycl0829@uw.edu>
Move the score/delta/task display to after chunk_step() completes so the user sees results of the just-executed chunk rather than a pre-execution summary. The Enter prompt now only shows the chunk number. Signed-off-by: thomas0829 <ycl0829@uw.edu>
When restarting, ports 1234/1235 may still be occupied by old follower processes. Now ensure_port_is_free() uses lsof to find and SIGTERM the stale process instead of immediately raising an error. Signed-off-by: thomas0829 <ycl0829@uw.edu>
The second Ctrl+C during shutdown caused cleanup() to return early via the CLEANING_UP guard, leaving the Python server process orphaned while it was still running return_to_home. Now cleanup() ignores SIGINT so the user cannot interrupt the return-home sequence. Also moved the force-kill (kill -9) for the server into the grace-period loop so it only fires if the server is actually stuck. Signed-off-by: thomas0829 <ycl0829@uw.edu>
…r, harden Ctrl+C - Revert SBS status display back to before chunk execution. - Send reward frames as lossless PNG instead of JPEG so TOPReward sees full-quality images. Include the final-step obs image in reward_frames so the client never feeds JPEG-compressed images to the VLM. - Save reward frames to /tmp/rlinf_sbs_reward_frames/ when SBS is active for local preview. - Add early return in ChunkStep when stop_event is set so no SBS prompt is printed during shutdown. Signed-off-by: thomas0829 <ycl0829@uw.edu>
…ames/ Signed-off-by: thomas0829 <ycl0829@uw.edu>
…review Signed-off-by: thomas0829 <ycl0829@uw.edu>
Instead of terminating the episode when the subtask planner recognizes the current goal is complete, the VLM now proposes a creative new task based on the current observation. The worker detects the "NEW TASK:" prefix in the VLM response, and the client rotates _initial_task_descriptions so subsequent planning calls use the new goal. This keeps the robot productively exploring without requiring an environment reset. Signed-off-by: Shirui Chen <chinsengi@gmail.com> Signed-off-by: Carl <sc256@uw.edu> Made-with: Cursor
…k display - On SBS startup, if sbs_reward_frames/ has existing content, prompt the user to delete it before starting. - Remove _base_task_description and the "task=... | subtask=..." split display. SBS now only shows the current task_description (which is the subtask when subtask planning is active). The policy model already only receives the current subtask — this makes the display consistent. Signed-off-by: thomas0829 <ycl0829@uw.edu>
…ignore Set use_orig_params: True in all four YAM OpenPI FSDP configs to match the expected FSDP behavior. Replace the overly broad .*/ gitignore pattern with explicit entries for .claude/, .ralph/, .pytest_cache/, and .ruff_cache/. Signed-off-by: Shirui Chen <chinsengi@gmail.com> Signed-off-by: Carl <sc256@uw.edu> Made-with: Cursor
…github.com/chinsengi/RLinf into feat/remote-env-sbs-status-and-timing-push
Measures latency, SSH throughput, and SCP speed between hosts. Supports iperf3 when available, falls back to SSH+dd. Signed-off-by: thomas0829 <ycl0829@uw.edu>
… mode - Save cumulative topreward_input.mp4 mirroring VLMPlannerClient episode frames, plus per-chunk PNGs, organized by episode directory - Write prompt.txt with reconstructed TOPReward prompt text and metadata - Add RobotEnvServicer class docstring covering lifecycle, modes, and thread safety - Fix _episode_count double-increment on recovery flows by only incrementing in Reset when not in a restart path - Guard _format_sbs_status/_format_chunk_task_context with _env_lock to prevent races with PushStatusInfo and SetEpisodeTiming Signed-off-by: Carl <sc256@uw.edu> Made-with: Cursor
…S status and use dummy obs on shutdown - Protect SetEpisodeTiming and PushStatusInfo writes with _env_lock to prevent race conditions with episode_timeout watchdog and SBS status formatter. - Add _env_lock snapshot in _format_sbs_status (read side) to avoid dict iteration crash from concurrent PushStatusInfo. - Use dummy obs in ChunkStep shutdown-exit paths to avoid portal AssertionError when connections are already closed. - Protect local Ctrl+C return_to_home with _env_lock to prevent concurrent hardware access with in-flight ChunkStep. - Fix pre-existing test failures: add missing _reward_frame_interval to fake env and correct task description assertion. Signed-off-by: thomas0829 <chinseng0829@gmail.com> Signed-off-by: thomas0829 <ycl0829@uw.edu>
Switch noise_method to flow_cps and set lr=0 in subtask configs. Add action_chunk_smoothing block (disabled) to async/sync subtask configs. Set return_home_minutes=20 for subtask envs. Fix dones vs prev_logprobs shape check to compare only batch/chunk dims. Refactor _get_staged_runtime_helpers() to import from rlinf.runners instead of examples. Move validate_cfg/get_logger imports into main() to avoid module-level side effects. Add test verifying staged runtime helpers import from shared runtime module. Add .qwen3_runtime/ to .gitignore. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: chinsengi <sc256@uw.edu>
…yboardInterrupt Use a timeout-based join loop so the main thread can process signals between iterations instead of blocking indefinitely in thread.join(). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: chinsengi <sc256@uw.edu>
Signed-off-by: thomas0829 <chinseng0829@gmail.com> Signed-off-by: thomas0829 <ycl0829@uw.edu>
Instead of re-raising KeyboardInterrupt, print a short message and let the finally block handle cleanup gracefully. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: chinsengi <sc256@uw.edu>
…port prepare_for_reconnection no longer resets _has_reset_once, so the next reset() goes through the normal PD-restore + interpolate-to-home path instead of the skip-home initial-reset path that only captures the current pose without motor commands. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: chinsengi <sc256@uw.edu>
…via sys.exit Keep the original raise behavior (which the robot server depends on for correct shutdown) but call sys.exit(0) at the end of the finally block to prevent the traceback from printing. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: chinsengi <sc256@uw.edu>
…rupt path Remove sys.exit(0) from finally block — the original raise behavior is confirmed to produce correct robot shutdown. Accept the traceback as cosmetic. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: chinsengi <sc256@uw.edu>
Revert b601fa0 — the change caused the robot to make large unexpected movements on the first chunk after beaker restart. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: chinsengi <sc256@uw.edu>
…oint Catch KeyboardInterrupt in the entry script so the runtime's raise propagates normally (preserving correct shutdown) but no traceback is printed to the user. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: chinsengi <sc256@uw.edu>
Replace the try/except approach with a custom excepthook that only changes the traceback display. The exception propagation and exit path remain completely unchanged. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: chinsengi <sc256@uw.edu>
The code fallback default was 16 while every YAML config and the vlm_planner_client already used 1000. Update the docstring, the VLMPlannerWorker constructor, and the TOPReward class to match. Signed-off-by: Carl <sc256@uw.edu> Made-with: Cursor
When a cooldown boundary falls mid-rollout, should_collect_previous and should_collect_current disagree, producing trajectories with mismatched tensor lengths (e.g. rewards has 1 entry but prev_values has 2). This caused a RuntimeError in compute_advantages_and_returns. Add prev_values length validation to _validate_trainable_rollout_batch and convert validation errors to _CooldownTransitionError so the decorator skips the training step instead of crashing. Also add observation shape validation in _proto_to_obs to catch corrupted gRPC responses early with a clear error message. Signed-off-by: chinsengi <sc256@uw.edu>
Stripped-down version of submit_yam_beaker_cluster.sh that submits a gantry job with GPUs and idles, leaving Ray/training startup to manual attach. Repo and .venv are expected on Weka. Signed-off-by: Carl <sc256@uw.edu> Made-with: Cursor
…down ChunkStep logs 1. reset_for_env_reset now restores task_description to the initial value so maybe_plan_initial_subtask can re-prime after cooldown. 2. Move _print_chunk_task_context after the _restart_required check so ChunkStep logs are suppressed during cooldown idle responses. Signed-off-by: chinsengi <sc256@uw.edu>
After cooldown transitions, a partial rollout with fewer steps than global_batch_size / world_size would crash in run_training. Add a minimum-size check in _has_trainable_rollout_batch to skip gracefully. Signed-off-by: chinsengi <sc256@uw.edu>
The per-slot reset_for_env_reset (called from on_env_step during cooldown) restored task_description to the main task but did not reset _steps_since_subtask_update to 0. This caused maybe_plan_initial_subtask to skip VLM re-planning after cooldown recovery because the should_prime condition required _steps_since_subtask_update == 0. Signed-off-by: chinsengi <chinsengi@gmail.com> Signed-off-by: chinsengi <sc256@uw.edu>
Log score, recent deltas, and trigger status on every adaptive check so we can see why adaptive subtask changes are not firing. Signed-off-by: chinsengi <chinsengi@gmail.com> Signed-off-by: chinsengi <sc256@uw.edu>
Log when VLM is called for initial subtask and what it returns, to diagnose why the initial subtask may not differ from main task. Signed-off-by: chinsengi <chinsengi@gmail.com> Signed-off-by: chinsengi <sc256@uw.edu>
…github.com/chinsengi/RLinf into feat/remote-env-sbs-status-and-timing-push
… thresholds - Replace silent except in RemoteEnv.task_description setter with a warning log so we can diagnose why the robot server does not see subtask updates from the VLM planner. - Adjust adaptive subtask thresholds to match actual TOPReward score range (~-20): plateau_threshold 0.01→1.0, score_threshold -0.5→-10. Signed-off-by: chinsengi <chinsengi@gmail.com> Signed-off-by: chinsengi <sc256@uw.edu>
Signed-off-by: chinsengi <chinsengi@gmail.com> Signed-off-by: chinsengi <sc256@uw.edu>
reset_for_env_reset was calling inner_env.task_description (the property setter) which sends a SetTaskDescription gRPC to the robot server. This overwrites the VLM-planned subtask with the main task right before maybe_plan_initial_subtask re-plans — causing the robot server to show the main task for the first several ChunkSteps after cooldown recovery. Fix: set _task_description directly (bypassing the setter and gRPC) so only the beaker-side state is updated for the should_prime condition check. The actual SetTaskDescription gRPC is sent later by apply_subtask_update inside maybe_plan_initial_subtask. Signed-off-by: chinsengi <chinsengi@gmail.com> Signed-off-by: chinsengi <sc256@uw.edu>
Force a one-time synchronous subtask re-prime when collection resumes after cooldown so rollout observations no longer stay on the main-task placeholder until the interval trigger. Made-with: Cursor
Constrain VLM subtask generation to produce concrete micro-steps instead of restating the episode goal, and remove stale worker/server code paths that are no longer used by the current planner flow. Made-with: Cursor
…on actions and enable training - Update VLM planner system prompt to require direct object manipulation actions (grasp, lift, fold, place) instead of navigation/approach actions - Enable actor learning rates across subtask_async, subtask_sync, and sync configs (lr=5e-6, value_lr=5e-5) - Increase return_home_minutes to 3 and set noise_level to 0.3 for subtask_async config Signed-off-by: chinsengi <sc256@uw.edu>
Signed-off-by: chinsengi <sc256@uw.edu>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
PushStatusInfoRPC so Beaker can push TOPReward score/delta (and other key-values) to the desktopRobotServer; shown in the SBS prompt before each chunk.SetEpisodeTimingRPC soRemoteEnvpushesepisode_duration_s/episode_cooldown_sat connect time — Beaker becomes the single timing source, no desktop restart needed to change the timer.touch /tmp/rlinf_approve_chunk) with an Enter-key prompt read from/dev/tty, matching the existing per-chunk SBS prompt UX.Files changed
rlinf/envs/remote/proto/robot_env.proto— new RPCs + messages (regenerated pb2 / pb2_grpc).rlinf/envs/remote/robot_server.py— servicer handlers, SBS status line formatter, Enter-based first-chunk approval.rlinf/envs/remote/remote_env.py—__init__pushes episode timing; newpush_status_info(values, text)method.rlinf/workers/env/vlm_planner_client.py— pushestop_reward_score/top_reward_deltato the matchingRemoteEnvafter each TOPReward computation.rlinf/workers/env/env_worker.py— threadsenv_list/slot_idthrough to the planner client so it can look up the right env.rlinf/config.py— mirrors top-levelenv.return_home_minutes/env.server_cooldown_minutesintoenv.train.episode_duration_s/env.train.episode_cooldown_ssoRemoteEnvcan pick them up.Test plan
RobotServer(without specifying timing), launch Beaker training, confirm server logsEpisode timing updated from client.--sbs, confirm the SBS prompt showschunk#N | ep_time=... | client(Ns ago): top_reward_score=..., top_reward_delta=...before each chunk.touch /tmp/rlinf_approve_chunkneeded).