Skip to content

feat(exec): buffer output before attach and separate execution wait#1022

Open
BatmanByte wants to merge 5 commits into
boxlite-ai:mainfrom
BatmanByte:codex/init-stdio-drain
Open

feat(exec): buffer output before attach and separate execution wait#1022
BatmanByte wants to merge 5 commits into
boxlite-ai:mainfrom
BatmanByte:codex/init-stdio-drain

Conversation

@BatmanByte

@BatmanByte BatmanByte commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Summary

Drain guest exec pipes when the process is created rather than when Attach starts, so a command can exit even when the caller never attaches. Buffer retained guest output for one late consumer, and keep execution completion on an independent wait path so unread output cannot block Execution::wait().

Builds on #988, which made the container main command the guest init process.

Changes

  • Add a guest OutputManager that continuously drains stdout and stderr into a bounded 1 MiB ring buffer.
  • Replay retained output to one late Attach consumer and send ExecOutput.dropped when older history was overwritten.
  • Bound portal and REST output queues so slow consumers apply backpressure instead of accumulating host memory.
  • Reset UTF-8 decoding at dropped-output boundaries so bytes across a loss boundary cannot form a fabricated character.
  • Add GET /executions/{id}/wait through boxlite serve, runner, API proxy, OpenAPI, and the REST SDK. Completion is independent from the WebSocket output pump.
  • Bound each server poll to 45 minutes, with a 50-minute proxy timeout and 55-minute SDK request timeout. The SDK renews running polls and retries transient transport failures within its total wait budget.
  • Fall back to GET /executions/{id} status polling when an older runner or boxlite serve instance returns 404 for the new wait route.

How to verify

  • BUILD_PROFILE=debug FILTER=main_command_exits_after_large_output_without_attach make test:integration:rust
  • BOXLITE_DEPS_STUB=1 cargo test -p boxlite --features rest --lib rest::litebox::tests:: -- --test-threads=1
  • make fmt:check:rust
  • make clippy

Risks / rollout

Completed executions currently retain their 1 MiB guest output ring through the execution registry. A follow-up will add terminal-session cleanup or a TTL. The wait-route fallback preserves SDK compatibility with older servers, but old servers still use short status polling rather than the newer long-poll route.

Summary by CodeRabbit

  • New Features
    • Improved command stdout/stderr streaming with bounded buffering, backpressure, and retained output for late attachment.
    • Added explicit output-dropped/overwritten events when output buffer capacity is exceeded.
    • Introduced “wait for completion” long-poll endpoints across REST/proxy/runner/CLI, plus OpenAPI + client support.
  • Bug Fixes
    • More reliable UTF-8 recovery across chunk boundaries.
    • Cancellation-safe draining/flush behavior during shutdown and reconnection.
  • Tests
    • Added/updated integration and controller tests to validate bounded backpressure, drop events, and the new wait behavior.

Drain exec pipes from process creation into a bounded guest ring so commands without an Attach consumer cannot block on stdout or stderr. Surface overwritten history as a control event and use bounded host queues to propagate slow-consumer backpressure.

Reset host UTF-8 decoding at dropped-output boundaries and cover the no-Attach and late-Attach VM paths.
@BatmanByte
BatmanByte requested a review from a team July 23, 2026 08:42
@boxlite-agent

boxlite-agent Bot commented Jul 23, 2026

Copy link
Copy Markdown

📦 BoxLite review — looks good · c7b2ce1

Review evidence

  • git diff --numstat origin/main...HEAD && git diff origin/main...HEAD — 19 files, ~1400 lines changed across TS/Go/Rust
  • grep status literals in Go runner/CLI-serve handlers — server only emits completed|running; rust's extra killed/timed_out arms are dead but harmless
  • cargo test --lib rest::litebox::tests:: (src/boxlite) — cargo not installed in sandbox
  • go test ./apps/runner/... — go not installed in sandbox

Risk notes

  • concurrency: bounded output channels — stdout/stderr moved from unbounded to mpsc::channel(1024); portal exec.rs adds shutdown_token-raced sends (route_output_until_cancelled) so backpressure can't hang shutdown, with a dedicated test (route_output_cancels_while_output_queue_is_full); verified logic by reading, not executed
  • wait/attach decoupling — ExecResult now delivered via new spawn_execution_wait long-poll task instead of the WS exit frame; attach_ws_pump no longer emits ExecResult on disconnect. Confirmed via test wait_returns_when_unread_output_fills_the_websocket_queue that wait completes even with a full output queue
  • guest output ring buffer — new OutputManager (output.rs) drains stdout/stderr into a 1MB shared-sequence ring buffer from process start regardless of attach, evicting oldest entries and emitting one Dropped marker per gap; index math (next_sequence - oldest_sequence) verified safe given single-attach invariant
  • REST wait fallback/backoff — wait_execution retries network errors with exponential backoff (1s-30s), treats status==running as immediate re-poll (server already long-polled), falls back permanently to plain status endpoint on 404 (old runner); bounded by 25h WAIT_TOTAL_TIMEOUT
  • coverage — openapi/box.openapi.yaml and serve/README.md changes are documentation-only, skimmed but not deeply verified against schema tooling
apps/api/src/boxlite-rest/boxlite-proxy.controller.ts
  proxyExecWait/proxyTimeout opt  +23/-2  new wait proxy route, 50m timeout
apps/runner/pkg/api/controllers/boxlite_exec.go
  BoxliteWaitExecution  +21/-0  long-poll handler, 45m server timeout
apps/runner/pkg/api/controllers/boxlite_exec_test.go
  TestBoxliteWaitExecution*  +69/-0  blocking + timeout wait tests
apps/runner/pkg/api/server.go
  route registration  +1/-0  wires GET .../wait
src/boxlite/src/litebox/exec.rs
  OutputReceiver enum  +42/-10  bounded/unbounded receiver abstraction
src/boxlite/src/portal/interfaces/exec.rs
  DecodedStream, route_output*_until_cancelled  +244/-47  async backpressure-aware send + cancellation
src/boxlite/src/rest/client.rs
  get_with_timeout  +9/-0  per-request timeout override
src/boxlite/src/rest/litebox.rs
  wait_execution/spawn_execution_wait  +397/-187  decouples completion from WS attach
src/boxlite/src/rest/types.rs
  ExecutionStatusResponse.error_message  +4/-4  carries server error through wait
src/boxlite/tests/run_main_command.rs
  main_command_exits_after_large_output_without_attach  +83/-0  integration coverage for buffering
src/guest/src/service/exec/output.rs
  OutputManager  +204/-0  new ring-buffer output manager
src/guest/src/service/exec/state.rs
  ExecutionState::attach  +21/-112  delegates to OutputManager, drops task bookkeeping
src/cli/src/commands/serve/handlers/executions.rs
  wait_execution/wait_for_completion  +73/-1  axum wait endpoint, watch-based wait
src/cli/src/commands/serve/mod.rs
  done_tx.send_replace  +5/-1  route wiring + watch send fix
src/shared/proto/boxlite/v1/service.proto
  OutputDropped  +5/-0  new oneof variant for eviction signal

reviewed c7b2ce1 in a BoxLite microVM · @boxlite-agent review to re-run · powered by BoxLite

@coderabbitai

coderabbitai Bot commented Jul 23, 2026

Copy link
Copy Markdown

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

Execution output now uses bounded channels and guest-side sequenced buffering. Dropped output is reported explicitly, WebSocket streaming is separated from completion reporting, and wait endpoints provide execution status across REST, runner, and CLI servers.

Changes

Execution output and completion flow

Layer / File(s) Summary
Host bounded output pipeline
src/boxlite/src/litebox/exec.rs, src/boxlite/src/portal/interfaces/exec.rs
Output channels and decoders support bounded asynchronous delivery, backpressure, cancellation, UTF-8 boundary handling, and dropped-frame flushing.
Guest output buffering and attach wiring
src/shared/proto/boxlite/v1/service.proto, src/guest/src/service/exec/*
Guest output is drained into a sequenced buffer, with evicted data represented by OutputDropped; attach returns the buffered stream directly.
REST attach streaming and wait completion
src/boxlite/src/rest/litebox.rs, src/boxlite/src/rest/client.rs, apps/api/src/boxlite-rest/*, apps/runner/pkg/api/*, openapi/box.openapi.yaml
WebSocket output uses bounded channels, while terminal results are retrieved through a dedicated HTTP wait endpoint with retry, proxy, and runner routing.
CLI wait endpoint and completion signaling
src/cli/src/commands/serve/*
The CLI server exposes execution waiting, shares status serialization, and observes completion through a watch signal.
Output buffering integration tests
src/boxlite/tests/run_main_command.rs
Tests cover large output without an attach consumer and dropped output after late attachment.

Estimated code review effort: 5 (Critical) | ~120 minutes

Sequence Diagram(s)

sequenceDiagram
  participant ExecProtocol
  participant OutputManager
  participant WebSocketAttach
  participant WaitEndpoint
  participant Execution
  ExecProtocol->>OutputManager: deliver decoded output
  WebSocketAttach->>OutputManager: attach to buffered stream
  OutputManager-->>WebSocketAttach: ordered output or OutputDropped
  WaitEndpoint->>Execution: await terminal completion
  Execution-->>WaitEndpoint: execution status and exit code
Loading

Possibly related PRs

Suggested labels: bug, rust, e2e-test

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title is concise and accurately reflects the main changes: buffering output before attach and separating execution wait.
Description check ✅ Passed The description matches the template well, with Summary, Changes, How to verify, and Risks / rollout sections all present.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@BatmanByte
BatmanByte force-pushed the codex/init-stdio-drain branch from b9f4787 to b0ee85c Compare July 23, 2026 08:49

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
src/boxlite/src/portal/interfaces/exec.rs (1)

338-388: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

route_output().await ignores shutdown_token — a stalled consumer blocks shutdown indefinitely.

The select! at lines 338-353 only guards stream.message(); the per-message Self::route_output(...).await at line 358 (and the terminal-path flushes/sends at lines 371-384) run outside it. Since mpsc::Sender::send().await on the now-bounded channel blocks until capacity frees up, a consumer that is alive but not polling (hasn't dropped its ExecStdout/ExecStderr) makes this task ignore shutdown_token.cancelled() for as long as the channel stays full — exactly the scenario the non-blocking try_flush() on the cancellation branch was designed to avoid.

🔧 Race the per-message routing against cancellation too
                         match output.transpose() {
                             Some(Ok(output)) => {
                                 message_count += 1;
-                                Self::route_output(output, &mut stdout, &mut stderr).await;
+                                tokio::select! {
+                                    biased;
+                                    _ = shutdown_token.cancelled() => {
+                                        stdout.try_flush();
+                                        stderr.try_flush();
+                                        break;
+                                    }
+                                    _ = Self::route_output(output, &mut stdout, &mut stderr) => {}
+                                }
                             }

The same gap applies to the error/EOF terminal flushes (lines 371-384), though its impact there is smaller since the loop is about to exit anyway.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/boxlite/src/portal/interfaces/exec.rs` around lines 338 - 388, Update the
attach-stream loop around route_output and the terminal error/EOF handling so
every potentially blocking flush or send is raced against
shutdown_token.cancelled(). Ensure cancellation exits promptly without awaiting
a bounded-channel operation indefinitely, while preserving normal output
ordering and existing cleanup behavior when no shutdown occurs.
🧹 Nitpick comments (1)
src/guest/src/service/exec/output.rs (1)

16-19: 🧹 Nitpick | 🔵 Trivial

Unbounded ring accumulation across completed executions.

Each OutputManager retains its 1 MiB ring after the process exits (held by Inner.output and any live attach-stream clone), so total guest memory grows with the number of completed-but-uncleaned executions. The PR already notes TTL/cleanup as future work; consider bounding this with an eviction/TTL on completed executions before it becomes a guest-OOM vector under high exec churn.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/guest/src/service/exec/output.rs` around lines 16 - 19, Bound completed
execution state retained by OutputManager so its 1 MiB ring is not kept
indefinitely after process exit. Add eviction or TTL cleanup for completed
executions, including releasing Inner.output and any attach-stream-held clones
when cleanup occurs, while preserving access for active executions.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@src/boxlite/src/portal/interfaces/exec.rs`:
- Around line 338-388: Update the attach-stream loop around route_output and the
terminal error/EOF handling so every potentially blocking flush or send is raced
against shutdown_token.cancelled(). Ensure cancellation exits promptly without
awaiting a bounded-channel operation indefinitely, while preserving normal
output ordering and existing cleanup behavior when no shutdown occurs.

---

Nitpick comments:
In `@src/guest/src/service/exec/output.rs`:
- Around line 16-19: Bound completed execution state retained by OutputManager
so its 1 MiB ring is not kept indefinitely after process exit. Add eviction or
TTL cleanup for completed executions, including releasing Inner.output and any
attach-stream-held clones when cleanup occurs, while preserving access for
active executions.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 104bcb7e-857c-4403-a78a-aac86bcf1d0f

📥 Commits

Reviewing files that changed from the base of the PR and between a5da5b4 and b0ee85c.

📒 Files selected for processing (9)
  • src/boxlite/src/litebox/exec.rs
  • src/boxlite/src/litebox/mod.rs
  • src/boxlite/src/portal/interfaces/exec.rs
  • src/boxlite/src/rest/litebox.rs
  • src/boxlite/tests/run_main_command.rs
  • src/guest/src/service/exec/mod.rs
  • src/guest/src/service/exec/output.rs
  • src/guest/src/service/exec/state.rs
  • src/shared/proto/boxlite/v1/service.proto

Bounded output delivery can wait indefinitely when a caller retains but does not poll an output stream. Race routing, terminal flushes, and attach error delivery against shutdown so the task exits promptly without bypassing normal backpressure.
@BatmanByte

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1e8b966028

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/boxlite/src/rest/litebox.rs
Keep the WebSocket pump responsible only for stdio. A full output receiver must not prevent callers from observing an execution's terminal result, so the REST backend now waits through a separate long-lived endpoint exposed by the CLI server, runner, and API proxy.

Use watch::Sender::send_replace so a server-side wait started after completion still observes the stored terminal state.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
src/cli/src/commands/serve/handlers/executions.rs (1)

110-140: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

wait/status responses never surface error_message, unlike the REST server and Go runner.

execution_status_body only emits execution_id, status, and exit_code. This isn't just a JSON-construction gap — ActiveExecution's wait task (mod.rs) only stores result.exit_code from Execution::wait(), discarding result.error_message entirely, so there's nothing to surface even if this function were extended. Both the Go runner (ExecutionInfoResponse.ErrorMessage) and the REST client's ExecutionStatusResponse.error_message carry this field — e.g. the container-init-death diagnosis attached when a process is SIGKILLed via PID-namespace teardown. The boxlite serve backend is now the one implementation of this shared "wait" contract that can't report it.

Consider storing error_message alongside exit_code on ActiveExecution (populated from the same wait() call) and including it in execution_status_body when present.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/cli/src/commands/serve/handlers/executions.rs` around lines 110 - 140,
Extend ActiveExecution to retain result.error_message alongside the exit code
when its wait task processes Execution::wait(). Update execution_status_body and
the related wait/status response path to include error_message when present,
preserving the existing execution_id, status, and exit_code fields and omitting
the new field when absent.
🧹 Nitpick comments (2)
src/boxlite/src/rest/litebox.rs (1)

73-90: 🩺 Stability & Availability | 🔵 Trivial | 🏗️ Heavy lift

Spawned WS pump task has no lifecycle handle; bounded output channel can block it forever.

tokio::spawn(...) return values are discarded in both wire_attach and exec(). Combined with the new bounded stdout_tx/stderr_tx channels, attach_ws_pump's stdout_tx.send(text).await (line 802/806) blocks the entire pump loop — including reading the eventual exit frame and sending keepalive pings — whenever the channel is full. The added test wait_returns_when_unread_output_fills_the_websocket_queue demonstrates exactly this and has to attach.abort() manually to clean up; production callers have no equivalent hook. A caller that keeps an Execution alive (e.g., just to call wait()/signal()) without draining stdout()/stderr() to completion will leave the WS connection and task stuck open until the Execution itself is dropped.

Worth confirming this is the intended tradeoff (bounded memory vs. potentially long-lived stuck connections for non-draining consumers), and if so, consider retaining the JoinHandle so it can be aborted once wait_execution resolves (or documenting the drain requirement on Execution).

Also applies to: 154-173

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/boxlite/src/rest/litebox.rs` around lines 73 - 90, Retain the JoinHandle
returned by the spawned attach_ws_pump task in both wire_attach and exec, and
tie its lifecycle to wait_execution so the pump is aborted once execution
completion resolves, including when output channels are not drained. Ensure the
resulting Execution retains or exposes the necessary handle and cleanup occurs
without changing normal output consumption behavior.
apps/api/src/boxlite-rest/boxlite-proxy.controller.ts (1)

131-151: 🩺 Stability & Availability | 🔵 Trivial

Confirm nothing upstream cuts this connection before the 25h proxyTimeout.

proxyTimeout only bounds the proxy→runner leg. If the Node http server (main.ts, not in this changeset) or any load balancer/reverse proxy in front of this service has a shorter idle/socket timeout, a long-running wait could still be truncated well before 25h, which would surface to SDK callers as a spurious network failure on the very endpoint meant to be authoritative for completion.

Also applies to: 255-269

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@apps/api/src/boxlite-rest/boxlite-proxy.controller.ts` around lines 131 -
151, The long-running proxyExecWait request may be terminated by shorter
upstream Node HTTP server or load-balancer idle/socket timeouts despite its
25-hour proxyTimeout. Inspect and update the relevant server and deployment
proxy timeout configuration so the connection remains valid for at least the
full wait duration, covering the equivalent configuration for the other affected
endpoint as well.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/boxlite/src/rest/litebox.rs`:
- Around line 974-1015: Update wait_execution to retry transient transport
errors from get_with_timeout using bounded backoff consistent with
WS_RECONNECT_BUDGET before producing an ExecResult failure. Preserve immediate
handling of terminal statuses and non-terminal responses, and send the synthetic
error result only after the retry budget is exhausted.

---

Outside diff comments:
In `@src/cli/src/commands/serve/handlers/executions.rs`:
- Around line 110-140: Extend ActiveExecution to retain result.error_message
alongside the exit code when its wait task processes Execution::wait(). Update
execution_status_body and the related wait/status response path to include
error_message when present, preserving the existing execution_id, status, and
exit_code fields and omitting the new field when absent.

---

Nitpick comments:
In `@apps/api/src/boxlite-rest/boxlite-proxy.controller.ts`:
- Around line 131-151: The long-running proxyExecWait request may be terminated
by shorter upstream Node HTTP server or load-balancer idle/socket timeouts
despite its 25-hour proxyTimeout. Inspect and update the relevant server and
deployment proxy timeout configuration so the connection remains valid for at
least the full wait duration, covering the equivalent configuration for the
other affected endpoint as well.

In `@src/boxlite/src/rest/litebox.rs`:
- Around line 73-90: Retain the JoinHandle returned by the spawned
attach_ws_pump task in both wire_attach and exec, and tie its lifecycle to
wait_execution so the pump is aborted once execution completion resolves,
including when output channels are not drained. Ensure the resulting Execution
retains or exposes the necessary handle and cleanup occurs without changing
normal output consumption behavior.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: a936e2d1-b314-4d66-b16f-c402e6108a55

📥 Commits

Reviewing files that changed from the base of the PR and between 1e8b966 and 6f87c18.

📒 Files selected for processing (11)
  • apps/api/src/boxlite-rest/boxlite-proxy.controller.ts
  • apps/runner/pkg/api/controllers/boxlite_exec.go
  • apps/runner/pkg/api/controllers/boxlite_exec_test.go
  • apps/runner/pkg/api/server.go
  • openapi/box.openapi.yaml
  • src/boxlite/src/rest/client.rs
  • src/boxlite/src/rest/litebox.rs
  • src/boxlite/src/rest/types.rs
  • src/cli/src/commands/serve/README.md
  • src/cli/src/commands/serve/handlers/executions.rs
  • src/cli/src/commands/serve/mod.rs

Comment thread src/boxlite/src/rest/litebox.rs
Limit each runner and serve wait request to 45 minutes so it completes before the API load balancer's idle timeout. The REST client renews running polls and retries transient transport failures within the execution wait budget.

Copy link
Copy Markdown
Contributor Author

Review follow-up:

  • The long-lived REST wait / proxy timeout finding is addressed in f28b16c. Each server poll is bounded to 45 minutes, the API proxy has a 50-minute limit, and the SDK has a 55-minute request limit. running responses renew the poll; transient transport failures retry with bounded exponential backoff.
  • The CLI error_message parity finding is valid but intentionally left for a separate follow-up. This change remains limited to making REST wait work across the existing one-hour ALB idle timeout.
  • I did not abort the REST WebSocket pump when wait() resolves. That would cut output for callers that read stdout/stderr after waiting. A separate output-consumer lifecycle/cancellation policy is needed for that resource-retention tradeoff.

Tests include REST wait retry coverage and the runner wait-timeout controller test.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@openapi/box.openapi.yaml`:
- Around line 644-647: Update the endpoint’s 200-response description in the
OpenAPI definition to document both outcomes: a terminal execution result and a
non-terminal status: "running" response when the 45-minute wait expires. Replace
the “Terminal execution result” wording while preserving the existing timeout
and retry behavior details.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 3709c682-a0e9-424e-a3a5-cc86ec7fa2b0

📥 Commits

Reviewing files that changed from the base of the PR and between 6f87c18 and f28b16c.

📒 Files selected for processing (7)
  • apps/api/src/boxlite-rest/boxlite-proxy.controller.ts
  • apps/runner/pkg/api/controllers/boxlite_exec.go
  • apps/runner/pkg/api/controllers/boxlite_exec_test.go
  • openapi/box.openapi.yaml
  • src/boxlite/src/rest/litebox.rs
  • src/cli/src/commands/serve/README.md
  • src/cli/src/commands/serve/handlers/executions.rs
🚧 Files skipped from review as they are similar to previous changes (4)
  • apps/api/src/boxlite-rest/boxlite-proxy.controller.ts
  • src/cli/src/commands/serve/README.md
  • src/cli/src/commands/serve/handlers/executions.rs
  • src/boxlite/src/rest/litebox.rs

Comment thread openapi/box.openapi.yaml
@BatmanByte

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Something went wrong. Try again later by commenting “@codex review”.

Unknown error
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f28b16c618

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/boxlite/src/rest/litebox.rs
New REST SDKs may call a runner or boxlite serve instance that predates the execution wait route. A 404 from that route previously produced a synthetic failed result even when the older status route could return the real exit code.

Fall back to the status route only after the initial wait-route 404, retaining the normal error path if the status route also reports the execution missing. Document that a successful wait response may be non-terminal after its poll timeout.
@BatmanByte

Copy link
Copy Markdown
Contributor Author

@codex review

@BatmanByte BatmanByte changed the title feat(exec): buffer output before attach feat(exec): buffer output before attach and separate execution wait Jul 24, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c7b2ce1a8c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +637 to 638
tracing::debug!(path = %path, error = %e, "WS attach: connect failed; output will not stream");
return;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Surface REST attach failures to waiters

When the initial WebSocket attach fails (for example a transient proxy/runner upgrade failure), this task now only logs at debug and returns while the separately spawned /wait task can still report a clean exit code with no error_message. In that case stdout/stderr were never streamed, but callers of Execution::wait() cannot distinguish “command produced no output” from “we lost the output”; the previous fallback preserved the attach failure in the result. Please propagate this attach failure to the execution result/status path or otherwise surface it to the caller.

AGENTS.md reference: AGENTS.md:L69-L70

Useful? React with 👍 / 👎.

@DorianZheng DorianZheng left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

most of these trace to two changes in this PR — the unbounded→bounded output channels, and the narrowed wait-retry predicate:

  • one task routes both streams through blocking bounded sends, so reading only one of stdout/stderr can stall the other (and on the ws pump a full channel also blocks stdin + the keepalive ping).
  • wait_execution retries only Network, so a transient 503/timeout/429 becomes a cached exit_code:-1.

the rest are resource retention (guest ring + leaked wait tasks/timers), the auto-pause vs long-poll interaction, and a few decoder/observability items.

}

fn route_output(output: ExecOutput, stdout: &mut DecodedStream, stderr: &mut DecodedStream) {
async fn route_output(

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a single task routes both stdout and stderr through the bounded(1024) channels here, and send_text awaits the send. a caller that reads only one stream (the pattern shown in the exec_handle docs) fills the other channel; route_output then parks on send().await, the task stops polling the guest stream, and the stream that is being read never advances. meanwhile the guest's 1 MiB ring overflows and drops output. the pre-change unbounded senders couldn't block here.

0x01 => {
tracing::trace!(len = text.len(), "WS attach: stdout frame");
let _ = stdout_tx.send(text);
let _ = stdout_tx.send(text).await;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this send is now bounded and sits inside the single select! loop, so a full stdout channel blocks stdin forwarding and the keepalive ping too (same for the stderr send just below). an interactive exec whose consumer stops reading stdout — or an exec().wait() that never reads output — fills the 1024 slots, the pump parks on send().await, stdin stops flowing, no ping goes out, and the idle watchdog can reap a healthy connection.

using_status_endpoint = true;
continue;
}
Err(error) => {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this terminal arm turns every non-Network, non-NotFound error into ExecResult{exit_code:-1}. an enveloped 503 (Portal), a timeout envelope (Internal), a bare 500 (Internal), and 429 (ResourceExhausted) all land here instead of retrying, and Execution::wait() caches the result via get_or_try_init — so a momentary gateway hiccup is permanently reported to the caller as the command having failed.

}

@Get(':boxId/executions/:execId/wait')
async proxyExecWait(

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this endpoint invites a wait-only client (no attach, no other calls), but proxyToRunner writes lastActivityAt once at request start. a 25-min command polled only through /wait has its activity stamped at t=0, so the auto-pause sweeper (900s) can pause the box mid-wait. with autoResume off the exec stays frozen; with it on the box flaps and time is lost.

use tokio::sync::{watch, Mutex};
use tonic::Status;

const BUFFER_CAPACITY_BYTES: usize = 1024 * 1024;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

every ExecutionState now retains an OutputManager ring up to this 1 MiB cap, and the guest ExecutionRegistry has no remove/evict path, so nothing frees it after the exec exits. a box that runs many output-heavy commands over its lifetime pins ~1 MiB each until it stops — guest RSS grows monotonically toward OOM.

});
}

async fn drain<S>(&self, mut stream: S, source: OutputSource)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this drain loop (and the spawn helpers) emit no tracing — the previous forwarding tasks logged per-exec (execution=…, "…forwarding task ended"). a stuck or prematurely-ended exec can no longer be correlated to an exec_id or a specific stream from guest logs.

stderr.send_bytes(chunk.data);
stderr.send_bytes(chunk.data).await;
}
Some(exec_output::Event::Dropped(_)) => {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the Dropped event is stream-agnostic and flushes both decoders. a loss on one stream then corrupts an in-flight split codepoint on the other: if stderr is mid-codepoint (holding e.g. 0xE2) when a stdout burst overflows the ring, stderr.flush() emits U+FFFD for the held byte and the surviving continuation bytes decode to more U+FFFD — a healthy on stderr becomes replacement chars though stderr lost nothing.

retry_backoff = retry_backoff.saturating_mul(2).min(WAIT_RETRY_BACKOFF_MAX);
delay
}
Err(BoxliteError::NotFound(_)) if !using_status_endpoint => {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

using_status_endpoint latches true on the first /wait 404 and never resets, and a route-unsupported 404 is indistinguishable from a transient/real-exec 404 here. one early 404 (exec not yet visible on the addressed runner, or a proxy-level 404) permanently downgrades the client to immediate status polling for the whole exec, even against a server that supports the long-poll.


/// Decode a wire chunk and forward any completed text to the receiver.
fn send_bytes(&mut self, data: Vec<u8>) {
async fn send_bytes(&mut self, data: Vec<u8>) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

on shutdown cancellation this path can lose output: send_bytes decodes and then awaits the send, so a cancel mid-send drops the already-decoded chunk; and try_flush only handles Closed, not Full, so a held partial's trailing U+FFFD is dropped when the channel is full. the normal EOF/error path awaits the flush and delivers these, so the shutdown path silently diverges.

_ => {
// Server says the exec is still running — surface the
// synthesized cause so the caller sees the disconnect.
Err(BoxliteError::Network(error)) => {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Network errors are retried until WAIT_TOTAL_TIMEOUT (25h) with no consecutive-failure cap, so a clearly-persistent outage (connection refused / bare 502 / DNS failure all map to Network) makes Execution::wait() hang for up to a day instead of failing fast.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants