Skip to content

fix(runner): drop dead-on-arrival toolbox HTTP readiness check#754

Merged
DorianZheng merged 1 commit into
mainfrom
fix/runner-drop-broken-toolbox-readiness
Jun 15, 2026
Merged

fix(runner): drop dead-on-arrival toolbox HTTP readiness check#754
DorianZheng merged 1 commit into
mainfrom
fix/runner-drop-broken-toolbox-readiness

Conversation

@G4614

@G4614 G4614 commented Jun 13, 2026

Copy link
Copy Markdown
Contributor

Draft. Targets chore/e2e-required-merge-gate (PR #724) so the e2e-cloud stack picks this up on the next dispatch.

waitForToolboxReady was introduced in fc88aa0b ("feat: add agent-ready runtime catalog") and landed on main via PR #715 on 2026-06-10. It HTTP-polls http://127.0.0.1:<hostPort>/version expecting a daemon on guest TCP port 2280 — the Daytona-daemon interface that was never reimplemented after the Rust guest agent rewrite in dbb11ec8 (2026-04-01). The new agent binds vsock://2695 for gRPC + notifies the host via vsock://2696; nothing in the VM listens on TCP:2280, so libkrun's port-forward accepts the SYN and the guest end immediately RST's, timing out every CREATE_BOX 30 s in.

Production data from the live Tokyo runner:

$ sudo journalctl -u boxlite-runner --since "24 hours ago" | grep -E "CREATE_BOX|toolbox is ready|toolbox not ready"
CREATE_BOX events: 490
toolbox-ready ok:  0
toolbox-ready fail: 181

The exec request that fires immediately after CREATE_BOX (over the same vsock gRPC channel) always succeeds — confirming the box VM and the guest agent are healthy. The readiness probe itself is the bug.

Test plan

Before After
make test:integration:* (PyO3 / FFI path, bypasses the runner) ✓ pass ✓ pass (unchanged)
go test ./pkg/boxlite/ 4 tests, 2 of them assert against the dead HTTP probe 2 tests (the dead two are deleted)
go build ./... + go vet ./...
Live Tokyo: CREATE_BOX for docker.io/library/alpine:3.23 via e2e-cloud-test always box toolbox not ready after 30s expected to succeed (next dispatch on this branch)
journalctl -u boxlite-runner post-deploy toolbox not ready per CREATE_BOX no toolbox lines at all (probe gone)

Iteration plan (no full e2e-cloud needed)

  1. Dispatch Deploy Runner Binary on this branch — rebuilds libboxlite.a + the runner Go binary and pushes via SSH+SCP.
  2. Dispatch Deploy API on this branch (no-op if no API changes, just to make sure the LB/health side is fresh).
  3. Dispatch E2E test (Tokyo) standalone — runs the pytest suite against the freshly-deployed runner.

That triple is ~30-40 min total vs running the full e2e-cloud.yml umbrella.

Files

  • apps/runner/pkg/boxlite/client.go — drop the two waitForToolboxReady call sites + the toolboxReadyTimeout field + plumbing.
  • apps/runner/pkg/boxlite/toolbox_ports.go — delete waitForToolboxReady, drop now-unused io / net/http / time imports.
  • apps/runner/pkg/boxlite/toolbox_ports_test.go — delete the two tests that asserted against the dead probe.
  • apps/runner/cmd/runner/main.go — drop ToolboxReadyTimeout: time.Duration(cfg.DaemonStartTimeoutSec) * time.Second.
  • apps/runner/cmd/runner/config/config.go — drop DaemonStartTimeoutSec env var.

@coderabbitai

coderabbitai Bot commented Jun 13, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 7d292ad8-8e3e-4031-b231-83518e42e5fc

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/runner-drop-broken-toolbox-readiness

Comment @coderabbitai help to get the list of available commands and usage tips.

@DorianZheng DorianZheng marked this pull request as ready for review June 13, 2026 09:53
@DorianZheng DorianZheng requested a review from a team as a code owner June 13, 2026 09:53
G4614 added a commit that referenced this pull request Jun 13, 2026
Run #14 failed all 34 tests at API image validation, not at the runner
toolbox readiness this PR is supposed to fix. The conftest on this branch
defaults to bare 'alpine:3.23'; Tokyo Api's supportedImages allowlist
matches by exact string and only accepts the fully-qualified
'docker.io/library/alpine:3.23'. Set the env so conftest sends the
allowed form; the box VM still boots alpine and exercises the
vsock-readiness path #754 is fixing.
waitForToolboxReady was introduced in fc88aa0 (2026-06-05,
"feat: add agent-ready runtime catalog", which then landed on main
via PR #715 on 2026-06-10). It HTTP-polls http://127.0.0.1:<hostPort>/version
expecting a daemon on guest TCP port 2280 — that interface dates from
the Daytona-daemon era and was never reimplemented after the Rust
guest agent rewrite landed in dbb11ec (2026-04-01). The new agent
binds **vsock://2695** for gRPC + notifies the host via vsock://2696;
nothing inside the VM listens on TCP:2280, so libkrun's port-forward
accepts the SYN and immediately reset-by-peer's, and every CREATE_BOX
times out 30 s in.

Production data from a Tokyo runner: in 24 h, 490 CREATE_BOX events,
0 toolbox-ready successes, 181 toolbox-ready failures. The exec path
that fires immediately afterward (via the same vsock gRPC channel)
**always succeeds** — confirming the box VM is healthy, the readiness
check itself is the bug.

Remove the dead probe: drop waitForToolboxReady from client.go's Create
and Start, drop the function + its TCP/HTTP imports, drop the
toolboxReadyTimeout field, drop the ToolboxReadyTimeout/
DaemonStartTimeoutSec config plumbing in main.go + config.go, drop
the two now-unreachable tests. Box readiness is now signalled by
bx.Start(ctx) returning (which itself blocks on the vsock notification
from the guest).

Branched off chore/e2e-required-merge-gate (PR #724) so the e2e-cloud
stack picks this up next dispatch.
@G4614 G4614 force-pushed the fix/runner-drop-broken-toolbox-readiness branch from ffc2b36 to 32b2eb4 Compare June 13, 2026 10:49
@G4614 G4614 changed the base branch from chore/e2e-required-merge-gate to main June 13, 2026 10:49
@DorianZheng DorianZheng merged commit 5db267d into main Jun 15, 2026
2 of 3 checks passed
@DorianZheng DorianZheng deleted the fix/runner-drop-broken-toolbox-readiness branch June 15, 2026 02:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants