fix(runner): drop dead-on-arrival toolbox HTTP readiness check#754
Merged
Conversation
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
G4614
added a commit
that referenced
this pull request
Jun 13, 2026
Run #14 failed all 34 tests at API image validation, not at the runner toolbox readiness this PR is supposed to fix. The conftest on this branch defaults to bare 'alpine:3.23'; Tokyo Api's supportedImages allowlist matches by exact string and only accepts the fully-qualified 'docker.io/library/alpine:3.23'. Set the env so conftest sends the allowed form; the box VM still boots alpine and exercises the vsock-readiness path #754 is fixing.
waitForToolboxReady was introduced in fc88aa0 (2026-06-05, "feat: add agent-ready runtime catalog", which then landed on main via PR #715 on 2026-06-10). It HTTP-polls http://127.0.0.1:<hostPort>/version expecting a daemon on guest TCP port 2280 — that interface dates from the Daytona-daemon era and was never reimplemented after the Rust guest agent rewrite landed in dbb11ec (2026-04-01). The new agent binds **vsock://2695** for gRPC + notifies the host via vsock://2696; nothing inside the VM listens on TCP:2280, so libkrun's port-forward accepts the SYN and immediately reset-by-peer's, and every CREATE_BOX times out 30 s in. Production data from a Tokyo runner: in 24 h, 490 CREATE_BOX events, 0 toolbox-ready successes, 181 toolbox-ready failures. The exec path that fires immediately afterward (via the same vsock gRPC channel) **always succeeds** — confirming the box VM is healthy, the readiness check itself is the bug. Remove the dead probe: drop waitForToolboxReady from client.go's Create and Start, drop the function + its TCP/HTTP imports, drop the toolboxReadyTimeout field, drop the ToolboxReadyTimeout/ DaemonStartTimeoutSec config plumbing in main.go + config.go, drop the two now-unreachable tests. Box readiness is now signalled by bx.Start(ctx) returning (which itself blocks on the vsock notification from the guest). Branched off chore/e2e-required-merge-gate (PR #724) so the e2e-cloud stack picks this up next dispatch.
ffc2b36 to
32b2eb4
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft. Targets
chore/e2e-required-merge-gate(PR #724) so the e2e-cloud stack picks this up on the next dispatch.waitForToolboxReadywas introduced infc88aa0b("feat: add agent-ready runtime catalog") and landed onmainvia PR #715 on 2026-06-10. It HTTP-pollshttp://127.0.0.1:<hostPort>/versionexpecting a daemon on guest TCP port 2280 — the Daytona-daemon interface that was never reimplemented after the Rust guest agent rewrite indbb11ec8(2026-04-01). The new agent binds vsock://2695 for gRPC + notifies the host via vsock://2696; nothing in the VM listens on TCP:2280, so libkrun's port-forward accepts the SYN and the guest end immediately RST's, timing out everyCREATE_BOX30 s in.Production data from the live Tokyo runner:
The exec request that fires immediately after CREATE_BOX (over the same vsock gRPC channel) always succeeds — confirming the box VM and the guest agent are healthy. The readiness probe itself is the bug.
Test plan
make test:integration:*(PyO3 / FFI path, bypasses the runner)go test ./pkg/boxlite/go build ./...+go vet ./...CREATE_BOXfordocker.io/library/alpine:3.23via e2e-cloud-testbox toolbox not ready after 30sjournalctl -u boxlite-runnerpost-deploytoolbox not readyper CREATE_BOXtoolboxlines at all (probe gone)Iteration plan (no full e2e-cloud needed)
Deploy Runner Binaryon this branch — rebuilds libboxlite.a + the runner Go binary and pushes via SSH+SCP.Deploy APIon this branch (no-op if no API changes, just to make sure the LB/health side is fresh).E2E test (Tokyo)standalone — runs the pytest suite against the freshly-deployed runner.That triple is ~30-40 min total vs running the full
e2e-cloud.ymlumbrella.Files
apps/runner/pkg/boxlite/client.go— drop the twowaitForToolboxReadycall sites + thetoolboxReadyTimeoutfield + plumbing.apps/runner/pkg/boxlite/toolbox_ports.go— deletewaitForToolboxReady, drop now-unusedio / net/http / timeimports.apps/runner/pkg/boxlite/toolbox_ports_test.go— delete the two tests that asserted against the dead probe.apps/runner/cmd/runner/main.go— dropToolboxReadyTimeout: time.Duration(cfg.DaemonStartTimeoutSec) * time.Second.apps/runner/cmd/runner/config/config.go— dropDaemonStartTimeoutSecenv var.