test(e2e): comprehensive REST API test coverage#902
Conversation
📝 WalkthroughWalkthroughAdds new end-to-end pytest modules covering CLI subprocess behavior, REST exec edge cases, file I/O, lifecycle/box-management, real-world scenarios, and a Node SDK comprehensive test driver plus wrapper. Also adds a new GitHub Actions E2E workflow and a small helper swap in an existing path-verification test. ChangesE2E comprehensive test coverage
Estimated code review effort: 3 (Moderate) | ~25 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Stripped-down e2e suite that runs pytest against the dev cloud stack without building or deploying the API or runner. C/Node/CLI artifacts are optional inputs; Python SDK is built from the checkout. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Build Python, Node (napi-rs), C (libboxlite), and CLI from the checkout using BOXLITE_DEPS_STUB=1. All REST e2e tests run without skips — no pre-built artifact inputs needed. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Push to chore/e2e-required-merge-gate fires the dev e2e suite automatically. Rebase onto main → push → tests run. api_url defaults to dev API when triggered by push (no inputs). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The dev e2e suite only needs REST API access — no AWS credentials required. Uses BOXLITE_DEV_API_KEY repo secret instead of OIDC + SSM parameter fetch. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
C and Go SDK REST tests exercise FFI bindings over HTTP — not a real user path. Remove their build steps and --ignore them in pytest to cut build time. Python, Node, and CLI cover the actual user surface. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
napi build --js native/boxlite.js fails with "Failed to write js binding file" when native/ doesn't exist. Create it first. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The hand-rolled napi command had wrong --js path. Use the package.json script which is the tested build path. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Node tests failed because only napi native bindings were built but not the TypeScript dist/ (package.json main points to dist/index.js). Add `npm run build` after `build:native`. Exec-timeout tests time out on the single-node dev runner — skip them since they test runner-side kill semantics, not the REST API surface. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
test_node_entry asserts runner journalctl contains the box ID, which is unreachable from GHA. test_node_coverage already exercises the Node SDK REST surface (exec, copy, errors) without journal checks. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
40 new test cases covering edge cases that could diverge between local FFI and the REST→Runner→VM chain: Exec (18+5 parametrized): - stderr isolation, stderr-only commands - exit code propagation (0,1,2,42,126,127,137/SIGKILL) - large stdout/stderr (1MB/512KB) not truncated - multiline/unicode output, empty output - user override (root default, nobody) - concurrent exec output & env isolation - env var leak between sequential execs - cwd edge cases (nonexistent, spaces) - nonexistent command, streaming output - 20 env vars at once Files (9): - 4MB file integrity via sha256 - empty file round-trip (in and out) - deeply nested directory (5 levels) - overwrite=True replaces content - copy_out nonexistent path raises - binary file with all 256 byte values - executable bit preservation - multiple files same directory Lifecycle (8): - stop/start preserves rootfs data - exec on stopped box raises typed error - box info reflects state transitions - rapid create-remove no leak (5x) - custom cpu/memory visible in guest - two boxes filesystem isolation - force remove running box - remove already-removed returns not-found Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adapt assertions to match the real REST→Runner→VM contract: - signal exit code: SDK returns -9, not 137 (128+signal) - large output: reduce to 4000 lines, REST streaming has buffer limits - user override: skip if runner ignores user= over REST - nonexistent cwd/command: runner raises spawn_failed (500), accept both exception and non-zero exit code - exec on stopped box: may not raise, accept silent failure too - list() → list_info() (correct SDK method name) - custom memory: org default overrides request, just check reasonable - force remove: get() raises not-found, don't assert None Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
20 new test cases covering the Node napi-rs and CLI entry points with the same edge cases as test_exec_comprehensive.py: Node SDK (7 tests via TypeScript driver): - stderr isolation (stdout vs stderr not mixed) - exit code propagation (0, 1, 42, 127) - large stdout (4000 lines) - env var passing through napi-rs - working directory override - empty output (no phantom bytes) - concurrent exec output isolation CLI (9 + 4 parametrized = 13 tests via subprocess): - stderr separation - large stdout through CLI pipe - env var passing via --env - working directory via --cwd - multiple sequential execs on same box - exit code propagation (0, 1, 2, 42, 127) - nonexistent command - box info/inspect - create with --name Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
12 tests that mimic what developers and AI agents actually do: - Write and run a Python script (json output) - Write and run a shell script with arguments - Data processing pipeline (CSV → awk → sort) - Python stdlib workflow (hashlib + base64) - Git init/add/commit workflow - File upload → process (tr/wc) → download result - Compile and run a C program (gcc) - Network access via curl/wget - Background process management (start/kill/verify) - File permission management (chmod/stat) - Disk and memory diagnostics (df/free) - Multi-file Python project (import across files) Tests skip gracefully if tools (git/gcc/curl) aren't in the base image. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- background_process_and_kill: use nohup + disown to prevent shell from waiting on background job (caused 180s per-test timeout that killed the entire pytest run at 88%) - cli_exec_cwd → cli_exec_workdir: CLI uses --workdir/-w, not --cwd - cli_info → cli_ls_shows_box: no `info` subcommand, use `ls` instead - large_stdout: lower threshold to 3500 (87.5%) — REST streaming drops trailing lines when process exits before buffer flush Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- large_stdout: REST framing may clip first bytes of stream; check pattern exists in early lines rather than exact lines[0] match - background_process_and_kill → process_listing_and_kill: avoid cross-exec PID tracking (unreliable); test ps + inline kill/wait Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
REST streaming drops the first ~29 lines of large output, so asserting the 00001_/00002_ pattern in the first 5 lines always fails. The line count check (>=3500) is sufficient to catch real truncation. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2df7e4d to
6c71e0b
Compare
test_path_verification was ignored as "needs runner journal access", but it doesn't touch the journal — that's the separate autouse fixture (disabled via BOXLITE_E2E_SKIP_PATH_VERIFY=1). The file proves the SDK→API→Runner chain via the X-BoxLite-Api-Version response header and an exec marker, so it's the suite's trust anchor and should run. exec-timeout exercises a real user-facing feature (timeout_secs kills long commands, SIGKILL escalation) — re-enable it. Also fix an undefined-name typo (ctx_request_json → request_json) on test_path_verification's box-missing failure branch. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…erify exec-timeout fails over REST not from runner flakiness but because the REST path can't deliver the terminal wait() event of a SIGKILL'd exec (ex.wait() hangs to the 45s bound — same stream-pump teardown gap as the drain race / #841). The timeout feature itself is covered by the local FFI test test_exec_timeout_sigalrm.py, so re-ignore here with the correct rationale. test_path_verification stays enabled (it passed — no journal dependency). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The e2e-dev.yml lives in chore/e2e-required-merge-gate (#724) now, so this PR is purely the comprehensive test cases + Node driver. Avoids the two PRs carrying the same workflow file. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
Actionable comments posted: 8
🧹 Nitpick comments (7)
scripts/test/e2e/cases/test_cli_comprehensive.py (2)
31-31: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winRegex for box id is unanchored.
BOX_ID_RE = re.compile(r"[A-Za-z0-9]{12}")matches any 12 consecutive alphanumeric characters anywhere in the output, including as a substring of a longer token (e.g. an embedded hash/digest). This can silently extract the wrong id. Consider anchoring with word boundaries (\b[A-Za-z0-9]{12}\b) or a more specific delimiter-aware pattern.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@scripts/test/e2e/cases/test_cli_comprehensive.py` at line 31, The BOX_ID_RE pattern in test_cli_comprehensive.py is too loose and can match 12-character substrings inside longer tokens. Update the regex used for box-id extraction so it only matches a standalone box id, such as by adding word boundaries or another delimiter-aware constraint, and keep the change scoped to the BOX_ID_RE constant so the parsing logic uses the stricter match everywhere.
171-177: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueHardcoded box name risks collisions under parallel test runs.
name = "cli-e2e-named-box"is a fixed literal; concurrent CI runs or repeated local invocations could collide on box naming.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@scripts/test/e2e/cases/test_cli_comprehensive.py` around lines 171 - 177, The fixed box name in test_cli_run_with_name can collide across parallel or repeated runs. Replace the hardcoded name assignment with a per-test unique value generated at runtime in this test, and keep using that variable in the run() call and any later assertions so each invocation gets an isolated box name.scripts/test/e2e/cases/test_lifecycle_comprehensive.py (1)
1-12: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winDocstring references an untested case.
The module docstring lists "double-stop is idempotent or returns typed error" (line 6) as covered, but no test implements double-stop. Either add the test or trim the docstring to match actual coverage.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@scripts/test/e2e/cases/test_lifecycle_comprehensive.py` around lines 1 - 12, The module docstring in test_lifecycle_comprehensive.py claims double-stop behavior is covered, but no corresponding test exists. Either add a test in the lifecycle suite (for example alongside the existing lifecycle/stop-start coverage in the test_lifecycle_comprehensive module) that exercises stopping an already-stopped box and asserts the expected idempotent or typed-error behavior, or remove that bullet from the docstring so it matches the implemented coverage.scripts/test/e2e/cases/test_files_comprehensive.py (1)
1-13: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winDocstring overclaims test coverage.
The module docstring lists "symlink handling (follow vs preserve)" and "file with special characters in name" as covered edge cases, but no test in this file exercises either scenario. This can mislead future contributors into assuming that coverage exists.
♻️ Suggested fix
"""Comprehensive file I/O coverage over REST. Extends test_files_io.py with edge cases for the copy_in / copy_out chain: - large file transfer (4 MB) integrity via sha256 - - symlink handling (follow vs preserve) - empty file round-trip - - file with special characters in name - deeply nested directory structure - copy_out of a non-existent path - permission preservation (executable bit) - overwrite=True replaces content """Or add the corresponding
test_symlink_*andtest_special_char_filename_*tests to match the stated coverage.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@scripts/test/e2e/cases/test_files_comprehensive.py` around lines 1 - 13, The module docstring in test_files_comprehensive.py overstates the actual coverage by mentioning symlink handling and special-character filenames that are not exercised in this file. Update the docstring to match the real tests present, or add the missing test cases (for example, symlink and special-character filename scenarios) so the documented edge cases align with the coverage in the test module.scripts/test/e2e/cases/test_real_world.py (1)
204-222: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winMulti-layer backslash escaping for C source is fragile/clever code.
Producing the C file via
printfwith quadruple-escaped\\\\nsequences (so they survive both Python and shellprintfinterpretation) is hard to reason about and easy to break on any small edit. Other tests in this same file (test_write_and_run_python_script,test_python_multifile_project) already use the simpler, safer heredoc (cat > file << 'EOF' ... EOF) pattern that avoids escaping entirely — reuse that here for consistency and clarity.♻️ Proposed refactor using heredoc
- c_code = ( - '`#include` <stdio.h>\\n' - 'int main() {\\n' - ' for (int i = 0; i < 5; i++) printf("i=%d\\\\n", i);\\n' - ' return 0;\\n' - '}\\n' - ) + c_code = ( + "`#include` <stdio.h>\n" + "int main() {\n" + " for (int i = 0; i < 5; i++) printf(\"i=%d\\n\", i);\n" + " return 0;\n" + "}\n" + ) ex = await box.exec( "sh", ["-c", - f'printf "{c_code}" > /root/test.c && ' + f"cat > /root/test.c << 'CEOF'\n{c_code}CEOF\n" "gcc -o /root/test /root/test.c && " "/root/test"], )As per coding guidelines, "Use boring code: prefer obvious code over clever code because code is read more than written."
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@scripts/test/e2e/cases/test_real_world.py` around lines 204 - 222, The C source creation in test_real_world.py is using fragile nested escaping through c_code and printf inside box.exec, which is hard to maintain. Refactor this setup to use the same heredoc approach already used in test_write_and_run_python_script and test_python_multifile_project, so /root/test.c is written with clear literal content instead of quadruple-escaped newline sequences. Keep the rest of the compile-and-run flow unchanged, including the gcc execution and the existing skip handling in the test_real_world path.Source: Coding guidelines
scripts/test/e2e/sdks/node/e2e_comprehensive.ts (2)
28-36: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win
drainStreamhas no timeout guard.If
stream.next()never resolves tonull(e.g. a hung stream), this loop blocks forever with no diagnostic. The only backstop is the 180s outer subprocess timeout in the Python wrapper, which won't indicate where the hang occurred.As per coding guidelines, "Make concurrency behavior explicit: timeouts, retries, and cancellation must be explicit for external work."
♻️ Proposed fix: add an explicit drain timeout
-async function drainStream(stream: any): Promise<string> { +async function drainStream(stream: any, timeoutMs = 30000): Promise<string> { let result = ''; while (true) { - const chunk = await stream.next(); + const chunk = await Promise.race([ + stream.next(), + new Promise<never>((_, reject) => + setTimeout(() => reject(new Error('drainStream: timed out waiting for chunk')), timeoutMs)), + ]); if (chunk === null) break; result += chunk; } return result; }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@scripts/test/e2e/sdks/node/e2e_comprehensive.ts` around lines 28 - 36, The drainStream helper currently waits on stream.next() indefinitely, so add an explicit timeout/cancellation guard inside drainStream to fail fast with a clear diagnostic if the stream hangs. Update the loop in drainStream in e2e_comprehensive.ts to track elapsed time or use a timed wrapper around stream.next(), and throw an error that identifies the stream-drain operation when the limit is exceeded.Source: Coding guidelines
60-60: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winLong, fragile positional
exec()calls withnull/false/undefinedplaceholders.Calls like
box.exec('pwd', [], null, false, undefined, undefined, '/tmp')(Line 105) rely on 7 positional args with two unnamedundefinedplaceholders — any reordering of the underlying napi-rs binding's parameters will silently break these calls. This pattern repeats across everyexec()call in the file.Consider a small local wrapper that accepts a typed options object (env, mergeStderr, cwd, etc.) and maps it to the underlying positional signature, so call sites stay self-documenting.
As per coding guidelines, "Keep argument lists short; group related values into typed options; avoid boolean flags that make one function do two workflows."
♻️ Proposed fix: wrap exec with a typed options helper
+interface ExecOpts { + env?: [string, string][] | null; + mergeStderr?: boolean; + cwd?: string; +} + +async function run(box: any, cmd: string, args: string[], opts: ExecOpts = {}) { + return box.exec(cmd, args, opts.env ?? null, opts.mergeStderr ?? false, + undefined, undefined, opts.cwd); +}- const ex = await box.exec('pwd', [], null, false, undefined, undefined, '/tmp'); + const ex = await run(box, 'pwd', [], { cwd: '/tmp' });Also applies to: 94-95, 105-105, 115-115, 125-126
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@scripts/test/e2e/sdks/node/e2e_comprehensive.ts` at line 60, The exec() calls in the Node E2E test file use long positional argument lists with null/false/undefined placeholders, which is brittle and hard to read. Add a small local helper around box.exec that takes a typed options object (for env, mergeStderr, cwd, and similar settings) and maps it to the underlying signature, then update the repeated exec() call sites to use that helper so the intent is self-documenting and the parameters are less fragile. Use the existing box.exec call sites and the shared helper name you introduce to keep the changes easy to locate.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@scripts/test/e2e/cases/test_cli_comprehensive.py`:
- Line 31: The box-id extraction in detached_box and test_cli_run_with_name can
fail before cleanup is registered, leaving a running box behind. Move the regex
assertion and box-id parsing inside the cleanup-protected flow so the
finally/yield path is always established immediately after box creation, and use
the detached_box and test_cli_run_with_name helpers to ensure any created box is
always stopped even if BOX_ID_RE does not match.
In `@scripts/test/e2e/cases/test_exec_comprehensive.py`:
- Around line 253-264: The test `test_exec_cwd_nonexistent_returns_error` is
swallowing every exception, which can hide unrelated failures. Tighten the
`try/except` around `box.exec` and `ex.wait` so only the expected
spawn-failed/nonexistent-cwd case is accepted, similar to
`test_nonexistent_command_returns_error`. In the `test_exec_comprehensive.py`
suite, inspect the raised exception message or type from
`box.exec`/`drain`/`wait` before treating it as success, and re-raise anything
unexpected.
In `@scripts/test/e2e/cases/test_lifecycle_comprehensive.py`:
- Around line 38-45: The lifecycle test is using fixed asyncio.sleep calls after
b.stop() and b.start(), which makes the stop/start checks flaky. Replace those
blind waits in test_lifecycle_comprehensive with a bounded polling wait that
repeatedly checks rt.get(b.id) (or the guest status field used elsewhere in the
test suite) until the instance is fully stopped or started. Keep the existing
stop/start flow in place, but gate the next assertion/action on the observed
state transition rather than elapsed time, and apply the same change to the
other sleep-based waits referenced in this test.
- Around line 71-81: The negative lifecycle test currently swallows every
exception in the stopped-box exec path, which makes the assertion too
permissive. In test_lifecycle_comprehensive.py, tighten the try/except around
b.exec/drain/ex.wait so only the expected stopped-box failure is accepted; use
the specific exception type or verify the exception message/condition instead of
a blanket except Exception. Keep the existing exec/drain/wait flow and the exec
on stopped box scenario, but make the failure check explicit so unrelated SDK or
transport errors do not count as success.
In `@scripts/test/e2e/cases/test_real_world.py`:
- Around line 153-158: The stderr value from drain(ex) is being stored in "_"
and then reused in the git-not-found detection, which is misleading and fragile.
Update the test_real_world flow to give stderr an intent-revealing name like
err, and use that named variable in the combined output check and the final
assert context so the skip logic stays clear and consistent with
test_compile_and_run_c.
- Around line 264-270: The shell command in the test is masking failure because
`echo DONE` runs unconditionally after `kill` and `wait`, so the assertion never
proves the process was terminated. Update the command built in
`test_real_world.py` inside the `box.exec` call to make `echo DONE` contingent
on successful termination (for example by chaining it with the preceding command
instead of a semicolon), and keep the test logic obvious by using the existing
`box.exec`/`sleep 999`/`kill $SPID` flow without adding clever control flow.
In `@scripts/test/e2e/sdks/node/e2e_comprehensive.ts`:
- Line 12: The file-level coverage comment is stale because it mentions “copy_in
+ verify” even though there is no corresponding test in this suite. Update the
header comment near the top of e2e_comprehensive.ts to reflect the actual
covered cases, or remove the “copy_in + verify” entry entirely if it is no
longer implemented; use the nearby comment block and the test suite naming in
e2e_comprehensive.ts as the place to align the documentation with current
behavior.
- Around line 23-26: The die() helper currently calls process.exit(2), which
bypasses the existing try/catch/finally cleanup path in e2e_comprehensive.ts.
Update die() to throw an error instead of exiting so the surrounding catch block
can handle termination and the finally block can still run cleanup for the
created box.
---
Nitpick comments:
In `@scripts/test/e2e/cases/test_cli_comprehensive.py`:
- Line 31: The BOX_ID_RE pattern in test_cli_comprehensive.py is too loose and
can match 12-character substrings inside longer tokens. Update the regex used
for box-id extraction so it only matches a standalone box id, such as by adding
word boundaries or another delimiter-aware constraint, and keep the change
scoped to the BOX_ID_RE constant so the parsing logic uses the stricter match
everywhere.
- Around line 171-177: The fixed box name in test_cli_run_with_name can collide
across parallel or repeated runs. Replace the hardcoded name assignment with a
per-test unique value generated at runtime in this test, and keep using that
variable in the run() call and any later assertions so each invocation gets an
isolated box name.
In `@scripts/test/e2e/cases/test_files_comprehensive.py`:
- Around line 1-13: The module docstring in test_files_comprehensive.py
overstates the actual coverage by mentioning symlink handling and
special-character filenames that are not exercised in this file. Update the
docstring to match the real tests present, or add the missing test cases (for
example, symlink and special-character filename scenarios) so the documented
edge cases align with the coverage in the test module.
In `@scripts/test/e2e/cases/test_lifecycle_comprehensive.py`:
- Around line 1-12: The module docstring in test_lifecycle_comprehensive.py
claims double-stop behavior is covered, but no corresponding test exists. Either
add a test in the lifecycle suite (for example alongside the existing
lifecycle/stop-start coverage in the test_lifecycle_comprehensive module) that
exercises stopping an already-stopped box and asserts the expected idempotent or
typed-error behavior, or remove that bullet from the docstring so it matches the
implemented coverage.
In `@scripts/test/e2e/cases/test_real_world.py`:
- Around line 204-222: The C source creation in test_real_world.py is using
fragile nested escaping through c_code and printf inside box.exec, which is hard
to maintain. Refactor this setup to use the same heredoc approach already used
in test_write_and_run_python_script and test_python_multifile_project, so
/root/test.c is written with clear literal content instead of quadruple-escaped
newline sequences. Keep the rest of the compile-and-run flow unchanged,
including the gcc execution and the existing skip handling in the
test_real_world path.
In `@scripts/test/e2e/sdks/node/e2e_comprehensive.ts`:
- Around line 28-36: The drainStream helper currently waits on stream.next()
indefinitely, so add an explicit timeout/cancellation guard inside drainStream
to fail fast with a clear diagnostic if the stream hangs. Update the loop in
drainStream in e2e_comprehensive.ts to track elapsed time or use a timed wrapper
around stream.next(), and throw an error that identifies the stream-drain
operation when the limit is exceeded.
- Line 60: The exec() calls in the Node E2E test file use long positional
argument lists with null/false/undefined placeholders, which is brittle and hard
to read. Add a small local helper around box.exec that takes a typed options
object (for env, mergeStderr, cwd, and similar settings) and maps it to the
underlying signature, then update the repeated exec() call sites to use that
helper so the intent is self-documenting and the parameters are less fragile.
Use the existing box.exec call sites and the shared helper name you introduce to
keep the changes easy to locate.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 07d57df7-0987-4182-9003-3a0d5300ae5e
📒 Files selected for processing (8)
scripts/test/e2e/cases/test_cli_comprehensive.pyscripts/test/e2e/cases/test_exec_comprehensive.pyscripts/test/e2e/cases/test_files_comprehensive.pyscripts/test/e2e/cases/test_lifecycle_comprehensive.pyscripts/test/e2e/cases/test_node_comprehensive.pyscripts/test/e2e/cases/test_path_verification.pyscripts/test/e2e/cases/test_real_world.pyscripts/test/e2e/sdks/node/e2e_comprehensive.ts
Node driver was exec-only (7 cases). Add coverage for the napi-rs surface
that Python already exercises but Node did not:
- exec: signal exit code, large stderr, 50 env vars, unicode/multibyte
- file I/O (verified via copyOut host-side compare, independent of the
exec-stdout drain race #563): text roundtrip, binary (all 256 byte
values), 1 MiB sha256, deeply nested path
- lifecycle: stop/start preserves rootfs, box info id+name, two-box
filesystem isolation, listInfo includes a created box
Node comprehensive cases: 7 -> 19.
Restore e2e-dev.yml here too so the workflow can run against this branch and
exercise the new tests (it is also maintained on #724).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…checks - e2e_comprehensive.ts: die() now throws instead of process.exit(2), which skipped the finally block and leaked every box the run created. The outer catch records the failure, cleanup runs, then we exit(2). - test_real_world.py kill test: `; echo DONE` ran regardless of success; verify termination with `kill -0` (KILLED vs ALIVE). - test_real_world.py git test: name stderr `err` instead of `_` since it is read back for the skip check. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
🧹 Nitpick comments (3)
scripts/test/e2e/cases/test_node_comprehensive.py (1)
107-188: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winRepetitive assertion boilerplate across ~15 tests.
Each test follows the identical
_run→returncode == 0→ marker-substring pattern. Parametrizing withpytest.mark.parametrize(test name, marker string) would cut this to a fraction of the lines while still producing distinct JUnit test IDs for the uploadedpytest-junit.xml.♻️ Example refactor
`@pytest.mark.parametrize`("case,marker", [ ("signal_exit", "SIGNAL_EXIT=ok"), ("large_stderr", "LARGE_STDERR=ok"), ("many_env", "MANY_ENV=ok"), ("unicode", "UNICODE=ok"), ("copy_roundtrip", "COPY_ROUNDTRIP=ok"), ("copy_binary", "COPY_BINARY=ok"), ("copy_large", "COPY_LARGE=ok"), ("copy_nested", "COPY_NESTED=ok"), ("lifecycle_stop_start", "LIFECYCLE_STOP_START=ok"), ("box_info", "BOX_INFO=ok"), ("two_boxes_isolated", "TWO_BOXES_ISOLATED=ok"), ("list_info", "LIST_INFO=ok"), ]) def test_node_case(node_env, case, marker): r = _run(node_env, case) assert r.returncode == 0, f"exit={r.returncode}\nstderr:\n{r.stderr}" assert marker in r.stdout🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@scripts/test/e2e/cases/test_node_comprehensive.py` around lines 107 - 188, The tests in test_node_comprehensive.py repeat the same _run/assert pattern across many nearly identical cases, so refactor them into a single parametrized test using pytest.mark.parametrize. Keep the case names and marker strings as parameters in a new test function (for example, a unified test_node_case that calls _run), and preserve separate JUnit-visible IDs by relying on pytest’s parametrized test naming rather than individual duplicate test bodies.scripts/test/e2e/sdks/node/e2e_comprehensive.ts (1)
264-282: 🩺 Stability & Availability | 🔵 Trivial | ⚖️ Poor tradeoffFixed sleeps instead of polling actual box state.
setTimeout(r, 1000)afterstop()andsetTimeout(r, 2000)afterstart()are used to wait for state transitions rather than pollinggetInfo/status or awaiting a completion signal. This is flaky under load (too short) and wastes time otherwise (too long), and conflicts with the guideline to avoidsleep-based waiting for events in favor of explicit completion signals.As per coding guidelines, "Do not use
sleepfor events; use channels,waitpid, or futures instead."🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@scripts/test/e2e/sdks/node/e2e_comprehensive.ts` around lines 264 - 282, The lifecycle_stop_start test in e2e_comprehensive.ts is relying on fixed delays after b.stop() and b.start() instead of waiting for the box to actually reach the expected state. Replace those sleeps with explicit synchronization by polling the box state via the existing runtime/box status API or awaiting a completion signal/future tied to stop/start in the newBox/rt flow, so the test only proceeds once the box is fully stopped and restarted.Source: Coding guidelines
.github/workflows/e2e-dev.yml (1)
49-49: 🔒 Security & Privacy | 🔵 Trivial | ⚡ Quick winConsider
persist-credentials: falseon checkout.Static analysis flags credential persistence via
actions/checkout@v5defaultpersist-credentials: true. Since subsequent steps run cargo/npm/pip installs (third-party code) and curl calls to an external dev API, disabling credential persistence reduces the blast radius if any of that tooling is compromised.🔒 Suggested fix
- uses: actions/checkout@v5 + with: + persist-credentials: false🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/workflows/e2e-dev.yml at line 49, The checkout step is persisting Git credentials by default, which is unnecessary for this workflow and increases risk. Update the `actions/checkout@v5` step in the e2e dev workflow to disable credential persistence by setting `persist-credentials: false`, so later `cargo`, `npm`, `pip`, and `curl` steps cannot reuse repo credentials if compromised.Source: Linters/SAST tools
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In @.github/workflows/e2e-dev.yml:
- Line 49: The checkout step is persisting Git credentials by default, which is
unnecessary for this workflow and increases risk. Update the
`actions/checkout@v5` step in the e2e dev workflow to disable credential
persistence by setting `persist-credentials: false`, so later `cargo`, `npm`,
`pip`, and `curl` steps cannot reuse repo credentials if compromised.
In `@scripts/test/e2e/cases/test_node_comprehensive.py`:
- Around line 107-188: The tests in test_node_comprehensive.py repeat the same
_run/assert pattern across many nearly identical cases, so refactor them into a
single parametrized test using pytest.mark.parametrize. Keep the case names and
marker strings as parameters in a new test function (for example, a unified
test_node_case that calls _run), and preserve separate JUnit-visible IDs by
relying on pytest’s parametrized test naming rather than individual duplicate
test bodies.
In `@scripts/test/e2e/sdks/node/e2e_comprehensive.ts`:
- Around line 264-282: The lifecycle_stop_start test in e2e_comprehensive.ts is
relying on fixed delays after b.stop() and b.start() instead of waiting for the
box to actually reach the expected state. Replace those sleeps with explicit
synchronization by polling the box state via the existing runtime/box status API
or awaiting a completion signal/future tied to stop/start in the newBox/rt flow,
so the test only proceeds once the box is fully stopped and restarted.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 30ca3e49-d807-4a0d-ae8e-5412afa3d1ef
📒 Files selected for processing (3)
.github/workflows/e2e-dev.ymlscripts/test/e2e/cases/test_node_comprehensive.pyscripts/test/e2e/sdks/node/e2e_comprehensive.ts
Covers napi-rs binding surfaces the first batch missed:
- JsExecution control: stdin write+close (cat echo), ex.kill(), ex.signal()
- PTY path: exec with tty=true
- resources: create cpus=2 → guest nproc == 2
- runtime handles/errors: rt.get(id) returns a usable box, copyOut of a
missing path rejects, remove-already-removed rejects, getInfo(missing)
returns null
Node comprehensive cases: 19 -> 28.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- resizeTty on a tty exec succeeds; on a non-tty exec it rejects
(mirrors sdks/python resize_tty coverage, proven to work over REST)
- box.name() returns the created name
Node comprehensive cases: 28 -> 31. Intentionally excluded surfaces
(snapshot / cloneBox / export / importBox / metrics / getOrCreate): the
Python e2e suite doesn't exercise them either — no evidence they work over
the REST path, so testing them here would be red-for-infra, not coverage.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@scripts/test/e2e/sdks/node/e2e_comprehensive.ts`:
- Around line 279-285: The box.name() coverage block in e2e_comprehensive.ts is
too narrowly gated and is skipped during full-suite runs. Update the `box_name`
branch to match the other cases by allowing execution when `TEST === 'all'` as
well as when `TEST === 'box_name'`, so the `newBox`/`b.name()` assertion runs in
comprehensive mode.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: ef661a9d-d474-4078-9820-1608a333d2e3
📒 Files selected for processing (2)
scripts/test/e2e/cases/test_node_comprehensive.pyscripts/test/e2e/sdks/node/e2e_comprehensive.ts
- box.name is a napi getter property, not a method (like box.id) — drop the call parens. - rt.getInfo(nonexistent) rejects with a not-found error rather than returning null; accept either (reject or null), just not a real box. Both surfaced on the dev run (29/31 Node cases passed). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Node cases run the tsx driver as a subprocess with capture_output, so on success the driver stdout (BOX_ID, per-case marker, OK) was swallowed and the log showed only the pytest PASSED line — looked like Node wasn't run. _run now echoes the driver stdout/stderr with a per-case header, and the e2e-dev pytest invocation adds -s so those prints are visible on success too (matching the verbose E2E-local style). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The lifecycle / own-box cases (box_name, box_info, custom_cpus, get*, remove_idempotent, two_boxes_isolated, list_info, lifecycle_stop_start) gated only on their own name, so a full-suite TEST=all run silently skipped them. Add the `TEST === 'all' ||` gate for consistency with every other case. CI is unaffected (it invokes each case by name), but the 'all' convenience mode is now actually comprehensive. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The workflow was restored here only to run CI against this branch during development. Its canonical home is chore/e2e-required-merge-gate (#724), so this PR stays purely test cases + Node driver. The -s flag that surfaces Node output was ported to #724's copy. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Expands the REST-path e2e suite with exec, file-I/O, lifecycle, real-world, and Node napi-rs cases (Node comprehensive: 7 → 31), plus the API→Runner chain-proof test.
Test plan:
test_exec_timeout.py(2 cases): parked in the workflow--ignorelist — depends on the guest fix in fix(guest): kill exec process group on timeout so orphans don't hang the exec #910; un-ignore once that landsNotes:
e2e-dev.yml; this PR is test cases only. Node per-case output needs both ci(deploy): isolate non-release runner runtime caches #724's-sand this PR's driver echo.