feat(bench): Python REST API benchmark suite — 8 scenarios#592
Open
G4614 wants to merge 4 commits into
Open
Conversation
G4614
added a commit
to G4614/boxlite
that referenced
this pull request
May 26, 2026
macOS clippy on PR boxlite-ai#592 caught `unused import: super::*` in the tests submodule: the only test was already `#[cfg(target_os = "linux")]`-gated, so on macOS the module compiled empty and the `use super::*` had nothing to import → `-D warnings` fails. Move the cfg to the mod declaration so the whole tests block is elided off-Linux. No behavior change on Linux. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
G4614
changed the title
G4614
changed the title
Contributor
Author
REST API Cold-Start Benchmark —
|
| Metric | p50 | p90 | p99 | mean | stdev |
|---|---|---|---|---|---|
api_create_wall_ms (client-side POST /boxes) |
1948 | 2893 | 3475 | 2222 | 619 |
runner_create_ms (runner-reported VM create) |
566 | 576 | 580 | 566 | 8 |
first_exec_wall_ms (first exec after boot) |
1579 | 2614 | 4261 | 1866 | 1051 |
total_ready_wall_ms (create + settle + first exec) |
5910 | 7177 | 8221 | 6088 | 1059 |
Observations
- Runner VM creation is very consistent — p50=566ms, stdev=8ms. The variance is nearly zero.
- API wall clock is ~4x runner create — the ~1.7s delta is API scheduling + network RTT + runner queue.
- First exec latency is high-variance — p99=4.3s, likely due to guest userspace init (systemd/agent-base services).
boot_duration_msnot reported by the runner metrics endpoint (field is omitted/zero).- Total time to first useful exec: ~6s p50, ~8s p99.
Raw per-iteration data
# api_wall runner_create first_exec
1* 2573ms 564ms 984ms (warmup)
2 1713ms 565ms 1779ms
3 2731ms 580ms 2156ms
4 2625ms 568ms 1285ms
5 1686ms 575ms 1338ms
6 1984ms 560ms 1051ms
7 1879ms 572ms 1579ms
8 1892ms 554ms 4444ms
9 1948ms 558ms 2117ms
10 3540ms 566ms 1048ms
Script: scripts/test/e2e/bench_rest_startup.py (added in PR #746)
Report JSON: schema-compatible with bench harness v1.0.
G4614
added a commit
that referenced
this pull request
Jun 17, 2026
Measures box creation latency through the full REST path by: 1. POST /boxes wall clock (client-side) 2. GET /boxes/:id/metrics (runner-reported create_duration_ms) 3. First exec latency after boot Emits schema v1.0 JSON report compatible with bench harness (#592). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…k suite
Replace the local-only Rust benchmark harness (libkrun, Linux-only) with a
Python REST API benchmark suite that runs against any remote BoxLite API.
8 scenarios covering latency, throughput, density, and stability:
- latency-cold-start: POST /boxes + runner metrics + first exec
- latency-exec: single exec on warm box
- latency-lifecycle: full create → exec → delete cycle
- throughput-exec-serial: 20 serial execs, execs/sec
- throughput-exec-parallel: 10 concurrent execs, batch wall time
- density-parallel-create: 5 concurrent box creates
- stability-churn: 10 create+exec+delete cycles, drift tracking
- stability-exec-loop: 50 execs on one box, degradation check
CLI: python -m bench {list,run,compare}
Auth: reuses e2e_auth.py (env vars or credentials.toml)
Report: schema v1.0 JSON with p50/p90/p99/mean/stdev aggregates
Includes baseline results from dev.boxlite.ai run (2026-06-17).
Contributor
Author
Benchmark Results — dev.boxlite.ai (2026-06-17)Replaced the Rust local-only bench harness with a Python REST API benchmark suite. Ran all 8 scenarios against Latency
Throughput
Density
Stability
Key Takeaways
How to reproducecd scripts/test/e2e
export BOXLITE_E2E_API_URL=https://api.dev.boxlite.ai/api
export BOXLITE_E2E_API_KEY=blk_live_...
export BOXLITE_E2E_AUTH=api-key
.venv/bin/python -m bench run all --runs 5 --out-dir bench/results/ |
G4614
changed the title
|
Note Currently processing new changes in this PR. This may take a few minutes, please wait... ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (4)
📝 WalkthroughWalkthroughAdds a Python E2E benchmark harness for the BoxLite REST API. The suite includes a shared authentication module ( ChangesE2E Benchmark Harness for BoxLite REST API
Sequence Diagram(s)sequenceDiagram
participant User
participant CLI as bench __main__.py
participant Harness as harness.py
participant Auth as e2e_auth.py
participant API as BoxLite /v1 API
User->>CLI: bench run --scenario latency-exec --runs 5
CLI->>Harness: run_scenario("latency-exec", runs=5, warmup=1)
loop each iteration
Harness->>Auth: auth_context()
Auth-->>Harness: E2EAuthContext (token, base_url)
Harness->>API: POST /v1/boxes (create_box)
API-->>Harness: box_id
Harness->>API: POST /v1/boxes/{id}/exec (exec_command)
API-->>Harness: exec result
Harness->>API: DELETE /v1/boxes/{id} (delete_box)
API-->>Harness: 204
Harness-->>CLI: metrics dict {exec_wall_ms: ...}
end
CLI->>Harness: build_report(samples)
Harness-->>CLI: report JSON
CLI->>User: print aggregates + write bench/results/latency-exec.json
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 15
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@scripts/test/e2e/bench/__main__.py`:
- Around line 110-112: The code writes to the file specified by args.out without
creating parent directories first, causing write_text() to fail if those
directories don't exist. Before writing to Path(args.out), create the parent
directories by calling mkdir(parents=True, exist_ok=True) on the parent property
of the Path object to ensure all nested directories are created and handle cases
where they already exist.
- Around line 164-165: The direction arrow logic in the comparison is inverted.
In the line where direction is assigned, the condition evaluates `(cv > bv) !=
higher`, which produces the opposite arrow for metrics where higher values are
better. Change the inequality operator != to == so that when higher_is_better is
true and the current value exceeds the baseline value, the up arrow is displayed
instead of the down arrow, correctly indicating an improvement.
- Around line 151-152: The field value retrieval for variables `bv` and `cv`
silently falls back to "p99" when the requested field is missing, which can hide
typos and misconfigurations. Instead of using nested get() calls with automatic
fallback, explicitly check if the requested field exists in both `ba` and `ca`
dictionaries before retrieving values. If the field is missing, raise an
informative error that identifies the missing field, ensuring that misconfigured
comparisons are caught immediately rather than producing incorrect results.
In `@scripts/test/e2e/bench/README.md`:
- Around line 33-53: The scenario tables in the README file are missing blank
lines before and after them, which violates the MD058 markdown linting rule. Add
blank lines between the section headers (like "### Latency", "### Throughput",
etc.) and their corresponding tables, and also add blank lines after each table
before the next section header or table begins. Ensure all four scenario table
sections (latency-cold-start, throughput-exec-serial, density-parallel-create,
and stability-churn tables) have proper spacing around them.
In `@scripts/test/e2e/bench/scenarios/density_parallel_create.py`:
- Around line 19-37: The exception handling in this code is insufficient because
if any future raises an exception during the results collection list
comprehension, the finally cleanup block will not execute, leaving orphan boxes.
Move the try-finally block to wrap the entire results collection and ID
extraction logic. Instead of collecting all results first and then extracting
bids, incrementally accumulate the box IDs (bids) and per-box times (per_box) as
each future completes successfully. This way, if any future fails partway
through, the finally block will still execute and delete any boxes that were
successfully created up to that point.
In `@scripts/test/e2e/bench/scenarios/latency_cold_start.py`:
- Around line 23-26: The code uses truthy checks with `if
rm.get("create_duration_ms"):` and `if rm.get("boot_duration_ms"):` which
incorrectly discard zero values as falsy. Replace these truthy checks with
explicit `is not None` comparisons so that valid zero duration metrics are
preserved in the metrics dictionary. Change both the create_duration_ms and
boot_duration_ms checks to use `is not None` instead of relying on truthiness
evaluation.
- Around line 29-30: The exec_command() function call returns a tuple of
(status, body) but the code does not capture or validate the status code before
recording the latency metric. Modify the code to unpack both the status and body
from the exec_command() call, validate that the status indicates success (a 2xx
HTTP status code), and only record the first_exec_wall_ms latency metric if the
execution was successful. This prevents non-successful responses from being
counted as valid samples.
In `@scripts/test/e2e/bench/scenarios/latency_exec.py`:
- Around line 17-21: The warmup and measured exec_command calls do not validate
that responses are successful, so failed requests with non-2xx status codes
still produce latency samples and skew benchmark results. Modify the
exec_command calls for both the warmup call (with "warmup" argument) and the
measured call (with "bench" argument) to check the response status code and
raise an exception if it is not in the 2xx range, ensuring that only successful
executions contribute to the latency measurements and failed requests are caught
immediately.
In `@scripts/test/e2e/bench/scenarios/latency_lifecycle.py`:
- Around line 20-25: The code records exec_ms and delete_ms metrics without
validating that the operations succeeded. Modify the exec_command call to check
its return value for success before recording the exec_ms metric, and check the
HTTP response status from the api DELETE call to ensure it returns a 2xx status
code before recording the delete_ms metric. If either operation fails, handle it
appropriately by raising an exception or logging an error to prevent metrics
from recording failed operations as successful.
In `@scripts/test/e2e/bench/scenarios/stability_churn.py`:
- Around line 22-29: The returned metrics dictionary has "cycles" as the first
key, but the CLI sample contract expects the first metric key to be a wall-time
value in milliseconds for proper sample.wall_ms derivation. Reorder the
dictionary keys in the return statement to place a total_wall_ms metric first
(which should represent the total elapsed time across all cycles in
milliseconds), followed by the existing metrics including cycles,
first_cycle_ms, last_cycle_ms, mean_cycle_ms, max_cycle_ms, and drift_ms.
- Line 17: The exec_command call at the churn cycle line is ignoring the
returned (status, body) tuple, which means failed command executions are still
counted as successful cycles and can skew stability metrics. Capture the
(status, body) return value from the exec_command function call, check that the
status indicates success, and only count that iteration as a successful cycle if
the exec was successful. If the exec fails, either skip counting that cycle or
handle the failure appropriately based on the test's stability validation logic.
In `@scripts/test/e2e/bench/scenarios/stability_exec_loop.py`:
- Around line 25-32: The returned dictionary in the stability_exec_loop.py file
has exec_count as the first key, but the CLI expects a wall-time metric to be
the first key to properly store it as sample.wall_ms. Reorder the dictionary
keys so that the mean execution time metric (mean_exec_ms) appears first,
followed by the other metrics like exec_count, max_exec_ms, first_10_mean_ms,
last_10_mean_ms, and degradation_ms. This ensures the CLI correctly captures and
aggregates wall-time data for this scenario.
- Around line 14-20: The exec_command calls in the warmup iteration and the
measurement loop do not validate the HTTP response status, allowing failed
executions to be silently timed and reported as successful samples. Add status
validation after both the warmup exec_command call at the beginning and the
exec_command call inside the EXEC_COUNT loop to ensure the commands executed
successfully before recording their timing measurements. This will prevent
failed executions from being included in the performance data.
In `@scripts/test/e2e/bench/scenarios/throughput_exec_parallel.py`:
- Around line 11-14: The _do_exec function does not validate whether
exec_command succeeds before recording its execution time in metrics. Add a
check after calling exec_command to verify it completed successfully, and only
return the timing measurement if the command succeeded. Apply the same
validation pattern to the other parallel worker function mentioned in the
comment (around lines 20-26) to ensure failed executions are not incorrectly
recorded as successful operations in the per_exec metrics.
In `@scripts/test/e2e/bench/scenarios/throughput_exec_serial.py`:
- Around line 14-19: The serial throughput benchmark is not validating the
response status from exec_command calls, so both successful and failed
executions are being counted in the elapsed time and throughput metrics. Capture
the return values from the exec_command calls (both the warmup call on line 14
and the calls within the loop on line 18), check that they represent successful
responses (2xx status), and only include successful executions in the throughput
calculation. Consider skipping or separately tracking failed executions so they
don't artificially inflate the throughput numbers.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: cf98cacc-90ac-4f1b-8dff-b9fe0fdf853c
📒 Files selected for processing (22)
bench/results/density-parallel-create.jsonbench/results/latency-cold-start.jsonbench/results/latency-exec.jsonbench/results/latency-lifecycle.jsonbench/results/stability-churn.jsonbench/results/stability-exec-loop.jsonbench/results/throughput-exec-parallel.jsonbench/results/throughput-exec-serial.jsonscripts/test/e2e/bench/README.mdscripts/test/e2e/bench/__init__.pyscripts/test/e2e/bench/__main__.pyscripts/test/e2e/bench/harness.pyscripts/test/e2e/bench/scenarios/__init__.pyscripts/test/e2e/bench/scenarios/density_parallel_create.pyscripts/test/e2e/bench/scenarios/latency_cold_start.pyscripts/test/e2e/bench/scenarios/latency_exec.pyscripts/test/e2e/bench/scenarios/latency_lifecycle.pyscripts/test/e2e/bench/scenarios/stability_churn.pyscripts/test/e2e/bench/scenarios/stability_exec_loop.pyscripts/test/e2e/bench/scenarios/throughput_exec_parallel.pyscripts/test/e2e/bench/scenarios/throughput_exec_serial.pyscripts/test/e2e/lib/e2e_auth.py
…pare - exec_command() now validates HTTP status centrally (fail-fast on non-2xx) - density_parallel_create: track created box IDs incrementally for cleanup - stability_churn/exec_loop: add total_wall_ms as first metric key - latency_cold_start: use `is not None` for zero-value runner metrics - __main__.py: mkdir parents for --out, strict --on field check, fix arrow - README.md: add blank lines around markdown tables
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
scripts/test/e2e/bench/__main__.py (1)
153-176:⚠️ Potential issue | 🟡 Minor | ⚡ Quick winTrack comparison count to detect false passes from field typos.
If
--oncontains a typo (e.g.,p9instead ofp99), every metric will printSKIP: missing 'p9'and the loop completes withregressions = [], producingPASS: no regressions. This hides the misconfiguration.Track how many metrics were actually compared and exit with an error if zero:
Suggested fix
all_names = sorted(set(base_aggs) | set(curr_aggs)) + compared = 0 for name in all_names: ba = base_aggs.get(name) ca = curr_aggs.get(name) if not ba or not ca: status = "NEW" if ca else "REMOVED" print(f" {name:35s} {status}") continue if field not in ba or field not in ca: print(f" {name:35s} SKIP: missing '{field}'") continue bv = float(ba[field]) cv = float(ca[field]) + compared += 1 higher = ba.get("higher_is_better", False) if bv == 0: ratio = 0.0 elif higher: ratio = (bv - cv) / bv else: ratio = (cv - bv) / bv regressed = ratio > threshold marker = " ← REGRESSION" if regressed else "" direction = "→" if cv == bv else ("↑" if cv > bv else "↓") print(f" {name:35s} {bv:10.1f} → {cv:10.1f} {direction} {ratio:+.1%}{marker}") if regressed: regressions.append(name) + if compared == 0: + print(f"\nERROR: No metrics were compared. Check that '{field}' exists in both reports.") + sys.exit(1) + if regressions:🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@scripts/test/e2e/bench/__main__.py` around lines 153 - 176, Add a counter variable to track how many metrics were actually compared (not skipped) in the loop where field values are checked and processed. Initialize this counter before the loop starts. Increment the counter whenever a metric passes the field existence check (when field is in both ba and ca) and the ratio calculation proceeds. After the loop completes, add a validation check that exits with an error if the counter is zero, indicating no metrics were compared due to possible field name typos. This prevents false passes when all metrics are skipped and regressions remains empty.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@scripts/test/e2e/bench/__main__.py`:
- Around line 153-176: Add a counter variable to track how many metrics were
actually compared (not skipped) in the loop where field values are checked and
processed. Initialize this counter before the loop starts. Increment the counter
whenever a metric passes the field existence check (when field is in both ba and
ca) and the ratio calculation proceeds. After the loop completes, add a
validation check that exits with an error if the counter is zero, indicating no
metrics were compared due to possible field name typos. This prevents false
passes when all metrics are skipped and regressions remains empty.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 367ab75a-21fb-49e3-9800-523e67985511
📒 Files selected for processing (7)
scripts/test/e2e/bench/README.mdscripts/test/e2e/bench/__main__.pyscripts/test/e2e/bench/harness.pyscripts/test/e2e/bench/scenarios/density_parallel_create.pyscripts/test/e2e/bench/scenarios/latency_cold_start.pyscripts/test/e2e/bench/scenarios/stability_churn.pyscripts/test/e2e/bench/scenarios/stability_exec_loop.py
✅ Files skipped from review due to trivial changes (1)
- scripts/test/e2e/bench/README.md
🚧 Files skipped from review as they are similar to previous changes (5)
- scripts/test/e2e/bench/scenarios/stability_exec_loop.py
- scripts/test/e2e/bench/scenarios/density_parallel_create.py
- scripts/test/e2e/bench/scenarios/stability_churn.py
- scripts/test/e2e/bench/scenarios/latency_cold_start.py
- scripts/test/e2e/bench/harness.py
…pass When --on field typo causes all metrics to SKIP, exit with error instead of reporting PASS with empty regressions list.
Measures per-stage wall time for stopping a running box and restarting it: stop_wall_ms, start_wall_ms, first_exec_after_restart_ms, total_restart_ms. Also adds stop_box(), start_box(), wait_box_status() helpers to harness.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
latency benchmark
Scenarios
latency-cold-startlatency-execlatency-lifecyclethroughput-exec-serialthroughput-exec-paralleldensity-parallel-createstability-churnstability-exec-loopTest plan
Summary by CodeRabbit
Summary
New Features
Documentation