ci(bench): rank families per OS on one machine per OS - #59
Merged
Conversation
Salvaged from an interrupted agent run. Adds a db-cpuset input to the bench-runner composite so an out-of-process database (the PostgreSQL service container, or SpacetimeDB under taskset) owns cores carved out of the SUT half, making load / server / database three disjoint sets instead of letting the database float unpinned while embedded engines stay confined. Hard-fails when the literal disagrees with nproc. Documents that macOS has no usable CPU-affinity API, so there is nothing to fall back to there. Incomplete: runners.yml still has the old per-family job layout. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A publish run is about to measure each family twice: once in its existing per-family job on the paced workload, and once in a new per-OS job on the saturation ramp that produces the cross-family ranking. Both invocations would otherwise write `bench-out/<family>-<platform>`, upload `runner-<family>-<platform>`, and save the same baseline cache key, so the second would clobber the first. `slug-suffix` is empty by default, so the per-family jobs keep their current artifact names and cache keys and branch-protection checks are unaffected. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`postgres-all` collapsed the three PostgreSQL families onto one VM so their leaderboard rows shared hardware, and skipped the per-family jobs to avoid artifact collisions. That bought comparability inside PostgreSQL only: SQLite, libSQL, Turso and SpacetimeDB still landed in the same table from four other VMs, and a publish run lost the parallel per-family latency readings. Replace it with `plan.outputs.cross_family`, which the per-OS `*-all` jobs in the next commit key off. The per-family jobs lose their `if:` guards and now always run, so every event keeps fast parallel feedback on the paced workload. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…sequence The per-OS ranking job runs every family on one VM back to back, so it is the one job here that can hit GitHub's hard 360-minute cancellation. At 27 linux targets the sequence estimated 356 minutes against a 350-minute timeout — it did not fit, and the way it would have announced that is a cancellation at minute 350 with most of the ranking missing. Drop the 1024-VU step. On a two-core server set every target is already in queueing collapse at 512, so the step has never produced a peak: it breaches the p99 SLO for anything that breached at 512, and its only remaining job is ruling out `did_not_saturate`, which 512 does. Published peaks all sit at 16-64 concurrency, so no existing curve changes shape. That buys 25s per trial per target and brings the estimate to 334 minutes. `plan` now computes that estimate from the specs themselves and fails in five minutes if it exceeds the timeout, so adding a target lengthens the sequence loudly instead of silently. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The per-OS ranking job carves the SUT half into a client set and a database set, applied by the caller via --cpuset-cpus on the PostgreSQL service or taskset on the SpacetimeDB daemon. The runner cannot pin a process it did not spawn, so it never saw that third set and the manifest described a three-way split as a two-way one — which is exactly the disclosure the dashboard reads to say whether a ranking's families were isolated from each other. BENCH_CPUSET_DB is read for the manifest only; nothing applies it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The leaderboard puts every family in one table, so a rank only means something if the rows came off one machine. They did not: eight families were measured on eight separate CI VMs, and `postgres-all` fixed that for three of them only. Add `linux-all` (all eight families) and `desktop-all` (the in-process engines on macOS and Windows). Each runs its families back to back on one VM under one cohort id, so every row in an OS ranking shares hardware. They run on publish events only, alongside — not instead of — the per-family jobs, which keep producing the paced latency reading in parallel on every event. Cross-OS folding stays impossible and is not attempted: service containers are Linux-only on GitHub-hosted runners, and libsql segfaults the benchmark on the other two, so macOS and Windows can only ever rank in-process engines. That is why the ranking is scoped per OS rather than merged. On linux the PostgreSQL service is pinned to the top core of the SUT half so load / client / database are three disjoint sets; SpacetimeDB starts last under taskset on the same core, by which point PostgreSQL is idle. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The ranking ran 01..N over every row in the set, and the set spans operating systems: the SQLite family alone is measured on Linux, macOS and Windows. So a Windows row could outrank a Linux row, under a heading that reads as a library comparison, when what separated them was a kernel and a CI machine. The field is not even the same shape across platforms — GitHub runs service containers on Linux only, so PostgreSQL and SpacetimeDB can never appear on the other two. Operating system becomes the ranking's outer scope, in the URL as `?os=`, with pills above the database ones. Rank, bar scale, the "vs drizzle-rs" baseline, the per-database verdict tiles, the harness strip and whether the set measured capacity at all are now computed inside the scope, because each is a within-scope claim. No rows are hidden: every platform is a pill away, and the pill says how many targets it holds. `osScopes` also carries the provenance the scope can honestly assert. More than one CPU model in a scope is proof its rows came off more than one machine and says so; one model is only consistent with a shared host and is worded that way. `topology.cpu_pinning` reaches the row now, so a Linux scope states its cpuset split and a macOS one states that nothing was pinned — Darwin exposes no usable CPU-affinity API — rather than leaving the absence to be read as isolation. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The audit table still listed `postgres-all` and claimed the per-family PostgreSQL jobs were skipped on publish runs, both of which stopped being true when the ranking moved to one job per OS. The CI topology section described core isolation as a two-way split, which it no longer is on the families whose engine is out of process. Also notes BENCH_CPUSET_DB in the manifest schema's description of `topology.cpu_pinning`, since the field's value now carries a third set. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
`cross_family` required `publish`, and `publish` is gated on `main`. That made the two jobs no pull request can run also the two jobs that would run for the first time only after merging — the topology would get debugged in production, on a five-hour serial sequence. A `workflow_dispatch` now enables them whether or not it publishes; R2 stays gated on `publish` independently. Dispatching `benchmark_size=preview` from a branch runs the whole sequence on the saturation-preview ramp at one trial. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The first per-OS run recorded `load=0-1 server=2-2 db=3` on the out-of-process families. Correct, and the parser takes either form, but this string is what the dashboard prints as a run's isolation disclosure and `server=2-2` reads like a typo rather than a deliberate single-core assignment. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…onflict The first per-OS run showed the copy was wrong. A cross-family linux scope has two splits by design — the in-process engines take the whole system-under-test half, the out-of-process ones hand a core to the database — and both halves own the same cores, which is the property that makes the ranking comparable. The scope detail reported that as "under more than one CPU-isolation setting, so no single split describes them", which reads as a warning about a correct state. It now lists what actually ran: "cores split load=0-1 server=2-3 and load=0-1 server=2 db=3 by engine". A scope mixing pinned and unpinned rows still stays `mixedPinning` and names only the splits it has. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`EntityFilter` is `Box<dyn Fn(..)>`, which is already `#[must_use]`, so the attribute is a no-op that `clippy::double_must_use` rejects. The lint started firing with the 2026-08-08 nightly, which turned Lint red on main and on every open branch — main's last green run predates it. The sibling `system_table_filter` returns `bool` and keeps its `#[must_use]`, which is where the attribute actually does something. Unrelated to this branch's subject; included because it blocks its CI. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The leaderboard puts every family in one table, so a rank only means something if the rows came off one machine. They did not — eight families were measured on eight separate CI VMs, and
postgres-allfixed that for three of them only. And the ranking itself ran01..Nacross operating systems, so a Windows row could outrank a Linux row under a heading that reads as a library comparison.CI
postgres-allis replaced by two publish-only jobs that run every family their OS can host, back to back on one VM under one cohort id:linux-all— all eight families. PostgreSQL's service container is pinned with--cpuset-cpusand SpacetimeDB starts last undertaskset, so load / client / database are three disjoint sets. Previously the database floated unpinned over the whole VM while the in-process engines were confined to the upper half — a silent handicap for every one of them in the same table.desktop-all— the three in-process families on macOS and Windows.Cross-OS folding stays impossible and is not attempted: GitHub runs service containers on Linux only, and libsql segfaults the benchmark on the other two, so macOS and Windows can only ever rank in-process engines.
These run alongside the per-family jobs, not instead of them. The per-family jobs lose their
if:guards and keep producing the paced latency reading in parallel on every event;slug-suffix: crosskeeps the two sets of artifacts and baseline caches apart.Fitting the 360-minute ceiling
linux-allis a serial sequence and is the one job that can hit GitHub's hard cancellation. At 27 targets it estimated 356 minutes against a 350-minute timeout — it did not fit, and the way it would have announced that is a cancellation at minute 350 with most of the ranking missing.The saturation ramp now ends at 512 VUs. On a two-core server set every target is already in queueing collapse there, so the 1024 step has never produced a peak — published peaks all sit at 16–64 concurrency, so no existing curve changes shape. That brings the estimate to 334 minutes.
plannow computes that estimate from the specs themselves and fails in five minutes if it exceeds the timeout, so adding a target lengthens the sequence loudly instead of silently.Dashboard
Operating system becomes the ranking's outer scope, in the URL as
?os=, with pills above the database ones. Rank, bar scale, the "vs drizzle-rs" baseline, the verdict tiles, the harness strip and whether the set measured capacity at all are computed inside the scope. No rows are hidden — every platform is a pill away, and the pill says how many targets it holds.osScopescarries the provenance a scope can honestly assert: more than one CPU model is proof its rows came off more than one machine and says so; one model is only consistent with a shared host and is worded that way.topology.cpu_pinningnow reaches the row (and now includes the database's cpuset), so a Linux scope states its split and a macOS one states that nothing was pinned — Darwin exposes no usable CPU-affinity API — rather than leaving the absence to read as isolation.Verification
cargo test -p bench-runner— 84 passedbun run check— 0 errors;vitest— 70 passed (8 new, coveringosScopes)bun run build— clean🤖 Generated with Claude Code