feat(bench/dashboard): capacity headline, honest outcomes, group-scoped fairness - #54
Merged
Conversation
Every healthy target reports ~485 rps because the paced workload caps offered load at VUs/think-time — the column measures the load generator's sleep timer, not the database. The saturation suite adds an unpaced concurrency ramp that produces a real capacity figure; this is the dashboard side of consuming it. Peak throughput becomes the primary number for targets that have one, and the paced rate stays visible beside it as "throughput at fixed load". The two are made impossible to confuse structurally rather than by labelling discipline: a capacity number and the objective it was measured at are one value (CapacityFigure in #lib/saturation), and CapacityFigure.svelte is the only component that can draw one — so there is no call site able to print a bare rps figure that reads like the paced number. No fallbacks. All four states render as themselves: saturated 12.4k at p99 < 50 ms, at 64 concurrent did_not_saturate at least 48.9k at p99 < 50 ms, knee not reached slo_never_met never met the p99 target (no number substituted) absent not measured Ordering cannot launder a non-measurement into a placement. Under ?sort=capacity a row without a measured peak never outranks one that has it — a 48.9k lower bound sorts below a measured 4.2k, because a ramp that ended early is not evidence of being faster — and only measured rows get a rank number at all; the rest show a dash, an empty bar track, and their state in their own cell. Lower bounds draw a faint open-ended bar rather than a solid one. A set that measured no capacity omits the column entirely and says why, instead of filling twenty rows with "not measured"; ?sort=capacity there resolves to the set's default rather than rendering an unranked table. The curve is the centrepiece of run detail: throughput vs concurrency over latency vs concurrency, sharing an ordinal x axis (concurrency ramps are geometric, so a linear axis crushes the early steps), SSR'd at reserved heights, with the objective as a rule and the chosen peak marked in both plots. Disqualified steps stay on the curve struck through, with the reason on hover and in a per-step table that works without scripting — the fixture's 128-concurrency step is faster than the peak AND held the objective, and is still not the headline, which is exactly the case the design has to get right. Two-axis fairness without splitting the single global table: a per-database harness strip above it (workers/pool/tuning, and whether within-family identity was verified), the same line inside each row's detail, and the delta label scoped in words — "vs drizzle-rs on SQLite", not "vs drizzle-rs". Harness blocks are merged across a set's shards; shards that disagree about a family produce a warning and mark the family unverified rather than one silently winning. A family that declared nothing says so. summary.saturation already existed with a different shape — the legacy knee_rps/knee_p95 heuristic computed off the paced run, which falls back to the busiest bucket when it finds no knee. `outcome` is the sole discriminator and the legacy shape reads as "not measured"; the two run-detail rows that displayed it are gone, because a paced number wearing the word "saturation" is what this change exists to end. Method gains "two suites, two headlines", the three outcomes, and the two meanings of fair, in the page's existing voice. Verified: check, build, 49 unit tests, and the route crawl green on fixtures covering every state (saturated / did_not_saturate / slo_never_met / absent / legacy-shaped, a disqualified step, a peak missing from its own curve, an undeclared harness, a family that failed within-family identity, and a mixed-harness cohort) and on real artifacts, which carry no saturation block and so exercise the "not measured" path end to end. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…eak" Follow-up to the saturation work. "Peak throughput" now names one specific thing — the capacity figure from the unpaced ramp — so every remaining place that used the word for `primary.rps.peak`, the fastest single sample bucket inside a paced run, is renamed to "busiest second". That covers the compare page's throughput column and metric option, the trends KPI and history table, and the two KPI builders. Each one gains a hint saying what it is and that it is not a capacity figure. Also replaces the descendant-selector emphasis the ranking row was applying to CapacityFigure's internals (`lg:[&_.text-lead]:text-foreground`) with an explicit `active` prop. The rule is "lift the number in the column the table is sorted by", and it has to keep working when the component changes its own type classes; reaching in from outside would have silently stopped matching. A lower bound is deliberately excluded from that emphasis — it is a floor, and the active column should not make it read like a measurement. Verified: check, build, 49 unit tests, route crawl green on the saturation fixtures (111 urls). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…e database Two coordinator rulings on the saturation contract, plus the first real runner output, land together because they touch the same scoping decision. **The peak is the fastest qualifying step**, not the highest qualifying concurrency. Legend, tooltip and step-table verdict all say so; Method says ties go to the lower concurrency, since the same throughput for less concurrency is the better result. The max-vs-peak check that the earlier rule needed is kept and repurposed. Under the new rule a well-formed artifact cannot trip it, so `CurveView.tallerThanPeak` is now a self-consistency disclosure in the same class as `peakMissing`: when an artifact names a peak while a faster *qualifying* step sits on its own curve, the curve says so with both numbers instead of drawing a mark that visibly contradicts the line. Not hypothetical — artifacts selected under the old rule exist and still render here (peak 28760@256 against 31457@16). Disqualified steps are excluded; their strike already explains them. Method also gains the non-obvious part: `outcome` is decided by whether the ramp's LAST step qualified, which is a different question from where the maximum landed. A curve can peak early, flatten, and still be "knee not reached" — a visible bend is not a measured limit. **`fair.family` is a comparison group, not an engine**, and it is now the scoping key. A group is the set of targets claiming to be directly comparable; it usually maps onto the database and splits where the harness genuinely cannot be equalised. bun:sqlite is synchronous on a single-threaded runtime, so a pool of 8 there is theatre — it sits in `sqlite-ts` with drizzle-orm, where same runtime, same pool of 1 and same pragmas make a real library comparison, while `sqlite` keeps the Rust stack. So: harness lookup keys on the group (a `sqlite-ts` row gets pool 1, never the Rust group's pool 8); `baselinesByFamily` replaces `baselinesByDb`, so a Bun row reads "vs Drizzle ORM on SQLite / TypeScript" rather than being measured against drizzle-rs across a language boundary; the delta scope moves into the visible label via `RankingRow.deltaLabel` instead of being hardcoded to drizzle-rs; the strip becomes "harness by comparison group"; and the row's harness line says "shared by every SQLite / TypeScript row", which the previous "every SQLite row" made outright false the moment one engine held two groups. Presentation deliberately does NOT follow the split: `?db=sqlite` still returns every SQLite row with `SQLite` in the database column. A split changes what a row is measured against, never whether it is shown. An absent `fair.family` falls back to the database profile. Not a guess — artifacts predating the field had exactly one group per database, which is what that expresses. Also handles `harness[].exempt`: `within_family_identical: true` beside a non-empty exempt list means "identical among the ones we checked", strictly weaker than the tick reads as, so the strip prints "N exempt" and the tooltip names them. Verified: check, build, 59 unit tests, route crawl 120/120 on fixtures that now include a `sqlite-ts` shard sharing an engine with the Rust sqlite shard, and against the runner's real output, which renders the old-rule peak discrepancy end to end. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
…ip by engine
familyLabel composes instead of looking up: a bare database id names that
database, and `<engine>-<stack>` becomes "PostgreSQL / TypeScript". It gets
postgres-ts right without having been told that group exists, and a future
postgres-go too — where a hardcoded map would have silently shipped
"Postgres Ts" the day the family was added. Unknown ids fall through
humanised with known engine tokens cased correctly ("SQLite Haskell"), so a
missing label looks slightly wrong on purpose: an unlabelled group should be
visible, the same way an unmeasured value reads as "not measured" rather
than zero.
The harness strip now orders by engine with a split group directly beneath
the group it split from — SQLite, SQLite / TypeScript, Turso, PostgreSQL,
PostgreSQL / TypeScript. Comparing two groups that share an engine is the
whole reason to read the strip, and alphabetical ordering parked each split
at the far end from the thing it should be read against.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
themixednuts
force-pushed
the
feat/saturation-ui
branch
from
August 3, 2026 23:36
9cc68b0 to
344a38b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Presents the saturation suite (#53) without letting its number be confused with the paced one, and makes the two fairness axes legible inside the single global table.
Two throughput numbers that cannot be confused. "Peak throughput — at p99 < 25ms, at 64 concurrent" sits beside "throughput at fixed load — median across trials". The separation is structural, not editorial:
CapacityFigureis a type whose number and qualifier are one object, and one component is the only thing on the site permitted to draw one, so no call site can emit a bare rps figure that reads like the other. The pacedrps.peakwas renamed "busiest second" site-wide, since "peak" now means something specific.Every outcome renders honestly, and sorting can't lie. Hard tiers — measured → lower-bound → never-met → not-measured — so a 48.9k lower bound ranks below a measured 4.2k peak, and only genuinely-measured rows get a rank number at all; the rest show a dash and state their own condition in-cell rather than having it inferred from position. A set that measured no capacity drops the column with an explanation instead of printing twenty "not measured" rows. Artifacts predating the suite (including the deleted legacy knee shape) read as "not measured" — never zero, never a relabeled paced number.
The curve is the centerpiece. Throughput and latency vs concurrency on a shared ordinal axis (concurrency ramps are geometric; a linear axis crushes the early steps), SSR'd at reserved heights, with the SLO threshold and chosen peak marked. Disqualified steps stay on the curve struck through with their reason on hover, plus a no-JS step table. Two self-consistency disclosures rather than silent smoothing: a peak whose concurrency is absent from its own curve withholds the marker and says so, and a peak with a faster qualifying step on its own curve is reported with both numbers.
Fairness legibility without splitting the table. Per-group harness strip, the same line in each row's detail,
within_family_identical: falsesurfaced in red, exempt counts named, and the delta scope written into the visible label ("vs Drizzle ORM on SQLite / TypeScript") rather than assumed. Harness lookup keys on the comparison group, so asqlite-tsrow is never handed the Rust group's pool.62 unit tests pinning each honesty rule, route crawl 120/120 on fixtures covering every state plus real artifacts, check/build clean.
🤖 Generated with Claude Code