bench(targets): tune each engine for the workload it is measured on

[themixednuts]

Stacked on #59 (ci/per-os-topology) — review or merge that first; this branch contains its commits.

"Fair" across families means nobody crippled. Every engine was running on defaults chosen for machines with less RAM than this dataset needs, so the ranking was partly measuring who has the better defaults rather than who is faster.

What was actually wrong

• SQLite ships a 2 MiB page cache and no memory map. The benchmark is ~10k customers / 50k orders / ~300k detail rows over query_only connections — it fits in RAM many times over, so the defaults meant re-reading pages the machine already held.
• PostgreSQL's stock image plans with random_page_cost = 4.0, a spinning-disk figure, against data that is in RAM after warmup. That biases the planner away from index scans. work_mem at 4 MB is what the /orders-with-details aggregate sorts inside.

What changed

Embedded engines take a 64 MiB page cache, and a 256 MiB mmap where the engine has one:

family	page cache	mmap
sqlite (rusqlite)	64 MiB	256 MiB
sqlite-ts (bun:sqlite)	64 MiB	256 MiB
libsql	64 MiB	256 MiB
turso	64 MiB	none — no mmap_size pragma exists

Turso's gap is an engine capability difference, not a configuration choice. I probed it rather than assumed: PRAGMA mmap_size reports success and reads back nothing. Its spec declares the absence instead of implying parity.

PostgreSQL gets random_page_cost=1.1, effective_cache_size=2GB, work_mem=16MB via ALTER DATABASE.

Not a container command, deliberately: a GitHub Actions services: block cannot pass one. Applying it from the seeding path instead means local runs and CI take the settings from the same code, and that the external ORM crates (sqlx, diesel, seaorm, toasty) — which open their own connections and never run runner code — are covered without knowing about any of it. shared_buffers is left at the image default because it needs a restart CI cannot perform, and tuning it locally only would mean local and published numbers came from different servers.

Every setting is read back

Each value is verified after being applied, and the run fails if the engine did not honour it. These values are declared to readers in each spec's fair.tuning, so a silently clamped pragma would publish a claim that is false of the run. SQLite reports success for a pragma it clamps or ignores, so the readback is the only real confirmation — the same reason enable_wal and enable_mvcc already read their modes back.

That readback found two declarations that were already false

• libsql declared query_only=ON and nothing ever set it — its pooled connections were writable while the rusqlite family's were not.
• turso declared temp_store=MEMORY and nothing ever set it.

Both are now applied rather than dropped from the claim, since both are what makes those families comparable to the rusqlite one. Within-family identity stays enforced and clean across all 7 families.

Verification

• cargo test -p bench-runner --bins — 65 passed, including a new test that proves a pooled connection honours the declared tuning on whatever platform it runs on
• cargo +nightly clippy --workspace --all-targets --all-features -- -D warnings — clean
• turso and libsql pragma support probed directly rather than assumed

Not verified locally against a live PostgreSQL — Docker was unavailable on the machine this was written on, so the ALTER DATABASE path and its readback are exercised first by the PostgreSQL CI jobs. If the syntax or permissions are wrong there, those jobs fail loudly rather than silently skipping the tuning.

Expect a step change in the numbers

This makes engines faster on purpose, so published results before and after are not comparable, and baseline regression checks should be expected to flag it once.

🤖 Generated with Claude Code

[cursor]

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

[themixednuts]

bench(saturation): find the ceiling by throughput turning over, not by latency

Measuring the tuned engines showed the ramp was answering the wrong question.
Throughput is flat across almost the whole ladder: every target reaches its
maximum at 4-16 concurrent requests and then holds it while p99 climbs from
0.2 ms to 185 ms. The steps at 256 and above bought no throughput information at
all; they existed only to make some step eventually breach the objective.

Keying did_not_saturate on that breach punished targets for being fast.
rusqlite peaked at 62.8k rps at concurrency 16 and fell to 55.6k by 1024 without
ever crossing a 25 ms p99, so its maximum was measured cleanly three steps in and
it was still reported as "knee not reached" - and a row with no peak gets no rank,
so the fastest SQLite target dropped out of the ranking entirely.

Saturation is a property of throughput. A closed-loop target is at its ceiling
once more in-flight requests stop buying throughput; the objective is a policy
filter on which operating points are acceptable, not the definition of the limit.
The outcome now turns on whether the curve turned over: a maximum that is both
interior to the ladder and measurably above the last step that held the objective.

Both halves of that are load-bearing. Requiring the maximum to be interior means
it is bracketed by a rise and a fall, so a curve that was still climbing below the
ladder's floor is not mistaken for a peak. Requiring a 2% margin over the last
step means an ordinary flat curve stays a lower bound instead of having run-to-run
wander manufacture a peak out of noise.

The ladder is re-cut to match where the information actually is: 1,2,4,8,16,32,
64,128 instead of 4..512. It costs the same wall clock, so the linux-all budget is
unchanged at 334 of 350 minutes, and it samples the region where throughput
changes rather than re-confirming a plateau eight times.

Verified end to end on the target this broke. Its curve now reads 9.1k at
concurrency 1, 18.8k at 2, 23.5k at 4, 27.3k at 8, then 26.6k, 26.8k, 26.4k,
25.9k - a bracketed interior maximum, reported as a peak, with no step ever
breaching the objective. The old ladder started at 4 and never saw the rise.

Co-Authored-By: Claude Opus 5 noreply@anthropic.com