Exp 233: reject async checkpoint worker

[danReynolds]

Hypothesis

Resqlite's writer WAL hook runs a PASSIVE checkpoint synchronously when a
commit crosses 500 frames. Exp 029 showed that proactive checkpoints improve
write tails, but the threshold-crossing caller pays the checkpoint I/O before
its write future can complete. Friday's moonshot tested whether the policy
could stay intact while a dedicated writable connection performed that work
after the commit released the write lock.

The acceptance target was at least a 10% first-crossing improvement. The
declared kill conditions were a sustained-write p99 that failed to improve, a
foreground-read regression, or a WAL that failed to settle.

Approach

The archived prototype replaced the inline hook work with a mutex-protected
idle / requested / running / rerun request state. After a successful
top-level commit, the writer isolate claimed that state and woke a dedicated
checkpoint isolate owning a separately keyed writable connection. That worker
ran SQLITE_CHECKPOINT_PASSIVE; close joined the writer and performed a final
unconditional drain so cross-sender message ordering could not strand a
claimed checkpoint.

The focused A/B paired exp 132's known 529-frame first crossing with 3,000
awaited 8 KiB writes, foreground reads every 100 writes, and immediate/250 ms
observational WAL samples. Full detail is in
experiments/233-async-checkpoint-worker.md.

The final PR is publication-only: all runtime code, worker-specific tests, and
the focused harness have been removed. The exact candidate remains at
archive/exp-233.

Results

Pair	Metric	Baseline	Candidate	Delta
baseline first	first crossing p50	61.883 ms	27.027 ms	-56.3%
baseline first	sustained write p50 / p95 / p99	0.041 / 0.109 / 0.659 ms	0.138 / 0.493 / 1.502 ms	+237% / +352% / +128%
baseline first	foreground read p95	0.218 ms	0.259 ms	+18.8%
candidate first	first crossing p50	66.178 ms	35.878 ms	-45.8%
candidate first	sustained write p50 / p95 / p99	0.044 / 0.112 / 0.429 ms	0.117 / 0.373 / 1.069 ms	+166% / +233% / +149%
candidate first	foreground read p95	0.207 ms	0.325 ms	+57.0%

The mechanism works once: the first threshold crossing is roughly 2x
faster, and the WAL samples prove the writer replies before the background
checkpoint finishes. Under sustained writes, though, every central/tail metric
regresses by multiples and foreground reads slow in both orderings. The
candidate WAL grows to 12,377 frames; two candidate-first repeats remain 2,689
and 3,625 frames behind after 250 ms.

The root cause is architectural: pages_in_wal >= 500 is a level trigger. Once
checkpoint work becomes asynchronous, later commits continue observing the
same true level and repeatedly request/rerun PASSIVE work. Coalescing bounds
messages but does not supply a WAL-generation or high-water re-arm rule, so the
worker contends with the foreground workload.

Outcome

Rejected. The prototype clears the one-shot ceiling and fails every
sustained kill condition. Shipping it would also retain a second connection,
isolate lifecycle, key propagation, FFI surface, four-state native scheduler,
and close protocol without representative evidence that inline checkpoint wall
is common enough to repay that complexity.

Reopen only with a reset-aware or high-water trigger that cannot remain armed
throughout a burst. The sustained write/read lane—not the attractive first
crossing—must be the load-bearing gate, and a downstream/release trace should
establish that synchronous checkpoint wall is materially user-visible.

Test plan

• Two order-flipped focused A/B comparisons, seven repeats per side
• Archived prototype: strict Clang warnings, native 792-frame smoke,
  checkpoint/encryption/close integration tests 3/3
• Archived prototype: focused database/transaction/encryption/portability/
  build-hook tests 101/101
• Publication branch: sources restored byte-for-byte to origin/main
• dart run benchmark/finalize_experiment.dart --experiment=experiments/233-async-checkpoint-worker.md
• dart run benchmark/check_experiment_dispositions.dart
• dart analyze --fatal-infos
• Full serial suite 326/326
• JSON validation and git diff --check