[glue] Remove Shared database ownership

[danlaine]

Part 4 of a 4-PR stack: #4531 → #4532 → #4533 → #4534.

Background

Glue's stateful actor drives consensus decisions into the databases. Consensus asks it to verify candidate blocks (is this block valid on its branch?), and each verification runs as an independently polled job executing the application's state transition on speculative batches (#4531). Completed verdicts are cached in a pending map keyed by block digest, so duplicate requests and children forking from a verified block reuse them. Marshal delivers finalized blocks. Applying one moves the anchor, the last finalized block reflected in the databases. Blocks form branches, and when a block finalizes, every competing branch has lost. Before this PR, the actor shared its databases through a lock, and a scheduler classified, cancelled, and waited out every in-flight job before each finalization could apply.

Summary

The stateful actor owns its databases outright, and a verification job whose branch loses (i.e. is invalid because a non-ancestor was committed) finds out from the database itself, through a typed error instead of being classified, cancelled, and awaited by the scheduler. This is the payoff the three preceding PRs set up.

Ownership

Shared<DB> (an Arc<RwLock<Option<DB>>>), WriteSlot, BatchContext, and the take-the-value-out-put-it-back mutation dance are deleted. In their place, split(db) returns a Writer and a Reader (glue/src/stateful/db/cell.rs). The Writer is the actor's field and the sole mutation authority. It is not Clone, and its mutations consume and produce the database, so they read as ordinary by-value code with no reachable poisoned state except a mutation future dropped mid-flight, which is actor shutdown. Dropping the Writer closes the cell, so leftover readers park instead of silently serving a database that can never advance. Reader is Clone. Verification jobs and test probes hold one, and a read guard covers exactly one storage call. The lock is write-preferring, so a readers cannot starve finalization.

Cancellation moves from the scheduler to the data

The Disposition/FinalizationBoundary/VerificationProgress classification, the invalidation channels, and the quiesce-before-apply barrier are deleted. Finalization applies immediately, never waiting for doomed jobs to unwind. A job on a losing fork now dies at whichever comes first: its next database read returns StaleRead through the gates, or its requester (consensus) drops the response channel it stopped caring about. What a doomed job can waste is bounded by one inter-read gap of application compute. Applications are no longer obligated to write cancellation-tolerant futures, because glue no longer cancels them -- only requester-drop remains, the ordinary async contract.

The error channel

Application::{propose, verify, apply} return Result<_, ExecutionError>, where ExecutionError is {Stale, Fatal(String)} with a From impl for storage errors -- applications write ? where they wrote .unwrap() and never interpret the error. Glue, the only layer that knows what each case means, does the mapping.

Stale means a finalization landed mid-work. Wherever an honest answer exists, glue re-answers from the new canonical chain rather than guessing:

• A verification whose own block finalized mid-verify answers true (this closes a race where the honest answer used to be ambiguous); a competing block answers false. Each retry consumes one finalization, so the loop is bounded.
• Ancestry that looked invalid re-checks the canonical chain before answering false, and retries when the anchor moved mid-walk -- a finalization landing mid-walk must not turn a valid descendant into a false verdict.
• A completed verdict the anchor overtook before it could be cached keeps its answer: caching is retention, not part of the verdict, so the wrapper discards the entry and the verdict stays true -- validity is relative to the supplied ancestry. A block finalized by its own descendant mid-verify answers true for the same reason.
• From apply on the finalize path it panics: finalize replays one block on fresh batches from applied state and mutation authority is unique, so seeing it means a violated invariant, and there is nobody to answer false to.
• During propose it warns and declines, with a debug-assert, since the actor defers every other message while a proposal is active, so no finalization can land mid-propose.

Fatal is any other storage failure and panics everywhere, proposals included: a node that has observed broken storage must not keep serving. A closed storage channel maps here too -- the honest outcome when a backend worker dies, instead of a mislabeled cancellation or a silent wedge.

Shutdown is the actor's job, not the application's: the loop's stop signal covers the application awaits (the finalize drive, the proposal drive, and the sync handoff's applies), so on stop the actor drops the in-flight future and exits cleanly, leaving the block unacknowledged so marshal redelivers it after restart -- exiting releases the actor's own stop signal instead of parking on the very future graceful shutdown waits for. The stop arm is ordered first, so a Fatal caused by shutdown tearing down the application's dependencies is never observed, while a Fatal that arrives before the stop signal is a real failure and panics as it should. A verification in flight at shutdown quietly never resolves: its caller parks, and consensus reaps it on teardown.

The finalizing window

fork_batches refuses to fork while a finalization is mid-flight and re-checks its anchor after taking per-database forks, so a batch set can never straddle a finalization boundary (each per-database gate would pass individually while the set as a whole mixes two states). Stale retries park until the anchor moves past what they have already seen, instead of spinning.

Consequences elsewhere

• Mailbox::subscribe_databases is gone. The actor no longer hands its databases to anyone; peers observe only the published snapshot.
• The syncer delivers its artifact exactly once on the completion channel: SyncResult loses Clone, and update_targets returns an UpdateOutcome enum instead of mutating shared state.
• DatabaseSet::committed_targets becomes applied_targets, matching what it returns.
• Mocks, the reshare harness, and the reshare example migrate from .unwrap() to ?.

[github-actions]

Benchmark results

Regressions: 0.

✅ `qmdb::merkleize/v=any::unordered::fixed::mmr k=10000 ch=false s=true cc=true` (2/2 gates passed)

Field	Value
Package	commonware-storage
Benchmark target	qmdb_gungraun
Variant	qmdb::merkleize/v=any::unordered::fixed::mmr k=10000 ch=false s=true cc=true
Filter	*::bench_merkleize::any_unordered_fixed_mmr
Baseline suite	commonware-storage
Gates	EstimatedCycles should decrease; tolerance 10.00%; blob_reads should decrease; tolerance 10.00%
Cargo flags	--features test-traits

Metric	Baseline	Current	Delta	Gate
Ir	13,193,266	13,172,460	-0.16%	-
L1hits	16,876,759	16,854,369	-0.13%	-
LLhits	97,477	99,135	+1.70%	-
RamHits	11,263	11,401	+1.23%	-
TotalRW	16,985,499	16,964,905	-0.12%	-
EstimatedCycles	17,758,349	17,749,079	-0.05%	EstimatedCycles should decrease; tolerance 10.00%
blob_reads	158	158	+0.00%	blob_reads should decrease; tolerance 10.00%

✅ `qmdb::merkleize/v=current::ordered::fixed::mmb chunk=256 k=10000 ch=false s=true cc=true` (2/2 gates passed)

Field	Value
Package	commonware-storage
Benchmark target	qmdb_gungraun
Variant	qmdb::merkleize/v=current::ordered::fixed::mmb chunk=256 k=10000 ch=false s=true cc=true
Filter	*::bench_merkleize::current_ordered_fixed_mmb_chunk_256
Baseline suite	commonware-storage
Gates	EstimatedCycles should decrease; tolerance 10.00%; blob_reads should decrease; tolerance 10.00%
Cargo flags	--features test-traits

Metric	Baseline	Current	Delta	Gate
Ir	15,506,247	15,545,370	+0.25%	-
L1hits	20,403,595	20,441,820	+0.19%	-
LLhits	145,436	146,165	+0.50%	-
RamHits	14,314	18,479	+29.10%	-
TotalRW	20,563,345	20,606,464	+0.21%	-
EstimatedCycles	21,631,765	21,819,410	+0.87%	EstimatedCycles should decrease; tolerance 10.00%
blob_reads	235	235	+0.00%	blob_reads should decrease; tolerance 10.00%

Baseline commit(s): 1ba964dcd45a

[cloudflare-workers-and-pages]

Deploying with    Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Updated (UTC)
✅ Deployment successful! View logs	commonware-mcp	ea467df	Aug 20 2026, 02:04 AM

[cloudflare-workers-and-pages]

Deploying monorepo with    Cloudflare Pages

Latest commit:	ea467df
Status:	✅  Deploy successful!
Preview URL:	https://5582c92f.monorepo-eu0.pages.dev
Branch Preview URL:	https://danlaine-glue-owned-db.monorepo-eu0.pages.dev

View logs

[codecov]

Codecov Report

❌ Patch coverage is 88.48684% with 105 lines in your changes missing coverage. Please review.
✅ Project coverage is 95.52%. Comparing base (079c093) to head (ea467df).

Files with missing lines	Patch %	Lines
glue/src/stateful/db/mod.rs	84.24%	46 Missing ⚠️
glue/src/stateful/actor/processor/verifier.rs	74.28%	16 Missing and 2 partials ⚠️
glue/src/stateful/actor/syncer/actor.rs	43.33%	17 Missing ⚠️
glue/src/stateful/actor/core/syncing.rs	89.77%	8 Missing and 1 partial ⚠️
glue/src/stateful/db/cell.rs	93.38%	7 Missing and 2 partials ⚠️
glue/src/stateful/actor/syncer/mailbox.rs	88.88%	2 Missing and 2 partials ⚠️
glue/src/stateful/db/any.rs	95.55%	1 Missing and 1 partial ⚠️

@@                        Coverage Diff                         @@
##           danlaine/db-snapshot-serving-2    #4534      +/-   ##
==================================================================
- Coverage                           95.52%   95.52%   -0.01%     
==================================================================
  Files                                 611      612       +1     
  Lines                              279203   278848     -355     
  Branches                             6663     6641      -22     
==================================================================
- Hits                               266713   266360     -353     
- Misses                              10295    10296       +1     
+ Partials                             2195     2192       -3     

Files with missing lines	Coverage Δ
glue/src/stateful/actor/core/mailbox.rs	96.03% <100.00%> (+0.20%)	⬆️
glue/src/stateful/actor/core/mod.rs	91.21% <100.00%> (-0.13%)	⬇️
glue/src/stateful/actor/core/processing.rs	96.45% <ø> (-1.05%)	⬇️
glue/src/stateful/actor/core/verifications.rs	100.00% <100.00%> (+5.17%)	⬆️
glue/src/stateful/actor/processor/mod.rs	92.99% <ø> (-1.73%)	⬇️
glue/src/stateful/actor/syncer/mod.rs	96.23% <100.00%> (-0.16%)	⬇️
glue/src/stateful/db/current.rs	78.21% <100.00%> (+2.33%)	⬆️
glue/src/stateful/db/immutable/compact.rs	93.14% <100.00%> (+1.17%)	⬆️
glue/src/stateful/db/immutable/standard.rs	43.86% <100.00%> (+7.81%)	⬆️
glue/src/stateful/db/keyless/compact.rs	91.20% <100.00%> (+0.97%)	⬆️
... and 9 more

... and 10 files with indirect coverage changes

---

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 079c093...ea467df. Read the comment docs.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.