[storage/qmdb] Add journal snapshots - #4532
Open
danlaine wants to merge 6 commits into
Open
Conversation
Benchmark resultsRegressions: ✅ `qmdb::merkleize/v=any::unordered::fixed::mmr k=10000 ch=false s=true cc=true` (2/2 gates passed)
✅ `qmdb::merkleize/v=current::ordered::fixed::mmb chunk=256 k=10000 ch=false s=true cc=true` (2/2 gates passed)
Baseline commit(s): |
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ✅ Deployment successful! View logs |
commonware-mcp | 91f162f | Aug 19 2026, 10:15 PM |
danlaine
force-pushed
the
danlaine/qmdb-snapshots-3
branch
from
August 19, 2026 17:26
119a9c6 to
0e8a682
Compare
Deploying monorepo with
|
| Latest commit: |
91f162f
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://68ed04ec.monorepo-eu0.pages.dev |
| Branch Preview URL: | https://danlaine-qmdb-snapshots-3.monorepo-eu0.pages.dev |
This was referenced Aug 19, 2026
danlaine
force-pushed
the
danlaine/qmdb-snapshots-3
branch
4 times, most recently
from
August 19, 2026 21:00
8adfa09 to
5aeecc3
Compare
danlaine
changed the title
Every authenticated database family gains snapshot(), an owned immutable capture of its state: full families freeze the authenticated journal at its current size (bounds pinned at capture, proofs byte-stable across later appends, rewrites, and prunes; applied-but-uncommitted operations are included, and reads from a later-rewound range are unspecified), and compact families capture the witness tip (the latest commit). Captures serve reads and proofs through the same Source interface as the live database, so peer serving can move off the live instance. Supporting rework: the merkle Storage trait keys on an associated Family and gains pinned_nodes_at; historical_proof is hoisted to qmdb so snapshots and live databases share one implementation; the compact witness's VerifiedWitness becomes Tip, implements Source directly, and now rejects non-canonical persisted commit bytes on rebuild (a corrupt journal fails with DataCorrupted instead of reopening). The in-memory key-to-location map is renamed from snapshot to index throughout (field, IndexBuild, IndexUndo, telemetry labels, and every comment and test name that used the old word for it), freeing the word for these captures. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q92PhqGoHbWCGHCsBJ7EBU
The remaining comments that used snapshot for the key index, including the fuzz crate outside the workspace, now say index. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q92PhqGoHbWCGHCsBJ7EBU
Six comments kept their old punctuation through the rename instead of trading colons for dashes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q92PhqGoHbWCGHCsBJ7EBU
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q92PhqGoHbWCGHCsBJ7EBU
Finish the index rename in three missed comments and inline a single-caller request check. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q92PhqGoHbWCGHCsBJ7EBU
Snapshot inherited the Storage trait's sequential get_nodes, one awaited journal read per proof node on the serving path. Mirror the live structure's override, serving memory-resident nodes directly and batching the rest through one read_many. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q92PhqGoHbWCGHCsBJ7EBU
danlaine
force-pushed
the
danlaine/qmdb-snapshots-3
branch
from
August 19, 2026 22:14
5aeecc3 to
91f162f
Compare
danlaine
marked this pull request as ready for review
August 19, 2026 22:30
Codecov Report❌ Patch coverage is @@ Coverage Diff @@
## danlaine/batch-read-gates #4532 +/- ##
=============================================================
- Coverage 95.56% 95.55% -0.02%
=============================================================
Files 610 610
Lines 278058 278774 +716
Branches 6654 6661 +7
=============================================================
+ Hits 265732 266380 +648
- Misses 10139 10203 +64
- Partials 2187 2191 +4
... and 11 files with indirect coverage changes Continue to review full report in Codecov by Harness.
🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part 2 of a 4-PR stack: #4531 → #4532 → #4533 → #4534.
Background
QMDB peers catch up by state sync: they fetch operations plus Merkle proofs and verify them against a root, and serving code hands out that data through the
Sourcetrait.What this adds
Every authenticated family gains
snapshot(), which returns an owned immutable snapshot of the database's journal state. The motivation is peer serving: today, serving state-sync requests reads the live database, which means serving contends with the writer and can observe state that is not yet on disk. A snapshot is a standalone object that keeps answering the same reads and proofs while the live database appends, commits, and prunes. #4533 moves peer serving onto these snapshots.Note this does not snapshot the QMDB's key-value index.
Three pieces of shared machinery change:
Storagetrait keys on an associatedFamilytype and gains a defaultpinned_nodes_at, so a frozen Merkle store and a live one expose the same node access.historical_proofis hoisted from per-family copies intoqmdb, so snapshots and live databases share one proof implementation.VerifiedWitnessbecomesTipand implementsSourcedirectly. Rebuilding a tip now also verifies the persisted commit bytes against the Merkle tip leaf and rejects non-canonical encodings, so a corrupt witness journal fails to reopen withDataCorrupted.Snapshot -> index rename
To prevent overloading terms, renames QMDB's in-memory index from "snapshot" to "index". Same for associated function names, etc.