Disk-backed SMT

[jait91]

Port the Rust aggregator’s disk-backed SMT design from rugregator to the Go aggregator to reduce SMT memory growth

[b3y0urs3lf]

For Issue #152 — Disk-backed SMT

QA verification complete — PR #156

Verified end-to-end on branch disk-backed-smt-port @ 89db142. All
acceptance criteria for porting the disk-backed SMT to reduce in-memory
growth are met.

Acceptance criteria

• Disk-backed SMT implemented — RocksDB-based backend behind the
  Backend interface (internal/smt/backend/disk_backend.go,
  internal/smt/disk/*)
• In-memory remains default — SMT_BACKEND=memory is the
  unchanged default; RocksDB opt-in via SMT_BACKEND=rocksdb
• Restricted to bft-shard mode — factory guards reject
  SHARDING_MODE=child/parent/standalone for RocksDB (R2)
• Startup verification against Mongo — every replica boot logs
  "Starting disk SMT startup verification" and "Disk SMT verified"
  before serving traffic
• Bounded replay/recovery — replayDiskSMTGap with
  SMT_STARTUP_REPLAY_LIMIT_BLOCKS (default 100)
• Proof metadata cache — proofMetadataCache keyed by stateID,
  matched against current published root (records cap 250k, blocks cap 8192)
• Performance diagnostics surfaced — rocksDBL0Files,
  rocksDBPendingCompactionMB, rocksDBBlockCacheHits,
  memRocksDBMemTableMB, memRocksDBBlockCacheMB, proofCacheRecords,
  proofCacheBlocks, all per-round PERF metrics
• Build tag rocksdb — guarded compilation, Dockerfile
  multi-stage with static rocksdb library
• Goal achieved: SMT memory growth reduced — see live evidence
  below

How it was tested

Aggregator-side unit tests (in disk-backed-smt-port, run via make test):

• internal/round/disk_smt_startup_test.go (752 lines) — bounded
  replay, replay-limit-exceeded fail-closed, root mismatch
  detection, recovery from unfinalized blocks
• internal/round/proof_metadata_cache_test.go — eviction order on
  add/refresh
• internal/smt/backend/disk_backend_test.go, ..._rocksdb_test.go —
  backend operations + RocksDB-specific behavior
• internal/smt/disk/persist/tree_test.go (788 lines) — disk SMT tree
  operations
• internal/smt/disk/rocksstore/store_test.go — RocksDB store
  primitives
• internal/round/smt_backend_factory_*_test.go — factory guards for
  invalid mode combinations
• internal/service/disk_bft_proof_rocksdb_test.go — proof generation
  under disk backend

All packages green: go test ./... → exit 0.

SDK BDD scenarios (state-transition-sdk-js,
feature/test-infrastructure, top f8ec96f, tagged
@disk-smt-coverage):

Scenario	Result
non-inclusion-proof.feature	✅
proof-self-validates.feature (freshly-finalized + never-submitted)	✅
append-only-inclusion.feature	✅
wire-format-stability.feature	✅
proof-stability-soak.feature (10k commits)	✅
cache-hit-against-rotated-root.feature	✅
cold-cache-proof.feature	✅

Run sequence: 9/9 PASS in 1m42s.

1-hour compaction-pressure soak against
bft-sharding-ha-compose-3nodes.yml (2-shard, 3-replica HA, RocksDB
backend, SMT_PRECOMPUTE_PROOFS=true):

Metric	Result
Duration	53m47s
Block production	3580 → 5924 (+2,344 blocks)
Commitments submitted	~150,000
ERROR / WARN / panic across 6 replicas	0 / 0 / 0
Leader churn	none — shard0-a + shard1-a held leadership entire run
RocksDB memtable flushes	2 per shard, observed in-place at blocks 4046 and 5507
Memory growth signal	none — heap stable 100–200 MB throughout
rocksDBL0Files	0 throughout (flushes compacted into higher levels immediately)

Exact flush evidence on shard0-a (raw PERF log lines):

block 4045: "memRocksDBMemTableMB":63 → block 4046: "memRocksDBMemTableMB":1
block 5506: "memRocksDBMemTableMB":63 → block 5507: "memRocksDBMemTableMB":1

Round timing held across both flushes
(finalization:18ms / 46ms — well under the 1.2s round budget).

Reviewer notes

• All review items from PR Add RocksDB-backed disk SMT for BFT shard mode #156 (Gemini + MastaP) verified resolved: FIFO
  cache eviction, snapshot leak fixes, proof availability race, tree
  insert error handling, RocksDB handle TOCTOU, Docker volume
  permissions, graceful empty proof for non-included states.
• One related bug surfaced during cold-cache testing and was fixed in
  325e323 fix: tolerate disk SMT replay gaps after HA failover —
  reproduced from QA side, fix verified live, regression test added on
  dev side.

Sign-off

Both functional acceptance and operational behavior verified.
Ready to close from QA perspective.