feat: reduce store dependencies (experimental) - #4047
Draft
darshankabariya wants to merge 13 commits into
Draft
Conversation
darshankabariya
marked this pull request as draft
July 17, 2026 13:01
Promote storeSyncConf from StoreServiceConf to a top-level WakuConf field so relay/full nodes can run RBSR reconciliation + transfer without mounting the store service. When no store service is present, back the sync protocols with a small archive (default sqlite://:memory:, hidden --store-sync-db-url override) bounded by a time:2*range retention policy. Drive the ENR Sync capability bit from the new top-level conf so full nodes discover each other for reconciliation. Existing CLI flags are unchanged; store-service nodes keep identical behavior, including --storenode sync-peer slotting (standalone nodes warn instead, so sync peer selection stays discovery-based and is not pinned to one store peer). Part of the "Store as a startup-only dependency" experiment (step 1). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Integration test wiring two archive-backed nodes exactly like mountStoreSync (shared peer manager + the three sync channels, no store service): a missed message is recovered end to end through reconciliation and transfer into the receiving node's archive, and one session converges both nodes to the union of their message sets. Part of the "Store as a startup-only dependency" experiment (step 2). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…e protocol Nodes running store sync standalone carry an archive but no store protocol; GET /store/v3/messages without peerAddr now falls back to the local archive (new queryArchive, same conversion path as mountStore's request handler so pagination semantics match). Only the archive's retention window is served; peerAddr still queries remote stores. This is the audit interface the simulator's delivery verifier uses. Part of the "Store as a startup-only dependency" experiment (step 3). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The **/vendor/* dockerignore rule strips vendored sources (e.g. nat_traversal's miniupnp) out of a host-resolved nimbledeps/ tree while keeping its stamp, so in-container builds skip dependency resolution and then fail on the missing vendor content. Excluding nimbledeps/ and build/ makes the container resolve dependencies fresh, matching CI. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Closes the transfer.nim TODO: received transfer messages pass through an injected validator before syncMessageIngress; rejects are dropped and counted (total_transfer_messages_rejected). mountStoreSync wires a nil-guarded closure that runs Rln.validateMessage with the new checkFreshness=false switch - synced messages are old by design, so the timestamp-recency bound is skipped while proof, membership root and timestamp/epoch binding still verify. Known gap: proofs against roots older than the acceptable root window are rejected, bounding history sync across heavy membership churn. Part of the "Store as a startup-only dependency" experiment (step 8). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New messaging option send-confirmation = store (default) | propagation. In propagation mode MessageSent fires from the publish path (relay mesh / lightpush relayPeerCount) right after MessagePropagated, with no store polling; finalized tasks no longer re-enter the task cache, so the confirmation is emitted exactly once. Store mode is bit-identical to before. The option flows through both the structured and legacy flat JSON config shapes. Part of the "Store as a startup-only dependency" experiment (step 9). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…offline) At startup the node estimates its offline gap from the persisted last-online timestamp: within the sync window it catches up via full-node reconciliation (retrying until a sync peer is found, with the periodic sync loop as safety net); beyond the window, or on fresh start, it falls back to a bounded store resume, waiting for a store peer before consuming retry attempts. The catch-up runs as a background future so node startup no longer blocks up to 90 s on store peers, and shutdown bounds its cancellation wait. StoreResume is also mounted on sync-enabled full nodes so they track last-online. Part of the "Store as a startup-only dependency" experiment (step 10). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Archives do not persist RLN proofs (no proof column in the message store schema), so every message served from a peer's archive via store-sync transfer arrives proofless and was rejected in an endless retry loop (observed in the simulator: same missing messages re-transferred and re-rejected every sync round). Validate only proof-carrying messages; count proofless ones via the new total_transfer_messages_unverified metric. Real enforcement requires persisting proofs in the archive - recorded as a production gap. Part of the "Store as a startup-only dependency" experiment (step 8 fix). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
preProcessPayload only skipped ranges ENTIRELY below the receiver's sync window, so when a wide-window peer (e.g. a store node) reconciled with a narrow-window peer (e.g. a core node), the recursive split bottomed out in a range straddling the narrow node's window floor and its below-window items were still offered for transfer. Now a straddling range is split: the below-window portion becomes Skip so the wide peer no longer offers history older than the narrow node retains. Only triggers on a window mismatch; equal windows never straddle, so store<->store and core<->core sync are unchanged (full suite 37/37). Added a reconciliation test proving a 2h-window initiator does not push a 45-min-old message to a 30-min-window peer. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…lse clamp Adversarial review found the previous clamp compared the incoming range's lower bound to a fresh now()-derived selfLowerBound. Because the responder samples now() later than the initiator, that floor is always slightly higher, so the clamp fired even between EQUAL-window peers (store<->store), skipping a sub-second sliver at the bottom of the window and risking permanent store-fleet inconsistency for a message landing in it. Compare range WIDTH to our syncRange instead: width is independent of each peer's now() (both bounds subtract the same now+jitter), so equal windows never clamp and only a genuinely wider-window peer is clamped, erring toward reconciling slightly extra rather than dropping. Added an equal-window near-floor guard test (suite 38/38). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
darshankabariya
force-pushed
the
feat/store-sync-standalone
branch
from
July 22, 2026 07:51
6922f27 to
9812f99
Compare
Resume-fetched history is older than the archive's 20 s live-traffic freshness filter by definition, so the whole catch-up was silently rejected: a node past the sync window fetched its gap from a store node and archived none of it (1,662/1,662 dropped in the mechanism probe; stuck at ~71% delivery in the 150-node reproduction). Catch-up now enters via syncMessageIngress - the same filter-free door the sync transfer uses - and also feeds the reconciliation index so the next sync round does not re-request what the store just provided. Also repairs tests/waku_store/test_resume.nim: its node/waku_node import stopped exporting the store api after the rebase (compile error), and the legacy test assumed node startup blocks on resume, which stopped being true when catch-up moved to the background. Adds a regression test that messages older than the freshness window survive resume. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… reads The sync server had no protection (the old transfer guard was 'removed DOS prototection until we can design something better'): any peer could open unlimited reconciliation sessions, claim an int.high-sized payload with one length prefix, and push unsolicited transfer messages for free. - reconciliation: a RequestRateLimiter now fronts the server handler (new 'storesync' bucket in --rate-limit, default 30 sessions/5 min, the cheap check running before any processing); one active session per peer at a time, enforced on both the serving and initiating side; payload reads bounded to 64 MiB (large legitimate diffs are ~40 B per difference, so the largest honest rounds stay well under) - transfer: messages are only accepted from peers with a reconciliation session in the last 10 minutes; each session (re)opens that peer's window, unsolicited pushes are dropped, counted and disconnected. A stricter accept-only-granted-hashes model was tried and rejected: for an empty or far-behind receiver the sender computes the diff, so the receiver cannot enumerate the hashes it is owed - re-enables the transfer test disabled 'until we impl. DOS protection again'; adds session-guard, rate-limit and unsolicited-drop regression tests Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…iable Live messages carry an RLN proof, but archives dropped it at insert time - so history handed over via store-sync transfer or store resume arrived proofless and had to be accepted on faith, sidestepping the spam limiter exactly where it should be load-bearing. A malicious peer could invent 'history you missed' and a victim would swallow and re-serve it. - archive: proof column (sqlite schema v11, postgres v8; nullable, so existing rows migrate as a no-op and in-memory archives need no migration); insert/select thread message.proof through both drivers; the queue driver stores whole messages and needed no change - transfer and store resume serve proof-bearing messages automatically (the wire codecs already carry the field) and both re-verify the original author's proof on arrival: the receiving node's FIRST verification of a message it missed on relay, not a repeat - same work the relay path would have done, deferred to catch-up - store resume gains the same validator the sync transfer uses, since a store node could also serve invented history - new --store-sync-require-proof flag (default off): the rollout lever; once the fleet persists and serves proofs, flipping it makes nodes reject proofless synced history when RLN is enabled - reconciliation untouched: proofs are not part of the message hash, so identity, fingerprints and cursors are unchanged Tested end-to-end: a proof survives full-node -> full-node sync transfer and the store-query -> resume -> archive trip intact. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
This PR may contain changes to database schema of one of the drivers. If you are introducing any changes to the schema, make sure the upgrade from the latest release to this change passes without any errors/issues. Please make sure the label |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.