feat(dedupe): retention per tenant and table, and an expiry sweep

[EricAndrechek]

Part of #613. Part of the remote-dedupe design, stacked on #629 (feat/dedupe-windowed-ingest).

Fixes #220

What changes

dedupe.retention (internal/settings/{settings,validate,store}.go). This is a new optional config.json key, and dedupe.tables.<table>.retention overrides it per table. It says how long a committed id stays a duplicate, as a Go duration string ("720h"). "0" keeps ids forever: that is the seed value and today's behaviour. A missing key means "0" (forever), and a table override without one inherits the tenant's, so an existing config.json needs no change.

• Validation. It refuses a value that is not a duration ("30d", a bare number), a negative one, and a finite one below settings.MinDedupeRetention = 2 min, which is the embedded queue's duplicate window. It refuses rather than clamps, so the file never means something other than what it says. The reason for the floor (as noted in fix(ingest): windowed reserve/publish/commit; 503 when dedupe down #629): a deduped record is published under a Nats-Msg-Id derived from its id. If an id is re-sent after a shorter retention but inside the window, it is claimed again, and then JetStream drops it as a copy while the client gets 200. TestIngest_MinDedupeRetentionCoversTheDuplicateWindow pins the constant ≥ mq.EmbeddedDuplicateWindow. settings does not import mq, which would pull NATS into the control plane's import of settings.
• Resolution. Store.DedupeFor(table) now returns a settings.Dedupe{Enabled, IDField, RequireID, Retention} struct, not three values, and resolves every field from one snapshot, as before. Ingest reads it per record, so a reload changes retention for the next record. Each pendingRecord carries its retention. commitClaims issues one Commit per distinct retention in a window: one in practice, two only if a reload lands mid-window.

Pebble sweep (internal/dedupe/sweep.go, embedded.go). The read side already honoured the committed 0x02 ‖ expiry value. Now:

• A background loop starts when the instance opens and is stopped (and waited for) before it closes. Its first pass runs one minute after open, so a legacy value goes promptly, and then it runs hourly.
• A pass walks the whole keyspace in chunks of 1,024 keys, with a 10 ms pause between chunks, reading each chunk without holding Embedded.commitMu. It collects candidates whose committed expiry has passed, plus any key whose value is not a commit at all (isCommit: a 9-byte committedMark value — only commits are ever stored in Pebble, so an older, 8-byte value from a prior key layout is never a commit). It then takes commitMu only to re-Get each candidate and delete those still expired or still not a commit (NoSync), so a key the sweep read as expired cannot be re-committed between that read and the delete. A Commit waits at most for about 1,024 point reads and one batch, however many stale values the read stepped over (measured under -race: a Commit racing a chunk over 300k stale values waited 241–278 ms with a single lock over the whole chunk, 5–11 ms with this two-phase read).
• Deletes are NoSync. A delete lost to a crash is redone on the next pass, so the sweep adds no fsync to ingest.
• A new metric, wavehouse_dedupe_swept_keys_total{reason="expired"|"version_0"}, counts what it deletes, and a pass that deletes anything logs one line at Info.
• expiry() saturates at MaxInt64 rather than wrapping into the past for an absurd retention.

Cadence is a constant (1 min first, then 1 h), not a config key. The sweep only reclaims space: correctness never waits on it, which makes it a candidate for a scheduled background worker later, per the design.

Deviations from / additions to the design

• A legacy (non-commit) value is deleted on the first pass, not "once older than the largest retention". Nothing reads a value that isn't a commit, so keeping it longer protects nothing.
• Refuse, not clamp, for a retention under the window (above).
• The design's Pebble value encodes expiry in unix seconds; fix(dedupe)!: reserve/commit ids, windowed ingest, retention, dynamodb #625 shipped UnixNano. This PR keeps fix(dedupe)!: reserve/commit ids, windowed ingest, retention, dynamodb #625's encoding, which is the on-disk format already.
• A reload mid-window splits the window's commit by retention. The design says only "changing retention affects ids committed after the change".
• Disk space of deleted keys comes back as Pebble compacts. This PR does not force a compaction, because an hourly manual compaction of the whole keyspace would rewrite it every hour, and a legacy value can be anywhere in the keyspace so the cost could not be bounded. When space is reclaimed is inferred, not measured.

Deliberately left to later PRs

• DynamoDB (feat(dedupe): add a DynamoDB backend, tested but not yet wired #628, not in this base): its Commit already writes ex when retention > 0, so the retention arrives through the same Commit argument with no change.
• External broker: the 2-minute floor is the embedded window. With an operator-owned stream whose duplicate_window is longer, the same hazard exists between 2 min and that window. The boot check that compares lease to duplicate_window (fix(ingest): windowed reserve/publish/commit; 503 when dedupe down #629's note, for feat(app): choose the DynamoDB dedupe backend at boot #635 and later) should also warn when a tenant's finite retention is below it. Settings validation is pure and cannot see the broker.
• A sweep for multi-region/shared-cluster deployments (design §8) is not built.
• dedupe.on_unavailable is design open question 2, and unchanged here.

Test evidence

• internal/dedupe/sweep_test.go:
  • A sweep over 2,058 expired keys interleaved with 2,058 live keys crosses chunk boundaries. It deletes exactly the expired ones and 3 legacy (version-0) keys from two tenants. It keeps a forever key and a key that expired and was committed again before the sweep, which is still a duplicate afterwards. A second pass finds nothing.
  • Retention is honoured on read to the nanosecond before any sweep.
  • The loop runs on its own while the instance is open, and closing waits for it.
  • A cancelled sweep stops after one chunk.
  • expiry saturates.
  • TestEmbedded_SweepNeverDeletesACommitLandingMidChunk: a test-only hook races a Commit into the gap between a chunk reading its keys and deleting them; mutation-checked (fails 3/3 with the lock removed).
  • TestEmbedded_SweepKeepsAKeyCommittedAfterItsRead, TestEmbedded_SweepReadRunsUnlocked (asserts the chunk read genuinely runs without holding commitMu): each fails on the regression it names, checked by mutation (skipping the re-read, releasing the lock before the delete, reading under the lock, or running the whole chunk under one lock).
• internal/settings:
  • A missing key loads and resolves to forever (tenant and inherited table level); a non-duration, a bare number, a negative value, and a value under the window are each refused, at the top level and per table.
  • "0", "0s", "2m" and table overrides (shorter, forever) are accepted.
  • DedupeFor cascades all four fields.
  • The seed resolves to retention 0.
• internal/api/ingest_retention_test.go:
  • Real settings directory. Each table's retention reaches Commit, including a per-table "0".
  • A reload changes it: new ids get the new retention, ids committed earlier keep theirs, and the removed override is gone.
  • A reload mid-window gives two Commits with the right retention each.
  • The floor ≥ the queue window.
• A mid-chunk Commit survives the sweep (above).
• The conformance suite's existing "a commit expires after its retention" case still covers read-side expiry on both clocks.
• make ci is green: unit 93.4%, integration 45.1%, e2e 61.6%, Go total 94.5%, and every coverage gate passed.

🤖 Generated with Claude Code

https://claude.ai/code/session_01G6Cz4H5k1spJAPk5CZeGds

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Advanced

Run ID: f5c41fb5-cd67-4f4e-9705-3ed284ea6f07

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}