MultiLog replication: drift-barrier throttling, inline tail-witness pulses, and AOF shipping backpressure

[Vasileios Zois (vazois)]

Summary

Reworks MultiLog (sharded AOF) replication with an end-to-end backpressure/throttling model that keeps a lagging replica from losing unshipped data while bounding cross-sublog replay drift for read consistency:

• Primary-side AOF shipping backpressure — a fail-safe AofBackpressure gate stalls primary appenders when a replica lags, so an in-memory AOF cannot recycle unshipped pages.
• Reader-side drift-align barrier — replica replay threads align on a ReplayAlignBarrier driven by the read-consistency manager, bounding how far one virtual sublog runs ahead of the others.
• Inline tail-witness time pulses — the primary emits in-band CLUSTER ADVANCE_TIME pulses directly from AofSyncTask, advancing idle-sublog time so readers/barriers don't wait on a silent sublog.

Current status

Functionally complete; still draft (test + cleanup items below). Recent work fixed two deadlock/stall regressions found under load:

• Replay-barrier lost wakeup (fixed) — the shared, re-armed ManualResetEventSlim could drop a release signal (Set() racing the next round's Reset()), stranding a parked replay thread until ReplicaSyncTimeout. Replaced with one reusable event per participant (virtual sublog), each with a single waiter and single resetter, so a release can never be lost to a concurrent reset. (ReplayAlignBarrier)
• Advance-time pulse starvation under backpressure (fixed) — the pulse was suppressed whenever no tail moved; under a backpressure stall (tails frozen) an idle replica sublog then never arrived at a barrier round, self-reinforcing the freeze. The pulse now also fires while the primary is backpressure-stalled (AofBackpressure.AnyStalled(), a global OR across sublogs), which lets idle sublogs arrive, the round complete, and the stall drain.
• Clarity renames — barrier API renamed for intent: TryActivate → TryOpenRound, CheckAndWait → SignalArrivalAndWait, CheckAndArrive → SignalArrival, private Arrive → WaitForAllArrivals.

Key changes

• libs/server/AOF/AofBackpressure.cs (new) — conservative per-sublog gate. Appenders self-check tail - shippedWatermark against a per-sublog budget; the shipping side publishes a min-shipped watermark. A stale watermark only over-estimates lag, so it is fail-safe. Adds read-only AnyStalled() used by the pulse path.
• GarnetLog / GarnetAppendOnlyFile — gate wired into every append path before sublog locks.
• AofSyncDriverStore / AofSyncTask — publish min-shipped watermark on ship (byte-progress gated) and on driver-set changes; emit inline advance-time pulses (with the backpressure-heartbeat behavior above).
• ReadConsistencyManager / ReplayAlignBarrier — reader-side alignment barrier, per-participant wakeup events, and AdvanceVirtualSublogTime pulse application.
• Removed the old advance-time throttle-driver plumbing (ReplicationManager, TaskType, AofSyncDriver).

Configuration — how each option affects MultiLog execution

General effect only (not recommended values).

Primary

Option	Effect on execution
ReplicaSyncDelayMs	Spin-wait between primary ship-loop polls for new records. Lower → tighter polling, lower shipping latency, higher CPU spin; higher → batchier shipping, less CPU, more latency.
AofTailWitnessFreqMs	Minimum spacing of advance-time pulses for idle sublogs. Lower/0 → idle-sublog time advances promptly (readers/barriers wait less) at the cost of more pulse traffic; higher → coarser idle-time advancement and fewer pulses.
AofSyncMaxLagBytes	Primary append backpressure budget (whole-log, split evenly per sublog). Lower → appenders stall sooner when a replica lags (tighter memory bound, more write throttling); higher → more unshipped data tolerated before stalling (higher write throughput, larger in-flight window); -1 disables — appends never throttle and an in-memory AOF may drop unshipped pages.

Replica

Option	Effect on execution
AofReplayMaxLagBytes	Replay mode/lag. 0 → synchronous replay (lockstep with receive); >0 → asynchronous background replay allowed up to that lag (higher throughput, reads may trail); -1 → unbounded async lag.
AofReplayDriftThreshold	Cross-sublog replay drift tolerated before a replay-align barrier round fires. Lower → tighter cross-sublog alignment (stronger read consistency, more frequent rounds → more coordination overhead); higher → looser alignment, fewer rounds, more drift; -1 disables the barrier.
AofReplayDriftCheckFreq	How often drift is proactively re-checked during replay, as a multiple of the threshold. 0 → only readers about to wait trigger rounds (lazy, minimal proactive scanning); >0 → replay scans for drift every freq × threshold (more proactive alignment, more scan overhead).
AofReplayBarrierSpinUs	How a replay thread waits at the barrier. -1 → spin forever (lowest wake latency, burns a core while parked); 0 → sleep immediately (lowest CPU, higher wake latency); >0 → spin up to that many microseconds, then sleep (hybrid).

[Copilot]

🟡 Changes recommended

There are correctness and concurrency issues in the updated iterator wait path and replay-state monotonic publishing that can lead to busy looping or stale/incorrect sequence maxima.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.

Pull request overview

This PR refactors MultiLog (sharded AOF) replication to add an explicit end-to-end backpressure/throttling model: reader-side replay drift alignment via a new ReplayAlignBarrier, primary-side AOF append backpressure via a new AofBackpressure gate, and in-band CLUSTER ADVANCE_TIME “tail-witness” pulses emitted inline from AofSyncTask.

Changes:

• Added primary-side AOF shipping backpressure (AofBackpressure) and wired it into all append paths and driver-store shipped-watermark publishing.
• Replaced prior advance-time/throttle-driver plumbing with in-band per-sublog CLUSTER ADVANCE_TIME pulses and replica-side pulse routing/application.
• Updated configuration surface (options/defaults/runtime config/docs/tests/bench) to rename and add replication tuning knobs (aof-replay-max-lag-bytes, aof-sync-max-lag-bytes, drift barrier options).

File summaries

File	Description
website/docs/getting-started/configuration.md	Documents new/renamed AOF replication/backpressure & drift-barrier options
test/standalone/Garnet.test/TestUtils.cs	Updates test server option wiring to AofReplayMaxLagBytes
test/standalone/Garnet.test/RespConfigTests.cs	Adds CONFIG round-trip coverage for new lag options (incl. 64-bit budget)
test/Garnet.fuzz/Targets/GarnetEndToEnd.cs	Updates fuzz target server options to new lag setting
libs/storage/Tsavorite/cs/src/core/TsavoriteLog/TsavoriteLogScanIterator.cs	Alters iterator wait behavior in BulkConsumeAllAsync
libs/server/TaskManager/TaskType.cs	Removes replica advance-time task type mapping
libs/server/StoreWrapper.cs	Adds runtime application hook for aof-sync-max-lag-bytes updates
libs/server/Servers/GarnetServerOptions.cs	Replaces drift-throttle option with drift-barrier options; renames/extends lag options
libs/server/Config/ServerConfigType.cs	Renames/expands runtime config keys for replay/sync lag budgets
libs/server/Config/RuntimeServerConfig.cs	Adds runtime-config entry + update action for aof-sync-max-lag-bytes
libs/server/AOF/ReadConsistency/VirtualSublogReplayState.cs	Adds waiter reuse, prefetch, min-waiter optimization; changes monotonic update logic
libs/server/AOF/ReadConsistency/ReplicaReadSessionContext.cs	Adds per-session reusable waiter lifetime management
libs/server/AOF/ReadConsistency/ReplayAlignBarrier.cs	Introduces replay drift alignment barrier implementation
libs/server/AOF/ReadConsistency/ReadConsistencyManager.cs	Wires drift barrier activation/arrival and time-pulse application
libs/server/AOF/GarnetLog.cs	Wires backpressure checks into all append/enqueue paths
libs/server/AOF/GarnetAppendOnlyFile.cs	Constructs/disposes backpressure gate; disables old manager safely on update
libs/server/AOF/AofBackpressure.cs	New backpressure gate with per-sublog budgets + shipped watermark publishing
libs/host/defaults.conf	Adds defaults for new drift-barrier and backpressure options
libs/host/Configuration/Options.cs	Adds CLI flags for drift barrier and AOF sync/replay lag budgets
libs/cluster/Session/RespClusterReplicationCommands.cs	Parses/routs per-sublog in-band ADVANCE_TIME pulses
libs/cluster/Server/Replication/ReplicationManager.cs	Removes advance-time work-queue background task plumbing
libs/cluster/Server/Replication/ReplicaOps/ReplicaDisklessSync.cs	Removes advance-time task stop/start during diskless sync
libs/cluster/Server/Replication/ReplicaOps/ReplicaDiskbasedSync.cs	Removes advance-time task stop/start during diskbased sync
libs/cluster/Server/Replication/ReplicaOps/AOFReplay/ReplicaReplayTask.cs	Applies staged pulses vs. normal record replay path
libs/cluster/Server/Replication/ReplicaOps/AOFReplay/ReplicaReplaySession.cs	Updates synchronous replay checks + routes throttling/pulse application
libs/cluster/Server/Replication/ReplicaOps/AOFReplay/ReplicaReplayDriver.cs	Replaces drift throttle with pulse staging and application logic
libs/cluster/Server/Replication/PrimaryOps/AofOperations/AofSyncTask.cs	Adds inline tail-witness pulses + shipped-watermark publishing
libs/cluster/Server/Replication/PrimaryOps/AofOperations/AofSyncDriverStore.cs	Publishes min shipped watermarks to backpressure gate on progress/driver changes
libs/cluster/Server/Replication/PrimaryOps/AofOperations/AofSyncDriver.cs	Removes separate advance-time background task; passes driver-store to sync tasks
libs/cluster/Server/Failover/ReplicaFailoverSession.cs	Removes cancellations related to removed replica advance-time task
libs/cluster/Server/ClusterProvider.cs	Adds replication-info diagnostics for vector lag and per-sublog sequence metrics
libs/client/ClientSession/GarnetClientSessionReplicationExtensions.cs	Makes ADVANCE_TIME fire-and-forget and per-sublog
benchmark/Resp.benchmark/OfflineBench/GarnetServerInstance.cs	Updates bench options to new replay lag setting
benchmark/Resp.benchmark/OfflineBench/AOFBench/AofGen.cs	Updates bench options to new replay lag setting
benchmark/Resp.benchmark/OfflineBench/AOFBench/AofBench.cs	Updates bench options to new replay lag setting

Review details

Suppressed comments (1)

libs/server/AOF/ReadConsistency/VirtualSublogReplayState.cs:122

• UpdateKeySequenceNumber has the same monotonicity race: multiple writers can observe a stale slot value and overwrite a newer/larger sequence number with a smaller one. This breaks the documented guarantee that updates are monotonically increasing.

        public void UpdateKeySequenceNumber(long hash, long sequenceNumber)
        {
            if (sequenceNumber > sketch[GetSketchSlot(hash)])
                Volatile.Write(ref sketch[GetSketchSlot(hash)], sequenceNumber);
            SignalWaiters();

• Files reviewed: 35/35 changed files
• Comments generated: 7
• Review effort level: Lite

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.