Validators OOM during marshal catch-up after brief downtime under load

[michaelkaplan13]

Summary

Any validator OOM-kills while catching up missed blocks after even a short offline window, under high load. A 3-minute outage was sufficient to kill an otherwise-healthy primary. The marshal's block backfill buffers are bounded by item count (MAX_REPAIR = 200, MAILBOX_SIZE = 1024 in crates/engine/src/engine.rs) while items are ~25 MB blocks, so catch-up admits far more block memory than a 32 GB node holds.

Environment

• deploy.sh config: c8a.4xlarge validators (16 vCPU / 32 GiB), external chain store, --spammer-accounts 49152, --max-propose-bytes 16777216, ~280k TPS sustained, ~100k txs/block, ~163 blocks/min. Stock engine.rs constants.

Reproduction

1. Run a network at the above load.
2. Stop one primary's binary for ~3 minutes (accrues ~500 blocks of backlog), then restart.
3. RSS climbs from ~8 GB baseline to a ~28-29 GB OOM kill within 2-3 minutes of catch-up, then crash-loops. A lagging primary also keeps its RoundRobin leader slots, so the network sees sustained nullifications (~129-159/min from one lagging node) and degraded view latency for the whole catch-up duration.

Investigation

• jemalloc heap profile of a (survivable) primary catch-up burst showed the spike is churn, not accumulation: live in-use heap stayed flat at ~7.2-7.4 GB while RSS ballooned to ~17 GB. The ~9 GB gap is allocator-retained freed pages from a high alloc/free rate during block application (churning paths: MMR/merkle digest work, journal appends, batch apply). Caveat: profiled with jemalloc; production uses mimalloc, which has different page-return behavior, so absolute RSS is allocator-dependent though the mechanism holds.
• This is distinct from the indexer's finalized-upload OOM, which is genuine live-heap accumulation; this one is rate/churn-driven RSS.
• Tried MAX_REPAIR = 16 alone (mailbox stock, mimalloc) against the lethal 3-minute backlog: OOM-killed at 29.2 GB, ~identical to stock. Bounding repair concurrency does not bound the churn, consistent with the mechanism (concurrency != allocation rate). A prior combined test (MAILBOX_SIZE = 64 + MAX_REPAIR = 16) did bound memory to ~6 GB but stalled block application (processed_height froze), so naive count-shrinking trades OOM for a hang.

Directions not yet tried

• Allocator page-return tuning on the stock mimalloc binary (e.g. MIMALLOC_PURGE_DELAY=0) to directly target the retained-page component — zero rebuild, applies to all validators.
• Byte-aware bounds on marshal backfill/delivery (upstream in commonware-consensus), analogous to the indexer's byte budget.
• Lag-aware leader election so a catching-up validator does not hold leader slots it cannot serve (separate consensus concern; would cap the network-wide nullification cost).