Summary
The chain indexer secondary OOM-kills during catch-up whenever it restarts with a backlog of finalized blocks to upload. Memory climbs ~320 MB/s to a ~31 GB kill on a 32 GB node, then crash-loops, because the finalized-upload pipeline admits work bounded by item count while items are large. At ~100k txs/block each in-flight block holds ~300 MB across its concurrent representations (decoded queue entry, prepared QMDB/merkle ops, staged SQL rows, wire-encoded commit), so a full pipeline is a multi-GB commitment independent of store health.
Environment
deploy.sh config: c8a.4xlarge validators (16 vCPU / 32 GiB), external chain store, --spammer-accounts 49152, --max-propose-bytes 16777216, ~280k TPS sustained, ~100k txs/block, ~163 blocks/min.- Indexer is a secondary validator with the full upload pipeline (
upload_buffer/count-bounded, default 64).
Reproduction
- Run a network at the above load with the indexer ingesting normally.
- Restart the indexer (or let it OOM once) so a backlog accrues on the durable finalized-upload queue.
- On restart it fills all ~64 count-bounded pipeline slots immediately; RSS climbs ~320 MB/s to ~31 GB and the kernel OOM-kills it. Because queue acks are gated on upload completion, each cycle makes little durable progress and it crash-loops.
Investigation
- jemalloc heap profile at the moment of death showed ~30.7 GB live (in-use) heap: ~17 GB in the upload pipeline (entry decode, QMDB prepare, commit staging, protobuf encoding), ~5.4 GB marshal delivery, ~6 GB legitimate fixed caches. Genuine accumulation, not allocator churn.
- Confirmed count-bounded, not store-bound: the store's commit latency was flat ~23 ms throughout; the client-side pipeline was the constraint.
Example solution shape
Byte-aware admission into the pipeline, prototyped on branch michael/indexer-byte-admission (commits 905c62d, 301a46d, 6775caf):
- Charge a byte budget (default 3 GiB) on each entry's encoded size × amplification factor before decode; release only after the upload is acked. Backpressure parks the backlog on disk instead of in memory.
- Validated live: restart under full load caught up 2,037 blocks, zero kills,
reserved_bytes pinned at the cap, peak RSS ~24 GB vs ~31 GB death.
- Note: the ~24 GB peak still includes a backlog-proportional marshal share; the pipeline budget bounds only the indexer-owned portion. The marshal's catch-up memory affects all validators and is a separate problem. Two review follow-ups remain: a benign spawn-order
WriterOutOfSync retry, and documenting the amplification constant's provenance.
Summary
The chain indexer secondary OOM-kills during catch-up whenever it restarts with a backlog of finalized blocks to upload. Memory climbs ~320 MB/s to a ~31 GB kill on a 32 GB node, then crash-loops, because the finalized-upload pipeline admits work bounded by item count while items are large. At ~100k txs/block each in-flight block holds ~300 MB across its concurrent representations (decoded queue entry, prepared QMDB/merkle ops, staged SQL rows, wire-encoded commit), so a full pipeline is a multi-GB commitment independent of store health.
Environment
deploy.shconfig: c8a.4xlarge validators (16 vCPU / 32 GiB), external chain store,--spammer-accounts 49152,--max-propose-bytes 16777216, ~280k TPS sustained, ~100k txs/block, ~163 blocks/min.upload_buffer/count-bounded, default 64).Reproduction
Investigation
Example solution shape
Byte-aware admission into the pipeline, prototyped on branch
michael/indexer-byte-admission(commits905c62d,301a46d,6775caf):reserved_bytespinned at the cap, peak RSS ~24 GB vs ~31 GB death.WriterOutOfSyncretry, and documenting the amplification constant's provenance.