-
Notifications
You must be signed in to change notification settings - Fork 35
Description
Description
Zeam crashes repeatedly with thread 1 panic: integer overflow during catch-up sync after checkpoint-sync startup. The crash occurs in the SSZ serialization path when writing a block to RocksDB inside processCachedDescendants.
The same crash recurs on every restart (the node auto-restarts, syncs partially, then crashes again at roughly the same point).
Stack Trace
thread 1 panic: integer overflow
/root/.cache/zig/p/ssz-0.0.9-.../src/lib.zig:45:9: 0x17d5ede in serializedSize__anon_10551 (zeam)
/root/.cache/zig/p/ssz-0.0.9-.../src/lib.zig:89:43: 0x18e5e50 in serialize__anon_1654536 (zeam)
/app/pkgs/database/src/rocksdb.zig:209:30: 0x18ebff3 in onBlock (zeam)
/app/pkgs/node/src/node.zig:410:57: 0x1a99f30 in processCachedDescendants (zeam)
/app/pkgs/node/src/node.zig:489:42: 0x17f1254 in onInterval (zeam)
/root/.cache/zig/p/libxev-.../src/backend/io_uring.zig:806:29: 0x17ba9eb in run (zeam)
/app/pkgs/cli/src/node.zig:426:27: 0x181404a in run (zeam)
/app/pkgs/cli/src/main.zig:780:26: 0x181c8bb in mainInner (zeam)
/app/pkgs/cli/src/main.zig:241:14: 0x17b7fdd in main (zeam)
The overflow occurs in ssz.serializedSize (lib.zig:45) when computing the byte size of a block before writing it to RocksDB. This suggests a block with an unexpectedly large or malformed field is being serialized — possibly a block received during the catch-up sync that has an oversized variable-length field (e.g. attestations list) whose byte-length overflows a u32 or similar integer type used in the SSZ size calculation.
Additional Errors Observed
Prior to each crash, the log repeatedly shows:
[error] [chain] invalid aggregated attestation data in block: error=error.UnknownSourceBlock
[warning] [node] Failed to process cached block <root>: error.OutOfMemory
The OutOfMemory warnings during catch-up sync suggest memory pressure from the large volume of cached blocks, which may be contributing to the corrupted state that triggers the overflow.
Reproduction Steps
- Start zeam with checkpoint sync:
--checkpoint-sync-url https://leanpoint.leanroadmap.org/lean/v0/states/finalized - Node starts at slot 10570, peers are at slot 10586+
- During catch-up sync (
processCachedDescendants), node crashes with integer overflow - Auto-restart → same crash repeats consistently
Environment
- Started at slot: 10570 (checkpoint sync anchor)
- Peers: ethlambda ×5, gean_0, nlean_0, qlean_0, lantern_0
- SSZ version:
ssz-0.0.9 - Crash consistently in
processCachedDescendants→onBlock→rocksdb.zig:209