-
Notifications
You must be signed in to change notification settings - Fork 35
Open
Labels
bugSomething isn't workingSomething isn't working
Description
Crash: integer overflow in SSZ serializedSize during processCachedDescendants
Severity: Critical — node crashes repeatedly during catch-up sync, preventing participation
Stack Trace
thread 1 panic: integer overflow
ssz-0.0.9/src/lib.zig:45:9: 0x17d5ede in serializedSize__anon_10551 (zeam)
ssz-0.0.9/src/lib.zig:89:43: 0x18e5e50 in serialize__anon_1654536 (zeam)
pkgs/database/src/rocksdb.zig:209:30: 0x18ebff3 in onBlock (zeam)
pkgs/node/src/node.zig:410:57: 0x1a99f30 in processCachedDescendants (zeam)
pkgs/node/src/node.zig:489:42: 0x17f1254 in onInterval (zeam)
pkgs/node/src/utils.zig:42:33: 0x1b6c398 in callback (zeam)
libxev/src/backend/io_uring.zig:806:29: 0x17ba9eb in run (zeam)
pkgs/cli/src/node.zig:426:27: 0x181404a in run (zeam)
pkgs/cli/src/main.zig:780:26: 0x181c8bb in mainInner (zeam)
pkgs/cli/src/main.zig:241:14: 0x17b7fdd in main (zeam)
The crash occurs consistently — same stack trace on every restart, hitting the same code path in processCachedDescendants → onBlock → ssz serialize.
Observed Behaviour
- Node starts with checkpoint sync from slot
10570. - Connects to peers and begins catch-up block sync.
- During
processCachedDescendants(triggered byonInterval), the SSZserializedSizecomputation overflows an integer when attempting to serialize a block for RocksDB storage. - Process panics and restarts. This repeats on every restart — the problematic block is re-fetched and re-encountered on each run.
Additional Errors (same session)
error.UnknownSourceBlock— repeated across many blocks during attestation validation inchain.zigerror.OutOfMemory—Failed to process cached blockwarnings during the same catch-up window
Likely Root Cause
The SSZ library uses u32 offsets, so serializedSize can overflow if the serialized size of a block exceeds 2^32 - 1 bytes (~4 GiB). The most plausible cause is a block with an abnormally large list field (e.g. attestations, deposits, or BLS changes) — possibly the node is accepting or constructing a malformed/oversized block during sync.
Could also indicate that multiple blocks are being concatenated or a slice is not properly bounded before being passed to serialize.
Reproduction
- Checkpoint sync from
https://leanpoint.leanroadmap.org/lean/v0/states/finalized(slot10570) - Devnet3 with 9 peers (ethlambda ×5, gean ×1, nlean ×1, qlean ×1, lantern ×1)
- Crash occurs reproducibly within ~1 minute of startup
Suggested Fix
- Add a bounds check in
rocksdb.zig:onBlockbefore callingssz.serialize— validate block size is withinu32range and log + skip if not. - Investigate why a block in the devnet3 chain has an abnormally large serialized size.
- Check if
processCachedDescendantsis accumulating blocks/data across iterations without bounds.
Environment
- zeam devnet3 (v0.3.3)
- Checkpoint sync mode
attestation-committee-count: not set (defaults to spec value)
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't working