Lossless event delivery: unbounded intrusive SPSC queue + node freelist (no drops, flat footprint)

[Ivansete-status]

Summary

Rework the event delivery queue so that no event can ever be dropped, while keeping the steady-state memory footprint flat (near-zero allocation on the hot path). The proposal is to replace the current fixed-capacity ring (introduced in #107, closing #92) with a single unbounded intrusive SPSC queue backed by a node freelist — reusing the mechanism already proven for FFI requests in #102, minus the sharding.

Motivation — the current design drops events, terminally

Event delivery today is a bounded ring (EventQueue, EventQueueCapacity = 1024). The FFI thread is the sole producer; the event thread is the sole consumer that invokes listener callbacks. When the ring is full, the overflowing event is discarded, and the failure is not transient:

• ffi/ffi_events.nim — enqueueOrMarkStuck: on a full ring it logs "event queue full; library marked stuck", sets the sticky eventQueueStuck flag, and breaks — the event is never enqueued, never retried, never delivered.
• ffi/event_thread.nim:117 — notifiedStuck is documented as "latched forever — eventQueueStuck is sticky terminal." Nothing clears it.
• ffi/ffi_thread.nim:16 — sendRequestToFFIThread starts with if ctx.eventQueueStuck.load(): return err(...). So once the ring overflows once, the context (a) loses that event and (b) rejects all future FFI requests — the library is effectively bricked.

A single slow/wedged listener that lets the ring fill is enough to trigger this. Losing events is unacceptable for our use case, and the terminal-stuck cascade makes it worse. This issue's primary goal is to make event delivery lossless.

Non-negotiables

1. No event is ever dropped. This is the top priority. Under sustained consumer backlog the queue must grow rather than discard. (We explicitly accept unbounded growth under a wedged listener — see "Backpressure / observability" — because dropping is not an option.)
2. Flat steady-state memory footprint. In normal operation (bounded in-flight events) the delivery path must allocate ~nothing — buffers are recycled, not malloc/free'd per event. This preserves the allocation win feat(ffi): slab-allocate event payloads instead of c_malloc per dispatch #107 was built for.
3. The FFI (producer) thread is never blocked by a slow listener. Enqueue is O(1) and always succeeds; listener callbacks run on the event thread only.
4. Strict FIFO event ordering (a started is always delivered before its stopped).

Proposed design

Reuse the request-queue mechanism from #102 (ffi/ffi_request_queue.nim), adapted to the events' single-producer/single-consumer shape:

What transfers from #102

• Unbounded intrusive linked list, c_malloc-backed, with a next link where the node is its own payload carrier — survives the FFI→event-thread hop (same TLS/MemRegion hazard already documented in alloc.nim / ffi_events.nim).
• Lock-guarded O(1) push that always succeeds → producer never blocks, events never dropped.
• Wake-on-empty + short poll signaling (already present).

What does NOT transfer: the sharding (bank of queues)

The #102 RequestQueueBank shards into 16 queues to spread multi-producer contention. Events have a single producer, so:

• there is no producer-producer contention to spread — a bank would be pure overhead (N locks/heads/tails), and
• sharding only guarantees per-shard order; a single queue gives strict global FIFO for free.

So the event queue is effectively one shard of the request bank — SPSC, one lock, one head/tail.

The piece #102 lacks: a node freelist (keeps the footprint flat)

The request queue mallocs a node per request and frees it after dispatch. To hold requirement #2, add recycling:

• Push (FFI thread): pop a node from the freelist (or c_malloc if empty), copy name + payload into it (reuse feat(ffi): slab-allocate event payloads instead of c_malloc per dispatch #107's inline-slot-vs-heap-fallback for oversize values via MaxEventPayloadBytes / MaxEventNameBytes), link into the tail.
• Commit (event thread): after the callback returns, return the node to the freelist instead of freeing it.

Steady state → the freelist recycles a small working set → zero allocation. Backlog → freelist drains, we c_malloc more → grows, never drops. No fixed upfront 576 KB slab; memory tracks the high-water mark and recycles from there.

Invariant to carry over from #107 (peek → dispatch → commit)

• pop-under-lock → release lock → run callback → re-acquire lock → return node to freelist.
• The callback MUST run outside the queue lock — holding the lock across a slow callback would silently re-couple the producer and consumer (the exact bug to avoid).
• While the callback runs, the node is off both the ready-list and the freelist (owned by the consumer), so the producer cannot grab and overwrite a buffer still being read. This is feat(ffi): slab-allocate event payloads instead of c_malloc per dispatch #107's slot-pinning contract, unchanged — only "clear the ring slot" becomes "return node to freelist."

Backpressure / observability (replacing the drop)

Dropping was doing crude backpressure; removing it means a wedged listener grows memory unbounded → eventual OOM. Since we will not drop, make the condition loud and visible instead:

• Track and expose the queue depth / high-water mark.
• Log a warning when depth crosses a threshold and when it recovers.
• The existing heartbeat watchdog (onNotResponding / onResponding) already surfaces a wedged consumer — keep it.
• If a hard ceiling is ever needed, apply backpressure at the event source (slow emission), never by discarding at the queue.

Remove the eventQueueStuck sticky-terminal machinery and the request-intake gate that depends on it (ffi/ffi_thread.nim:16), since overflow-to-drop no longer exists.

Acceptance criteria

• No code path discards an event; a full/backlogged consumer causes the queue to grow, not drop.
• Steady-state emission (in-flight ≤ working set) allocates nothing on the hot path (freelist recycles nodes + buffers); verified under ASan/UBSan/LSan (detect_leaks=1) and TSan, matching feat(ffi): slab-allocate event payloads instead of c_malloc per dispatch #107's bar.
• Producer (FFI thread) is never blocked by a listener; callback runs outside the queue lock.
• Strict FIFO delivery preserved.
• eventQueueStuck and its request-intake gate are removed; a wedged listener is surfaced via watchdog + depth logging, not a terminal stuck state.
• Existing event tests stay green (orc + refc): test_event_dispatch.nim, test_event_thread.nim, test_event_listener.nim.
• New tests: sustained backlog grows and fully drains with zero loss and no leak; oversize name/payload heap-fallback still works; freelist reuse is exercised (steady state does not grow RSS).

Notes / future

• Long term, the FFI request bank could adopt the same node freelist to eliminate its per-request c_malloc too — the two subsystems would then share one primitive (unbounded intrusive queue), parameterized only by shard count (N for requests, 1 for events) and whether recycling is enabled.

References

• feat(ffi): slab-allocate event payloads instead of c_malloc per dispatch #107 (current slab ring), Slab-allocate event payloads instead of c_malloc per dispatch #92 (allocation issue feat(ffi): slab-allocate event payloads instead of c_malloc per dispatch #107 closed), feat(codegen): C binding generator (-d:targetLang=c) #102 (request queue bank / ffi/ffi_request_queue.nim).
• Current drop path: ffi/ffi_events.nim enqueueOrMarkStuck / tryEnqueueEvent.
• Sticky-terminal + watchdog: ffi/event_thread.nim:117, :126.
• Request-intake gate on stuck: ffi/ffi_thread.nim:16.