alarm: "lost more while already alarming" fires when nothing more was lost

[JSONbored]

lane_lost_again is the alarm's way of saying a dead-letter queue lost further messages while its issue was already open. It has fired twice in the last 14 days and both times nothing more had been lost — once while the loss was actively shrinking.

Observed

Production, probe-jobs-dlq, from the $exception stream (chronological):

2026-08-14 22:58Z  lane_stale       3 message(s)  (netuid=108, netuid=29, sn-51-lium-revenue-for-validators)
2026-08-14 23:58Z  lane_lost_again  2 message(s)  (netuid=108, netuid=29)   <- count DECREASED
2026-08-15 00:28Z  lane_lost_again  2 message(s)  (netuid=108, netuid=29)   <- count UNCHANGED
2026-08-15 01:58Z  lane_stale       2 message(s)  (netuid=108, netuid=29)

The first lost_again reports new loss on a tick where the queue went from 3 messages to 2. The second reports new loss on a tick where the detail string is byte-identical to the previous one.

Each firing also posts a comment on the open alarm(lane): probe-jobs-dlq issue, so the false signal reaches a human on the noisier of the two channels.

Why

laneAlarmPlan decides "lost more" on a timestamp alone (src/lane-alarm.ts, the update loop):

const record = latest[lane];
if (record.checked_at <= openIssue.updatedAt) continue;
update.push({ lane, issue: openIssue.issue, record });

The block's own comment says "Further losses on a queue whose alarm is already open", and the gate is careful about the right things — it re-checks qualified, the residue guard and the stale floor. But none of those look at what was lost. The only test is that a verdict row exists newer than the last comment.

A dead-letter lane writes a verdict every tick by construction — src/dead-letter.ts is explicit that it "writes stale and never writes ok", because a lost message has no recovery. So checked_at advances past updatedAt on every tick, and the condition is true every tick, whatever the queue is doing.

Intent and implementation disagree: the intent is "more was lost", the implementation is "time passed".

Why it isn't already obvious

Volume is low — 2 firings in 14 days against 156 lane_stale — because the update path also requires an open GitHub alarm for that lane, and those get closed promptly. So the bug only shows during the window an alarm is open, which is exactly when someone is looking at it.

The fix is a comparison, not a clock

The verdict's detail already enumerates the messages (2 dead-lettered message(s) on probe-jobs-dlq (netuid=108,netuid=29)), so the information needed is present — it is just not consulted. A recurrence should require the loss to have changed, and specifically to have grown:

• comparing detail to the previously reported one catches both observed cases (3→2 differs but shrank; 2→2 is identical);
• shrinking must not report loss at all. The 3→2 transition shows the count is not monotonic — queue retention ages messages out — so "different" is not sufficient on its own, and a naive inequality would report every ageing-out as a new loss.

Where the previously reported detail lives is the design question. lane_health holds the prior row, so the plan builder could take the previous verdict alongside latest rather than needing new storage.

Not urgent

Two false comments on one issue, and the underlying alarm is correct that the queue has lost messages — netuid=108 and netuid=29 are genuinely dead-lettered and genuinely unrecovered. What is wrong is the claim that they were lost again, which is the kind of thing that teaches a reader to skim the channel.

Found while sweeping live lane health after #11364; probe-jobs-dlq is the one lane currently not ok (1 of 67).

[linear-code]

JSO-8313

[JSONbored]

Closing as invalid — I diagnosed this wrong, and the alarm is correct.

I read summarizeDeadLetterBatch's output as a running total of what is sitting on the queue. It is not: handleDeadLetterBatch is a queue consumer, so each row describes one arriving batch. The actual lane_health rows:

04:31:06   2 dead-lettered message(s) ... (netuid=108,netuid=29)
04:34:24   1 dead-lettered message(s) ... (sn-51-lium-revenue-for-validators)
05:30:31   3 dead-lettered message(s) ... (netuid=108,netuid=29,sn-51-...)
06:30:14   2 dead-lettered message(s) ... (netuid=108,netuid=29)
07:26:50   2 dead-lettered message(s) ... (netuid=108,netuid=29)

Five distinct batches. The 3 -> 2 I called a shrinking queue is a batch of 3 followed by a batch of 2 — both new losses. checked_at only advances when a batch actually arrives, which is exactly what makes the existing timestamp comparison right: one comment per loss, not one per tick, as the test at tests/lane-alarm.test.ts already states.

Both lane_lost_again firings were true positives. The fix I proposed — requiring the count to have grown — would have suppressed real loss reports whenever a later batch happened to be smaller, which on this evidence is most of them.

The real finding underneath

The same two subjects recur across four separate batches:

netuid=108, netuid=29   at 04:31, 05:30, 06:30, 07:26

That is a poison pill, not five unrelated incidents: the compute-declaration reads for netuid 108 and 29 are being re-enqueued, exhausting their retry budget, and dead-lettering again roughly hourly. The alarm is doing its job by saying so each time.

Filed separately as the thing actually worth fixing. Apologies for the noise on this one — the diagnosis was confidently argued and wrong.