Ding refreshes status for a dead provider — an unarmed sidecar forges liveness

[schickling-assistant]

Problem

A Ding sidecar that never armed still refreshes its target's status. When the provider process dies, Ding keeps writing liveness, so the agent reads available indefinitely — for a process that no longer exists.

This makes status unsafe to trust as a liveness signal, which is the one job it has.

Why this matters

For an always-on supervised role (a Chief of Staff, an orchestrator), the failure is silent and permanent:

• the provider crashes
• Ding keeps the status fresh, so every consumer reads the agent as healthy
• nothing alerts, nothing restarts

The agent is dead and the bus says it is fine. An operator has no signal until they notice the work stopped. This is worse than a missing feature, because the system actively reports the opposite of the truth during the exact failure the signal exists to expose.

It also breaks the intended contract that a sidecar keeps liveness fresh and lets status decay honestly when the session is no longer running. Only the first half currently holds.

Evidence

Observed on a Linux host, convoy 0.2.0-ts.0, running a managed session with two limbs — provider <host>.w1 and sidecar <host>.w1.ding — in an isolated network.

1. Status advanced while the provider was dead. After killing the provider process, status mtime kept moving forward and both st agents and convoy ls --tree reported available.

2. Ding was the writer. Killing Ding froze the mtime permanently.

3. Observer effect ruled out. With both limbs dead, running st agents and convoy ls repeatedly left the mtime untouched. So the refresh was not a side effect of reading — Ding was actively writing.

4. Ding never armed. Its entire event log for the session is 206 bytes containing a single session_start record. The log shows:

target session "<host>.w1" not yet registered

despite the target being registered at that point. Ding never armed, never delivered a message — and still wrote liveness for the whole session.

So the status write is not gated on the sidecar having successfully attached to its target.

Proposed direction

The invariant I would suggest: a sidecar must not write liveness it has not earned. Concretely, one of:

• (a) Arm-or-exit. If Ding cannot register against its target, it exits non-zero rather than running in a degraded state that still writes status. Simplest, and makes the failure loud.
• (b) Gate the write on armed state. Ding keeps running but writes status only while it holds a verified attachment to a live target.
• (c) Liveness derives from the provider, not the sidecar. Ding reports what it observes rather than being the source of truth.

(a) is the smallest change and would have surfaced this immediately. (c) is the most principled but the largest.

What I would like consensus on

1. Is Ding intended to be the writer of status at all, or is that an accident of it being the long-lived limb?
2. Should an unarmed sidecar be a hard failure, or a degraded-but-running state? If degraded is intentional, what should it be allowed to write?
3. Does status mean "this identity has a live session" or "something is tending this folder"? The two readings diverge exactly here, and consumers currently assume the first.

Happy to put up a PR once there is agreement on which of (a)/(b)/(c) is wanted — the fix differs a lot depending on the answer to Q3.

Related

• coord mcp does not assert available on connect/resume — supervised sessions come back alive but offline #12 is the inverse symptom (failing to assert available on connect/resume). Both point at status not having a single owner with a clear contract.

• Supervisor detects provider death but never replays the manifest — no recovery convoy#82 — the supervisor does not restart the dead provider. Together with this issue, a crashed session stays dead and reads healthy.

[schickling-assistant]

Production reproduction

This now reproduces on a real hosted agent, not a harness — and it is the blocker on that agent being reachable at all.

Setup: a supervised agent (<host>.cos) hosted by convoy up, two limbs running (provider + ding sidecar), bus folder materialized with status = available.

Sequence:

1. The agent cold-booted and ran its boot ritual, correctly reporting inbox empty.
2. A message was sent to it a minute later. It landed in inbox/ — delivery to the folder works.
3. Nothing woke the agent. It continued working on an unrelated task; the message sat unread indefinitely.

The sidecar's entire event log for the session is a single record:

{"session":"<host>.cos.ding","type":"session_start","ts":"...","tags":{"role":"ding",...}}

No arm, no delivery, no error — the same signature as the original report. The sidecar process is alive the whole time.

So the practical effect is broader than a stale status: an agent that boots, drains, and then goes idle is unreachable for the rest of its life. The bus accepts mail that is never surfaced. For an always-on role this is the difference between a working agent and an ornamental one.

Worth noting the two failure modes compound in opposite directions. The sidecar does not arm (so it never delivers) yet still refreshes status (so the agent reads available). A sender sees a healthy, available recipient and gets no response.

[schickling-assistant]

Correcting my earlier comment

Two claims in the production report above are wrong, and the diagnosis they pointed at is wrong with them. Root-causing this properly (see the linked PR) found something different.

"The sidecar never arms" is false. It arms correctly. The single session_start record I cited is not written by smalltalk at all — it is convoy/pty telemetry — so it was never evidence about the sidecar's own state. And the target session "<id>" not yet registered line does not gate delivery; a ding against a target with no pidfile delivers normally, since the session-watch only governs exit.

"Unreachable for the rest of its life" overstates it. An idle pane is byte-static, and a held message lands within roughly twenty seconds of the agent going idle. The accurate statement is that an agent is unreachable while its pane keeps changing — which, for a working agent, is most of the time, and for a busy one is indefinitely.

The actual mechanism is a hold with no upper bound. On the no-input branch of guardedDeliver, a changing pane frame returns held unconditionally; forceCap is computed at the top of the function but only consulted on the un-submitted-input branch. assessPane byte-compares two peek frames 300 ms apart, so any agent that is doing work looks "changing" forever. The hold is logged only under a debug flag that production does not set, while the status refresher — decoupled from delivery — keeps bumping the mtime.

So the invariant this issue should assert is narrower and more precise than the one I proposed. Not "an unarmed sidecar must not write liveness", but an armed sidecar that is not delivering must not keep writing available. The gate belongs on delivery state, not arm state.