Supervisor detects provider death but never replays the manifest — no recovery

[schickling-assistant]

Problem

When a managed session's provider process dies, the supervisor detects it precisely and then does nothing. There is no manifest replay and no restart. Detection is implemented; recovery is not.

Evidence

Linux host, convoy 0.2.0-ts.0, isolated network, one declared agent with both limbs running under a hosted convoy up.

Killing the provider process produced an immediate and correct diagnosis:

worker crash <host>.w1 (vanished (hard death — no exit record)) — dinged 0 orchestrator(s)

The supervisor then ran many 30-second reconcile cycles without restarting anything. The provider stayed dead. .convoy/pty.toml was present and valid the whole time.

Worth stating explicitly, because it is the good news: the detection is genuinely precise — it distinguished a hard death with no exit record, and it did so promptly. The gap is purely what happens next.

Why this matters

The manifest is described as a launch spec that a session can be replayed from — that is what makes a managed session survivable rather than something an operator has to babysit. Without replay:

• an unattended role does not come back from a crash, a provider OOM, or a host reboot
• the manifest's value is limited to initial launch
• an operator has to notice and intervene manually, which is exactly what hosting was meant to remove

This compounds badly with the sidecar liveness defect (compoundingtech/smalltalk#101): the provider stays dead and the bus keeps reporting it healthy. Together they mean a crashed always-on session fails silently and stays failed. Either one alone is recoverable by an attentive operator; both together are not.

Proposed direction

The reconcile loop already knows the session is dead and already has the manifest. The minimal change is for the crash path to replay .convoy/pty.toml rather than only reporting.

Things worth deciding rather than assuming:

• Restart policy. Unconditional restart risks a crash loop against a genuinely broken config. Some backoff and a give-up threshold probably belong here — there is already a flapping cap in convoy up, so possibly this reuses it.
• Which limbs. If only the provider died, does the existing Ding get reused or torn down and relaunched with it? Relaunching both is simpler to reason about; reusing risks a sidecar bound to a stale target.
• Cold boot on replay. Replay should presumably cold-boot from external context rather than pinning a provider transcript, consistent with the manifest being a launch spec rather than a conversation pin. Worth confirming that is the intent.

What I would like consensus on

1. Is recovery-on-death in scope for the supervisor, or is it deliberately delegated to an external process manager? If delegated, that should probably be documented, since the crash log reads as though convoy intends to handle it.
2. If in scope: restart policy — unconditional, backoff, or capped attempts before it stays down and escalates?
3. On a provider-only death, relaunch both limbs or reattach the survivor?

I can put up a PR for the minimal version (replay the manifest, reuse the existing flapping cap, relaunch both limbs) if that shape sounds right — but the answers to 2 and 3 change the implementation enough that I would rather agree first.

Notes from the same session

Smaller things observed while testing this, filed here as context rather than as asks — happy to split any of them out:

• convoy up --once does not pre-trust the agent directory, so the provider sits at the harness trust prompt forever. Hosted convoy up does pre-trust. As-is, --once is not usable headlessly.
• convoy init exits 0 after failing to create the CoS, leaving an empty catalog behind.
• Every supervisor log line is printed twice.

[schickling-assistant]

Correcting the framing above

Reproducing this properly (see the linked PR) found the diagnosis in this issue is wrong in two ways, and the real defect is worse than reported.

The evidence line is misattributed. The worker crash … — dinged 0 orchestrator(s) message I quoted fires only on the non-permanent branch, which is deliberately no-respawn. That line is the ephemeral path working as designed. The production concern — a declared always-on agent that dies and stays dead — is the permanent branch, a different code path.

"Detection works, recovery is missing" is wrong. Recovery was wired up. It could never succeed, because of a deadlock between two components:

• pty's cmdRestart refuses any role=agent session and directs the caller to "cycle it through its supervisor (e.g. convoy up)".
• convoy's respawn primitive was pty restart -y — that exact refused command, without --force.

pty defers to the supervisor; the supervisor calls back into pty's refusal. Neither side is individually wrong, which is why it survived review on both.

It is broader than hard death. Killing the daemon (vanished) and killing only the inner provider (clean exit, code 137) both fail identically. Every permanent-agent respawn was broken, on every tick — not only after an unclean death.

A second defect falls out of the first. The flapping cap infers a fast fail from the leaf's exit record, which is blind when the spawn never happened at all. So the counter stays at 0/3 across unbounded cycles: the session never parks, never dings, and retries silently forever. The cap that exists to bound this case cannot see it.

The fix in the PR replays the manifest rather than passing --force, on the reasoning that manifest replay is the supervisor path pty's guard defers to, whereas forcing would reintroduce the wedge that guard exists to prevent. That judgement is the thing worth pushing back on: if the preference is instead that pty grows a supervisor-blessed restart mode, the PR is the wrong shape.