fix(client): recover from stuck operator suspend

[baixiaohang]

Summary

• bound manual operator suspend settlement to 30 seconds so a lost provider completion cannot fence later inbox delivery forever
• establish the inbox operator-suspend boundary on timeout, resolving only the proven processing prefix and retaining the unresolved tail as recovery debt before any fresh provider route
• abandon the timed-out handler and its current route-producer join at an explicit generation boundary: late output remains fenced, ordinary routes and daemon shutdown stay available, and Reset still fails closed until stop/settlement is proven
• keep abandoned teardown retryable: Reset refuses to join a still-pending lost callback, but starts a fresh strict shutdown after a failed attempt has settled and clears debt only on confirmed success
• fence that strict retry behind abandoned-producer settlement so a late stale completion must materialize its after-prior teardown debt before Reset can retry or clear handler-keyed debt
• cover real dispatch/ledger custody, a never-settling route producer, repeated resume, Reset safety and convergence, and manager shutdown with regression tests

Why this shape

A laptop sleep and wake commonly continues the same daemon process, so startup-only cleanup does not run in the reported failure mode. A true daemon restart already drops the in-memory suspend barrier and uses normal inbox recovery. This keeps the timeout local to manual pause while preserving the durable recovery and Reset safety boundaries.

Testing

• pnpm --filter @first-tree/client test (2415 passed, 7 skipped)
• pnpm check && pnpm typecheck
• pnpm test was also attempted on the earlier head; full parallel execution hit unrelated existing Web and CLI hook timeouts. Both failed files passed when rerun independently.

Fixes #2144

[yuezengwu]

Requesting changes for two blocking recovery gaps:

1. The timeout branch returns before establishing the inbox settlement/recovery boundary. At suspendSession() lines 4207-4218, an established session with a timed-out suspend skips prepareOperatorSuspend(). Its SessionEntry remains suspended (not evicted) and the coordinator has no recoveryDebt, so both dispatch() and recoverDebtBeforeResume() allow the next inbox row to enter the fresh handler immediately. If the ledger still contains an older provider-entered or queued row, that later row executes before the prefix is resolved; only its eventual finishTurn() notices the non-terminal prefix gap and requests recovery, after execution, making the later row eligible for redelivery/replay. Please establish the correct coordinator boundary before any fresh provider re-entry (resolve only the provable consumed prefix and retain the rest as recovery debt), and extend the regression test with real ledger custody to prove recovery happens before freshHandler.resume().

2. A permanently stuck provider callback can still fence the chat again after the first rescue. The timeout path marks the old handler retired, but handlerForRouteTransition() registers its shutdown as ordinary pendingTeardowns only after the current settleTeardownDebtBeforeRoute() call has passed. This permits exactly one fresh route. If the old shutdown() also never settles—which is the relevant case for handlers such as Claude where shutdown() delegates to the same suspend() wait—the pending raw shutdown remains forever. After the fresh handler later suspends, the next resume joins that old promise in settleTeardownDebtBeforeRoute() and hangs indefinitely (and manager shutdown waits on it too). The abandoned timed-out generation needs a bounded/non-route-blocking teardown policy that still preserves late-output fencing and Reset safety. Please add a regression with both old suspend() and shutdown() never settling, then suspend the fresh handler and prove a second resume can proceed.

The generation fence itself is the right containment mechanism, but the current test only covers a handler with immediately successful shutdown and no inbox ledger state, so it misses both production failure modes above.

[baixiaohang]

Addressed the blocking recovery gaps in eb101cbe9:

1. The timeout branch now completes prepareOperatorSuspend() before releasing the suspend barrier. The proven processing prefix is ACKed, the unresolved/queued tail remains recovery debt, and redelivery recovery is requested before a fresh provider route can start.
2. A timed-out handler is now explicit abandoned teardown debt. Its best-effort shutdown is not joined by ordinary route admission or manager shutdown, so a permanently stuck shutdown() cannot fence the chat again. The handler stays retired/generation-fenced, and Reset fails closed while its stop is unconfirmed.
3. The route producer present at the timeout boundary is explicitly abandoned as part of the same generation transition. Fresh route admission and manager shutdown do not wait on that lost producer join. It remains tracked for Reset safety; if it completes late, stale adoption is rejected and its materialized handler flows into teardown debt.

The regression coverage now:

• creates a real never-settling producer through dispatch() and real inbox-ledger custody;
• proves prefix ACK + tail recovery occur before the fresh handler enters;
• leaves both the old suspend() and shutdown() unresolved, then proves a fresh route and a later second resume both proceed;
• proves Reset rejects while the abandoned producer/handler is unconfirmed;
• proves manager shutdown returns without joining the abandoned callbacks.

Validation: full @first-tree/client suite (2413 passed, 7 skipped), pnpm check, and pnpm typecheck all pass. Please take another look.

[yuezengwu]

The two previously reported routing/custody blockers are addressed, but the new abandoned-debt path introduces one blocking Reset convergence issue.

abandonTimedOutSuspendHandler() clears abandonedSuspendHandlers only when its first best-effort shutdown() resolves. If that shutdown rejects, the rejection branch deliberately leaves both the marker and pendingTeardowns entry in place. Every later session:terminate then sees the marker at lines 1146-1150 / 1191-1195 and throws before attempting shutdownHandler() again. Since the completed rejected shutdown record is otherwise retryable, this turns a transient teardown failure into permanent Reset failure: even if the next shutdown attempt would succeed, no Reset retry can ever make it.

That conflicts with the existing Reset contract and with the nearby teardown comments that a failed apply remains retryable and teardown retry converges. Please let a later Reset perform a strict retry for an abandoned handler once the original shutdown promise has settled (while still refusing to join a genuinely pending lost callback), and add a regression where the timed-out handler's first shutdown rejects, the next shutdown succeeds, and a subsequent Reset completes. The currently added test covers only a shutdown promise that never settles, so it cannot catch this branch.

[baixiaohang]

Addressed the Reset convergence blocker in d8a511382.

The abandoned marker now distinguishes the two states using the existing coalesced shutdown record:

• If the original raw shutdown is still registered, Reset fails closed immediately and does not join the potentially lost callback.
• If that shutdown has settled with rejection, Reset starts a fresh strict shutdown() attempt. Confirmed success clears both pendingTeardowns and the abandoned marker; another rejection preserves both so a later Reset can retry again.

This applies to both an abandoned handler still attached to the current session and one detached into pending teardown debt. Ordinary route admission continues to skip abandoned debt, so this does not reintroduce the route hang.

The new regression times out suspend(), makes the first best-effort shutdown() reject, verifies the debt remains, then proves the next session:terminate performs a second shutdown and fully clears the session/debt.

Validation:

• focused edge suite: 146 passed
• full @first-tree/client suite: 2414 passed, 7 skipped
• pnpm check && pnpm typecheck: passed

Please take another look.

[yuezengwu]

Approved. The latest change resolves the remaining Reset convergence blocker: a still-pending abandoned shutdown is not joined, while a settled rejection can be retried strictly; success clears both the teardown debt and abandoned marker, and another rejection remains retryable. The regression covers the reject-then-succeed path, and the earlier inbox-custody, repeated-resume, late-generation fencing, and manager-shutdown concerns remain addressed.

Static diff review only; I did not rerun tests or QA.

[baixiaohang]

Addressed the late-materialization Reset race in 60cdde8d1.

session:terminate now checks for an abandoned route producer before starting any abandoned-handler shutdown retry. While that producer remains tracked, Reset fails closed immediately. When it eventually completes, stale-route handling registers and starts the new after-prior teardown debt before the producer settle callback removes it from the tracked set. Only a later Reset can retry teardown, so a successful earlier attempt can no longer delete debt that materializes while it is awaiting.

The post-quiesce abandoned-producer check remains as a defensive boundary for the rest of terminate.

The new regression uses a real dispatch() resume producer and covers the exact sequence:

1. suspend times out and the first best-effort shutdown rejects;
2. Reset is attempted while the producer remains gated, and no retry shutdown starts;
3. the producer completes late and materializes an after-prior shutdown that remains pending;
4. a later Reset still rejects and preserves the teardown debt.

Validation:

• focused edge suite: 147 passed
• full @first-tree/client suite: 2415 passed, 7 skipped
• pnpm check && pnpm typecheck: passed (repository baseline warnings only)

Please take another look.

[yuezengwu]

Requesting changes for one blocking late-materialization path on 60cdde8d18d1772c36ccae4e8d332b36b38b92e0.

abandonTimedOutSuspendHandler() and detachHandlerWithPendingTeardown() clear abandonedSuspendHandlers as soon as a best-effort shutdown succeeds (lines 2350-2354 and 2369-2373), even if the handler still has an abandoned route producer. That success is not final stop proof: discardStaleRouteTransition() explicitly documents that a pre-materialization shutdown may be a no-op, and starts a new after-prior shutdown when the producer returns (lines 2413-2427).

If the first shutdown succeeds while the producer is still pending, then the producer materializes late and the new after-prior shutdown never settles, its debt no longer has the abandoned marker. settleTeardownDebtBeforeRoute() therefore joins it as ordinary route-blocking debt, and manager shutdown joins it too. The chat is permanently fenced again even though the late generation remains safely retired. The new early Reset check does not cover this: once the producer settles it disappears from routeProducers, leaving only the now-ordinary pending shutdown.

Please retain/propagate the abandoned status until every abandoned producer for that handler has settled and its post-materialization stop is confirmed. Add a regression where the first pre-materialization shutdown resolves, the producer returns late, and the second after-prior shutdown never settles; prove ordinary later routing and manager shutdown remain available while Reset still fails closed.

The newly added test covers only an initial shutdown rejection, which keeps the marker present and misses this branch. Static diff review only; Client & Web CI passed, while the current Server job failed in two unchanged Reset tests.

[first-tree-hub-staging]

Reviewed head 60cdde8d18d1772c36ccae4e8d332b36b38b92e0 and requested changes for one remaining blocker.

A successful pre-materialization shutdown currently clears the abandoned-handler marker before its abandoned producer settles. If that producer later materializes and the required after-prior shutdown never settles, the debt becomes ordinary again, so both later route admission and manager shutdown join it forever. The fix should retain or propagate abandonment until producer settlement plus post-materialization stop is confirmed, with a regression for “first shutdown succeeds, late producer returns, second shutdown never settles.”

Formal review: #2205 (review)

Static diff review only. Client & Web CI passed; the current Server job failed in two unchanged Reset tests.

[baixiaohang]

Superseded by #2212, which restarts from current main with the narrower #2144 contract: chat-level quarantine, inbox recovery before a fresh handler, restart-required Reset, and no same-daemon teardown retry/convergence path.