fix(client): quarantine timed-out operator suspend

[baixiaohang]

Summary

• bound manual operator suspend settlement to 30 seconds, then permanently quarantine the exact handler/generation whose callback was lost
• run prepareOperatorSuspend() before releasing the boundary so only the provider-entered prefix is ACKed and the unentered tail becomes recovery debt before a fresh handler route
• keep ordinary messages recoverable in the same daemon without waiting for or retrying the quarantined handler shutdown; late output stays behind the existing handler/generation fence
• return a concrete restart-required Reset failure (applied:false) while quarantine exists, and keep provider admission fail closed when the timeout caught an unfinished start/resume transition
• emit a structured timeout event/log with provider, chat, generation, and whether a route transition was in flight

Scope

This intentionally does not provide same-daemon Reset convergence after provider teardown becomes unobservable. Restarting the agent daemon clears the in-memory quarantine and lets bind-reset recover unACKed inbox work before Reset is retried.

This replacement supersedes #2205 and keeps the two required parts of its review: the inbox custody boundary and never joining the lost provider callback again. The reject-then-succeed Reset retry and late-materialization Reset race paths are removed from this scope.

Testing

• pnpm exec vitest run src/__tests__/session-manager-edge-coverage.test.ts --reporter=dot --testTimeout=10000 --hookTimeout=10000 (147 passed)
• pnpm --filter @first-tree/client test (2421 passed, 7 skipped)
• pnpm --filter @first-tree/client typecheck
• pnpm check && pnpm typecheck (passed; repository-baseline warnings only)

Closes #2144

[yuezengwu]

Approved.

This change fixes the post-sleep/manual-suspend deadlock by bounding operator suspend settlement at 30 seconds and permanently quarantining the exact timed-out handler generation. The timeout path establishes the inbox custody boundary before releasing suspension, so only the provider-entered prefix is ACKed and the unentered tail remains recovery debt. Ordinary routing can then install a fresh handler without joining the lost callback, while an in-flight start/resume timeout keeps provider admission closed and Reset consistently fails with a restart-required result.

The narrower permanent-quarantine model also removes the late-materialization convergence race from #2205: a late completion matching the quarantined handler/generation is ignored instead of becoming ordinary teardown debt that later route admission or manager shutdown could join.

Static diff review only; I did not rerun tests or QA.

[baixiaohang]

Blocking review finding on 2f26181c3f36528fd8e0cb5c11aa2cb262ee6368 (please address before merge).

[P1 / R4 — packages/client/src/runtime/session-manager.ts:4284] Preserve terminal-notice debt on the timeout path before ACKing the processing prefix.

markNoticeRequiredForProcessingPrefix() currently runs only after waitForHandlerSuspend() resolves. In the exact new timeout scenario, the handler may emit a structured terminal/retry-exhausted provider failure during its drain and then never settle. SessionContext.emitEvent() records that payload in entry.pendingRuntimeFailureNotice while the settlement lease is still valid, but the timeout rejection skips lines 4284–4285. Execution then reaches prepareOperatorSuspend() at line 4333 with no requiresDurableNotice marker, so the coordinator promotes and ACKs the provider-entered prefix. That permanently consumes the inbox row without the durable chat notice required by the existing provider-failure contract; the fresh handler/recovery path can no longer reconstruct the lost evidence.

Please transfer pendingRuntimeFailureNotice to the processing prefix on both successful settlement and timeout before invalidating the generation / calling prepareOperatorSuspend() (for example, move the existing marking into their shared continuation). Add a regression where suspend() emits a terminal provider-failure event and then never resolves: after 30 seconds the entry must remain unACKed, and recovery/redelivery must post the notice before its only ACK.

The quarantine shape itself is substantially narrower than #2205 and the normal inbox-recovery, late-generation fence, restart-required Reset, and bounded manager-shutdown paths otherwise look consistent. Static diff review only; I did not rerun tests or QA. CI is currently green.

[yuezengwu]

Requesting changes; this supersedes my earlier approval after confirming the newly identified P1 on the same head.

The quarantine design and the normal custody/recovery paths remain sound, but the timeout branch can consume a terminal provider failure without its required durable chat notice. pendingRuntimeFailureNotice is transferred to the inbox ledger only after waitForHandlerSuspend() resolves. If suspend() emits the structured terminal/retry-exhausted event and then never settles, the 30-second timeout skips that transfer and continues to prepareOperatorSuspend(), which can promote and ACK the processing prefix. The inbox row is then permanently consumed even though the session event is only a live trace and the durable notice was never posted.

Please move the pending-notice transfer into the continuation shared by successful settlement and timeout, before invalidating the generation and before prepareOperatorSuspend(). Add the proposed regression proving that a terminal event emitted by a never-settling suspend() keeps the row unACKed and that recovery/redelivery posts the notice before its only ACK.

Static diff review only; I did not rerun tests or QA.

[baixiaohang]

Addressed the P1 notice-debt finding in 81b3e61.

• The pending terminal provider-failure notice is now transferred to the inbox ledger on both successful suspend settlement and operator-suspend timeout, before generation invalidation and prepareOperatorSuspend().
• Added a regression where suspend() emits provider_failure_terminal and then never resolves. After 30 seconds the processing entry remains unACKed; recovery/redelivery posts the durable notice before its single ACK and never re-enters the old provider.

Validation: focused regression passed; all 148 session-manager edge tests passed; full client suite passed (2422 passed, 7 skipped); repo typecheck passed; touched files pass Biome. Please re-review when convenient.

[yuezengwu]

Approved on 81b3e616c9bc6fd1276e7e51182bf9f9a029a74f.

The goal remains to bound a lost manual-suspend callback without losing inbox custody or letting the quarantined handler/generation block ordinary routing and manager shutdown. The new commit resolves the outstanding P1: terminal provider-failure notice debt is now transferred on both successful settlement and timeout, before generation invalidation and before prepareOperatorSuspend() can promote the processing prefix.

The added regression covers the exact failure sequence: suspend() emits provider_failure_terminal and never resolves; the entry remains unACKed with recovery debt after timeout; redelivery persists the durable notice before the single ACK; and the old handler is not re-entered. I found no remaining blocker in the incremental diff.

This PR still introduces the core in-memory quarantinedSessions lifecycle state and has no database migration. Static diff review only; I did not rerun tests or QA.

[baixiaohang]

Re-reviewed 81b3e616c9bc6fd1276e7e51182bf9f9a029a74f; no remaining blocker found.

The P1 is resolved at the correct boundary: pendingRuntimeFailureNotice is transferred to the inbox processing prefix for both successful settlement and timeout, before the route generation is invalidated and before prepareOperatorSuspend() can promote any processing entry to ACK-eligible terminal work.

The regression covers the missing branch end to end: suspend() emits provider_failure_terminal and never settles; timeout leaves the processing entry unACKed with recovery debt; redelivery persists the durable notice before the single ACK; and the old provider handler is not re-entered. The incremental production change is limited to moving the existing transfer into that shared continuation.

Static incremental-diff review only; I did not rerun tests or QA.