feat: orchestrate on-demand runtime residency

[schickling-assistant]

Problem

The residency model and provider resume capabilities need one host-owned policy that decides when to go cold, completes safe teardown, and wakes without weakening message delivery semantics.

Goal

Orchestrate on-demand residency with explicit host policy, durable demand, complete-group teardown, exact launch-attempt fencing, and user-facing wake commands.

Decisions

• Configure one host idle threshold and warm capacity; no policy means on-demand launches are suppressed.
• Wake only from durable unread inbox work, st2 wake, or st2 pty attach <agent>.
• Once checkpointing starts, finish stop and verify complete absence before servicing racing demand.
• Mint an independent host launch-attempt incarnation. Rotate it when the canonical provider is absent; retain it while that provider is live and sidecars converge.
• Require Codex, OMP, and Claude readiness to match both generation and the exact host attempt incarnation.
• Treat the dedicated residency wake watcher as required startup infrastructure. Best-effort Resource watchers retain their timer/refresh fallback.
• Emit timestamped transition events, but define no latency SLO before a live canary is measured.

Verification

• CI=1 nix build --builders '' --max-jobs 8 --cores 8 .#st2 --no-link --print-build-logs — full native Darwin package and test suite passed; the library reported 799 passed and 1 ignored, followed by all selected integration suites.
• axe vrs check --profile strict docs/vrs — passed.
• Residency state, host orchestration, Codex, OMP, Claude, task inventory, watcher error propagation, and VRS ledger tests pass in the package suite.
• The portable watcher test proves recursive registration failures are reported instead of silently degrading immediate wake.
• Final independent review found no correctness blocker after the watcher fix.

Complexity

• Desired lifecycle and runtime residency remain independent instead of creating a second lifecycle model.
• Provider-specific native resume protocols stay separate because their evidence differs, while shared launch orchestration owns generation and incarnation fencing.
• The integration removes provider convenience wrappers, reuses shared fsatomic::replace, and keeps one dedicated residency wake channel.
• Required residency wake failures abort startup. Existing best-effort Resource observers explicitly discard watcher construction errors because they retain independent fallback paths.

Concerns

Draft PR #519 overlaps provider session and specification files and will need a semantic rebase after this stack lands.

Friction & bottlenecks

cargo fmt --check reports broad pre-existing formatting drift, so this stack avoids a repo-wide formatting rewrite. This repository has no devenv quick-check task.

Follow-ups

• Add one downstream host canary configuration and deploy only with explicit operator approval.
• Measure cold-to-ready latency and resident resource reduction before setting an SLO.

References

Depends on PR #493. Part of stack #495. Related test deflake: PR #318.

Posted on behalf of @schickling

field	value
agent_identity	mbp2025.direct.omp.bn5hrz39
session	mbp2025.bn5hrz39
agent_persona	generalist
agent_supervisor	unavailable
agent_tool	OMP
agent_tool_version	18.1.7
agent_runtime	OMP 18.1.7
tooling_profile	dotfiles@4816a7a