diff --git a/rfcs/0013-cloud-serializable-sqlite-state.md b/rfcs/0013-cloud-serializable-sqlite-state.md index efbd97ec..c4efca53 100644 --- a/rfcs/0013-cloud-serializable-sqlite-state.md +++ b/rfcs/0013-cloud-serializable-sqlite-state.md @@ -3,7 +3,7 @@ title: SQLite Snapshot Backup Artifacts authors: - giodl created: 2026-06-18 -last_updated: 2026-07-21 +last_updated: 2026-08-13 status: completed issue: https://github.com/openclaw/openclaw/pull/105718 rfc_pr: https://github.com/openclaw/rfcs/pull/20 @@ -55,6 +55,7 @@ The first landed implementation uses SQLite `VACUUM INTO` to capture committed W - This RFC does not require OpenClaw to own upload, tenant routing, retention, or encryption policy. - This RFC does not require hot writes over a network filesystem. - This RFC does not define WAL bundles, leases, promotion, fencing, or standby orchestration. +- This RFC does not define another Gateway pause, drain, or suspension API. - This RFC does not change `openclaw backup create` archive behavior. ## Proposal @@ -172,6 +173,437 @@ flowchart LR The diagram is a responsibility split. The default local runtime can ignore the host box entirely. Hosted deployments can use the snapshot directory as the sync boundary without copying live SQLite sidecars. +### Optional Follow-On Composition + +RFC 0013 is the completed one-database artifact contract. It is also the +owner-authored substrate for optional recovery workflows, but those workflows +must compose the landed command rather than reinterpret live SQLite files or +duplicate snapshot creation, verification, repository, or restore behavior. + +For scale-to-zero, the goal is simple: let an idle OpenClaw Gateway stop using +compute, then wake a replacement when work arrives without losing state or +sending work to it before it is ready. + +Most of the OpenClaw-side foundation already exists. Vincent's snapshot work +provides durable state capture and restore. Peter's Gateway suspension and Cron +work provides a cooperative way to stop and reconcile tracked work. The +remaining need is a small handoff between OpenClaw and its host: + +1. the host remembers that work is waiting and wakes the replacement; +2. OpenClaw restores the accepted state and reports when the Gateway is ready; +3. the host waits for that readiness before sending the queued work. + +This follow-on does not add a host scheduler, Teams transport, compute +placement service, or retained-payload store to OpenClaw. It defines the +application-consistency facts a host needs to use its existing infrastructure +safely. The detailed contracts below preserve generation fencing, replay, and +failure handling for implementations. + +The scale-to-zero outcome and user evidence are tracked in +[openclaw/openclaw#114145](https://github.com/openclaw/openclaw/issues/114145). +This follow-on deliberately builds on, rather than replaces, work Vincent and +Peter already added: Vincent's verified SQLite snapshots in +[openclaw/openclaw#105718](https://github.com/openclaw/openclaw/pull/105718), +Peter's cooperative host suspension in +[openclaw/openclaw#103618](https://github.com/openclaw/openclaw/pull/103618), +and Vincent's suspension validation repair in +[openclaw/openclaw#103925](https://github.com/openclaw/openclaw/pull/103925). +It also composes with Josh Lehman's SQLite-backed session and transcript runtime +in [openclaw/openclaw#98236](https://github.com/openclaw/openclaw/pull/98236) +and restart-recovery admission behavior in +[openclaw/openclaw#111869](https://github.com/openclaw/openclaw/pull/111869). + +The three implementation PRs linked below demonstrate one bounded composition; +they are not a required decomposition. If maintainers prefer a smaller or +different core seam that satisfies the same recovery and restored-admission +outcomes, that is a valid resolution of the umbrella issue. + +#### What snapshots and suspension do not finish + +Per-user and event-driven hosts can stop paying for resident compute only when +they can retire one Gateway generation and later admit work on a replacement +without losing accepted state. A SQLite snapshot alone cannot prove that +transaction. It does not identify the complete set of state owners, prove that +the host durably accepted every required byte, authorize retirement of the +source generation, or prove that the replacement restored the same recovery +point before accepting work. + +Without those semantics, hosts must choose between keeping idle Gateways warm +or maintaining private shutdown, copying, restore-ordering, and readiness +inference paths. Managed-host experience has exposed the resulting failure +classes: accepted ingress can outlive the compute that should process it, +scheduled wake can race idle retirement, replacement can lose scheduler +continuity, and a cold runtime can appear ready before its required owner state +is restored. + +Existing reports ground those outcomes in OpenClaw operator needs: + +- [openclaw/openclaw#13616](https://github.com/openclaw/openclaw/issues/13616) + requests unified backup and restore for config, cron, and session state. +- [openclaw/openclaw#63392](https://github.com/openclaw/openclaw/issues/63392) + requests per-agent backup and restore rather than whole-instance rollback. +- [openclaw/openclaw#104412](https://github.com/openclaw/openclaw/issues/104412) + shows recurring cron work can be skipped silently while Gateway is stopped. +- [openclaw/openclaw#101290](https://github.com/openclaw/openclaw/issues/101290) + shows why hosts need owner-safe snapshot boundaries instead of touching live + SQLite state from another process. +- [openclaw/openclaw#107433](https://github.com/openclaw/openclaw/issues/107433) + requests explicit protected and reconstructable-state obligations. + +[openclaw/openclaw#113306](https://github.com/openclaw/openclaw/issues/113306) +tracks adjacent crash-durability and identity hardening in the underlying +SQLite snapshot implementation; the follow-on lifecycle does not claim to fix +that separate issue. + +Infrastructure platforms already provide the compute lifecycle primitives: + +- [Fly Machines](https://fly.io/docs/launch/autostop-autostart/) can stop all + Machines and autostart them for traffic. +- [E2B](https://e2b.dev/docs/sandbox/auto-resume) pauses and automatically + resumes persistent sandboxes. +- [Daytona](https://www.daytona.io/docs/en/persistence/) preserves sandbox + files across stop/start and offers archive or VM pause/resume paths. +- [Azure Container Apps](https://learn.microsoft.com/azure/container-apps/scale-app) + scales to zero and wakes from configured event sources. +- [Cloudflare Durable Objects](https://developers.cloudflare.com/durable-objects/best-practices/websockets/) + hibernate while retaining wakeable WebSocket delivery. +- [Modal](https://modal.com/docs/guide/scale) scales Functions to zero by + default and offers + [Sandbox snapshots](https://modal.com/docs/guide/sandbox-snapshots). + +Those platforms do not know OpenClaw's owner inventory, SQLite invariants, +scheduler state, or readiness boundary. The follow-on contracts define that +application-consistent layer. A host remains responsible for retained ingress, +wake registration, compute placement, and external durability; OpenClaw and +its state owners provide the exact recovery point and restored-admission proof +that make those host primitives safe to use. + +The implementer-facing follow-on contracts are deliberately split by owner: + +- [Recovery Point Components v1](0013/recovery-point-components-v1-spec.md): + compose verified SQLite snapshots with explicit non-SQLite owner artifacts + and external or reconstruction obligations. +- [Portable Handoff v1](0013/portable-handoff-v1-spec.md): combine the existing + cooperative Gateway suspension fence with final owner capture, durable host + acceptance, and generation-bound source-compute retirement authority. +- [Restored Admission v1](0013/restored-admission-v1-spec.md): restore exact + accepted components into fresh paths and keep admission closed until + scheduler and required owner readiness complete. + +The split is mechanical rather than architectural: + +| Contract | Owner-side input | Durable output | Stops before | +| --- | --- | --- | --- | +| Recovery Point Components | Verified owner artifacts plus the closed selected-owner inventory | `recoveryPointId` and `acceptanceSetId` | Suspension, storage, or wake | +| Portable Handoff | Suspension-ready source generation plus final owner capture | Host acceptance receipt and generation-scoped `sourceComputeRetirementAuthorized` | Restore or destination admission | +| Restored Admission | Exact accepted recovery point plus destination generation | Restore receipt and `gateway.restore.status = ready` | Retained-work delivery | + +The IDs join the contracts; no sidecar owns another sidecar's work. + +These sidecars do not change `openclaw backup sqlite`. They do not make every +ordinary snapshot a portable recovery point, add a continuity-specific storage +provider, or make Lobster part of the core contract. + +OpenClaw implementation evidence is available for the three owner-side slices: + +- [openclaw/openclaw#112385](https://github.com/openclaw/openclaw/pull/112385) + composes verified global and owner-selected per-agent RFC 0013 snapshots into + one deterministic `host-protected` recovery point and exact acceptance byte + inventory. +- [openclaw/openclaw#112865](https://github.com/openclaw/openclaw/pull/112865) + is stacked on #112385. It adds one hidden offline final + capture operation with operation-scoped SQLite intent, exact + committed-result replay, and fail-closed quarantine for conflicting or + incomplete attempts. +- [openclaw/openclaw#112896](https://github.com/openclaw/openclaw/pull/112896) + is stacked on #112865. It restores one exact accepted aggregate to + fresh canonical paths, holds when required owner evidence is absent, and + keeps Gateway work admission closed through scheduler reconciliation and + owner readiness. Restore intent, results, startup descriptors, and ready + evidence share the dedicated SQLite recovery journal rather than JSON + runtime sidecars. It also implements the proposed read-only + `gateway.restore.status` projection as review evidence. The public Gateway + Protocol surface remains an explicit owner decision before merge or ship. + +These PRs are evidence for owner review, not normative dependencies. They do +not move Gateway suspension, external ingress fencing, clean process shutdown, +durable host acceptance, publication, host wake, or source-compute retirement +into +OpenClaw. + +#### What already exists and what remains + +OpenClaw has added useful owner-level foundations since this follow-on was +first drafted: + +- [openclaw/openclaw#118393](https://github.com/openclaw/openclaw/pull/118393) + now binds Cron cancellation settlement to each active run and keeps + unresolved work visible to Gateway suspension through bounded shutdown. + This strengthens the source-work fence; it does not register or deliver a + wake after compute reaches zero. +- [openclaw/openclaw#117705](https://github.com/openclaw/openclaw/pull/117705) + keeps Gateway-backed agent turns on a cold CLI path. This reduces replacement + startup overhead; it does not provide retained ingress, compute wake, or + restored-readiness gating. +- Vincent's snapshot durability campaign in + [openclaw/openclaw#113306](https://github.com/openclaw/openclaw/issues/113306) + has landed durable parent publication and pending-snapshot recovery. The + generic helper's staging-cleanup ownership contract remains an explicit + maintainer decision. +- Vincent's open + [openclaw/openclaw#117258](https://github.com/openclaw/openclaw/pull/117258) + isolates post-commit auth snapshot publication per runtime owner. It is + relevant evidence for owner-local reconciliation, but it is not a portable + recovery receipt or a restored-admission signal. + +The remaining gaps are narrower and sit at the host/OpenClaw boundary: + +1. agree who owns the selected-agent inventory, recovery journal, and public + `gateway.restore.status` method; +2. let the host find the accepted recovery point and choose one fenced + replacement generation; +3. remember and combine Teams, API, and Cron reasons to wake while compute is + absent; and +4. send each queued item only after OpenClaw reports exact restored readiness, + with durable acknowledgement and crash replay. + +There is also an RFC-ownership decision before merge: repository review has +asked whether this lifecycle belongs in a standalone draft RFC rather than as +an optional follow-on to completed RFC 0013. Until maintainers settle that +placement, RFC 0013 remains authoritative only for the completed SQLite +artifact contract and these lifecycle sections remain review material. + +The intended observable outcome of the complete host composition is: + +- idle compute may reach zero without treating a raw live filesystem copy as a + recovery point; +- accepted ingress remains retained until restored admission succeeds; +- autonomous scheduled work does not require an unrelated user message to + recover from absent compute; +- source compute is not retired before exact durable acceptance; and +- replacement readiness names the accepted recovery point it restored. + +The three OpenClaw PRs prove only the owner-side recovery-point, final +capture, and restored-admission slices. Host acceptance, retained ingress, +wake, and source-compute retirement remain separate review and implementation +work. + +#### Keep the host/OpenClaw handoff small + +The recommended V1 surface is one bidirectional protocol contract without an +OpenClaw-owned lifecycle coordinator or abstract `Host` base class: + +- current-main `gateway.suspend.prepare|status|resume` supplies the live-source + fence; +- the final-capture and accepted-restore operations supply the offline owner + work; and +- a new read-only `gateway.restore.status` supplies the exact live-destination + admission fact. + +`gateway.restore.status` follows the design Peter established for +`gateway.suspend.status`: a closed TypeBox request and result, one stable +operation identity, `operator.read` scope, no mutation, and typed conflict +behavior. The request names the expected `restoreOperationId`. An ordinary +Gateway returns `not-restored`; the matching restored Gateway returns `held` +or `ready`; a different active restore operation fails rather than returning a +success-shaped result. The `ready` result names the destination generation, +accepted recovery point and byte set, restore receipt, scheduler and owner +readiness evidence, admission identity, and final readiness identity. + +The method projects the durable restored-admission record already required by +this RFC. It does not add another journal or state machine. Offline restore and +preflight failures remain offline failures; they are not represented as a +live `restoring` phase. A quarantined start does not expose a live Gateway and +therefore is not a success result. The detailed request, result, availability, +and error contract is normative in +[Restored Admission v1](0013/restored-admission-v1-spec.md#gateway-restore-status). + +There is deliberately no `gateway.restore.admit` method. OpenClaw computes +readiness, durably records it, and opens its own admission. The host may only +observe the result and compare it with the operation and generation it owns. +Normal user-work methods remain unavailable while restored admission is held; +the status method is allowed only through an authenticated pre-admission +control path, following the existing suspension-control pattern. That path +must admit only a bounded, non-enrolling control handshake, track it as active +root work until authentication and connection completion, and lose to any +restart fence. It must not admit nodes, device pairing or enrollment, remote +clients, presence-bearing sessions, or any RPC other than the read-only status +query while work admission is held. + +#### RFC 0018 readiness composition + +When the opt-in canonical readiness facility from +[RFC 0018](https://github.com/openclaw/rfcs/pull/33) is available, restored +admission composes with it rather than creating a second evaluator. A restored +start publishes one required `RecoveryPointRestored` condition whose primary +subject is the stable `openclaw/gateway` role. Related stable subject references +identify the restore operation, accepted recovery point, destination runtime +generation, scheduler reconciliation, and required state-owner roles; their +current IDs and generations live in RFC 0018's identity package rather than in +the references. The condition remains `Unknown` or `False` while the restore +hold is active and becomes `True` only from the same durable record that opens +admission. + +`gateway.restore.status` remains the stronger operation-fenced query: its +caller supplies the expected restore operation and receives typed conflict +behavior. `/readyz`, `openclaw ready`, Status, and Gateway readiness RPC expose +the ordinary RFC 0018 projection for the active runtime. Both surfaces must +project the same readiness generation and subject identities. Neither may +infer restore completion from process health, database-open success, or a +container probe. This composition is contingent on RFC 0018 acceptance and +activation; `gateway.restore.status` remains independently useful and does not +depend on RFC 0018. + +One host-owned coordinator may compose those contracts as +`prepareHibernate`, `ensureRuntimeReady`, and `inspectLifecycle`. Those are +host operations, not new OpenClaw Gateway methods. The resulting evidence +chain is: + +```text +GatewaySuspensionReady + -> SourceWritersClosed + -> RecoveryPointCaptured + -> RecoveryPointAccepted + -> generation-scoped SourceComputeRetirementAuthorized + -> retained wake cause + -> RecoveryPointRestored + -> AdmissionReady + -> owner delivery and acknowledgement +``` + +These are distinct facts, not aliases for one global `synced` state: + +- `GatewaySuspensionReady` proves OpenClaw's cooperative tracked-work fence for + one source generation. +- `SourceWritersClosed` proves the host supervisor closed the Gateway process + and every other authoritative writer it owns for that generation. +- `RecoveryPointCaptured` proves immutable owner artifacts and one closed + recovery-point identity. +- `RecoveryPointAccepted` proves the host durability boundary accepted the + exact logical byte set. +- `SourceComputeRetirementAuthorized` permits removal of only the named source + compute generation. It never authorizes persistent-state deletion. +- `RecoveryPointRestored` proves exact fresh-target restore and owner + reconciliation for one destination generation. +- `AdmissionReady` proves that same destination may accept ordinary work. + +Each authority contributes only its own fact. +`SourceComputeRetirementAuthorized` is derived from durable acceptance rather +than supplied by a caller. Wake callers cannot +select a recovery point, destination generation, or readiness result. Teams, +cron, and API owners retain their own payload, retry, deduplication, and +acknowledgement state. Unknown outcomes retain work and hold or quarantine; +timeouts do not imply success. + +The diagram shows the success path, not an irreversible linear workflow. A new +retained-work cause may race any pre-retirement step. The host must atomically +revoke or defeat `SourceComputeRetirementAuthorized` while source compute still +exists; after retirement, the same cause joins the single destination wake and +restore operation. + +Planned and forced transitions deliberately have different guarantees: + +| Transition | New recovery point | Source retirement | Recovery guarantee | +| --- | --- | --- | --- | +| Planned handoff | Requires Gateway suspension, closed source writers, final capture, and exact durable acceptance | Only after generation-scoped authorization | The newly accepted point | +| Forced source loss | None may be inferred from process absence or partial files | Host policy handles already-lost compute; no clean-retirement claim is created | The last previously accepted point and its declared RPO | + +This contract simplifies the evidence implementation: reuse the existing +suspension methods, scheduler reconciliation hooks, recovery operations, +admission fence, and readiness journal; keep one host lifecycle state machine; +and avoid a second pause API, generic host callback registry, duplicate +Gateway admission state, or central retained-payload store. The Gateway +Protocol package should own the new wire schemas beside `gateway-suspend`; a +separate lifecycle SDK package is not required for V1. + +#### Wake and deliver work after compute reaches zero + +The current three-PR evidence stack intentionally stops at restored admission. +A complete scale-to-zero host also needs to wake without relying on an +unrelated user request and to retain accepted work while no Gateway process +exists. + +OpenClaw already exposes the owner mechanism Peter added for this job: +post-commit `cron_changed: scheduled` signals in +[openclaw/openclaw#103647](https://github.com/openclaw/openclaw/pull/103647) +and lifecycle-owned `cron_reconciled` snapshots in +[openclaw/openclaw#104368](https://github.com/openclaw/openclaw/pull/104368). +The documented safe external projection pattern uses `cron_reconciled` to adopt +the exact scheduler, treats `cron_changed` only as a coalescible reread hint, +and completes its `replaceAll` callback only after the host durably accepts the +projection. Those callbacks remain tracked root work, so the existing +`gateway.suspend.*` fence cannot report idle while a projection write is still +in flight. + +The host should bind the durably accepted projection revision, accepted final +recovery point, and revocable sleep authority in one transaction. That host +wake registration contains a nullable `nextRequiredAt` and a bounded reason +class, not cron expressions, job names, prompts, or payloads. The host may +subtract a configured cold-start lead from that timestamp, but it must not +decide whether a cron job is due, invent catch-up work, or suppress duplicates. +No continuous callback beyond the shipped reconciliation hooks is required. + +A retained-ingress owner such as Teams must durably store its payload or opaque +reference and deduplication identity before acknowledging upstream delivery. +That accepted cause revokes an in-progress sleep authorization or joins an +idempotent wake request. Teams messages, API work, semantic deadlines, and +operator requests may coalesce one compute-provisioning attempt, but each owner +retains its independent delivery, retry, and acknowledgement state. + +The host then selects an accepted recovery point, grants one destination +generation, restores through the existing contract, and waits for the exact +restored-admission readiness identity. Only then may retained-work owners use +their existing generation-fenced delivery paths. After wake, OpenClaw remains +authoritative for due and missed-run reconciliation, catch-up policy, duplicate +suppression, and the next semantic deadline; a host alarm is only a provisioning +trigger and must not invoke cron jobs directly. + +Wake callers cannot select a recovery point or destination generation and +cannot supply readiness. Central lifecycle records contain bounded cause and +generation identities only; retained payloads and credentials remain with +their existing owners. + +Unknown provisioning, restore, readiness, or delivery outcomes retain the wake +causes and hold or quarantine. A timeout is not permission to acknowledge work, +start a second authoritative generation, or open admission. + +The bounded host follow-on has three independently reviewable responsibilities: + +1. **Project scheduler wake metadata:** consume `cron_reconciled`, use + `cron_changed` only as a reread hint, and durably bind the complete accepted + projection to the final recovery point and sleep authority. Fork-only + evidence is in Lobster PRs + [#38](https://microsoft.ghe.com/giodl/lobster/pull/38) through + [#45](https://microsoft.ghe.com/giodl/lobster/pull/45) (Microsoft access + required). +2. **Retain and coalesce wake causes:** keep Teams/API payloads with their + owners, revoke sleep atomically, and grant one destination generation for + concurrent causes. Fork-only evidence is in Lobster PRs + [#46](https://microsoft.ghe.com/giodl/lobster/pull/46) and + [#47](https://microsoft.ghe.com/giodl/lobster/pull/47). +3. **Deliver only after exact readiness:** match `gateway.restore.status` + against the host-owned restore operation and destination generation before + invoking each owner's existing delivery path. Fork-only replay evidence is + in Lobster PR [#51](https://microsoft.ghe.com/giodl/lobster/pull/51). + +These are review and evidence slices, not a required repository decomposition. +The host slices may be combined if the same ownership, race, replay, and +conformance boundaries remain independently reviewable. OpenClaw does not gain +a Teams transport, compute scheduler, retained-payload store, or placement API. +If end-to-end proof exposes an invariant the shipped hooks and suspension fence +cannot express, the follow-up should be the smallest extension of those owner +seams rather than a parallel continuity scheduler or wake API. + +OpenClaw `main` already provides the host-neutral +`gateway.suspend.prepare|status|resume` contract from +[openclaw/openclaw#103618](https://github.com/openclaw/openclaw/pull/103618), +with the validation and import-boundary repair from +[openclaw/openclaw#103925](https://github.com/openclaw/openclaw/pull/103925). +A follow-on handoff must reuse that cooperative tracked-work fence. It must not +introduce another Gateway pause API. The existing contract intentionally leaves +external ingress, third-party Channel transports, unregistered background work, +and full process/filesystem consistency to the host. + ### Snapshot Semantics The unit of snapshotting is one existing OpenClaw-owned SQLite database: @@ -267,6 +699,16 @@ The original contributor prototype was [openclaw/openclaw#94805](https://github. The stress harness remains tracked separately in [openclaw/openclaw#94967](https://github.com/openclaw/openclaw/pull/94967). Broader state ownership and continuity work remains related to [openclaw/openclaw#101290](https://github.com/openclaw/openclaw/issues/101290). +Adjacent shipped lifecycle foundations are: + +- [openclaw/openclaw#103618](https://github.com/openclaw/openclaw/pull/103618), + which added cooperative host suspension; and +- [openclaw/openclaw#103925](https://github.com/openclaw/openclaw/pull/103925), + which restored its architecture and validation gates. + +The optional follow-on sidecars consume those contracts as implemented on +current `main`. + ## Rationale This approach solves the reliability problem at the correct boundary. SQLite remains local and authoritative while OpenClaw is running. Durability is handled by verified artifacts, manifests, and explicit restore procedures. @@ -288,3 +730,6 @@ Deferring WAL bundles is intentional. Full snapshots provide the first correct r - restore-on-boot host integration - leases, promotion, fencing, and managed failover - dedicated snapshot targets for future owner stores + +The three RFC 0013 sidecars narrow the first portable follow-on without +promoting those future items into this completed SQLite contract. diff --git a/rfcs/0013/portable-handoff-v1-spec.md b/rfcs/0013/portable-handoff-v1-spec.md new file mode 100644 index 00000000..c96e50ee --- /dev/null +++ b/rfcs/0013/portable-handoff-v1-spec.md @@ -0,0 +1,219 @@ +# Portable Handoff v1 Specification + +This document is the implementer-facing planned-handoff specification for RFC +0013 follow-on recovery points. It composes the shipped cooperative Gateway +suspension contract with final owner capture and durable host acceptance. + +Status: draft, tied to RFC 0013. + +## Implementation Evidence + +[openclaw/openclaw#112865](https://github.com/openclaw/openclaw/pull/112865) +is the OpenClaw evidence slice for final owner capture, stacked on +[openclaw/openclaw#112385](https://github.com/openclaw/openclaw/pull/112385). +It uses the existing RFC 0013 SQLite provider, binds the exact runtime lineage, +source generation, handoff identity, selected-agent inventory, and +host-supplied closure evidence, and durably replays one committed recovery +point after response loss. + +The operation is deliberately offline and host-invoked. It does not suspend or +stop Gateway, fence external ingress, accept or publish bytes, or authorize +source-compute retirement. The pull request is implementation evidence; this +specification remains the normative responsibility boundary. + +## Scope + +This specification defines: + +- host ingress fencing before planned handoff; +- reuse of `gateway.suspend.prepare|status|resume`; +- final capture after the tracked Gateway work fence is ready; +- immutable host acceptance of one aggregate recovery point; +- exact replay after transport or coordinator uncertainty; +- generation-bound source-compute retirement authority; +- hold and quarantine behavior. + +This specification does not define: + +- another Gateway pause, drain, or suspension API; +- storage-provider plugin registration; +- object-store, filesystem, sidecar, or queue implementation; +- retention, placement, encryption-key, or idle policy; +- forced-termination recovery beyond the last accepted recovery point; +- restored admission or Elastic wake. + +## Shipped Suspension Contract + +Current OpenClaw `main` provides: + +```text +gateway.suspend.prepare +gateway.suspend.status +gateway.suspend.resume +``` + +The contract was introduced by +[openclaw/openclaw#103618](https://github.com/openclaw/openclaw/pull/103618) +and its architecture and validation gates were restored by +[openclaw/openclaw#103925](https://github.com/openclaw/openclaw/pull/103925). + +`prepare`: + +- accepts a stable host operation ID; +- closes new tracked root, session, and command admission; +- pauses automatic cron scheduling; +- synchronously inspects tracked active work; +- returns `busy` and reopens normal operation when work remains; or +- returns a renewable two-minute suspension lease when ready. + +The lease is cooperative and refuse-only. It does not: + +- persist external ingress; +- stop third-party Channel transports; +- prove unregistered plugin or background work idle; +- stop the process tree; +- snapshot the filesystem; or +- authorize source-compute retirement. + +The handoff binding must preserve those boundaries. + +## Planned Handoff + +The host executes: + +```text +fence host-controlled ingress + -> call gateway.suspend.prepare with stable handoff identity + -> retry busy without stopping the runtime + -> receive ready suspension lease + -> stop the Gateway and remaining authoritative writers cleanly + -> create and verify final owner components + -> assemble and verify one aggregate recovery point + -> durably accept exact recovery-point bytes and identity + -> finalize the source generation + -> derive sourceComputeRetirementAuthorized +``` + +If the handoff is abandoned before process stop, the controller calls +`gateway.suspend.resume` with the exact suspension ID. A lost control path does +not authorize progress; the host waits for lease expiry or proves the process +is already stopped through its existing supervisor contract. + +The host must complete clean Gateway stop before the ready lease expires or +renew the lease by repeating `gateway.suspend.prepare` with the same stable +request ID while the Gateway remains responsive. `gateway.suspend.status` does +not renew the lease. If the host cannot complete or renew the clean-stop window, +it abandons the planned handoff, resumes when possible, and leaves source +compute intact. After the process stops, the host must keep every supervisor, +health-recovery, and autoscaling start path fenced until handoff completes or +is explicitly abandoned. + +The host must not extend the suspension lease semantics by treating a stale +lease, health probe, process absence, or local snapshot path as handoff success. + +## Final Capture + +Final capture reuses owner primitives: + +- RFC 0013 `backup sqlite` for global and per-agent databases; +- the Recovery Point Components v1 contract for aggregate composition; and +- existing owner-specific capture for non-SQLite components. + +An aggregate made directly from ordinary RFC 0013 snapshots remains +`host-protected`. It may move only into a host protection domain whose +encryption, access control, and credential exposure policy accepts those exact +bytes. A deployment must not call the point credential-free portable unless it +also verifies owner-authored portability receipts bound to every component and +artifact digest. An obligation list alone is not portability evidence. + +Final capture must not run while an authoritative writer can still mutate the +captured source. A force kill, uncertain process termination, conventional +shutdown warning, capture blocker, verification failure, or unknown writer +state blocks the planned handoff. Recovery after such a failure uses the last +previously accepted recovery point and does not claim a new clean final point. + +The final recovery point is immutable. Retrying the same handoff identity must +return the same recovery-point identity or a typed conflict. + +## Host Acceptance + +The host durably records: + +- logical runtime and continuity lineage; +- source runtime generation; +- handoff identity; +- aggregate recovery-point identity and manifest digest; +- acceptance-set identity; +- exact aggregate-manifest digest and size; +- the canonical component acceptance inventory, including each owner-manifest + and artifact digest and size; +- durability class; +- accepted-at time; and +- storage-owned opaque reference. + +The opaque reference may identify a file, blob, object, or service record. It +must not expose credentials in OpenClaw metadata. + +Local artifact creation and host acceptance are separate facts. A host may +claim acceptance only after its selected durability boundary confirms every +byte sequence in the closed acceptance inventory. The storage implementation +may package those bytes as files, blobs, or service records, but must return +the same canonical acceptance-set identity. No digest of an unspecified +directory, archive, or concatenation is valid evidence. + +## Replay And Uncertainty + +Every mutating step is idempotent under the handoff identity. + +- Lost response after capture: replay returns the same local recovery point. +- Lost response after host acceptance: replay queries the same acceptance + operation and returns the same receipt. +- Different bytes under the same identity: quarantine. +- Unknown storage outcome: hold; do not capture a replacement and infer the + first attempt failed. +- Source process or adapter replacement before local state is durable: + quarantine unless the host can prove the exact accepted recovery point. + +## Source Compute Retirement Authority + +`sourceComputeRetirementAuthorized` is a host-owned durable conclusion bound +to: + +- runtime lineage; +- source generation; +- handoff identity; +- accepted recovery point; and +- the current lifecycle revision. + +It authorizes retirement of only the source compute generation. It never +authorizes deletion of recovery points, persistent tenant data, external +credentials, registry records, or another generation. + +New retained work or an operator cancellation must revoke or race with source +compute retirement through the host's durable lifecycle authority. OpenClaw +does not poll for that race after it has stopped. + +This fact is not a general `synced` or clean-filesystem state. It is valid only +for the exact runtime lineage, source generation, handoff, recovery point, and +lifecycle revision named by the record. Process absence, an expired suspension +lease, a local snapshot path, or a successful health probe cannot create it. + +## Conformance + +V1 conformance must prove: + +- current-main `gateway.suspend.*` is called rather than reimplemented; +- busy leaves the runtime running; +- ready closes tracked admission and pauses scheduling; +- lease renewal repeats `prepare` with the same request ID; +- host ingress is fenced separately; +- third-party and unregistered work limitations remain explicit; +- final capture occurs only after authoritative writers stop; +- one exact recovery point is accepted durably; +- the accepted logical byte set is closed and deterministically identified; +- host-protected snapshots cannot be reported as credential-free portable + without exact owner portability receipts; +- response loss replays the same acceptance; +- digest conflict and unknown outcome block source compute retirement; and +- `sourceComputeRetirementAuthorized` is generation-scoped and cannot purge + persistent data. diff --git a/rfcs/0013/recovery-point-components-v1-spec.md b/rfcs/0013/recovery-point-components-v1-spec.md new file mode 100644 index 00000000..d8479886 --- /dev/null +++ b/rfcs/0013/recovery-point-components-v1-spec.md @@ -0,0 +1,291 @@ +# Recovery Point Components v1 Specification + +This document is the implementer-facing component-composition specification for +RFC 0013, SQLite Snapshot Backup Artifacts. RFC 0013 defines one verified +SQLite snapshot directory. This sidecar defines how a recovery workflow may +compose those directories with other owner-authored artifacts without changing +the SQLite snapshot contract. + +Status: draft, tied to RFC 0013. + +## Implementation Evidence + +[openclaw/openclaw#112385](https://github.com/openclaw/openclaw/pull/112385) +is the paired evidence slice. It implements the V1 global and +owner-selected per-agent SQLite composition path, deterministic recovery-point +and acceptance-set identities, exact owner-manifest and artifact binding, +closed obligation validation, and the `host-protected` classification. + +The pull request is evidence for review of this contract. Its types and file +layout are not normative, and the specification does not require that pull +request to merge before the ownership decisions can be accepted. + +## Scope + +This specification defines: + +- one aggregate recovery point made from immutable owner components; +- use of RFC 0013 global and per-agent snapshot directories as SQLite + components; +- explicit non-SQLite owner components; +- a closed required-component inventory supplied by the runtime state owner; +- captured, external, reconstructed, and ephemeral obligations; +- an explicit protection classification that cannot imply credential-free + portability; +- exact component identity, ordering, compatibility, and verification; +- fail-closed aggregate manifest parsing and conformance. + +This specification does not define: + +- another SQLite snapshot command, repository, or manifest; +- mutation of an RFC 0013 snapshot after publication; +- upload, storage transport, retention, or encryption implementation; +- Gateway suspension, final handoff, or source-compute retirement; +- restore-on-boot, restored admission, hibernation, or wake; +- arbitrary restore hooks or a generic capture-provider registry. + +## Shipped Foundation + +An RFC 0013 component is a completed directory produced and verified by: + +```text +openclaw backup sqlite create +openclaw backup sqlite verify +openclaw backup sqlite restore +``` + +The aggregate layer must treat the directory and its strict `manifest.json` and +`database.sqlite` bytes as immutable. It must not: + +- copy a live `.sqlite`, `-wal`, `-shm`, or `-journal` file; +- reimplement `VACUUM INTO`; +- weaken role, owner, schema, index, ACL, DACL, link, or race validation; +- claim that a locally present path is durable host acceptance; or +- treat a successful SQLite snapshot as complete OpenClaw recovery state. + +## Ownership + +| Owner | Authority | +| --- | --- | +| SQLite owner | RFC 0013 capture, manifest, verification, and fresh restore. | +| Non-SQLite owner | Artifact contents, compatibility, and restore semantics for its state. | +| Recovery-point composer | Required-component inventory, exact identities, dependency order, and obligation closure. | +| Host or operator | Durable destination, encryption, access control, retention, and recovery-point selection. | + +The composer does not inspect private SQLite schema to infer application state. +It consumes the verified owner result. + +### Required-component inventory authority + +The runtime state owner, not the host and not the aggregate composer, selects +the complete required-component inventory. The owner must derive it from the +same activation-pinned runtime and selected-agent state that governs the source +generation, then bind that inventory revision and source generation into the +recovery-point input. A caller-supplied list of agent IDs, a filesystem scan, +or whatever databases happen to exist is not authoritative. + +The composer validates and records the owner result but cannot add, remove, or +reinterpret components. Retrying the same capture operation with a different +owner inventory conflicts. A changed selected-agent set requires a new owner +inventory revision and a new recovery point. Missing owner evidence fails +closed rather than treating an empty or partial inventory as complete. + +## Component Model + +A recovery point contains a closed, ordered component list. V1 component kinds +are: + +```text +sqlite-global +sqlite-agent +config +workspace +plugin-source +``` + +The SQLite kinds reference RFC 0013 snapshot directories. The other kinds must +be produced by their existing semantic owner from bounded regular-file +inventories. V1 does not allow third-party component registration. + +| Kind | Semantic owner | Eligibility | +| --- | --- | --- | +| `sqlite-global` | OpenClaw shared-state database owner | Eligible through RFC 0013. | +| `sqlite-agent` | OpenClaw per-agent database owner | Eligible through RFC 0013. | +| `config` | OpenClaw config loader and include resolver | Reserved until the owner exposes bounded capture and restore verification. | +| `workspace` | OpenClaw workspace owner | Reserved until the owner exposes bounded capture and restore verification. | +| `plugin-source` | OpenClaw plugin installation owner | Reserved until the owner exposes bounded capture and restore verification. | + +Reserved kinds are specified so the aggregate contract has stable ownership. +They are not eligible components until their named owner contract exists and +passes this specification's verification requirements. A host must not invent +path-copy behavior to make a reserved kind appear implemented. + +Every component records: + +- stable component ID; +- kind and owner; +- immutable artifact digest and size; +- owner manifest digest and size when the component has its own manifest; +- compatibility identity; +- dependency IDs; +- capture time; +- required or optional status. + +Paths are materialization details, not component identity. + +## Aggregate Manifest + +Illustrative V1 shape: + +```json +{ + "version": "openclaw-recovery-point/v1", + "recoveryPointId": "recovery-point-42", + "createdAt": "2026-07-21T15:00:00.000Z", + "inventory": { + "version": "openclaw-runtime-sqlite-inventory/v1", + "owner": "openclaw-state", + "sourceRuntimeGeneration": "runtime-generation-17", + "revision": "inventory-revision-9", + "requiredComponentIds": ["sqlite/global", "sqlite/agent/main"] + }, + "protection": { + "mode": "host-protected" + }, + "components": [ + { + "id": "sqlite/global", + "kind": "sqlite-global", + "owner": "openclaw-state", + "artifactSha256": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "artifactSizeBytes": 1048576, + "ownerManifestSha256": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "ownerManifestSizeBytes": 512, + "compatibility": "openclaw-state-schema/7", + "dependsOn": [], + "required": true + } + ], + "obligations": { + "external": [], + "reconstructed": [], + "ephemeral": [] + } +} +``` + +The exact serialization must be deterministic. `recoveryPointId` is the +lowercase hexadecimal SHA-256 digest of the canonical UTF-8 JSON serialization +of the aggregate manifest with the `recoveryPointId` field omitted. Canonical +JSON sorts object keys recursively, preserves array order, and emits no +insignificant whitespace. + +Unknown required fields, +unknown component kinds, duplicate IDs, missing dependencies, dependency +cycles, digest mismatches, and unsupported major versions fail closed. + +The state owner supplies the complete required component IDs for the selected +runtime before composition. V1 requires exactly one global component and the +exact selected set of per-agent components. The inventory is canonicalized, +stored in the manifest with its owner, source runtime generation, and revision, +and covered by `recoveryPointId`. Composition fails on a missing or extra +component; successfully composing one agent cannot imply that every agent +owned by the selected runtime was captured. + +The aggregate manifest does not copy the complete RFC 0013 manifest. It binds +that owner manifest by digest and preserves it beside the component artifact. + +## Portability And Sensitive State + +An ordinary RFC 0013 snapshot can contain auth profiles, session state, plugin +state, and credentials-adjacent records. This sidecar does not silently remove +or rewrite those rows. A recovery point composed directly from ordinary RFC +0013 snapshots therefore has `protection.mode = host-protected`. The aggregate +manifest cannot label itself credential-free or portable. + +A deployment may call a recovery point credential-free portable only when a +separate owner-authored portability receipt proves that every component is an +approved portable projection, binds the exact component and artifact digest, +names the owner contract and version, and proves every sensitive surface has +one explicit treatment: + +- **captured**: the owner approves non-reissuable runtime-owned bytes for the + selected host protection domain; +- **external**: the destination re-resolves the value from its existing owner; +- **reconstructed**: the owner establishes a safe default or rebuilds from + declared inputs; or +- **ephemeral**: normal startup recreates it and readiness does not depend on + its captured bytes. + +Host-managed or reissuable credentials, OAuth tokens, provider tokens, and +short-lived session credentials must be `external` or omitted. They cannot use +the `captured` treatment in V1. + +Manifest obligation arrays imply the `external`, `reconstructed`, or +`ephemeral` treatment. Entries carry a closed obligation kind, stable owner, +identifier, and readiness requirement. V1 initially supports only named +SecretRef, plugin-dependency, and runtime-cache owner pairs; unknown owner, +kind, or treatment combinations fail closed. Obligations must not carry secret +values, credential bytes, raw prompts, message payloads, or arbitrary commands. +Captured state is represented by its owner component, not by an obligation +entry. + +If a credential-free portable projection is required, its state owner must +define and verify that projection as a separate operation. It must not change +ordinary `backup sqlite` behavior or mutate the live database. + +## Acceptance Byte Set + +The recovery point defines a closed logical acceptance byte set without +standardizing a tarball, directory tree, object-store key, or transport: + +1. the canonical aggregate-manifest bytes; +2. each exact owner-manifest byte sequence; and +3. each exact component-artifact byte sequence. + +An acceptance inventory records the aggregate-manifest digest and size plus, +for every canonically ordered component, its component ID, owner-manifest +digest and size, and artifact digest and size. Its identity is the SHA-256 of +the canonical inventory with that identity field omitted. Storage packaging +and paths remain host-owned and are not part of component identity. + +## Verification + +Aggregate verification must: + +1. parse the closed manifest; +2. verify deterministic identity; +3. verify every component digest and size from content-pinned reads; +4. invoke the owner verifier for every required component; +5. require the component IDs to equal the closed owner inventory; +6. validate dependency ordering and compatibility; +7. validate every obligation against the supported owner/treatment pairs; and +8. return one exact verified recovery-point identity and acceptance inventory. + +Verification failure must not create restore targets or return a +success-shaped partial result. + +## Conformance + +V1 conformance must prove: + +- one global and the exact owner-selected per-agent RFC 0013 components; +- the inventory is bound to the source runtime generation and owner revision; +- caller-supplied IDs and filesystem discovery cannot replace owner selection; +- rejection of missing and extra components against the bound inventory; +- deterministic aggregate identity; +- exact owner-manifest binding; +- canonical recovery-point identity; +- rejection of missing, duplicate, unknown, corrupt, or reordered components; +- rejection of dependency cycles and unsupported compatibility; +- explicit sensitive-state treatment and rejection of reissuable credential + bytes; +- ordinary snapshot composition remaining `host-protected` until an exact + owner portability receipt exists; +- deterministic acceptance inventory over the closed logical byte set; +- no secret values in aggregate metadata; and +- no duplicate SQLite capture or verification implementation. + +The first reserved non-SQLite kind gains its own conformance cases only after +its named semantic owner exposes the bounded capture and restore-verification +contract required by this specification. diff --git a/rfcs/0013/restored-admission-v1-spec.md b/rfcs/0013/restored-admission-v1-spec.md new file mode 100644 index 00000000..053e83bf --- /dev/null +++ b/rfcs/0013/restored-admission-v1-spec.md @@ -0,0 +1,455 @@ +# Restored Admission v1 Specification + +This document is the implementer-facing restore and admission specification for +accepted RFC 0013 follow-on recovery points. It defines the ordering and +evidence required before a restored Gateway can receive work. + +Status: draft, tied to RFC 0013. + +## Implementation Evidence + +[openclaw/openclaw#112896](https://github.com/openclaw/openclaw/pull/112896) +is the OpenClaw evidence slice for exact fresh-target restore and +readiness-gated admission, stacked on final-capture PR +[openclaw/openclaw#112865](https://github.com/openclaw/openclaw/pull/112865). +It reuses the RFC 0013 SQLite provider, durably binds the restore receipt and +private startup descriptor, reconciles the canonical scheduler, and opens work +admission only after owner readiness. + +The PR also implements the proposed `gateway.restore.status` projection over +that same durable record. The method below is the recommended V1 design and +representative proof for owner review; it is not an approved public contract +until the Gateway protocol owner accepts it. + +The draft deliberately holds when required external or reconstructed +obligations lack accepted owner evidence. It does not add wake routing, +placement, idle policy, host acceptance, or a public restore-hook registry. +This pull request is implementation evidence; this specification remains the +normative responsibility boundary. + +## Scope + +This specification defines: + +- a durable restore hold before target mutation or process start; +- exact accepted recovery-point selection; +- fresh-path component restore and owner verification; +- external and reconstructed obligation handling; +- scheduler reconciliation; +- required owner and generic Gateway readiness; +- RFC 0018 readiness projection for the restored recovery point; +- one-time restored admission; +- same-child replay after coordinator failure. + +This specification does not define: + +- another SQLite restore implementation; +- a public generic restore-hook or readiness-provider registry; +- capture, host storage, retention, or source-compute retirement; +- host placement, proxy transport, or Channel delivery; +- Elastic wake policy. + +This specification applies to any recovery point previously accepted through a +conforming durability boundary. That includes restore of the latest accepted +point after forced source loss. It does not create an accepted recovery point, +define the forced-loss RPO, or permit restore from an unaccepted local path. + +## Shipped Foundation + +SQLite components restore through RFC 0013: + +```text +openclaw backup sqlite verify +openclaw backup sqlite restore --target +``` + +The restore path must preserve RFC 0013's content-pinned verification, +fresh-target requirement, stale-sidecar rejection, and owner validation. It +must not copy snapshot bytes directly into an existing live database. + +The host may use current-main `gateway.suspend.*` for planned source handoff. +Restored admission does not add or modify that suspension API. + +## Restore Hold + +Before creating any original target or starting a Gateway, the host acquires a +durable hold bound to: + +- logical runtime and continuity lineage; +- lifecycle owner generation; +- destination runtime generation; +- accepted recovery-point identity; +- aggregate manifest digest; +- acceptance-set identity; +- restore operation identity; and +- destination owner. + +Every launcher, health recovery, restart, warm-up, and autoscaling path must +reject ordinary startup while the hold is active. + +The hold is released only by: + +- exact restored-admission completion; +- explicit operator rollback before any restored process becomes runnable; or +- quarantine and a separately authorized recovery action. + +Timeout does not silently release a partially applied restore. + +Quarantine has no automatic exit in V1. A lifecycle owner may exit only through +an explicit durable revocation record that binds the quarantined hold and +operation identity. A later attempt uses a new restore-operation identity and a +newly selected accepted recovery point. A newer attempt must not implicitly +supersede, delete, or reuse the quarantined hold or its evidence. + +## Restore Ordering + +The binding executes: + +```text +verify accepted aggregate manifest + -> verify every required component and compatibility identity + -> create fresh owner-private destination roots + -> restore components in declared dependency order + -> resolve external obligations through existing owners + -> perform declared reconstruction through existing owners + -> start one restored Gateway with private restore evidence + -> reconcile scheduler state + -> satisfy required owner readiness + -> satisfy generic Gateway readiness + -> durably record restored-ready evidence + -> consume that exact evidence to open admission +``` + +The exact private transport into the restored process is an implementation +binding, not a new public RFC 0013 command. It must be bounded, owner-private, +versioned, and absent during ordinary startup. + +## Required Evidence + +The restored-start binding must preserve: + +- accepted recovery-point and aggregate manifest identity; +- exact acceptance-set identity; +- destination runtime generation; +- lifecycle owner generation; +- component restore receipt identities; +- stable, generation-bound scheduler-reconciliation evidence identity; +- normalized required-owner readiness evidence identity; +- Gateway incarnation identity; +- one admission identity; and +- one canonical readiness generation. + +These identities are independent. A process ID, worker name, health probe, +container generation, or local path cannot substitute for them. + +The durable completion record contains identities and bounded disposition +metadata only. It must not contain credential values, raw artifact locations, +message payloads, prompts, or arbitrary commands. + +V1 may derive the scheduler and readiness evidence identities only after the +semantic owner operation succeeds. The derivation binds a versioned evidence +label, the exact restore receipt, recovery-point and acceptance-set identities, +the destination runtime generation, and the normalized outcome. It is not a +digest of mutable scheduler status, does not claim to identify the complete +scheduler definition set, and does not substitute for a richer owner-authored +receipt when an owner later exposes one. + +## Scheduler And Owner Readiness + +OpenClaw remains authoritative for: + +- restored scheduler definitions; +- due and missed-run reconciliation; +- duplicate suppression; +- catch-up policy; +- earliest semantic deadline; and +- readiness to admit ordinary work. + +The host must not parse cron expressions or synthesize due work. + +External or reconstructed obligations are evaluated by their existing owners. +Examples include credential availability and plugin dependency reconstruction. +Missing required evidence holds readiness closed. An obligation classified as +not required must be decided by its owner from restored effective +configuration, not by the host. + +## Admission + +Restore completion, process startup, `/healthz`, container readiness, and a +successful SQLite open do not independently authorize work. + +Admission opens only after one canonical restored-ready record binds: + +- the accepted recovery point; +- the accepted logical byte set; +- destination and lifecycle generations; +- component restore receipts; +- generation-bound scheduler-reconciliation evidence; +- normalized required-owner readiness evidence; +- Gateway readiness; and +- the one-time admission identity. + +Ordinary startup remains unchanged because it has no restored-start evidence. +Ordinary startup cannot consume a committed restore hold, and restored startup +cannot use an ordinary admission path. + +## Canonical Readiness Projection + +When enabled, restored admission uses the opt-in readiness result defined by +[RFC 0018](https://github.com/openclaw/rfcs/pull/33). It does not define a +parallel health evaluator or provider registry. This composition is contingent +on RFC 0018 acceptance and activation; the restore hold, admission fence, and +`gateway.restore.status` remain independently complete without it. + +A restored start publishes one required condition and adds the referenced +objects to RFC 0018's identity package. Stable refs name roles; current IDs and +generations name their occupants: + +```text +type: RecoveryPointRestored +subjectRef: openclaw/gateway +relatedSubjectRefs: + - openclaw/restore-operation + - openclaw/recovery-point + - openclaw/runtime-generation + - openclaw/scheduler-reconciliation + - openclaw/state-owner/... + +identity.subjects: + - { ref: openclaw/gateway, kind: gateway, id: } + - { ref: openclaw/restore-operation, kind: restore-operation, + id: } + - { ref: openclaw/recovery-point, kind: recovery-point, + id: } + - { ref: openclaw/runtime-generation, kind: runtime-generation, + generation: } +``` + +The related subject list is bounded by RFC 0018 and uses its canonical subject +deduplication and lifetime rules. `RecoveryPointRestored` is `Unknown` while +required durable evidence is unavailable, `False` when current evidence proves +the selected restore is not admissible, and `True` only after the same durable +record that authorizes admission commits. A required non-`True` result keeps +aggregate readiness false. + +`gateway.restore.status` and RFC 0018 readiness are two projections of one +owner record. The status method adds caller-supplied operation fencing and +typed conflict behavior; `/readyz`, `openclaw ready`, Status, and Gateway +readiness RPC report the active runtime's canonical result. They must expose +the same readiness generation and subject identities. No projection may infer +restore completion from `/healthz`, process existence, database-open success, +or container readiness. + +## Gateway Restore Status + +V1 adds one read-only Gateway Protocol method: + +```text +gateway.restore.status +``` + +The method is the live-destination observation seam for a host. It follows the +existing `gateway.suspend.status` conventions: + +- core-owned and host-neutral; +- closed TypeBox request and result schemas in the Gateway Protocol package; +- `operator.read` scope; +- no control-plane-write classification; +- no config key or environment variable; +- idempotent and side-effect free; and +- exposed by authenticated Gateway control transports that opt into core RPC + methods, including the existing Admin HTTP RPC path. + +The request is: + +```ts +type GatewayRestoreStatusParams = { + restoreOperationId: string; +}; +``` + +`restoreOperationId` must satisfy the same bounded non-empty token rules as +the restored-admission operation. It is required even though only one restored +operation can own a Gateway incarnation. Requiring it prevents an observation +for one host attempt from being mistaken for another generation's readiness. + +The successful result is a closed union: + +```ts +type GatewayRestoreStatusResult = + | { + status: "not-restored"; + } + | { + status: "held"; + reason: "scheduler-reconciliation" | "owner-readiness" | "ready-commit"; + retryAfterMs: number; + runtimeLineage: string; + lifecycleOwnerGeneration: string; + destinationRuntimeGeneration: string; + restoreOperationId: string; + destinationOwner: string; + admissionIdentity: string; + recoveryPointId: string; + acceptanceSetId: string; + restoreReceiptIdentity: string; + } + | { + status: "ready"; + runtimeLineage: string; + lifecycleOwnerGeneration: string; + destinationRuntimeGeneration: string; + restoreOperationId: string; + destinationOwner: string; + admissionIdentity: string; + recoveryPointId: string; + acceptanceSetId: string; + restoreReceiptIdentity: string; + schedulerIdentity: string; + ownerReadinessIdentity: string; + readinessIdentity: string; + }; +``` + +`not-restored` means the running Gateway has no restored-start evidence. It is +not permission for a host restore operation to deliver retained work. `held` +means the expected restored incarnation is live but ordinary work admission is +still closed. `ready` is returned only after the exact ready record is durable +and the same admission identity has opened Gateway work admission. + +All strings use explicit protocol bounds. `retryAfterMs` is a +non-negative bounded integer and only a polling hint. Results contain no local +paths, artifact locations, credentials, retained payloads, prompts, or owner +diagnostic text. + +The handler must derive `held` and `ready` from the same in-memory startup +binding and operation-scoped SQLite journal that enforce admission. It must not +create a second restore-status file, infer readiness from `/healthz`, or hash a +mutable runtime status object. A replayed call returns the same identity fields +for the same durable record. The Gateway may retain the already validated +status projection in memory; polling must not synchronously reopen or rehash +the recovery journal on every request. Transition to `ready` occurs only after +the durable record commit succeeds. + +If a different restore operation owns the running Gateway, the method returns +`UNAVAILABLE` with bounded details reason `restored-admission-conflict`; it +must not return that operation's identity as a success result. Invalid or +oversized tokens return `INVALID_REQUEST`. Journal corruption, contradictory +evidence, or failure to read a required committed record returns `UNAVAILABLE` +and keeps work admission closed. No error is success-shaped. + +The method must remain callable while restored admission is held through an +authenticated pre-admission control path. The path may reserve tracked work to +complete authentication, but the pre-auth request shape must be limited to a +non-enrolling host probe and the resulting connection must remain scoped to +read-only status inspection. The lease lasts through handshake completion so +a concurrent restart drains it rather than racing it. Restart always wins; +nodes, pairing or enrollment, remote clients, presence-bearing sessions, and +ordinary RPC methods remain fenced. A host without such a control path may use +`/readyz` only as a backoff hint and call `gateway.restore.status` after +readiness, but it must not deliver retained work until the exact `ready` result +matches its operation and destination generation. + +For a fresh destination, the recommended held-state transport is the existing +authenticated Admin HTTP RPC route bound to the host-controlled loopback path. +The status method does not justify a new unauthenticated probe, public listener, +bearer-token scheme, or sidecar process. A mismatched-operation error must not +disclose the active operation's identities. + +There is no `gateway.restore.admit`, `resume`, or caller-supplied readiness +method. Admission remains an OpenClaw-owned consequence of durable owner +evidence, not a host command. + +## Crash Replay + +The required crash boundary is: + +```text +restored-ready durably recorded + -> coordinator exits before completing its host operation +``` + +A fresh coordinator must: + +- resolve the same destination child; +- establish a fresh process-local control or proxy connection; +- read the retained restored-ready evidence; +- verify the same readiness generation; and +- complete without repeating destination allocation, preparation, component + restore, scheduler reconciliation, or admission. + +If the child, generation, recovery point, or retained evidence differs, the +operation conflicts or quarantines. It must not allocate a second winner while +the first can still be authoritative. + +All persistent journals and paths are scoped by stable operation identity. +Fixed shared paths across independent attempts are invalid. + +OpenClaw-owned recovery intent, committed restore results, startup descriptors, +and restored-ready evidence use an operation-scoped SQLite journal. They are +not steady-state JSON sidecars. A dedicated journal is required because final +capture must commit intent before snapshotting the shared state database, and +restore must commit intent before the fresh shared state database exists. + +### Recovery journal ownership and lifecycle + +OpenClaw core owns the recovery journal schema, versioning, atomic transitions, +migrations, integrity checks, and repair classification. Hosts may retain, +protect, and back up the journal with the runtime protection domain, but must +not write rows, advance transitions, or infer success from file presence. + +The journal must be opened and integrity-checked before a restored start can +publish readiness. Unsupported schema versions, failed migrations, malformed +terminal records, and contradictory operation identities hold or quarantine; +they never fall back to ordinary startup. Repair is an explicit offline owner +operation and cannot fabricate committed acceptance, restore, or readiness +evidence. + +An operation record remains available while its restore hold, destination +generation, status query, crash replay, or retained-delivery reference can +still be active. Cleanup may remove a terminal record only after the owner can +prove that no supported replay or status consumer references it and the host's +retention obligation has expired. Cleanup is idempotent and cannot remove the +currently active restored-admission record. Orphaned or partially committed +records are quarantined rather than age-deleted. + +## Failure Dispositions + +- **retry same incarnation**: transient response loss with the same child and + exact retained evidence; +- **hold**: missing dependency, temporary owner unavailability, or a timeout + before any destination target mutation; +- **quarantine**: corrupt artifact, identity conflict, stale generation, + contradictory replay, or a timeout or partially applied state after target + mutation when the winner is uncertain. + +No failure opens admission or emits a success-shaped ready result. + +## Conformance + +V1 conformance must prove: + +- RFC 0013 verify and fresh restore are reused; +- ordinary startup is unchanged; +- the restore hold fences every start path; +- no target mutation occurs before aggregate verification; +- component dependency order is preserved; +- external and reconstructed obligations remain owner-evaluated; +- scheduler reconciliation precedes readiness; +- `RecoveryPointRestored` uses RFC 0018 subjects, lifetimes, aggregation, and + bounded projection rules; +- canonical readiness and `gateway.restore.status` project the same durable + readiness generation and identities; +- admission opens exactly once from exact durable evidence; +- process health alone cannot open admission; +- `gateway.restore.status` is read-only, operation-fenced, and returns exact + durable identities for `ready`; +- ordinary and mismatched restored Gateways cannot return success-shaped + readiness for the requested operation; +- the status method remains available through an authenticated control path + while ordinary restored work admission is held; +- coordinator crash replay reuses the same child and readiness generation; +- preparation and restore execute exactly once; and +- journal schema, migration, corruption, repair, retention, and cleanup remain + OpenClaw-owner operations and fail closed; and +- stale, contradictory, corrupt, and fixed-path collision cases fail closed.