feat: add durable generic agent checkpoints

[N0xMare]

Summary

Adds a generic, durable checkpoint contract for stateful Smithers agents.

Checkpoint-aware agents can publish opaque JSON state during or after generation, resume it on retries and process restarts, and carry exact checkpoint lineage through snapshots, rewind, replay, and historical forks.

This is backend-agnostic infrastructure. Existing agents remain unchanged unless they opt in. It also provides the durable foundation needed by future stateful adapters such as Nanocodex.

Why

Smithers persisted provider-native CLI session IDs, but had no generic mechanism for agents that expose richer resumable state or snapshots. That prevented stateful library-backed agents from preserving high-fidelity execution state across retries, schema correction, engine crashes, process restarts, rewind, reset, and historical forks.

Agent contract

Introduces:

• AgentCheckpoint, AgentCheckpointMode, AgentCheckpointFormat, and AgentCheckpointCapability
• checkpointFormats for states an agent may produce
• checkpointCapabilities for codec/version/mode combinations an agent may consume
• discriminated resumeCheckpoint and checkpointMode continuation inputs
• awaited onCheckpoint(...) publication as a durability fence

Checkpoint envelopes use strict JSON:

{
  codec: string;
  version: number;
  payload: JsonValue;
}

Generic checkpoints and provider-native session continuation remain mutually exclusive.

Persistence and engine behavior

Migration 0035_agent_checkpoints adds:

• content-addressed immutable checkpoint storage
• attempt-owned references and atomic sequence allocation
• SHA-256 integrity validation and deduplication
• ownership/cancellation fencing
• bounded cursor pagination and orphan-content GC
• SQLite, external-SQLite, PGlite, and PostgreSQL support

The engine supports checkpoints across task retries, schema-correction turns, failed attempts, progress publication, process restart, task forks, cache identity, and same-task workspace restoration. The maximum checkpoint size is 16 MiB. History scans fail explicitly at their safety bound instead of silently starting fresh.

Time travel

Snapshots preserve the exact attempt rows and checkpoint bytes visible to each frame. Rewind and fork can therefore:

• restore exact historical checkpoint provenance
• trim references above exact sequence horizons
• rehydrate content reclaimed after reset/GC
• serialize destructive operations through the durable rewind lease
• reject corrupt or oversized provenance before materialization

Legacy snapshots retain their timestamp fallback.

Live validation finding

A live Codex workflow exposed an existing fast path that rewrote completed nodes with lastAttempt: null, causing later snapshots to lose checkpoint lineage. This PR preserves or reconstructs the durable attempt number and adds a regression proving completed-run resume and latest-frame fork retain checkpoint references.

Compatibility

• Existing agents require no changes.
• Existing native CLI-session continuation remains supported.
• Migration 0035 is additive.
• Agents participate only when they declare checkpoint formats/capabilities.

Testing

Focused local validation:

• Agent checkpoint contract: 10 passed
• Engine checkpoint suite: 21 passed
• Engine/cache/workspace suites: 65 passed
• Database and migration suites: 70 passed
• Time-travel suites: 84 passed
• Snapshot suite and benchmark: 20 passed
• Fresh-process SQLite SIGKILL/resume E2E: passed
• Root typecheck, lint, formatting, declarations, and docs gates: passed
• Independent final reviews: no remaining P0/P1 findings

Live gpt-5.3-codex-spark validation covered:

1. Medium structured output.
2. Forced schema failure and generic checkpoint persistence.
3. Exact Codex-session continuation on attempt two.
4. Xhigh review of the resumed output.
5. Fresh-process reopen with zero additional model calls.
6. Latest-frame historical fork inheriting both checkpoint references.

PostgreSQL CI gates

Real-PostgreSQL-only tests are wired into CI for migration repair, concurrent sequence allocation, writer/GC overlap, publication versus owner takeover/cancellation, and fresh-process SIGKILL/resume. They require SMITHERS_TEST_PG_URL and therefore run in the GitHub PostgreSQL job rather than the local fallback suite.

Follow-up

A subsequent PR can add NanocodexAgent on top of this contract without coupling Nanocodex-specific state or execution semantics into Smithers core.

Stack and merge order

This is layer 2 of 3. It is based on and requires #1463 at 92ca474344a194c188e8a6981ec5e51d907e341e; this PR's verified head is f0c287dfb5c0f37f4657d25148c8ef5bc4cb6f30.

Merge order: #1463 → #1449 → #1461. Because these are cross-repository PRs, the GitHub base intentionally remains main, so the displayed diff is cumulative until #1463 lands.

[roninjin10]

Automated code review

Reviewed 48 file(s) in the diff against main. 5 major, 12 minor, 1 info.

packages/db/src/adapter.js:2530 — major/correctness (confirmed)

Use Effect.tryPromise here and in the two new cleanup operations below. Effect.promise converts rejected database promises into defects, so withSqliteWriteRetryEffect cannot retry transient SQLITE_BUSY/SQLITE_IOERR failures and callers do not receive the declared SmithersError. A transient checkpoint write error therefore fails immediately instead of retrying the transaction.

packages/db/src/schema-migrations.js:2304 — major/correctness (confirmed)

Make this trigger repair atomic. ensureSchema() invokes this block for every PostgreSQL runtime, but the DROP and CREATE are separate autocommit statements. Two concurrent startups can both finish DROP before either CREATE; the second CREATE then fails with duplicate_object, aborting schema initialization. Use CREATE OR REPLACE TRIGGER or serialize the pair in one locked transaction.

await pgConn.query({
  text: `CREATE OR REPLACE TRIGGER _smithers_agent_checkpoint_refs_delete
         AFTER DELETE ON _smithers_agent_checkpoints
         FOR EACH ROW EXECUTE FUNCTION _smithers_delete_orphan_agent_checkpoint_content()`,
});

packages/time-travel/src/jumpToFrame.js:539 — major/correctness (confirmed)

Delete the entire snapshot-owned attempt scope before restoring references. Retry resets numbering to attempt 1, so after a snapshot whose lastAttempt is greater than 1, a reused older attempt can contain new sequences. This loop replaces only sequences already present in the snapshot, while the later horizon trims only lastAttempt; extra sequences on the reused attempt therefore survive the rewind. Delete all references for every attempt in provenance.attempts, then reinsert the snapshot references.

for (const tuple of provenance.attempts) {
    await storage.deleteWhere(
      "_smithers_agent_checkpoints",
      "run_id = ? AND node_id = ? AND iteration = ? AND attempt = ?",
      [runId, tuple[0], tuple[1], tuple[2]],
    );
  }

packages/time-travel/src/jumpToFrame.js:547 — major/performance (confirmed)

Batch this rehydration. The snapshot format permits 100,000 checkpoint references, but this loop performs two serial database round trips per reference; the surrounding delete and reference-insert loops add two more. A valid checkpoint-rich snapshot can therefore issue roughly 400,000 statements while holding the rewind transaction and pausing execution, making PostgreSQL rewinds likely to time out. Deduplicate content hashes and use chunked VALUES operations for deletion, content validation, and reference insertion.

packages/time-travel/src/resetCancelMarker.js:38 — major/correctness (confirmed)

This uses the session-corruption flag as an unscoped reset boundary. retry-task stamps every prior attempt, while listAttempts returns attempts descending and the engine vetoes legacy resume when any newest failed/reset attempt has this flag. If a task previously reached attempt 2+, fresh post-reset attempt 1 can create a healthy CLI session and then fail, but old reset attempt 2 still carries this flag when the new attempt 2 starts. The engine consequently rejects the new attempt 1 session, conversation, and smithers.cli-session checkpoint. This can make every retry in the reset execution start from scratch and repeat work or side effects. Scope the veto to resume state from before the reset instead of stamping the general discardResumeSession flag on every cancelled attempt.

packages/agents/src/agent-checkpoint.js:160 — minor/correctness (confirmed)

-0 passes this validation, but JSON.stringify(-0) emits 0, so cloneAgentCheckpoint silently changes an allowed finite payload despite its lossless-JSON contract. Reject negative zero or serialize it without coercion.

if (typeof value === "number") {
    if (!Number.isFinite(value)) throw new TypeError(`Agent checkpoint contains a non-finite number at ${path}.`);
    if (Object.is(value, -0)) throw new TypeError(`Agent checkpoint contains negative zero at ${path}.`);
    return;
  }

packages/db/src/adapter.js:2603 — minor/correctness (plausible)

The conflict path has a race with orphan GC. ON CONFLICT DO NOTHING neither returns nor locks the existing content row, leaving a gap before FOR KEY SHARE. If GC deletes that unreferenced row during the gap, the SELECT returns nothing and a valid checkpoint publication rolls back as corruption. Read and lock the row first; only insert when it is absent, retaining the final locked read to handle a concurrent inserter.

content = await this.internalStorage.queryOne(
  `SELECT * FROM _smithers_agent_checkpoint_contents
   WHERE content_hash = ? LIMIT 1 FOR KEY SHARE`,
  [contentHash],
);
if (!content) {
  content = await this.internalStorage.queryOne(
    `INSERT INTO _smithers_agent_checkpoint_contents
       (content_hash, checkpoint_json, size_bytes, created_at_ms)
     VALUES (?, ?, ?, ?)
     ON CONFLICT (content_hash) DO NOTHING
     RETURNING content_hash, checkpoint_json, size_bytes, created_at_ms`,
    [contentHash, row.checkpointJson, sizeBytes, createdAtMs],
  );
  if (!content) {
    content = await this.internalStorage.queryOne(
      `SELECT * FROM _smithers_agent_checkpoint_contents
       WHERE content_hash = ? LIMIT 1 FOR KEY SHARE`,
      [contentHash],
    );
  }
}

packages/db/src/index.d.ts:3350 — minor/correctness (confirmed)

Both newly exported tables erase their schema with SQLiteTableWithColumns<any>. This makes nonexistent columns type-check and turns $inferSelect into any, unlike the other exported schema tables. Preserve the inferred column configuration in these declarations and add a type-consumer assertion for their columns/row types.

packages/db/src/schema-migrations.js:329 — minor/correctness (confirmed)

Do not accept a nullable content identity. In an ordinary SQLite rowid table, TEXT PRIMARY KEY does allow multiple NULL values; bun:sqlite confirms this behavior. This assertion therefore blesses a schema that does not enforce the content-address invariant, and NULL content rows cannot be referenced or reached by the existing hash-ordered orphan collector. Define the canonical column as TEXT NOT NULL PRIMARY KEY and require non-nullability here.

packages/db/src/schema-migrations.js:670 — minor/correctness (confirmed)

Require agent_id to be nullable, not merely exempt it from the non-null check. As written, an agent_id TEXT NOT NULL column passes validation. putAgentCheckpoint() accepts an omitted/null agent ID and inserts NULL, so such a schema is reported as applied and then fails normal checkpoint writes at runtime.

column.is_nullable !== (column.name === "agent_id" ? "YES" : "NO")

packages/errors/src/smithersErrorDefinitions.js:103 — minor/docs (confirmed)

This description drops a live trigger for this code. validateForkSources() still throws TASK_FORK_SESSION_UNAVAILABLE when a task uses fork without an agent, regardless of the source state. Because when feeds the public error reference, users encountering that authoring error are now given an incomplete explanation.

when: "A <Task fork> cannot obtain usable agent state — either the forking task is not an agent task, or the source produced neither a compatible checkpoint nor a forkable conversation.",

packages/smithers/tests/fixtures/agentCheckpointRestartWorkflow.js:61 — minor/tests (confirmed)

This cutpoint expires after 60 seconds, but the parent test allows listAgentCheckpointRefs alone to wait that long and performs several more database checks before sending SIGKILL. On a slow run, this callback can return successfully first, so the child exits normally and the later expect(killed.signal).toBe("SIGKILL") fails even though checkpoint recovery works. Keep the fixture blocked until the parent terminates it.

await new Promise(() => {
  setInterval(() => {}, blockMs);
});

packages/time-travel/src/fork/forkRunEffect.js:294 — minor/correctness (confirmed)

The fallback reads current attempt rows without constraining them to the selected snapshot. If a retry/reset reused an attempt key after this frame, an older fork copies the replacement row's state, response, and metadata. resolveForkAgentState later consumes agentCheckpoint or agentConversation from that copied row, so the child can resume from state created after frameNo. Timestamp-filter fallback rows at minimum; exact inheritance should require embedded attempt provenance.

packages/time-travel/src/fork/forkRunEffect.js:349 — minor/performance (confirmed)

Checkpoint inheritance performs sequential per-row writes: one insert per attempt and up to three statements per checkpoint reference. The accepted provenance limits allow 100,000 attempts and 100,000 references, making a valid fork execute hundreds of thousands of statements while holding the transaction and rewind fence. Batch attempts, contents, validation reads, and references into bounded multi-row operations.

packages/time-travel/src/jumpToFrame.js:547 — minor/correctness (plausible)

Atomically lock the existing content row on PostgreSQL. When another run references the same hash, insertIgnore can no-op and the following plain SELECT holds no key lock. Concurrent deletion of that run's last reference can then trigger orphan cleanup before the restored reference is inserted, causing a foreign-key failure and aborting the rewind. Use a single no-op INSERT ... ON CONFLICT DO UPDATE ... RETURNING statement, as snapshot content persistence does, or an equivalent locked read with retry.

packages/time-travel/src/snapshot/agentCheckpointProvenance.js:221 — minor/data-loss (confirmed)

Return null only when the provenance property is absent. A present block with an unsupported version or malformed shape is currently treated as legacy; jumpToFrame then skips rehydration and fork falls back to mutable live rows. After attempt reuse or checkpoint GC, the operation can succeed without restoring the snapshot-owned lineage. Throw a corruption error for present-but-invalid metadata.

const encoded = outputs?.__smithersAgentCheckpointProvenance;
if (encoded === undefined) return null;
if (
  !encoded ||
  typeof encoded !== "object" ||
  encoded.version !== PROVENANCE_VERSION ||
  !Array.isArray(encoded.attempts) ||
  !Array.isArray(encoded.checkpoints)
) {
  throw new Error("Snapshot agent checkpoint provenance is corrupt");
}

packages/time-travel/src/snapshot/agentCheckpointProvenance.js:237 — minor/correctness (confirmed)

Apply MAX_SNAPSHOT_CHECKPOINT_ATTEMPT_BYTES while parsing embedded attempts. Capture rejects an attempt whose persisted text fields exceed 16 MiB, but this decoder accepts a single 16–64 MiB attempt as long as the overall encoded block remains under 64 MiB, then callers rehydrate that oversized row. The per-attempt safety bound should be symmetric across capture and restore.

const attemptTextBytes = [7, 8, 9, 11, 12, 13].reduce(
  (total, index) => total + (tuple[index] === null ? 0 : Buffer.byteLength(tuple[index], "utf8")),
  0,
);
assertWithinLimit(attemptTextBytes, MAX_SNAPSHOT_CHECKPOINT_ATTEMPT_BYTES, "attempt text size");

packages/driver/src/RunOptions.ts:65 — info/docs (confirmed)

Document that 16 MiB is also the hard ceiling, not merely the default. validateRunOptions rejects larger values, so the current public JSDoc can lead callers to believe this limit may be raised and then encounter a runtime RangeError.

/** Maximum UTF-8 JSON bytes accepted for one durable agent checkpoint (default and hard ceiling: 16 MiB; may only be lowered). */
  maxAgentCheckpointBytes?: number;

_{Generated by smithers review (Codex sol), findings adversarially verified.}