Skip to content

Schema-version mismatch permanently bricks deployed Durable Objects — "reset-only" has no implemented reset #532

Description

@FredKSchott

Problem

Persisted stores are reset-only pre-1.0: any stored FLUE_SCHEMA_VERSION other than the runtime's is rejected at open with PersistedSchemaVersionError ("stores must be cleared"). The policy is sound, but no clear/reset operation exists anywhere — no DROP, no storage.deleteAll(), nothing — and on the primary production target there is no operator-reachable way to perform one:

  • On Cloudflare, each agent instance's store lives in its Durable Object's SQLite and is opened in the DO constructor (runtime.prepare() runs before super()). A version mismatch is therefore a constructor throw: every request to every existing instance fails forever, every alarm tick (drain wakes, user scheduleEvery automations) fails and is platform-retried forever, and no code path can clear the storage because reaching the object re-runs the failing constructor.
  • The failure isn't even loud in the right way: the typed error is swallowed twice — createSqlAgentExecutionStore re-wraps it in a plain Error, and the constructor throw crosses the DO stub untyped — landing as a generic 500. The operator sees "internal error", not "store is v7, runtime supports v8". The error's dev guidance ("point the runtime at a different database") is Node-shaped and meaningless for a DO.
  • The only documented remedy is retiring the DO class via deleted_classes — which destroys every instance of the agent (healthy ones included), kills co-located Agents-SDK state, and takes a two-deploy dance.
  • Lesser corners: Cloudflare local dev crashloops the same way after every bump (remedy: knowing to delete .wrangler/state); flue run's Flue-owned cache DB (node_modules/.cache/flue/run.db) is never wiped and the CLI prints the raw error with no remediation pointer.

Node is mostly fine today (in-memory deployed default; dev DB wiped per cold start; external DBs clearable with their own tooling; mismatch fails loudly once at boot).

The schema version was bumped several times in the current cycle alone, so this fires for anyone tracking nightlies — and it will fire for real deployments at any release boundary that crosses a schema bump.

Proposed direction (researched design available)

  1. One open-time decision point (openFlueSqlStore()): returns ready/mismatch instead of throwing, so the constructor structurally cannot crash on version state.
  2. Mismatch becomes a held, observable fault state on Cloudflare: typed 503 envelope (persisted_schema_version_unsupported, stored/supported versions in meta) on every request/dispatch; alarm ticks log-and-complete instead of platform-retrying; user schedules keep running (Flue-touching callbacks fail loudly per-tick).
  3. Explicit, version-pinned reset consent in config — e.g. schemaReset: { fromVersions: [7] }, exact versions only, no wildcard — with each instance healing lazily at its next contact (request, dispatch, or alarm tick; the only shape that reaches scheduled-automation instances). Every reset emits an error-level structured log plus durable audit keys. Reset drops the whole flue_% table set (the version stamp covers the transcript format itself; under reset-only there is no reader for old-format data). Listing a newer stored version is the deliberate rollback story.
  4. Throwaway stores stop demanding surgery: flue run's cache auto-resets with a notice; Cloudflare local dev can treat older stores as implicitly consented (Node dev already wipes per cold start).
  5. 1.0 evolution: the opener becomes the migration ladder's front door (exact match → migration steps → consent reset → held state) — the fault state, consent shape, audit stamps, and adapter seam survive verbatim; nothing is throwaway.

Timing note: this matters most at and after the next major release — both because real users will then cross version boundaries, and because any post-release schema bump without this in place reproduces the bricked-fleet scenario at production scale.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions