Problem
Submission payloads carry their attachment bytes inline (as 256 KiB text chunks in flue_submission_chunks, reassembled on read), and parseSubmission unconditionally rehydrates the full chunk set on every row read: getSubmission, listUnreadySubmissions, listRunnableSubmissions, listRunningSubmissions, listExpiredSubmissions, claimJoinableSubmissions (sql-agent-execution-store.ts). The Cloudflare drain's reconcile pass calls three of those lists per pass — so every pending submission's attachments are resident simultaneously before any work starts.
The severe shape is a crash loop, reported from production-style workloads: an image-heavy instance's DO wakes (armed schedule row re-fires the drain), the reconcile pass OOMs while listing hydrated submissions — before any CAS increments attemptCount or settles anything — and the isolate restarts into the identical wake. The retry budget cannot break the loop because the OOM precedes claim, and even the give-up path (failInterruptedSubmission → recordSubmissionTerminal) opens a session, which materializes the entire visible image context first (tracked separately). Every escape hatch requires the memory that is already exhausted.
Two aggravating facts:
- Bytes are stored twice, forever: admission chunks the payload into
flue_submission_chunks, then admission-side materialization writes the same bytes to the canonical attachment store — and submission chunk rows are never deleted, even after settlement.
- Nothing here is partially solved: the chunked storage exists for DO SQLite row limits, not memory.
Proposed direction
After admission-side materialization succeeds, the attachment store is the durable byte authority and the payload's inline bytes are pure duplication. Introduce a shed step at the canonical-ready seam: rewrite the stored payload so each attachment becomes a byte-free ref (marker + size/digest sidecar sufficient to rebuild the canonical AttachmentRef with zero byte loads) and delete the submission's chunk rows in the same transaction. Bytes then ride the payload only across the short admission crash window, which the existing unready-row recovery pass already re-materializes.
- One new optional method on
AgentSubmissionStore (adapters without it keep today's byte-carrying payloads — correct, heavier).
- Downstream consumers that genuinely need bytes (the waking delivery render, join paths) hydrate ref-marked attachments from the attachment store, bounded by the per-message admission cap; idempotent replay comparison matches digests against the ref sidecars.
- Payload shape + chunk lifecycle change ⇒ schema-version bump (reset-only per policy).
Result: wake-time hydration cost collapses from Σ(all pending attachments) to Σ(unshed unready rows), and the crash loop cannot form because no reconcile step needs unbounded memory.
Related: admission-side aggregate budgets (#529); model-context image materialization (companion issue).
Problem
Submission payloads carry their attachment bytes inline (as 256 KiB text chunks in
flue_submission_chunks, reassembled on read), andparseSubmissionunconditionally rehydrates the full chunk set on every row read:getSubmission,listUnreadySubmissions,listRunnableSubmissions,listRunningSubmissions,listExpiredSubmissions,claimJoinableSubmissions(sql-agent-execution-store.ts). The Cloudflare drain's reconcile pass calls three of those lists per pass — so every pending submission's attachments are resident simultaneously before any work starts.The severe shape is a crash loop, reported from production-style workloads: an image-heavy instance's DO wakes (armed schedule row re-fires the drain), the reconcile pass OOMs while listing hydrated submissions — before any CAS increments
attemptCountor settles anything — and the isolate restarts into the identical wake. The retry budget cannot break the loop because the OOM precedes claim, and even the give-up path (failInterruptedSubmission→recordSubmissionTerminal) opens a session, which materializes the entire visible image context first (tracked separately). Every escape hatch requires the memory that is already exhausted.Two aggravating facts:
flue_submission_chunks, then admission-side materialization writes the same bytes to the canonical attachment store — and submission chunk rows are never deleted, even after settlement.Proposed direction
After admission-side materialization succeeds, the attachment store is the durable byte authority and the payload's inline bytes are pure duplication. Introduce a shed step at the canonical-ready seam: rewrite the stored payload so each attachment becomes a byte-free ref (marker +
size/digestsidecar sufficient to rebuild the canonicalAttachmentRefwith zero byte loads) and delete the submission's chunk rows in the same transaction. Bytes then ride the payload only across the short admission crash window, which the existing unready-row recovery pass already re-materializes.AgentSubmissionStore(adapters without it keep today's byte-carrying payloads — correct, heavier).Result: wake-time hydration cost collapses from Σ(all pending attachments) to Σ(unshed unready rows), and the crash loop cannot form because no reconcile step needs unbounded memory.
Related: admission-side aggregate budgets (#529); model-context image materialization (companion issue).