Async operations need poison quarantine / degraded queue health

[ai-ag2026]

Summary

During a post-crash Hindsight recovery incident, /health continued to report healthy while async worker lanes were degraded by deterministic poison operations.

The immediate data-shape bug is tracked separately in #1669 and a narrow validation fix is open in #1670. This issue is the operational follow-up: deterministic async failures should be quarantined or surfaced as degraded health/status instead of requiring SQL + journal inspection.

Observed failure classes from the incident:

asyncpg.exceptions.DataError: different vector dimensions 384 and 0
index "idx_memory_links_to_type_weight" contains unexpected zero page at block 33008
HINT:  Please REINDEX it.

What happened

After an unclean VM shutdown and WebUI session recovery, Hindsight had derived async work stuck/failing in the background queue. The REST API stayed healthy:

{"status":"healthy","database":"connected"}

But worker logs and async_operations showed repeated deterministic failures and pending backlog. In the local cleanup pass, we had to manually:

• cancel active poison operations with different vector dimensions 384 and 0;
• reindex a crash-damaged PostgreSQL index;
• cancel batch_retain parent operations with task_payload is null that remained as pending queue work;
• restart the worker and watch WORKER_STATS until global: pending=0.

Final local state after manual cleanup:

pending/processing async_operations: 0
active vector/zero-page errors: 0
WORKER_STATS ... global: pending=0 ... my_active: none

Impact

• /health can report healthy while retain/consolidation work is functionally degraded.
• Deterministic failures can retry until max retries or keep reappearing as queue work.
• Operators cannot distinguish normal backlog from poison backlog without SQL and journal access.
• Null-payload parent operations can make the queue look pending even when there is no executable payload.
• The remediation path is operationally risky because it requires direct DB inspection/mutation.

Suggested behavior

Add an explicit async-operation quarantine/degraded-status path:

1. Classify deterministic/non-transient failures after N retries or by known error classes:
   • invalid embedding/vector dimension;
   • schema/data-shape errors;
   • invalid/null payload operations;
   • corruption/index errors that require operator intervention;
   • oversized single-item payloads when splitting cannot reduce them.
2. Move those operations out of the active retry lane as quarantined or failed_permanent.
3. Store safe metadata, not retained content:
   • operation id;
   • operation type;
   • error class/signature;
   • retry count;
   • payload byte size / payload-null flag;
   • created/updated timestamps.
4. Expose queue degradation via /health or a dedicated status endpoint:
   • active poison/quarantined operation count;
   • oldest retrying operation age;
   • recent deterministic failure classes;
   • pending operations by type/status/payload-null;
   • whether reserved worker lanes are blocked.
5. Keep privacy intact: status output should never expose retained text or payload contents.

Regression test ideas

• A retain/consolidation operation that repeatedly raises invalid vector dimension is moved out of active retry flow after the configured threshold.
• A null-payload parent batch_retain cannot remain indefinitely as normal pending executable work.
• /health or /status reports degraded queue state without leaking payload contents.
• Transient LLM/network failures still use ordinary retry/backoff.
• Quarantined operations do not block unrelated later retain operations.

Relationship to existing work

• Complements batch_retain can pass zero-length embeddings to pgvector (vector(384) vs vector(0)) #1669: zero-length embeddings reaching pgvector.
• Complements fix: validate embedding dimensions before pgvector writes #1670: validates retain embedding dimensions before pgvector writes.
• Related to Sub-batch splitter announces 10K-token chunks but processes full payload as 1/1, triggering OOM on large single-item retains #1571: oversized single retain items can also become worker poison.

The key point: validation fixes individual poison sources; quarantine/status prevents the next deterministic poison source from silently degrading the worker lane.

[ai-ag2026]

Additional crash-recovery evidence from a local deployment:

• /health continued to report healthy while async operations were retrying poisoned work.
• After cancelling existing poisoned operations, consolidation was able to create fresh different vector dimensions 384 and 0 failures in the observation-create path.
• Null-payload batch_retain parent operations also accumulated while child retain operations were still active.
• Temporary containment required disabling consolidation processing; endpoint health alone did not reflect the degraded async-worker state.

This strengthens the need for queue health/degraded reporting and poison quarantine in addition to the embedding validation fix in #1670.

[ai-ag2026]

Implementation PR opened: #1674

It adds worker-side quarantine for pending async operations with task_payload=NULL that cannot ever be claimed, while preserving legitimate batch_retain parent rows that still have child operations. This handles the orphaned/null-payload parent shape from this issue separately from the embedding-dimension validation in #1670.

[ai-ag2026]

Follow-up from local queue repair: deterministic retain poison should not retry

Additional metadata-only evidence from a second local repair pass:

• fix: validate embedding dimensions before pgvector writes #1670 covers the vector-dimension validation boundary before pgvector writes.
• fix(worker): quarantine invalid async operations #1674 covers pending task_payload=NULL operations that have no child references.
• The remaining live shape was slightly different: batch_retain parent rows had children, and one child had already hit different vector dimensions 384 and 0 while the rest were still pending. The parent staying pending was formally consistent because children were not terminal yet; the operational problem was that the child failure was deterministic but still went through retry/backoff.

Current code already has a useful precedent in tests/test_integrity_violation_not_retried.py: deterministic DB integrity errors are treated as non-retryable because retrying them only burns worker capacity. The vector-dimension failure appears to need the same treatment after #1670 turns it into a controlled validation error.

Suggested follow-up scope:

1. Introduce/reuse a typed deterministic task failure, e.g. NonRetryableTaskError / PermanentTaskError.
2. Have invalid embedding dimension / empty embedding validation raise that class, or classify it in MemoryEngine.execute_task().
3. Mark the operation failed immediately instead of raising RetryTaskAt for 3 x 60s retries.
4. Preserve ordinary retry behavior for transient LLM/network/backend errors.
5. Let existing parent propagation handle batch_retain parent finalization once children are terminal.

Regression tests that would pin the incident shape:

• invalid embedding dimension error does not raise RetryTaskAt and marks the child operation failed;
• if that child is the last/only child, the parent batch_retain becomes failed through existing propagation;
• a generic transient embedding/backend error still retries.

Privacy note: I have only metadata rows and sanitized error classes from the local incident; no retained text or payload content should be needed for the tests.

[nicoloboschi]

Quick question before we land #1674 — trying to figure out whether the null-payload rows you cleaned up manually are legacy data or a live bug.

The known code paths that could create batch_retain parents (or any other rows) with task_payload IS NULL were all closed before May 20:

• Fix non-atomic async operation creation #619 (Mar 19) — atomic INSERT in _submit_async_operation
• fix: files/retain upload problems and orphaned retains #1091 (Apr 16) — same fix for files/retain
• fix(worker): make submit_task idempotent when payload already set #1097 (Apr 16) — idempotent submit_task UPDATE
• fix(async-ops): atomically commit batch_retain parent and child rows #1343 (Apr 30) — atomic parent+child INSERT in submit_async_retain

Reading the current code I can't see a path that still produces these. So either your install was first deployed on a pre-April binary and carries leftover rows from then, or there's a hole we haven't found yet.

Two things that would help nail this down:

1. When was the affected install first deployed? Specifically, did it ever run a version older than fix: files/retain upload problems and orphaned retains #1091 (Apr 16, 2026)?
2. If you still have access: do the stuck rows' created_at timestamps predate the upgrade to the post-fix(async-ops): atomically commit batch_retain parent and child rows #1343 binary?

If it's legacy data, we'd ship the cleanup as a one-shot Alembic migration rather than a recurring sweep in the worker poll loop — same effect, zero ongoing cost, and the row shape can't recur. If it's a live bug we want to find it before papering over with a sweep. Either way the broader poison-quarantine + health story you outlined here is its own follow-up.