tracking(widget): clarification form submission failure family

[codeacme17]

tracking(widget): clarification form submission failure family

Umbrella for the customer-reported "Widget forms do not submit" incident: a visitor answers an agent's clarification/interaction form (e.g. parameters for a pending tool/MCP call), the submit fails with a generic "Failed to send response", and the task stays parked in WAITING_FOR_USER — the pending tool call never runs. Production triage found multiple independent failure legs plus adjacent defects discovered while fixing them. This issue tracks the whole family so nothing gets lost across the two repos.

Failure legs (seed issues)

• fix(files): durable storage upload failures swallow the underlying exception #1467 — fix(files): durable storage upload failures swallow the underlying exception
  • PR fix(files): log the provider cause behind durable-storage faults #1476 — log the provider cause behind durable-storage faults (base of enh(files): classify the provider fault behind durable-storage failures #1480 — must land first)
  • PR enh(files): classify the provider fault behind durable-storage failures #1480 — classify the provider fault behind durable-storage failures (contains fix(files): log the provider cause behind durable-storage faults #1476; target main only after fix(files): log the provider cause behind durable-storage faults #1476 lands)
• fix(frontend): clarification form gives up on transient upload failures and hides the real rejection reason #1468 — fix(frontend): clarification form gives up on transient upload failures and hides the real rejection reason
  • PR fix(frontend): retry transient clarification uploads and surface the real failure reason #1472 — retry transient clarification uploads and surface the real failure reason (renders ack text verbatim — needs security(websocket): redact incidental exception text from chat clients #1514 first)
• fix(web): form-response command fails permanently after ~19s when the resume reservation is held #1469 — fix(web): form-response command fails permanently after ~19s when the resume reservation is held
  • PR fix(web): defer a form-response command when the resume slot is held #1484 — defer a form-response command when the resume slot is held
  • fix(web): a contended durable RESUME command is persisted as completed even though nothing resumed #1499 — the same lie on the sibling branch: a contended durable RESUME is persisted as completed (same code path as fix(web): defer a form-response command when the resume slot is held #1484)
• security(websocket): agent RuntimeError text is surfaced verbatim to public chat visitors #1479 — security(websocket): agent RuntimeError text is surfaced verbatim to public chat visitors
  • security(websocket): raw exception text still reaches chat clients through unrecognized producer shapes #1497 — residual: raw exception text still reaches clients through unrecognized producer shapes
  • PR security(websocket): redact incidental exception text from chat clients #1514 — redact incidental exception text from chat clients (split out of fix(frontend): retry transient clarification uploads and surface the real failure reason #1472; the redaction contract both other legs depend on)
• xorbitsai/xagent-saas#680 — feat(session-ws): widget session WS rejects WAITING_FOR_USER form responses as BUSY
  • PR xorbitsai/xagent-saas#686 — classify waiting-for-user turn refusals distinctly (taxonomy split-out) — merged
  • PR xorbitsai/xagent-saas#713 — let a continuation record its delivery under the caller's id (shared-code fix behind fix: scope custom API tools to agent config, not all user APIs #687's critical finding; fix: scope custom API tools to agent config, not all user APIs #687 rebases on it)
  • PR xorbitsai/xagent-saas#687 — route parked-task chat frames to the interaction continuation
• xorbitsai/xagent-saas#651 — fix(widget-session): the widget can show an ask-user question but has no frame to answer it — same customer incident, second framing; see "Open question" below
  • PR xorbitsai/xagent-saas#658 — closed, split into the stack below (4.5k lines)
  • PR xorbitsai/xagent-saas#699 — bound the input-continuation turn id to the delivery column (APPROVED, conflicted — root of the stack)
  • PR xorbitsai/xagent-saas#700 — hold agent ownership through the input-continuation dispatch
  • PR xorbitsai/xagent-saas#712 — make one logical answer claimable exactly once
  • PR xorbitsai/xagent-saas#701 — closed; still the base of feat: add conversation logs API and frontend integration #702/ref(kb): H05.1 move collection delete/rename data-plane into handle #703, which need re-basing
  • PR xorbitsai/xagent-saas#702 — read answer receipts on the transport (draft)
  • PR xorbitsai/xagent-saas#703 — typed answer identity on the chat frame (draft)

Derived while fixing (spawned issues)

• fix(frontend): taskless Session send drops message attachments silently instead of rejecting them #1470 — fix(frontend): taskless Session send drops message attachments silently instead of rejecting them (P0 — silent data loss; no PR yet)
• fix(web): a submitted clarification form stays disabled when its turn does not run #1500 — fix(web): a submitted clarification form stays disabled when its turn does not run (no PR yet; blocks the "Done means" below even after the legs land)
• enh(frontend): give transport.uploadFiles a cancellation hook so abandoned chat uploads stop #1471 — enh(frontend): give transport.uploadFiles a cancellation hook so abandoned chat uploads stop
• enh(frontend): give ClarificationForm's onSend the same delivery contract as the internal path #1485 — enh(frontend): give ClarificationForm's onSend the same delivery contract as the internal path
• fix(files): delete_file maps permanent storage-scope faults to a retryable 503 #1473 — fix(files): delete_file maps permanent storage-scope faults to a retryable 503 (unassigned)
• fix(triggers): trigger run is stamped finished_at but left running when the task is neither completed nor failed #1482 — fix(triggers): trigger run is stamped finished_at but left running when the task is neither completed nor failed
• xorbitsai/xagent-saas#685 — fix(external): acceptance classifier reports unrecognized V1 error codes as UNAUTHORIZED, closing widget sockets as revoked (unassigned)

Split out of the durable-storage reviews (#1476 / #1480)

• fix(files): delete_durable never wraps provider faults, so deletion failures escape unclassified #1489 — delete_durable never wraps provider faults, so deletion failures escape unclassified
• enh(files): route the remaining durable-fault absorbers through provider classification #1515 — route the remaining durable-fault absorbers through provider classification
• fix(files): provider-fault classifier hardening residuals (chain bound, BaseException escapes, code whitespace) #1517 — provider-fault classifier hardening residuals (chain bound, BaseException escapes, code whitespace)
• security(kb): StorageKeyScopeError discloses two user ids to the model through the file-ingestion tool result #1518 — security(kb): StorageKeyScopeError discloses two user ids to the model through the file-ingestion tool result (P1, security)
• fix(web): the durable-fault field sanitiser misses eight line-break characters its own test predicate honours #1519 — the durable-fault field sanitiser misses eight line-break characters its own test predicate honours (P2, security)
• fix(web): the durable-fault logging contract is enforced by weak proxies (kwargs collision, unsanitised label, AST-parsed closed set) #1520 — the durable-fault logging contract is enforced by weak proxies (P2)
• ref(web): decide where durable-storage 503 translation belongs — ASGI boundary handler or per call site #1521 — decide where durable-storage 503 translation belongs — ASGI boundary handler or per call site (P2)
• fix(web): log records are forgeable through exception text, not just rendered fields #1516 — log records are forgeable through exception text, not just rendered fields (app-wide, out of enh(files): classify the provider fault behind durable-storage failures #1480's scope)

Split out of the WebSocket redaction work (#1479 / #1497)

• test(ws): make the control-char filter drift guard shape-agnostic and cover all five copies #1490 — test(ws): make the control-char filter drift guard shape-agnostic and cover all five copies
• PR perf(ws): replace per-character control-char stripping with str.translate #1481 — replace per-character control-char stripping with str.translate — merged

Split out of the saas continuation work (saas#651 / saas#680)

• xorbitsai/xagent-saas#662 — typed answer identity cutover for the session WebSocket (PR ref(kb): H05.1 move collection delete/rename data-plane into handle #703)
• xorbitsai/xagent-saas#663 — bound the input-continuation runtime handoff and reconcile ambiguous outcomes
• xorbitsai/xagent-saas#675 — make pending-question consumption atomic with input dispatch (PR fix(kb): recover parse method from parser column for candidate semantic_id #712)
• xorbitsai/xagent-saas#682 — recover input-continuation claims stranded by a worker crash
• xorbitsai/xagent-saas#678 — sequence numbers are allocated before envelope serialization can fail

Test/CI debt this family created

• PR test(web): stop flaky one-slot QueuePool contention budgets #1492 — stop flaky one-slot QueuePool contention budgets (the fix(web): form-response command fails permanently after ~19s when the resume reservation is held #1469 leg's own tests)
• PR xorbitsai/xagent-saas#709 — wait for the turn to settle before writing from the test thread

Adjacent, deliberately not in this family

• fix: bound durable upload concurrency #1464 / enh: enforce aggregate upload intake capacity #1478 — durable upload concurrency and aggregate intake capacity. Same subsystem, different incident (capacity, not error surfacing). Tracked separately so this family stays closable.

Cross-PR contracts (agreed so parallel work does not conflict)

1. Backend message_rejected ack message field is always user-presentable text; the frontend renders it verbatim (fix(frontend): clarification form gives up on transient upload failures and hides the real rejection reason #1468 ↔ fix(web): form-response command fails permanently after ~19s when the resume reservation is held #1469 ↔ saas#680). Security caveat: security(websocket): agent RuntimeError text is surfaced verbatim to public chat visitors #1479/security(websocket): raw exception text still reaches chat clients through unrecognized producer shapes #1497 define what must NOT flow into that field — the redaction (PR security(websocket): redact incidental exception text from chat clients #1514) must land before or with fix(frontend): retry transient clarification uploads and surface the real failure reason #1472, or the frontend change turns a storage fault into a tenant-internals disclosure.
2. The session transport gains its own distinct waiting-for-user refusal reason (saas#686, merged) so the frontend surfacing from fix(frontend): clarification form gives up on transient upload failures and hides the real rejection reason #1468 works unchanged on both transports.
3. Delivery identity is caller-supplied (delivery_turn_id, saas#713) rather than derived, so a client-chosen dedup key probes the id space its delivery was recorded in. Every continuation caller must adopt it; a caller that keeps the derived id re-delivers on ack loss.

Landing order (updated 2026-08-20)

Every open PR in this family is either conflicted, un-reviewed, or the base of another PR, so the order matters more than the count.

Wave 0 — unblock the trains. saas#699 (approved, needs a rebase; root of a 5-PR stack) · saas#713 (clean, needs review; #687 waits on it) · #1476 (base of #1480) · #1514 (needs human review; contract dependency for #1472).

Wave 1 — the legs, in dependency order. #1480 (after #1476) · #1472 (after #1514) · #1484 (fold in or immediately follow #1499) · saas#687 (after #713, rebased and shrunk to routing) · saas#700 → saas#712 (after #699).

Wave 2 — P0 items with no PR yet. #1470 · #1500 · and a disposition for #1479 (does #1514 close it, or does it stay open for the residual?).

Wave 3 — P1. #1482 · #1485 · saas#685 · saas#663 · saas#675 · saas#682 · #1518 · plus the CI debt (#1492, saas#709) so the legs' own tests stop flaking.

Wave 4 — P2 hygiene. #1471 · #1473 · #1489 · #1515 · #1516 · #1517 · #1519 · #1520 · #1521 · #1490.

Open question: saas#651 vs saas#680

Both issues are the same customer report ("Failed to send response" in a session-mode widget) approached from opposite ends, and both now have open PRs that add parked-task routing to session_websocket.py (#687 from the #680 side; #702/#703 from the #651 side) and change answer/delivery identity in external_tasks.py (#713 vs #712). They will conflict textually and may overlap semantically. Decide one owner for the routing change and one for answer identity before any of them merges, and close the loser as superseded. Secondary cleanup: #701 is closed but is still the git base of #702 and #703.

Non-code follow-ups (ops, tracked here so they are not lost)

• Investigate the S3-side cause of the intermittent durable-storage write failures once fix(files): durable storage upload failures swallow the underlying exception #1467's logging lands (infra dashboard/metrics work, internal). Related: PR fix(storage): use standard S3 retries #1494 (standard S3 retries) has merged.
• Nudge affected embed conversations stuck in WAITING_FOR_USER (support follow-up, internal).

Related / upstream

• hitl epic: feat(interactions): add canonical task interaction schema #1074-feat(frontend): submit structured clarification responses #1081 (native task interactions). The fixes above harden the legacy chat-message path production widgets use today; the session-WS continuation (saas#680, saas#651) should converge with feat(api): expose typed task interaction continuation #1079's typed continuation API when it lands.

Done means

A form submission on both widget transports either delivers (with retries absorbing transient storage faults and reservation contention) or fails with a specific, actionable, non-sensitive message — and never strands a task in WAITING_FOR_USER silently. The form must also become re-answerable when its turn does not run (#1500), and an answer must never be delivered twice or against a superseded question (saas#712, saas#713).

[codeacme17]

Priority triage (criterion: does the incident stay open without it?):

P0 — required to close the incident: #1467, #1468, #1469, xorbitsai/xagent-saas#680 (the four failure legs), #1470 (silent attachment loss), #1479 (error-text disclosure; also constrains the ack-passthrough contract).

P1 — real defects, ship after the legs: #1482 (trigger runs left running), #1485 (delivery-identity parity — matters more once #1468's retries land), xorbitsai/xagent-saas#685 (unrecognized codes close sockets as revoked; the waiting-for-user case is already split out in saas PR #686).

P2 — deferred hygiene: #1471 (upload cancellation hook; downgraded from P0 — no user-visible breakage), #1473 (delete_file status-code mapping on a cleanup path).

[codeacme17]

Refreshed the body against both repos (2026-08-20). What changed since the initial triage:

Newly linked, in-family

• The whole xagent-saas#651 line was missing: it is the same customer report as saas#680 from the other end, and it now carries a five-PR stack (feat: concurrency-safe tool coverage #699 → chore: bump compose images to 0.5.1 #700 → fix(kb): recover parse method from parser column for candidate semantic_id #712, plus drafts feat: add conversation logs API and frontend integration #702/ref(kb): H05.1 move collection delete/rename data-plane into handle #703) after ref(kb): H03 move embedding lifecycle into handle #658 was split. Spawned issues saas#662, ref: batch agent model lookups into a single query #663, feat(cache): PromptSegment contract and canonical hashing #675, feat(cache): checkpoint and child context compatibility #678, feat(cache): provider-neutral cache capability model #682 came out of it.
• saas#713 — split out of fix: scope custom API tools to agent config, not all user APIs #687 to carry the shared-code fix behind that PR's critical id-space finding. fix: scope custom API tools to agent config, not all user APIs #687 rebases on it.
• security(websocket): raw exception text still reaches chat clients through unrecognized producer shapes #1497 + PR security(websocket): redact incidental exception text from chat clients #1514 — the redaction work moved out of fix(frontend): retry transient clarification uploads and surface the real failure reason #1472 into its own PR; security(websocket): raw exception text still reaches chat clients through unrecognized producer shapes #1497 is the residual (producers the AST guard cannot see). security(websocket): agent RuntimeError text is surfaced verbatim to public chat visitors #1479 now has no PR of its own.
• fix(web): a contended durable RESUME command is persisted as completed even though nothing resumed #1499 (contended durable RESUME persisted as completed) and fix(web): a submitted clarification form stays disabled when its turn does not run #1500 (submitted form stays permanently disabled) — both from the fix(web): form-response command fails permanently after ~19s when the resume reservation is held #1469 leg, both still without a PR. fix(web): a submitted clarification form stays disabled when its turn does not run #1500 matters for "Done means": the legs can all land and the form can still be stuck until a reload.
• Durable-storage review residuals: fix(files): delete_durable never wraps provider faults, so deletion failures escape unclassified #1489, enh(files): route the remaining durable-fault absorbers through provider classification #1515, fix(web): log records are forgeable through exception text, not just rendered fields #1516, fix(files): provider-fault classifier hardening residuals (chain bound, BaseException escapes, code whitespace) #1517, security(kb): StorageKeyScopeError discloses two user ids to the model through the file-ingestion tool result #1518, fix(web): the durable-fault field sanitiser misses eight line-break characters its own test predicate honours #1519, fix(web): the durable-fault logging contract is enforced by weak proxies (kwargs collision, unsanitised label, AST-parsed closed set) #1520, ref(web): decide where durable-storage 503 translation belongs — ASGI boundary handler or per call site #1521.
• Test/CI debt this family created: PR test(web): stop flaky one-slot QueuePool contention budgets #1492, test: test_cancel_after_committed_create_revokes_the_undelivered_exact_key flakes on a 2s cross-thread wait #1488, saas#709, test(ws): make the control-char filter drift guard shape-agnostic and cover all five copies #1490.
• Merged and closed out: saas#686, perf(ws): replace per-character control-char stripping with str.translate #1481, fix(storage): use standard S3 retries #1494.
• Marked adjacent-but-out-of-family: fix: bound durable upload concurrency #1464 / enh: enforce aggregate upload intake capacity #1478 (upload capacity, a different incident in the same subsystem).

The actual bottleneck is review depth, not PR count

PR	Leg	Size	State	Unresolved threads
#1476	#1467	+1071/-26	CHANGES_REQUESTED, round 4	8 (one carried forward, NOT FIXED)
#1480	#1467	+2202/-27 (incl. #1476)	CHANGES_REQUESTED	11 (4 major)
#1472	#1468	+1780/-61	CHANGES_REQUESTED	9 (2 major)
#1484	#1469	+1265/-56	CHANGES_REQUESTED	5 (3 major)
#1514	#1479/#1497	+384/-29	REVIEW_REQUIRED	8 (no human review yet)
saas#687	saas#680	+653/-81	CHANGES_REQUESTED	13 (1 critical)
saas#713	saas#680	+276/-7	REVIEW_REQUIRED	0
saas#699	saas#651	+399/-24	APPROVED, conflicted	0
saas#700	saas#651	+230/-4	needs review	1
saas#712	saas#651	+799/-16	needs review	1

Two of these (#1480 at 2202 lines, saas#702 at 2409) are over the 2000-line review budget, and #1480's base is main rather than #1476's branch, so its diff double-counts its own base.

Blocking decision before anything merges: the saas#651 vs saas#680 overlap in the "Open question" section above. #687 and #702/#703 both add parked-task routing to session_websocket.py; #713 and #712 both change answer/delivery identity in external_tasks.py. One owner each, and the loser closed as superseded.