fix(mcp): fan_out answers with a fan_out_id, and a batch can be polled (#2670)

[dolho]

Closes the third and last route of the #914 gateway-timeout class.

Stacked on #2675 (#2661). This branches off feature/2661-task-queued-timeout-receipt — it reuses that PR's attribution rules, its bounded recovery read, and its 409 reader, and AC 5 asks it to drop a note that PR added. Merge #2675 first and this diff reduces to the files below.

The defect

client.ts::fanOut() carried the identical unbounded ceiling #2661 removed from sync /task:

const timeout = (options?.timeout_seconds ?? 7200) + 60;

The MCP client's own 30–60s gateway timeout kills the JSON-RPC call long before that, so fan_out surfaced a bare fetch failed while every dispatched task kept running. It is the route that hits this most reliably: a fan-out dispatches N tasks and by construction runs longer than any single one of them.

Why the #914/#2661 receipt did not transfer

The id. Those receipts name one execution_id. A fan-out creates N rows sharing one fan_out_id, so a single execution id could only ever name an arbitrary member of the batch. The receipt is {status: "fan_out_timeout", agent, fan_out_id, execution_ids, task_count, message} — and execution_ids is evidence, not a manifest, since a slot-starved subtask has no row yet.

The ambiguity rule. #2661 refuses when more than one row survives, because on /task it cannot tell which row is the caller's. Here N survivors is the expected shape — all of them are the caller's — so the unit that must be unambiguous is the batch: pickRecentFanOut returns nothing when more than one distinct fan_out_id survives. Same rule ("a wrong id is worse than none"), measured on the right thing.

The status filter. /chat and /task require a non-terminal row, because a terminal one is evidence the receipt is unnecessary. By abort time a fan-out is normally a mix — some subtasks finished, others running — so requiring non-terminal rows would drop exactly the batches furthest along. The derived window (timeout + 10s, never a constant — #2661's rule) is what bounds staleness.

The polling surface

GET /api/agents/{name}/fan-out/{fan_out_id} folds schedule_executions, where fan_out_id has been stamped on every subtask since FANOUT-001.

status	means
running	any subtask can still change — outranks every verdict, because reporting one early is what makes a polling caller stop polling
completed	all succeeded
partial	some did. Best-effort is the fan-out's default policy, so this is a normal outcome
failed	none did

Per-task status is the execution status verbatim (queued/running/success/…), not the dispatch response's two-value pair — a live batch has to distinguish "waiting for a slot" from "running", and two values force a healthy queued subtask to be reported as a failure. deadline_exceeded is absent by construction: it is the dispatcher's verdict on its own outer deadline, not a property of any row.

Enumeration-safe (Invariant #8): malformed, unknown, and belonging-to-another-agent are one uniform 404.

It does not read the idempotency snapshot — and this is why #2670 does not collapse into #2671

The issue notes that routers/fan_out.py:140 already stores the entire aggregate under the call's key, so a read-only (scope, key) probe would give fan_out a receipt "for free". I checked that first, as suggested, and it does not hold: complete() runs only once the batch has finished, so the snapshot cannot answer the question a timed-out caller is actually asking — what is happening right now — which is the state every batch that issues a receipt is in. #2671 remains worth doing and would retire the executions scan; it would not have replaced this read surface.

Backend contracts the client depends on

• The in-flight 409 matches /chat and /task — {error, message, execution_id} instead of a bare string, so the bug(mcp): the sync /task route has no queued_timeout receipt — the MCP gateway kills the held fetch and the caller sees 'fetch failed' while the target keeps running (#914, second route) #2661 client that reads that field can turn API error (409) into a pollable receipt.
• The batch id is attached when it is minted, via FanOutService.execute(on_started=…), not at complete(). Attaching at the end records the id exactly when nobody needs it any more: the window in which a concurrent duplicate arrives, and in which this call's own gateway gives up, is the whole run. Best-effort — bookkeeping must not be able to fail a dispatch that is otherwise fine.

Acceptance criteria

• sync fanOut() bounded by MCP_CHAT_TIMEOUT_MS
• structured receipt carrying fan_out_id (+ the execution_ids known at abort)
• a polling surface resolves it to the batch's aggregate status/results
• the in-flight 409 returns the same {error, execution_id} shape as /chat and /task
• the fan_out description states the contract; mcp-orchestration.md drops the "still unfixed" note and gains route three

Tests

13 TS cases on the matcher: N rows is the normal shape and not ambiguity, two distinct batches refuses, a part-finished batch is still found, rows from other routes and rows with no fan_out_id are ignored, key scoping, the messages discriminator, the derived window, and the per-call-site trigger set (fan_out is in neither /chat's nor /task's set).

26 backend cases: the fold (running outranks every verdict; pending_retry and queued are live, not failed; cancelled/skipped are failures, not successes; deadline_exceeded unreachable; the fold is pure), the read surface (7 malformed-id shapes are the same 404 as an unknown one and never reach the DB; an unknown batch is 404, not an empty "completed" aggregate; the read is scoped by agent as well as by id), and the two contracts above.

354 MCP tests, 15072 backend tests pass. The full-suite run shows 15 failures in DNS-dependent SSRF tests (test_ent399_ipv6_origin, test_mcp_validator::TestCgnatSsrfGuard, …) — these fail identically on the base branch with this change stashed, i.e. sandbox DNS, not this PR.

Live harness: npx tsx src/mcp-server/scripts/verify_914.ts <agent> fanout (three tasks, so it exercises batch identification rather than a single row).

Fixes #2670

🤖 Generated with Claude Code

https://claude.ai/code/session_01RSjVEjay9ztC1oDXXkh9uN

[dolho]

Self re-review. Nothing further from me. Two things I went back and checked because they are where a silent truncation or a wrong count would hide.

• The 200-row cap cannot truncate a real batch. MAX_TASKS = 50 bounds a fan-out at creation, so the cap is 4× the largest legal batch and exists only to keep an unbounded SELECT off the read path. It is deliberately not exact — a re-queued subtask (redelivery_count, RETRY-001) can add a row under the same fan_out_id — which is why the docstring says "a belt" rather than a bound, and why the aggregate counts rows rather than comparing against a declared total. Had it compared, a re-queue would have made a healthy batch read as over-count.
• running genuinely outranks every verdict. _NON_TERMINAL covers queued, running and pending_retry, and the last is the one that would have been easy to drop: a subtask awaiting a feat(scheduler): retry mechanism for failed scheduled executions #271 retry is neither done nor lost, and counting it as failed reports a batch finished while a row is about to run again — i.e. it tells a polling caller to stop polling. There is a test on exactly that status.

The finding I would keep from building it is the one in the PR body: the issue suggested this might collapse into #2671's idempotency probe, and it does not — complete() writes the snapshot only once the batch has finished, so the snapshot cannot describe a batch that is still running, which is the state every batch issuing a receipt is in. Worth carrying into #2671 so that one is not scoped as a replacement for this read surface.

Merge after #2675 (#2661) — this reuses its attribution rules, its bounded recovery read and its 409 reader, and drops the "still unfixed" note it added.

[dolho]

Deeper self re-review. No defect found; four things checked rather than assumed, and one inherited assumption worth naming.

• The 409 receipt actually has an id to carry. It depends on a chain I had not verified end to end: on_started → attach_execution (which early-returns on decision.replay or a falsy id, neither true for the first caller) → the second caller's begin() returning execution_id=res.get("execution_id") on the in-flight branch. Confirmed at each link. Without the attach, the 409 would carry null and the client would fall through to API error (409) — the shape this PR exists to remove.
• The 200-row cap cannot truncate a real batch. MAX_TASKS = 50, so it is 4× the largest legal fan-out. It is deliberately not exact — a re-queued subtask (RETRY-001) adds a row under the same fan_out_id — which is why the aggregate counts rows rather than comparing against a declared total. Had it compared, a re-queue would make a healthy batch read as over-count.
• pending_retry counts as live. Easy to drop from _NON_TERMINAL, and dropping it reports a batch finished while a row is about to run again — i.e. it tells a polling caller to stop polling.
• Only two callers of fanOut(), both handling the widened union (the tool stringifies whatever comes back; the harness classifies on status). A third caller destructuring .results would now break at runtime on a receipt, which is worth knowing before one is added.

The inherited assumption. The receipt says "the batch is still running". That is the same claim #914 and #2661 make on their routes, and it rests on the backend handler surviving the client disconnect — the MCP server aborts its fetch, and whether the FastAPI coroutine continues is a property of the server, not of this change. I did not verify it for the fan-out path, and I am flagging it rather than quietly relying on it. Two mitigations already hold regardless: get_fan_out_result reads rows, so it reports whatever actually happened (cancelled / failed if the subtasks were torn down, running if not) rather than repeating the receipt's claim; and the idempotency snapshot is still written by whichever path finishes. If the assumption turns out to be false it is false for all three routes at once, and that is a finding about #914's contract, not about this PR.

Everything else from the first pass re-read clean.

[vybe]

merge-train 2026-09-10 (evening run): not on this train. This branch is stacked on #2675 (b1cd0885 / 8ccc686b are its first two commits), so it carries #2675's diff and inherits the reason #2675 was held this afternoon — two exits in chat_execution_service.py still wedge the 24h idempotency claim, and closing them is a behaviour decision. Rides once #2675 lands and this is re-based on a dev that contains it.

[dolho]

Re-merged from #2675 at 10228b395 (b9ae1a741), which closes the two chat_execution_service.py exits this branch inherited the hold for — see #2675's comment for the behaviour decision. learnings.md was the only conflict (append-only keep-both; the duplicated Tailwind entry deduped). MCP suite 363 pass, tsc clean; test_2661/test_chat_sync_backlog/test_2280 35 pass. Still stacked — rides once #2675 lands.

🤖 Generated with Claude Code

https://claude.ai/code/session_015owqMKD5QDjzZrUTF2Joht

[vybe]

merge-train 2026-09-11: on this train (after #2675, which it is stacked on), with one mechanical push to your branch — 115528af.

What I pushed and why. src/backend/routers/fan_out.py:1 # mcp: header now reads chat.ts (fan_out) + executions.ts (get_fan_out_result → …) — the router gained a second covering tool module and the header is how /validate-architecture tells "unexposed on purpose" from "forgotten" (Invariant #13). And docs/memory/feature-flows/mcp-orchestration.md:717 advertised verify_914.ts … [chat|task|both] while the script takes [chat|task|fanout|all].

Validation of the #2670 delta (not the inherited #2661 material): no criticals, CSO clean, no schema change (fan_out_id is FANOUT-001; the delta adds a SELECT), the new poll tool is on the OPERATOR_SCOPES allowlist, and I executed every link — temp SQLite → FanOutService.execute → FastAPI TestClient → stubbed-fetch client — so fan_out_id and every fold field reach an MCP caller.

One genuine design gap, non-blocking, worth its own issue rather than a fix on this PR: total/status on the poll surface are row-count-derived, and a slot-starved subtask has no row. fan_out_service.py:136 takes the semaphore before execute_task creates the row, so an outer timeout_seconds that cancels waiters before they insert leaves GET /fan-out/{id} reporting failed/partial with total < N permanently (the undispatched tasks are invisible), and with max_concurrency=1 a poll between the last rowed subtask's terminal write and the next insert can read completed, total: k and stop polling early. The receipt carries task_count; the poll model doesn't. A caveat on FanOutBatchStatus.total is the cheap mitigation; persisting the manifest is the real one.

Cheap follow-ups: nothing executes client.ts::fanOut()'s abort/409 branches, fanOutReceipt, findRecentFanOut or getFanOutResult (only pickRecentFanOut is driven) — #2675's task-receipt.test.ts already has the stubFetch/hangUntilAbort harness for exactly this; and test_2670_fan_out_receipt.py:189-215 pins the 409 shape by inspect.getsource while db.get_fan_out_executions is never executed (test_2423's temp-SQLite fixture is the pattern).

[vybe]

merge-train: one more push to your branch — the stacked-base re-merge after #2675 squashed onto dev. git merge origin/dev conflicted in seven files because dev now carries #2675 as one squash while this branch carries its original commits plus your delta on the same lines. Resolved as: merge #2675's final head af9ce7a9 first (shared ancestry — brought the #2714 learnings.md entries in cleanly), then merge dev taking this branch's version for every file where dev == af9ce7a9. Verified: the branch now differs from dev by exactly the reviewed #2670 delta (10228b39 → 115528af) — same 17 files, hunk-for-hunk identical.

[vybe]

merge-train: batch validated on train/20260911-1635 (#2729)