bug: chat_with_agent MCP tool returns 'fetch failed' but still queues execution — silent duplicates burn budget on naive retry

[pandore]

Summary

mcp__trinity__chat_with_agent in default sync mode (parallel=false) holds the MCP gateway → backend → agent HTTP chain open for the entire agent execution. When the agent takes longer than the gateway's request timeout (~30-60s observed), the tool call returns Tool 'chat_with_agent' execution failed: fetch failed to the caller — but the request was successfully queued on Trinity and the agent IS running it.

A caller that doesn't know this will retry, queueing duplicates that Trinity then kills as concurrent conflicts. Today's example: a 3-lead dry-run on bdr-agent cost ~$2-4 of wasted compute and ~12min of agent time across 4 attempts (1 legitimate + 3 retries from fetch failed responses, 2 of which were killed mid-execution by the concurrent-duplicate guard).

This is the MCP-client surface of #408 (long-running backend→agent dispatch). The concurrent-duplicate kill is related to #912 (OrphanSweep false-kills).

Component

MCP server / Trinity gateway

Priority

P2 — has a workaround (parallel=true, async=true) but the default sync path is the obvious thing to reach for, and the silent-queue behavior is invisible until you check list_recent_executions.

Reproduction Steps

1. Call mcp__trinity__chat_with_agent against any agent with a task that takes longer than the MCP gateway's request timeout (anything enrichment-heavy: e.g., daily-lead-outreach's Step 4 Apollo+Apify pass). Default sync mode, no parallel/async flags.
2. Observe: tool returns Tool 'chat_with_agent' execution failed: fetch failed within ~30-60s.
3. Assume the request didn't land. Retry.
4. Call mcp__trinity__list_recent_executions — you'll see the original request is still running, plus your retry now queued behind it.

Observed evidence (bdr-agent, 2026-05-22)

A single intended dry-run produced 4 execution rows:

execution_id	status	started	duration	notes
fZv-iXtUXSolY1wzPO7T6w	running	12:28:29Z	ongoing (~17min+ at obs)	the legitimate one
pjayrwMmiY1WY5HQLzeiBQ	failed	12:33:45Z	520s	killed at 12:42:26Z (concurrent dup)
umrJziHm_AVsGj8-5cxPOg	failed	12:39:14Z	191s	killed at 12:42:26Z (concurrent dup, same second)
DGJvCjPH4faeNDccQy_HrA	queued	12:44:48Z	n/a	my eventual parallel=true, async=true retry

The caller (Claude Code session) got fetch failed after each of the first three calls and retried in good faith. Two of the resulting duplicates burned ~12min of agent compute before Trinity killed them simultaneously at 12:42:26Z (suspect: OrphanSweep / concurrent-claude-code-subprocess kill per #912).

Expected behavior

One of:

• (a) chat_with_agent returns an execution_id immediately even in sync mode when the gateway timeout is about to fire, so the caller can poll instead of retry. Mirrors how parallel=true, async=true works today.
• (b) chat_with_agent returns a structured error ({queued: true, execution_id: \"...\", reason: \"gateway timeout — task still running\"}) instead of the generic fetch failed, giving the caller a signal that retry is wrong.
• (c) Documented loudly on the tool description that fetch failed in default sync mode does NOT mean the request was dropped, and callers should list_recent_executions before retrying.

Workaround (in use now)

Use parallel=true, async=true for any task that might exceed the gateway timeout. Returns execution_id immediately; poll via get_execution_result. Wrote this up in the local Claude memory (memory/reference_trinity_chat_timeout.md).

Side benefit: also avoids the duplicate-queue problem entirely if used from the start.

Related

• #408 — same root cause: long-running synchronous HTTP held open across backend↔agent. The fix proposed there (refactor: consolidate ExecutionQueue + SlotService + BacklogService into CapacityManager #428 CAPACITY-CONSOLIDATE + push-completion via Redis Streams) would dissolve this issue too. This issue is the MCP-client-surface version of the same problem.
• #912 — OrphanSweep false-kills concurrent legitimate claude-code subprocesses. The two failed rows at the exact same second (12:42:26Z) look like this mechanism kicking in on the duplicates.
• #531 — drain_reader_threads silently loses final result line on long agentic tasks. Different surface (final result vs. queued retry) but same broad family (long-task plumbing fragility).

Suggested minimal fix

If a full push-completion redesign (#428) is a way off, the cheapest interim fix is (c) + (b): update chat_with_agent's gateway timeout handler to return {ok: false, queued: true, execution_id: \"...\", message: \"Task still running — poll get_execution_result/{id}\"} instead of letting fetch failed propagate. Mirrors the existing Agent at capacity; task queued. Poll GET /api/agents/{name}/executions/{id} for results. response that parallel=true, async=true already returns when the agent is busy.

Net: caller knows the request landed, has the execution_id, and won't retry. Zero behavior change for fast tasks.

🤖 Generated with Claude Code

[Keesan12]

This is a classic retry ambiguity: the caller sees failure, but the backend accepted work. The fix probably needs an idempotency key or queued-job receipt at the MCP boundary.

If chat_with_agent returns fetch failed after the gateway timeout, it should still be possible to answer: was the request accepted, what job id was created, and should the caller retry or poll?

Without that, naive retry is rational from the caller side and dangerous from the budget side. A stable request id plus accepted/queued receipt would make duplicate suppression much easier.

[Keesan12]

The dangerous part here is the invisible handoff between transport failure and queue admission. If fetch failed can still enqueue execution, I’d rather see the tool return an explicit terminal state plus a stable task id, and only admit a retry when the previous attempt is confirmed absent. That keeps duplicate burn from looking like success.