feat: agent orchestration education — teach agents when to use chat_with_agent vs fan_out vs async

[vybe]

Summary

Surfaced during work on #498. Agents calling other agents have no in-band guidance on which delegation pattern to use, leading to predictable misuse (e.g., chat_with_agent parallel=true async=false for fan-out, which doesn't queue and produces terminal capacity rejections).

The platform offers three distinct patterns but documents only one of them in the agent's system prompt and undersells the trade-offs in the MCP tool descriptions.

Component

Backend / Platform Prompt Service / MCP Server (chat tool)

Priority

P2 — improves observable agent behavior platform-wide; complements the platform fix in #498.

What agents see today

src/backend/services/platform_prompt_service.py:38 mentions only:

- mcp__trinity__chat_with_agent(agent_name, message) - Delegate tasks to other agents

No mention of parallel, async, fan_out, or capacity behavior.

src/mcp-server/src/tools/chat.ts — the chat_with_agent description hints at the modes but doesn't tell agents that sync parallel mode does NOT queue at capacity. The fan_out tool exists right next to it but the chat_with_agent description doesn't pointer to it.

Proposed Changes

1. platform_prompt_service.py — Replace the Agent Collaboration subsection

Replace the single-bullet chat_with_agent mention with three named patterns:

• Conversational handoff — chat_with_agent(name, message). Sequential, queue-backed, maintains context.
• Single parallel task — chat_with_agent(name, message, parallel=true, async=true). Stateless, returns execution_id; poll for result. Capacity overflow IS absorbed.
• Fan-out workload — fan_out(name, [tasks]). Designed for N independent tasks with built-in concurrency control. Use whenever there are ≥3 independent tasks for the same agent.

Plus a short capacity note explaining when each mode queues vs fails.

2. src/mcp-server/src/tools/chat.ts — Tighten chat_with_agent description

Add a 'Capacity behavior' subsection that names the three (parallel, async) combinations and what each does at capacity. Add explicit pointer: 'For fan-out (≥3 tasks to same agent), use fan_out instead.'

3. Optional — surface fan_out in list_agents response

When an agent lists its collaborators, hint at the recommended delegation pattern based on permissions.

Acceptance Criteria

• Platform system prompt distinguishes the three orchestration patterns with capacity model.
• chat_with_agent MCP tool description states the at-capacity behavior of each mode and pointers to fan_out.
• Manual smoke test: a fresh agent's first attempt at fan-out picks fan_out or async-parallel, not sync-parallel.
• Token-cost delta on platform prompt is bounded (≤ ~30 lines added).

Out of Scope

• Platform behavior changes — see bug: synchronous parallel chat_with_agent bypasses backlog (BACKLOG-001) — fan-out callers see terminal capacity rejections instead of queueing #498 for the sync-bypasses-backlog platform fix.
• Per-agent dashboard guidance / UI surfaces.

Related

• bug: synchronous parallel chat_with_agent bypasses backlog (BACKLOG-001) — fan-out callers see terminal capacity rejections instead of queueing #498 — Platform-side fix (sync parallel calls now spill to backlog). This issue is the agent-education companion.

[vybe]

Field evidence — and a fourth pattern this issue does not yet cover

Evidence from a real orchestration workload, routed through product intake 2026-09-14. No priority or label changed by this comment.

The workload

An operator is running 40–50 research tasks through one orchestrator agent delegating to 6 research agents. That is well past the "≥3 independent tasks" threshold this issue already names as the fan_out trigger, so it is a direct test of the education gap described above.

The failure

The orchestrator delegated a task to a sub-agent inside its own container, did not wait for it, assumed the task was done, and ended its turn — at which point the sub-agent stopped too. No error surfaced; the run reported clean with no work product.

The numbers, verified on dev

The capacity note proposed in this issue should carry the real values rather than describing the model qualitatively:

• default per-agent parallelism is 3 — canary/snapshot.py: func.coalesce(agent_ownership.c.max_parallel_tasks, 3)
• there is a fleet ceiling applied on top of the stored value — _clamp_to_ceiling / clamp_to_ceiling (feat(capacity): admin-configurable system-wide ceiling + agent-settings UI for max_parallel_tasks #506), effectively capping concurrency around 10
• the binding constraint is per-container memory/CPU, so the cap is not arbitrary and raising it is not free

An operator planning 40–50 parallel tasks has no way to see either number before hitting it, and the failure it produces does not name the cap.

The fourth pattern (proposed scope addition)

The three patterns in "Proposed Changes" are all cross-agent (chat_with_agent / parallel+async / fan_out). The failure above is in-container sub-agents, which the platform prompt addresses in exactly one parenthetical, inside a section about background commands:

"(Subagents are the one exception: the process waits for agents you launch before exiting — but a command you background is not waited for.)"

That sentence is true about container lifetime and is readable as permission to fire and forget — which is how it failed here. What is missing is the model-facing rule: a sub-agent's completion must be read, not assumed. The pattern known to work today is "spin up sub-agents, then poll their state", because a turn cannot tell from the inside when a 3–5 minute child process has finished.

Proposed as a fourth bullet in the same prompt subsection, not a separate issue — this issue already owns "agent orchestration education", and a duplicate would be the wrong artifact.

Status of the working pattern

Two engineers are assigned to validate the poll-state pattern against this workload. The intent stated at assignment is that a validated pattern becomes a best practice adopted platform-wide, which makes this issue its landing site rather than a per-user convention repeated by hand.

Promotion of this issue from P2 is on the table for the next /roadmap session — proposed there, not decided here.