Follow-up to #554: remaining items for #511

[wyuc]

Context

#554 lands a focused director-layer fix for one of #511's three symptoms (premature END). Scope was narrowed during review and the regression eval was deferred. This tracks what's outstanding.

1. Agent-side attribution clarification (covers #511 symptoms ① and ②)

#554 only modifies the director path. The other two #511 symptoms originate in the agent path and are unchanged:

• ① Off-topic replies — agent ignores the actual content of the prior turn.
• ② Role confusion — teacher misattributes a human turn to a peer agent (the "照镜子" case).

In director-graph.ts:290, convertMessagesToOpenAI(state.messages, agentId) re-encodes peer agents as role:'user' with a [AgentName]: prefix. The human's turn is also role:'user' with [You]: (or its localised equivalent) prefix. The agent system prompt doesn't currently document this encoding, so the model has no structural signal to distinguish them.

Fix: add one or two sentences in lib/orchestration/prompt-builder.ts clarifying that role:'user' messages prefixed [Name]: are AI peers, not the human student. This was item 3 of @ashutoshrana's original plan in #511; it was dropped from #554 to keep scope tight.

2. End-to-end regression for premature-END

Deferred from #554. The unit tests there verify the director's inputs (summary labels), not its behaviour. The premature-END symptom is model-dependent — a deterministic e2e fixture that fails on main and passes with the fix is the real regression guard.

Frame as "verify #511 is actually fixed", not "add fixtures": reproduce the bug deterministically first (which models / prompts / seeds?), then encode the regression. If it doesn't reproduce reliably on any shipped model config, that's data we need before trusting the fix.

Lands under a new /eval/orchestration/ domain dir (not tests/eval/), following the outline-language pattern: runner.ts + judge.ts + scenarios/*.json + an eval:orchestration npm script.

3. Director Rule #11 wording

Rule #11 in director/system.md implicitly treats every [Student (Human)] turn as a question, so it over-blocks END when the human just acknowledges ("ok continue", "got it"). Suggested rewording: gate on "the most recent [Student (Human)] line asks a question or raises an objection AND appears after the last substantive [Agent] answer".

4. Test coverage for […]:-leading content

SENDER_PREFIX_RE = /^\[[^\]]+\]:\s*/ in conversation-summary.ts greedily strips the first […]: pattern. Content like "[FYI]: my answer is…" loses [FYI]: in the summary. Low-probability in practice; a single test pinning the chosen behaviour (strip vs preserve) would document the contract.

5. JSDoc contract on summarizeConversation

Add a line clarifying that system role messages don't appear in director-path input (convertMessagesToOpenAI filters non-user/assistant), so the 'System' label branch is defensive-only.

---

Items 1 and 2 are substantive. 3–5 are polish on #554's own work. Happy to coordinate via comments if anyone wants to bundle multiple.

[wyuc]

Working on it — starting from item 2 (premature-END e2e regression under /eval/orchestration/).

[wyuc]

Item 2 (premature-END regression eval) → #599 (draft).

Empirical finding: across 9 models × 5 #511-faithful scenarios, no scenario produced a non-zero END rate in either pre- or post-fix variant — so the prompt-layer rules in #554 do not measurably affect director behavior on any tested model. The eval is wired as a regression guard. Discussion in the PR. Items 1, 3-5 will land separately.

[wyuc]

Progress on item 2:

Two evals now land in #599 — the premature-END regression guard originally scoped, plus a new frustration-handling eval after the investigation surfaced a different shape of the problem.

The premature-END eval is a null finding: across 9 tested models × 5 scenarios × 5 samples per variant, neither pre-#554 nor post-#554 prompts produce END on any scenario. So the prompt-layer rules in #554 do not measurably change director END behavior on these inputs. The eval stays as a regression guard.

The frustration eval probes a different hypothesis: when the user expresses frustration after agents have drifted off-topic, does the director recognize the signal and route to USER or back to the teacher, or does it keep picking new peer agents for variety?

On 5 synthesized scenarios, baseline (current director prompt) picks OTHER_AGENT 0% on 4/5 cases — the bug reproduces. Adding a "Handling User Feedback" rule lifts the mean correct rate from 16% to 88% (Δ 72pp). Two scenarios don't yet pass the strict per-scenario bar; follow-ups noted in the PR.

Next steps (separate PRs): tighten rule wording so bio_dark_reaction also passes, then actually inject the rule into lib/prompts/templates/director/system.md. The agent-side system prompt likely needs a parallel "responding to user feedback" rule — its current SP has no handling for user dissatisfaction.

[ashutoshrana]

Happy to pick up items 1, 3, 4, 5 and the Rule 13 injection.

For the polish items (3–5) I'll bundle them — tighten Rule 11 to only block END when the student actually asked something (not just "got it"), add a test locking down the SENDER_PREFIX_RE strip behavior, and drop a note in the summarizeConversation JSDoc that the system branch is defensive-only since convertMessagesToOpenAI never passes system messages through.

For item 1, the agent system prompt says nothing about how the conversation history is encoded — peer agent turns show up as role:'user' with a [Name]: prefix and there's no hint of that anywhere in the template. Planning a short section explaining the peer-vs-human distinction so the model isn't guessing.

On Rule 13 — ready to inject it into director/system.md. Before I do, do you have revised wording in mind for the bio scenario, or should I iterate on the constant to push it past 70% first?

Will also add a parallel user feedback handling section to the agent system prompt since that SP has nothing on it right now.

On the status conflation — will file a separate issue.

[wyuc]

Update on #599 after a substantial revision — the PR was force-pushed today with a broader framing.

The investigation surfaced that the deeper issue isn't quite "premature END" or "frustration handling" — it's that agents drift onto adjacent topics rather than answering the user's literal question, and the director then picks peer agents for "variety" instead of routing back to the teacher. Frustration signals are just one sub-case.

Current state of #599 (5 commits):

1. premature-END regression guard (the original null-finding eval, kept as a guard)
2. director question-answering eval — 5 scenarios, 3 first-turn drift + 2 escalated frustration; baseline mean 52% correct → with_rule 100% on google:gemini-3-flash-preview
3. director rule 13 injected — "Answering the User's Question", constrained-output (literal role: teacher route by default, USER cue when ambiguous, forbid peer-agent variety routing)
4. agent SP section added — "Answering the User's Question", lead with the concrete answer to the literal question, frustration recovery as a sub-bullet
5. refactor — rip maxTurns end-to-end (server topology + client UI + settings + i18n) and distinguish empty_turns / no_done from natural completion (new error SessionStatus + AlertCircle icon)

Mapping to the items you offered:

• Items 1, 3, 4, 5: all still unclaimed — please do! These don't overlap with test+fix(orchestration): rip maxTurns; restore answer-the-user's-question behavior (#598 follow-up) #599.
• Rule 13 injection: already done in test+fix(orchestration): rip maxTurns; restore answer-the-user's-question behavior (#598 follow-up) #599 (different framing than what your original plan suggested — broader, covers first-turn drift not just frustration). Happy to discuss the wording on the PR if you'd prefer a different approach.
• Agent SP user-feedback rule: subsumed by the broader "Answering the User's Question" section in test+fix(orchestration): rip maxTurns; restore answer-the-user's-question behavior (#598 follow-up) #599.
• Status conflation as a separate issue: don't need to file — already implemented in commit 5 of test+fix(orchestration): rip maxTurns; restore answer-the-user's-question behavior (#598 follow-up) #599.

Easiest split would be: you take 1, 3, 4, 5 in your own PR(s); #599 lands the broader answering fix + the maxTurns/error cleanup. We can coordinate on the Rule 13 wording in #599 review.