Skip to content

test(e2e): resolve dynamic task_id in continue_subagent spec via mock templating (#4517) #4701

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
senamakel merged 1 commit into from
Jul 8, 2026
Merged

test(e2e): resolve dynamic task_id in continue_subagent spec via mock templating (#4517) #4701

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
127 changes: 86 additions & 41 deletions app/test/e2e/specs/chat-harness-subagent-continue.spec.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,17 +7,34 @@
* 3. The harness exits early, returns [SUBAGENT_AWAITING_USER] to the
* orchestrator.
* 4. Orchestrator surfaces the question to the user.
* 5. User replies in the composer ("the main repo").
* 5. User replies in the composer.
* 6. Orchestrator calls `continue_subagent` with the user's answer.
* 7. Researcher resumes from checkpoint, produces final answer.
* 8. Orchestrator produces final synthesis with canary.
*
* Scripting: `llmKeywordRules` + mock `{{DYNAMIC_*}}` template substitution.
* A prior version used `llmForcedResponses` but that FIFO drains one entry
* per `/chat/completions` call regardless of who called; the researcher's
* own harness loop plus any ancillary summarisation/memory-prep call
* shifts responses out of order and the scripted final canary lands on
* the wrong turn or never renders (tinyhumansai/openhuman#4517). Keyword
* rules route each call by a substring of its latest user/tool message
* and are never consumed.
*
* The orchestrator's `continue_subagent` call needs a real `task_id`
* (`sub-<uuid>` generated at delegate time in
* `src/openhuman/agent_orchestration/tools/dispatch.rs`). Tests can't
* know it ahead of time, so the mock substitutes `{{DYNAMIC_TASK_ID}}` /
* `{{DYNAMIC_AGENT_ID}}` from the latest `[SUBAGENT_AWAITING_USER]`
* envelope in the message history (see
* `scripts/mock-api/routes/llm/shared.mjs::renderDynamicPlaceholders`).
*
* Verifies:
* - The tool timeline shows a subagent entry.
* - The final canary text renders in the DOM.
* - The mock LLM received ≥4 POST requests (orchestrator initial +
* researcher initial + orchestrator relay + researcher resumed +
* orchestrator final).
* researcher initial + orchestrator relay + orchestrator continue +
* researcher resumed + orchestrator final).
* - Persisted thread JSONL contains the final canary text.
*/
import { waitForApp } from '../helpers/app-helpers';
Expand All @@ -37,33 +54,66 @@ import { navigateViaHash } from '../helpers/shared-flows';
import { getRequestLog, setMockBehavior, startMockServer, stopMockServer } from '../mock-server';

const USER_ID = 'e2e-chat-harness-subagent-continue';
const PROMPT = 'Research the answer and tell me a marker phrase.';
// Per-turn probes carry disjoint distinctive tokens so `pickProbeText`'s
// substring match routes each of the six model calls to exactly one rule
// regardless of any ancillary summarisation / title-gen calls the harness may
// issue on top of the happy-path turns.
const PROMPT =
'Please delegate a research task with the chromatophore-request badge and return a marker phrase.';
const DELEGATE_PROMPT = 'Return the harlequin-token phrase after searching.';
const CLARIFICATION_QUESTION = 'Which repo should I search for the citrine-cliff badge?';
const USER_ANSWER = 'Please try the wisteria-tag project.';
const RESEARCHER_FINAL_REPLY = 'In the wisteria-tag project I found the zenith-beacon marker.';
const CANARY_FINAL = 'subagent-continue-canary-9bc3f';
const CLARIFICATION_QUESTION = 'Which repo should I search?';
const USER_ANSWER = 'the main repo';
const RESEARCHER_FINAL_REPLY = 'After searching the main repo, the answer is 42.';

// Five forced responses, popped in order by the mock LLM streamer:
// 1. Orchestrator: emits `research` tool call.
// 2. Researcher: calls `ask_user_clarification` (triggers early exit).
// 3. Orchestrator: sees [SUBAGENT_AWAITING_USER], asks user the question.
// 4. Orchestrator (after user reply): calls `continue_subagent` with user's answer.
// 5. Researcher (resumed): produces final text answer.
// 6. Orchestrator: final synthesis containing the canary.
const FORCED_RESPONSES = [
// 1. Orchestrator: delegate to researcher.

// Content-addressed keyword rules — never depleted, immune to extra
// ancillary /chat/completions calls (#4517).
//
// ORDER MATTERS: the researcher's resumed probe includes both the framing
// `[User's answer to your clarification question]` prepended by
// `src/openhuman/agent_orchestration/tools/continue_subagent.rs` AND the raw
// user answer text (`wisteria-tag`). Its rule must appear BEFORE the
// orchestrator-continue rule (keyed on `wisteria-tag`) so the researcher's
// resumed call hits the researcher rule first.
const KEYWORD_RULES = [
// Researcher's resumed turn — the harness reconstructs history and appends
// "[User's answer to your clarification question]\n<message>" as a user
// message (continue_subagent.rs). Key on the framing so this rule only
// matches the researcher, not the orchestrator's own continue turn (which
// sees the composer message without framing).
{ keyword: "user's answer to your clarification question", content: RESEARCHER_FINAL_REPLY },
// Orchestrator's final synthesis — probe is the tool result carrying the
// researcher's resumed output. `zenith-beacon` is unique to that reply.
{ keyword: 'zenith-beacon', content: `Done. The result is: ${CANARY_FINAL}` },
// Orchestrator's continue_subagent turn — user answered the clarification
// in the composer; the latest user message is USER_ANSWER. The mock
// substitutes `{{DYNAMIC_TASK_ID}}` / `{{DYNAMIC_AGENT_ID}}` from the
// preceding `[SUBAGENT_AWAITING_USER]` envelope in message history so the
// resulting tool_call carries the real runtime-generated task_id.
{
keyword: 'wisteria-tag',
content: '',
toolCalls: [
{
id: 'call_research_1',
name: 'research',
arguments: JSON.stringify({ prompt: 'Tell me a marker phrase' }),
id: 'call_continue_1',
name: 'continue_subagent',
arguments: JSON.stringify({
task_id: '{{DYNAMIC_TASK_ID}}',
agent_id: '{{DYNAMIC_AGENT_ID}}',
message: USER_ANSWER,
}),
},
],
},
// 2. Researcher: ask clarification (tool call).
// Orchestrator's relay turn — probe is the `[SUBAGENT_AWAITING_USER]`
// envelope; `citrine-cliff` is embedded in the clarification question and
// does not appear in any later probe.
{ keyword: 'citrine-cliff', content: `The researcher needs to know: ${CLARIFICATION_QUESTION}` },
// Researcher's initial turn — dispatch renders "Task:\n<DELEGATE_PROMPT>"
// (archetype_delegation.rs render_structured_handoff) so `harlequin-token`
// is unique to the researcher's incoming user message.
{
keyword: 'harlequin-token',
content: '',
toolCalls: [
{
Expand All @@ -73,27 +123,22 @@ const FORCED_RESPONSES = [
},
],
},
// 3. Orchestrator: relay the question to the user.
{ content: `The researcher needs to know: ${CLARIFICATION_QUESTION}` },
// 4. Orchestrator (after user replies): continue_subagent.
// Orchestrator's initial turn — user PROMPT. Shared with the fire-and-forget
// title-gen call (threadSlice.ts, tools: None) which sees the same probe;
// `chat_with_system` consumes `content` and ignores unexpected tool_calls,
// so a delegation-triggering rule here is safe for both callers (benign
// "Delegating to researcher." title).
{
content: '',
keyword: 'chromatophore-request',
content: 'Delegating to researcher.',
toolCalls: [
{
id: 'call_continue_1',
name: 'continue_subagent',
arguments: JSON.stringify({
task_id: '{{DYNAMIC_TASK_ID}}',
agent_id: 'researcher',
message: USER_ANSWER,
}),
id: 'call_research_1',
name: 'research',
arguments: JSON.stringify({ prompt: DELEGATE_PROMPT }),
},
],
},
// 5. Researcher (resumed): final answer.
{ content: RESEARCHER_FINAL_REPLY },
// 6. Orchestrator: final synthesis.
{ content: `Done. The result is: ${CANARY_FINAL}` },
];

interface RuntimeSnapshot {
Expand Down Expand Up @@ -132,12 +177,12 @@ describe('Chat harness — orchestrator → subagent continuation flow', () => {
await waitForApp();
await resetApp(USER_ID);

setMockBehavior('llmForcedResponses', JSON.stringify(FORCED_RESPONSES));
setMockBehavior('llmKeywordRules', JSON.stringify(KEYWORD_RULES));
setMockBehavior('llmStreamChunkDelayMs', '10');
});

after(async () => {
setMockBehavior('llmForcedResponses', '');
setMockBehavior('llmKeywordRules', '');
setMockBehavior('llmStreamChunkDelayMs', '');
await stopMockServer();
});
Expand Down Expand Up @@ -226,9 +271,9 @@ describe('Chat harness — orchestrator → subagent continuation flow', () => {
const llmHits = log.filter(
r => r.method === 'POST' && r.url.includes('/openai/v1/chat/completions')
);
// Orchestrator turn 1 + researcher turn + orchestrator turn 2 (relay question)
// + orchestrator turn 3 (continue) + researcher turn 2 + orchestrator turn 4 (synthesis)
// = 6, but accept ≥4 for robustness.
// Orchestrator turn 1 + researcher turn 1 + orchestrator relay
// + orchestrator continue + researcher resumed + orchestrator final
// = 6, but accept ≥4 for robustness against harness optimisations.
expect(llmHits.length).toBeGreaterThanOrEqual(4);
});

Expand Down
196 changes: 196 additions & 0 deletions scripts/mock-api/routes/__tests__/llm.test.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,10 @@ import assert from "node:assert/strict";

import { handleLlmCompletions } from "../llm.mjs";
import { handleIntegrations } from "../integrations.mjs";
import {
applyDynamicPlaceholdersToResponse,
renderDynamicPlaceholders,
} from "../llm/shared.mjs";
import {
listMockLlmThreads,
resetMockBehavior,
Expand Down Expand Up @@ -339,3 +343,195 @@ test("records thread state for multi-turn mock LLM sessions", () => {
assert.equal(thread.lastFamily, "summarization");
assert.equal(thread.turnCount, 1);
});

// ── {{DYNAMIC_*}} placeholder substitution (#4517) ──────────────────
//
// Envelope shape matches awaiting_user_envelope() in
// src/openhuman/agent_orchestration/tools/awaiting_user.rs — the parser
// must handle the exact production format (JSON-encoded question,
// `worker_thread_id: (none)`, trailing instruction block).
function subagentAwaitingUserEnvelope({
taskId = "sub-abc123-fake-uuid",
agentId = "researcher",
workerThreadId = "(none)",
question = "Which repo should I search?",
} = {}) {
return `[SUBAGENT_AWAITING_USER]
task_id: ${taskId}
agent_id: ${agentId}
worker_thread_id: ${workerThreadId}
question: ${JSON.stringify(question)}
[/SUBAGENT_AWAITING_USER]

The sub-agent needs clarification before it can continue. Surface the above question to the user. When the user responds, call continue_subagent with the task_id, agent_id, and the user's answer as the message parameter.`;
}

test("renderDynamicPlaceholders substitutes task_id and agent_id from history", () => {
const parsedBody = {
messages: [
{ role: "user", content: "please research the codex marker" },
{
role: "tool",
content: subagentAwaitingUserEnvelope({
taskId: "sub-runtime-42",
agentId: "researcher",
}),
},
{ role: "user", content: "the main repo" },
],
};
const rendered = renderDynamicPlaceholders(
'{"task_id":"{{DYNAMIC_TASK_ID}}","agent_id":"{{DYNAMIC_AGENT_ID}}","message":"the main repo"}',
parsedBody,
);
assert.equal(
rendered,
'{"task_id":"sub-runtime-42","agent_id":"researcher","message":"the main repo"}',
);
});

test("renderDynamicPlaceholders returns input unchanged when no envelope present", () => {
const parsedBody = {
messages: [{ role: "user", content: "no envelope here" }],
};
assert.equal(
renderDynamicPlaceholders("{{DYNAMIC_TASK_ID}}", parsedBody),
"{{DYNAMIC_TASK_ID}}",
);
});

test("renderDynamicPlaceholders short-circuits when text has no placeholders", () => {
// Guardrail: no envelope scan for content without placeholders.
assert.equal(renderDynamicPlaceholders("plain text", null), "plain text");
});

test("applyDynamicPlaceholdersToResponse renders content and toolCalls arguments without mutating input", () => {
const parsedBody = {
messages: [
{
role: "tool",
content: subagentAwaitingUserEnvelope({ taskId: "sub-xyz" }),
},
],
};
const original = {
content: "prefix {{DYNAMIC_TASK_ID}}",
toolCalls: [
{
id: "call_1",
name: "continue_subagent",
arguments: '{"task_id":"{{DYNAMIC_TASK_ID}}"}',
},
],
};
const rendered = applyDynamicPlaceholdersToResponse(original, parsedBody);
assert.equal(rendered.content, "prefix sub-xyz");
assert.equal(rendered.toolCalls[0].arguments, '{"task_id":"sub-xyz"}');
// Purity: input must not be mutated (call sites re-read the rule on the
// next request; mutation would let a first substitution leak into later
// turns that don't have an envelope).
assert.equal(original.content, "prefix {{DYNAMIC_TASK_ID}}");
assert.equal(
original.toolCalls[0].arguments,
'{"task_id":"{{DYNAMIC_TASK_ID}}"}',
);
});

test("llmKeywordRules substitute {{DYNAMIC_TASK_ID}} in continue_subagent tool_call args", () => {
setMockBehaviors(
{
llmKeywordRules: JSON.stringify([
{
keyword: "user answer",
content: "",
toolCalls: [
{
id: "call_continue_1",
name: "continue_subagent",
arguments: JSON.stringify({
task_id: "{{DYNAMIC_TASK_ID}}",
agent_id: "{{DYNAMIC_AGENT_ID}}",
message: "the main repo",
}),
},
],
},
]),
},
"replace",
);

const ctx = makeCtx({
parsedBody: {
model: "e2e-mock-model",
// No tools → not a "primary turn" for the forced-response FIFO, but
// keyword rules still fire and this validates the substitution site.
messages: [
{
role: "tool",
content: subagentAwaitingUserEnvelope({
taskId: "sub-realtime-777",
agentId: "researcher",
}),
},
{ role: "user", content: "here is my user answer" },
],
},
});

assert.equal(handleLlmCompletions(ctx), true);
const body = JSON.parse(ctx.res.body);
const args = body.choices[0].message.tool_calls[0].function.arguments;
const parsed = JSON.parse(args);
assert.equal(parsed.task_id, "sub-realtime-777");
assert.equal(parsed.agent_id, "researcher");
assert.equal(parsed.message, "the main repo");
// Placeholder text must be fully consumed.
assert.ok(!args.includes("{{DYNAMIC_"), `args should not contain unresolved placeholders: ${args}`);
});

test("llmKeywordRules leave the rule unmutated across successive requests", () => {
// Regression guard for applyDynamicPlaceholdersToResponse purity: two
// requests, only the first has an envelope in history. Second must still
// see the raw `{{DYNAMIC_TASK_ID}}` (which then falls through unchanged
// since there is no envelope to substitute from).
setMockBehaviors(
{
llmKeywordRules: JSON.stringify([
{
keyword: "answer",
content: "raw {{DYNAMIC_TASK_ID}}",
},
]),
},
"replace",
);

const first = makeCtx({
parsedBody: {
model: "e2e-mock-model",
messages: [
{
role: "tool",
content: subagentAwaitingUserEnvelope({ taskId: "sub-first" }),
},
{ role: "user", content: "my answer" },
],
},
});
assert.equal(handleLlmCompletions(first), true);
const firstBody = JSON.parse(first.res.body);
assert.equal(firstBody.choices[0].message.content, "raw sub-first");

const second = makeCtx({
parsedBody: {
model: "e2e-mock-model",
messages: [{ role: "user", content: "another answer" }],
},
});
assert.equal(handleLlmCompletions(second), true);
const secondBody = JSON.parse(second.res.body);
// No envelope → helper leaves `{{DYNAMIC_TASK_ID}}` verbatim (identity
// fast-path). Prior substitution must not have poisoned the stored rule.
assert.equal(secondBody.choices[0].message.content, "raw {{DYNAMIC_TASK_ID}}");
});
Loading
Loading