Desktop chat tool-round E2E specs fail: TinyAgents harness model-call sequence desyncs the mock forced-response queue

[senamakel]

Summary

In the desktop Release CI E2E suite, the chat "tool round → final answer" specs fail: the scripted final assistant answer (a canary string) never renders. The DOM shows the agent instead firing unscripted tool calls (analyze_image, a citation-schema subagent) that error out, because the vendored TinyAgents harness's per-turn model-call sequence no longer matches the E2E mock's one-forced-response-per-model-call queue. The queue desyncs, the scripted responses are consumed out of order, and later calls fall through to the mock's default/keyword behavior.

This is one of the failures surfaced now that Release CI runs the full desktop-E2E matrix for the first time (the lane has never been green since it was introduced in #4486). The login/auth cascade and most stale-fixture failures are addressed separately in #4511; this issue tracks the one candidate real behavior change.

Problem

What happens: app/test/e2e/specs/user-journey-full-task.spec.ts J1.2 (tool-call timeline appears) and J1.3 (final answer with canary renders) both fail. The canary canary-journey-fetch-j1k2l3 is absent from the DOM. Related tool-round specs fail the same way: chat-tool-call-flow, chat-multi-tool-round, and (intermittently) others in the chat shard.

What we expected: the test scripts exactly two forced model responses and the mock pops one per model call:

// user-journey-full-task.spec.ts
FORCED_RESPONSES = [
  { toolCalls: [ web_fetch(url=https://example.com) ] },   // model call 1
  { content: 'Here is the fetched page content: canary-journey-fetch-j1k2l3' }, // model call 2
]
setMockBehavior('llmForcedResponses', JSON.stringify(FORCED_RESPONSES))

The mock (scripts/mock-api/routes/llm.mjs:245-253) does forced.shift() once per /chat/completions call. So the test assumes the harness makes exactly 2 model calls: emit web_fetch, then emit the canary answer.

What actually rendered (ground truth from the failure DOM dump, run 28718166785, Linux chat shard, failure-User-journey-full-research-task-J1.3-*.source.xml): two assistant bubbles, neither containing the canary:

1. Tool result received. The relevant outcome is: invalid arguments for analyze_image: tool analyze_image arguments.prompt is required.
2. Expected schema: {"properties":{"citation_requirement":{"description":"Citation/evidence style the child must preserve in its result…

analyze_image and the citation-requirement subagent are not in the forced responses — they're the harness improvising once the forced queue no longer lines up with the model-call count, so the scripted web_fetch/canary pair never drives the turn. The core did produce content and render assistant bubbles (so this is not an empty-full_response / renderer bug — an earlier hypothesis that the DOM disproves); the content is just the wrong (error) content.

Impact: several desktop chat/agent E2E specs fail deterministically, blocking the Release CI Gate (Desktop E2E has no continue-on-error). Whether the underlying change is user-facing (the harness making extra/different model calls per turn) or purely a test-scripting assumption that the TinyAgents migration invalidated needs a maintainer with harness knowledge to decide.

Steps to reproduce:

1. Run the desktop E2E chat shard (or just user-journey-full-task.spec.ts) against the mock backend on Linux (Release CI Desktop E2E (full suite, 3 OS) / E2E (Linux full / chat)).
2. Observe J1.2/J1.3 fail; the canary never appears; the assistant bubbles contain analyze_image/subagent schema errors.

Environment: desktop (CEF), Linux CI runner, mock backend; branch at TinyAgents migration parity (#4451–#4502). Reproduced on fork Release CI run senamakel/openhuman 28716355551 and 28718166785.

Solution (optional)

Two candidate directions — a maintainer decision is needed on which is correct:

1. If the harness legitimately makes more/different model calls per turn now (e.g. an added planning/summarization call, or subagent orchestration): update the E2E forced-response scripting to match the new sequence (either add the intervening forced responses, or switch these specs to keyword-rule scripting keyed on the tool/message so ordering is robust to extra calls). This is a test-fixture fix.
2. If the harness is over-calling / improvising when it shouldn't (calling analyze_image with no prompt, spawning a citation subagent for a simple fetch task): that's a genuine harness regression from the migration and should be fixed in src/openhuman/agent/harness/ / the tinyagents integration, with a Rust-level assertion on the model-call sequence for a scripted tool-round turn.

Diagnostic to disambiguate on one run: log the number of /chat/completions calls the mock serves per turn (mock side) and the tool names the harness selects; compare against the 2 the specs assume.

Acceptance criteria

• Repro gone — the chat tool-round specs (user-journey-full-task J1.2/J1.3, chat-tool-call-flow, chat-multi-tool-round) render the scripted canary and pass on the Linux chat shard, or the harness behavior is confirmed intentional and the mock scripting is updated to match.
• Root cause documented — decision recorded on whether the extra/different model calls are intended harness behavior or a migration regression.
• Regression safety — a Rust unit/integration assertion on the model-call sequence for a scripted tool-round turn, and/or the E2E specs made robust to the harness's real call pattern.
• Diff coverage ≥ 80% on the fix PR.

Related

• fix(ci): unblock release lane — Rust E0063, macOS E0463, and E2E login-consume fixtures #4511 — unblocks the release lane (Rust E0063, macOS E0463, auth-consume E2E fixtures) and clears the stale-fixture / under-budgeted-hook E2E failures; commerce shard green, foundation 5→1, chat 14→4, webhooks 8→5.
• feat(ci): two-branch CI — changed-files-only PR lane, full-suite release lane, release-branch cuts, cancellation watchdog #4486 — introduced Release CI (full desktop-E2E matrix); has never been green, which is why this pre-existing desync only surfaces now.
• Broader Release-CI E2E tail still open after fix(ci): unblock release lane — Rust E0063, macOS E0463, and E2E login-consume fixtures #4511 (separate follow-ups): CEF shared-session instability — the harness runs one CEF app per shard (app/test/wdio.conf.ts maxInstances: 1, debuggerAddress attach) and it dies mid-shard under load (A session is either terminated or not started, webhooks harness-*/providers), which specFileRetries cannot rescue (retry re-attaches to the dead session); and onboarding-modes wizard (2 tests: Phase-B RuntimeChoice step advance + Voice Routing staged-save persistence) needing an app-run to debug.

https://claude.ai/code/session_01UAetDHQCyFtWnyP7RwGdJU

[senamakel]

Root cause confirmed + partial fix in #4519

Mechanism (traced to source): the desktop chat agent is the orchestrator archetype (session/builder/factory.rs → from_config_for_agent(config, "orchestrator")), which synthesizes delegate_* tools including analyze_image (vision agent's delegate) whose schema requires prompt + citation_requirement (agent_orchestration/tools/archetype_delegation.rs). The mock's llmForcedResponses is a process-global one-shot FIFO consumed one entry per /chat/completions (llm.mjs), unscoped to thread/purpose.

The desync driver is an extra racing model call: fire-and-forget thread-title generation (chat_with_system, tools: None) dispatched on user send (app/src/store/threadSlice.ts:162-163 → threads/ops.rs:387) races the visible turn and drains a scripted response. The queue then falls through to buildDynamicCompletion → agentic inferToolCalls, which fabricates a call to declared[0] (= analyze_image) with {url} args → the exact DOM error "invalid arguments for analyze_image: arguments.prompt is required" + the citation-schema bubble. Not empty-full_response; not cross-spec bleed (per-spec after() reset is adequate; mock runs in-process).

Classification: test-fixture/mock issue exposed by an intended harness change — no core change needed.

Fix (#4519, mock-side):

1. Only the primary interactive turn (request advertises tools) drains the forced/stream queue — ancillary no-tools completions (title/summary) no longer steal scripted responses.
2. inferToolCalls only fabricates for simple worker tools (no required prompt); delegate-only turns return benign text, so residual desync is legible.

Validated against the running mock (no-tools call leaves queue intact; consecutive tool turns get web_fetch→canary in order; delegate-only drained turn returns text not a broken analyze_image call).

Remaining under this issue: chat-harness-subagent (genuine orchestrator→researcher fan-out) still needs its FORCED_RESPONSES extended to cover the sub-agent's own model calls, or conversion to llmKeywordRules (content-addressed, never shifted). #4519 fixes the single-turn specs (user-journey-full-task, chat-tool-call-flow, chat-thread-todo-strip, likely chat-multi-tool-round).