reference-agent

Reference agent: the complete hotato example

A self-contained, runnable example of the whole hotato loop, offline. It is a reference voice agent under test: 25 realistic jobs, most handled correctly and a handful with genuine defects, so the suite surfaces failures the way a working QA system does.

25 scenarios  ×  5 caller behaviours (speaking paces 0.7–1.4×)  ×  3 audio
environments (clean / cafe / street)  =  375 offline simulated runs

Everything runs through the deterministic scripted-caller simulator: no live agent, no network, no model. Each scenario declares a scripted caller plus a deterministic agent_mock (tool calls + a post-call state sandbox), so the conversation-tests exercise the outcome and policy authorities offline:

• Authority 1 - the trace: agent_mock.tools render as tool_call spans; tool_result / tool_error / sequence read them, never the agent's words.
• Authority 2 - the state sandbox: agent_mock.state is a {resource: rows} post-call system of record; state / state_change query it.

A mock is never a live agent: every produced conversation is origin=simulated, and its evidence is labelled the simulator's.

Run it

make reference          # regenerate the files, run the 375-run suite, print counts + wall time
make serve              # browse the recorded workspace (hotato serve)
make clean              # remove ./.workspace and ./.out

Or directly:

python generate.py                       # write scenarios/, tests/, suite.json
python run_reference.py --parallel 8     # run the suite; record a browsable workspace
hotato serve --workspace reference --registry ./.workspace

run_reference.py prints the real counts (runs, valid, per-dimension pass/fail/inconclusive, SIMULATOR_INVALID) and the wall time, records the Release / Suite / Scenario / Run / Conversation / Evaluation rows into a local fleet registry under ./.workspace, and writes the per-test conversation artifacts + the suite report under ./.out.

Layout

scenarios/<job>.scenario.json   # 25 hotato.scenario.v1 files (caller + agent_mock)
tests/<job>.test.json           # 25 hotato.conversation-test.v1 files (deterministic assertions)
suite.json                      # the hotato.suite.v1 binding them (required_for_release, inconclusive_policy: fail)
generate.py                     # the data-driven builder (source of truth for the files)
run_reference.py                # runs the suite + records the workspace
Makefile                        # make reference / serve / clean

What the tests check, per dimension

Dimension	Assertion kinds used
Outcome	tool_result (Authority 1), state (Authority 2)
Policy	sequence (identity verified before the sensitive action), tool_call (required disclosure), handoff (escalation), termination, tool_error (absent)
Conversation	phrase (the caller stated their need), count, sequence (ordered flow)
Speech	latency (a tool responded within its budget)
Reliability	pass@1 / pass@k / pass^k over the runs (its own axis, never blended)

Success is a boolean over named conditions; the scorecard is per-dimension. There is no blended or overall score anywhere.

The built-in defects

Four jobs carry genuine agent bugs, so the suite reports real failures - the raw material for the failure-cluster view and the production-to-regression flow:

• refund-claimed-not-issued - the caller asks for a refund; the agent never calls issue_refund and the order's refund_status stays none → outcome FAIL (tool_result + state).
• identity-skipped-before-lookup - records looked up before identity is verified → policy FAIL (sequence).
• escalate-not-handed-off - a manager was requested but no handoff occurred → policy FAIL (handoff).
• payment-declined-handled-wrong - the charge errored but the flow required no error → policy FAIL (tool_error).

The suite therefore exits non-zero, and hotato serve clusters these failures by observable signature.

Determinism - what is byte-identical, and what is not

Two seeded runs of the reference procedure write byte-identical conversation artifacts under ./.out: the transcript.json and trace.jsonl carry no timestamp, and the conversation.json manifest's created_at defaults to a reproducible instant rather than the wall clock. That default follows the reproducible-builds SOURCE_DATE_EPOCH convention - set SOURCE_DATE_EPOCH (integer seconds) to pin a specific instant; unset, it is a fixed placeholder (1970-01-01T00:00:00Z), never now().

The fleet registry (./.workspace/fleet.db) is deliberately outside that byte-identical claim: it is mutable runtime index state, and its bookkeeping columns (every *_at timestamp, plus last_watermark) are wall-clock. The defined comparison for the registry is therefore its deterministic content with those columns excluded - two seeded runs agree on every id, edge, and per-dimension verdict, differing only on the excluded runtime clocks. The canonicalization and both claims are pinned by tests/test_determinism_reference.py.

See ../../docs/SUITE-RUN.md, ../../docs/CONVERSATION-TEST.md, and ../../docs/STATE-ADAPTERS.md.