docs

hotato docs

Find what broke in your agent calls. Pin it and CI stays red until you fix it. The loop: autopsy one recording (or scan a folder, or pull a platform's recent calls with vapi health), pin the incident as a portable failure check, and prove re-measures that stored evidence on every push, holding the build red until a fresh recording of the same moment clears it. This index maps every doc to the step it belongs to.

New here? Run hotato autopsy ./call.wav on one recording (AUTOPSY.md), or start with GETTING-STARTED.md. The whole loop on one page: LIFECYCLE.md.

The loop: find what broke, then pin it

• AUTOPSY.md - autopsy, scan, and the <stack> health commands: one recording (or a folder, or your platform's recent calls) in, the incident list and health report out; pin graduates an incident
• EVIDENCE-CONTRACT.md - the four-tier evidence policy: what every verdict stands on
• CONTRACTS.md - failure contracts: a portable CI bundle of one call moment
• CI.md - prove and the CI gate: fail the pull request on the pinned evidence, offline
• GETTING-STARTED.md - one path from first touch to a CI gate
• START.md - guided first run on the bundled demo data
• WHY.md - four timing failures a text-level eval cannot see

Continuous use

• CONNECT.md - store a stack's credentials once; feed every hotato command from your platform
• ADAPTER-STATUS.md - per-stack pull, endpoint, and channel-separation status
• PRODUCTION-MONITORING.md - turn production call events into offline regression candidates
• WORKSPACE.md - hotato serve and hotato console: the self-hosted team web workspace and the live call console
• SELF-HOST.md - run the full workspace in your own VPC

Working with audio

• TRUST.md - is this recording even scorable?
• TRUST-MATRIX.md - the input-condition-to-behaviour contract for the trust check
• TRUST-GALLERY.md - eight recordings, eight verdicts, verbatim output
• DIARIZE.md - diarize a mono recording to make it scorable
• TRANSCRIBE.md - attach a transcript beside the timing score
• FULL-DUPLEX.md - score the moment both sides speak at once

Reference

• API.md - the stdlib-only scoring core, Python API
• SDK.md - the typed Python SDK facade over the CLI
• MCP.md - the hotato MCP server and its tools, over stdio
• METHODOLOGY.md - how the timing measurement works, end to end
• THREAT-MODEL.md - which commands are offline, which reach the network
• EGRESS.md - every network call site mapped to its command
• VALIDATION.md - the three separate jobs hotato is validated on
• EVIDENCE-PACK.md - the reproducible proof artifacts, ranked
• GALLERY.md - every image and worked example, each reproducible
• COMPARE.md - where hotato sits next to broad QA platforms
• evidence/README.md - the evidence standard: what counts, and ranking
• case-studies/README.md - the honesty standard every case study meets

Lab: the deep toolkit

Every command in this section lives under hotato lab (hotato lab --help). The public commands are durable. hotato lab moves faster, and every command name that worked before 1.17 still runs unchanged.

Investigate and evaluate

• INVESTIGATE.md - one recording in, ranked candidate moments out
• ANALYZE.md - drop a folder, rank and hear the worst moments
• ASSERTIONS.md - deterministic typed assertions over transcript, trace, and timing
• RUBRIC.md - the model-judged rubric lane, scored with a pinned local model
• EXPLAIN.md - root-cause-by-layer attribution from existing results
• STATE-ADAPTERS.md - ground state assertions in your system of record
• scenarios/dtmf-verification.md - verify DTMF reached the far end
• scenarios/echo-self-interruption.md - diagnose self-interruption from echo bleed

Observe

• OBSERVE.md - LLM and voice observability from your OpenTelemetry spans, locally
• TRACE.md - voice traces: the pipeline-event timeline
• OTEL.md - ingest OTel traces into the voice_trace span format
• latency-waterfall.md - per-hop latency waterfall from a scored call

Test and simulate

• SIMULATE.md - render a scenario into a deterministic labelled conversation
• CONVERSATION-TEST.md - one file, one call, a per-dimension scorecard
• SUITES.md - four tiered deterministic corpus suites
• SUITE-RUN.md - execute a suite, per-dimension report
• GENERATIVE-CALLER.md - the bounded caller engine: scripts, graphs, replay
• DRIVE-A-CALL.md - originate a call against a live agent, then score
• CALLER-LOAD.md - replay a bounded caller program under load
• LOAD-AND-RECOVERY.md - schedule calls under load, keep per-call evidence
• COUNTEREXAMPLES.md - reduce a scripted failure to a minimal repro
• PIPER-CALLER-TTS.md - local Piper TTS adapter for caller speech
• scenarios/browser-vs-pstn.md - score the same moment through telephony degradation
• scenarios/load-and-recovery.md - behaviour under concurrent load, with receipts

Fix and gate, the long way

• BAD-CALL-TO-CI.md - turn one bad call into a CI gate, step by step
• STARTER.md - hotato lab init starter scaffolds a CI gate and config
• PYTEST.md - the pytest fixture and opt-in session gate
• FIX-LOOP.md - the closed loop: find, fix, prove it is fixed
• FIX-PLANS.md - the guarded fix ladder: diagnose, inspect, plan, apply
• FIX-TRIAL.md - before/after fix proof, fail-closed and clone-only
• APPLY.md - guarded, clone-only staged apply of a fix patch
• RECAPTURE.md - prove the current agent, not the frozen recording
• RELEASE-COMPARE.md - diff two releases per dimension
• CARDS.md - render one measured moment as a PR-native SVG card
• scenarios/false-interruption-replay.md - a false-stop becomes a contract replayed in CI

Pipe your stack in

• INGEST.md - a passive webhook on-ramp scanning completed calls
• SET-AND-FORGET.md - a passive scheduled sweep for regression monitoring
• TRANSPORT-RUNTIME.md - lifecycle, delivered media, and assertion facts across transports
• CALLER-SIDECAR-PROTOCOL.md - the caller/transport sidecar WebSocket protocol
• LIVEKIT-CALLER-SESSION.md - direct LiveKit room transport for the caller engine

Fleet, reports, and benchmarks

• GUARDIAN-FLEET.md - a control plane running the evidence workflow continuously
• REPORTS.md - reporting surfaces: doctor, report, team, export
• BENCHMARK.md - the measurement-error harness over labelled recordings
• BENCH-SPEC.md - frozen batteries, scoring protocol, verify
• BENCHMARK-STACKS.md - run one battery through your configured stacks

Contributing

• SUBMITTING.md - the full path from a call to a merged corpus entry
• CORPUS-GOVERNANCE.md - consent, PII, and publishing rules for contributed calls
• RFC-ROLEPLAY-FIXTURES.md - a share-safe role-play fixture format
• RELEASE-CHECKLIST.md - the maintainer gates to clear before a release

See also the repository CONTRIBUTING.md, SECURITY.md, and CHANGELOG.md.