diff --git a/CHANGELOG.md b/CHANGELOG.md index 00c7af53..1e150ab9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,49 @@ Every entry reports millisecond measurement error and a confusion matrix. See `d ## [Unreleased] +### Added +- **`hotato lab` -- the deep toolkit gets its own front door.** The + top-level `--help` now shows only the public surface (`autopsy`, `scan`, + `pin`, `prove`, `connect`, the five ` health` commands, `start`, + `demo`, `doctor`, `console`, `production`, `serve`, `contract`, + `describe`); every other command lives under `hotato lab `, and + `hotato lab --help` lists each one with its one-line description. One + registration layer (`_SurfaceRouter`) decides visibility: a lab command + registers exactly as before -- same name, same parser, same behavior -- + but without a parent help line, so every pre-1.17 top-level spelling + keeps working unchanged as a compat alias, and the `lab` prefix is pure + argv dispatch onto the same parser. The stability statement (public + commands durable; lab evolves faster; old spellings keep working) + renders in the `--help` epilog tail, `hotato lab --help`, README, + CONTRIBUTING (with the monthly-stable release cadence), and the + `describe` manifest. +- **`hotato describe` records both spellings per command.** Every + top-level manifest entry now carries `surface` (`public` or `lab`), its + `canonical` spelling, and `compat_alias` (the still-working pre-1.17 + top-level name; `null` for public commands), plus a top-level + `stability` field. The text rendering shows lab commands under + `hotato lab ...` with the alias beside them. Exit codes, args, and the + command set itself are unchanged. +- **`docs/EVIDENCE-CONTRACT.md` -- the four-tier evidence policy, stated + once.** Tier 1 dual-channel audio (deterministic, the only + verdict-eligible tier), tier 2 mono plus provider metadata (attributable + findings, each carrying its source's declared authority), tier 3 raw + mixed mono (symptom detection with a measured confidence), tier 4 + insufficient evidence (refused, exit 2, with the remediation). README, + AUTOPSY.md, the trust docs, and the docs index link to it instead of + restating it; `tests/test_evidence_contract.py` holds the page and its + referencing surfaces in lockstep. + +### Changed +- **docs index restructured to the loop.** `docs/README.md` now leads with + the loop (autopsy -> scan -> pin -> prove -> ` health`), then + continuous use, then a Lab section grouping the moved commands' docs. + Every doc stays listed. +- **Superlative sweep.** Unsupported superlatives replaced with measured + framing: the README say-do bullet now reads "a bug that fires actions + the caller canceled", and `docs/STARTER.md` opens with what the one + command does. + ## [1.16.0] - 2026-07-24 ### Added diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 99173461..3cda795e 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,10 +1,11 @@ # Contributing to hotato -Thanks for being here. hotato is local-first testing and observability for AI agents: it simulates, -evaluates, reviews, and tracks calls across five dimensions (outcome, policy, -conversation, speech, reliability), with the evidence behind every result. -Deterministic checks stay separate from the model-judged rubric, and no output is -ever a single blended score. +Thanks for being here. hotato finds what broke in your agent calls and pins +it so it never ships again: it simulates, evaluates, reviews, and tracks +calls across five dimensions (outcome, policy, conversation, speech, +reliability), with the evidence behind every result. Deterministic checks +stay separate from the model-judged rubric, and no output is ever a single +blended score. Every kind of contribution is welcome. This guide gets your first one merged with the least friction, then points you at the contribution that helps most. @@ -63,11 +64,32 @@ Try the tool end to end with no account, keys, or network: ```bash hotato start --demo # sweep two bundled calls, write a dashboard, verify a contract -hotato --help # every subcommand +hotato --help # the public surface +hotato lab --help # the deep toolkit behind it ``` Then keep the diff small: one scenario, one fix, or one recording per PR. +## Surface stability and release cadence + +The CLI has two surfaces, and they carry different stability promises: + +- **Public** (everything `hotato --help` lists: `autopsy`, `scan`, `pin`, + `prove`, `connect`, the ` health` commands, `start`, `demo`, + `doctor`, `console`, `production`, `serve`, `contract`, `describe`). + These commands, their flags, and their exit codes are durable: a script + or CI job written against them keeps working across releases. +- **Lab** (everything `hotato lab --help` lists). The lab surface evolves + faster between releases. Every pre-1.17 top-level spelling keeps working + unchanged as a compat alias, and `hotato describe` records both + spellings for every command. + +Releases follow a **monthly-stable cadence**: a versioned release ships +about once a month, carries the full CHANGELOG entry for everything that +landed, and is the unit the stability promise attaches to. Any change to a +public command's flags or exit codes is called out in the CHANGELOG at +that release. + ## The highest-value contribution: a labeled call recording hotato gets more credible with every consented, de-identified call clip in the diff --git a/README.md b/README.md index cd630388..4a6f482a 100644 --- a/README.md +++ b/README.md @@ -29,7 +29,7 @@ No judges. No cloud. No bill. MIT. ## What it finds -- **Barge-in → Say-do gaps**: Caller interrupts to cancel; agent says "canceled" but the booking tool still fires. The most expensive voice AI bug. (Timing from the audio; the tool-fire check reads your call's tool log: hotato ingests Vapi/OTel traces.) +- **Barge-in → Say-do gaps**: Caller interrupts to cancel; agent says "canceled" but the booking tool still fires: a bug that fires actions the caller canceled. (Timing from the audio; the tool-fire check reads your call's tool log: hotato ingests Vapi/OTel traces.) - **Latency spikes**: 800ms → 5s unpredictability that makes users hang up. - **Dead air**: Long silences that kill conversation flow. - **Talk-over**: Agent speaks over the caller; never yields. @@ -122,8 +122,14 @@ The whole loop, command by command: [`docs/LIFECYCLE.md`](docs/LIFECYCLE.md). First touch to a CI gate: [`docs/GETTING-STARTED.md`](docs/GETTING-STARTED.md). Feed it what you already have: [`docs/CONNECT.md`](docs/CONNECT.md) · [`docs/TRACE.md`](docs/TRACE.md) · [`docs/SIMULATE.md`](docs/SIMULATE.md). +What every verdict stands on: [`docs/EVIDENCE-CONTRACT.md`](docs/EVIDENCE-CONTRACT.md). Next to the hosted alternatives: [`docs/COMPARE.md`](docs/COMPARE.md). +The deep toolkit -- capture, simulation, load, benchmarking, the fix ladder, +the fleet control plane -- lives under `hotato lab` (`hotato lab --help`). +The public commands are durable; `hotato lab` evolves faster; every pre-1.17 +top-level spelling keeps working unchanged. + ## Specifications | Property | Value | @@ -144,7 +150,7 @@ PYTHONPATH=src python3 -m hotato.benchmark \ On 13 recorded AMI Meeting Corpus clips, the median error between measured caller-onset and the human word-alignment label is **20 ms**. Provenance: [`corpus/real/README.md`](corpus/real) · method: [`METHODOLOGY.md`](METHODOLOGY.md). -Timing is measurable only when the two voices arrive on separate channels; a mono or mixed export is marked **NOT SCORABLE** and refused (`hotato trust --stereo call.wav`). +Timing is measurable only when the two voices arrive on separate channels; a mono or mixed export is marked **NOT SCORABLE** and refused (`hotato trust --stereo call.wav`). The full four-tier evidence policy (what each verdict stands on, per input) is [`docs/EVIDENCE-CONTRACT.md`](docs/EVIDENCE-CONTRACT.md). diff --git a/README.pypi.md b/README.pypi.md index 1b330cd8..e780a745 100644 --- a/README.pypi.md +++ b/README.pypi.md @@ -29,7 +29,7 @@ No judges. No cloud. No bill. MIT. ## What it finds -- **Barge-in → Say-do gaps**: Caller interrupts to cancel; agent says "canceled" but the booking tool still fires. The most expensive voice AI bug. (Timing from the audio; the tool-fire check reads your call's tool log: hotato ingests Vapi/OTel traces.) +- **Barge-in → Say-do gaps**: Caller interrupts to cancel; agent says "canceled" but the booking tool still fires: a bug that fires actions the caller canceled. (Timing from the audio; the tool-fire check reads your call's tool log: hotato ingests Vapi/OTel traces.) - **Latency spikes**: 800ms → 5s unpredictability that makes users hang up. - **Dead air**: Long silences that kill conversation flow. - **Talk-over**: Agent speaks over the caller; never yields. @@ -122,8 +122,14 @@ The whole loop, command by command: [`docs/LIFECYCLE.md`](https://github.com/att First touch to a CI gate: [`docs/GETTING-STARTED.md`](https://github.com/attenlabs/hotato/blob/main/docs/GETTING-STARTED.md). Feed it what you already have: [`docs/CONNECT.md`](https://github.com/attenlabs/hotato/blob/main/docs/CONNECT.md) · [`docs/TRACE.md`](https://github.com/attenlabs/hotato/blob/main/docs/TRACE.md) · [`docs/SIMULATE.md`](https://github.com/attenlabs/hotato/blob/main/docs/SIMULATE.md). +What every verdict stands on: [`docs/EVIDENCE-CONTRACT.md`](https://github.com/attenlabs/hotato/blob/main/docs/EVIDENCE-CONTRACT.md). Next to the hosted alternatives: [`docs/COMPARE.md`](https://github.com/attenlabs/hotato/blob/main/docs/COMPARE.md). +The deep toolkit -- capture, simulation, load, benchmarking, the fix ladder, +the fleet control plane -- lives under `hotato lab` (`hotato lab --help`). +The public commands are durable; `hotato lab` evolves faster; every pre-1.17 +top-level spelling keeps working unchanged. + ## Specifications | Property | Value | @@ -144,7 +150,7 @@ PYTHONPATH=src python3 -m hotato.benchmark \ On 13 recorded AMI Meeting Corpus clips, the median error between measured caller-onset and the human word-alignment label is **20 ms**. Provenance: [`corpus/real/README.md`](https://github.com/attenlabs/hotato/blob/main/corpus/real) · method: [`METHODOLOGY.md`](https://github.com/attenlabs/hotato/blob/main/METHODOLOGY.md). -Timing is measurable only when the two voices arrive on separate channels; a mono or mixed export is marked **NOT SCORABLE** and refused (`hotato trust --stereo call.wav`). +Timing is measurable only when the two voices arrive on separate channels; a mono or mixed export is marked **NOT SCORABLE** and refused (`hotato trust --stereo call.wav`). The full four-tier evidence policy (what each verdict stands on, per input) is [`docs/EVIDENCE-CONTRACT.md`](https://github.com/attenlabs/hotato/blob/main/docs/EVIDENCE-CONTRACT.md). diff --git a/docs/AUTOPSY.md b/docs/AUTOPSY.md index b8076385..ad8fcd33 100644 --- a/docs/AUTOPSY.md +++ b/docs/AUTOPSY.md @@ -74,9 +74,10 @@ mixed channel; talk-over attribution comes from a two-channel recording -- the scope line states this once per run). Mono calls report into the best-effort mono observations block with their own counts and never enter the Voice Stability denominator, so the mono stacks' reports carry -observations without a stability score. The analysis runs on this -machine; recordings download straight from the platform and go nowhere -else. +observations without a stability score +([EVIDENCE-CONTRACT.md](EVIDENCE-CONTRACT.md) states the whole tier +policy). The analysis runs on this machine; recordings download straight +from the platform and go nowhere else. A window with no calls, a pull in which every recording failed to fetch, or a pulled set with zero analyzable calls refuses with the reason (exit @@ -120,7 +121,10 @@ nothing is guessed and no confidence is invented. The stricter commands keep their bar: `run`, `scan`, `trust`, and the contract path still refuse mono as NOT SCORABLE. Autopsy is discovery; -the CI gate stays deterministic and dual-channel. +the CI gate stays deterministic and dual-channel. The full four-tier +policy behind this split -- dual-channel deterministic, mono with +provider metadata, raw mixed mono, refused -- is stated once in +[EVIDENCE-CONTRACT.md](EVIDENCE-CONTRACT.md). An unreadable input -- a text file, a truncated header, a non-audio blob -- is refused with the reason (exit 2), never scored. diff --git a/docs/EVIDENCE-CONTRACT.md b/docs/EVIDENCE-CONTRACT.md new file mode 100644 index 00000000..ee20583e --- /dev/null +++ b/docs/EVIDENCE-CONTRACT.md @@ -0,0 +1,64 @@ +# The evidence contract: four tiers, one policy + +Every hotato verdict states what evidence it stands on. This page is the +single source of truth for that policy; the README, [AUTOPSY.md](AUTOPSY.md), +the platform health reports, and the trust docs +([TRUST.md](TRUST.md) · [TRUST-MATRIX.md](TRUST-MATRIX.md)) all apply it +and link back here. + +The four tiers, from strongest evidence to none: + +## Tier 1 -- dual-channel audio: deterministic + +Caller on one channel, agent on the other, separated at capture. Overlap and +timing are facts of the recording, exact to the sample, so the deterministic +timing walk runs end to end: byte-identical output for the same input, every +threshold exposed, every frame inspectable. This is the only tier that is +**verdict-eligible**: contracts, `hotato pin`, `hotato prove`, and the CI +gate all stand on it, and it alone enters the Voice Stability denominator. + +## Tier 2 -- mono plus provider metadata: attributable, with declared authority + +A mixed single channel cannot attribute energy to a speaker by itself, but +metadata a provider or pipeline supplies alongside it can: a +speaker-attributed transcript, a voice-pipeline trace, tool-call logs, or a +diarizer's turn labels (`hotato run --mono call.wav --diarize`, +[DIARIZE.md](DIARIZE.md)). Findings on this tier are **attributable**, and +each one carries the **declared authority** of the source that attributed it +-- the production evidence plane records it explicitly (`submitted`, +`adapter_reported`, `provider_export`, `signed_attestation`, `measured`), +and a diarized verdict is tiered and stamped (`diarized-mono`; +`indicative_only` below the confidence bar). Attribution is only as strong +as its source, so the authority prints with the finding and this tier never +merges into Tier 1's deterministic counts. + +## Tier 3 -- raw mixed mono: symptom detection, with measured confidence + +One mixed channel with no metadata still measures what silence shows: +dead air and latency gaps. `hotato autopsy` and the mono-stack health +reports run this path best-effort, and every finding carries a **measured +confidence** with its derivation printed beside it. A mono gap says +everything stopped, not who stopped -- talk-over and barge-in attribution +comes from the tiers above, that scope is stated once per run, and Tier 3 +findings report in their own block, outside the Voice Stability +denominator. + +## Tier 4 -- insufficient evidence: refused, with the remediation + +An input that supports none of the above -- an unreadable file, a silent +required channel, a mixed export where a deterministic verdict was asked +for -- is **refused** (exit `2`) with the reason and the next step: the +recording scaffold for your stack (`hotato setup`), the input health check +(`hotato trust --stereo call.wav`), or the mono escapes above. A refusal +leaves no artifact and never becomes a number; a green or red build always +means something. + +## Reading a report against the tiers + +The health and scan reports render the split directly: the Voice Stability +Score and the `health:` share count **dual-channel calls only** (Tier 1); +mono findings sit in the *Best-effort mono observations* block with their +own counts (Tier 3, or Tier 2 where metadata attributed them); refused +files are listed with their reasons (Tier 4), never scored. The per-lane +**evidence coverage** block states which tiers this run actually had -- +a lane whose evidence was absent never renders as assessed. diff --git a/docs/README.md b/docs/README.md index 11fef2be..5f4e1a8b 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,34 +1,66 @@ # hotato docs -Find what broke in your agent calls. Pin it so it never ships again. Every -production failure becomes a portable test, every candidate runs against it, -and every release carries evidence. This index maps every doc to the step it -belongs to. +Find what broke in your agent calls. Pin it so it never ships again. The +loop: `autopsy` one recording (or `scan` a folder, or pull a platform's +recent calls with `vapi health`), `pin` the incident as a portable failure +check, and `prove` re-runs the stored evidence in CI so it never ships +again. This index maps every doc to the step it belongs to. New here? Run `hotato autopsy ./call.wav` on one recording (**[AUTOPSY.md](AUTOPSY.md)**), or start with **[GETTING-STARTED.md](GETTING-STARTED.md)**. The whole loop on one page: **[LIFECYCLE.md](LIFECYCLE.md)**. -## Getting started +## The loop: find what broke, then pin it +- [AUTOPSY.md](AUTOPSY.md) - `autopsy`, `scan`, and the ` health` commands: one recording (or a folder, or your platform's recent calls) in, the incident list and health report out; `pin` graduates an incident +- [EVIDENCE-CONTRACT.md](EVIDENCE-CONTRACT.md) - the four-tier evidence policy: what every verdict stands on +- [CONTRACTS.md](CONTRACTS.md) - failure contracts: a portable CI bundle of one call moment +- [CI.md](CI.md) - `prove` and the CI gate: fail the pull request on the pinned evidence, offline - [GETTING-STARTED.md](GETTING-STARTED.md) - one path from first touch to a CI gate - [START.md](START.md) - guided first run on the bundled demo data -- [STARTER.md](STARTER.md) - `hotato init starter` scaffolds a CI gate and config -- [BAD-CALL-TO-CI.md](BAD-CALL-TO-CI.md) - turn one bad call into a CI gate - [WHY.md](WHY.md) - four timing failures a text-level eval cannot see -## Observe +## Continuous use -- [OBSERVE.md](OBSERVE.md) - LLM and voice observability from your OpenTelemetry spans, locally -- [TRACE.md](TRACE.md) - voice traces: the pipeline-event timeline -- [OTEL.md](OTEL.md) - ingest OTel traces into the `voice_trace` span format -- [latency-waterfall.md](latency-waterfall.md) - per-hop latency waterfall from a scored call +- [CONNECT.md](CONNECT.md) - store a stack's credentials once; feed every hotato command from your platform +- [ADAPTER-STATUS.md](ADAPTER-STATUS.md) - per-stack pull, endpoint, and channel-separation status - [PRODUCTION-MONITORING.md](PRODUCTION-MONITORING.md) - turn production call events into offline regression candidates +- [WORKSPACE.md](WORKSPACE.md) - `hotato serve` and `hotato console`: the self-hosted team web workspace and the live call console +- [SELF-HOST.md](SELF-HOST.md) - run the full workspace in your own VPC + +## Working with audio + +- [TRUST.md](TRUST.md) - is this recording even scorable? +- [TRUST-MATRIX.md](TRUST-MATRIX.md) - the input-condition-to-behaviour contract for the trust check +- [TRUST-GALLERY.md](TRUST-GALLERY.md) - eight recordings, eight verdicts, verbatim output +- [DIARIZE.md](DIARIZE.md) - diarize a mono recording to make it scorable +- [TRANSCRIBE.md](TRANSCRIBE.md) - attach a transcript beside the timing score +- [FULL-DUPLEX.md](FULL-DUPLEX.md) - score the moment both sides speak at once + +## Reference + +- [API.md](API.md) - the stdlib-only scoring core, Python API +- [SDK.md](SDK.md) - the typed Python SDK facade over the CLI +- [MCP.md](MCP.md) - the hotato MCP server and its tools, over stdio +- [METHODOLOGY.md](../METHODOLOGY.md) - how the timing measurement works, end to end +- [THREAT-MODEL.md](THREAT-MODEL.md) - which commands are offline, which reach the network +- [EGRESS.md](EGRESS.md) - every network call site mapped to its command +- [VALIDATION.md](VALIDATION.md) - the three separate jobs hotato is validated on +- [EVIDENCE-PACK.md](EVIDENCE-PACK.md) - the reproducible proof artifacts, ranked +- [GALLERY.md](GALLERY.md) - every image and worked example, each reproducible +- [COMPARE.md](COMPARE.md) - where hotato sits next to broad QA platforms +- [evidence/README.md](evidence/README.md) - the evidence standard: what counts, and ranking +- [case-studies/README.md](case-studies/README.md) - the honesty standard every case study meets + +## Lab: the deep toolkit + +Every command in this section lives under `hotato lab` (`hotato lab --help`). +The public commands above are durable; the lab surface evolves faster, and +every pre-1.17 top-level spelling keeps working unchanged. -## Evaluate +### Investigate and evaluate -- [AUTOPSY.md](AUTOPSY.md) - one recording, zero config: the incident list and report; the folder health report (`scan DIR`) and `pin` to a contract - [INVESTIGATE.md](INVESTIGATE.md) - one recording in, ranked candidate moments out - [ANALYZE.md](ANALYZE.md) - drop a folder, rank and hear the worst moments - [ASSERTIONS.md](ASSERTIONS.md) - deterministic typed assertions over transcript, trace, and timing @@ -38,7 +70,14 @@ New here? Run `hotato autopsy ./call.wav` on one recording - [scenarios/dtmf-verification.md](scenarios/dtmf-verification.md) - verify DTMF reached the far end - [scenarios/echo-self-interruption.md](scenarios/echo-self-interruption.md) - diagnose self-interruption from echo bleed -## Test +### Observe + +- [OBSERVE.md](OBSERVE.md) - LLM and voice observability from your OpenTelemetry spans, locally +- [TRACE.md](TRACE.md) - voice traces: the pipeline-event timeline +- [OTEL.md](OTEL.md) - ingest OTel traces into the `voice_trace` span format +- [latency-waterfall.md](latency-waterfall.md) - per-hop latency waterfall from a scored call + +### Test and simulate - [SIMULATE.md](SIMULATE.md) - render a scenario into a deterministic labelled conversation - [CONVERSATION-TEST.md](CONVERSATION-TEST.md) - one file, one call, a per-dimension scorecard @@ -53,10 +92,10 @@ New here? Run `hotato autopsy ./call.wav` on one recording - [scenarios/browser-vs-pstn.md](scenarios/browser-vs-pstn.md) - score the same moment through telephony degradation - [scenarios/load-and-recovery.md](scenarios/load-and-recovery.md) - behaviour under concurrent load, with receipts -## Gate +### Fix and gate, the long way -- [CONTRACTS.md](CONTRACTS.md) - failure contracts: a portable CI bundle of one call moment -- [CI.md](CI.md) - gate a pull request on turn-taking timing, offline +- [BAD-CALL-TO-CI.md](BAD-CALL-TO-CI.md) - turn one bad call into a CI gate, step by step +- [STARTER.md](STARTER.md) - `hotato lab init starter` scaffolds a CI gate and config - [PYTEST.md](PYTEST.md) - the pytest fixture and opt-in session gate - [FIX-LOOP.md](FIX-LOOP.md) - the closed loop: find, fix, prove it is fixed - [FIX-PLANS.md](FIX-PLANS.md) - the guarded fix ladder: diagnose, inspect, plan, apply @@ -67,49 +106,21 @@ New here? Run `hotato autopsy ./call.wav` on one recording - [CARDS.md](CARDS.md) - render one measured moment as a PR-native SVG card - [scenarios/false-interruption-replay.md](scenarios/false-interruption-replay.md) - a false-stop becomes a contract replayed in CI -## Connect your stack +### Pipe your stack in -- [CONNECT.md](CONNECT.md) - connect, pull, sweep: score every call across stacks -- [ADAPTER-STATUS.md](ADAPTER-STATUS.md) - per-stack pull, endpoint, and channel-separation status - [INGEST.md](INGEST.md) - a passive webhook on-ramp scanning completed calls - [SET-AND-FORGET.md](SET-AND-FORGET.md) - a passive scheduled sweep for regression monitoring - [TRANSPORT-RUNTIME.md](TRANSPORT-RUNTIME.md) - lifecycle, delivered media, and assertion facts across transports - [CALLER-SIDECAR-PROTOCOL.md](CALLER-SIDECAR-PROTOCOL.md) - the caller/transport sidecar WebSocket protocol - [LIVEKIT-CALLER-SESSION.md](LIVEKIT-CALLER-SESSION.md) - direct LiveKit room transport for the caller engine -## Working with audio +### Fleet, reports, and benchmarks -- [TRUST.md](TRUST.md) - is this recording even scorable? -- [TRUST-MATRIX.md](TRUST-MATRIX.md) - the input-condition-to-behaviour contract for the trust check -- [DIARIZE.md](DIARIZE.md) - diarize a mono recording to make it scorable -- [TRANSCRIBE.md](TRANSCRIBE.md) - attach a transcript beside the timing score -- [FULL-DUPLEX.md](FULL-DUPLEX.md) - score the moment both sides speak at once - -## Self-host and team - -- [SELF-HOST.md](SELF-HOST.md) - run the full workspace in your own VPC -- [WORKSPACE.md](WORKSPACE.md) - `hotato serve`: the self-hosted team web workspace - [GUARDIAN-FLEET.md](GUARDIAN-FLEET.md) - a control plane running the evidence workflow continuously - [REPORTS.md](REPORTS.md) - reporting surfaces: doctor, report, team, export - -## Reference - -- [API.md](API.md) - the stdlib-only scoring core, Python API -- [SDK.md](SDK.md) - the typed Python SDK facade over the CLI -- [MCP.md](MCP.md) - the hotato MCP server and its tools, over stdio - [BENCHMARK.md](BENCHMARK.md) - the measurement-error harness over labelled recordings - [BENCH-SPEC.md](BENCH-SPEC.md) - frozen batteries, scoring protocol, verify - [BENCHMARK-STACKS.md](BENCHMARK-STACKS.md) - run one battery through your configured stacks -- [METHODOLOGY.md](../METHODOLOGY.md) - how the timing measurement works, end to end -- [THREAT-MODEL.md](THREAT-MODEL.md) - which commands are offline, which reach the network -- [EGRESS.md](EGRESS.md) - every network call site mapped to its command -- [TRUST-GALLERY.md](TRUST-GALLERY.md) - eight recordings, eight verdicts, verbatim output -- [VALIDATION.md](VALIDATION.md) - the three separate jobs hotato is validated on -- [EVIDENCE-PACK.md](EVIDENCE-PACK.md) - the reproducible proof artifacts, ranked -- [GALLERY.md](GALLERY.md) - every image and worked example, each reproducible -- [COMPARE.md](COMPARE.md) - where hotato sits next to broad QA platforms -- [evidence/README.md](evidence/README.md) - the evidence standard: what counts, and ranking -- [case-studies/README.md](case-studies/README.md) - the honesty standard every case study meets ## Contributing diff --git a/docs/STARTER.md b/docs/STARTER.md index b5c9728a..3a091cd1 100644 --- a/docs/STARTER.md +++ b/docs/STARTER.md @@ -1,8 +1,8 @@ # The starter kit: `hotato init starter` -The fastest way to add hotato to an existing voice-agent repo: one command -scaffolds the CI gate, a stack-tuned config file, and the three directories -the rest of the docs assume already exist. +One command adds hotato to an existing voice-agent repo: it scaffolds the +CI gate, a stack-tuned config file, and the three directories the rest of +the docs assume already exist. ```bash hotato init starter --stack vapi --out . diff --git a/docs/TRUST-MATRIX.md b/docs/TRUST-MATRIX.md index c4486a8b..ed75411e 100644 --- a/docs/TRUST-MATRIX.md +++ b/docs/TRUST-MATRIX.md @@ -7,7 +7,9 @@ contract: input condition on the left, Hotato's behavior on the right. It covers input health only; a turn-taking verdict is `scan`'s and `run`'s job. Worked examples with CLI output for every row are in -[TRUST-GALLERY.md](TRUST-GALLERY.md). +[TRUST-GALLERY.md](TRUST-GALLERY.md). The four-tier evidence policy the +rows below implement is stated once in +[EVIDENCE-CONTRACT.md](EVIDENCE-CONTRACT.md). ## The contract diff --git a/docs/TRUST.md b/docs/TRUST.md index 5c01815e..ea3e9a35 100644 --- a/docs/TRUST.md +++ b/docs/TRUST.md @@ -4,7 +4,8 @@ The input-health check: inspect one recording and learn whether the audio is good enough to score, before you scan or run it. `trust` catches a bad export -- mono file, silent channel, swapped channel map, hot capture -- up front, before it becomes a confident-looking but meaningless verdict -downstream. +downstream. What each input tier supports downstream is the four-tier +policy in [EVIDENCE-CONTRACT.md](EVIDENCE-CONTRACT.md). ```bash hotato trust --stereo your-call.wav diff --git a/llms-full.txt b/llms-full.txt index 39917fdf..fee27df4 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -33,7 +33,7 @@ No judges. No cloud. No bill. MIT. ## What it finds -- **Barge-in → Say-do gaps**: Caller interrupts to cancel; agent says "canceled" but the booking tool still fires. The most expensive voice AI bug. (Timing from the audio; the tool-fire check reads your call's tool log: hotato ingests Vapi/OTel traces.) +- **Barge-in → Say-do gaps**: Caller interrupts to cancel; agent says "canceled" but the booking tool still fires: a bug that fires actions the caller canceled. (Timing from the audio; the tool-fire check reads your call's tool log: hotato ingests Vapi/OTel traces.) - **Latency spikes**: 800ms → 5s unpredictability that makes users hang up. - **Dead air**: Long silences that kill conversation flow. - **Talk-over**: Agent speaks over the caller; never yields. @@ -126,8 +126,14 @@ The whole loop, command by command: [`docs/LIFECYCLE.md`](docs/LIFECYCLE.md). First touch to a CI gate: [`docs/GETTING-STARTED.md`](docs/GETTING-STARTED.md). Feed it what you already have: [`docs/CONNECT.md`](docs/CONNECT.md) · [`docs/TRACE.md`](docs/TRACE.md) · [`docs/SIMULATE.md`](docs/SIMULATE.md). +What every verdict stands on: [`docs/EVIDENCE-CONTRACT.md`](docs/EVIDENCE-CONTRACT.md). Next to the hosted alternatives: [`docs/COMPARE.md`](docs/COMPARE.md). +The deep toolkit -- capture, simulation, load, benchmarking, the fix ladder, +the fleet control plane -- lives under `hotato lab` (`hotato lab --help`). +The public commands are durable; `hotato lab` evolves faster; every pre-1.17 +top-level spelling keeps working unchanged. + ## Specifications | Property | Value | @@ -148,7 +154,7 @@ PYTHONPATH=src python3 -m hotato.benchmark \ On 13 recorded AMI Meeting Corpus clips, the median error between measured caller-onset and the human word-alignment label is **20 ms**. Provenance: [`corpus/real/README.md`](corpus/real) · method: [`METHODOLOGY.md`](METHODOLOGY.md). -Timing is measurable only when the two voices arrive on separate channels; a mono or mixed export is marked **NOT SCORABLE** and refused (`hotato trust --stereo call.wav`). +Timing is measurable only when the two voices arrive on separate channels; a mono or mixed export is marked **NOT SCORABLE** and refused (`hotato trust --stereo call.wav`). The full four-tier evidence policy (what each verdict stands on, per input) is [`docs/EVIDENCE-CONTRACT.md`](docs/EVIDENCE-CONTRACT.md). @@ -2078,9 +2084,10 @@ mixed channel; talk-over attribution comes from a two-channel recording -- the scope line states this once per run). Mono calls report into the best-effort mono observations block with their own counts and never enter the Voice Stability denominator, so the mono stacks' reports carry -observations without a stability score. The analysis runs on this -machine; recordings download straight from the platform and go nowhere -else. +observations without a stability score +([EVIDENCE-CONTRACT.md](EVIDENCE-CONTRACT.md) states the whole tier +policy). The analysis runs on this machine; recordings download straight +from the platform and go nowhere else. A window with no calls, a pull in which every recording failed to fetch, or a pulled set with zero analyzable calls refuses with the reason (exit @@ -2124,7 +2131,10 @@ nothing is guessed and no confidence is invented. The stricter commands keep their bar: `run`, `scan`, `trust`, and the contract path still refuse mono as NOT SCORABLE. Autopsy is discovery; -the CI gate stays deterministic and dual-channel. +the CI gate stays deterministic and dual-channel. The full four-tier +policy behind this split -- dual-channel deterministic, mono with +provider metadata, raw mixed mono, refused -- is stated once in +[EVIDENCE-CONTRACT.md](EVIDENCE-CONTRACT.md). An unreadable input -- a text file, a truncated header, a non-audio blob -- is refused with the reason (exit 2), never scored. @@ -2457,7 +2467,8 @@ The input-health check: inspect one recording and learn whether the audio is good enough to score, before you scan or run it. `trust` catches a bad export -- mono file, silent channel, swapped channel map, hot capture -- up front, before it becomes a confident-looking but meaningless verdict -downstream. +downstream. What each input tier supports downstream is the four-tier +policy in [EVIDENCE-CONTRACT.md](EVIDENCE-CONTRACT.md). ```bash hotato trust --stereo your-call.wav @@ -2889,9 +2900,9 @@ FILE: docs/STARTER.md # The starter kit: `hotato init starter` -The fastest way to add hotato to an existing voice-agent repo: one command -scaffolds the CI gate, a stack-tuned config file, and the three directories -the rest of the docs assume already exist. +One command adds hotato to an existing voice-agent repo: it scaffolds the +CI gate, a stack-tuned config file, and the three directories the rest of +the docs assume already exist. ```bash hotato init starter --stack vapi --out . @@ -9689,6 +9700,76 @@ local test recording server on `127.0.0.1` needs [`docs/DRIVE-A-CALL.md`](DRIVE-A-CALL.md). +================================================================================ +FILE: docs/EVIDENCE-CONTRACT.md +================================================================================ + +# The evidence contract: four tiers, one policy + +Every hotato verdict states what evidence it stands on. This page is the +single source of truth for that policy; the README, [AUTOPSY.md](AUTOPSY.md), +the platform health reports, and the trust docs +([TRUST.md](TRUST.md) · [TRUST-MATRIX.md](TRUST-MATRIX.md)) all apply it +and link back here. + +The four tiers, from strongest evidence to none: + +## Tier 1 -- dual-channel audio: deterministic + +Caller on one channel, agent on the other, separated at capture. Overlap and +timing are facts of the recording, exact to the sample, so the deterministic +timing walk runs end to end: byte-identical output for the same input, every +threshold exposed, every frame inspectable. This is the only tier that is +**verdict-eligible**: contracts, `hotato pin`, `hotato prove`, and the CI +gate all stand on it, and it alone enters the Voice Stability denominator. + +## Tier 2 -- mono plus provider metadata: attributable, with declared authority + +A mixed single channel cannot attribute energy to a speaker by itself, but +metadata a provider or pipeline supplies alongside it can: a +speaker-attributed transcript, a voice-pipeline trace, tool-call logs, or a +diarizer's turn labels (`hotato run --mono call.wav --diarize`, +[DIARIZE.md](DIARIZE.md)). Findings on this tier are **attributable**, and +each one carries the **declared authority** of the source that attributed it +-- the production evidence plane records it explicitly (`submitted`, +`adapter_reported`, `provider_export`, `signed_attestation`, `measured`), +and a diarized verdict is tiered and stamped (`diarized-mono`; +`indicative_only` below the confidence bar). Attribution is only as strong +as its source, so the authority prints with the finding and this tier never +merges into Tier 1's deterministic counts. + +## Tier 3 -- raw mixed mono: symptom detection, with measured confidence + +One mixed channel with no metadata still measures what silence shows: +dead air and latency gaps. `hotato autopsy` and the mono-stack health +reports run this path best-effort, and every finding carries a **measured +confidence** with its derivation printed beside it. A mono gap says +everything stopped, not who stopped -- talk-over and barge-in attribution +comes from the tiers above, that scope is stated once per run, and Tier 3 +findings report in their own block, outside the Voice Stability +denominator. + +## Tier 4 -- insufficient evidence: refused, with the remediation + +An input that supports none of the above -- an unreadable file, a silent +required channel, a mixed export where a deterministic verdict was asked +for -- is **refused** (exit `2`) with the reason and the next step: the +recording scaffold for your stack (`hotato setup`), the input health check +(`hotato trust --stereo call.wav`), or the mono escapes above. A refusal +leaves no artifact and never becomes a number; a green or red build always +means something. + +## Reading a report against the tiers + +The health and scan reports render the split directly: the Voice Stability +Score and the `health:` share count **dual-channel calls only** (Tier 1); +mono findings sit in the *Best-effort mono observations* block with their +own counts (Tier 3, or Tier 2 where metadata attributed them); refused +files are listed with their reasons (Tier 4), never scored. The per-lane +**evidence coverage** block states which tiers this run actually had -- +a lane whose evidence was absent never renders as assessed. + + ================================================================================ FILE: docs/EVIDENCE-PACK.md ================================================================================ @@ -11454,35 +11535,67 @@ FILE: docs/README.md # hotato docs -Find what broke in your agent calls. Pin it so it never ships again. Every -production failure becomes a portable test, every candidate runs against it, -and every release carries evidence. This index maps every doc to the step it -belongs to. +Find what broke in your agent calls. Pin it so it never ships again. The +loop: `autopsy` one recording (or `scan` a folder, or pull a platform's +recent calls with `vapi health`), `pin` the incident as a portable failure +check, and `prove` re-runs the stored evidence in CI so it never ships +again. This index maps every doc to the step it belongs to. New here? Run `hotato autopsy ./call.wav` on one recording (**[AUTOPSY.md](AUTOPSY.md)**), or start with **[GETTING-STARTED.md](GETTING-STARTED.md)**. The whole loop on one page: **[LIFECYCLE.md](LIFECYCLE.md)**. -## Getting started +## The loop: find what broke, then pin it +- [AUTOPSY.md](AUTOPSY.md) - `autopsy`, `scan`, and the ` health` commands: one recording (or a folder, or your platform's recent calls) in, the incident list and health report out; `pin` graduates an incident +- [EVIDENCE-CONTRACT.md](EVIDENCE-CONTRACT.md) - the four-tier evidence policy: what every verdict stands on +- [CONTRACTS.md](CONTRACTS.md) - failure contracts: a portable CI bundle of one call moment +- [CI.md](CI.md) - `prove` and the CI gate: fail the pull request on the pinned evidence, offline - [GETTING-STARTED.md](GETTING-STARTED.md) - one path from first touch to a CI gate - [START.md](START.md) - guided first run on the bundled demo data -- [STARTER.md](STARTER.md) - `hotato init starter` scaffolds a CI gate and config -- [BAD-CALL-TO-CI.md](BAD-CALL-TO-CI.md) - turn one bad call into a CI gate - [WHY.md](WHY.md) - four timing failures a text-level eval cannot see -## Observe +## Continuous use -- [OBSERVE.md](OBSERVE.md) - LLM and voice observability from your OpenTelemetry spans, locally -- [TRACE.md](TRACE.md) - voice traces: the pipeline-event timeline -- [OTEL.md](OTEL.md) - ingest OTel traces into the `voice_trace` span format -- [latency-waterfall.md](latency-waterfall.md) - per-hop latency waterfall from a scored call +- [CONNECT.md](CONNECT.md) - store a stack's credentials once; feed every hotato command from your platform +- [ADAPTER-STATUS.md](ADAPTER-STATUS.md) - per-stack pull, endpoint, and channel-separation status - [PRODUCTION-MONITORING.md](PRODUCTION-MONITORING.md) - turn production call events into offline regression candidates +- [WORKSPACE.md](WORKSPACE.md) - `hotato serve` and `hotato console`: the self-hosted team web workspace and the live call console +- [SELF-HOST.md](SELF-HOST.md) - run the full workspace in your own VPC + +## Working with audio + +- [TRUST.md](TRUST.md) - is this recording even scorable? +- [TRUST-MATRIX.md](TRUST-MATRIX.md) - the input-condition-to-behaviour contract for the trust check +- [TRUST-GALLERY.md](TRUST-GALLERY.md) - eight recordings, eight verdicts, verbatim output +- [DIARIZE.md](DIARIZE.md) - diarize a mono recording to make it scorable +- [TRANSCRIBE.md](TRANSCRIBE.md) - attach a transcript beside the timing score +- [FULL-DUPLEX.md](FULL-DUPLEX.md) - score the moment both sides speak at once + +## Reference + +- [API.md](API.md) - the stdlib-only scoring core, Python API +- [SDK.md](SDK.md) - the typed Python SDK facade over the CLI +- [MCP.md](MCP.md) - the hotato MCP server and its tools, over stdio +- [METHODOLOGY.md](../METHODOLOGY.md) - how the timing measurement works, end to end +- [THREAT-MODEL.md](THREAT-MODEL.md) - which commands are offline, which reach the network +- [EGRESS.md](EGRESS.md) - every network call site mapped to its command +- [VALIDATION.md](VALIDATION.md) - the three separate jobs hotato is validated on +- [EVIDENCE-PACK.md](EVIDENCE-PACK.md) - the reproducible proof artifacts, ranked +- [GALLERY.md](GALLERY.md) - every image and worked example, each reproducible +- [COMPARE.md](COMPARE.md) - where hotato sits next to broad QA platforms +- [evidence/README.md](evidence/README.md) - the evidence standard: what counts, and ranking +- [case-studies/README.md](case-studies/README.md) - the honesty standard every case study meets + +## Lab: the deep toolkit + +Every command in this section lives under `hotato lab` (`hotato lab --help`). +The public commands above are durable; the lab surface evolves faster, and +every pre-1.17 top-level spelling keeps working unchanged. -## Evaluate +### Investigate and evaluate -- [AUTOPSY.md](AUTOPSY.md) - one recording, zero config: the incident list and report; the folder health report (`scan DIR`) and `pin` to a contract - [INVESTIGATE.md](INVESTIGATE.md) - one recording in, ranked candidate moments out - [ANALYZE.md](ANALYZE.md) - drop a folder, rank and hear the worst moments - [ASSERTIONS.md](ASSERTIONS.md) - deterministic typed assertions over transcript, trace, and timing @@ -11492,7 +11605,14 @@ New here? Run `hotato autopsy ./call.wav` on one recording - [scenarios/dtmf-verification.md](scenarios/dtmf-verification.md) - verify DTMF reached the far end - [scenarios/echo-self-interruption.md](scenarios/echo-self-interruption.md) - diagnose self-interruption from echo bleed -## Test +### Observe + +- [OBSERVE.md](OBSERVE.md) - LLM and voice observability from your OpenTelemetry spans, locally +- [TRACE.md](TRACE.md) - voice traces: the pipeline-event timeline +- [OTEL.md](OTEL.md) - ingest OTel traces into the `voice_trace` span format +- [latency-waterfall.md](latency-waterfall.md) - per-hop latency waterfall from a scored call + +### Test and simulate - [SIMULATE.md](SIMULATE.md) - render a scenario into a deterministic labelled conversation - [CONVERSATION-TEST.md](CONVERSATION-TEST.md) - one file, one call, a per-dimension scorecard @@ -11507,10 +11627,10 @@ New here? Run `hotato autopsy ./call.wav` on one recording - [scenarios/browser-vs-pstn.md](scenarios/browser-vs-pstn.md) - score the same moment through telephony degradation - [scenarios/load-and-recovery.md](scenarios/load-and-recovery.md) - behaviour under concurrent load, with receipts -## Gate +### Fix and gate, the long way -- [CONTRACTS.md](CONTRACTS.md) - failure contracts: a portable CI bundle of one call moment -- [CI.md](CI.md) - gate a pull request on turn-taking timing, offline +- [BAD-CALL-TO-CI.md](BAD-CALL-TO-CI.md) - turn one bad call into a CI gate, step by step +- [STARTER.md](STARTER.md) - `hotato lab init starter` scaffolds a CI gate and config - [PYTEST.md](PYTEST.md) - the pytest fixture and opt-in session gate - [FIX-LOOP.md](FIX-LOOP.md) - the closed loop: find, fix, prove it is fixed - [FIX-PLANS.md](FIX-PLANS.md) - the guarded fix ladder: diagnose, inspect, plan, apply @@ -11521,49 +11641,21 @@ New here? Run `hotato autopsy ./call.wav` on one recording - [CARDS.md](CARDS.md) - render one measured moment as a PR-native SVG card - [scenarios/false-interruption-replay.md](scenarios/false-interruption-replay.md) - a false-stop becomes a contract replayed in CI -## Connect your stack +### Pipe your stack in -- [CONNECT.md](CONNECT.md) - connect, pull, sweep: score every call across stacks -- [ADAPTER-STATUS.md](ADAPTER-STATUS.md) - per-stack pull, endpoint, and channel-separation status - [INGEST.md](INGEST.md) - a passive webhook on-ramp scanning completed calls - [SET-AND-FORGET.md](SET-AND-FORGET.md) - a passive scheduled sweep for regression monitoring - [TRANSPORT-RUNTIME.md](TRANSPORT-RUNTIME.md) - lifecycle, delivered media, and assertion facts across transports - [CALLER-SIDECAR-PROTOCOL.md](CALLER-SIDECAR-PROTOCOL.md) - the caller/transport sidecar WebSocket protocol - [LIVEKIT-CALLER-SESSION.md](LIVEKIT-CALLER-SESSION.md) - direct LiveKit room transport for the caller engine -## Working with audio - -- [TRUST.md](TRUST.md) - is this recording even scorable? -- [TRUST-MATRIX.md](TRUST-MATRIX.md) - the input-condition-to-behaviour contract for the trust check -- [DIARIZE.md](DIARIZE.md) - diarize a mono recording to make it scorable -- [TRANSCRIBE.md](TRANSCRIBE.md) - attach a transcript beside the timing score -- [FULL-DUPLEX.md](FULL-DUPLEX.md) - score the moment both sides speak at once - -## Self-host and team +### Fleet, reports, and benchmarks -- [SELF-HOST.md](SELF-HOST.md) - run the full workspace in your own VPC -- [WORKSPACE.md](WORKSPACE.md) - `hotato serve`: the self-hosted team web workspace - [GUARDIAN-FLEET.md](GUARDIAN-FLEET.md) - a control plane running the evidence workflow continuously - [REPORTS.md](REPORTS.md) - reporting surfaces: doctor, report, team, export - -## Reference - -- [API.md](API.md) - the stdlib-only scoring core, Python API -- [SDK.md](SDK.md) - the typed Python SDK facade over the CLI -- [MCP.md](MCP.md) - the hotato MCP server and its tools, over stdio - [BENCHMARK.md](BENCHMARK.md) - the measurement-error harness over labelled recordings - [BENCH-SPEC.md](BENCH-SPEC.md) - frozen batteries, scoring protocol, verify - [BENCHMARK-STACKS.md](BENCHMARK-STACKS.md) - run one battery through your configured stacks -- [METHODOLOGY.md](../METHODOLOGY.md) - how the timing measurement works, end to end -- [THREAT-MODEL.md](THREAT-MODEL.md) - which commands are offline, which reach the network -- [EGRESS.md](EGRESS.md) - every network call site mapped to its command -- [TRUST-GALLERY.md](TRUST-GALLERY.md) - eight recordings, eight verdicts, verbatim output -- [VALIDATION.md](VALIDATION.md) - the three separate jobs hotato is validated on -- [EVIDENCE-PACK.md](EVIDENCE-PACK.md) - the reproducible proof artifacts, ranked -- [GALLERY.md](GALLERY.md) - every image and worked example, each reproducible -- [COMPARE.md](COMPARE.md) - where hotato sits next to broad QA platforms -- [evidence/README.md](evidence/README.md) - the evidence standard: what counts, and ranking -- [case-studies/README.md](case-studies/README.md) - the honesty standard every case study meets ## Contributing @@ -13932,7 +14024,9 @@ contract: input condition on the left, Hotato's behavior on the right. It covers input health only; a turn-taking verdict is `scan`'s and `run`'s job. Worked examples with CLI output for every row are in -[TRUST-GALLERY.md](TRUST-GALLERY.md). +[TRUST-GALLERY.md](TRUST-GALLERY.md). The four-tier evidence policy the +rows below implement is stated once in +[EVIDENCE-CONTRACT.md](EVIDENCE-CONTRACT.md). ## The contract diff --git a/src/hotato/cli.py b/src/hotato/cli.py index 870900b2..488df4d3 100644 --- a/src/hotato/cli.py +++ b/src/hotato/cli.py @@ -5350,6 +5350,20 @@ def _describe_subcommand(name: str, parser: argparse.ArgumentParser, prefix: str "purpose": parser.description or parser.format_usage().strip(), "args": args, } + if not prefix: + # Top-level commands carry their surface classification: the canonical + # spelling (public commands at the top level, everything else under + # ``hotato lab``) and, for lab commands, the pre-1.17 top-level + # spelling that keeps working as a compat alias. Nested subcommands + # inherit the surface of their top-level parent. + if name in _PUBLIC_SURFACE: + out["surface"] = "public" + out["canonical"] = name + out["compat_alias"] = None + else: + out["surface"] = "lab" + out["canonical"] = f"lab {name}" + out["compat_alias"] = name if full_name in _EXIT_CODES: out["exit_codes"] = [ {"code": code, "meaning": meaning} @@ -5391,6 +5405,7 @@ def _schema_id(filename: str) -> str: "tool": _errors.TOOL, "schema_version": _errors.SCHEMA_VERSION, "version": __version__, + "stability": " ".join(_STABILITY_STATEMENT.split()), "core_loop": [ {"step": i, "command": cmd, "purpose": blurb} for i, (cmd, blurb) in enumerate(_CORE_LOOP_STEPS, 1) @@ -5411,6 +5426,8 @@ def _render_describe_text(manifest: dict) -> str: lines.append("schemas: " + " ".join( f"{name}={url}" for name, url in manifest["schemas"].items() )) + if manifest.get("stability"): + lines.append(manifest["stability"]) lines.append("") if manifest.get("core_loop"): lines.append("core loop (start here; every other command is advanced):") @@ -5420,9 +5437,16 @@ def _render_describe_text(manifest: dict) -> str: ) lines.append("") - def _walk(cmds, indent=""): + def _walk(cmds, indent="", lab=False): for c in cmds: - lines.append(f"{indent}hotato {c['name']}") + # A lab command renders under its canonical `hotato lab` spelling + # (nested subcommand names embed the parent's, so the prefix + # composes), with the still-working top-level spelling beside it. + lab_here = lab or c.get("surface") == "lab" + path = f"lab {c['name']}" if lab_here else c["name"] + lines.append(f"{indent}hotato {path}") + if c.get("compat_alias"): + lines.append(f"{indent} (also invocable as: hotato {c['compat_alias']})") if c.get("purpose"): lines.append(f"{indent} {c['purpose']}") for a in c["args"]: @@ -5432,7 +5456,7 @@ def _walk(cmds, indent=""): codes = ", ".join(f"{e['code']}={e['meaning']}" for e in c["exit_codes"]) lines.append(f"{indent} exit codes: {codes}") if c.get("subcommands"): - _walk(c["subcommands"], indent + " ") + _walk(c["subcommands"], indent + " ", lab_here) _walk(manifest["subcommands"]) return "\n".join(lines) + "\n" @@ -5550,6 +5574,104 @@ def _get_started_block() -> str: return "\n".join(lines) +# --- the public / lab surface split (1.17.0, "The Narrowing") --------------- +# +# ONE registration layer decides every command's visibility. `hotato --help` +# lists ONLY the durable public surface below. Every other command registers +# exactly as before -- same name, same parser object, same flags, same +# behavior -- but WITHOUT a help line: an argparse subparser is listed in the +# parent's help only when ``add_parser`` received a ``help=`` kwarg, so +# dropping it hides the entry while keeping the command fully callable. That +# unlisted registration IS the back-compat alias: a pre-1.17 invocation +# parses through the identical parser and stays byte-identical. +# +# ``hotato lab `` is the canonical spelling for every hidden command. +# The ``lab`` prefix is pure dispatch: main() strips the leading ``lab`` +# token before parsing (the same argv-rewrite technique +# :func:`_route_bare_folder` and :func:`_route_investigate_label` already +# use), so both spellings run one parser. ``hotato lab --help`` renders the +# lab listing from the help lines this layer collects at registration time, +# and ``hotato describe`` records both spellings per command. +_PUBLIC_SURFACE = frozenset({ + # Start here + "autopsy", "scan", "pin", "prove", "connect", + # health (five stacks) + "vapi", "retell", "bland", "synthflow", "millis", + # Onboarding + "start", "demo", "doctor", + # Continuous use + "console", "production", "serve", + # Checks + "contract", + # Agent-native + "describe", +}) + +# The stability statement, stated once and rendered on every surface that +# names the split: the --help epilog tail, `hotato lab --help`, README's +# Go-deeper section, CONTRIBUTING.md, and the describe manifest. +_STABILITY_STATEMENT = ( + "The public commands are durable. hotato lab evolves faster, and every\n" + "pre-1.17 top-level spelling keeps working unchanged." +) + + +class _SurfaceRouter: + """The single registration layer for the public/lab split. + + Wraps the top-level subparsers action. A public command registers + untouched (and so is listed in ``hotato --help``). Any other command has + its ``help=`` recorded for the ``hotato lab`` listing and then stripped, + so argparse registers the parser without listing it -- the hidden + back-compat spelling and the ``hotato lab`` spelling are one parser. + Everything else (``choices`` for :func:`ops_cli.register`'s collision + check, the manifest walk) delegates to the wrapped action. + """ + + def __init__(self, action: argparse._SubParsersAction): + self._action = action + self.lab_commands = [] # (name, one-line help) in registration order + + def add_parser(self, name: str, **kwargs) -> argparse.ArgumentParser: + if name not in _PUBLIC_SURFACE: + help_line = kwargs.pop("help", None) or "" + self.lab_commands.append((name, " ".join(help_line.split()))) + return self._action.add_parser(name, **kwargs) + + def __getattr__(self, attr): + return getattr(self._action, attr) + + +def _render_lab_help(parser: argparse.ArgumentParser) -> str: + """The ``hotato lab --help`` screen: every lab command with its one-line + description, from the help lines :class:`_SurfaceRouter` collected.""" + import textwrap + + lines = [ + "usage: hotato lab ...", + "", + "hotato lab: the deep toolkit behind the public surface -- capture,", + "simulation, load, benchmarking, the fix ladder, the fleet control", + "plane, and every other specialist command.", + "", + ] + lines.extend(textwrap.wrap(_STABILITY_STATEMENT, width=76)) + lines.append("") + lines.append("commands:") + for name, help_line in sorted(getattr(parser, "_lab_commands", ())): + if not help_line: + lines.append(f" {name}") + continue + wrapped = textwrap.wrap(help_line, width=58) or [""] + lines.append(f" {name.ljust(19)} {wrapped[0]}") + lines.extend(f"{' ' * 22}{cont}" for cont in wrapped[1:]) + lines.append("") + lines.append("Full machine-readable list of every command: " + "hotato describe --format json") + lines.append("Exit codes: 0 pass, 1 regression, 2 refuse.") + return "\n".join(lines) + "\n" + + def build_parser() -> argparse.ArgumentParser: p = argparse.ArgumentParser( prog="hotato", @@ -5570,20 +5692,25 @@ def build_parser() -> argparse.ArgumentParser: " scan the folder health report over a directory of recordings\n" " vapi health pull recent Vapi calls and write the health report\n" " (same shape: retell, bland, synthflow, millis)\n" - " pin pin one autopsy incident as a portable failure check\n\n" + " pin pin one autopsy incident as a portable failure check\n" + " prove the CI check: every evidence lane composed, fail-closed\n" + " connect store a stack's credentials once (0600, local only)\n\n" + "Onboarding:\n" + " start guided, credential-less first run on the bundled demo calls\n" + " demo run the packaged battery of two failing calls, open its report\n" + " doctor score a recording (or self-test), render the report, open it\n\n" "Continuous use:\n" " vapi health run it on a schedule and watch the trend\n" " console the call console over the production evidence store\n" " production the durable production evidence plane\n" " serve the self-hosted local team workspace\n\n" - "CI and checks:\n" - " prove the CI check: every evidence lane composed, fail-closed\n" + "Checks:\n" " contract create, verify, and pack pinned failure checks\n" - " suite run a suite of conversation-tests, per-dimension report\n" - " verify re-execute a bench result's pinned battery, hash-compare\n" - " gauntlet the bundled turn-taking stress suite\n\n" - "Everything else is advanced. Full machine-readable list of every " - "command: hotato describe --format json\n" + " (pin's artifacts verify through it)\n\n" + "Agent-native:\n" + " describe the generated machine-readable manifest of every command\n\n" + "Everything else lives under hotato lab (see: hotato lab --help).\n" + + _STABILITY_STATEMENT + "\n" "Exit codes: 0 pass, 1 regression, 2 refuse." ), formatter_class=argparse.RawDescriptionHelpFormatter, @@ -5591,10 +5718,14 @@ def build_parser() -> argparse.ArgumentParser: p.add_argument("--version", action="version", version=f"hotato {__version__}") # Not required: bare `hotato` prints the first-run guide (score your OWN call), # rather than an argparse usage error. - # metavar collapses the 50+ command names into one token in the usage line and + # metavar collapses the command names into one token in the usage line and # the positional-arguments header, so a newcomer reads the GET STARTED loop - # first; each command still lists (with its help) below, and all stay callable. - sub = p.add_subparsers(dest="command", required=False, metavar="") + # first. The router narrows the listing below it to the public surface; + # every other command registers unlisted and stays callable (see + # _SurfaceRouter above). + sub = _SurfaceRouter( + p.add_subparsers(dest="command", required=False, metavar="") + ) r = sub.add_parser( "run", @@ -10927,6 +11058,10 @@ def _fleet_parser(parent, name, dotted, help_text): _ops_cli.register(sub, epilog_factory=_exit_codes_epilog) + # The lab listing (`hotato lab --help`) renders from the (name, help) + # pairs the router collected while the commands above registered. + p._lab_commands = tuple(sub.lab_commands) + return p @@ -10966,9 +11101,40 @@ def _route_investigate_label(argv) -> "list | None": return None +def _route_lab(argv, parser) -> "list | int | None": + """``hotato lab ...``: the canonical spelling for every command the + top-level help no longer lists. The ``lab`` prefix is pure dispatch -- + strip it and parse the remainder through the same registered command, so + ``hotato lab simulate`` and the pre-1.17 ``hotato simulate`` run one + parser and stay byte-identical (the same argv-rewrite technique + :func:`_route_bare_folder` uses). Returns the rewritten argv; an int exit + code when ``lab`` itself handled the invocation (its --help listing, or + an unknown-command refusal); None to leave the argv untouched.""" + if not argv or argv[0] != "lab": + return None + rest = argv[1:] + if not rest or rest[0] in ("-h", "--help"): + print(_render_lab_help(parser), end="") + return 0 + known = set() + for action in parser._actions: + if isinstance(action, argparse._SubParsersAction): + known = set(action.choices) + break + if rest[0] not in known: + print(f"error: unknown lab command {rest[0]!r} " + "(see: hotato lab --help)", file=sys.stderr) + return 2 + return list(rest) + + def main(argv=None) -> int: parser = build_parser() raw = sys.argv[1:] if argv is None else list(argv) + lab = _route_lab(raw, parser) + if isinstance(lab, int): + return lab + raw = lab if lab is not None else raw raw = _route_investigate_label(raw) or raw rerouted = _route_bare_folder(raw, parser) args = parser.parse_args(rerouted if rerouted is not None else raw) diff --git a/tests/test_counterexample_cli.py b/tests/test_counterexample_cli.py index 11e5d35e..b659f631 100644 --- a/tests/test_counterexample_cli.py +++ b/tests/test_counterexample_cli.py @@ -82,8 +82,11 @@ def test_cli_budget_exhaustion_writes_capsule_and_exits_one(tmp_path, capsys): assert output.is_dir() -def test_parent_help_promises_explicit_proof_status_not_unconditional_minimality(): - help_text = " ".join(cli.build_parser().format_help().split()) +def test_lab_listing_promises_explicit_proof_status_not_unconditional_minimality(): + # counterexample lives on the lab surface (1.17.0 narrowing), so its + # one-line description renders in `hotato lab --help` rather than the + # top-level listing. + help_text = " ".join(cli._render_lab_help(cli.build_parser()).split()) assert "offline regression capsule with explicit proof status" in help_text assert "into a minimal, portable regression capsule" not in help_text diff --git a/tests/test_describe_cli.py b/tests/test_describe_cli.py index 068ff77f..5520e1ed 100644 --- a/tests/test_describe_cli.py +++ b/tests/test_describe_cli.py @@ -206,6 +206,36 @@ def _flat(cmds): assert [(e["code"], e["meaning"]) for e in manifest_codes] == list(codes) +def test_describe_json_classifies_every_top_level_command_by_surface(capsys): + """Every top-level command carries its surface (public or lab), its + canonical spelling, and -- for lab commands -- the pre-1.17 top-level + spelling that keeps working as a compat alias. The registry stays + truthful about BOTH paths (1.17.0 narrowing).""" + cli.main(["describe", "--format", "json"]) + manifest = json.loads(capsys.readouterr().out) + + surfaces = {c["name"]: c for c in manifest["subcommands"]} + assert {c["surface"] for c in surfaces.values()} == {"public", "lab"} + for name, c in surfaces.items(): + if c["surface"] == "public": + assert name in cli._PUBLIC_SURFACE + assert c["canonical"] == name + assert c["compat_alias"] is None + else: + assert name not in cli._PUBLIC_SURFACE + assert c["canonical"] == f"lab {name}" + assert c["compat_alias"] == name + # The split covers the whole registered top level. + assert {n for n, c in surfaces.items() if c["surface"] == "public"} \ + == cli._PUBLIC_SURFACE + + +def test_describe_json_carries_the_stability_statement(capsys): + cli.main(["describe", "--format", "json"]) + manifest = json.loads(capsys.readouterr().out) + assert manifest["stability"] == " ".join(cli._STABILITY_STATEMENT.split()) + + def test_describe_json_is_deterministic(capsys): cli.main(["describe", "--format", "json"]) first = capsys.readouterr().out @@ -221,9 +251,14 @@ def test_describe_text_is_readable_and_deterministic(capsys): assert code == 0 first = capsys.readouterr().out assert "hotato" in first - assert "hotato run" in first - assert "hotato benchmark compare" in first - assert "hotato fixture create" in first + # Lab commands render under their canonical `hotato lab` spelling, with + # the still-working top-level spelling stated beside them. + assert "hotato lab run" in first + assert "hotato lab benchmark compare" in first + assert "hotato lab fixture create" in first + assert "(also invocable as: hotato run)" in first + # Public commands render at the top level, unprefixed. + assert "hotato autopsy" in first assert "exit codes:" in first cli.main(["describe", "--format", "text"]) diff --git a/tests/test_evidence_contract.py b/tests/test_evidence_contract.py new file mode 100644 index 00000000..7d1066aa --- /dev/null +++ b/tests/test_evidence_contract.py @@ -0,0 +1,55 @@ +"""docs/EVIDENCE-CONTRACT.md is the ONE statement of the four-tier evidence +policy; every surface that narrates the policy links to it instead of +restating it. This lockstep keeps the page and its referencing surfaces from +drifting apart: the page must state all four tiers, and each surface that +applies the policy must point at the page. +""" + +import pathlib + +ROOT = pathlib.Path(__file__).resolve().parent.parent +PAGE = ROOT / "docs" / "EVIDENCE-CONTRACT.md" + +# Every surface that narrates the mono/dual-channel evidence split must link +# to the one policy page rather than growing its own divergent copy. +_REFERENCING_SURFACES = [ + "README.md", + "docs/README.md", + "docs/AUTOPSY.md", + "docs/TRUST.md", + "docs/TRUST-MATRIX.md", +] + +# The four tiers, by the phrase that names each one's behavior. +_TIER_MARKERS = [ + "dual-channel audio: deterministic", + "attributable, with declared authority", + "symptom detection, with measured confidence", + "refused, with the remediation", +] + + +def test_the_page_states_all_four_tiers(): + text = PAGE.read_text(encoding="utf-8") + for marker in _TIER_MARKERS: + assert marker in text, f"EVIDENCE-CONTRACT.md lost its tier: {marker!r}" + + +def test_every_referencing_surface_links_to_the_page(): + missing = [] + for rel in _REFERENCING_SURFACES: + if "EVIDENCE-CONTRACT.md" not in (ROOT / rel).read_text(encoding="utf-8"): + missing.append(rel) + assert not missing, ( + "these surfaces narrate the evidence policy but no longer link to " + f"docs/EVIDENCE-CONTRACT.md (the one source of truth): {missing}") + + +def test_the_page_keeps_the_exit_code_and_denominator_facts(): + """The two load-bearing facts the page asserts about the runtime: a + refusal is exit 2, and only dual-channel calls enter the Voice Stability + denominator. If either claim leaves the page, the surfaces linking here + lose the policy they defer to.""" + text = PAGE.read_text(encoding="utf-8") + assert "exit `2`" in text + assert "Voice Stability denominator" in text diff --git a/tests/test_lab_surface.py b/tests/test_lab_surface.py new file mode 100644 index 00000000..57b0b9a0 --- /dev/null +++ b/tests/test_lab_surface.py @@ -0,0 +1,147 @@ +"""The 1.17.0 public/lab surface split ("The Narrowing"). + +One registration layer (``cli._SurfaceRouter``) decides every command's +visibility: the top-level ``--help`` lists ONLY the durable public surface; +every other command registers unlisted under its pre-1.17 name (the compat +alias, byte-identical behavior) and gets ``hotato lab `` as its +canonical spelling. Pinned here: + + * the top-level help listing is exactly the public surface -- no lab + command leaks in, no public command drops out; + * every lab command stays fully callable under its old top-level spelling; + * ``hotato lab `` and ``hotato `` produce byte-identical output; + * ``hotato lab --help`` lists every lab command with its one-line + description plus the stability statement; + * the stability statement renders in the top-level --help epilog tail. +""" + +import argparse + +import pytest + +from hotato import cli + + +def _top_subparsers(parser): + for action in parser._actions: + if isinstance(action, argparse._SubParsersAction): + return action + raise AssertionError("no top-level subparsers action") + + +def _listed_commands(parser): + """The command names the top-level --help actually lists (argparse renders + one pseudo-action per add_parser call that passed ``help=``).""" + return {a.dest for a in _top_subparsers(parser)._choices_actions} + + +def _lab_names(parser): + return {name for name, _ in parser._lab_commands} + + +# --- top-level help = exactly the public surface --------------------------- + +def test_top_level_help_lists_exactly_the_public_surface(): + parser = cli.build_parser() + assert _listed_commands(parser) == cli._PUBLIC_SURFACE + + +def test_every_registered_command_is_public_or_lab_never_both(): + parser = cli.build_parser() + registered = set(_top_subparsers(parser).choices) + lab = _lab_names(parser) + assert lab == registered - cli._PUBLIC_SURFACE + assert not (lab & cli._PUBLIC_SURFACE) + # the public surface is fully registered, not aspirational + assert cli._PUBLIC_SURFACE <= registered + + +def test_top_level_help_epilog_carries_the_stability_statement(capsys): + with pytest.raises(SystemExit): + cli.main(["--help"]) + out = capsys.readouterr().out + for line in cli._STABILITY_STATEMENT.splitlines(): + assert line in out + assert "hotato lab --help" in out + + +# --- the compat aliases: old spellings keep working ------------------------ + +def test_lab_commands_stay_callable_under_their_old_top_level_spelling(): + parser = cli.build_parser() + choices = _top_subparsers(parser).choices + for name in _lab_names(parser): + assert name in choices, f"compat alias {name!r} lost its registration" + + +def test_lab_spelling_and_compat_spelling_are_byte_identical(capsys): + # --help output is generated from the one shared parser, so comparing it + # end to end proves both spellings dispatch through identical machinery. + with pytest.raises(SystemExit) as exc: + cli.main(["gauntlet", "--help"]) + assert exc.value.code in (0, None) + via_alias = capsys.readouterr().out + + with pytest.raises(SystemExit) as exc: + cli.main(["lab", "gauntlet", "--help"]) + assert exc.value.code in (0, None) + via_lab = capsys.readouterr().out + + assert via_alias == via_lab + + +def test_lab_investigate_label_routes_like_the_top_level_spelling(capsys): + with pytest.raises(SystemExit) as exc: + cli.main(["lab", "investigate", "label", "--help"]) + assert exc.value.code in (0, None) + via_lab = capsys.readouterr().out + + with pytest.raises(SystemExit) as exc: + cli.main(["investigate", "label", "--help"]) + assert exc.value.code in (0, None) + assert via_lab == capsys.readouterr().out + + +def test_lab_run_executes_the_command(capsys): + # a real execution through the lab spelling, not just help text + assert cli.main(["lab", "run", "--suite", "barge-in", + "--format", "json"]) == 0 + capsys.readouterr() + + +# --- hotato lab --help ------------------------------------------------------ + +def test_lab_help_lists_every_lab_command_with_a_description(capsys): + assert cli.main(["lab", "--help"]) == 0 + out = capsys.readouterr().out + parser = cli.build_parser() + for name, help_line in parser._lab_commands: + assert f"\n {name}" in out, f"lab listing omits {name!r}" + if help_line: + # the one-line description survives (wrapped, so match its head) + assert help_line.split()[0] in out + for line in cli._STABILITY_STATEMENT.splitlines(): + # the statement is re-wrapped in the listing; match word-wise + for word in ("durable", "pre-1.17"): + assert word in out + + +def test_bare_lab_prints_the_listing_too(capsys): + assert cli.main(["lab"]) == 0 + assert "hotato lab" in capsys.readouterr().out + + +def test_unknown_lab_command_refuses_with_exit_2(capsys): + assert cli.main(["lab", "definitely-not-a-command"]) == 2 + err = capsys.readouterr().err + assert "unknown lab command" in err + assert "hotato lab --help" in err + + +# --- no public command is reachable only through lab ------------------------ + +def test_public_commands_keep_their_top_level_help(capsys): + with pytest.raises(SystemExit) as exc: + cli.main(["autopsy", "--help"]) + assert exc.value.code in (0, None) + assert "autopsy" in capsys.readouterr().out