Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 43 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,49 @@ Every entry reports millisecond measurement error and a confusion matrix. See `d

## [Unreleased]

### Added
- **`hotato lab` -- the deep toolkit gets its own front door.** The
top-level `--help` now shows only the public surface (`autopsy`, `scan`,
`pin`, `prove`, `connect`, the five `<stack> health` commands, `start`,
`demo`, `doctor`, `console`, `production`, `serve`, `contract`,
`describe`); every other command lives under `hotato lab <cmd>`, and
`hotato lab --help` lists each one with its one-line description. One
registration layer (`_SurfaceRouter`) decides visibility: a lab command
registers exactly as before -- same name, same parser, same behavior --
but without a parent help line, so every pre-1.17 top-level spelling
keeps working unchanged as a compat alias, and the `lab` prefix is pure
argv dispatch onto the same parser. The stability statement (public
commands durable; lab evolves faster; old spellings keep working)
renders in the `--help` epilog tail, `hotato lab --help`, README,
CONTRIBUTING (with the monthly-stable release cadence), and the
`describe` manifest.
- **`hotato describe` records both spellings per command.** Every
top-level manifest entry now carries `surface` (`public` or `lab`), its
`canonical` spelling, and `compat_alias` (the still-working pre-1.17
top-level name; `null` for public commands), plus a top-level
`stability` field. The text rendering shows lab commands under
`hotato lab ...` with the alias beside them. Exit codes, args, and the
command set itself are unchanged.
- **`docs/EVIDENCE-CONTRACT.md` -- the four-tier evidence policy, stated
once.** Tier 1 dual-channel audio (deterministic, the only
verdict-eligible tier), tier 2 mono plus provider metadata (attributable
findings, each carrying its source's declared authority), tier 3 raw
mixed mono (symptom detection with a measured confidence), tier 4
insufficient evidence (refused, exit 2, with the remediation). README,
AUTOPSY.md, the trust docs, and the docs index link to it instead of
restating it; `tests/test_evidence_contract.py` holds the page and its
referencing surfaces in lockstep.

### Changed
- **docs index restructured to the loop.** `docs/README.md` now leads with
the loop (autopsy -> scan -> pin -> prove -> `<stack> health`), then
continuous use, then a Lab section grouping the moved commands' docs.
Every doc stays listed.
- **Superlative sweep.** Unsupported superlatives replaced with measured
framing: the README say-do bullet now reads "a bug that fires actions
the caller canceled", and `docs/STARTER.md` opens with what the one
command does.

## [1.16.0] - 2026-07-24

### Added
Expand Down
34 changes: 28 additions & 6 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,11 @@
# Contributing to hotato

Thanks for being here. hotato is local-first testing and observability for AI agents: it simulates,
evaluates, reviews, and tracks calls across five dimensions (outcome, policy,
conversation, speech, reliability), with the evidence behind every result.
Deterministic checks stay separate from the model-judged rubric, and no output is
ever a single blended score.
Thanks for being here. hotato finds what broke in your agent calls and pins
it so it never ships again: it simulates, evaluates, reviews, and tracks
calls across five dimensions (outcome, policy, conversation, speech,
reliability), with the evidence behind every result. Deterministic checks
stay separate from the model-judged rubric, and no output is ever a single
blended score.

Every kind of contribution is welcome. This guide gets your first one merged with
the least friction, then points you at the contribution that helps most.
Expand Down Expand Up @@ -63,11 +64,32 @@ Try the tool end to end with no account, keys, or network:

```bash
hotato start --demo # sweep two bundled calls, write a dashboard, verify a contract
hotato --help # every subcommand
hotato --help # the public surface
hotato lab --help # the deep toolkit behind it
```

Then keep the diff small: one scenario, one fix, or one recording per PR.

## Surface stability and release cadence

The CLI has two surfaces, and they carry different stability promises:

- **Public** (everything `hotato --help` lists: `autopsy`, `scan`, `pin`,
`prove`, `connect`, the `<stack> health` commands, `start`, `demo`,
`doctor`, `console`, `production`, `serve`, `contract`, `describe`).
These commands, their flags, and their exit codes are durable: a script
or CI job written against them keeps working across releases.
- **Lab** (everything `hotato lab --help` lists). The lab surface evolves
faster between releases. Every pre-1.17 top-level spelling keeps working
unchanged as a compat alias, and `hotato describe` records both
spellings for every command.

Releases follow a **monthly-stable cadence**: a versioned release ships
about once a month, carries the full CHANGELOG entry for everything that
landed, and is the unit the stability promise attaches to. Any change to a
public command's flags or exit codes is called out in the CHANGELOG at
that release.

## The highest-value contribution: a labeled call recording

hotato gets more credible with every consented, de-identified call clip in the
Expand Down
10 changes: 8 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ No judges. No cloud. No bill. MIT.

## What it finds

- **Barge-in → Say-do gaps**: Caller interrupts to cancel; agent says "canceled" but the booking tool still fires. The most expensive voice AI bug. (Timing from the audio; the tool-fire check reads your call's tool log: hotato ingests Vapi/OTel traces.)
- **Barge-in → Say-do gaps**: Caller interrupts to cancel; agent says "canceled" but the booking tool still fires: a bug that fires actions the caller canceled. (Timing from the audio; the tool-fire check reads your call's tool log: hotato ingests Vapi/OTel traces.)
- **Latency spikes**: 800ms → 5s unpredictability that makes users hang up.
- **Dead air**: Long silences that kill conversation flow.
- **Talk-over**: Agent speaks over the caller; never yields.
Expand Down Expand Up @@ -122,8 +122,14 @@ The whole loop, command by command: [`docs/LIFECYCLE.md`](docs/LIFECYCLE.md).
First touch to a CI gate: [`docs/GETTING-STARTED.md`](docs/GETTING-STARTED.md).
Feed it what you already have: [`docs/CONNECT.md`](docs/CONNECT.md) &#183;
[`docs/TRACE.md`](docs/TRACE.md) &#183; [`docs/SIMULATE.md`](docs/SIMULATE.md).
What every verdict stands on: [`docs/EVIDENCE-CONTRACT.md`](docs/EVIDENCE-CONTRACT.md).
Next to the hosted alternatives: [`docs/COMPARE.md`](docs/COMPARE.md).

The deep toolkit -- capture, simulation, load, benchmarking, the fix ladder,
the fleet control plane -- lives under `hotato lab` (`hotato lab --help`).
The public commands are durable; `hotato lab` evolves faster; every pre-1.17
top-level spelling keeps working unchanged.

## Specifications

| Property | Value |
Expand All @@ -144,7 +150,7 @@ PYTHONPATH=src python3 -m hotato.benchmark \

On 13 recorded AMI Meeting Corpus clips, the median error between measured caller-onset and the human word-alignment label is **20 ms**. Provenance: [`corpus/real/README.md`](corpus/real) &#183; method: [`METHODOLOGY.md`](METHODOLOGY.md).

Timing is measurable only when the two voices arrive on separate channels; a mono or mixed export is marked **NOT SCORABLE** and refused (`hotato trust --stereo call.wav`).
Timing is measurable only when the two voices arrive on separate channels; a mono or mixed export is marked **NOT SCORABLE** and refused (`hotato trust --stereo call.wav`). The full four-tier evidence policy (what each verdict stands on, per input) is [`docs/EVIDENCE-CONTRACT.md`](docs/EVIDENCE-CONTRACT.md).

</details>

Expand Down
10 changes: 8 additions & 2 deletions README.pypi.md

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

12 changes: 8 additions & 4 deletions docs/AUTOPSY.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,9 +74,10 @@ mixed channel; talk-over attribution comes from a two-channel recording
-- the scope line states this once per run). Mono calls report into the
best-effort mono observations block with their own counts and never
enter the Voice Stability denominator, so the mono stacks' reports carry
observations without a stability score. The analysis runs on this
machine; recordings download straight from the platform and go nowhere
else.
observations without a stability score
([EVIDENCE-CONTRACT.md](EVIDENCE-CONTRACT.md) states the whole tier
policy). The analysis runs on this machine; recordings download straight
from the platform and go nowhere else.

A window with no calls, a pull in which every recording failed to fetch,
or a pulled set with zero analyzable calls refuses with the reason (exit
Expand Down Expand Up @@ -120,7 +121,10 @@ nothing is guessed and no confidence is invented.

The stricter commands keep their bar: `run`, `scan`, `trust`, and the
contract path still refuse mono as NOT SCORABLE. Autopsy is discovery;
the CI gate stays deterministic and dual-channel.
the CI gate stays deterministic and dual-channel. The full four-tier
policy behind this split -- dual-channel deterministic, mono with
provider metadata, raw mixed mono, refused -- is stated once in
[EVIDENCE-CONTRACT.md](EVIDENCE-CONTRACT.md).

An unreadable input -- a text file, a truncated header, a non-audio blob
-- is refused with the reason (exit 2), never scored.
Expand Down
64 changes: 64 additions & 0 deletions docs/EVIDENCE-CONTRACT.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
# The evidence contract: four tiers, one policy

Every hotato verdict states what evidence it stands on. This page is the
single source of truth for that policy; the README, [AUTOPSY.md](AUTOPSY.md),
the platform health reports, and the trust docs
([TRUST.md](TRUST.md) &#183; [TRUST-MATRIX.md](TRUST-MATRIX.md)) all apply it
and link back here.

The four tiers, from strongest evidence to none:

## Tier 1 -- dual-channel audio: deterministic

Caller on one channel, agent on the other, separated at capture. Overlap and
timing are facts of the recording, exact to the sample, so the deterministic
timing walk runs end to end: byte-identical output for the same input, every
threshold exposed, every frame inspectable. This is the only tier that is
**verdict-eligible**: contracts, `hotato pin`, `hotato prove`, and the CI
gate all stand on it, and it alone enters the Voice Stability denominator.

## Tier 2 -- mono plus provider metadata: attributable, with declared authority

A mixed single channel cannot attribute energy to a speaker by itself, but
metadata a provider or pipeline supplies alongside it can: a
speaker-attributed transcript, a voice-pipeline trace, tool-call logs, or a
diarizer's turn labels (`hotato run --mono call.wav --diarize`,
[DIARIZE.md](DIARIZE.md)). Findings on this tier are **attributable**, and
each one carries the **declared authority** of the source that attributed it
-- the production evidence plane records it explicitly (`submitted`,
`adapter_reported`, `provider_export`, `signed_attestation`, `measured`),
and a diarized verdict is tiered and stamped (`diarized-mono`;
`indicative_only` below the confidence bar). Attribution is only as strong
as its source, so the authority prints with the finding and this tier never
merges into Tier 1's deterministic counts.

## Tier 3 -- raw mixed mono: symptom detection, with measured confidence

One mixed channel with no metadata still measures what silence shows:
dead air and latency gaps. `hotato autopsy` and the mono-stack health
reports run this path best-effort, and every finding carries a **measured
confidence** with its derivation printed beside it. A mono gap says
everything stopped, not who stopped -- talk-over and barge-in attribution
comes from the tiers above, that scope is stated once per run, and Tier 3
findings report in their own block, outside the Voice Stability
denominator.

## Tier 4 -- insufficient evidence: refused, with the remediation

An input that supports none of the above -- an unreadable file, a silent
required channel, a mixed export where a deterministic verdict was asked
for -- is **refused** (exit `2`) with the reason and the next step: the
recording scaffold for your stack (`hotato setup`), the input health check
(`hotato trust --stereo call.wav`), or the mono escapes above. A refusal
leaves no artifact and never becomes a number; a green or red build always
means something.

## Reading a report against the tiers

The health and scan reports render the split directly: the Voice Stability
Score and the `health:` share count **dual-channel calls only** (Tier 1);
mono findings sit in the *Best-effort mono observations* block with their
own counts (Tier 3, or Tier 2 where metadata attributed them); refused
files are listed with their reasons (Tier 4), never scored. The per-lane
**evidence coverage** block states which tiers this run actually had --
a lane whose evidence was absent never renders as assessed.
Loading