Scoreboard cannot distinguish model failures from check bugs - proposal: append-only amendment rows

[jjrdar]

Problem

The executed check's exit code is the only verdict ringer records. When the check itself is wrong (over-strict grep, unsatisfiable assertion), the FAIL lands on the model's eval row, and since first_try_pass_rate drives the promotion ladder, check bugs demote models invisibly.

The only correction channel today is a prose annotation in docs/MODEL-NOTES.md - human-read, not machine-read. The scoreboard math never hears about it.

Evidence

• Run stm-nav-restructure (2026-07-17): two recorded FAILs (task-05-pipeline-trim-admin, task-09-condense-tableau) were orchestrator check bugs; the worker output was audited fully correct both times. One check's negative phrase-grep matched the exact link-out stub the spec requested; the other demanded a repo state the spec forbade the worker from creating.
• docs/MODEL-NOTES.md already carries at least three earlier "recorded FAIL was a check bug" annotations from other runs - this is a recurring class, not a one-off.

Proposal

Append-only amendment rows, written by an explicit command, excluded from aggregation:

./ringer.py amend <run_id> <task_key> --reclassify check_bug --note "why"

• Appends {"type": "amendment", "run_id", "task_key", "reclassify", "note", "amended_at", "identity"} to runs.jsonl; never edits existing rows.
• aggregate_model_log_rows (ringer.py:5357) skips attempts whose (run_id, task_key) is amended; groups gain a visible amended count on the models table and HTML page.
• Amendment covers the whole task (all attempts) - if the check was wrong, everything it graded is meaningless.
• Non-goal: auto-detecting check bugs. Attribution requires judgment; the command trusts its caller and requires the note for the audit trail.

Full design note with schema, touch points, and acceptance criteria: docs/AMENDMENT-ROWS.md (committed alongside a planned patch).

[barthballard]

Strong +1 on this, and on the append-only / caller-trusted / note-required shape specifically. We built basically the same thing in our fork a couple weeks ago (a mark --check-fault retro-mark) after hitting this class of bug more than once, so a few notes from running it:

1. Put the skip before the group is created, not after. In aggregate_model_log_rows both rates share one denominator (group["tasks"]), so a whole-task skip drops the task from pass_rate and first_try_pass_rate together, which is clean. The thing to watch is placement: if the skip lands after the groups.setdefault(...), a (model, task_type) pair whose only task was amended survives as a group with tasks == 0, and the if tasks_count else 0.0 guard renders it as 0.00 first-try rather than dropping out of the table. That reads as total failure on exactly the scoreboard you are trying to correct.

2. Void, do not flip to PASS. Your --reclassify check_bug already implies this, but worth saying out loud: the attempt should be evidence-void, neither credit nor blame. Flipping to PASS over-credits a model whose output was never actually validated, which is the same scoreboard distortion pointed the other way.

3. Whole-task granularity is the right default. Your reasoning ("if the check was wrong, everything it graded is meaningless") holds for the common case. The one edge we hit: a task where the check was only wrong on a subset of attempts (the check got corrected mid-run). Rare, and whole-task is still the sane default, but the (run_id, task_key) key leaves room for an optional attempt filter later if anyone ever needs it.

4. Surface the amendment on the run view, not just the aggregate models table. Someone auditing a specific run wants to see "task-05: amended, check_bug, <note>" inline where the FAIL was recorded, so the correction sits next to the thing it corrects. Otherwise the per-run report still shows a bare FAIL and only the rollup knows better.

One question on the shape of amend itself: a check bug (a wrong FAIL) and a human rejection of a bad PASS are mirror images. Both are an after-the-fact correction to a recorded verdict, and both want the same primitive: append-only, caller-trusted, note-required, audit-logged. If a human-override path ever lands next to this, building amend general enough to carry both directions from the start leaves you one audit model instead of two. Worth not foreclosing even if you don't build it now.

The non-goal is right, attribution needs judgment and should stay human-driven. Happy to dig into any of these.