fix: grading and triage defects the first full-60 sweep exposed - #117
Merged
Conversation
Both bugs surfaced on the first full-60 sweep, in the two scripts that only run once a sweep has finished — so neither could have been caught before now. `sweep_status.py` classified a model's non-terminating code as an infrastructure transient. The transient pattern matched a bare `timed out`, which covers both a provider call failing and the harness's own 30s budget on the model's COMPILED code. Those are opposites: the first is worth retrying, the second is a deterministic wrong answer — the program never returned. So the sweep retried it to its retry limit, the target read RE-RUN forever, and a surgical repair re-ran it to identical effect. Per-test failures carry the harness's `test N:` prefix and run locally with no network in play, so no message carrying that prefix can be infrastructure; it now settles the bucket before any transient word is consulted. Across the 46 swept targets this moves exactly 2 rows and leaves 0 transients. The concrete case is worth keeping: Opus 4.8 wrote a `list_reverse` that passes `vera check` — contracts and types verified — and then does not terminate. It scores as not-solved, which is correct; the fix only stops the harness re-running it. `rerun_failed.py` could not find its target at all once a second release existed. It globbed the canonical name only as far as `-bench-`, which is unambiguous with one era on disk and ambiguous the moment another lands — `results/` keeps every release side by side. With 0.0.16 and 0.0.18 present, every repair exited `ambiguous — 2 files match`. The prefix now pins the bench version, defaulting to the installed one, with --bench-version to reach a superseded era; the compiler segment stays a wildcard because a target's Vera version is whatever produced it, not whatever is installed now. Same defect class as the sweep runner's hardcoded version, in the one script that only ever runs after a sweep. Refs #101. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughThe changes refine sweep classification, bench-version result repair, stored-result regrading, ADT rendering, and narrative chart generation. They also update coverage metrics, generation annotations, delta legend placement, and the changelog. ChangesBenchmark harness updates
Estimated code review effort: 4 (Complex) | ~60 minutes Possibly related PRs
Suggested labels: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@scripts/rerun_failed.py`:
- Around line 248-250: The --apply flow around find_canonical must acquire and
validate ownership through the shared sweep lifecycle/lock before mutating the
selected results file with splice(). Replace the raw row-count cleanliness check
with unique problem-ID coverage and transient-row status, and ensure --force
cannot bypass this protection; reject targets currently being processed by the
sweep to prevent unlink/write races.
- Line 76: Update the benchmark filename matching logic in the rerun flow so the
generated bench-version token is terminated by the filename separator,
preventing 0.0.18 from matching 0.0.180. Add a regression case using only
0.0.180 and verify that applying 0.0.18 does not modify those rows.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 26fd5007-d30e-4315-b11f-45cec079424a
📒 Files selected for processing (5)
CHANGELOG.mdscripts/rerun_failed.pyscripts/sweep_status.pytests/test_rerun_failed.pytests/test_sweep_status.py
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #117 +/- ##
==========================================
- Coverage 86.92% 86.62% -0.30%
==========================================
Files 15 15
Lines 2845 2871 +26
==========================================
+ Hits 2473 2487 +14
- Misses 372 384 +12
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Five defects, all found by reading the first full-60 sweep's own output. A truncation was published as a model refusal. `sweep_status.py` knew only OpenAI's `finish_reason=length`; Anthropic spells a token wall `stop_reason=max_tokens`, and its message also contains "no text block" — which the refusal pattern matches, and which was tested first. So an Anthropic truncation was relabelled a decline: a recoverable failure (raise --max-tokens, re-run) was kept as a real verdict and counted in the published refusal figure. LENGTH now covers both spellings and a shared `is_refusal()` excludes truncations, so the monitor and the refusal chart — which read the same pattern through different filters, and disagreed — cannot diverge again. Across 46 targets: refusals 5 -> 4. The delta chart's legend covered the largest bar. Placed `lower right` INSIDE the axes, which on a diverging chart is exactly where the biggest positive delta ends, so the strongest Vera result and its value label sat under a semi-transparent box — and the bar showing through made hidden data read as a rendering artefact. Moved to the sparse upper-left, in both the slide and the canonical chart. The generation slide asserted findings it no longer had. Its title and subtitle were hardcoded — "Three flagships, one trajectory", "the Vera line rises at every step" — both true of a three-link chain and false the moment a fourth landed. Both are computed from the plotted points now. Two slides in one deck disagreed about the same two models. The chain read Opus 4.8 and Opus 5 from 0.0.16 while the controlled-pair slide read them from 0.0.18 — 36 graded problems against 60 — so that pair fell 6 points in TypeScript on one slide and rose 3 on the other. Every link with 0.0.18 data now reads it, leaving both confounds on the single step where Opus 4 is pinned to the only release that swept it. The coverage slide outlived its premise. It argued pass@1 could not see the problems without test cases; #107 gave every problem test cases, so it rendered "0 of 60" beside a 0% hero stat that read as Vera failing everything. It now measures where `vera check` and `vera run` disagree: the 6 programs that cleared the static gate and still failed, named individually, against 0 working programs wrongly refused. Assets are deliberately not in this commit; they ship with the README rewrite. But the generators are, so a fresh clone cannot produce the old charts. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… lineup Claude Fable 5 was appended as a fourth link on my mistaken belief that it was the newest Anthropic model. It is the CEILING tier: more capable than Opus 5, not later than it. Three things followed from that error. The slope into the final point read as generational progress that never happened — and Fable 5 falls in three of four languages, so the chart contradicted its own "one trajectory" title. Any caption ordering the chain chronologically was false. It also displaced the controlled pair. With four links, Opus 4.8 -> Opus 5 sat at positions 2 and 3, so the companion --pair-only slide showed the chain's MIDDLE segment and looked like a zoom on an arbitrary interior step rather than the end of the line. And it was no longer earning its place: re-pinning Opus 4.8 and Opus 5 to 0.0.18 already put the current release in the chain three times over, so the link supplied nothing the rest did not. The chain is Opus 4 -> Opus 4.8 -> Opus 5 again, one family in release order, and the controlled pair is now literally its last step. The comment says what may be added: a successor, not a bigger sibling. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Actionable comments posted: 5
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@CHANGELOG.md`:
- Around line 65-70: Correct the changelog entry describing the delta-chart
legend placement: do not claim it was moved above the axes or cannot collide
with data. Update the wording to accurately state that the legend uses the
upper-left position inside the axes in both slide and canonical charts, without
implying collision is impossible.
In `@scripts/plot_narrative.py`:
- Around line 1040-1066: Adjust the escape-list layout in the rows_sorted
rendering loop so the y positions remain within the panel for any number of
entries, scaling the row step or text sizing based on len(rows_sorted). Preserve
the existing ordering, columns, and styling while ensuring all named escapes
remain visible instead of being clipped.
- Around line 677-683: Build the subtitle’s moved entries from the actually
plotted modes rather than wanted or CORE_MODES, using the rebound modes
collection from the plotting flow around extract_data. Keep the existing _net
calculation and None filtering, but ensure languages absent from the plotted
data cannot produce fabricated deltas in moved.
- Around line 1121-1148: The narrative contains hardcoded zero-value claims that
can contradict false_alarms. In the plot-building code around the ax2.text
caption and the fig.text subtitle, derive both messages from false_alarms:
retain the existing zero wording when it is zero, otherwise state the actual
count and avoid claiming the gate never refused correctly running code.
In `@scripts/plot_results.py`:
- Around line 742-752: Move the legends below their axes so they cannot overlap
plotted data: in scripts/plot_results.py lines 742-752, update the legend
anchoring and columns using comparison_modes, then extend the figure’s bottom
margin; in scripts/plot_slide.py lines 361-376, apply the same below-axis
anchoring with columns based on legend_handles and widen the tight_layout rect
bottom to prevent clipping.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: b435c050-9c64-4be3-8354-7783cdf4469a
📒 Files selected for processing (6)
CHANGELOG.mdscripts/plot_narrative.pyscripts/plot_results.pyscripts/plot_slide.pyscripts/sweep_status.pytests/test_sweep_status.py
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
scripts/plot_narrative.py (1)
738-742: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winCorrect the later Vera version in the caption.
The generation chain is pinned to bench version
0.0.18, but the slide states0.1.8. This publishes an incorrect toolchain version and can mislead interpretation of the plotted results.🐛 Proposed fix
- "Opus 4 ran under Vera 0.0.112, every later point under 0.1.8; " + "Opus 4 ran under Vera 0.0.112, every later point under 0.0.18; "🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@scripts/plot_narrative.py` around lines 738 - 742, Update the caption text in the fig.text call to report Vera version 0.0.18 for the later points instead of 0.1.8, preserving the surrounding wording and formatting.
♻️ Duplicate comments (3)
scripts/plot_narrative.py (3)
681-691: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick winBuild the subtitle from the modes actually plotted.
modeshas just been filtered to languages present in every generation link, but this comprehension iterateswanted/CORE_MODES. A dropped mode can therefore produce a fabricated_netvalue from the absent-data sentinel and appear in the subtitle despite having no plotted series.🐛 Proposed fix
- moved = [f"{m} {_net(m):+d}" for m in (wanted or CORE_MODES) if _net(m) is not None] + moved = [f"{m} {_net(m):+d}" for m in modes if _net(m) is not None]🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@scripts/plot_narrative.py` around lines 681 - 691, Update the subtitle’s `moved` comprehension to iterate over the already-filtered `modes` collection, matching the series actually plotted. Keep the existing `_net` filtering and formatting unchanged so dropped modes cannot appear in the subtitle.
1042-1070: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winKeep every named escape inside the panel.
The fixed
0.125decrement displays at most seven rows; the eighth reaches approximatelyy=-0.02and is clipped. Because the list is data-derived, larger sweeps can silently omit named failures.🐛 Proposed fix
y = 0.855 + step = min(0.125, 0.80 / max(len(rows_sorted), 1)) for model, pid, cause in rows_sorted: ... - y -= 0.125 + y -= step🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@scripts/plot_narrative.py` around lines 1042 - 1070, Update the escape-row layout around rows_sorted and the y decrement so every data-derived named escape remains within the panel, including lists longer than seven rows. Compute spacing or otherwise adapt the available vertical range to rows_sorted’s length while preserving the existing text ordering and formatting.
1125-1152: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick winDerive both false-alarm captions from
false_alarms.The count is already rendered and coloured red when non-zero, but the captions always claim that no working programme was rejected and that the cost is zero. That makes the slide self-contradictory whenever
false_alarms > 0.[details]
🐛 Proposed fix
- "the gate never refused code that ran correctly", + ( + "the gate never refused code that ran correctly" + if false_alarms == 0 + else f"{false_alarms} working programmes were rejected" + ), ... - "The cost of that gate is the number worth watching, and it is zero.", + ( + "The cost of that gate is the number worth watching, and it is zero." + if false_alarms == 0 + else f"The cost of that gate is {false_alarms} rejected programmes." + ),🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@scripts/plot_narrative.py` around lines 1125 - 1152, Update the captions in the narrative plotting section to derive their wording from false_alarms: make the gate-refusal caption accurately reflect the false-alarm count and replace the hard-coded “cost ... is zero” statement with the corresponding value. Preserve the existing zero-count wording where false_alarms is zero and keep the current styling and layout unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@scripts/plot_narrative.py`:
- Around line 738-742: Update the caption text in the fig.text call to report
Vera version 0.0.18 for the later points instead of 0.1.8, preserving the
surrounding wording and formatting.
---
Duplicate comments:
In `@scripts/plot_narrative.py`:
- Around line 681-691: Update the subtitle’s `moved` comprehension to iterate
over the already-filtered `modes` collection, matching the series actually
plotted. Keep the existing `_net` filtering and formatting unchanged so dropped
modes cannot appear in the subtitle.
- Around line 1042-1070: Update the escape-row layout around rows_sorted and the
y decrement so every data-derived named escape remains within the panel,
including lists longer than seven rows. Compute spacing or otherwise adapt the
available vertical range to rows_sorted’s length while preserving the existing
text ordering and formatting.
- Around line 1125-1152: Update the captions in the narrative plotting section
to derive their wording from false_alarms: make the gate-refusal caption
accurately reflect the false-alarm count and replace the hard-coded “cost ... is
zero” statement with the corresponding value. Preserve the existing zero-count
wording where false_alarms is zero and keep the current styling and layout
unchanged.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 5d2c5e6c-97fc-4a62-af22-eb3f5e87c754
📒 Files selected for processing (2)
CHANGELOG.mdscripts/plot_narrative.py
aallan
changed the title
…erived captions Six findings verified against current code; four were live. `rerun_failed.py`'s bench-version token was not terminated, so `0.0.18` also matched `bench-0-0-180-...`. On a lookup that would be harmless; this path ends in a splice, so it could have written fresh rows into a different release's results file. The glob now accepts only the two shapes a real name takes after the bench segment — a compiler segment, or the extension — with a regression case that puts 0.0.180 on disk alone and asserts 0.0.18 does not select it. Its in-flight guard counted rows. A fix attempt emits a second row for the same problem, so a file can reach 60 rows while still missing problems — and the guard exists precisely to stop a repair racing a sweep mid-write. It now counts unique problem ids, the same denominator sweep_status uses. The coverage slide's captions hardcoded their own zeroes: "the gate never refused code that ran correctly" and "the cost of that gate ... is zero" would have contradicted a non-zero false-alarm count sitting beside them. Both derive from the count now. The escape list's row pitch was fixed at 0.125, which put an eighth escape below the panel — so a WORSE result would have hidden the extra rows, flattering the number the slide exists to report. Pitch scales with the count. The generation subtitle reported net change for `wanted` rather than the `modes` actually plotted, which could have put a number on the slide with no series behind it. Declined two. Moving the delta legend below the axes is a design decision already taken the other way — upper-left, stacked, inside the axes, which is where the maintainer asked for it. And the generation caption's "0.1.8" is correct: 0.0.112 and 0.1.8 are Vera compiler versions, not bench versions, which is what that sentence is about. Also declined the sweep-lifecycle lock proposed alongside the coverage guard: no such lifecycle exists here, and inventing one is the feature tracked in #101, not a minimal fix. The coverage half of that finding is what landed. None of these change a published number — verified: the two affected charts regenerate byte-identical. They are latent fixes that matter when the data changes shape. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Owner
Author
|
@coderabbitai — replying to the outside-diff finding on Declining this one: The caption is about the Vera compiler, not the bench release:
Substituting the bench version into the first sentence would claim Opus 4 ran under "Vera 0.0.112" and the rest under "Vera 0.0.18", which is not a Vera version at all. The two numbers are close enough to look like a typo — |
|
Tip For best results, initiate chat on the files or code changes.
✏️ Learnings added
|
Error messages are truncated to a fixed length, and the sandbox path sits BEFORE the cut — so a path of a different length moves where the sentence ends, and the same advice comes back as "at compile t" instead of "at compile time." Normalising the path was not enough; the strings still differed by where they had been severed. That mattered for honesty rather than tidiness. The first full run reported 177 changed rows when 111 were real, and that inflated number was headed straight into the PR as evidence. It would also have rewritten dozens of rows' messages with paths from a re-grade rather than the sweep that produced them. `_same_verdict` now discounts both: every non-message field must match exactly, and only the message may be a truncation of its counterpart. Same-verdict rows keep their original text. 177 -> 129 changed, with the 111 real transitions untouched. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@scripts/regrade.py`:
- Around line 89-112: Update _same_verdict so the error_message comparison
requires a true truncation relationship: after confirming all non-message fields
match, return true only when ma is a prefix of mb or mb is a prefix of ma, while
preserving the existing empty-message behavior as appropriate.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 71e7f46a-89d0-4bf8-a284-82db184aaa1c
📒 Files selected for processing (1)
scripts/regrade.py
…el was blamed The harness splices `export` onto the entry point by string-matching `function name(`. A generic solution writes `function listLength<T>(list: List<T>)`, which does not contain `listLength(` — so no export was added, the wrapper's import resolved to undefined, and every test threw "is not a function". That is a WRAPPING failure recorded as the model's wrong answer, and it is not a rare shape: generics are idiomatic exactly where the ADT problems live. Re-grading TypeScript from stored code moves 51 rows from wrong to solved. Nothing moves the other way. It also nearly published a much stronger claim than the data supports. With these 51 counted as failures, Vera's measured lead over TypeScript roughly doubled — Fable 5 read +13 and Kimi K2.6 +15 against a previously published +7. Those numbers were an artefact of this bug. Same defect class as the ADT mapper fixed earlier in this PR: a literal string match that generics walk straight past. The injection now matches the declaration rather than a literal call shape, and handles arrow function entry points (`const f = <T>(...)`) as well. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…SSUES Both found while verifying the v0.0.18 re-grade, both with an exit condition rather than a shrug. #118 is the one that has already cost data: baselines unlinks its output at startup and writes at the end, so any interruption destroys a file that results/ being gitignored makes unrecoverable. aver-baseline.jsonl went three times in one night. #119 is smaller and mostly about honesty of route: a class with no __init__ and no fields is unambiguously nullary, and reading it as unverifiable lets the shape guard render a call it should have refused. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The min-length comparison is a prefix test, and the bool(n) guard reads like a typo until you notice that an empty string is a prefix of everything — so startswith() would treat a message appearing or disappearing as the same verdict. It is not. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Everything here was found by running the first full-60 sweep and reading what it produced. None of it was reachable from the test suite, because every existing gate — baselines,
validate, evaluator parity — runs against canonical solutions, and canonical solutions never hit any of these paths.One of them was silently biasing the published cross-language comparison.
1. Tier 3 was not being graded for Python or TypeScript
The harness builds a test wrapper by reading the model's own type declaration. When it cannot map that declaration it correctly declines — records
run_correct=None, leaves the problem ungraded rather than guessing. That decline path is right. What was wrong is how often it fired:All twelve missing problems are Tier 3 (ADT + exhaustive match) and Tier 4. So the headline "does Vera beat Python/TypeScript" chart was comparing Vera over 60 problems against Python over ~51 — and the missing tier is exactly the one Vera's design targets.
Two causes, both in
adt_render.py:Python — a union alias is the ADT's name:
selfishcollected the class and itsast.Namebases.Generic[T]is aSubscript, not aName, andLinkedListis a module-level alias rather than a base — so the self-reference never resolved and the shape guard saw('t', 'linkedlist')where it wanted('int', 'SELF'). Self-names now include module-level aliases that reference the class.TypeScript — a type variable is unreadable, not wrong:
Tis not a class name, so it collapsed to"object"and could never matchInt's"number"._ts_kind()now strips generics and returnsNonefor anything unreadable; alignment matches readable types first and only then fills gaps in declared order — so a genuine swap of two readable fields is still caught. This is the same judgement_type_mismatchalready made on the Python side.Result across the sweep: 111 of 113 declines resolve. Both halves are reported, because 35 of the newly-gradeable answers are simply wrong — those lower the comparison languages' scores relative to leaving them ungraded, which is why the fix has to be applied wholesale rather than to the rows that flatter the result.
2.
scripts/regrade.py— apply a grading fix without paying for the sweep againSince #109 every row carries a
code_path, so the model's actual output is on disk and a verdict can be recomputed: same code, same problems, fixed harness. That turns a ~$100 overnight re-run into a few minutes of local subprocesses.It re-grades every row, not just the ones a fix was expected to help — grading only the latter would import every improvement while hiding any regression the same change caused. That makes it a free regression test, and the first full run is the evidence this PR rests on: no
solved →transition of any kind. Nothing that passed was broken.Only verdict fields are replaced. Token counts, timings and model identity describe the original run and carry through untouched, so a re-graded file still records what that sweep actually cost. Verdicts are compared with sandbox paths normalised, since those differ on every run by construction — without that, 64 rows looked "changed" and would have had their original error messages overwritten with paths from a run that never happened.
3. Sweep triage: two misclassifications that changed what the sweep did
A model's non-terminating code was triaged as an infrastructure fault. The transient pattern matched a bare
timed out, which covers both a provider call failing and the harness's own 30s budget on the model's compiled code. Those are opposites: one is worth retrying, the other is a deterministic wrong answer. So the sweep retried it to its limit, the target readRE-RUNforever, and a surgical repair reproduced it exactly. Per-test failures carry the harness'stest N:prefix and run locally with no network in play, so that prefix now settles the bucket first. Concretely: Opus 4.8 wrote alist_reversethat passesvera checkand then does not terminate.A truncation was published as a model refusal.
LENGTHknew only OpenAI'sfinish_reason=length; Anthropic spells a token wallstop_reason=max_tokens, and its message also contains "no text block" — which the refusal pattern matched, and which was tested first. A recoverable failure was kept as a real verdict and counted in the published refusal figure. A sharedis_refusal()now excludes truncations, so the monitor and the refusal chart cannot disagree again. Refusals 5 → 4.4.
rerun_failed.py— the repair tool broke exactly when it was neededThree defects, all in the path that only runs after a sweep:
<model>-bench-*, unambiguous with one release on disk and ambiguous the moment a second landed. With 0.0.16 and 0.0.18 present, every repair exitedambiguous — 2 files match. The prefix now pins the bench version (--bench-versionreaches older eras); the compiler segment stays a wildcard, since a target's Vera version is whatever produced it.0.0.18also matchedbench-0-0-180-…— a repair aimed at one release could splice rows into another's file.sweep_statususes.5. Charts that asserted things the data no longer supported
lower rightinside the axes — on a diverging chart, exactly where the biggest positive delta ends. Moved to the sparse upper-left in both the slide and the canonical chart.vera checkandvera rundisagree — the programs that cleared the static gate and still failed, named individually, against the count of working programs wrongly refused. Both numbers are derived, so the captions cannot contradict them.Scope
Generators only. The regenerated assets ship with the README rewrite in a separate PR — but the generators are here, so a fresh clone cannot produce the old charts.
ruff check . && ruff format --check . && ruff check --select S vera_bench/clean; full suite green.Refs #101, #115.
🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Bug Fixes
Style