docs: rewrite the results narrative for the first full-60 sweep - #120
Merged
Conversation
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughThe documentation updates refresh VeraBench results and analysis for Vera v0.1.8, expand language comparisons, clarify evaluation metrics and benchmark workflows, document regrading and reporting, and update plot regeneration instructions. ChangesBenchmark documentation
Estimated code review effort: 2 (Simple) | ~15 minutes Possibly related PRs
Suggested labels: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
Every number is new. The v0.0.16 sweep output-graded 36 of the 60 problems, and the 24 it could not grade were the ADT, pattern-matching and effect-handler problems, so those numbers were computed on the subset that suited Python and TypeScript best. The two sets are not comparable, and the README says so up front. The story changed with them. Vera beats Python for six of the nine models and averages 98.7% against Python's 96.7%; against TypeScript it wins once, draws five and loses three, because TypeScript now scores 100% for six models and there is no headroom left to win. That is a fact about saturation rather than about Vera, and the saturation section says so. The zero-training-data comparison is the one saturation has not eaten, and it is now the strongest result in the release: Vera 98.2% against AILANG 96.8% and Aver 92.4% on the five models that ran all three, with Vera ahead of Aver on every one. A figure version of that three-language chart is new (assets/fig-ztd.png), sitting below the Vera-versus-Aver section. The coverage section is rewritten rather than updated. It used to be about the problems pass@1 could not see; that gap is closed, so it now reports where `vera check` and `vera run` disagree: six of 539 graded pairs cleared the static gate and were still wrong, against zero working programs refused. assets/GRAPHS.md was rewritten to match, but it is gitignored (talk notes rather than a published asset), so it does not appear in this diff. Drafted against the writing-voice skill and audited with its checker. assets/README.md comes back clean; README.md's remaining findings are markdown image syntax read as exclamation marks. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
aallan
force-pushed
the
docs/v0-0-18-results
branch
from
July 28, 2026 02:10
e197c86 to
5f63493
Compare
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 6
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@assets/README.md`:
- Around line 88-94: Update the asset documentation surrounding the regeneration
commands to consistently reference version 0.0.18, revise the denominator from
the outdated 36-problem description, and add fig-ztd.png to the figure
inventory. Keep the inventory aligned with all three commands shown in the diff.
In `@scripts/plot_narrative.py`:
- Around line 1158-1164: Update the narrative count in the `fig.text` block to
use the number of model targets that actually contribute Vera rows, matching the
filtering performed by the tally loop around `mode != "Vera"`; do not use the
unfiltered `len(rows_by_target)` value.
In `@scripts/regrade.py`:
- Line 154: Update the worker function one() to return its transition label
instead of mutating the shared tally, then collect those labels from
ThreadPoolExecutor.map and increment tally on the main thread after the worker
calls complete. Preserve the existing transition labels and printed output while
removing all worker-thread writes to tally.
- Around line 243-247: Update the target-processing loop around path.read_text()
to detect files with no lines before accessing the first line, and continue to
the next target when the file is empty. Preserve the existing language filter
and regrade_file flow for non-empty target files.
In `@scripts/rerun_failed.py`:
- Line 269: Update the covered-count calculation in the rerun-failed sweep guard
to count only truthy problem IDs, matching the filtering used by
sweep_status.py. Preserve unique-ID counting while excluding missing or empty
IDs so the guard cannot treat an incomplete sweep as clean.
In `@vera_bench/adt_render.py`:
- Around line 557-564: Update _ts_kind to match the intended handling of unknown
nominal and capitalized types: either preserve the original casing for the
capitalization check before lowercasing, or explicitly treat all unknown names
as "object" and remove the unreachable base[0].isupper() condition. Also
simplify the final return by removing the unreachable else branch, since known
_TS_CLASS values already return earlier.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 577dae0c-52ed-4554-87c2-0a0e517f0eab
⛔ Files ignored due to path filters (9)
assets/fig-coverage.pngis excluded by!**/*.pngassets/fig-delta.pngis excluded by!**/*.pngassets/fig-generation.pngis excluded by!**/*.pngassets/fig-reasoning.pngis excluded by!**/*.pngassets/fig-refusal.pngis excluded by!**/*.pngassets/fig-saturation.pngis excluded by!**/*.pngassets/fig-vera-vs-aver.pngis excluded by!**/*.pngassets/fig-ztd.pngis excluded by!**/*.pngassets/results-graph.pngis excluded by!**/*.png
📒 Files selected for processing (14)
CHANGELOG.mdKNOWN_ISSUES.mdREADME.mdassets/README.mdscripts/plot_narrative.pyscripts/plot_results.pyscripts/plot_slide.pyscripts/regrade.pyscripts/rerun_failed.pyscripts/sweep_status.pytests/test_rerun_failed.pytests/test_sweep_status.pyvera_bench/adt_render.pyvera_bench/runner.py
There was a problem hiding this comment.
Caution
Inline review comments failed to post. This is likely due to GitHub's internal server error or limits when posting large numbers of comments. If you are seeing this consistently it is likely a permissions issue. Please check "Moderation" -> "Code review limits" under your organization settings.
Actionable comments posted: 6
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@assets/README.md`:
- Around line 88-94: Update the asset documentation surrounding the regeneration
commands to consistently reference version 0.0.18, revise the denominator from
the outdated 36-problem description, and add fig-ztd.png to the figure
inventory. Keep the inventory aligned with all three commands shown in the diff.
In `@scripts/plot_narrative.py`:
- Around line 1158-1164: Update the narrative count in the `fig.text` block to
use the number of model targets that actually contribute Vera rows, matching the
filtering performed by the tally loop around `mode != "Vera"`; do not use the
unfiltered `len(rows_by_target)` value.
In `@scripts/regrade.py`:
- Line 154: Update the worker function one() to return its transition label
instead of mutating the shared tally, then collect those labels from
ThreadPoolExecutor.map and increment tally on the main thread after the worker
calls complete. Preserve the existing transition labels and printed output while
removing all worker-thread writes to tally.
- Around line 243-247: Update the target-processing loop around path.read_text()
to detect files with no lines before accessing the first line, and continue to
the next target when the file is empty. Preserve the existing language filter
and regrade_file flow for non-empty target files.
In `@scripts/rerun_failed.py`:
- Line 269: Update the covered-count calculation in the rerun-failed sweep guard
to count only truthy problem IDs, matching the filtering used by
sweep_status.py. Preserve unique-ID counting while excluding missing or empty
IDs so the guard cannot treat an incomplete sweep as clean.
In `@vera_bench/adt_render.py`:
- Around line 557-564: Update _ts_kind to match the intended handling of unknown
nominal and capitalized types: either preserve the original casing for the
capitalization check before lowercasing, or explicitly treat all unknown names
as "object" and remove the unreachable base[0].isupper() condition. Also
simplify the final return by removing the unreachable else branch, since known
_TS_CLASS values already return earlier.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 577dae0c-52ed-4554-87c2-0a0e517f0eab
⛔ Files ignored due to path filters (9)
assets/fig-coverage.pngis excluded by!**/*.pngassets/fig-delta.pngis excluded by!**/*.pngassets/fig-generation.pngis excluded by!**/*.pngassets/fig-reasoning.pngis excluded by!**/*.pngassets/fig-refusal.pngis excluded by!**/*.pngassets/fig-saturation.pngis excluded by!**/*.pngassets/fig-vera-vs-aver.pngis excluded by!**/*.pngassets/fig-ztd.pngis excluded by!**/*.pngassets/results-graph.pngis excluded by!**/*.png
📒 Files selected for processing (14)
CHANGELOG.mdKNOWN_ISSUES.mdREADME.mdassets/README.mdscripts/plot_narrative.pyscripts/plot_results.pyscripts/plot_slide.pyscripts/regrade.pyscripts/rerun_failed.pyscripts/sweep_status.pytests/test_rerun_failed.pytests/test_sweep_status.pyvera_bench/adt_render.pyvera_bench/runner.py
🛑 Comments failed to post (5)
scripts/plot_narrative.py (1)
1158-1164: 🎯 Functional Correctness | 🔵 Trivial | 💤 Low value
Model count is unfiltered while the pair count is filtered.
The tally loop skips any
mode != "Vera"(Line 992), butlen(rows_by_target)counts every target key including a non-Vera one, shouldload_rowsever return one. The two numbers in the same sentence would then be drawn from different populations.♻️ Count the models that actually contributed
escapes: list[tuple[str, str, str]] = [] # (model, problem, cause) + models_seen: set[str] = set() agree = false_alarms = graded = 0 for (model, mode), rows in sorted(rows_by_target.items()): if mode != "Vera": continue + models_seen.add(model)- f"program — {graded} pairs across {len(rows_by_target)} models.\n" + f"program — {graded} pairs across {len(models_seen)} models.\n"📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.fig.text( 0.5, 0.865, f"`vera check` and `vera run` are independent verdicts on the same " f"program — {graded} pairs across {len(models_seen)} models.\n" "Contracts bound what a program may do; they do not say everything " "it must do, so a few satisfy them and are still wrong.\n"🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@scripts/plot_narrative.py` around lines 1158 - 1164, Update the narrative count in the `fig.text` block to use the number of model targets that actually contribute Vera rows, matching the filtering performed by the tally loop around `mode != "Vera"`; do not use the unfiltered `len(rows_by_target)` value.scripts/regrade.py (2)
154-154: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
tallyis mutated from up to eight threads — the census can lose counts.
Counter.__iadd__on a key (tally["changed"] += 1) is a load-add-store, not an atomic operation, so concurrentone()calls underThreadPoolExecutorwill drop increments. The printed transition table is the entire output of a dry run, and an undercount here reads as "the fix touched fewer rows than it did" — the opposite of the honest census the docstring promises. The row data itself is safe; only the tally is racy.Cheapest fix: have
one()return the label and fold the counts on the main thread.🔒 Fold the tally after the map, not inside the workers
- def one(row: dict) -> dict: + def one(row: dict) -> tuple[dict, list[str]]: cp = row.get("code_path") problem = problems.get(row.get("problem_id")) if not cp or problem is None: - tally["no-code"] += 1 - return row + return row, ["no-code"] src = results_dir / cp if not src.exists(): - tally["missing-file"] += 1 - return row + return row, ["missing-file"] before = {k: row.get(k) for k in VERDICT_FIELDS} with tempfile.TemporaryDirectory(prefix="vb-regrade-") as tmp: try: fresh = evaluate( row, src.read_text(encoding="utf-8"), problem, Path(tmp), vera ) except Exception as exc: # a grader crash is not a verdict - tally["grader-error"] += 1 row["regrade_error"] = f"{type(exc).__name__}: {exc}"[:200] - return row + return row, ["grader-error"] new = dict(row) new.update({k: v for k, v in fresh.items() if k in VERDICT_FIELDS}) after = {k: new.get(k) for k in VERDICT_FIELDS} if _same_verdict(before, after): - tally["unchanged"] += 1 - return row - else: - tally["changed"] += 1 - was, now = _bucket(before), _bucket(after) - tally[f"{was} -> {now}"] += 1 - return new + # Same verdict, different sandbox path. Return the ORIGINAL row + # so the file keeps the message the sweep actually recorded. + return row, ["unchanged"] + return new, ["changed", f"{_bucket(before)} -> {_bucket(after)}"] if workers > 1: with ThreadPoolExecutor(max_workers=workers) as pool: - out = list(pool.map(one, rows)) + done = list(pool.map(one, rows)) else: - out = [one(r) for r in rows] + done = [one(r) for r in rows] + out = [] + for row, labels in done: + out.append(row) + tally.update(labels) return out, tallyAlso applies to: 163-198
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@scripts/regrade.py` at line 154, Update the worker function one() to return its transition label instead of mutating the shared tally, then collect those labels from ThreadPoolExecutor.map and increment tally on the main thread after the worker calls complete. Preserve the existing transition labels and printed output while removing all worker-thread writes to tally.
243-247: 🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win
An empty target file aborts the whole run with
IndexError.
path.read_text().splitlines()[0]assumes at least one line.vera-bench rununlinks and recreates its output file, so a zero-byte target inresults/is an ordinary mid-sweep state — and here it kills the regrade of every remaining target, after the earlier ones may already have been rewritten under--apply. Skip it instead.🛡️ Skip a file with no rows
for path in targets: - head = json.loads(path.read_text().splitlines()[0]) - if args.language and head.get("language") != args.language: + lines = [ln for ln in path.read_text().splitlines() if ln.strip()] + if not lines: + print(f" {path.name[:64]:66} empty — skipping", flush=True) + continue + head = json.loads(lines[0]) + if args.language and head.get("language") != args.language: continue📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.for path in targets: lines = [ln for ln in path.read_text().splitlines() if ln.strip()] if not lines: print(f" {path.name[:64]:66} empty — skipping", flush=True) continue head = json.loads(lines[0]) if args.language and head.get("language") != args.language: continue rows, tally = regrade_file(path, results_dir, problems, vera, args.parallel)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@scripts/regrade.py` around lines 243 - 247, Update the target-processing loop around path.read_text() to detect files with no lines before accessing the first line, and continue to the next target when the file is empty. Preserve the existing language filter and regrade_file flow for non-empty target files.scripts/rerun_failed.py (1)
269-269: 🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
Nonecounts as a problem, so the in-flight guard is one short ofsweep_status.
{r.get("problem_id") for r in rows}admitsNonefor any row without an id, inflatingcoveredby one — which is enough to let a file that is one problem short read as complete and be spliced while the sweep is still writing it.sweep_status.py(Line 220) filters:{r["problem_id"] for r in rows if r.get("problem_id")}. Match it, given the comment claims the same denominator.🛡️ Use the same denominator it claims to
- covered = len({r.get("problem_id") for r in rows}) + covered = len({r["problem_id"] for r in rows if r.get("problem_id")})Based on the guideline that
rerun_failed.pymust never run against a target the sweep is still processing, since both processes write the same output file, and on the guideline to use unique problem-ID coverage when determining sweep cleanliness.📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.covered = len({r["problem_id"] for r in rows if r.get("problem_id")})🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@scripts/rerun_failed.py` at line 269, Update the covered-count calculation in the rerun-failed sweep guard to count only truthy problem IDs, matching the filtering used by sweep_status.py. Preserve unique-ID counting while excluding missing or empty IDs so the guard cannot treat an incomplete sweep as clean.Source: Coding guidelines
vera_bench/adt_render.py (1)
557-564: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Two dead branches in
_ts_kind— the case check can never fire.
baseis already lower-cased on Line 557, sobase[0].isupper()on Line 562 is alwaysFalse. The docstring's rule that an unknown capitalised name (alias, union) is unreadable therefore never applies:Listbecomes"object"and is aligned as a readable type. Only thelen(base) <= 2clause is actually rejecting anything (type variables). Likewise on Line 564, theelse basearm is unreachable because Line 558 already returned for any base in_TS_CLASS.values().If aligning unknown nominal types as
"object"is the intent (the recursive-alias test suggests it is), say so and drop the dead clause; if capitalised unknowns really should be unreadable, test the case before lower-casing.♻️ Make the intent match the behaviour
- base = re.sub(r"<[^>]*>", "", t or "").strip().lower() - if base in _TS_CLASS.values(): - return base - if base in {k.lower() for k in _TS_CLASS}: - return _TS_CLASS[next(k for k in _TS_CLASS if k.lower() == base)] - if not base or len(base) <= 2 or base[0].isupper(): - return None - return "object" if base not in _TS_CLASS.values() else base + raw = re.sub(r"<[^>]*>", "", t or "").strip() + base = raw.lower() + if base in _TS_CLASS.values(): + return base + for k, v in _TS_CLASS.items(): + if base == k.lower(): + return v + # A short name is a type variable (`T`, `K`) — unreadable, never + # "object". Any other nominal type aligns with a non-scalar argument. + if not base or len(base) <= 2: + return None + return "object"📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.raw = re.sub(r"<[^>]*>", "", t or "").strip() base = raw.lower() if base in _TS_CLASS.values(): return base for k, v in _TS_CLASS.items(): if base == k.lower(): return v # A short name is a type variable (`T`, `K`) — unreadable, never # "object". Any other nominal type aligns with a non-scalar argument. if not base or len(base) <= 2: return None return "object"🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@vera_bench/adt_render.py` around lines 557 - 564, Update _ts_kind to match the intended handling of unknown nominal and capitalized types: either preserve the original casing for the capitalization check before lowercasing, or explicitly treat all unknown names as "object" and remove the unreachable base[0].isupper() condition. Also simplify the final return by removing the unreachable else branch, since known _TS_CLASS values already return earlier.
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@README.md`:
- Around line 55-68: Correct the full-spec score-range statement in the README
to account for the Aver values shown in the model results table, including its
actual 90%–95% range, or explicitly limit the claim to Vera, Python, and
TypeScript. Update both the referenced summary and the corresponding statement
around the table’s earlier range description.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 7bb4b93f-4804-4de5-bea0-e33512d3832d
⛔ Files ignored due to path filters (9)
assets/fig-coverage.pngis excluded by!**/*.pngassets/fig-delta.pngis excluded by!**/*.pngassets/fig-generation.pngis excluded by!**/*.pngassets/fig-reasoning.pngis excluded by!**/*.pngassets/fig-refusal.pngis excluded by!**/*.pngassets/fig-saturation.pngis excluded by!**/*.pngassets/fig-vera-vs-aver.pngis excluded by!**/*.pngassets/fig-ztd.pngis excluded by!**/*.pngassets/results-graph.pngis excluded by!**/*.png
📒 Files selected for processing (2)
README.mdassets/README.md
The Vera-TypeScript-Python ordering reads as a puzzle until the languages are grouped by when they check types. Python is dynamically typed and finds the error at runtime; TypeScript rejects it first; Vera is on the TypeScript side of that line and adds contracts and a prover on top. Grouped that way the result is the thesis rather than a counterexample: Vera 98.7% and TypeScript 99.7% against Python's 96.7%, so both constrained languages finish ahead of the loose one, and the language nobody trained on is in the leading group. The failure modes agree. Python's failures land as runtime wrong answers rather than compile rejections, thirteen against three across the sweep; TypeScript has one of each. A loose language cannot reject a bad program up front, so the mistake survives to execution. The section states its own caveat rather than leaving it to be found. Some of Python's deficit is ours: TypeScript is structurally typed, so at the boundary where the grader builds a test value it accepts an object carrying a field the model's type never declared, where Python's constructor raises. On two problems the identical modelling choice scored solved in TypeScript and not solved in Python (#121). The direction holds; the three points are provisional. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both findings were mine from this session, and both were live. I bumped the regeneration block in assets/README.md to v0.0.18 and added the fig-ztd.png command, but left the prose around it describing the v0.0.16 charts and a 36-problem denominator, and did not add the new figure to the inventory. A regeneration block that produces one set of assets beside prose describing another is worse than either alone. The provenance, the denominator (1.7 points per problem, not 2.8), the coverage chart's description and the ztd entry are all brought into line, and every figure the README embeds is now documented. The README claimed "every full-spec column bunches between 93% and 100%". Expanding that table to include Aver and AILANG made it false: Aver runs 90% to 95%. The claim is now scoped to the three columns it actually describes, with Aver's range stated rather than quietly excluded. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
README.md (1)
380-395: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winQualify the regrading scope.
regrade.pyleaves API-error/no-code rows, rows with missing code files, unknown problems, and grader-error rows unchanged. Therefore, “Every result row records” a code path and “It re-grades every row” are too broad; document that regrading applies only to eligible rows with valid stored code.As per path instructions,
**/*.mdfiles are reviewed for factual accuracy and broken links, not grammar or prose style.Suggested wording
-Every result row records the path to the code the model actually wrote, under +Eligible result rows record the path to the code the model actually wrote, under ... -It re-grades every row rather than only the ones a fix was expected to help, +It re-grades every eligible row with stored code rather than only the ones a fix was expected to help,🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@README.md` around lines 380 - 395, Update the regrading documentation around regrade.py to qualify that only eligible rows with valid stored code are regraded. State that API-error/no-code rows, rows with missing code files, unknown problems, and grader-error rows remain unchanged, and replace the broad claims that every result row records code and every row is regraded.Source: Path instructions
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@README.md`:
- Around line 380-395: Update the regrading documentation around regrade.py to
qualify that only eligible rows with valid stored code are regraded. State that
API-error/no-code rows, rows with missing code files, unknown problems, and
grader-error rows remain unchanged, and replace the broad claims that every
result row records code and every row is regraded.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 39634429-9e93-4764-ac18-d2d909c9dcd1
📒 Files selected for processing (1)
README.md
The Results section opened by asking the reader to compare against numbers they have never seen: an earlier sweep, its 36-problem denominator, and a warning that the two are not comparable. None of that is published anywhere, so it was meta-commentary in the position where the finding should be. Cut. Two other passages were framed the same way and are now framed as properties of the chart instead. The generation caveat says Claude Opus 4 was measured against an older compiler and a smaller set of graded problems, which is what a reader needs in order to distrust that step; it no longer cites bench versions to say so. The saturation section states that six models score 100% in TypeScript and six in Vera, rather than reporting that the number moved since last time. The full-results table drops the Aver and AILANG columns. Both languages have two sections of their own at the end, and carrying them in the headline table put a five-model subset beside a nine-model one, with dashes standing in for targets that were never run. The range claim beneath it now describes exactly the three columns that remain, verified against the table rather than asserted. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The static-typing section grouped the three languages correctly and then stopped short of the point. Vera is the only one of the three that no model has read: TypeScript earns 99.7% with an enormous amount of the language in every training set, and Vera reaches 98.7% from a single document in the prompt. Whatever familiarity is buying TypeScript, Vera appears to supply by other means, and the candidate is the structure it imposes. The zero-training-data comparison tests that rather than asserting it, because it holds exposure at zero for all three languages and varies only the design. Vera 98.2%, AILANG 96.8%, Aver 92.4%, with Vera also ahead of Python on the same five models. The most constrained of the three scores highest. Stated as compensation rather than superiority, because the data does not support superiority: Vera is a point behind TypeScript and the delta chart shows it. The claim is that mandatory contracts and typed slot references are worth roughly what a training corpus is worth, which is both defensible and the more interesting result. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
README.md (2)
356-358: 🩺 Stability & Availability | 🟡 Minor | ⚡ Quick winDistinguish provider timeouts from test-execution timeouts.
scripts/sweep_status.pytreatstest N:execution failures as real results even when their message contains “timed out”; only provider or infrastructure timeouts are transient. Clarify this here to avoid re-running non-terminating solutions.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@README.md` around lines 356 - 358, Update the “Watch the scoreboard” guidance in README.md to clarify that only provider or infrastructure timeouts are transient; test-execution failures reported as “test N:” results, including messages containing “timed out,” are real results that must be kept and not re-run.
374-389: 🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick winLimit the regrading claim to rows with stored code.
scripts/regrade.pyleaves API-error rows, missing code paths, and runs performed with--no-store-codeunchanged. Therefore, “Every result row records” and “It re-grades every row” are too broad; qualify them as applying to rows with an existing stored code path, and state that other rows are skipped.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@README.md` around lines 374 - 389, Update the README description of scripts/regrade.py to limit its claims to result rows with an existing stored code path. Revise “Every result row records” and “It re-grades every row” accordingly, and explicitly state that API-error rows, missing code paths, and runs using --no-store-code are skipped and left unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@README.md`:
- Around line 94-95: Update the TypeScript saturation count in the README
analysis from six of nine models to eight of nine, matching the table and
excluding only Claude Fable 5.
---
Outside diff comments:
In `@README.md`:
- Around line 356-358: Update the “Watch the scoreboard” guidance in README.md
to clarify that only provider or infrastructure timeouts are transient;
test-execution failures reported as “test N:” results, including messages
containing “timed out,” are real results that must be kept and not re-run.
- Around line 374-389: Update the README description of scripts/regrade.py to
limit its claims to result rows with an existing stored code path. Revise “Every
result row records” and “It re-grades every row” accordingly, and explicitly
state that API-error rows, missing code paths, and runs using --no-store-code
are skipped and left unchanged.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 1c048977-9abe-476b-9315-f4b8a61765bd
📒 Files selected for processing (1)
README.md
'and they score no better in Python' had an ambiguous subject and, read the obvious way, claimed no model does better in Python. Two do: Opus 4.8 by five points and Sonnet 5 by one. Replaced with the three averages, which carry the same point without the ambiguity and set up the section that follows. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…eresting Five of them, all the same move: a preamble telling you a fact matters, followed by the fact, which would have carried itself. "The losses are worth naming." "The number worth watching is the other cell, and it is zero." "That grouping is what makes the Vera number interesting, because..." "...which is the substance of the saturation problem discussed below." "...worth roughly what a training corpus is worth, which is a stranger and more useful result." Each is now just the statement. The paragraph about Opus 4.8 and Sonnet 5 opens on the five-point gap, the coverage section says "The other cell is zero", and the compensation argument opens on Vera being the only one of the three no model has read. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
It was built backwards. Two sentences of analytical scaffolding ("which
makes every model-and-problem pair a two-by-two: the static gate passed or it
did not... Two of those cells are agreement"), then the data, then a callback
that only parsed if you had held the grid in your head ("The other cell is
zero").
The grid was never the point and the reader does not need it. The section now
opens on the number: six programs out of 539 compiled, satisfied their
contracts, and were still wrong. The four that matter are described by what
they did rather than by which quadrant they fall in, and the closing
paragraph states the second finding in its own terms instead of referring
back to a diagram: nothing that ran correctly was refused, and nothing
refused ran correctly.
Also dropped "the honest limit", which was doing the same job as the
"worth naming" sentences cut in bf57d6b. The four programs kept every promise
they made and returned the wrong answer; that is the limit, stated.
Same treatment applied to assets/GRAPHS.md, which carried the same framing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #120 +/- ##
=======================================
Coverage 86.62% 86.62%
=======================================
Files 15 15
Lines 2871 2871
=======================================
Hits 2487 2487
Misses 384 384
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Applying the shape of the index.html rewrite. Three problems, all of them
padding rather than content.
The three averages were stated three separate times: in the opener, again at
"Group the three languages that way", and again at "TypeScript's 99.7% is
earned with...". They now appear once, in the opener, and the static-typing
section argues from them instead of restating them.
The static-typing section was five paragraphs saying what index.html says in
two. It opens on the fact now ("The difference between the Python and
TypeScript results is not random") rather than on a definition, and the
payoff about sorting by constraint gets its own short paragraph instead of
trailing a 98-word one. Cut the hedging with it: "appears to be supplying by
other means, and the obvious candidate is the structure it imposes" became
"Vera earns very nearly the same result without that, from a single skill
file in context", which is your phrasing and better than mine.
Four other paragraphs ran past 75 words for no reason. The two-modes
explanation is two paragraphs because it describes two modes; the generations
caveat drops a clause it had already made; the reasoning-mode paragraph stops
explaining itself twice; and "turns out to be a materially easier task than"
is now "is easier than".
No paragraph in the section is over 75 words. It is 1,928 words, down from
2,290, with nothing removed but repetition.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A three-point gap on a single run per model, with some of it traceable to a grading asymmetry we have not fixed yet (#121), does not support a flat claim. 'Probably' is what the evidence carries. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Better narrative throughout: the delta section now points at the saturation section rather than asserting the ceiling twice, the two Vera modes are labelled with the column names they produce, and the caveats that were paragraphs are blockquotes. One factual correction, and it was my error rather than the rewrite's. "TypeScript scores 100% for six of the nine models" is wrong in both places it appears: it is eight of nine, with only Claude Fable 5 short at 97%. Vera is the one at six. That strengthens the saturation argument rather than weakening it, since there is even less room at the top than the text claimed. The 1.1% moved out of a parenthetical that made it read as though 1.1% of programs compiled. It is now its own sentence after the finding. Typos: "were ran", "an skills file", "this results says", "wrote program", "While benchmark", "Typescript", "langauges", a stray "and" in the refusals paragraph, and a sentence that named Vera twice for one comparison. Trailing whitespace on five lines. Every numeric claim re-verified against the v0.0.18 result rows: the win/draw/loss counts, the 13-against-3 failure split, the pro-path deltas, Vera over Aver by one to ten points, the zero-training-data averages, the 87-100% range, and both anchor links. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Splicing paragraphs in left ragged line breaks. Prose only; headings, tables, blockquotes, images and code fences untouched, and every markdown link still resolves. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The section compares two verdicts, static and runtime, so it can only use rows that have both. One program never compiled and therefore never ran, which is why the denominator is 539 rather than 9 x 60. A reader doing the multiplication now finds the answer in the sentence. It still counts as not solved in the headline: GPT-5.6 Sol reads 98% in Vera because of it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…rade moved
assets/README.md was still describing the v0.0.16 sweep in four places, all of
them wrong against the current data:
"Vera wins outright for four of the nine models" -> six against Python,
one against TypeScript, and the reason is that TypeScript reaches 100%
for eight of nine and leaves nothing to win
"Five refusals in the whole run" -> four
"Nothing moves in any of the four languages" -> Vera +2, Vera NL -2
"The seven README figures" -> eight, since fig-ztd
It also claimed the ztd slide had no committed counterpart, which stopped
being true when fig-ztd.png landed, and described fig-coverage as counting
problems when it now counts programs.
README.md carried a fifth: "Python gains 2 and TypeScript loses 4" across the
generation chain. Those were the numbers before the TypeScript export fix was
re-graded. The chart regenerated and the prose did not, so the two disagreed.
It is Python +1 and TypeScript flat, verified against a freshly rendered chart
that is byte-identical to the committed one.
Both files rewritten in the voice of the Results section rather than patched:
the finding first, no announced significance, caveats as blockquotes,
contractions kept.
Every figure the README embeds is documented in assets/README, checked
mechanically. All eight.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Depends on #117, which fixed the grading defects these numbers were produced under. Merge that first.
The numbers are new, not updated
The v0.0.16 sweep output-graded 36 of the 60 problems. The 24 it could not grade were the ADT, pattern-matching and effect-handler problems, so every previously published figure was computed on the subset that suited Python and TypeScript best. This sweep grades all 60 in all five languages. The two sets are not comparable and the README opens by saying so.
What changed in the story
Against Python, Vera wins for six of nine models. Against TypeScript it wins one, draws five and loses three — because TypeScript now scores 100% for six models and there is no headroom left to win. That is a fact about saturation rather than about Vera, and the saturation section says so plainly: closing the coverage gap made saturation worse, since the comparison languages handle the newly-graded ADT problems well.
The zero-training-data comparison is the one saturation has not eaten, and it is now the strongest result in the release: Vera 98.2%, AILANG 96.8%, Aver 92.4% across the five models that ran all three, with Vera ahead of Aver on every one of them. A figure version of that three-language chart is new and sits below the Vera-versus-Aver section.
The coverage section is rewritten, not updated
It used to argue that pass@1 could not see the problems without test cases. #107 closed that gap, so the old chart degenerated to "0 of 60" beside a 0% hero stat that read as Vera failing everything. It now reports where
vera checkandvera rundisagree: six of 539 graded pairs cleared the static gate and were still wrong (1.1%), against zero working programs refused.Scope
README.md,assets/README.md, 9 regenerated figures, 1 new figure.assets/GRAPHS.mdwas rewritten to match but is gitignored (talk notes), so it is not in this diff.Drafted against the
writing-voiceskill and audited with its checker.assets/README.mdcomes back clean;README.md's remaining findings are markdown image syntax read as exclamation marks.🤖 Generated with Claude Code
Summary by CodeRabbit
--skill-mdoverride.fig-ztd.pngfor theztdslide type.