docs: update the VeraBench section for the v0.0.18 sweep - #1169
Conversation
The first sweep to grade all 60 problems in every language, so every number in the section moves and the denominator halves in significance: one problem is now worth 1.7 percentage points rather than just under three. Six of nine models write Vera perfectly, down from seven, and the headline stat says so. The table carries the new scores with the same win/loss marking, and the delta chart is the v0.0.18 render. The summary line under the table changes shape rather than just its numbers. "Vera wins outright for four" no longer describes the result: TypeScript now reaches 100% for six of the nine models, so there is very little room at the top for anything to win by, and claiming a win there would be reading noise. What the data does support is that Vera is the highest score or level with it for six of nine. Two paragraphs are new, below the chart, because the Python and TypeScript columns now differ in a way worth explaining. Python is dynamically typed and finds a type error when the code runs; TypeScript rejects it first; Vera sits on the TypeScript side and goes further with mandatory contracts and typed slot references. Sorted by how much each constrains the model rather than by how much of it the model has read, the ordering stops looking accidental. That leads to the point the section was already gesturing at. TypeScript earns its score with an enormous amount of the language in every training set; Vera earns very nearly the same score with none of it. Among the three languages here that no model has been trained on, the most constrained scores highest. Numbers verified against the VeraBench v0.0.18 result rows rather than transcribed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Plus Run ID: ⛔ Files ignored due to path filters (2)
📒 Files selected for processing (1)
🔗 Linked repositories identifiedCodeRabbit considers these linked repositories for cross-repo context during reviews:
📝 WalkthroughWalkthroughThe VeraBench documentation is updated from v0.0.16 to v0.0.18, with revised benchmark results, full 60-problem grading details, updated scoring impact, and refreshed generated site content. ChangesVeraBench documentation refresh
Estimated code review effort: 2 (Simple) | ~10 minutes Possibly related PRs
Suggested labels: 🚥 Pre-merge checks | ✅ 8✅ Passed checks (8 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1169 +/- ##
=======================================
Coverage 93.81% 93.81%
=======================================
Files 99 99
Lines 33457 33457
Branches 458 458
=======================================
Hits 31389 31389
Misses 2055 2055
Partials 13 13
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
'The Python and TypeScript columns are worth reading against each other, because they differ in exactly the way that matters here' and 'Which is the whole argument in one column' both told the reader what to think before saying anything. The paragraphs now open on the claim itself: Python and TypeScript differ in exactly the way that matters, and TypeScript earns its score with the language in every training set. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The section said 'four modes each' above a table with three columns, so a reader counts and comes up short. The missing one is the spec-from-NL run, where the model infers the types and writes its own contracts before writing any code, and it is the mode with by far the widest spread across models. Named all four and said the table shows three of them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
One instance against seven correct ones elsewhere on the page. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The HTML edit reaches one of five surfaces that carry these numbers. `docs/index.md` is not derived from the HTML -- it is generated by `build_index_md()` in `scripts/build_site.py`, which holds the benchmark prose as its own copy. Editing the page therefore leaves the Markdown companion agents fetch showing the previous sweep, and `check_site_assets.py` cannot see it: the check regenerates from the same function it compares against, so both sides move together and a stale generator validates as up to date. That is #1154. The generator now carries the v0.0.18 prose and table, and index.md and llms-full.txt are regenerated from it. `README.md` and `FAQ.md` each state the figures independently and were still on v0.0.16 -- seven of nine at 100%, "wins outright for four", 36 output-gradeable problems at 2.8 points each. Every number was cross-checked against vera-bench#120, the pending results rewrite in the benchmark repo, before being copied anywhere: all 27 published cells agree, "six of nine at 100%" agrees, and "highest or level for six of nine" agrees. The two headline claims were also recomputed from the table itself rather than taken on trust. Also restores one space of indentation on the VeraBench section comment, which the HTML edit dropped. Co-Authored-By: Claude <noreply@anthropic.invalid>
|
Completed the surfaces the HTML edit doesn't reach. Pushed at The data is sound — verified, not assumedRecomputed both headline claims from the table itself before touching anything else: six models at 100% Vera, and Vera highest-or-level for six of nine. Both match the prose exactly. Then cross-checked every figure against vera-bench#120, the pending results rewrite in the benchmark repo. All 27 published cells agree, as do both headline counts. The website's numbers and the benchmark's are the same measurement, not two transcriptions of it. What was missing
Left alone deliberately: Also restored one space of indentation on the VeraBench section comment, which the HTML edit dropped. One sequencing notevera-bench#120 should land first, or together with this. v0.0.18 is tagged, but the benchmark README on This PR points Gates: |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@scripts/build_site.py`:
- Line 562: Update the benchmark commentary sentence in the generated-site
source to state that each problem is worth 1.7 percentage points instead of
“just under two percentage points,” then regenerate the derived documentation so
all published surfaces use the precise figure consistently.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 6acf99b1-de82-453d-8050-76954ca5f62e
⛔ Files ignored due to path filters (4)
docs/index.htmlis excluded by!docs/**docs/index.mdis excluded by!docs/**docs/llms-full.txtis excluded by!docs/**docs/verabench-delta-chart.pngis excluded by!**/*.png,!docs/**
📒 Files selected for processing (4)
CHANGELOG.mdFAQ.mdREADME.mdscripts/build_site.py
🔗 Linked repositories identified
CodeRabbit considers these linked repositories for cross-repo context during reviews:
aallan/vera-bench(manual) → reviewed against open PR#120docs/v0-0-18-resultsinstead of the default branch
"is not random" asserts a statistical result the sweep cannot support: a single run per model, no pass@k, and gaps that are mostly one or two problems wide. "probably not random" is what the data carries. Also aligns the wording across the two copies. The HTML read "The difference we see between..." and the generator "The difference between..." -- a drift introduced when the paragraph was transcribed into `build_index_md()`. Both now read the same sentence. Co-Authored-By: Claude <noreply@anthropic.invalid>
1 participant
Updates the VeraBench section for the v0.0.18 sweep, the first to output-grade all 60 problems in all five languages. Numbers were read out of the result rows rather than transcribed, and I verified the rendered table against them cell by cell.
What moved
The delta chart is the v0.0.18 render, same dimensions, so the
width/heightattributes are unchanged.The summary line changes shape, not just its numbers
"Vera wins outright for four of the nine models, draws with three and loses two" no longer describes the result. TypeScript now reaches 100% for six of the nine, so there is very little room left at the top and claiming a win there would be reading noise. It now says Vera is the highest score or level with it for six of nine, and names the ceiling as the reason the margins are small.
Two new paragraphs, below the chart
The Python and TypeScript columns now differ in a way worth explaining to a reader looking at the chart, and the difference is the same axis Vera sits on:
The second paragraph carries that to the point the section was already gesturing at with "enough structure to compensate for zero training data", now with the evidence attached: TypeScript earns its score with an enormous amount of the language in every training set, Vera earns very nearly the same score with none of it, and among the three zero-training-data languages the most constrained one scores highest.
Stated as compensation rather than superiority, deliberately. Vera is a point behind TypeScript on average and the chart shows it; the claim is that the constraints are worth roughly what a training corpus is worth, which is both defensible and the more interesting result.
Related
Companion PR in the benchmark repo: aallan/vera-bench#120
🤖 Generated with Claude Code
Summary by CodeRabbit