Gate the facts docs/index.html and docs/index.md both state - #1174
Merged
Conversation
docs/index.md is not derived from the landing page: it is written out by build_index_md() in scripts/build_site.py, which holds the HTML's substance as a hand-maintained f-string. check_site_assets.py verified it by calling that generator and comparing the result against the committed file, putting the generator on both sides of the comparison — so a generator that missed an edit to the hand-designed HTML produced a committed asset stale in exactly the same way, and the gate passed. A v0.0.7-era benchmark section survived every intervening release that way. check_fact_coherence() extracts the load-bearing facts from each file independently and fails when they diverge, naming the fact, both values and both paths. A fact that cannot be located is itself a failure rather than a silent skip, so a reworded sentence cannot switch its own check off. Co-Authored-By: Claude <noreply@anthropic.invalid>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Plus Run ID: 📒 Files selected for processing (4)
🔗 Linked repositories identifiedCodeRabbit considers these linked repositories for cross-repo context during reviews:
📝 WalkthroughWalkthroughThe site asset checker now independently validates shared facts in ChangesLanding-page coherence
Estimated code review effort: 3 (Moderate) | ~25 minutes Sequence Diagram(s)sequenceDiagram
participant Checker as check_site_assets.py
participant HTML as docs/index.html
participant Markdown as docs/index.md
Checker->>HTML: Extract landing-page facts
Checker->>Markdown: Extract landing-page facts
Checker->>Checker: Compare facts and report errors
Possibly related PRs
Suggested labels: 🚥 Pre-merge checks | ✅ 7 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (7 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1174 +/- ##
=======================================
Coverage 93.81% 93.81%
=======================================
Files 99 99
Lines 33457 33457
Branches 458 458
=======================================
Hits 31389 31389
Misses 2055 2055
Partials 13 13
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Owner
Author
|
Adversarial review (maintainer-side, pre-CI): clean. Verified against the diff at cdbe94c:
One non-blocking observation: the editor-support check matches names file-wide, so an editor name surviving in unrelated prose (e.g. a historical mention elsewhere on the page) could satisfy the set even if the actual support claim dropped it. That failure mode errs conservative-quiet rather than false-alarm, and tightening it to the claim's sentence would re-introduce wording-fragility, so I think the trade is right as shipped — noting it so the choice is on the record. Note for whoever merges second: this PR and the in-flight #1111 fix both edit TESTING.md's headline test counts (this one 8,568→8,587, that one →8,578; post-both reality is 8,597) — the trailing rebase reconciles them. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tests/test_build_site.py`:
- Around line 633-647: Add tests in tests/test_build_site.py covering both
untested check_fact_coherence failure paths: duplicate the Kimi K3 Markdown row
to exercise _index_rows and assert the error mentions “twice”, the model name,
and the Markdown path; then mutate that row to have an extra cell to exercise
_bench_rows_md and assert the error mentions “cells” and the Markdown path.
- Around line 470-471: Update the version-badge mutation in the test around
_landing_pair and _edit to anchor on the version’s digit pattern rather than the
literal “0.1” prefix. Follow the sibling tests’ _sub-based approach so the
assertion remains valid across releases while still changing the intended
version value.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: ab2c097e-4dd1-4cd5-805a-1c1dfb1216da
📒 Files selected for processing (9)
CHANGELOG.mdCLAUDE.mdCONTRIBUTING.mdKNOWN_ISSUES.mdREADME.mdROADMAP.mdTESTING.mdscripts/check_site_assets.pytests/test_build_site.py
🔗 Linked repositories identified
CodeRabbit considers these linked repositories for cross-repo context during reviews:
aallan/vera-bench(manual)
💤 Files with no reviewable changes (1)
- KNOWN_ISSUES.md
…ge mutation Review round: the duplicate-model-row (_index_rows) and wrong-cell-count (_bench_rows_md) error paths had no exercising test. Two tests now mutate the Markdown benchmark table located via the checker's own header regex, asserting the "twice" / "cells" messages name the row and the file. The version-badge mutation anchored on a literal "[0.1" prefix that would go stale at the next minor release; it is digit-anchored via _sub like its sibling tests. Per-file and headline test counts move 8,587 -> 8,589. Co-Authored-By: Claude <noreply@anthropic.invalid>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The blind spot
docs/index.mdis the Markdown companion agents fetch viarel="alternate"andllms.txt— an LLM asking about Vera's benchmark results is served that file, not the landing page. It is not derived fromdocs/index.html. It is written out bybuild_index_md()inscripts/build_site.py, which holds the landing page's substance as a hand-maintained f-string.scripts/check_site_assets.pyverified it by calling that generator and comparing the result against the committed file. Both sides of that comparison come from the same function, so a generator that missed an edit to the hand-designed HTML produced a committed asset stale in exactly the same way, and the check was satisfied. The gate proved the asset matched the generator; nothing proved the generator matched the HTML it mirrors. A v0.0.7-era benchmark section survived every intervening release until it was caught by hand.This is option 2 from the issue: keep the generator, gate the pair. It extends
check_site_assets.pyrather than adding a script — this is that script's documented blind spot, and it inherits the existing CI wiring (.github/workflows/ci.yml, lint job). No new CI job or pre-commit entry.Facts gated
check_fact_coherence(html_path, md_path)extracts each fact from both files independently and compares:VeraBench v0.0.18in bothVera v0.1.8in both<span>v<a …>0.1.8</a>, MD**Current version:** [0.1.8]A 60-problem benchmarkacross 5 difficulty tiersNine models, three providersSix of nine frontier modelsCompared as facts, not wording: number words fold onto digits (
Nine=9), HTML entities and tags are stripped from table cells, Markdown emphasis is stripped (**100%**=100%— the win/loss marking is the presentational equivalent of the HTML's CSS class), and percentages compare as strings. The two files' prose registers are left alone.check_version_sync.pyis not duplicated: it pins the HTML badge topyproject.tomland never readsdocs/index.md. The new check compares the two documents to each other.Extraction failure is gate failure. If a fact cannot be located in either file, that is an error naming the fact and the file — a reworded sentence must not be able to switch its own check off. A file stating one fact two different ways is also an error.
An example failure, from a deliberately drifted
docs/index.html:RED-first evidence
19 tests were written in
tests/test_build_site.pybeforecheck_fact_coherenceexisted, and all 19 failed. Each drift case copies the committed pair intotmp_pathand mutates the copy — the real files are never touched — and the mutation helper asserts its anchor is unique, so a reworded landing page turns the test red rather than silently no-op'ing.Proved to fail: a bumped percentage in the HTML table, and in the Markdown table; a bumped VeraBench version; a bumped tested-Vera version; a diverged version badge; a changed problem count; a changed model count (which also trips the rows-vs-prose cross-check); a removed HTML table row, and a removed Markdown row; a renamed model; a changed tier; a dropped editor. Extraction failures proved to fail: a removed HTML
<table class="bench-table">, a removed Markdown table header, a deleted results caveat (both facts on that line named), a removed editor claim, a missing file, and one file stating the VeraBench version two different ways.Then mutation-validated: six extraction regexes were broken one at a time — the HTML row extractor, the HTML table anchor, the VeraBench version pattern, the Markdown table header, the Vim editor pattern, the Markdown emphasis stripper — and each turned at least one test red. None is dead weight.
The committed
docs/index.html/docs/index.mdpair passes: all 9 table rows match cell for cell, all 8 prose facts agree, and all three editors are named in both. No existing divergence was found.Full gate green:
pytest tests/(8,471 passed),check_site_assets.py,check_doc_counts.py,check_explicit_encoding.py,check_version_sync.py,ruff check .,ruff check --select S vera/,mypy vera/.Docs lockstep
CHANGELOG
[Unreleased]gains a### Fixedentry; the#1154row is removed fromKNOWN_ISSUES.md(no matching row existed inSKILL.md);CLAUDE.mdandTESTING.mddescribe what the script now checks;CONTRIBUTING.mdsays plainly thatdocs/index.mdhas to be edited inbuild_index_md()alongside the HTML. TheTESTING.md/README.md/ROADMAP.mdtest counts move with the 19 new tests.Closes #1154
🤖 Generated with Claude Code
Summary by CodeRabbit
Bug Fixes
Documentation
Tests