feat(eval): spreadsheetBench V2 quality gate — read-only official-score auditor#210
Merged
Merged
Conversation
…ot promoted) Recovered from the pre-split working tree (wip/full-tree-backup-20260717): a self-contained V2 quality gate — categories, expected task count (321), accepted baselines, artifact/receipt audits, pass|blocked verdict, dense formatter — plus its standalone CLI and 42-assertion test. Verified against current origin/main: typecheck clean, 9/9 tests pass. Distinct concern from upstream's spreadsheetBenchOfficialV2Acceptance.ts (a 2-export type guard), so additive, not redundant. Zero collision with upstream. NOT wired into package.json and NOT pushed: this is a grading/certification asset, which CLAUDE.md holds locked pending explicit promotion. Held as a ready-to-PR branch for that decision. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
benchmark:spreadsheetbench:v2-quality-gate — read-only auditor over accepted official receipts and model-run receipts. Default emits one deterministic dense line (exit 0); --strict exits 1 unless every sub-gate passes. Never invokes a model, provider, or scorer. Live run against current repo receipts: status=BLOCKED with named blockers (scorer_authenticity, editing_plan_integrity, category_performance, visualization_vlm) — the gate fails closed on today's data by design. The sibling chart-contract draft (spreadsheetBenchChartContract) is deliberately NOT promoted; it remains parked on wip/full-tree-backup-20260717 until a benchmark lane needs chart-task grading. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Scaffold Handoff — For Your Coding Agent
Immutability CheckMode: advisory ✅ No immutable files were modified in this branch. Changed Files
Needs Adversarial Review — Do NOT Apply YetThese proposals passed the reject check but have not been approved by
Safety Boundary
Immutable files (never modify):
Scaffold files (safe to modify):
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Promotes the spreadsheetBench V2 quality gate salvaged from the pre-fleet working
tree, verified against current main before promotion:
src/eval/spreadsheetBenchV2QualityGate.ts(1,458 L) — deterministic auditor overthe accepted official V2 scorer receipt + model-run receipts: coverage vs expected
task counts, trace/candidate/scorer receipt presence, authenticity, editing-plan
integrity, per-category performance, VLM coverage.
scripts/spreadsheetbench-v2-quality-gate.ts— CLI (--densedefault,--json,--json-out,--strict). Read-only; never invokes a model, provider, or scorer.tests/spreadsheetBenchV2QualityGate.test.ts— 9 scenario tests.benchmark:spreadsheetbench:v2-quality-gate.Promotion provenance (certification-lock note)
This is a human-directed promotion of a locked grading asset, not the harness
grading itself. The gate was authored in exploration, stranded un-committed when the
codex fleet's PRs #191–#208 superseded the surrounding draft work, and salvaged only
after passing verification against current
main: 0 typecheck errors, 9/9 tests.Merge authority stays with the repo owner.
Live proof (current repo receipts)
Default exit 0 (report mode);
--strictexit 1. BLOCKED is the correct verdict ontoday's receipts — the gate fails closed by design and is not tuned to pass.
Deliberately excluded
spreadsheetBenchChartContract(+ tests) is NOT in this PR: 2/13 of its tests failagainst current main (receipt-emission contract drifted during the fleet rewrite).
It stays parked on
wip/full-tree-backup-20260717until a benchmark lane needschart-task grading; reconciliation must keep strictness opt-in.
🤖 Generated with Claude Code