You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The Python and TypeScript test wrappers are each built in two places: the LLM evaluator (runner.py) and the baseline runner (baseline_runner.py). Four near-identical builders that must stay in lockstep by hand.
This session demonstrated the hazard twice in one day:
The TypeScript array-comparison fix (JSON.stringify for arrays, loose == for primitives) had to be applied at both TS sites, and the Python stdout-shielding at both Python sites — PR Grade 46 of 60: the Vera test wrapper, the baselines gate, and v0.0.17 #110. Missing one copy would have produced the exact class of asymmetry the PR exists to remove: canonical solutions validated under different rules than graded code.
The two copies had already drifted cosmetically (the bool-coercion comments differ), which is how real drift starts.
The Vera path avoids this by construction: both callers route through the shared vera_bench/vera_wrapper.py.
Proposed
Extract the per-language wrapper generation into shared helpers alongside vera_wrapper.py (or a wrapper_gen.py), parameterised by the small real differences (module stem, attempt suffix). runner.py and baseline_runner.py import the same implementation, and a change to comparison semantics happens exactly once.
Flagged by CodeRabbit on PR #110; deferred there because a four-site refactor of both evaluation paths is not a minimal review-response change, and both copies are currently in sync with tests.
Problem
The Python and TypeScript test wrappers are each built in two places: the LLM evaluator (
runner.py) and the baseline runner (baseline_runner.py). Four near-identical builders that must stay in lockstep by hand.This session demonstrated the hazard twice in one day:
JSON.stringifyfor arrays, loose==for primitives) had to be applied at both TS sites, and the Python stdout-shielding at both Python sites — PR Grade 46 of 60: the Vera test wrapper, the baselines gate, and v0.0.17 #110. Missing one copy would have produced the exact class of asymmetry the PR exists to remove: canonical solutions validated under different rules than graded code.The Vera path avoids this by construction: both callers route through the shared
vera_bench/vera_wrapper.py.Proposed
Extract the per-language wrapper generation into shared helpers alongside
vera_wrapper.py(or awrapper_gen.py), parameterised by the small real differences (module stem, attempt suffix).runner.pyandbaseline_runner.pyimport the same implementation, and a change to comparison semantics happens exactly once.Flagged by CodeRabbit on PR #110; deferred there because a four-site refactor of both evaluation paths is not a minimal review-response change, and both copies are currently in sync with tests.
🤖 Generated with Claude Code