_ts_declaration_regions in vera_bench/adt_render.py finds TypeScript type and interface declarations by scanning characters and tracking brace depth. It is not a lexer, and CodeRabbit is right that it does not have to be wrong to be unsafe:
braces/semicolons in quoted strings, template literals, or string content containing type X can affect depth or create false declaration starts. I did not find a current benchmark fixture that makes this a direct grading failure, but it is not robust for the full TypeScript surface.
Concretely, a solution containing any of these can perturb the scan:
const label = "a { b ;"; // unbalanced brace inside a string
const msg = `type List = ${x}`; // a false declaration start in a template
const doc = "interface X { y: 1 }"; // a false declaration regionThe consequence is not a crash but a wrong discriminant, and a wrong discriminant means the wrapper builds { kind: … } objects against a tag union — the solution's own switch reads undefined and a correct answer grades 0. That is a mis-grade, which this harness treats as strictly worse than a decline.
Why this is filed rather than patched
The last two review rounds on #114 were both defects in the previous round's fix to this exact function: the region regex terminated at the first ;, which silently rendered the declaration-priority rule inert whenever the first union member carried fields. Extending the scanner again, against inputs we are imagining rather than observing, is the process that produced those.
What the fix should look like
Given the decline-over-misgrade policy, either:
- Skip quoted and template literals during the scan so string content cannot affect depth or start a region; or
- Accept a restricted declaration grammar and decline anything outside it, so an unparseable shape becomes an honest ungraded row rather than a guess.
Option 2 is more in keeping with the rest of the module, where every unreadable declaration declines.
Evidence to gather first
#109 stores every attempt's generated code, so the first full-60 sweep will show which TypeScript declaration shapes models actually write — whether template literals and brace-bearing string constants appear in solutions at all, and in what proportion. That determines whether this needs a tokenizer or a tighter grammar with a decline path.
Raised by CodeRabbit on #114.
🤖 Generated with Claude Code
_ts_declaration_regionsinvera_bench/adt_render.pyfinds TypeScripttypeandinterfacedeclarations by scanning characters and tracking brace depth. It is not a lexer, and CodeRabbit is right that it does not have to be wrong to be unsafe:Concretely, a solution containing any of these can perturb the scan:
The consequence is not a crash but a wrong discriminant, and a wrong discriminant means the wrapper builds
{ kind: … }objects against atagunion — the solution's ownswitchreadsundefinedand a correct answer grades 0. That is a mis-grade, which this harness treats as strictly worse than a decline.Why this is filed rather than patched
The last two review rounds on #114 were both defects in the previous round's fix to this exact function: the region regex terminated at the first
;, which silently rendered the declaration-priority rule inert whenever the first union member carried fields. Extending the scanner again, against inputs we are imagining rather than observing, is the process that produced those.What the fix should look like
Given the decline-over-misgrade policy, either:
Option 2 is more in keeping with the rest of the module, where every unreadable declaration declines.
Evidence to gather first
#109 stores every attempt's generated code, so the first full-60 sweep will show which TypeScript declaration shapes models actually write — whether template literals and brace-bearing string constants appear in solutions at all, and in what proportion. That determines whether this needs a tokenizer or a tighter grammar with a decline path.
Raised by CodeRabbit on #114.
🤖 Generated with Claude Code