Two defects that compound, and together they account for most of the Python-versus-TypeScript gap in the v0.0.18 results.
1. The problem under-specifies the data type
VB-T3-002 (and VB-T3-008) show the model only this:
Define a binary tree type with leaf and branch constructors, then write a function that computes the depth. A leaf has depth 1.
Nothing there says a leaf carries a value, and for computing depth a leaf value is irrelevant. But the adt block declares Leaf: ['Int'] and the test cases feed {"Leaf": [5]}, so a model that writes
@dataclass(frozen=True, slots=True)
class Leaf:
passhas read the description correctly and is marked wrong. Four of nine models took that reading on VB-T3-002 and four on VB-T3-008.
The description_neutral text needs to specify the shape the test cases assume, in every problem where the model defines the data type.
2. The same reading is graded differently in each language
This is the sharper half. On VB-T3-008, four TypeScript solutions declared a leaf with no value and all four were scored solved; four Python solutions made the identical choice and all four were scored not solved.
The cause is where each language's type discipline actually bites:
- TypeScript is structurally typed. The generated wrapper passes an object literal, and a function whose type has no
value field simply ignores the extra property at runtime. The mismatch never surfaces.
- Python's dataclass constructor is nominal and positional.
Leaf(5) against a fieldless class raises TypeError: Leaf.__init__() takes 1 positional argument but 2 were given, and the problem is recorded as a wrong answer.
So at the boundary the grader uses (constructing the test value), Python is the stricter language, and it is penalised for it. That inverts the usual reading, where TypeScript is the more constrained of the two.
Scale
Python has 13 runtime wrong answers across the v0.0.18 sweep against TypeScript's 1; nine of Python's are this case. The published gap is 96.7% against 99.7%, about three points, and this accounts for most of it.
Related: #119 is the narrower harness half (a nullary Python class read as unverifiable rather than arity 0, so the shape guard renders a call it should refuse). Fixing #119 converts these from a raw TypeError into a clean decline, which is better but still not equal treatment; the grading asymmetry needs both problems respecified and the two languages brought into line.
Two defects that compound, and together they account for most of the Python-versus-TypeScript gap in the v0.0.18 results.
1. The problem under-specifies the data type
VB-T3-002(andVB-T3-008) show the model only this:Nothing there says a leaf carries a value, and for computing depth a leaf value is irrelevant. But the
adtblock declaresLeaf: ['Int']and the test cases feed{"Leaf": [5]}, so a model that writeshas read the description correctly and is marked wrong. Four of nine models took that reading on
VB-T3-002and four onVB-T3-008.The
description_neutraltext needs to specify the shape the test cases assume, in every problem where the model defines the data type.2. The same reading is graded differently in each language
This is the sharper half. On
VB-T3-008, four TypeScript solutions declared a leaf with no value and all four were scored solved; four Python solutions made the identical choice and all four were scored not solved.The cause is where each language's type discipline actually bites:
valuefield simply ignores the extra property at runtime. The mismatch never surfaces.Leaf(5)against a fieldless class raisesTypeError: Leaf.__init__() takes 1 positional argument but 2 were given, and the problem is recorded as a wrong answer.So at the boundary the grader uses (constructing the test value), Python is the stricter language, and it is penalised for it. That inverts the usual reading, where TypeScript is the more constrained of the two.
Scale
Python has 13 runtime wrong answers across the v0.0.18 sweep against TypeScript's 1; nine of Python's are this case. The published gap is 96.7% against 99.7%, about three points, and this accounts for most of it.
Related: #119 is the narrower harness half (a nullary Python class read as unverifiable rather than arity 0, so the shape guard renders a call it should refuse). Fixing #119 converts these from a raw
TypeErrorinto a clean decline, which is better but still not equal treatment; the grading asymmetry needs both problems respecified and the two languages brought into line.