[ci-flake] TestReferenceSolutions_JS/fizzbuzz hits its 60s node-startup timeout on Windows — red-lit dev on two unrelated commits in one window

[sunholo-voight-kampff]

Found by the V1 mission loop, iteration 139, while confirming Gate 3b on the #498 Lane B M1 merge. Not caused by either commit that it red-lit.

What happened

Build and Release → Build windows-latest failed on dev at f5ebcc0b5 with:

--- FAIL: TestReferenceSolutions_JS (92.30s)
    --- FAIL: TestReferenceSolutions_JS/fizzbuzz (60.59s)
    --- PASS: TestReferenceSolutions_JS/recursion_fibonacci (31.07s)
    --- PASS: TestReferenceSolutions_JS/graph_bfs (0.06s)
    --- PASS: TestReferenceSolutions_JS/binary_tree_sum (0.06s)
    ... every remaining subtest 0.06-0.07s
FAIL	github.com/sunholo-data/ailang/internal/eval_harness	147.808s

(Isolated with a --- FAIL grep against a 13,067-line --- PASS control, so the 2 FAIL lines are a measurement, not a scan artifact.)

internal/eval_harness/reference_solutions_test.go:89 sets timeout := 60 * time.Second. fizzbuzz failed at 60.59s — it hit that bound exactly.

Why it is not the commits

Two independent lines of evidence:

1. A descendant commit with the identical code passed. 434e3e53a is a docs-only child of f5ebcc0b5 (git merge-base --is-ancestor → true; git ls-tree 434e3e53a serveapi/ confirms the new package is present). Its Build windows-latest is success. Same Go code, green.
2. An unrelated concurrent commit failed the same way. f574c4b58 ("chore(eval): v0.32.0 curation cycle") changes benchmark YAML and a skill markdown file — zero Go — and its Build windows-latest also failed, in the same ~30-minute window.

Recent Build windows-latest on dev: f574c4b58 failure · 434e3e53a success · f5ebcc0b5 failure · 8567cd57e success · d1b851026 success · 6e82d2a1b success. Interleaved red/green across commits that cannot have caused it.

Root cause shape

The timing profile is the tell: fizzbuzz 60.59s and recursion_fibonacci 31.07s, while all eight other subtests are 0.06-0.07s. That is node cold-start stalling, not program work — these are tiny reference programs.

The test's own comment already anticipates this:

Reference solutions are tiny programs; wall-clock is dominated by interpreter startup (~slow on Windows CI runners — node alone can take >20s cold). recursion_fibonacci was getting the only 60s slot, but fizzbuzz hits the same 30s cliff on Windows. Give every benchmark the same generous slot...

So 60s was already a raise from a lower bound after the same class of failure. The Windows runner has now exceeded it too. Raising it again buys time but does not change the shape — the first JS subtest to run pays the cold-start cost, and the budget is per-subtest wall-clock that includes it.

Fix shapes worth considering (not prescribing): warm the interpreter once before the subtest loop and exclude startup from the per-case budget; or measure startup separately and make the per-case budget cover execution only; or gate the JS lane behind an explicit opt-in on Windows the way TestNetHttpPost (#561) gates on CI.

Class

Third in a recognised family: #583 (live GitHub clone), #494 (unbounded Windows subprocess), #509 (macOS cold-start guardrail). Each is a check whose verdict depends on runner or third-party conditions rather than on the code under test, and each has red-lit dev on a commit that could not have caused it. #561 is the same shape locally.

Filed by the V1 mission loop, iteration 139.

🤖 Generated with Claude Code

[sunholo-voight-kampff]

Correction from iteration 140 — evidence line 2 is wrong, retracted

The headline conclusion stands (the fizzbuzz Windows timeout is real and was not caused by f5ebcc0b5), but it now rests on evidence line 1 only. Line 2 was measured wrong, and the error was not harmless.

What line 2 claimed: "An unrelated concurrent commit failed the same way. f574c4b58 ... its Build windows-latest also failed, in the same ~30-minute window."

What is actually true, SHA-addressed via repos/.../commits/<sha>/check-runs:

• On f574c4b58, Build windows-latest was cancelled, not failed. The job that failed was Build macos-latest (plus test and test-windows in the CI workflow). The cancellations are matrix fail-fast reacting to the macos failure.
• It did not fail "the same way" at all. f574c4b58 failed on TestAllBenchmarksHaveTierAndTags and TestFilterBenchmarksByTier — a deterministic tier-distribution regression, reproducible locally in 0.5s, on both ubuntu and windows. Nothing to do with TestReferenceSolutions_JS/fizzbuzz or with node cold-start.
• Isolated with a --- FAIL grep against a 6,468-line --- PASS control in the same log, so the two FAIL lines are a measurement.

Why the mistake mattered. f574c4b58 (the v0.32.0 curation cycle) moved 12 benchmarks between tiers without updating the two Go gates that pin the tier distribution. That reddened dev on every subsequent commit. Because iteration 139 read its red as another instance of this flake, a real, deterministic, trivially-fixable regression sat on dev for roughly two hours and was reported to the human as a known runner flake.

Fixed in d434a9be9 (re-centers both gates; adds the missing follow-through step to benchmarks/CURATION.md §5, which never mentioned that these gates exist).

The generalisable lesson, recorded in the mission log: two commits going red in the same window is not evidence they failed for the same reason. Attribution to a known flake has to match on the failing test name and the platform, not on redness and adjacency — matching on co-occurrence is what let this one hide inside an existing issue.

🤖 Generated with Claude Code