pipeline: a degraded verdict re-opens cleanly finished work — heuristic fallback ignores a confirmed verify_done flip and demands flip evidence non-code tasks cannot produce

[macanderson]

Problem

When the model verdict degrades (verifier response unparseable), the heuristic fallback can return passed: false against a task the worker has already finished and cleanly stopped on — re-opening execution and converting solved tasks into loop-kills and harness timeouts. Three distinct defects compound here, all observed in match cc00894779ff (SUT 447f339, 2026-08-07):

1. The fallback does not consume verify_done results. In extract-elf__G9aMhUC, verify_done proved a genuine baseline-pinned fail→pass flip and printed WITNESS CONFIRMED (twice, byte-identical) — yet both verdict ladders assert "flip_achieved": false and fail on "no flip". A confirmed flip and flip_achieved: false coexist in the same trace. 780s of 1022s (76%) was rework driven by a verifier that never spoke.
2. The heuristic is structurally biased to FAIL non-code tasks. In count-dataset-tokens__zitF9qC the deliverable is a one-line answer.txt; there are no tests to flip. The worker solved it, emitted finish_reason=stop with an accurate summary — and the fallback ("no flip, touched tests not confirmed green") re-opened execution anyway. ~610s and 45% of the trial's spend went to re-deriving an answer that never changed. Same shape in log-summary-date-ranges__Z338iwW: solved at t=36s, "Done." at t=48s, degraded verdict re-opened it at t=77s, loop-detector killed it at t=346s (the recorded DeliberateStopError is that kill, not a clean stop).
3. Degradation is under-reported. In extract-elf, round 1's degradation emitted an error + verdict_degraded proof; round 2's identical degradation emitted neither — invisible in aggregate telemetry (fix(stella-pipeline,stella-protocol): record verifier degradation per candidate in a fan-out (#1787) #1951 deduplicated the warning; the proof event should still be per-occurrence).

Secondary waste multiplier: identical-argument verify_done re-runs are not memoized — extract-elf paid 127.8s + 122.1s for two byte-identical calls returning byte-identical output (24% of its budget).

Evidence

Traces under ~/.arenabench/matches/cc00894779ff/jobs/cc00894779ff-stella-sonnet-5-full-pipeline/:

• extract-elf__G9aMhUC/agent/stella-events.jsonl — verify_done WITNESS CONFIRMED at idx 339/478 vs verdicts at idx 249/374 with flip_achieved: false; round-2 degradation with no proof event.
• count-dataset-tokens__zitF9qC/agent/stella-events.jsonl — worker finish_reason=stop (line 122) before the degraded verdict re-opened execute (line 135).
• log-summary-date-ranges__Z338iwW/agent/stella-events.jsonl — verdict summary verbatim: "verifier unavailable; heuristic fallback failed (no flip, touched tests not confirmed green)".

All three verdicts fired only because the verdict model returned empty output (companion issue: reasoning-starved role output caps).

Fix directions

• The fallback ladder must read the run's verify_done/oracle results before asserting flip_achieved: false.
• A degraded verdict should not be able to re-open work on evidence it structurally cannot obtain: when triage classified the task without a witness (or no touched tests exist), the heuristic should weigh the healthy signals it does have (diff produced, diagnostics clean, worker self-stop) or abstain rather than fail.
• Emit verdict_degraded proof per occurrence, even when the warning is deduplicated.
• Memoize verify_done on identical (test_cmd, test_files, tree state).

Related

• Verdict output contract is fragile: no structured-output path, unbounded reasoning, degradations warned once per run across a fan-out #1787 (verdict contract; the heuristic's fail-bias is named in its item 1), The plan-step loop walks every step even when step 1 finished the task — 63% of a run's cost re-confirming finished work #1702 (re-confirming finished work), fix(stella-pipeline,stella-protocol): record verifier degradation per candidate in a fan-out (#1787) #1951 (degradation warning dedup), pipeline: add a shell runner to RUNNER_VOCABULARY so the witness stage is reachable on runner-less workspaces #2064/verify_done: the shadow baseline is HEAD, which auto-snapshots already advanced past — no flip is provable, and a missing build artifact fakes one #2067 (the flip machinery the fallback ignores).

Definition of done

A trace where the worker stops cleanly and verify_done has confirmed a flip can no longer produce a failing heuristic verdict; a replay-fixture test pins that. Round-N degradations each emit a proof event.