LoopGain stopped but my error wasn't zero — bug or intended?

[fitz2882]

[fitz2882]

Intended, and it's the most important thing to understand about what LoopGain is telling you.

There are two different terminal states that both look like "it stopped":

TARGET_MET — observed error hit or dropped below target_error. This is a clean success. The loop finished the job.

STALLING (surfacing as outcome == "stalled") — the error stopped moving in a statistically detectable way, two readings in a row, while still above your target. LoopGain is saying more iterations won't help, which is not the same as the answer is good. You've plateaued at, say, 2 failing tests and burning 15 more iterations won't clear them.

That distinction is the whole design. LoopGain decides when to stop; you decide whether the answer is good enough. It has no way to see the second thing.

So check the result, don't just trust termination:

result = lg.result
if result.outcome == "converged" and result.best_error <= my_threshold:
    ship(result.best_output)
else:
    escalate(result.best_output, result.best_error)   # plateau, oscillation, or divergence

result.best_output is always the lowest-error iteration in the buffer, not the last one — so even on DIVERGING you get the best thing the loop ever produced rather than the garbage it ended on.

The sharper version of this problem, which the docs are explicit about: if your verifier reports zero errors on output that's actually wrong, LoopGain trusts it and stops with a clean TARGET_MET. Nothing in the convergence signal can flag that. On the benchmark's code-gen workload, 4.5% of converged runs (16/355) passed every check the loop ran and still failed the full held-out test suite — and that's a floor, because the in-loop verifier there was strong.

Pair it with the strongest verifier you can afford: executable tests over a sampled subset, a schema check over a vibe, a held-out check the loop didn't optimize against.