vera-bench baselines deletes its output file before it has a replacement:
output_path = output_dir / f"{language}-baseline.jsonl"
if output_path.exists():
output_path.unlink() # cli.py — then the run happensAny interruption between that unlink and the write at the end leaves no file at all. results/ is gitignored, so there is no recovery except re-running.
results/aver-baseline.jsonl disappeared three separate times during the v0.0.18 sweep work on 27 July 2026. It was regenerated and verified at 60 rows / 60 passing each time, and was absent again later. The other three baselines survived, which fits: aver's run is the one most often interrupted, because it is slow enough to still be going when something else is stopped.
The trigger is not the interesting part. The design is: a tool that destroys good data before it has produced the data meant to replace it will lose that data eventually.
The fix is the pattern already used in scripts/rerun_failed.py and scripts/regrade.py — write to a temporary file in the same directory and os.replace() it into position. The existing file survives every failure mode, and the swap is atomic.
Worth applying to vera-bench run for the same reason; it unlinks its output at startup too, which is why an interrupted sweep target loses everything rather than being resumable.
vera-bench baselinesdeletes its output file before it has a replacement:Any interruption between that unlink and the write at the end leaves no file at all.
results/is gitignored, so there is no recovery except re-running.results/aver-baseline.jsonldisappeared three separate times during the v0.0.18 sweep work on 27 July 2026. It was regenerated and verified at 60 rows / 60 passing each time, and was absent again later. The other three baselines survived, which fits: aver's run is the one most often interrupted, because it is slow enough to still be going when something else is stopped.The trigger is not the interesting part. The design is: a tool that destroys good data before it has produced the data meant to replace it will lose that data eventually.
The fix is the pattern already used in
scripts/rerun_failed.pyandscripts/regrade.py— write to a temporary file in the same directory andos.replace()it into position. The existing file survives every failure mode, and the swap is atomic.Worth applying to
vera-bench runfor the same reason; it unlinks its output at startup too, which is why an interrupted sweep target loses everything rather than being resumable.