SFT: run evaluators on post-final-optim weights before final checkpoint

[Butanium]

Summary

• In tinker_cookbook/supervised/train.py, add an evals_final block inside the if did_train: branch that re-runs evaluators on the post-final-optim weights at step = total_steps, just before checkpoint_mgr.save_final_async(...).
• Only fires when evaluators is non-empty and config.eval_every > 0 (i.e. when eval is enabled at all), so default behavior for runs without evaluators is unchanged.

Motivation

The in-loop eval check in the SFT loop is

if evaluators and config.eval_every > 0 and step % config.eval_every == 0:
    ...run_evals(evaluators, training_client, step)...

and runs before that iteration's forward_backward_async / optim_step_async, so it snapshots pre-step weights. Two consequences:

1. The last in-loop eval logs metrics at step = floor((total_steps - 1) / eval_every) * eval_every, on weights that have undergone that many optim steps.
2. The final checkpoint, saved by checkpoint_mgr.save_final_async immediately after the loop, captures weights that have undergone total_steps optim steps.

So even in the vanilla case (eval_every=10, total_steps=100), the last in-loop eval fires at step=90 and reports metrics for weights ~10 optim steps stale relative to the final checkpoint. In the worst case (when total_steps is one short of a multiple of eval_every), the in-loop eval can be eval_every - 1 steps stale. For users who treat the final-checkpoint metrics as the model's "final" evaluation, this is a silent mismatch between what got logged and what got saved.

This PR closes that gap by running the configured evaluators once more at step = total_steps, immediately before the final checkpoint save, matching the checkpoint's step label.

Notes / alternatives considered

• Could be opt-out via a new final_eval: bool = True config flag. Happy to add one if reviewers prefer — the current design treats "if you have evals configured, you almost certainly want one at the genuinely-final weights" as the obvious default.
• Could instead change the in-loop condition to evaluate post-step weights instead of pre-step, but that would shift the step labels of every existing run's metrics — a much more invasive change.

🤖 Drafted with Claude Code — Co-Authored-By: Claude Opus 4.7 noreply@anthropic.com

[Butanium]

pretty useful if you want to ensure the final checkpoint is evaled

[Butanium]

check failed because of hf api error in some unrelated test, reopening to retrigger test