eval: CheckpointEvalFn — one contract for checkpoint-consuming eval backends

[yueming-yuan]

Chained on #1740. Refactors the eval backend surface it introduced; no change to the shared-engine default path (byte-identical behavior, guarded by a regression test).

Design

Eval has exactly two postures: against the live training engines (version pinned by blocking call order) or against a checkpoint file (version pinned by the file itself). This PR gives the second posture one contract:

class CheckpointEvalFn(abc.ABC):
    async def evaluate_checkpoint(self, checkpoint_dir, input) -> RolloutFnEvalOutput: ...
    def dispose(self): ...          # tear down anything launched in __init__
# raise EvalSkip(reason) for an attributable skipped point (eval/skipped_{reason})

__init__ prepares the backend (launch or attach); each call receives a snapshot path and returns results. The trainer owns everything else: per-point HF export, async dispatch/overflow policy, logging at the snapshot's step, snapshot GC. Because the fn runs in-job, all eval config (datasets, sampling, rm, lora) comes from the real training args — nothing is hand-copied.

• FleetEvalFn (core): the dedicated fleet (--eval-num-gpus) as a backend of this contract — absorbs the health-probe/pin/router logic that lived in RolloutManager and EvalFleetSession, then delegates generation to the inner eval fn with the fleet's GenerateState (custom eval fns keep working on the fleet).
• ExternalSglangEvalFn (example): self-contained black box — launches its own sglang server on user-named GPUs (MILES_EXTERNAL_EVAL_GPUS, extra flags via MILES_EXTERNAL_EVAL_SERVER_ARGS) or attaches to one (MILES_EXTERNAL_EVAL_URL); pins each snapshot (load → read back version → retry) and runs the standard eval datasets.
• A non-sglang black box implements the same contract by submitting checkpoint_dir to its API and mapping the response into RolloutFnEvalOutput.

Routing collapses to one discrimination point (resolve_checkpoint_eval_fn at manager construction, fail-fast validation included) and one snapshot path (RolloutManager._eval_checkpoint); EvalDispatcher no longer loads the eval fn in the driver — it asks the manager once via eval_uses_snapshots().

examples/fully_async/run_qwen3_5_4b_fully_async_eval.py launches either backend behind one flag (--eval-backend fleet|external), replacing the fleet-only shell script.

Removed (deliberate capability regressions vs #1740)

tools/checkpoint_eval_service.py is deleted rather than kept as a second driver: its hand-copied EVAL_ARG_DEFAULTS config was silently wrong for any run whose eval-relevant args differ from miles defaults (e.g. LoRA evals ran the bare base weights). This drops two detached use cases the service carried:

• post-hoc backlog eval without a training job (the fn is importable if someone needs a custom driver);
• external eval for --colocate runs (they use train.py, which never exports snapshots).

Testing

• tests/fast: 30 tests (fleet pin ordering/all-must-match/retry, dead-engine recovery, EvalSkip attribution, GC ring + never-delete-outside-staging, shared-path shape regression, dispatcher overflow policies, example fn pin discipline and launch mode).
• pre-commit run --all-files clean.
• Not verified here: GPU e2e for the self-launched server (CUDA_VISIBLE_DEVICES under a Ray-hosted subprocess); needs the same 8×H200 pass as eval: checkpoint-interfaced evaluation for fully-async training (dedicated fleet / pause-the-world / external service) #1740.

Commit 84359491f (weight_target fold) is a pure mechanical move with a reproducible transform script (mechanical-refactor-verify PASS); later commits then collapsed that abstraction into the two backends.

[gemini-code-assist]

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.