feat: controller-rollout robustness + explicit StopCommand shutdown + UCXX teardown

[shuangwu]

Summary

Unifies multi-rank rollout shutdown and hardens teardown across the controller,
policy, and rollout paths. The root cause was a design coupling, not a missing
guard: the controller forced a final P2R→R2R weight sync purely as an "ending
signal", held open by a hardcoded time.sleep(30) in the policy main loop —
which raced rollouts that had already drained and NCCL-aborted, wedging
ncclCommAbort on the P2R recipient. This branch also adds the Slurm full-CI
tooling and reworks the UCXX payload transport for correctness.

What changed

Controller / rollout shutdown (fix: harden controller-rollout shutdown in CI)

• Unified stop on the is_end path: _get_batched_prompt_impl forces
  is_end=True once training_finished() (non-validation), routing step-bounded
  runs through the same NCCL-free path as data-bounded runs. Starvation-safe:
  every needed rollout is already buffered by training-finish.
• Suppress the never-consumed final-step weight sync (non-validation) via a
  pure need_weight_sync helper; validation keeps it (final validation is driven
  by that R2R broadcast).
• Bounded policy wait replaces time.sleep(30) (gated on validation.enable).
• Explicit NCCL-free StopCommand broadcast once every policy replica
  unregisters — reaches every rollout via consume_command→handle_stop
  regardless of fetch state, unblocking a rank wedged on the version_fail gate;
  broadcast_object_cpu gives race-free lockstep exit for world_size>1.
• async_r2r_sync barrier fixes: authoritative per-round participant count
  (shrinks as replicas finish, killing the 120s barrier timeout) + abortable
  barrier wait (checks _stop each iteration).
• Phased shutdown: explicit JobPhase (RUNNING → DRAINING) + zero-work
  TrainingCompleteCommand ack so completion is separated from teardown.
• Strict final-validation guard: _maybe_finalize logs an error if the
  controller finalizes while validation is still active (covered by
  test_finalize_logs_error_if_validation_is_still_active).
• Teardown defense-in-depth: shared bounded_drain_or_abort (WST + inference
  streams); rollout-aware controller checkout gate; UCXX teardown
  drain→stop→reset ordering — _drain_inflight_requests cancels in-flight UCP
  requests with the progress thread alive so callbacks fire under a healthy
  interpreter, fixing a GC-time use-after-free SIGSEGV.
• Log hygiene: removed temporary Trace A–F dispatch/main-loop instrumentation
  and demoted high-volume counters to DEBUG.

Test plan

• tests/test_multirank_shutdown.py — 58 tests (StopCommand round-trip/handler,
  weight-sync status.ended exclusion, nccl_abort_all, bounded_drain_or_abort,
  rollout-checkout gate, R2R barrier world-size, _maybe_finalize validation guard)
• tests/test_ucxx_transport.py::TestResetUCXXContext (drain→stop→reset order,
  cancel-API fallback)
• tests/test_process_flow.py, test_custom_class.py, test_colocated.py,
  test_teacher_model.py (import-layout fix verified)
• Full 8-GPU Slurm CI suite green except known test_deepep.py CUDA
  (layout.cu:138 'named symbol not found') infra failure

[lfengad]

overall LGTM. Seems also among PRs some contents are shared and may conflict? We might merge them carefully with that order?

[shuangwu]

#698 --> #695 --> #696 (this one) --> #697 --> #694

[shuangwu]

After #695 is merged, I will rebase this one.