[Feature] Dynamo KV router as an alternative rollout backend

[AndyDai-nv]

Overview

Integrate NVIDIA Dynamo as an alternative rollout backend behind --rollout-backend dynamo, drop-in replacement for the existing miles-router + sglang path. SGLang remains the underlying inference engine — only routing and dispatch change. Default backend (sglang) is untouched.

Dynamo brings:

• KV-aware routing — block-hash prefix tree over NATS-published KV events; matches or beats sgl-router's cache_aware in multi-tenant / multi-prefix workloads, and preserves it for RL where n_samples_per_prompt sticky routing already handles prefix reuse.
• OpenAI-compatible frontend — /v1/completions etc., serves Miles rollouts and external clients through the same HTTP surface.
• etcd-based worker discovery — no static router config; workers self-register.
• Predict-on-route — sibling-burst routing that reduces tail latency when the same prompt fans into many samples (very common in RL).
• Path to PD-disaggregation — prefill/decode split with mooncake/nixl KV transfer.

Miles-side integration is packaged as a series of small, independently reviewable PRs so the abstraction can be discussed before implementation, and each capability can land / roll back independently. No patches applied to the Dynamo repo — the integration sits entirely on top of upstream ai-dynamo/dynamo.

Architecture

                 ┌─────────────────────────────────────────┐
                 │  Miles RolloutManager                    │
                 │  (Ray driver + train actors)             │
                 └───────────────────┬─────────────────────┘
                                     │
                       ── --rollout-backend ──
                       │                     │
              "sglang" │                     │ "dynamo"
       ┌───────────────▼──┐        ┌─────────▼───────────┐
       │ miles-router or  │        │ dynamo.frontend      │
       │ sgl-router       │        │ (KV router + OpenAI  │
       │                  │        │  compat + etcd disc.)│
       └────┬─────────────┘        └──────┬───────────────┘
            │                             │
      SGLang HTTP native            OpenAI /v1/completions
            │                             │
       ┌────▼─────┐                  ┌───▼──────────────┐
       │ sglang   │                  │ dynamo.sglang    │
       │ worker   │                  │ worker (SGLang + │
       │          │                  │  discovery + KV  │
       │          │                  │  events + RL     │
       │          │                  │  control plane)  │
       └──────────┘                  └──────────────────┘

Miles' Ray-actor / (ip, port) / weight-broadcast contracts are preserved end-to-end. DynamoEngine mirrors SGLangEngine's public surface so ServerGroup swaps ray.remote(SGLangEngine) for ray.remote(DynamoEngine) with no other changes downstream.

Non-goals (out of scope for this series)

• Modifying the default sglang backend or miles-router implementation.
• Any patch to the Dynamo repo (ai-dynamo/dynamo).
• Fault-tolerance improvements orthogonal to the router swap.
• Comparative benchmarking beyond what's needed to prove parity.

Wall-clock numbers (single-tenant RL, Qwen2.5-0.5B + GSM8K, 8-GPU non-colocate)

Validated on feat/dynamo-kv-router (the pre-split branch); numbers reproduced with the same code that lands across this series.

router	mean wall-clock (n=5)	range	Δ vs miles-router
miles-router (baseline)	6470 s	6361 – 6627 s	—
Dynamo KV router + --stream-interval=50	5877 s	5848 – 5906 s	−9.2%
sgl-router cache_aware	5893 s	5828 – 5977 s	−8.9%

Dynamo KV router and sgl-router both are ~9% faster than the miles-router baseline.

Implementation Plan

• Review Pending: --rollout-backend flag + dynamo_utils scaffold: feat(rollout): --rollout-backend flag + dynamo_utils scaffold (1/N) #1646
• Review Pending: dynamo.frontend launcher (round-robin): feat(rollout): dynamo.frontend launcher (round-robin) (2/N) #1647
• To open: DynamoEngine subprocess lifecycle (spawn + health + shutdown)
• To open: OpenAI-style generation dispatch (_generate_via_dynamo in sglang_rollout.py)
• To open: Weight-update methods (init_weights_update_group, update_weights_from_*, begin/end_weight_update, destroy_weights_update_group)
• To open: KV-aware routing (--dynamo-router-mode kv, etcd + NATS sidecars, KV events, predict-on-route, --dynamo-page-size)
• To open: DYN_SGLANG_STREAM_INTERVAL env passthrough (throttles per-token chunk emission for non-streaming RL clients; ~13% wall-clock win)
• To open: Example k8s Job yaml

[guapisolo]

The OpenAI-style generation dispatch should impl for
https://github.com/radixark/miles/blob/main/miles/rollout/generate_hub/agentic_tool_call.py

Miles  agentic_tool_call.py start connection with external harness 
(e.g. https://github.com/radixark/miles/tree/main/examples/experimental/swe-agent-v2)

/chat/completion request flow:
Current: harness -->  session server --> sglang-router / miles-router
Target: harness -->  session server --> Dynamo

[guapisolo]

DynamoEngine subprocess lifecycle (spawn + health + shutdown)

No need to simulate SGLangEngine impl. Dynamo already has its own launch logic, so what we need, is add a seperate dynamo launch script.

After weight update support for external rollout, dynamo should be very easy to integrate to miles.

[guapisolo]

To open: KV-aware routing (--dynamo-router-mode kv, etcd + NATS sidecars, KV events, predict-on-route, --dynamo-page-size)

It's only arg parsing. Almost, no modifications to miles.