recipes: add vLLM DSV4-Pro agentic disagg references

[esmeetu]

Summary

Adds four DeepSeek-V4-Pro AgentX GB200 disaggregated reference recipes under recipes/vllm/deepseek-v4-pro/GB200/agentic/ and five GB300 references under recipes/vllm/deepseek-v4-pro/GB300/agentic/.

The recipes use MooncakeStore + NIXL MultiConnector, sparse DSV4 attention, model-specific block size 256, and the 062126 AgentX trace. Host paths and RDMA device names are documented placeholders and must be adapted to the target cluster.

All nine recipes now use the same attention configuration on prefill and decode:

{"backend":"FLASHINFER_MLA_SPARSE_DSV4","use_prefill_query_quantization":true,"use_fp4_indexer_cache":true}

The completed GB300 job 564 regression adds a portable 1xPDEP4 + 1xDDEP8, c256 reference pinned to Dynamo 1.3.0.dev20260720. It keeps job 564's uniform per-node P/D launch and job-scoped DeepGEMM caches while replacing cluster-local mounts and the benchmark command with documented placeholders and the bundled driver. Job 564 completed 13,311 requests with 0 errors and no cancellation at 7.23 req/s and 928.95k active total tok/s (77.41k per serving GPU); p50/p99 TTFT was 12.68s/93.67s and p50/p99 ITL was 39.72ms/46.10ms.

The two latest GB300 scaling references are derived from job 522's saved runtime configuration:

• 3xPDEP8 + 1xDDEP16, c1536 (40 serving GPUs)
• 1xPDEP8 + 1xDDEP8, c512 (16 serving GPUs)

They preserve job 522's single fastokens frontend, per-node P/D process layout, P/D deep_gemm_amxf4_mega_moe, NUMA binding, scheduler limits, and Dynamo v1.2.1. Job 522 did not set use_fp4_indexer_cache; that requested attention change and these exact two topology/concurrency points have not been benchmarked, so no performance result is claimed for them.

The new GB200 3xPDEP8 + 1xDDEP16 c1280 variant preserves the existing job-9374 topology and scheduler limits while switching both roles to srt-slurm's default per-GPU launch, explicit P/D MegaMoE, and P/D TORCH_SYMMEM=NVSHMEM plus VLLM_USE_NCCL_SYMM_MEM=1. The validated mixed-launch/all-to-all recipe remains unchanged apart from the requested attention setting. The new all-per-GPU variant is not performance-validated.

The two 515d6e9 GB300 references intentionally use only vllm-container-deps.sh; they do not inject the job-522-era Mooncake watchdog, DeepGEMM scale-factor assert workaround, or FastAPI/Instrumentator shim. The four runtime patch files were removed from this PR. The exact no-patch configs were submitted as jobs 525 (3xPDEP8 + 1xDDEP16, c1536) and 526 (1xPDEP8 + 1xDDEP8, c512) on 2026-07-17; both reached RUNNING at submission time, but no startup-completion or performance result is claimed yet.

The bundled configs/agentx_aiperf.sh driver is self-contained and only requires an existing AIPerf virtual environment at AIPERF_VENV; it no longer requires an InferenceX checkout. It provides a 600-second AgentX cache warmup default and a 1800-second warmup grace default, optional OSL and context limits, and feature-gated Dynamo conversation-routing controls.

Every recipe uses the same job-scoped DeepGEMM JIT cache path on prefill and decode: /tmp/dg-cache-inf-yasong-{job_id}.

Dependency

The recipes using backend.dp_launch_mode or per-role launch overrides depend on NVIDIA/srt-slurm#90.

The launch-mode-dependent recipes include:

• GB200 c1280 prefill: per_node, producing six four-GPU node processes across three PDEP8 workers; decode remains per_gpu
• GB200 c640 P/D MegaMoE: per_node prefill plus per_gpu decode with NVSHMEM symmetric-memory plumbing
• GB300 c960 and c1280: per_node prefill plus per_gpu decode
• GB300 c512 and c1536 derived from job 522: per_node on both P/D roles

The new GB200 c1280 MegaMoE+NVSHMEM variant intentionally omits launch-mode override fields and therefore uses the existing default per_gpu behavior without PR #90.

The selected vLLM image must include connector-prefetch support for VLLM_CONNECTOR_PREFETCH_DEPTH and VLLM_CONNECTOR_PREFETCH_KV_CAP. The GB300 job-522-derived recipes pin Dynamo to the public v1.2.1 tag commit 919682da679aa699d5bca9c872f4c1d9a530bbc0; the job 564 regression instead pins Dynamo 1.3.0.dev20260720.

Historical performance provenance

The metrics below describe the source runs/topologies. Except for the job 564 regression, recipes derived from older runs add use_fp4_indexer_cache=true, so those older metrics are not validation of the exact updated YAML unless rerun with that attention setting.

1xPDEP8 + 1xDDEP16 at c480 on 24 GB200 GPUs:

• 65.18k active total TPGS per physical GPU
• p50 TTFT 18.50s
• p50 ITL/TPOT 39.76ms

2xPDEP8 + 1xDDEP24 at c800 on 40 GB200 GPUs:

• 63.64k active total TPGS per physical GPU
• 20.03 req/s
• stable decode B mean 32.5, p99 48 with no preemption or OOM

3xPDEP8 per-node + 1xDDEP16 per-GPU at c1280 on 40 GB200 GPUs:

• 84.37k active total TPGS per physical GPU
• formal AIPerf HTTP rate 26.93 req/s
• p50 TTFT 9.31s
• p50 ITL/TPOT 65.75ms

2xPDEP8 per-node + 1xDDEP12 per-GPU P/D MegaMoE at c640 on 28 GB200 GPUs (job 9637):

• 16.10 req/s
• p50 TTFT 8.10s
• p50 ITL/TPOT 51ms
• 29,619 completed / 447 cancelled at the grace boundary / 0 errors

3xPDEP8 per-node + 1xDDEP12 per-GPU at c960 on 36 serving GB300 GPUs (job 448):

• 74.43k active total TPGS per serving GPU
• 20.89 req/s
• p50 TTFT 13.87s
• p50 ITL/TPOT 61.89ms
• 38,447 completed requests / 2 errors

2xPDEP8 per-node + 1xDDEP12 per-GPU at c1280 with three affinity-aware frontends on 28 serving GB300 GPUs (job 487):

• 95.63k active total TPGS per serving GPU
• 20.91 req/s
• p50 TTFT 35.78s
• p50 ITL/TPOT 58.97ms
• 38,475 completed / 1,313 cancelled at the grace boundary / 0 errors

Job 522 is configuration provenance for the two newest GB300 references, not their performance validation. Its 5xPDEP8 + 2xDDEP12 c2400 parent shape completed warmup with 43,243 completed / 0 cancelled / 1 error. Profiling reached 79,804 completed / 1,991 cancelled / 0 errors and hit the grace-period boundary; those numbers are not attributed to the c1536 or c512 recipes.

Configuration validation

• All nine YAML files and embedded KV-transfer, attention, and compilation JSON documents parse successfully.
• The portable job 564 recipe loads with the current schema, passes the recipe linter, keeps matching P/D DG_JIT_CACHE_DIR=/tmp/dg-cache-inf-yasong-{job_id}, sets AIPERF_MAX_OSL: "none", and omits the recipe-level AIPERF_WARMUP_GRACE_PERIOD override.
• All nine P/D attention documents exactly match the requested three-field sparse DSV4 configuration; no old two-field value remains.
• The two latest GB300 recipes were service-field-compared against job 522's saved config. Intentional differences are topology/DDEP/concurrency, public paths/install mechanics, removal of cluster-local diagnostics, the mandatory per-job DG cache path, and the requested FP4 indexer-cache field.
• With PR feat(vllm): add per-node DP launch mode #90's schema, the two job-522-derived recipes expand to the intended per-node layouts: 6 prefill + 4 decode processes/registrations, and 2 prefill + 2 decode. The current PR schema rejects their launch-mode fields as expected until feat(vllm): add per-node DP launch mode #90 lands.
• The new GB200 default-launch variant loads with the current schema and expands to 24 prefill + 16 decode one-GPU processes/registrations. Its structural delta from the job-9374 recipe is limited to launch mode, symmetric-memory environment, and decode MegaMoE selection.
• The two no-patch GB300 recipes pass YAML/embedded-JSON parsing and PR-feat(vllm): add per-node DP launch mode #90 schema dry-run, render only vllm-container-deps.sh, and have no remaining references to the removed runtime patch files.
• P/D DeepGEMM cache values are identical and contain {job_id} in all nine recipes.
• git diff --check, YAML/JSON validation, both current and PR-feat(vllm): add per-node DP launch mode #90 schema checks, and the AIPerf-driver test pass for the latest delta.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (main@f6eb42a). Learn more about missing BASE report.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #229   +/-   ##
=======================================
  Coverage        ?   67.26%           
=======================================
  Files           ?       69           
  Lines           ?     9041           
  Branches        ?        0           
=======================================
  Hits            ?     6081           
  Misses          ?     2960           
  Partials        ?        0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[esmeetu]

Added four validated GB300 MTP disaggregated references under a new recipes/vllm/deepseek-v4-pro/gb300-mtp/ directory (commit ec95807). All four use MTP speculative decoding (num_speculative_tokens=3, synthetic rejection sampling, acceptance length 2.49) on the same c188b96 image, Dynamo 1.3.0.dev20260720, MooncakeStore + NIXL MultiConnector, per-node P/D launch, and the 062126 AgentX trace. Host paths and RDMA devices remain documented placeholders.

MTP needs three additions over the non-MTP references, present in every recipe:

1. Mooncake eagle-group-split patch (configs/patches/vllm_mooncake_mtp_eagle_group_split.py via post_install_script) — on c188b96, DeepSeek V4 + MTP zeroes all MooncakeStore external reads because the eagle-flagged KV group is merged with same-spec non-eagle groups and the cross-group hit intersection collapses. Transitional only: images containing vllm-internal e5827fabc1 + 89510b05dc carry the official fixes and should drop the patch.
2. PYTHONHASHSEED=0 for cross-process prefix-block hash (Mooncake key) consistency.
3. DYN_ROUTER_ACTIVE_REQUEST_EXPIRY_SECS=3600 so long AgentX requests survive the kv-router's 300 s default active-request expiry.

Validated performance (active total tok/s per physical serving GPU, 1800 s AgentX 062126 runs):

Recipe	Source job	GPUs	TPGS	req/s	p50/p99 TTFT	p90 ITL	actual/theoretical prefix hit
1p1d-pdep4-ddep8-mtp-c128	149	12	72.05k	6.43	5.91 s / 38.9 s	18.67 ms	96.5% / 97.1%
1p1d-pdep8-ddep8-mtp-c384	133	16	110.08k	12.91	17.09 s / 65.3 s	24.70 ms	96.0% / 96.7%
2p1d-pdep8-ddep8-mtp-c512	139	24	104.32k	18.35	8.16 s / 49.4 s	30.81 ms	96.2% / 96.8%
2p1d-pdep8-ddep12-mtp-c1024	151	28	125.53k	25.70	28.75 s / 94.4 s	32.31 ms	95.7% / 96.8%

🤖 Generated with Claude Code