Skip to content

feat(vllm): add per-node DP launch mode - #90

Merged
alec-flowers merged 1 commit into
NVIDIA:mainfrom
esmeetu:yasong/vllm-hybrid-dp-lb
Jul 18, 2026
Merged

feat(vllm): add per-node DP launch mode#90
alec-flowers merged 1 commit into
NVIDIA:mainfrom
esmeetu:yasong/vllm-hybrid-dp-lb

Conversation

@esmeetu

@esmeetu esmeetu commented Apr 27, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Add opt-in backend.dp_launch_mode: per_node for vLLM data-parallel endpoints.
  • Launch one process per physical node with the full local GPU set while preserving the existing per_gpu default.
  • Derive --data-parallel-size-local, --data-parallel-start-rank, --data-parallel-address, and DP-RPC settings from the allocated topology.
  • Reserve the per-local-rank KV-event port range and validate that configured global DP size matches allocated GPUs.

Per-role mixed launch overrides were split into the separate stacked follow-up #271.

Motivation

The existing per-GPU launcher restricts every process to one GPU. DeepGEMM MegaMoE and CUDA symmetric-memory paths can require node-local DP ranks to share one process and CUDA namespace. The opt-in per-node layout provides that namespace without changing existing recipes.

backend:
  type: vllm
  dp_launch_mode: per_node
  vllm_config:
    prefill:
      data-parallel-size: 8
      data-parallel-hybrid-lb: true
    decode:
      data-parallel-size: 16
      data-parallel-hybrid-lb: true

Validation

  • make check
  • Ruff and formatting pass.
  • Full test suite: 865 passed, 2 skipped, 6 deselected.
  • DeepSeek V4 Pro AgentX job 9700 validated uniform per-node launch for 3xPDEP8 + 1xDDEP16 with decode AMXF4 MegaMoE: 26.85 req/s, 91,193 TPGS on 40 GPUs, p50 TTFT 8.24s, and p50 TPOT 58.73ms.

@esmeetu
esmeetu marked this pull request as draft April 27, 2026 06:40
@esmeetu
esmeetu force-pushed the yasong/vllm-hybrid-dp-lb branch from f9e60ff to b21248d Compare July 11, 2026 01:42
@esmeetu esmeetu changed the title feat(vllm): switch DP+EP launch to hybrid_lb (per-node process) feat(vllm): add optional per-node DP launch mode Jul 11, 2026
@esmeetu
esmeetu marked this pull request as ready for review July 11, 2026 05:43
@esmeetu
esmeetu force-pushed the yasong/vllm-hybrid-dp-lb branch from b21248d to 975e1e6 Compare July 13, 2026 13:36
@esmeetu esmeetu changed the title feat(vllm): add optional per-node DP launch mode feat(vllm): add per-node and mixed DP launch modes Jul 13, 2026
@alec-flowers

Copy link
Copy Markdown
Collaborator

Is there a reason to mix the launch modes? Or is 1 way better than the other? If we have a reason to choose one way then I would prefer to be opinionated.

@esmeetu
esmeetu force-pushed the yasong/vllm-hybrid-dp-lb branch from 975e1e6 to 1a0f9e3 Compare July 18, 2026 02:27
@esmeetu esmeetu changed the title feat(vllm): add per-node and mixed DP launch modes feat(vllm): add per-node DP launch mode Jul 18, 2026
@esmeetu

esmeetu commented Jul 18, 2026

Copy link
Copy Markdown
Contributor Author

Split as suggested: #90 now contains only the opt-in global per_node launch mode. The per-role mixed-DP overrides moved to the separate stacked draft #271. per_node is useful when node-local ranks need a shared CUDA namespace (for example MegaMoE/symmetric-memory paths); this does not assert a universal performance advantage, and the existing per_gpu behavior remains the default.

@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 90.19608% with 5 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (main@c1b6b5c). Learn more about missing BASE report.

Files with missing lines Patch % Lines
src/srtctl/backends/vllm.py 90.69% 4 Missing ⚠️
src/srtctl/core/topology.py 87.50% 1 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##             main      #90   +/-   ##
=======================================
  Coverage        ?   67.34%           
=======================================
  Files           ?       69           
  Lines           ?     9091           
  Branches        ?        0           
=======================================
  Hits            ?     6122           
  Misses          ?     2969           
  Partials        ?        0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@alec-flowers
alec-flowers merged commit b93276c into NVIDIA:main Jul 18, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants