feat(vllm): support multi-node aggregate jobs with the direct vllm frontend - #1
Open
functionstackx wants to merge 2 commits into
Open
feat(vllm): support multi-node aggregate jobs with the direct vllm frontend#1functionstackx wants to merge 2 commits into
functionstackx wants to merge 2 commits into
Conversation
…ontend Extend frontend.type: vllm beyond single-node aggregate jobs. vLLM natively supports multi-node serve for TP/PP topologies (torchrun-style --nnodes/--node-rank with --master-addr; non-leader ranks run headless engine workers), so emit those flags from build_worker_command and drop the single-node validation. The leader keeps ownership of the OpenAI port; rank > 0 processes get --headless and no host/port args. Validated on a 2-node TP8xPP2 Kimi-K3 deployment on B200 (InferenceX agentic bring-up): both ranks boot, the model loads, and the leader serves /health and /v1/chat/completions. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add TestDirectVllmMultiNode: schema accepts agg_nodes > 1 with frontend.type: vllm while still rejecting disaggregated layouts; the leader command keeps --host/--port and gains --master-addr/--nnodes/--node-rank 0; ranks > 0 run --headless without binding the API port; single-node commands keep their original shape with no multi-node flags. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Extends
frontend.type: vllm(from NVIDIA#278,kylliang/direct-aggregate-vllm) beyond single-node aggregate jobs:src/srtctl/backends/vllm.py: in the direct-serve branch ofbuild_worker_command, emit vLLM's native multi-node serve flags instead of raising — leader gets--host/--portplus--master-addr <leader_ip> --nnodes N --node-rank 0; ranks > 0 get--node-rank R --headless(headless engine workers, no API server, no host/port args).src/srtctl/core/schema.py: drop theagg_nodes != 1load-time rejection for the vllm frontend.vLLM supports this topology natively (torchrun-style multi-node PP/TP serve: the leader owns the OpenAI server, non-leader nodes run headless engine workers), so no orchestration beyond the existing per-node srun launch is needed. The single-frontend/aggregate-only/nginx validations are unchanged.
Motivation
Models too large for one node (e.g. Kimi-K3, ~1.4TB MXFP4 weights, TP8×PP2 across 2×8 B200) cannot use the direct-serve path with the single-node restriction, but don't need Dynamo either.
Testing
TestDirectVllmMultiNode(5 new tests intests/test_configs.py, following the existing direct-vllm test style):agg_nodes: 2withfrontend.type: vllm; disaggregated layouts are still rejected--host/--portand gains--master-addr/--nnodes/--node-rank 0(no--headless, nodynamo.vllm, no--request-plane)--headlesswith--node-rank 1and does not bind--host/--portRun locally:
uv run pytest -q tests/test_configs.py -k DirectVllmMultiNode→ 5 passeduv run pytest -q→ 893 passed, 5 failed — all 5 failures are pre-existing environment issues (bash-script and/proc/Slurm-dependent probes on macOS) and fail identically on the base branchuv run ruff checkon the changed files → no new findings (remaining findings are pre-existing on base)Validation
Deployed on a 2-node TP8×PP2 Kimi-K3 aggregate job on B200 (SemiAnalysisAI/InferenceX agentic bring-up, PR 2359): srtctl generated the expected leader/headless commands, both ranks booted, the model loaded, and the leader served
/healthand/v1/chat/completionsfor the AIPerf trace replay.🤖 Generated with Claude Code