Skip to content

feat(benchmark): expose logical worker endpoints - #276

Merged
ishandhanani merged 1 commit into
NVIDIA:mainfrom
cquil11:agent/expose-worker-endpoints
Jul 28, 2026
Merged

feat(benchmark): expose logical worker endpoints#276
ishandhanani merged 1 commit into
NVIDIA:mainfrom
cquil11:agent/expose-worker-endpoints

Conversation

@cquil11

@cquil11 cquil11 commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Inject mode-specific logical worker leader IPs and endpoints into custom benchmark processes.
  • Give benchmark.type: custom an ordered AIPERF_SERVER_METRICS_URLS without requiring Slurm hostlist parsing or duplicated topology metadata.
  • Preserve the existing physical-process metrics behavior for built-in AIPerf runners, including vLLM data-parallel layouts.
  • Document the environment contract and test disaggregated, aggregated, multinode, co-located, and built-in-AIPerf cases.

Motivation

srt-slurm already owns endpoint allocation, node placement, port assignment, heterogeneous-job topology, and network-interface-aware IP resolution. A custom benchmark currently cannot consume that authoritative runtime mapping. Consumers must reconstruct it from SLURM_JOB_NODELIST, recipe-specific node counts, and the system-port allocation algorithm.

That reconstruction is brittle and cannot distinguish a logical worker leader from a multinode follower rank. InferenceX PR #2291 is a concrete consumer: its custom AgentX/AIPerf replay needs SGLang worker metrics to drain requests correctly between concurrency points while running against an unmodified upstream srt-slurm release.

Environment contract

When a mode is present, custom benchmark processes receive:

Mode IPs Endpoints
Prefill SRT_PREFILL_IPS SRT_PREFILL_ENDPOINTS
Decode SRT_DECODE_IPS SRT_DECODE_ENDPOINTS
Aggregated SRT_AGG_IPS SRT_AGG_ENDPOINTS
  • IP and endpoint lists are comma-separated and preserve logical topology/index order.
  • Endpoints use IP:port format.
  • Dynamo uses each logical leader's DYN_SYSTEM_PORT; other frontends use the worker HTTP port.
  • Multinode follower ranks are excluded.
  • Co-located logical workers retain repeated IPs and distinct ports, keeping list positions aligned.
  • Custom benchmarks also receive AIPERF_SERVER_METRICS_URLS as fully qualified /metrics URLs in the same order. Configured KVBM metrics URLs remain appended after worker URLs.
  • benchmark.env is applied last, so recipes can override any generated value explicitly.

This contract is computed from backend_processes, the same cached topology used to launch workers. It does not hardcode SGLang recipe layouts or Slurm hostlist rules and therefore also covers aggregated/disaggregated layouts across upstream backends.

Compatibility

This is additive for benchmark environments and requires no recipe schema change. Built-in AIPerf runners retain their historical physical-process URL list because vLLM data-parallel layouts rely on it; only custom commands use the new logical-worker AIPerf view.

Validation

  • uv run pytest tests/test_benchmarks.py::TestCustomBenchmarkRunner tests/test_node_placement.py -q — 15 passed.
  • uv run ruff check src/srtctl/ — passed.
  • uv run ruff format --check src/srtctl/ — passed.
  • Full local suite: 883 passed, 2 skipped, 5 failed. All five failures reproduce unchanged on clean main at 38723f6 and are macOS/environment baseline failures (sched_getaffinity, CPU allocation metadata, and the existing SA-Bench profiling-script path test).
  • DCO sign-off is present on the commit.

@cquil11
cquil11 force-pushed the agent/expose-worker-endpoints branch from 36be8a9 to 3fe5508 Compare July 21, 2026 00:16
@cquil11 cquil11 changed the title feat(benchmark): expose logical worker endpoints / 向基准测试公开逻辑 Worker 端点 feat(benchmark): expose logical worker endpoints Jul 21, 2026
@cquil11
cquil11 marked this pull request as ready for review July 27, 2026 22:56
Signed-off-by: Cam Quilici <cjquilici@gmail.com>
@cquil11
cquil11 force-pushed the agent/expose-worker-endpoints branch from aedc846 to de78dfa Compare July 28, 2026 21:49
@ishandhanani
ishandhanani merged commit c180328 into NVIDIA:main Jul 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants