[Fix] Drop stale init_batch_invariance(backend) call introduced by PR #45208 rebase drift - #14
Open
terafin wants to merge 9 commits into
Open
[Fix] Drop stale init_batch_invariance(backend) call introduced by PR #45208 rebase drift#14terafin wants to merge 9 commits into
terafin wants to merge 9 commits into
Conversation
Backport of vllm-project#41896. Fixes AttributeError: 'list' object has no attribute 'zero_' when calling /wake_up on instances with PP>1 + --kv-cache-dtype fp8. The pre-fix zero-out loop in init_fp8_kv_scales iterated self.kv_caches directly and called .zero_() on each top-level entry; with pipeline parallelism the entries can themselves be nested containers (list / tuple / Mapping of tensors), which the fix's new _iter_kv_cache_tensors helper recursively descends. Regression test added at tests/v1/worker/test_gpu_model_runner_fp8_wake_up.py — direct unit tests on the _iter_kv_cache_tensors helper (flat list, nested list, Mapping, mixed, None-skipping, TypeError on unexpected leaf) and integration tests on init_fp8_kv_scales (nested kv_caches no longer AttributeErrors, flat case still zeroes, non-quantized short-circuit). Uses GPUModelRunner.__new__ + monkeypatch so no model load and no GPU are required. Signed-off-by: terafin <terafin@users.noreply.github.com>
…llm-project#44395) EngineCore.wake_up calls resume_scheduler() unconditionally after model_executor.wake_up(tags). When tags is non-None and does not include every still-asleep resource (e.g. wake_up(tags=["weights"]) restores weights but leaves the KV cache asleep), the executor is still partially asleep — but the scheduler resumes and the next forward pass writes into released GPU memory, crashing with CUDA illegal memory access. Reported in issue vllm-project#44395. The crash is reachable two ways: 1. Data parallelism: idle DP ranks' EngineCore busy-loop runs execute_dummy_batch() (a forward pass) as soon as scheduling resumes — so even without an external request, a partial wake followed by resume_scheduler crashes on the first dummy step. 2. Without DP: an external request arriving in the brief window between a partial wake and the follow-up full wake hits the resumed scheduler and the same release-then-write fault. Gate the resume on model_executor.is_sleeping (which already exists and is used by EngineCore.is_sleeping). After a partial wake the executor reports still-asleep, scheduling stays paused, and the caller's follow-up full wake (no tags) clears is_sleeping and resumes scheduling safely. Regression test added at tests/v1/engine/test_wake_up_resume_gate.py — direct mock on EngineCore.wake_up; four cases including the partial-wake bug surface. Fails on pre-fix code, passes after the gate. No model load, no GPU required. Validated on a downstream stack (4× RTX 3090 with Qwen3.6-27B AWQ TP=2 PP=2) across multiple sleep/wake cycles with no regressions. Co-authored-by: Claude Signed-off-by: terafin <terafin@users.noreply.github.com>
Adds a standalone offline diagnostic script that measures per-process GPU memory state across vLLM sleep/wake cycles. After each cycle it captures torch.cuda.memory_stats() from inside the worker (via collective_rpc) plus nvidia-smi per-PID residual, then writes a JSONL trace and prints a delta summary. Useful for verifying fixes for vllm-project#36651 and similar cumem / caching-allocator leaks. The probe disambiguates whether residual growth lives in PyTorch's caching allocator (reserved_bytes grows in lockstep with nvidia-smi) or outside it (reserved_bytes flat, but nvidia-smi grows -- pointing at cumem handles or other CUDA subsystem state). Used to add a third repro reference on vllm-project#36651 comparing a clean lab harness (Qwen3-Embedding-0.6B, util=0.08, 20 cycles, all metrics flat) against a production rerank deployment that showed ~412 MiB -> ~5184 MiB residual growth over a day of cycling. Posting upstream so others investigating sleep-mode allocator behaviour have the same tool. Test plan: * `python examples/features/pause_resume/cumem_probe_offline.py --help` prints expected argparse output without vllm installed (lazy import). * `VLLM_ALLOW_INSECURE_SERIALIZATION=1 python examples/features/pause_resume/cumem_probe_offline.py --model Qwen/Qwen3-Embedding-0.6B --cycles 20 --gpu-mem-util 0.4 --kv-cache-bytes 1073741824 --runner pooling --inference-between-cycles` runs cleanly on RTX 3090 in <5 min and emits 41 JSONL records. AI-assistance disclosure (per AGENTS.md): the script and this commit message were drafted with the assistance of Claude (Anthropic). All methodology, the empirical results referenced above, and this commit were reviewed by the human submitter before posting. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: terafin <terafin@users.noreply.github.com>
…veness The standard /health endpoint only asks "is the engine task alive?". It does not probe whether the engine is actually decoding. When the engine deadlocks at the GPU layer — for example the NCCL P2P deadlock in TP>1 / PP>1 that survives a container restart, tracked in vllm-project#45094 — the FastAPI task stays alive, /health continues returning 200, and every in-flight request hangs forever. Orchestrators (k8s readiness, docker healthcheck, custom watchdogs) have no signal to act on. This adds GET /health/decode, which returns: * 200 {"status": "ok", ...} — running > 0 AND token recent * 200 {"status": "idle", ...} — running == 0, OR engine has never decoded (cold start, or long-prefill on first request) * 503 {"status": "stalled", ...} — running > 0 AND last decoded token is older than the stall threshold Stall threshold defaults to 60s and is overridable via the new env var VLLM_DECODE_LIVENESS_STALL_SECONDS. The endpoint always includes "running", "last_token_age_seconds", and "stall_threshold_seconds" so operators can dashboard the raw values. Implementation taps the existing per-step record() flow on StatLoggerManager (which already sees SchedulerStats.num_running_reqs and IterationStats.num_generation_tokens on every output_handler iteration) — no new wiring through the engine core, no new synchronization. The accessor is added to the EngineClient protocol with a safe default of (0, None) so non-v1 implementations remain healthy "idle" without override. Backwards compatible: /health semantics are unchanged; /health/decode is a new orthogonal route. Auth-bypass is automatic (the path doesn't match GUARDED_PREFIX). Prometheus instrumentor is excluded from the new route to match /health's behavior. Refs: vllm-project#45094 🤖 Generated with assistance from Claude Co-Authored-By: Claude <noreply@anthropic.com>
vLLM unconditionally pops NCCL_ASYNC_ERROR_HANDLING in ``Worker.init_device`` (gpu_worker.py:253) because the env (originally set by Ray) historically caused exceptions during CUDA graph capture. The side effect is that torch's NCCL watchdog never auto-aborts hung collectives — PP/TP P2P send/recv loops block the CUDA stream forever instead of surfacing a timeout. This is one of the reasons issue vllm-project#45094 (TP=2 PP=2 P2P decode deadlock) is hard to detect: there is no operator-controllable knob to re-enable the watchdog without patching vLLM. This change adds an opt-in env, ``VLLM_NCCL_ENABLE_ASYNC_ERROR_HANDLING`` (default ``False`` = current behavior, no change for existing users): - When unset/false, vLLM still pops both ``NCCL_ASYNC_ERROR_HANDLING`` and ``TORCH_NCCL_ASYNC_ERROR_HANDLING`` (the post-rename name) so CUDA graph capture is undisturbed. - When set to ``1``/``true``, vLLM preserves the envs and emits a startup warning documenting the tradeoff so the operator's choice shows up in logs. The pop logic is extracted into a small module-level helper (``_apply_nccl_async_error_handling_policy``) so the policy can be unit-tested without standing up a full Worker / GPU / distributed init. The new test file covers all four cells (default vs opt-in × warning vs no-warning) plus truthy/falsey env parsing and the no-NCCL-envs-set idempotency case. End-to-end NCCL watchdog behavior is necessarily out of unit-test scope (multi-GPU integration territory); the operator opting into this env is expected to verify behavior on their hardware. Followups (intentionally out of scope here to keep the PR small): - ``parallel_state.GroupCoordinator.recv_tensor_dict`` uses ``torch.distributed.isend/irecv`` and waits indefinitely; wrapping with a deadline that surfaces "P2P timeout on rank K" in logs would give a second detection path even when the watchdog isn't active. That plumbs through several subsystems and warrants its own PR. This work was AI-assisted (Claude). Manual integration testing on multi-GPU hardware is pending and will be reported in the PR thread. Refs: vllm-project#45094 Refs: vllm-project#45097 (decode-liveness HTTP endpoint, orthogonal detection) Signed-off-by: terafin <terafin@users.noreply.github.com> Co-authored-by: Claude
In-the-trenches operational notes from running 9 co-tenant vLLM containers (1 pinned large generate-runner + a 3-member swap-group of heavy generate-runners + small always-warm sidecars including pooling runners) on a 4-GPU host with continuous wake/sleep cycling. Covers: sleep level tradeoffs observed across runner types, the cross-cycle residual growth pattern (vllm-project#36651), deadlock detection via /health/decode (vllm-project#45094, vllm-project#45097, vllm-project#45105), co-tenancy planning, PP rank asymmetry, and a short wishlist of orchestrator-friendly primitives. The new page is framed throughout as one deployment's experience, not universal guidance — readers are pointed at calibrating per-model residuals in their own environments. A small cross-link is added from the existing sleep_mode.md to make the new page discoverable; no other content in sleep_mode.md is changed. AI assistance was used (Claude) to draft this page per AGENTS.md. The operational observations, calibrated numbers, and referenced issues are from a human-supervised production deployment; every concrete claim traces back to either the public vllm-jukebox repo's calibrated config or a referenced upstream issue/PR. Signed-off-by: terafin <terafin@users.noreply.github.com>
When the cumem allocator's cleanup bypasses PyTorch's allocator tracking via direct cuMemUnmap, memory_reserved() becomes inflated, making non_torch_memory negative and underestimating non-KV cache memory usage. This causes OOM on large models (e.g. gpt-oss-120b on GH200 144GB). Replace the memory_reserved()-based formula with a mem_get_info()-based measurement (total_consumed) that is always accurate regardless of which allocator is used. Use transient_peak_headroom (torch_peak - torch_allocated) instead of torch_peak_increase to avoid double-counting persistent torch allocations already included in total_consumed. Fixes vllm-project#37096 Signed-off-by: haosdent <haosdent@gmail.com>
…llm-project#45208 rebase drift PR vllm-project#45208 (haosdent's cumem OOM fix) was rebased against a tree where init_batch_invariance() had been changed to accept an attention backend argument. When cherry-picked onto a stock upstream/main (the function's signature remained 0-arg), the call site `init_batch_invariance(attention_config.backend)` crashes worker init with: TypeError: init_batch_invariance() takes 0 positional arguments but 1 was given This restores the call to the upstream 0-arg form. Also drops the now- unused `attention_config = vllm_config.attention_config` line introduced by the same commit. Carrier-PR for the intarweb fork's FORK_CARRIED_COMMITS cherry-pick stack, which carries c9427de (cumem) + 88af8e6 (/health/decode). Without this fix, the canonical image fails to cold-load with TP/PP > 1. Tested: rolled back fleet on Jun 12 02:00 UTC after re-test confirmed deterministic crash; this carry-fix applied on top of cumem produces a healthy boot. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
intarweb-sync-bot
Bot
force-pushed
the
intarweb-dev
branch
21 times, most recently
from
June 13, 2026 04:58
aee39d2 to
7acf243
Compare
intarweb-sync-bot
Bot
force-pushed
the
intarweb-dev
branch
29 times, most recently
from
June 19, 2026 23:09
f02f651 to
1f3d93f
Compare
intarweb-sync-bot
Bot
force-pushed
the
intarweb-dev
branch
from
June 27, 2026 03:00
1f3d93f to
855f142
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Carry-fix for the FORK_CARRIED_COMMITS cherry-pick stack. PR vllm-project#45208 (haosdent's cumem OOM fix) was rebased against a tree where
init_batch_invariance()had been changed to accept an attention backend argument. When cherry-picked onto a stock upstream/main where the function's signature remained 0-arg, the call site crashes worker init:Repro
sha256:c28b3f2d9d6c...(head_shaf1bdf3e33)r2-e7f40cfFix
One-line: drop the unused
attention_config = vllm_config.attention_configand revert the call back to upstream 0-arg forminit_batch_invariance().Carrier-PR
This is a fork-local carrier PR for the FORK_CARRIED_COMMITS mechanism. Source-of-truth tracking: not an upstream PR. Upstream's
init_batch_invariance()signature stays 0-arg; only the cumem cherry-pick needed the call-site reverted.🤖 Generated with Claude Code