feat(ENV-AGNOSTIC-W1-TOOLING): resolve or refuse in scripts/ and tools/, and keep the 9 recorded hosts literal

.agents/issue-index.md

@@ -420,3 +420,4 @@ rather than merged. `scripts/check-agent-record.py` gates both.
 | [#1303](https://github.com/mudler/vllm.cpp/issues/1303) | `ENG-CUDAGRAPH-DEDUP` | **The decode graphs are TWO topologies, not one: 376 and 404 nodes, mixed across the shipped padded bucket set.** Found by the `ENG-CUDAGRAPH-DEDUP` W6 device-byte run (`origin/main` `2c8f53d93`, GB10, driver `580.173.02`, nvcc 13.0.88, `rc` job `93f783de`, 2026-08-19) with a `cudaGraphGetNodes` shim on every `cudaGraphInstantiate`: `w32_off_a` captured `404 404 376 376 404 404 404` and `w64_coarse_a` saw `376 376 376 376 376 376 404 404 376 …` over 11 buckets. **The measured consequence:** with the coarse key the driver refuses **43% of probes at 7 buckets and 73% at 11**, and EVERY refusal in the run gave one reason — `probe refused a fold (err=910 result=2)`, `cudaErrorGraphExecUpdateFailure` / `cudaGraphExecUpdateErrorTopologyChanged`. Not one was about a parameter. **This INVERTS the premise of [#1226](https://github.com/mudler/vllm.cpp/issues/1226) (now closed):** `cudaGraphExecUpdate` permits parameter changes and requires topology to match, so dropping the launch dimensions from the key was expected to fold more; it produces more FALSE CANDIDATES instead, because the key cannot see the difference the driver rejects on. W5's `refused=0` is explained as an artefact of workloads whose buckets only ever SHRANK, so exactly one pair was ever presented. **Why it matters beyond the key:** at 11 buckets the coarse arm ran 27 instantiate calls against OFF's 11 (2.45x) plus 28 update calls, to retain 6 fewer executables for no measurable byte saving; if the split has a removable cause, every refusal disappears and the fold ratio becomes the whole bucket set rather than 3/7 and 5/11, which is the only route by which the default flip could become supportable on this model class. First suspect, NOT verified: the FA-2 split-KV grid. Owed: attribute the split to a node or subgraph by DUMPING and diffing the two node sets (`cudaGraphNodeGetType`, `cudaGraphDebugDotPrint`) rather than reading source; say whether a capture that fixes the node set exists and what it costs; re-run the W6 A/B if it does. Evidence `/mnt/nas_share/rc/dedup-bytes/` (`RESULT.md`, `logs-bytes/bytes.log` `### PER-INSTANTIATE BYTE DISTRIBUTION`, `mem/mem_<tag>.csv` column 8); recorded in [`benchmark-record.md`](benchmark-record.md) entry `ENG-CUDAGRAPH-DEDUP W6` and [`eng-cudagraph-dedup.md`](specs/eng-cudagraph-dedup.md) `## Outcome` (W6) and `## Owed` | perf |
 | [#1291](https://github.com/mudler/vllm.cpp/issues/1291) | `ENG-CUDAGRAPH-BREAK` | W3 of the break-point capture seam: the three remaining PLAIN BATCHED decode drivers migrate onto `vt::GraphCaptureScope` + `vt::BreakableGraph` — `Qwen3MoeDecodeGraph` (`qwen3_moe.cpp`), `VoxtralDecodeGraph` (`voxtral.cpp`) and `DeepseekV2DecodeGraph` (`deepseek_v2.cpp`) — one commit each, each landable alone, each with its own RED-first G2 gate. Four of the nine drivers are now on the seam and the six batched-driver `VLLM_CPP_CUDAGRAPH` reads the spec's `## Our baseline` item 1 counted are down to TWO, both in `qwen3_5.cpp` (W4). The two per-model rollback switches STAY (`VT_QWEN3MOE_CUDAGRAPH`, `VT_DEEPSEEK_CUDAGRAPH`): each is an A/B lever for exactly one driver, not a copy of the shared one. **Each driver owes its own gate because nothing else can see the difference** — a driver that kept its hand-rolled `BeginCapture`/`EndCaptureGraph` pair produces IDENTICAL logits, an IDENTICAL backend log and an identical `replay_count()`, so `segments_captured` and `replays` are the only observables that separate "captured a graph" from "captured a graph THROUGH THE SEAM". Red-first on four assertions each (`test_qwen3_moe_decode_graph_seam` 222/226, `test_voxtral_decode_graph_seam` 224/228, `test_deepseek_v2_decode_graph_seam` 224/228, all exit 1), 3/3 green after; the G2 mutation restoring each pre-W3 driver file (25/102, 23/92 and 25/94 lines, each compiled clean, each restored and verified by sha256) reddens ONLY its own gate and leaves `test_breakable_graph` 216/216 and W2's `test_qwen3_decode_graph_seam` 231/231 GREEN. The two swapped seams every such gate needs now live ONCE, in `tests/vllm/models/decode_graph_seam_harness.h`, with W2's file including it: three more copies inside `tests/` would have reproduced the duplication this row removes from `src/`, and two copies of a harness diverge invisibly because both files stay green while measuring different things. **G1 IS DELIVERED, retiring the item W1 and W2 both carried as owed.** `tests/vllm/models/test_decode_graph_seam_g1_cuda.cpp` drives each migrated driver COLD, CAPTURE and THREE consecutive replays and compares every step bit for bit against the driver's own EAGER arm, which is selected by `max_num_reqs == 0` (`PadToCaptureSize` returns -1 and `Step` falls out to its plain forward) so both arms are ONE binary on ONE device rather than two builds, each with its OWN device KV cache so neither can read the other's writes. Measured on `thor:gpu0` through an `rc` lease — NVIDIA Thor sm_110, driver 595.78, nvcc 13.0.88, source `c905bb536`, CUDA-ON build for arch 110, 32 `.cu.o` objects, binary resolving `libcudart.so.13` and `libcublasLt.so.13`, build 489 s at `-j4`: **3 cases, 1600 assertions, exit 0**, `5 steps x 100 logits, 0 differing, 4 replays` for each driver. The ASSERTION COUNT carries that claim and the status line does not: with no CUDA backend every case skips and the same file prints `Status: SUCCESS!` over `assertions: 0`, which the file's own header names as a skip wearing a pass. Bounded honestly: the models are the synthetic tiny ones the CPU forward gates use rather than a checkpoint, and W2's `Qwen3DenseDecodeGraph` is covered by shared-seam ARGUMENT and not by one of the three cases. **W3 ALSO FOUND A GATE THAT COULD NOT FAIL, and closing it is part of the stage.** The three gates assert `breaks_registered == 0` to hold the capture to vLLM's decode arm (`FULL_AND_PIECEWISE`, `vllm/config/compilation.py:63,65-66,630-632` @ `5559679229`). That is a TAUTOLOGY for these models: the counter moves only when a `vt::GraphBreak` registers into a splitting scope, and the ONE production break point in the tree is W1's, in `qwen3.cpp` — W2's driver runs through it, none of W3's three does. Measured rather than reasoned: flipping `kFull` to `kPiecewise` in `qwen3_moe.cpp`, one token, compiled clean and left that driver's whole gate GREEN at 226/226, and the same flip was equally invisible in the other two. The mode was UNOBSERVABLE from outside a driver — the scope is a `Step` local, the container is private to the driver's `Impl`, and a token gate cannot see a segment count — so `vt::GraphBreakStats` gains `full_scopes` and `piecewise_scopes`, counted in `GraphCaptureScope`'s constructor on the ACTIVE path only (an inert scope makes no backend call in either mode, so counting it would report a mode that never reached a backend), cleared by `ResetGraphBreakStats()`, gated in `tests/vt/test_breakable_graph.cpp` with the inert arm as the control that stops them degenerating into "scopes constructed". Re-measured after the fix, the same one-token flip now REDS each of the three gates on exactly those two assertions (226/228, 228/230, 228/230, all exit 1). **NO break point is registered in these three models, and that is a decision rather than an omission**: under `kFull` a break point takes the same pass-through arm it takes outside a scope, so registering one would land machinery no gate can exercise; establishing each model's break-point set is what the PIECEWISE arm needs, that arm is blocked on replay-safe closure inputs (W4), and W6 is where the eligibility predicate moves. **The async device-token decline, stated per driver rather than left as an omission.** `VoxtralDecodeGraph` needs none: its only construction site is `VoxtralGenerateGreedy`, the single-sequence multimodal greedy loop, which is not reached from `runner.cpp` and has no asynchronous device mirror. The other two are a NEW FINDING, filed [#1305](https://github.com/mudler/vllm.cpp/issues/1305) with W4 as owner and NOT fixed in flow, because adding a decline trades a shipped default-ON capability away on a measurement W3 cannot make and the fix `qwen3.cpp:1106`'s own comment names is `StepDevInputs` as a seam capability. **G5's ROCm/Tenstorrent arm is NOT discharged**, and the owner moves from W3 to W5 with the reason stated: the fleet carries no ROCm and no Tenstorrent device, so it is BLOCKED on hardware rather than unattempted. What W3 can say instead is that the seam's CUDA arm now runs on TWO architectures, sm_110 here and sm_121a for W1's exit criterion. Still NO throughput claim anywhere: the prefill refutation stands (3.8% host idle, >96% GPU-busy, 92.5% glue) and `kFull` keeps each migrated step's shape the one it already had | feature |
 | [#1305](https://github.com/mudler/vllm.cpp/issues/1305) | `ENG-CUDAGRAPH-BREAK` | Three decode-graph registrations route an ASYNCHRONOUS step into a host-vector replay with NO `device_token_ids` decline. Found while migrating two of them onto the break-point capture seam ([#1291](https://github.com/mudler/vllm.cpp/issues/1291)) and NOT caused by that migration — the shape is present at its base commit `5d9fe332c`. `src/vllm/model_executor/models/qwen3.cpp:1106` returns `std::nullopt` whenever `ModelForwardInput::device_token_ids` is non-null, on a measured four-concurrent battery (`depth-1 graph ON PASS 78/78`, `depth-2 graph OFF PASS 82/82`, `depth-2 graph ON FAIL, slots 1-3 degenerate`, #323/#1179), because `Step()` replays against persistent HOST `token_ids` while the asynchronous combine has patched the DEVICE ids and deliberately left the host vector stale for decode rows (`src/vllm/v1/worker/gpu/runner.cpp:1265-1309,1523`). That comment calls the hazard "latent for EVERY classic-dense model, since the graph is default-ON". `qwen3_moe_registry.cpp:107`, `deepseek_v2_registry.cpp:106` and `glm4_moe_lite_registry.cpp:125` (the last constructs the DeepSeek driver) admit a pure-decode step to a driver with the same host-vector replay and carry NO such check; `grep -c StepDevInputs` returns 0 for `qwen3_moe.cpp`, `deepseek_v2.cpp` and `voxtral.cpp` against 41 for `qwen3_5.cpp`, which is the divergence the row's spec already records as the cost of nine drivers. `VoxtralDecodeGraph` is NOT affected: its only construction site is `VoxtralGenerateGreedy`, which the runner does not reach and which has no device mirror. NOT fixed in flow, for two reasons stated rather than assumed: adding a decline is a behaviour change that trades a shipped, default-ON capability away, and W3 has no four-concurrent battery for these two models to justify it; and the correct fix is already assigned — `StepDevInputs` (`qwen3_5.cpp:3894`) as a SEAM capability, which is `## Work breakdown` W4 of [eng-cudagraph-break.md](specs/eng-cudagraph-break.md) and the only version that reaches these registrations at all. Owner: **W4**, with [#1179](https://github.com/mudler/vllm.cpp/issues/1179) as the standing tracker. What would settle it: run that same battery against Qwen3-Coder and DeepSeek-V2-Lite with `VT_ASYNC_DEVICE_MIRROR` live and the decode graph ON — either it reproduces the depth-2 degeneration, or the reason it does not has to be found and written down rather than assumed | bug |
+| [#1308](https://github.com/mudler/vllm.cpp/issues/1308) | — | ENV-AGNOSTIC: the campaign's derivation query `git grep -cIE 'dgx\.casa\|nas_share\|192\.168\.\|thor:gpu0'` names a HOST, and a hard-coded default names a PATH, so the query cannot see the defect [#1190](https://github.com/mudler/vllm.cpp/issues/1190) was opened to remove. Measured at `5c8671c50` over `scripts/` and `tools/`, which is `ENV-AGNOSTIC-W1-TOOLING`'s ownership: the four campaign patterns match 19 files and 22 hits, and widening to `/home/mudler`, `~/venvs/vllm-oracle`, `~/work/vllm.cpp`, `cutlass-4.5.0` and `cutlass_probe` matches 73 files and 174 hits. Twelve of those lines in six shell scripts are the exact `${KEY:-<one operator's path>}` shape the campaign's worked example removed from `scripts/dgx-bringup.sh`, and `scripts/upstream-inventory.py:38-40` carries the Python form `os.environ.get("VLLM_SOURCE", str(Path.home() / "_git/vllm"))`. `ENV-AGNOSTIC-W1-TOOLING` converts `scripts/regen-triton-aot.sh`, because the campaign row wrote that one debt into the file's own text at `:20-23` rather than into a count, and leaves the rest: two of them read as provenance on a first pass and must not be swept blind, since `scripts/cpu-x86-llamacpp-floor.sh:33` states that "every recorded leg used these defaults verbatim" and `scripts/dgx-gdn-packed-bridge-ab.sh:4` dates its prerequisites, which is the same instrument-identity argument that keeps `scripts/mtp-k-gt-1-neartie-gap.py` literal. The blind spot is campaign-level: `ENV-AGNOSTIC-W3-CODE` and `ENV-AGNOSTIC-W4-RECORDS` derive their sets from the same query, so the campaign's claim that its five waves partition 227 files is a claim about the query's 227 and not about the tree. Owed by [`env-agnostic-w1-tooling.md`](specs/env-agnostic-w1-tooling.md) under `## Owed` | bug |