ci: a group that never cancels must never queue, or GitHub discards the baseline (#274)

.agents/issue-index.md

@@ -421,4 +421,5 @@ rather than merged. `scripts/check-agent-record.py` gates both.
 | [#1291](https://github.com/mudler/vllm.cpp/issues/1291) | `ENG-CUDAGRAPH-BREAK` | W3 of the break-point capture seam: the three remaining PLAIN BATCHED decode drivers migrate onto `vt::GraphCaptureScope` + `vt::BreakableGraph` — `Qwen3MoeDecodeGraph` (`qwen3_moe.cpp`), `VoxtralDecodeGraph` (`voxtral.cpp`) and `DeepseekV2DecodeGraph` (`deepseek_v2.cpp`) — one commit each, each landable alone, each with its own RED-first G2 gate. Four of the nine drivers are now on the seam and the six batched-driver `VLLM_CPP_CUDAGRAPH` reads the spec's `## Our baseline` item 1 counted are down to TWO, both in `qwen3_5.cpp` (W4). The two per-model rollback switches STAY (`VT_QWEN3MOE_CUDAGRAPH`, `VT_DEEPSEEK_CUDAGRAPH`): each is an A/B lever for exactly one driver, not a copy of the shared one. **Each driver owes its own gate because nothing else can see the difference** — a driver that kept its hand-rolled `BeginCapture`/`EndCaptureGraph` pair produces IDENTICAL logits, an IDENTICAL backend log and an identical `replay_count()`, so `segments_captured` and `replays` are the only observables that separate "captured a graph" from "captured a graph THROUGH THE SEAM". Red-first on four assertions each (`test_qwen3_moe_decode_graph_seam` 222/226, `test_voxtral_decode_graph_seam` 224/228, `test_deepseek_v2_decode_graph_seam` 224/228, all exit 1), 3/3 green after; the G2 mutation restoring each pre-W3 driver file (25/102, 23/92 and 25/94 lines, each compiled clean, each restored and verified by sha256) reddens ONLY its own gate and leaves `test_breakable_graph` 216/216 and W2's `test_qwen3_decode_graph_seam` 231/231 GREEN. The two swapped seams every such gate needs now live ONCE, in `tests/vllm/models/decode_graph_seam_harness.h`, with W2's file including it: three more copies inside `tests/` would have reproduced the duplication this row removes from `src/`, and two copies of a harness diverge invisibly because both files stay green while measuring different things. **G1 IS DELIVERED, retiring the item W1 and W2 both carried as owed.** `tests/vllm/models/test_decode_graph_seam_g1_cuda.cpp` drives each migrated driver COLD, CAPTURE and THREE consecutive replays and compares every step bit for bit against the driver's own EAGER arm, which is selected by `max_num_reqs == 0` (`PadToCaptureSize` returns -1 and `Step` falls out to its plain forward) so both arms are ONE binary on ONE device rather than two builds, each with its OWN device KV cache so neither can read the other's writes. Measured on `thor:gpu0` through an `rc` lease — NVIDIA Thor sm_110, driver 595.78, nvcc 13.0.88, source `c905bb536`, CUDA-ON build for arch 110, 32 `.cu.o` objects, binary resolving `libcudart.so.13` and `libcublasLt.so.13`, build 489 s at `-j4`: **3 cases, 1600 assertions, exit 0**, `5 steps x 100 logits, 0 differing, 4 replays` for each driver. The ASSERTION COUNT carries that claim and the status line does not: with no CUDA backend every case skips and the same file prints `Status: SUCCESS!` over `assertions: 0`, which the file's own header names as a skip wearing a pass. Bounded honestly: the models are the synthetic tiny ones the CPU forward gates use rather than a checkpoint, and W2's `Qwen3DenseDecodeGraph` is covered by shared-seam ARGUMENT and not by one of the three cases. **W3 ALSO FOUND A GATE THAT COULD NOT FAIL, and closing it is part of the stage.** The three gates assert `breaks_registered == 0` to hold the capture to vLLM's decode arm (`FULL_AND_PIECEWISE`, `vllm/config/compilation.py:63,65-66,630-632` @ `5559679229`). That is a TAUTOLOGY for these models: the counter moves only when a `vt::GraphBreak` registers into a splitting scope, and the ONE production break point in the tree is W1's, in `qwen3.cpp` — W2's driver runs through it, none of W3's three does. Measured rather than reasoned: flipping `kFull` to `kPiecewise` in `qwen3_moe.cpp`, one token, compiled clean and left that driver's whole gate GREEN at 226/226, and the same flip was equally invisible in the other two. The mode was UNOBSERVABLE from outside a driver — the scope is a `Step` local, the container is private to the driver's `Impl`, and a token gate cannot see a segment count — so `vt::GraphBreakStats` gains `full_scopes` and `piecewise_scopes`, counted in `GraphCaptureScope`'s constructor on the ACTIVE path only (an inert scope makes no backend call in either mode, so counting it would report a mode that never reached a backend), cleared by `ResetGraphBreakStats()`, gated in `tests/vt/test_breakable_graph.cpp` with the inert arm as the control that stops them degenerating into "scopes constructed". Re-measured after the fix, the same one-token flip now REDS each of the three gates on exactly those two assertions (226/228, 228/230, 228/230, all exit 1). **NO break point is registered in these three models, and that is a decision rather than an omission**: under `kFull` a break point takes the same pass-through arm it takes outside a scope, so registering one would land machinery no gate can exercise; establishing each model's break-point set is what the PIECEWISE arm needs, that arm is blocked on replay-safe closure inputs (W4), and W6 is where the eligibility predicate moves. **The async device-token decline, stated per driver rather than left as an omission.** `VoxtralDecodeGraph` needs none: its only construction site is `VoxtralGenerateGreedy`, the single-sequence multimodal greedy loop, which is not reached from `runner.cpp` and has no asynchronous device mirror. The other two are a NEW FINDING, filed [#1305](https://github.com/mudler/vllm.cpp/issues/1305) with W4 as owner and NOT fixed in flow, because adding a decline trades a shipped default-ON capability away on a measurement W3 cannot make and the fix `qwen3.cpp:1106`'s own comment names is `StepDevInputs` as a seam capability. **G5's ROCm/Tenstorrent arm is NOT discharged**, and the owner moves from W3 to W5 with the reason stated: the fleet carries no ROCm and no Tenstorrent device, so it is BLOCKED on hardware rather than unattempted. What W3 can say instead is that the seam's CUDA arm now runs on TWO architectures, sm_110 here and sm_121a for W1's exit criterion. Still NO throughput claim anywhere: the prefill refutation stands (3.8% host idle, >96% GPU-busy, 92.5% glue) and `kFull` keeps each migrated step's shape the one it already had | feature |
 | [#1305](https://github.com/mudler/vllm.cpp/issues/1305) | `ENG-CUDAGRAPH-BREAK` | Three decode-graph registrations route an ASYNCHRONOUS step into a host-vector replay with NO `device_token_ids` decline. Found while migrating two of them onto the break-point capture seam ([#1291](https://github.com/mudler/vllm.cpp/issues/1291)) and NOT caused by that migration — the shape is present at its base commit `5d9fe332c`. `src/vllm/model_executor/models/qwen3.cpp:1106` returns `std::nullopt` whenever `ModelForwardInput::device_token_ids` is non-null, on a measured four-concurrent battery (`depth-1 graph ON PASS 78/78`, `depth-2 graph OFF PASS 82/82`, `depth-2 graph ON FAIL, slots 1-3 degenerate`, #323/#1179), because `Step()` replays against persistent HOST `token_ids` while the asynchronous combine has patched the DEVICE ids and deliberately left the host vector stale for decode rows (`src/vllm/v1/worker/gpu/runner.cpp:1265-1309,1523`). That comment calls the hazard "latent for EVERY classic-dense model, since the graph is default-ON". `qwen3_moe_registry.cpp:107`, `deepseek_v2_registry.cpp:106` and `glm4_moe_lite_registry.cpp:125` (the last constructs the DeepSeek driver) admit a pure-decode step to a driver with the same host-vector replay and carry NO such check; `grep -c StepDevInputs` returns 0 for `qwen3_moe.cpp`, `deepseek_v2.cpp` and `voxtral.cpp` against 41 for `qwen3_5.cpp`, which is the divergence the row's spec already records as the cost of nine drivers. `VoxtralDecodeGraph` is NOT affected: its only construction site is `VoxtralGenerateGreedy`, which the runner does not reach and which has no device mirror. NOT fixed in flow, for two reasons stated rather than assumed: adding a decline is a behaviour change that trades a shipped, default-ON capability away, and W3 has no four-concurrent battery for these two models to justify it; and the correct fix is already assigned — `StepDevInputs` (`qwen3_5.cpp:3894`) as a SEAM capability, which is `## Work breakdown` W4 of [eng-cudagraph-break.md](specs/eng-cudagraph-break.md) and the only version that reaches these registrations at all. Owner: **W4**, with [#1179](https://github.com/mudler/vllm.cpp/issues/1179) as the standing tracker. What would settle it: run that same battery against Qwen3-Coder and DeepSeek-V2-Lite with `VT_ASYNC_DEVICE_MIRROR` live and the decode graph ON — either it reproduces the depth-2 degeneration, or the reason it does not has to be found and written down rather than assumed | bug |
 | [#1308](https://github.com/mudler/vllm.cpp/issues/1308) | — | ENV-AGNOSTIC: the campaign's derivation query `git grep -cIE 'dgx\.casa\|nas_share\|192\.168\.\|thor:gpu0'` names a HOST, and a hard-coded default names a PATH, so the query cannot see the defect [#1190](https://github.com/mudler/vllm.cpp/issues/1190) was opened to remove. Measured at `5c8671c50` over `scripts/` and `tools/`, which is `ENV-AGNOSTIC-W1-TOOLING`'s ownership: the four campaign patterns match 19 files and 22 hits, and widening to `/home/mudler`, `~/venvs/vllm-oracle`, `~/work/vllm.cpp`, `cutlass-4.5.0` and `cutlass_probe` matches 73 files and 174 hits. Twelve of those lines in six shell scripts are the exact `${KEY:-<one operator's path>}` shape the campaign's worked example removed from `scripts/dgx-bringup.sh`, and `scripts/upstream-inventory.py:38-40` carries the Python form `os.environ.get("VLLM_SOURCE", str(Path.home() / "_git/vllm"))`. `ENV-AGNOSTIC-W1-TOOLING` converts `scripts/regen-triton-aot.sh`, because the campaign row wrote that one debt into the file's own text at `:20-23` rather than into a count, and leaves the rest: two of them read as provenance on a first pass and must not be swept blind, since `scripts/cpu-x86-llamacpp-floor.sh:33` states that "every recorded leg used these defaults verbatim" and `scripts/dgx-gdn-packed-bridge-ab.sh:4` dates its prerequisites, which is the same instrument-identity argument that keeps `scripts/mtp-k-gt-1-neartie-gap.py` literal. The blind spot is campaign-level: `ENV-AGNOSTIC-W3-CODE` and `ENV-AGNOSTIC-W4-RECORDS` derive their sets from the same query, so the campaign's claim that its five waves partition 227 files is a claim about the query's 227 and not about the tree. Owed by [`env-agnostic-w1-tooling.md`](specs/env-agnostic-w1-tooling.md) under `## Owed` | bug |
+| [#1316](https://github.com/mudler/vllm.cpp/issues/1316) | — | `scripts/main-baseline.py` renders a scheduled run that executed ZERO jobs as `RED` with all 11 covered jobs `missing`, so `NEWEST BASELINE: RED at <sha>` names a tree the run never checked out. Measured at `origin/main` `250db75a2`: runs `32206456661` and `32140419182` both return `startedAt: null` for every job, because GitHub cancelled them while they were pending in the single `ci-schedule-refs/heads/main-mudler/vllm.cpp` group, whose queue holds one run ([#274](https://github.com/mudler/vllm.cpp/issues/274)). Fail-closed, and the `missing (expected, never ran)` line is accurate about the jobs; the defect is the verdict word, because a run that executed nothing is NOT RUN rather than RED, and the newest verdict should fall through to the newest run that actually ran. NOT fixed in flow: `test_an_expected_job_the_payload_never_mentions_is_red` and `test_a_narrowed_run_reports_red_and_names_what_never_ran` deliberately assert missing-is-red so a narrowed run cannot pass, and separating "narrowed" from "never started" changes what the verdict means, which owes its own spec, red-before evidence and a fresh reviewer. Owed under `## Owed` of [baseline-lane-eviction.md](specs/baseline-lane-eviction.md), which removes the only observed producer of a zero-job run | bug |
 | [#1314](https://github.com/mudler/vllm.cpp/issues/1314) | `SPEC-DFLASH2` | **DFlash2 (`DFlash2DraftModel`) is unported**, and one config rule would run the published checkpoint wrong in silence. Upstream carries DFlash2 as a SECOND architecture beside DFlash rather than as a change to it ([vllm#52816](https://github.com/vllm-project/vllm/pull/52816), OPEN at head `19c9351904df4c63042671bc67a866ca48dc7d6f`, base `9842d701`, 755+/5-, 11 files, plus the stacked guard fix [vllm#52883](https://github.com/vllm-project/vllm/pull/52883)): DFlash1 gains two subclass seams and keeps every behaviour, and the new architecture adds a GROUPED DYNAMIC DEPTHWISE CONVOLUTION around each attention and each MLP sublayer plus a CANDIDATE SELECTOR that replaces the independent per-slot argmax with a scored path walk over the target head's top-K. Shapes taken from the published checkpoint rather than from the diff: `z-lab/Qwen3.8-27B-DFlash2`, safetensors header range-read 2026-08-19, 81 tensors -- DFlash1's set plus `layers.N.{attention,mlp}_conv.{base_kernel (2,2,5120), kernel_projection.weight (1280,5120)}` x5 and `candidate_selector.{hidden_projection.weight (256,5120), predecessor_codebook, successor_codebook}` at `(248320,256)` bf16 each, ~254 MB resident the DFlash1 lane never allocates; `conv_kernel_size 2`, `conv_group_size 16`, `selector_rank 256`, `selector_top_k 16`, `block_size 8`. **The silent one:** that config declares all five layers `sliding_attention` AND `is_causal false`, while our resolution mirrors the OLD upstream rule (causal iff SWA, unless `dflash_config.causal`, `include/vllm/model_executor/models/qwen3_dflash.h:22-24`), so every layer would run CAUSAL -- plausible tokens, a token gate against our own output sees nothing, and only ACCEPTANCE moves, which the lossless verify hides. Upstream changes `_dflash_layer_causal` to read `is_causal` first, in the same commit. Three further things are owed and none of them is silent: no route for the `DFlash2DraftModel` architecture string (the same classification code as the open `DSparkDraftModel` gap, [#1193](https://github.com/mudler/vllm.cpp/issues/1193)); no top-k that EMITS the surviving (id, value) pairs, where the decision is to extend the sort-free pivot-bracket threshold search already ported from the same FlashInfer approach at `src/vt/cuda/cuda_sample.cu:297-506` rather than port FlashInfer's 3380-line general radix kernel; and the path walk must run ON DEVICE from the first landing, because the identical sequential shape in DSpark shipped host-side and measured 28% of the 27B draft step ([#436](https://github.com/mudler/vllm.cpp/issues/436)) before `SampleSequentialDevice` moved it. Already reusable unchanged: `vt::DFlashBlockAttention`, the DFlash runner/rejection/GDN-rollback lane, and the loader's target-shared `embed_tokens`+`lm_head`, which is already what a DFlash2 checkpoint needs. BEYOND-PIN by developer decision 2026-08-19 (mirror the open PR now, reconcile if review moves it), in the same posture `SPEC-DSPARK-QWEN3-ROUTING` takes toward vllm#52197; the parity pin `555967922` does not carry the architecture and is NOT advanced. Gate arm is bf16 27B plus the GGUF drafter arm in the same wave, oracle = vLLM built at the PR head, acceptance measured SAME-TRAJECTORY because `SPEC-DFLASH` D8 spent a campaign on a divergent-trajectory confound that D9 refuted. Spec [dflash2-spec-decode.md](specs/dflash2-spec-decode.md) | feature |