feat(ENG-CUDAGRAPH-BREAK): migrate the last three plain batched decode drivers onto the seam, and pay G1 on a device (#1291, #1163)

[localai-bot]

W3 of ENG-CUDAGRAPH-BREAK migrates the three remaining PLAIN BATCHED decode
drivers onto the break-point capture seam: Qwen3MoeDecodeGraph (Qwen3-Coder
MoE), VoxtralDecodeGraph (Voxtral text) and DeepseekV2DecodeGraph (MLA, also
constructed by the GLM4-MoE-Lite registration). Each Step opens a
vt::GraphCaptureScope over a per-slot vt::BreakableGraph in kFull and
replays through BreakableGraph::Replay, so three more copies of the hand-rolled
BeginCapture/EndCaptureGraph pair, the raw void* handle, the bool captured flag, the DestroyGraph loop and the driver's own VLLM_CPP_CUDAGRAPH
read are gone. Four of the nine drivers are now on the seam, and the six
batched-driver reads of that variable the spec's ## Our baseline item 1 counted
are down to TWO, both in qwen3_5.cpp, which W4 owns.

Each driver landed as its own commit with its own gate, because they share W2's
shape and nothing about one depends on another. The two per-model rollback
switches STAY: VT_QWEN3MOE_CUDAGRAPH and VT_DEEPSEEK_CUDAGRAPH are A/B levers
for exactly one driver each, not copies of the shared one.

kFULL, INHERITED FROM W2 AND NOT RE-ARGUED. vLLM's v1 default
FULL_AND_PIECEWISE (vllm/config/compilation.py:63 @ pin 5559679229) is
documented at :630-632 as a FULL graph for DECODE batches and a piecewise one
for prefill and mixed batches, and decode_mode() (:65-66) returns the full
half. W2 measured what the alternative costs: a kPiecewise scope over a decode
driver does not merely slow the step down, it FAULTS on the first replay, because
W1's break closure captures its layer frame by reference. No throughput is
claimed here and none was measured. kFull is what keeps each migrated step's
shape the one it already had.

Each driver owes its own gate, because nothing else can see the difference

A driver that kept its raw pair produces IDENTICAL logits, an IDENTICAL backend
log and an identical replay_count(). vt::GraphBreakStats::segments_captured
moves only when a vt::GraphCaptureScope closes a segment and replays only
inside vt::BreakableGraph::Replay, so those two are the only observables that
separate "captured a graph" from "captured a graph THROUGH THE SEAM". Each gate
drives Step through cold, capture, replay and a second replay and asserts them.

RED FIRST, on four assertions each, before its migration:

gate	red	green
test_qwen3_moe_decode_graph_seam	222/226, exit 1	3/3, 228/228
test_voxtral_decode_graph_seam	224/228, exit 1	3/3, 230/230
test_deepseek_v2_decode_graph_seam	224/228, exit 1	3/3, 230/230

G2 REACHABILITY MUTATION, one per driver: restore the pre-W3 driver file in a
scratch copy (25/102, 23/92 and 25/94 changed lines; each compiled clean, exit
0). Each reddens ONLY its own gate on those four assertions and leaves
test_breakable_graph 216/216 and W2's test_qwen3_decode_graph_seam 231/231
GREEN. Each file restored and verified by sha256sum -c.

Two cases per file were green BEFORE the change, deliberately, and they are
controls rather than red-first cases. G4 — the capture step bit-identical to the
eager arm, 100 values, 0 differing — is what makes each migration reversible, so
it must not move. The capture-FAILURE case passed against each pre-W3 driver
because s.graph = b.EndCaptureGraph(...) was unguarded and propagated; it is
here because W2's first head turned exactly that into a silent return of
pool-recycled memory, and a migration that reads only captured() cannot tell a
FAILED capture from an INERT scope.

THE HARNESS IS NOW SHARED. W2's gate carried its own capture-capable CPU backend
and static-graph CPU platform, and W3 adds three more gates of that shape.
tests/vllm/models/decode_graph_seam_harness.h holds them once and W2's file
includes it — a fourth copy inside tests/ would reproduce the duplication this
row removes from src/, and two copies of a harness diverge invisibly because
both files stay green while measuring different things. W2's gate is unchanged in
behaviour: 4/4, 231 assertions, the same numbers its own commit recorded.

G1 is DELIVERED, on a leased device

This is the item W1 and W2 both carried as owed, and it is owed for a structural
reason: every other gate this row owns runs against a CPU backend that implements
the capture vocabulary by LOGGING, and a CPU kernel is a direct function call
rather than a backend submission, so nothing a CPU harness calls a replay
recomputes anything.

tests/vllm/models/test_decode_graph_seam_g1_cuda.cpp runs each migrated driver
COLD, CAPTURE and THREE consecutive replays and compares every step bit for bit
against that driver's own EAGER arm. The eager arm is selected WITHOUT an
environment variable: PadToCaptureSize(b, max_num_seqs) returns -1 when b
exceeds max_num_seqs, and every migrated Step falls out to its plain forward
on that value, so max_num_reqs == 0 gives the same binary on the same device
taking the eager path. Each arm gets its OWN device KV cache, so neither can read
the other's writes and call the agreement a result.

Measured on thor:gpu0 through an rc lease — NVIDIA Thor sm_110, driver
595.78, nvcc 13.0.88, source c905bb536, CUDA-ON build for arch 110, 32
.cu.o objects, binary resolving libcudart.so.13 and libcublasLt.so.13,
build 489 s at -j4:

[doctest] test cases:    3 |    3 passed | 0 failed | 0 skipped
[doctest] assertions: 1600 | 1600 passed | 0 failed |
G1 Qwen3MoeDecodeGraph  on CUDA: 5 steps x 100 logits, 0 differing, 4 replays
G1 VoxtralDecodeGraph   on CUDA: 5 steps x 100 logits, 0 differing, 4 replays
G1 DeepseekV2DecodeGraph on CUDA: 5 steps x 100 logits, 0 differing, 4 replays

The assertion count carries that claim and the status line does not. With no
CUDA backend every case skips and the same file prints Status: SUCCESS! over
assertions: 0, which the file's own header names as a skip wearing a pass.

TWO LIMITS, recorded rather than left to be discovered. The models are the
synthetic tiny ones the CPU forward gates already use: the run exercises the real
CUDA kernels and the real capture and replay, and NOT a checkpoint's weights or a
long context. And W2's Qwen3DenseDecodeGraph is not one of the three cases — it
shares the seam and the shape, which is an argument and not a measurement.

W3 found a gate that could not fail, and closing it is part of this stage

The three gates assert breaks_registered == 0 to hold each capture to vLLM's
decode arm. For these models that assertion is a TAUTOLOGY: it moves only when a
vt::GraphBreak registers into a splitting scope, and the ONE production break
point in the tree is W1's, in qwen3.cpp. W2's driver runs through it, so for W2
the guard was load-bearing; none of W3's three registers one. Measured, not
reasoned: flipping kFull to kPiecewise in qwen3_moe.cpp — one token —
compiled clean and left that driver's whole gate GREEN at 226/226.

The mode was UNOBSERVABLE from outside a driver: the scope is a Step local, the
container is private to the driver's Impl, and a token gate cannot see a
segment count. vt::GraphBreakStats gains full_scopes and piecewise_scopes,
counted in GraphCaptureScope's constructor on the ACTIVE path only — an inert
scope makes no backend call in either mode, so counting it would report a mode
that never reached a backend. Gated in tests/vt/test_breakable_graph.cpp with
the inert arm as the control (29 cases, 229 assertions). RE-MEASURED after the
fix: the same one-token flip now REDS each of the three driver gates on exactly
those two assertions (226/228, 228/230, 228/230, all exit 1).

NO BREAK POINT IS REGISTERED IN THESE THREE MODELS, and that is a decision. Under
kFull a break point takes the same pass-through arm it takes outside a scope,
so registering one would land machinery no gate can exercise. Establishing each
model's break-point set is what the PIECEWISE arm needs; that arm is blocked on
replay-safe closure inputs, which W4 owns, and W6 is where the eligibility
predicate moves.

The async device-token decline, stated per driver

• VoxtralDecodeGraph needs none. Its only construction site is
  VoxtralGenerateGreedy, the single-sequence multimodal greedy loop, which is
  not reached from runner.cpp and has no asynchronous device mirror.
• Qwen3MoeDecodeGraph and DeepseekV2DecodeGraph are a NEW FINDING, filed
  as Three decode-graph registrations route async steps into a host-vector replay with no device_token_ids decline #1305 and NOT fixed in flow. qwen3_moe_registry.cpp:107,
  deepseek_v2_registry.cpp:106 and glm4_moe_lite_registry.cpp:125 route a
  pure-decode step into a host-vector replay with NO device_token_ids check,
  while qwen3.cpp:1106 declines for exactly that condition on a measured
  battery whose own comment calls the hazard "latent for EVERY classic-dense
  model, since the graph is default-ON". The shape is present at this branch's
  base commit 5d9fe332c, so W3 found it rather than caused it. Adding a decline
  would trade a shipped, default-ON capability away on a measurement this stage
  cannot make, and the fix qwen3.cpp names is StepDevInputs as a SEAM
  capability — W4, and the only version that reaches these registrations at all.
  Owner W4, with The decode-graph driver count recorded in 9bc4d7f44 is eight; it is nine, and the ninth predates the record #1179 as the standing tracker.

Two owed items are corrected rather than quietly inherited

• G5's ROCm and Tenstorrent arm read Owner: W3 and W3 did not discharge it.
  The fleet carries no ROCm and no Tenstorrent device, so it is BLOCKED on
  hardware rather than unattempted, and it moves to W5 — the stage that
  migrates the driver family whose Tenstorrent recapture path is the only place a
  ttnn mesh trace meets this seam. What W3 can say instead: the seam's CUDA arm
  now runs on TWO architectures, sm_110 here and sm_121a for W1's exit criterion.
• The async serving battery read Owner: W3. The lease was obtainable; the
  CHECKPOINT was not, because the G1 run built from a clean clone on the leased
  box's local disk against synthetic weights. It moves to W4, which owns the
  fix the battery exists to validate.

Gates run

Focused, on the final head: test_breakable_graph 29/29 (229),
test_qwen3_decode_graph_seam 4/4 (231), test_qwen3_break_point 2/2 (516),
test_qwen3_moe_decode_graph_seam 3/3 (228), test_voxtral_decode_graph_seam
3/3 (230), test_deepseek_v2_decode_graph_seam 3/3 (230),
test_decode_graph_sizes 5/5 (478), test_qwen3_forward 10/10 (1575),
test_qwen3_moe_forward 3/3 (504), test_deepseek_v2_forward 11/11 (1052),
test_graph_safe_scratch (4). ctest -R 'voxtral|glm4|deepseek|qwen3_moe|graph'
46/46 passed. Full CPU build exit 0.

Component checkers green: commit trailers and commit style over
origin/main..HEAD, agent record (ENGINE=164 MODEL=377 QUANT=84 KERNEL=52 BACKEND=85), issue-index append-only against origin/main, public doc tables
(docs/STATUS.md inside its size ratchet), symbol anchors (649/649), test
registration, gate commands, NOW currency, role discipline, prompt contracts,
model checklist, oracle pins.

Negative mutations, each printed with its git diff --stat, its compile status
and its exit status, each detected and each restored byte-for-byte and verified
by sha256: the three G2 driver mutations above; the capture_failed() branch
disabled in qwen3_moe.cpp (CHECK_THROWS_AS(...) did NOT throw at all!,
225/226, exit 1); and the kFull -> kPiecewise flip on all three drivers,
UNDETECTED before the mode counters and detected after.

Closes #1291. Parent #1163 stays open: W4 lifts StepDevInputs into the seam and
owns both the async decline and #1305, W5 takes the three single-shape drivers
and G5's other backends, and W6 moves the eligibility predicate off
pure_decode.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [Claude Code]