L4: the kernel gate is a tracked exception, and the oracle denominator is now gated (#607, #414)

.agents/benchmark-record.md

@@ -25988,3 +25988,52 @@ matched the GEMMs' measured 3.98 TFLOP/s, its 0.140 TFLOP would cost **35.2 ms
 instead of 688.1 ms**: forward 1569 -> 916 ms (**1.71x on the DiT**), bucket
 370.510 -> 216.3 s, run 598.207 -> 444.0 s (**1.35x end to end**). An upper
 bound from a flop ratio on a kernel nobody has written.
+### The Qwen3.5-4B `1.0283x` row ran against an UNFUSED denominator (#414, #1345)
+
+Found 2026-08-19 while landing [#607](https://github.com/mudler/vllm.cpp/issues/607)
+wave L4, which gated the CLI oracle-launch surface. This leg is on the OTHER
+surface, so the new checker does not see it, and it is the one published figure
+the L4 sweep found still carrying the #414 defect unmarked.
+
+The chain, each link checked rather than assumed:
+
+1. `docs/BENCHMARKS.md` cites `docs/bench-evidence/qwen35-4b-sm120-main-20260807.md`,
+   whose reproduction identity names `Qwen/Qwen3.5-4B` and the exact workload
+   (128 requests, 128 output tokens, concurrency 32,
+   `max_num_batched_tokens=2048`, greedy) that `tools/bench/run_qwen35_4b_compare.sh`
+   drives.
+2. `tools/bench/run_qwen35_4b_compare.sh:120` runs the vLLM arm through
+   `tools/bench/vllm_closed_loop_metrics.py`. Its `LLM(...)` at `:160` never
+   passes `language_model_only` and exposes no argument that could set it.
+3. Upstream registers `Qwen/Qwen3.5-4B` under `Qwen3_5ForConditionalGeneration`
+   (`tests/models/registry.py:1322-1324`, `extras={"4b": "Qwen/Qwen3.5-4B"}`), so
+   `multimodal_config` is non-`None` and `text_only` resolves `False`
+   (`vllm/config/multimodal.py:78`, `vllm/model_executor/models/qwen3_next.py:325`).
+4. The conjunct was live at the time of the run. `git log -S'use_fused_qk_norm_rope_gate'`
+   over `vllm/model_executor/models/qwen3_next.py` dates its introduction to
+   `16282a9c4` on 2026-06-10, two months before the 2026-08-07 measurement.
+5. The remaining conjuncts hold: sm_120 is CUDA, and Qwen3.5-4B carries
+   `attn_output_gate` with NeoX-style RoPE.
+
+So the oracle issued four ops per full-attention layer where its own production
+configuration issues one, against our single fused launch.
+
+**Direction: it flatters us**, for the reason #414 gives. **Magnitude: NOT
+MEASURED, and deliberately not estimated.** The model is GDN-hybrid, so only its
+full-attention layers are exposed, and the L4 wave took no measurement: both
+fleet devices were held and it carried no lease authority. Quoting a corrected
+ratio here would be a number nobody took.
+
+The row keeps its values and gains the attribution, per AGENTS.md: evidence is
+annotated, never deleted. Re-measurement is owed with
+[#1345](https://github.com/mudler/vllm.cpp/issues/1345), which also owns the
+repair to the three in-process harnesses.
+
+**Not affected, so the next reader does not re-derive it.** #414 reaches a figure
+only where the full-attention layers are `Qwen3NextAttention` AND the checkpoint
+loads as a `*ForConditionalGeneration`. That is the Qwen3.5/3.6/3.8 family alone.
+The OPT, GLM-4, InternLM2, Qwen3-dense, Qwen3-Coder and DeepSeek-V2-Lite legs
+never construct `Qwen3NextAttention`. The `Qwen3.8-27B` rows ran through
+`tools/bench/run_serve_low.py`, which has passed `--language-model-only` since it
+was written. The Qwen3.5-4B GDN prefill kernel row is conv and post-conv timing
+on the LINEAR-attention path, which the full-attention preamble does not touch.

.agents/specs/sm120-qwen35-pareto-2026-08-09.md

@@ -328,7 +328,8 @@ file moved and the row stays `GATING`. What git shows:
 - The exact 4B series has been **rerun six times since** — `qwen35-4b-*`
   `20260727`, `20260728`, `20260729`, `20260803`, `20260805`, `20260807` in
   `docs/bench-evidence/` — and the throughput axis moved from **0.9864x** to
-  **1.0283x PASS** (6831.71 vs 6643.40 tok/s, three repetitions).
+  **1.0283x PASS** (6831.71 vs 6643.40 tok/s, three repetitions) — OPTIMISTIC,
+  because that denominator ran UNFUSED ([#1345](https://github.com/mudler/vllm.cpp/issues/1345), #414).
 - The successor lever on the same wait was tried and **rejected**: geometric
   argmax scratch migrated the wait rather than removing it and cost +1% TPOT
   ([spec](sm120-qwen35-argmax-scratch-growth-2026-08-09.md)).

.github/workflows/ci.yml

@@ -262,6 +262,15 @@ jobs:
         run: |
           python3 scripts/check-surface-coverage.py
           python3 tests/scripts/test_check_surface_coverage.py
+      - name: No harness launches the pinned oracle on a handicapped path (#414)
+        # Without `--language-model-only` a Qwen3.6 checkpoint gives the oracle
+        # `text_only == False` (qwen3_next.py:324-331 at the pin), so it runs its
+        # UNFUSED QK-norm+RoPE+gate path against our fused one and every ratio it
+        # produces is FLATTERED. The 2026-08-13 series repaired the RUN; this
+        # repairs the HARNESS, which is what the next campaign reaches for.
+        run: |
+          python3 scripts/check-oracle-denominator-flags.py
+          python3 tests/scripts/test_check_oracle_denominator_flags.py
       - name: The x86_64 CPU floor harness runs from a clean checkout
         # It shipped unable to run at all: `OUT=evi` with no `mkdir -p`, so every
         # redirection failed, every leg was discarded for a non-zero exit, and

docs/BENCHMARKS.md

@@ -58,7 +58,7 @@ The first series free of both, at the pin, graphed, and at a pinned clock is in
 | Qwen3.6-27B | NVFP4 (`nvidia` @`0893e160`, ModelOpt `modelopt_mixed`) | 0.25.0 ROLLBACK, SUPERSEDED | 0/6 | BEHIND, uniformly 0.94x on decode, flat c1-c32, but **VOID as a ratio** (#520, #414). At the pin, clocks pinned: **0.976x c1 / 0.946x c4 TPOT**. Different model from the `unsloth` row (NVFP4 MLP + FP8 W8A8 tower) |
 | Qwen3.6-35B-A3B | NVFP4 `modelopt_mixed` | 0.25.0 ROLLBACK, SUPERSEDED | 2/18 | 3-rep grid 2026-08-05 @`1ea26427`: 0.93-1.03x, VOID as ratios (#520, #414). At the pin, clocks pinned: **0.995x c1 / 0.946x c4 TPOT**. ★ probe found a prod async batch-1 greedy DEGENERATION bug the mirror fixes |
 | DeepSeek-V2-Lite | bf16 MLA | 0.25.0 ROLLBACK, SUPERSEDED | 4/25 | Attributed miss, row stays `ACTIVE` |
-| Qwen3.5-4B | bf16 direct-load | 0.26.0.dev0 | **1.0283x tput, `PENDING`** | OPEN: TTFT/TPOT/E2E 1.085/1.017/1.029x, VRAM +118.7 MiB ([data](bench-evidence/qwen35-4b-sm120-main-20260807.md)) |
+| Qwen3.5-4B | bf16 direct-load | 0.26.0.dev0, UNFUSED and vision-tower-carrying denominator ([#1345](https://github.com/mudler/vllm.cpp/issues/1345)) | **1.0283x tput, `PENDING` and OPTIMISTIC** | OPTIMISTIC on all axes (#414). TTFT/TPOT/E2E 1.085/1.017/1.029x; VRAM **+118.7 MiB** OPEN, not like-for-like: the oracle built the tower the flag elides ([data](bench-evidence/qwen35-4b-sm120-main-20260807.md)) |
 | Qwen3.8-27B | bf16 (@`1d4bf0f2`) | 0.26.0.dev0 at the pin, graphed; c4 at a pinned 2184 MHz, the 2026-08-19 re-measure SAMPLED only | **1 of 3 concurrency cells** | Token gate PASSES. c4 like-for-like: tput **0.963x**, ITL **1.008x**. c1 re-measured COMPLETE, pairing DISCARDED on clock spread; c8 vLLM denominator NOT MEASURABLE ([#915](https://github.com/mudler/vllm.cpp/issues/915)) |
 
 ### GDN prefill kernels by GPU

scripts/agent-preflight.sh

@@ -106,6 +106,7 @@ CHECKERS=(
   check-test-registration
   check-snapshot-pins
   check-oracle-pins
+  check-oracle-denominator-flags
   check-now-current
   check-gate-commands
   check-symbol-anchors
@@ -163,6 +164,7 @@ SUITES=(
   test_agent_preflight_skip_report
   test_agent_pr_body
   test_check_symbol_anchors
+  test_check_oracle_denominator_flags
   test_check_conflict_markers
 )
 

scripts/check-oracle-denominator-flags.py

@@ -0,0 +1,254 @@
+#!/usr/bin/env python3
+"""Refuse a harness that launches the pinned vLLM oracle SERVER on a handicapped path.
+
+This is the structural half of
+[#414](https://github.com/mudler/vllm.cpp/issues/414), and the gate that #607
+wave L4 owes (`.agents/specs/multimodal-track.md` §1.6).
+
+The defect it removes
+---------------------
+At the pin, `vllm/config/multimodal.py:78` defaults `language_model_only` to
+`False`. A Qwen3.6 checkpoint loads as `Qwen3_5*ForConditionalGeneration`, so its
+`multimodal_config` is non-`None`, so
+`vllm/model_executor/models/qwen3_next.py:324-331` computes `text_only == False`
+and DISABLES `use_fused_qk_norm_rope_gate`. The oracle then issues four ops per
+full-attention layer where its own production configuration issues one, while
+our arm has issued the single fused launch by default throughout
+(`src/vllm/model_executor/models/qwen3_5.cpp::FuseAttnPreambleOn`).
+
+A ratio measured that way compares two different algorithms, which AGENTS.md
+§Gates forbids, and the error runs in OUR favour, hardest on TTFT. The
+2026-08-13 clock-controlled series repaired it by passing
+`--language-model-only`. That repaired the RUN, not the HARNESS:
+`scripts/dgx-online-serving.sh` still launched the oracle without the flag while
+`tools/bench/run_serve_low.py` passed it, so the two disagreed about the
+oracle's configuration and the next canonical campaign would have reproduced the
+defect. A grep for the flag reads as coverage while the canonical driver lacks
+it, which is why this is a checker and not a comment.
+
+Scope, and what is deliberately outside it
+------------------------------------------
+This gate covers the ONE surface where the published parity ratios come from: a
+command that launches the oracle SERVER through its CLI. A token `serve`
+immediately following an oracle client token (`vllm`, `.../vllm`, or a
+`${client}`-style variable) is a launch. `vllm bench serve` is the timed CLIENT
+rather than a server, and is not matched, because `serve` there follows `bench`.
+
+The same defect exists on the IN-PROCESS surface: `LLM(...)` in
+`tools/bench/{profile_vllm_online_gate,vllm_closed_loop_metrics,dump_vllm_tokens}.py`
+constructs the oracle with `language_model_only` left at its `False` default and
+exposes no way to set it. Those harnesses take `--model` as a path, so no static
+rule can know whether a given run points at a multimodal checkpoint; the repair
+is to thread the knob through and record the resolved value beside the
+measurement. That is tracked separately and is NOT enforced here, because a
+checker that cannot decide its own question is worse than none.
+
+Each matched launch must pass `--language-model-only` or carry an explicit
+exemption naming a reason:
+
+    # ORACLE-DENOMINATOR-EXEMPT: <reason>
+
+An exemption is for an architecture with no `multimodal_config` at all, where
+`text_only` is already `True` and the flag decides nothing. It is not a place to
+park a multimodal checkpoint.
+
+Why this is not a lock
+----------------------
+The checker holds its expectation in its own code and reads harness sources. No
+pull request has to edit a shared record to pass it, and adding a harness costs
+nothing unless that harness launches an oracle server.
+
+    scripts/check-oracle-denominator-flags.py            # gate
+    scripts/check-oracle-denominator-flags.py --json     # the discovered set
+    scripts/check-oracle-denominator-flags.py --root DIR # scan another tree
+
+`--json` exists so a test can pin the DISCOVERED set, not only the violation
+count. A detector whose terms stopped matching would otherwise report a clean
+tree, which is the failure this project has already paid for elsewhere: a null
+grep proves the terms wrong, never an absence.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import re
+import sys
+from pathlib import Path
+
+
+ROOT = Path(__file__).resolve().parents[1]
+
+# Harness surfaces. A file outside these globs cannot launch a gate oracle
+# without first becoming a harness, at which point it lands here in the same
+# change.
+SCAN_GLOBS = ("scripts/*.sh", "scripts/*.py", "tools/bench/*.py", "tools/bench/*.sh")
+
+# This checker reads itself out of the scan: its own prose names every token it
+# looks for, and matching that prose would be a tautology.
+SELF = "check-oracle-denominator-flags.py"
+
+REQUIRED_FLAG = "--language-model-only"
+EXEMPT_MARKER = "ORACLE-DENOMINATOR-EXEMPT:"
+
+# A token naming the pinned oracle CLI: either the binary spelled out, or ANY
+# shell or Python variable holding it. Naming the two variables this tree
+# happens to use today would have made the detector a list of spellings, and a
+# harness calling its binary `${ORACLE}` or `${server_bin}` would have launched
+# the oracle unscanned while the checker reported a clean tree. What identifies
+# a launch is `serve` sitting directly after the binary, so the binary may be
+# anonymous. `bench serve` is unaffected and stays a client: the token before
+# `serve` there is `bench`, which is no expansion at all.
+#
+# Case-sensitive on purpose: prose writes "vLLM", commands write "vllm". The
+# optional closing brace is not cosmetic -- `_TOKEN_STRIP` eats `}` off the end
+# of a token, so `"${client}"` reaches this pattern as `${client`.
+_ORACLE_CLIENT = re.compile(r"\A(?:(?:.*/)?vllm|\$\{?[A-Za-z0-9_]+\}?)\Z")
+_TOKEN_STRIP = "\"'`,()[]{}\\"
+# A line that opens a command block, so a flag placed before `serve` still counts.
+_OPENER = re.compile(r"[(\[]\s*$")
+# A line that closes one.
+_CLOSER = re.compile(r"\A\s*[)\]]\s*,?\s*\Z")
+
+
+def _tokens(line: str) -> list[str]:
+    return [t.strip(_TOKEN_STRIP) for t in line.split()]
+
+
+def _is_comment(line: str) -> bool:
+    stripped = line.lstrip()
+    return stripped.startswith("#") or stripped.startswith("*")
+
+
+def _block(lines: list[str], index: int) -> tuple[int, int]:
+    """The command block containing line ``index`` (0-based), as [start, end)."""
+    start = index
+    while start > 0:
+        if _OPENER.search(lines[start - 1]):
+            start -= 1
+            break
+        if not lines[start - 1].strip():
+            break
+        start -= 1
+    end = index + 1
+    while end < len(lines):
+        if _CLOSER.match(lines[end]):
+            end += 1
+            break
+        if not lines[end].strip():
+            break
+        end += 1
+    return start, end
+
+
+def _exempt(lines: list[str], start: int, end: int) -> str | None:
+    """The exemption reason for a block, searching the block and its 8-line lead."""
+    lead = max(0, start - 8)
+    for line in lines[lead:end]:
+        if EXEMPT_MARKER in line:
+            reason = line.split(EXEMPT_MARKER, 1)[1].strip().strip(_TOKEN_STRIP).strip()
+            return reason or None
+    return None
+
+
+def _serve_launches(lines: list[str]) -> list[int]:
+    """0-based indices of lines launching the oracle SERVER through its CLI."""
+    found = []
+    for i, line in enumerate(lines):
+        if _is_comment(line):
+            continue
+        tokens = _tokens(line)
+        for j, token in enumerate(tokens):
+            if token != "serve" or j == 0:
+                continue
+            if _ORACLE_CLIENT.match(tokens[j - 1]):
+                found.append(i)
+                break
+    return found
+
+
+def scan(root: Path) -> list[dict]:
+    """Every oracle server launch in the tree, with its verdict."""
+    results: list[dict] = []
+    paths: list[Path] = []
+    for glob in SCAN_GLOBS:
+        paths.extend(sorted(root.glob(glob)))
+    for path in paths:
+        if path.name == SELF:
+            continue
+        try:
+            lines = path.read_text(encoding="utf-8", errors="replace").splitlines()
+        except OSError:
+            continue
+        for index in _serve_launches(lines):
+            start, end = _block(lines, index)
+            body = "\n".join(lines[start:end])
+            results.append(
+                {
+                    "file": str(path.relative_to(root)),
+                    "line": index + 1,
+                    "has_flag": REQUIRED_FLAG in body,
+                    "exempt_reason": _exempt(lines, start, end),
+                }
+            )
+    results.sort(key=lambda r: (r["file"], r["line"]))
+    return results
+
+
+def violations(results: list[dict]) -> list[dict]:
+    return [r for r in results if not r["has_flag"] and not r["exempt_reason"]]
+
+
+def main(argv: list[str] | None = None) -> int:
+    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
+    parser.add_argument("--root", type=Path, default=ROOT)
+    parser.add_argument("--json", action="store_true", help="print the discovered set")
+    args = parser.parse_args(argv)
+
+    results = scan(args.root)
+    if args.json:
+        print(json.dumps(results, indent=2, sort_keys=True))
+
+    if not results:
+        print(
+            "check-oracle-denominator-flags: found NO oracle server launch at all. "
+            "That is a broken detector, not a clean tree -- this repository has at "
+            "least one harness that launches the pinned oracle.",
+            file=sys.stderr,
+        )
+        return 2
+
+    bad = violations(results)
+    if bad:
+        for r in bad:
+            print(
+                f"{r['file']}:{r['line']}: launches the pinned vLLM oracle server "
+                f"without {REQUIRED_FLAG}. The oracle then runs its UNFUSED "
+                f"QK-norm+RoPE+gate path (qwen3_next.py:324-331 at the pin) while "
+                f"our arm runs the fused one, so every ratio from this harness is "
+                f"FLATTERED (#414). Pass the flag, or mark the block "
+                f"'{EXEMPT_MARKER} <reason>' when the architecture has no "
+                f"multimodal_config and the flag decides nothing.",
+                file=sys.stderr,
+            )
+        print(
+            f"\n{len(bad)} handicapped oracle launch(es) of {len(results)} "
+            f"discovered. Never weaken this checker to make a harness pass: the "
+            f"denominator it protects is the whole comparison.",
+            file=sys.stderr,
+        )
+        return 1
+
+    exempt = [r for r in results if r["exempt_reason"]]
+    print(
+        f"check-oracle-denominator-flags: {len(results)} oracle server launch(es), "
+        f"{len(results) - len(exempt)} carrying {REQUIRED_FLAG}, "
+        f"{len(exempt)} exempt."
+    )
+    for r in exempt:
+        print(f"  exempt {r['file']}:{r['line']} -- {r['exempt_reason']}")
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

scripts/dgx-online-serving.sh

@@ -481,6 +481,9 @@ start_server() {
     # backend aborts engine start (BackendSupportedError mm_fp4 cap 121), so we
     # force the Marlin W4A16 keep-quant kernel our arm mirrors. Dense Qwen3 has no
     # mamba SSM state, so the hybrid-only SSM cache-dtype flag is omitted here.
+    # ORACLE-DENOMINATOR-EXEMPT: dense Qwen3-8B has no multimodal_config, so text_only is already True
+    # and the flag decides nothing. multimodal.py:78 never applies, qwen3_next.py:325 already resolves
+    # True, and adding the flag to a measured harness that does not need it is noise, not rigour.
     server_cmd=(
       env "PATH=$(dirname "${client}"):${PATH}"
       "VLLM_DISABLED_KERNELS=FlashInferMxFp4LinearKernel"
@@ -493,10 +496,24 @@ start_server() {
       --port "${port}"
     )
   else
+    # 27 / 27n / 35 are Qwen3.6 checkpoints, which load as
+    # Qwen3_5*ForConditionalGeneration. Their multimodal_config is therefore
+    # non-None, and multimodal.py:78 defaults language_model_only to False, so
+    # without the flag qwen3_next.py:325 computes text_only == False and the
+    # oracle DISABLES use_fused_qk_norm_rope_gate: four ops per full-attention
+    # layer where its own production configuration issues one, against our arm's
+    # single fused launch. That is a handicapped denominator whose cost scales
+    # with prompt tokens, so it lands hardest on TTFT, and it flatters us (#414).
+    # `tools/bench/run_serve_low.py` has passed the flag since it was written;
+    # this driver did not, so the two harnesses disagreed about the oracle's own
+    # configuration. The 2026-08-13 clock-controlled series passed it and is the
+    # binding record; this makes the canonical driver agree with it, and
+    # scripts/check-oracle-denominator-flags.py keeps them agreeing.
     server_cmd=(
       env "PATH=$(dirname "${client}"):${PATH}"
       "${client}" serve "${snapshot}"
       --served-model-name gate
+      --language-model-only
       --gpu-memory-utilization 0.6
       --max-num-seqs "${max_num_seqs}"
       --max-num-batched-tokens "${max_num_batched_tokens}"