From a66eca34df5b0ef842ba56153ee5fc53aa293318 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Wed, 19 Aug 2026 08:22:59 +0000 Subject: [PATCH 1/3] spec(BACKEND-ATTN-REGISTRY): scope the capability surface, and say plainly that a device predicate cannot answer a question about the binary (#1332) The attention-backend selector is a faithful mirror of vLLM's priority walk whose result nothing dispatches on, and the capability layer it is missing is the same layer that passed on a GB10 and then failed at launch. This spec scopes M0 (reconcile) and M1 (`validate_configuration` plus the `supports_*` predicates) of #1332 and records the rest as owed, so that landing M1 cannot be read as fixing selection. M0 found the gap real and unowned: one commit ever names the row, no open pull request touches the selector paths, and the 25 remote branches matching `attn|validate|registry` are all kernel or model-factory work. Both owning rows already exist, so none is opened. Four headers cited `@ pin e24d1b24`, retired at W5. Re-read against a checkout of the tree pin `5559679229`, they carry one content drift rather than a line shift: the non-MLA sm_100 arm gained a `use_non_causal` guard, filed as #1333 and fixed in the same flow because `use_non_causal` is a field M1 adds regardless. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-4-6 [Claude Code] --- .agents/issue-index.md | 2 + .agents/specs/attn-validate-configuration.md | 204 +++++++++++++++++++ 2 files changed, 206 insertions(+) create mode 100644 .agents/specs/attn-validate-configuration.md diff --git a/.agents/issue-index.md b/.agents/issue-index.md index 928abcf7d..b199f3376 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -423,3 +423,5 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#1308](https://github.com/mudler/vllm.cpp/issues/1308) | — | ENV-AGNOSTIC: the campaign's derivation query `git grep -cIE 'dgx\.casa\|nas_share\|192\.168\.\|thor:gpu0'` names a HOST, and a hard-coded default names a PATH, so the query cannot see the defect [#1190](https://github.com/mudler/vllm.cpp/issues/1190) was opened to remove. Measured at `5c8671c50` over `scripts/` and `tools/`, which is `ENV-AGNOSTIC-W1-TOOLING`'s ownership: the four campaign patterns match 19 files and 22 hits, and widening to `/home/mudler`, `~/venvs/vllm-oracle`, `~/work/vllm.cpp`, `cutlass-4.5.0` and `cutlass_probe` matches 73 files and 174 hits. Twelve of those lines in six shell scripts are the exact `${KEY:-}` shape the campaign's worked example removed from `scripts/dgx-bringup.sh`, and `scripts/upstream-inventory.py:38-40` carries the Python form `os.environ.get("VLLM_SOURCE", str(Path.home() / "_git/vllm"))`. `ENV-AGNOSTIC-W1-TOOLING` converts `scripts/regen-triton-aot.sh`, because the campaign row wrote that one debt into the file's own text at `:20-23` rather than into a count, and leaves the rest: two of them read as provenance on a first pass and must not be swept blind, since `scripts/cpu-x86-llamacpp-floor.sh:33` states that "every recorded leg used these defaults verbatim" and `scripts/dgx-gdn-packed-bridge-ab.sh:4` dates its prerequisites, which is the same instrument-identity argument that keeps `scripts/mtp-k-gt-1-neartie-gap.py` literal. The blind spot is campaign-level: `ENV-AGNOSTIC-W3-CODE` and `ENV-AGNOSTIC-W4-RECORDS` derive their sets from the same query, so the campaign's claim that its five waves partition 227 files is a claim about the query's 227 and not about the tree. Owed by [`env-agnostic-w1-tooling.md`](specs/env-agnostic-w1-tooling.md) under `## Owed` | bug | | [#1316](https://github.com/mudler/vllm.cpp/issues/1316) | — | `scripts/main-baseline.py` renders a scheduled run that executed ZERO jobs as `RED` with all 11 covered jobs `missing`, so `NEWEST BASELINE: RED at ` names a tree the run never checked out. Measured at `origin/main` `250db75a2`: runs `32206456661` and `32140419182` both return `startedAt: null` for every job, because GitHub cancelled them while they were pending in the single `ci-schedule-refs/heads/main-mudler/vllm.cpp` group, whose queue holds one run ([#274](https://github.com/mudler/vllm.cpp/issues/274)). Fail-closed, and the `missing (expected, never ran)` line is accurate about the jobs; the defect is the verdict word, because a run that executed nothing is NOT RUN rather than RED, and the newest verdict should fall through to the newest run that actually ran. NOT fixed in flow: `test_an_expected_job_the_payload_never_mentions_is_red` and `test_a_narrowed_run_reports_red_and_names_what_never_ran` deliberately assert missing-is-red so a narrowed run cannot pass, and separating "narrowed" from "never started" changes what the verdict means, which owes its own spec, red-before evidence and a fresh reviewer. Owed under `## Owed` of [baseline-lane-eviction.md](specs/baseline-lane-eviction.md), which removes the only observed producer of a zero-job run | bug | | [#1314](https://github.com/mudler/vllm.cpp/issues/1314) | `SPEC-DFLASH2` | **DFlash2 (`DFlash2DraftModel`) is unported**, and one config rule would run the published checkpoint wrong in silence. Upstream carries DFlash2 as a SECOND architecture beside DFlash rather than as a change to it ([vllm#52816](https://github.com/vllm-project/vllm/pull/52816), OPEN at head `19c9351904df4c63042671bc67a866ca48dc7d6f`, base `9842d701`, 755+/5-, 11 files, plus the stacked guard fix [vllm#52883](https://github.com/vllm-project/vllm/pull/52883)): DFlash1 gains two subclass seams and keeps every behaviour, and the new architecture adds a GROUPED DYNAMIC DEPTHWISE CONVOLUTION around each attention and each MLP sublayer plus a CANDIDATE SELECTOR that replaces the independent per-slot argmax with a scored path walk over the target head's top-K. Shapes taken from the published checkpoint rather than from the diff: `z-lab/Qwen3.8-27B-DFlash2`, safetensors header range-read 2026-08-19, 81 tensors -- DFlash1's set plus `layers.N.{attention,mlp}_conv.{base_kernel (2,2,5120), kernel_projection.weight (1280,5120)}` x5 and `candidate_selector.{hidden_projection.weight (256,5120), predecessor_codebook, successor_codebook}` at `(248320,256)` bf16 each, ~254 MB resident the DFlash1 lane never allocates; `conv_kernel_size 2`, `conv_group_size 16`, `selector_rank 256`, `selector_top_k 16`, `block_size 8`. **The silent one:** that config declares all five layers `sliding_attention` AND `is_causal false`, while our resolution mirrors the OLD upstream rule (causal iff SWA, unless `dflash_config.causal`, `include/vllm/model_executor/models/qwen3_dflash.h:22-24`), so every layer would run CAUSAL -- plausible tokens, a token gate against our own output sees nothing, and only ACCEPTANCE moves, which the lossless verify hides. Upstream changes `_dflash_layer_causal` to read `is_causal` first, in the same commit. Three further things are owed and none of them is silent: no route for the `DFlash2DraftModel` architecture string (the same classification code as the open `DSparkDraftModel` gap, [#1193](https://github.com/mudler/vllm.cpp/issues/1193)); no top-k that EMITS the surviving (id, value) pairs, where the decision is to extend the sort-free pivot-bracket threshold search already ported from the same FlashInfer approach at `src/vt/cuda/cuda_sample.cu:297-506` rather than port FlashInfer's 3380-line general radix kernel; and the path walk must run ON DEVICE from the first landing, because the identical sequential shape in DSpark shipped host-side and measured 28% of the 27B draft step ([#436](https://github.com/mudler/vllm.cpp/issues/436)) before `SampleSequentialDevice` moved it. Already reusable unchanged: `vt::DFlashBlockAttention`, the DFlash runner/rejection/GDN-rollback lane, and the loader's target-shared `embed_tokens`+`lm_head`, which is already what a DFlash2 checkpoint needs. BEYOND-PIN by developer decision 2026-08-19 (mirror the open PR now, reconcile if review moves it), in the same posture `SPEC-DSPARK-QWEN3-ROUTING` takes toward vllm#52197; the parity pin `555967922` does not carry the architecture and is NOT advanced. Gate arm is bf16 27B plus the GGUF drafter arm in the same wave, oracle = vLLM built at the PR head, acceptance measured SAME-TRAJECTORY because `SPEC-DFLASH` D8 spent a campaign on a divergent-trajectory confound that D9 refuted. Spec [dflash2-spec-decode.md](specs/dflash2-spec-decode.md) | feature | +| [#1332](https://github.com/mudler/vllm.cpp/issues/1332) | `BACKEND-ATTN-REGISTRY` | The attention-backend selector routes nothing, and a capability check is not a runnability check. `vllm::v1::SelectAttentionBackendName` (`src/vllm/v1/attention/registry.cpp`) mirrors vLLM's priority walk faithfully, and its result reaches only `attn_backend_names_`, a `VT_ATTN_SELECT_LOG` print and `CheckKvCacheShape` in `runner.cpp::GpuModelRunner::InitializeKvCache`; `dense_attn::AttnBlock` calls `vt::PagedAttention` unconditionally and the real arm choice is an env-flag + shape + dtype ladder in `src/vt/cuda/cuda_paged_attn.cu`. Deleting the whole selector would leave every emitted token identical, which is `.agents/reachability.md`'s "unselected branch" and "unpassed parameter" at once. Separately, measured on a GB10 (capability 12,1): vLLM selects `FLASH_ATTN` because `supports_compute_capability` is `capability >= (8,0)`, while the shipped FA2 binary carries `sm_80` SASS plus `compute_80` PTX alone, so every launch needs a driver JIT that fails with `cudaErrorUnsupportedPtxVersion` — `grep -rn get_arch_list vllm/` returns zero hits, so vLLM never asks what its own fatbins contain. We have written the same class of check: `src/vllm/platforms/cuda.cpp::CudaPlatform::supports_fa2_attention` returns true for every CUDA device while `CMakeLists.txt` defaults `VLLM_CPP_CUDA_ARCHITECTURES` to `121a` alone. The invariant: no backend may be declared valid on the strength of a property of the DEVICE alone; every predicate naming a compute capability must be paired with one naming the binary. M0 (reconcile) and M1 (the `validate_configuration` capability surface) land in [`attn-validate-configuration.md`](specs/attn-validate-configuration.md); the compiled-arch manifest, the launch probe, the dispatch wiring and the ABI override remain owed there under `## Owed` | bug | +| [#1333](https://github.com/mudler/vllm.cpp/issues/1333) | `BACKEND-ATTN-REGISTRY` | The non-MLA sm_100 attention priority row misses the `use_non_causal` guard the pin added. `include/vllm/platforms/cuda_attn_priority.h::AttnPriorityTable` keys rows on `(use_mla, major)` and cites `vllm/platforms/cuda.py:143-166` `@ pin e24d1b24`; at the tree pin `5559679229` that arm reads `if device_capability.major == 10 and not use_non_causal` (`cuda.py:148`) with `use_non_causal` added as a fifth `_get_backend_priorities` parameter (`cuda.py:88`), because SM100f's non-causal cutlass path — the one DFlash attention uses — is known-bad (`cuda.py:145-147`). No selected name changes in this tree today, since `FLASHINFER` is registered for no device and both orderings fall through to `FLASH_ATTN`; it becomes a real divergence the moment a FlashInfer backend registers, and it is already a divergence in the record because the header claims a faithful, complete port of that function. Found while reconciling the four `@ pin e24d1b24` header anchors under [#1332](https://github.com/mudler/vllm.cpp/issues/1332) M0 and fixed in the same flow, since `use_non_causal` is a field M1 adds to `AttnSelectorConfig` regardless. Spec [`attn-validate-configuration.md`](specs/attn-validate-configuration.md) | bug | diff --git a/.agents/specs/attn-validate-configuration.md b/.agents/specs/attn-validate-configuration.md new file mode 100644 index 000000000..b00f48729 --- /dev/null +++ b/.agents/specs/attn-validate-configuration.md @@ -0,0 +1,204 @@ +# SPEC — `validate_configuration` and the `supports_*` capability surface + +Issue: [#1332](https://github.com/mudler/vllm.cpp/issues/1332) +Owning rows: `BACKEND-ATTN-REGISTRY` (`.agents/backend-matrix.md`, `ANCHOR-BACKFILL`, +claim `CLAIM-ATTN-REGISTRY-1`) owns the selector seam; `BACKEND-CUDA-COMP-FA` +(`PARTIAL`) owns the FA2 arms. +Scope of this change: **M0 and M1 of #1332 only.** + +## Now + +`BACKEND-ATTN-REGISTRY` stays `ANCHOR-BACKFILL`. This change adds the declared +capability layer the row recorded as deferred and re-anchors four headers onto +the tree pin. It does not move the row's lifecycle state, because the row's +open item — that the resolved name does not drive dispatch — is untouched and is +owed to M4. + +## M0 — reconciliation, performed before any code + +Checked at `origin/main` `f06b9e93d`: + +- `git log --oneline --grep 'BACKEND-ATTN-REGISTRY'` returns one commit, + `b3a4ca963` (the seam itself). No later landing. +- `git log -S'validate_configuration' --oneline` returns six commits, every one + of them a spec, audit or spike that *names* the upstream method. `git log + -S'supports_compute_capability'` returns two, both records-only spikes. The + capability surface has never been implemented. +- `gh pr list --state open --limit 100`: 24 open pull requests, none touching + `src/vllm/v1/attention/**`, `include/vllm/v1/attention/**` or + `include/vllm/platforms/**`. +- `git branch -r | grep -iE 'attn|validate|registry'`: 25 remote branches match, + every one of them a kernel or model-factory branch (`perf/attn-prefill-*`, + `opt/decode-paged-attn`, `wmma-attn`, `row/MODEL-FACTORY-registry`, + `row/KERNEL-ATTN-DECODE-D128`, `row/PERF-CPU-ATTN-DTYPE`). None is a selector + branch. +- `.agents/NOW.md` names no attention-selector gate. + +**Outcome: the gap is real, unowned by any branch in flight, and both rows that +own it already exist.** No new roadmap row is opened. + +## Owed — recorded debt this change does NOT close + +1. **The selector still routes nothing.** `SelectAttentionBackendName`'s result + reaches exactly three places in + `src/vllm/v1/worker/gpu/runner.cpp::GpuModelRunner::InitializeKvCache`: + `attn_backend_names_` (`:1002`), the `VT_ATTN_SELECT_LOG` debug print + (`:1003-1024`), and `vllm::v1::CheckKvCacheShape` (`:1025-1029`). + `dense_attn::AttnBlock` calls `vt::PagedAttention` unconditionally + (`include/vllm/model_executor/models/dense_attn_block.h:532-536`) and + `AttentionBackend::get_impl_cls()` returns `nullptr` for every dense backend. + The real arm choice is the env-flag + shape + dtype ladder in + `src/vt/cuda/cuda_paged_attn.cu:2696-2848`. Deleting the selector today would + leave every emitted token identical. **Owner: #1332 M4, row + `BACKEND-ATTN-REGISTRY`.** +2. **This layer is necessary and NOT sufficient.** M1 reproduces upstream's + *declared* capability layer, and that layer is precisely what passed on a + GB10 (capability 12,1) and then failed at launch: upstream's + `FlashAttentionBackend.supports_compute_capability` is `capability >= (8,0)` + (`vllm/v1/attention/backends/flash_attn.py:200-202` @ `5559679229`) while the + shipped fatbin carries `sm_80` SASS plus `compute_80` PTX and nothing else, so + every launch needs a driver JIT that fails with + `cudaErrorUnsupportedPtxVersion`. **A predicate over the DEVICE cannot answer a + question about the BINARY.** Landing M1 while claiming it fixes selection would + be the defect #1332 exists to correct. The compiled-arch manifest (M2) and the + launch probe (M3) are what make the layer sound. **Owner: #1332 M2/M3, row + `BACKEND-CUDA-COMP-FA`.** +3. **`AttnSelectorConfig::dtype` is not supplied by the runner.** The production + call site fills `head_size`, `block_size` and `kv_cache_dtype` from the + resolved per-layer KV geometry, but the model/query dtype is not available at + `InitializeKvCache` (the runner resolves only `ResolveKvCacheDType()` there), + so `dtype` keeps its `kBF16` default in production and the `supports_dtype` + predicate is exercised by tests alone. **Owner: #1332 M4.** +4. **The non-MLA sm_100 priority arm gained a `use_non_causal` guard upstream** + between the two pins; see "Anchor reconciliation" below. Fixed in flow, tracked + by its own issue, because the anchor being corrected points straight at it. + +## Anchor reconciliation + +Verified against a checkout of the tree pin. `git rev-parse HEAD` in that +checkout returns `5559679229bc961848b121ccdeaa8fa5d79bec98`, which equals the +`vllm_commit` in the `parity-pin` block of `.agents/upstream-sync.md`. Four +headers cited `@ pin e24d1b24`, the pin retired at W5 (`bc415a3e`). + +| File | Cited at `e24d1b24` | Verified at `5559679229` | +|---|---|---| +| `include/vllm/v1/attention/backend.h:1` | `vllm/v1/attention/backend.py` | same file; `validate_configuration` is `:320-393`, the `supports_*` predicates `:154-317` | +| `include/vllm/v1/attention/registry.h:3` | `cuda.py:361-470` (`get_valid_backends` / `get_attn_backend_cls`) | `get_valid_backends` is `:359-394`, `get_attn_backend_cls` is `:397-492` | +| `include/vllm/platforms/cuda_attn_priority.h:2` | `cuda.py:84-176` `_get_backend_priorities`, MLA `:93-142`, non-MLA `:143-166`; `mla/prefill/selector.py:47-76` | `_get_backend_priorities` is `:83-163`, MLA `:93-143`, non-MLA `:144-163`; `selector.py:48-77` | +| `include/vllm/platforms/interface.h:2` | `interface.py:134-229` (`class Platform`) | `class Platform` is `:134-1290`; the cited `:409-439` capability accessors and `:181-187` `supported_dtypes` moved | + +**One content drift, not just a line shift.** At `e24d1b24` the non-MLA sm_100 +arm was `if device_capability.major == 10`. At `5559679229` it is +`if device_capability.major == 10 and not use_non_causal` (`cuda.py:148`), with +`use_non_causal` added as a fifth `_get_backend_priorities` parameter +(`cuda.py:88`); the comment at `:145-147` gives the reason — SM100f's non-causal +CUTLASS path is known-bad for DFlash. Our table keyed rows on `(use_mla, major)` +alone, so a non-causal request on sm_100 got upstream's causal ordering. Today +this changes no selected name in this tree, because `FLASHINFER` is not +registered and both orderings fall through to `FLASH_ATTN`; it would change one +the moment FlashInfer lands. Fixed here rather than deferred: `use_non_causal` is +a field this change adds to `AttnSelectorConfig` regardless, the fix is one row +plus one predicate in the same header whose anchor is being corrected, and it is +CPU-gateable through the existing `FakeCudaPlatform`. + +The MLA rows, the sm_12x two-entry row, and the MLA-prefill lists are unchanged +in content at the new pin and were re-read line by line to confirm it. + +**Not reconciled here, recorded instead.** `FlashAttentionBackend::get_kv_cache_shape` +mirrors `flash_attn.py::get_kv_cache_shape` at `e24d1b24` +(`num_blocks, 2, block_size, num_kv_heads, head_size`); at `5559679229` upstream +returns `(num_blocks, num_kv_heads, block_size, 2 * head_size)`. That is the KV +memory format the whole engine allocates and every paged-attention kernel reads, +so re-anchoring it is a kernel campaign, not a comment edit. Left at its +`e24d1b24` anchor with the divergence named in the header. Owner: +`BACKEND-ATTN-REGISTRY`. + +## Design + +Mirror upstream's structure exactly: predicates are virtual member functions +returning `bool`, `validate_configuration` collects reason strings, and an empty +list means valid. + +**Where the config lives.** Upstream's `AttentionSelectorConfig` +(`vllm/v1/attention/selector.py:24-39`) is one named tuple that feeds both +`_get_backend_priorities` and `validate_configuration`. Our equivalent is +`vllm::platforms::AttnSelectorConfig` (`include/vllm/platforms/interface.h`), +already threaded through `Platform::get_attn_backend_priority`. It gains the +remaining upstream fields, every one with upstream's default, so no existing call +site changes meaning. + +**`attn_type` is a string.** Upstream's `AttentionType` is a `str` enum and +`supports_attn_type` takes a `str` (`backend.py:292-298`). Our `AttentionType` is +an `enum class` in `vllm::v1`, and `platforms/` must not depend on `v1/`. The +config therefore carries upstream's string, which is the faithful shape, and +`backend.h` gains `AttentionTypeName()` to convert. + +**Compute capability is checked only when the platform reports one.** Upstream +runs `validate_configuration` from `CudaPlatform.get_attn_backend_cls`, which +asserts `device_capability is not None` first (`cuda.py:403-404`); `CpuPlatform` +has its own selector (`cpu.py:75-87`) and never reaches this code. Our selector +is shared across every `DeviceType`, and `DeviceCapability::present()` is already +false (`major == -1`) for every non-CUDA platform. The predicate is therefore +applied only when `present()` is true, which reproduces upstream's precondition +rather than inventing a new rule. Recorded because it is the one structural +adaptation in this change. + +**The refusal names what failed.** `SelectAttentionBackendName` keeps its two +paths. An explicit override that fails validation throws with the full reason +list, mirroring `cuda.py:416-420`. The priority walk skips an invalid candidate +and, on exhaustion, throws naming each candidate with its reasons, mirroring +`cuda.py:432-446`. + +**Per-backend overrides ported.** `FlashAttentionBackend` gets upstream's +`supported_kv_cache_dtypes`, `get_supported_kernel_block_sizes` (`MultipleOf(16)`), +`supports_head_size` (`% 8 == 0 && <= 256`; upstream's FA4 `<= 512` arm is +unreachable here and says so), `supports_compute_capability` (`>= 8.0`), +`supports_sliding_window`, `supports_batch_invariance`, `supports_non_causal`, +`supports_attn_type` (all four types) and the `supports_combination` sink rule. +`TritonMLABackend` keeps `MultipleOf(16)` and its existing `is_mla()`. + +**One deliberate divergence from upstream's per-backend data.** +`FlashAttentionBackend::supports_per_head_quant_scales()` is upstream's +`fa_version >= 3`; this tree ships FA2 only (`CMakeLists.txt` +`VLLM_CPP_CUDA_ARCHITECTURES`, `BACKEND-CUDA-COMP-FA` records FA3 as unported), +so it returns `false` — which is upstream's own answer for FA2 and is what makes +the ported `test_per_head_quant_scales_backend_selection` case assert a refusal. + +## Tests + +`tests/vllm/v1/attention/test_attn_validate_configuration.cpp`, a port of +`tests/kernels/attention/test_attention_selector.py` @ `5559679229`. That file +monkeypatches the device capability and asserts a chosen name, so every case +selected below runs on CPU against the existing `FakeCudaPlatform`, which takes +its capability by constructor argument. + +| Upstream case | Ported as | +|---|---| +| `test_backend_selection` (cuda/cpu arms) | selection under a fully populated config still resolves `FLASH_ATTN` on sm_121 and sm_100 | +| `test_fp32_fallback` | an `f32` request finds no valid backend and the refusal names `dtype not supported`; upstream lands on `FLEX_ATTENTION`, which this tree does not register | +| `test_flash_attn` (upstream `pytest.skip`s it) | its five assertions ported as direct predicate cases: capability `(7,5)`, dtype `fp8`, `kv_cache_dtype="fp8"`, `block_size=8`, `head_size=17` | +| `test_per_head_quant_scales_backend_selection` | `FLASH_ATTN` + `use_per_head_quant_scales` is refused and the reason names it | +| `test_non_causal_backend_selection`, `test_non_causal_autoselect_backend` | `FLASH_ATTN` advertises `supports_non_causal()` and a non-causal request selects it | +| `test_flash_attn_rejects_unhandled_kv_cache_dtypes` | all six upstream parameters asserted false | +| `test_flash_attn_accepts_handled_fp8_variants` | both upstream parameters asserted true | +| `test_invalid_backend`, `test_auto_backend_string` | not applicable: our override is a `std::string` with `""` for automatic, so there is no enum lookup to fail | + +Plus, beyond upstream: the reason list is asserted to be a LIST — a request that +violates three predicates at once produces three reasons, so a refusal message +cannot collapse to the first failure; the sm_100 `use_non_causal` ordering; and +`AttnSelectorConfig{}` (every default) still validates, which is the +behavior-preserving control for the existing production call sites. + +## Gates + +`scripts/agent-preflight.sh --fail-on-skip`, plus the focused +`test_attn_validate_configuration` and `test_attn_backend_registry` cases. No GPU: +every predicate is host code and the capability is injected. + +## Stop conditions + +- Stop and report `NEEDS_DECISION` if a ported predicate would refuse a + configuration the gate models use today, because that is a behavior change, not + a capability port. +- Stop if `validate_configuration` cannot be reached from the production call + site without also wiring dispatch; wiring dispatch is M4 and out of scope. From bb5ce1f3a6486e1e5b105c8f2aebab646a24eb55 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Wed, 19 Aug 2026 09:07:19 +0000 Subject: [PATCH 2/3] feat(BACKEND-ATTN-REGISTRY): port validate_configuration, and say where a declared capability stops being an answer (#1332) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `vllm/v1/attention/backend.py`'s `supports_*` predicates and the `validate_configuration` that collects them were recorded as deferred at `backend.h:26-29`, so the selector asked two questions inline — `is_mla` and `is_sparse` — and had no way to refuse a backend for a head size, a block size, a dtype or a compute capability it cannot serve. This ports the surface as upstream shapes it: predicates returning bool, one reason string per failure, and an empty list meaning valid. The reason strings are upstream's byte for byte, so a refusal here and a refusal from the oracle read the same. The refusal is a LIST, not a first failure. A request that violates four predicates at once produces four reasons, because a message that stopped at the first would send a reader chasing one cause of four, and because upstream prints the whole list at `cuda.py:416-420` and `:432-446`. `AttnSelectorConfig` grows into the full `AttentionSelectorConfig` (`selector.py:24-39`) with upstream's defaults, so every existing caller keeps its exact selection behavior; `initialize_kv_cache` fills the head size, block size and KV-cache dtype it has just resolved. `TritonMLABackend`'s `static supports_block_size` becomes the supported-size LIST, because a static of that name would have hidden the new base virtual and answered differently through a base reference than through the concrete type. READ THE SPEC BEFORE READING THIS AS A FIX FOR SELECTION. It is not one. Every predicate here describes what a backend CLAIMS, and none can see what the shipped binary contains: upstream's `supports_compute_capability` is `>= (8,0)`, it returned true on a GB10 whose FA2 fatbin holds sm_80 SASS and compute_80 PTX alone, and every launch then failed a driver JIT. That is the failure #1332 was opened for. This layer is necessary and NOT sufficient; the compiled-arch manifest (M2) and the launch probe (M3) are what make it sound, and the selected name still dispatches nothing because `dense_attn::AttnBlock` calls `vt::PagedAttention` unconditionally (M4). All three are named under `## Owed`. `docs/USAGE.md` says the same thing to a user, beside the refusal message. Four headers cited the pin retired at W5 and are re-anchored onto `5559679229`, which surfaced one content drift rather than a line shift: the non-MLA sm_100 priority arm gained a `use_non_causal` guard, fixed here as #1333. `FlashAttentionBackend::get_kv_cache_shape` is deliberately left at its `e24d1b24` anchor with the divergence named, because that layout is what the whole engine allocates. `kernel-matrix.md`'s `KERNEL-ATTN-MLA-SPARSE` cell is repaired for the lines this change moved. Ports tests/kernels/attention/test_attention_selector.py @ 5559679229. Red first at 12 cases / 10 failed; green at 21 cases / 76 assertions, with `test_attn_backend_registry` 17/62 and `test_runner` 19/543 unmoved. Six mutations, each with its `compile_rc` and its restored `git diff --stat`; corrupting the head size the runner feeds the selector fails all 19 `test_runner` cases, which is the reachability proof, and deleting the fill outright leaves them green, which is the honest negative and is recorded as such. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-4-6 [Claude Code] --- .agents/kernel-matrix.md | 2 +- .agents/specs/attn-validate-configuration.md | 59 ++- docs/USAGE.md | 31 ++ include/vllm/platforms/cuda_attn_priority.h | 33 +- include/vllm/platforms/interface.h | 71 ++- include/vllm/v1/attention/backend.h | 196 ++++++- include/vllm/v1/attention/registry.h | 6 +- src/vllm/v1/attention/backend.cpp | 163 +++++- src/vllm/v1/attention/registry.cpp | 71 ++- src/vllm/v1/worker/gpu/runner.cpp | 24 +- tests/CMakeLists.txt | 2 + .../test_attn_validate_configuration.cpp | 481 ++++++++++++++++++ 12 files changed, 1075 insertions(+), 64 deletions(-) create mode 100644 tests/vllm/v1/attention/test_attn_validate_configuration.cpp diff --git a/.agents/kernel-matrix.md b/.agents/kernel-matrix.md index b90469977..c4e96c5a5 100644 --- a/.agents/kernel-matrix.md +++ b/.agents/kernel-matrix.md @@ -137,7 +137,7 @@ host/sched. Detail: state `KERNEL-FA2-GQA-SWAP-FLIP`. | `KERNEL-ATTN-FA3-FA4` | FlashAttention-3/4 | FA3/FA4 build `setup.py:1113-1124`; runtime version selection `vllm/v1/attention/backends/fa_utils.py:132-250` | - | - | [inventory](specs/kernel-family-inventory.md) | `INVENTORIED` | - | | `KERNEL-ATTN-FLASHINFER-TRTLLM` | FlashInfer paged attention, XQA, and TensorRT-LLM generation kernels | availability/dispatch `vllm/utils/flashinfer.py:206-342,373-511`; SM90 decode and SM100+ prefill/decode gates | - | - | [inventory](specs/kernel-family-inventory.md) | `INVENTORIED` | - | | `KERNEL-ATTN-TRITON-FLEX-HPC` | Triton, FlexAttention, TurboQuant, and HPC attention families | registry `registry.py:34-120`; Triton contract `triton_attn.py:250-383`; HPC gate `hpc_attn.py:235-247` | - | - | [inventory](specs/kernel-family-inventory.md) | `INVENTORIED` | - | -| `KERNEL-ATTN-MLA-SPARSE` | MLA and sparse attention: CUTLASS, FlashMLA, FlashInfer, FA, Triton, MSA **W6: the MLA attention BLOCK + WEIGHT ABSORPTION — the layer that COMPOSES W3+W4+W5** — [mla_attention.h](../include/vllm/model_executor/models/mla_attention.h) + [mla_attention.cpp](../src/vllm/model_executor/layers/attention/mla_attention.cpp) <- `mla.py:119-181` (`MultiHeadLatentAttentionWrapper.forward`) over `mla_attention.py:553-620` (the cache-update-BEFORE-attention order), `:624-874` (`forward_impl`: the dispatch + the absorbed decode) and `:2344-2425` (`forward_mha`); `AbsorbKvBProjBf16` <- `:875-962 process_weights_after_loading` (split `:892-900`, permutes `:959-962`); `MakeMlaUpProjectFn` <- `:2141-2170` (the `kv_b_proj` callback W5 left open); `BuildDeepseekRopeCosSinCache` <- `deepseek_scaling_rope.py:76-118` over `rotary_embedding/common.py:34-70`; `MlaAttentionScale` <- `deepseek_v2.py:995,1067-1075` (the mscale^2 correction, kept SEPARATE from the rope's own rotation mscale). **Absorption is a LOAD-TIME weight transform plus TWO batched GEMMs, not a fused kernel**, so it needed only two new general primitives: **`vt::BatchedMatmul`** <- `torch.bmm` at `mla_attention.py:789` (q-side W_UK fold) and `:1034` (`_v_up_proj`), CUDA impl = cuBLASLt STRIDED-BATCHED [cuda_matmul.cu](../src/vt/cuda/cuda_matmul.cu) (the cuBLASLt form of the cuBLAS `gemmStridedBatchedEx` torch.bmm resolves to; the only upstream alternatives are ROCm-only aiter fp8/fp4 bmm branches) + CPU ref [cpu_ops.cpp](../src/vt/cpu/cpu_ops.cpp), stride-driven because BOTH call sites pass `.transpose(0,1)` views; and **`vt::ConcatMlaNopeRope`** <- `ConcatMLAQKernel` (`csrc/libtorch_stable/concat_mla_q.cuh`) + wrapper `cache_kernels.cu:1555-1600`, GENERALIZED to arbitrary nope/rope widths and a head-BROADCAST rope operand so one op also serves `_concat_k_nope_k_pe` (`:2063-2092`) — CUDA [cuda_mla_attn.cu](../src/vt/cuda/cuda_mla_attn.cu), CPU [cpu_ops.cpp](../src/vt/cpu/cpu_ops.cpp). Two ADDITIVE relaxations of existing ops, integer-identical for contiguous tensors: `vt::RopeFromCache` stride-driven on q/k (DeepSeek rotates the TRAILING 64-dim slice and its `k_pe` is a column block of the fused kv_a projection) and `vt::MatmulBT` accepting a row-strided ACTIVATION (`kv_b_proj` applied to a 512-column slice of the 576-wide workspace, `:2160`) | CUDA priority `vllm/platforms/cuda.py:84-176` (`_get_backend_priorities`, both branches); MLA classes `vllm/v1/attention/backends/mla/*.py`; MLA prefill selector `mla/prefill/selector.py:47-76`; capability filter `vllm/v1/attention/backend.py:307-360`; CUTLASS build `CMakeLists.txt:1037-1061` **W6** [test_mla_attention_block.cpp](../tests/vllm/model_executor/layers/attention/test_mla_attention_block.cpp) **10/10 cases / 2,372,644 assertions** and [test_ops_mla_absorb.cpp](../tests/vt/test_ops_mla_absorb.cpp) **9/9 / 1,644,807 assertions** on dgx sm_121 — ports of `tests/kernels/test_concat_mla_q.py` (BOTH arms incl. the NON-CONTIGUOUS transposed-nope case, compared bit-exactly since a concat is a pure copy), the MLA-geometry sweep of `tests/v1/attention/test_mla_backends.py`, and the two-pass-oracle discipline of `tests/kernels/attention/test_mla_decode_cpu.py`. **THE ABSORBED-vs-UNABSORBED EQUIVALENCE IS PROVEN NUMERICALLY, THREE WAYS:** an INDEPENDENT double-precision block oracle computing the attention BOTH ways agrees to **< 1e-11** (the identity itself, at both query branches); our absorbed decode reproduces the UNABSORBED oracle to **< 2e-4** (f32); and the SAME batch driven once through the ABSORBED MQA decode kernel and once through the UNABSORBED materialized-MHA prefill path agrees to **< 3e-4** (CPU f32) / **< 4e-2** (CUDA bf16) — two code paths sharing nothing but the weights. Real geometry throughout (V2-Lite 512/128/64/128/16-head, plus V3's 7168 / 128-head / `q_lora_rank=1536` for the lora branch, which has NO e2e coverage and says so). Decode-only / prefill-only-no-context / chunked-prefill-with-context / MIXED (decode packed FIRST) all gated; NaN-poisoned outputs; run-to-run BIT-exact; CUDA cases proven to EXECUTE (124,941 + 290,835 assertions when run alone). memcheck **0 errors**, racecheck **0 hazards**, synccheck **0 errors** (the last requires `--num-cuda-barriers 65536`: the default table OVERFLOWS on a binary driving this many kernel families and the tool then emits a bogus `unspecified launch failure`). Clean CUDA build 0 warn/0 err; regression set UNCHANGED (27B 235/235, 35B 315/315, Coder 138/138, Qwen3-dense 664/664, OPT 36/36). No speed number — W9 owns tuning | **SELECTION (W2) + the DECODE KERNEL (W4) + the PREFILL PATH and CHUNKED-CONTEXT LOOP (W5). What is still absent is the MLA attention BLOCK and MODEL (W6/W7).** Priority TABLE [cuda_attn_priority.h:49](../include/vllm/platforms/cuda_attn_priority.h#L49) (both branches, one row per upstream arch arm) + lookup [cuda_attn_priority.h:86](../include/vllm/platforms/cuda_attn_priority.h#L86); `is_mla()`/`is_sparse()` filter [registry.cpp:63](../src/vllm/v1/attention/registry.cpp#L63); `TritonMLABackend` NAME + 3-D `get_kv_cache_shape` [backend.h:271](../include/vllm/v1/attention/backend.h#L271), [backend.cpp:83](../src/vllm/v1/attention/backend.cpp#L83), registration [backend.cpp:108](../src/vllm/v1/attention/backend.cpp#L108) — **W4: `vt::MlaDecodeAttention`** — `OpId::kMlaDecodeAttention` + args/validation [ops.h](../include/vt/ops.h), [ops.cpp](../src/vt/ops.cpp); CPU single-pass REFERENCE [cpu_mla_attn.cpp](../src/vt/cpu/cpu_mla_attn.cpp) (numerics from `csrc/cpu/mla_decode.cpp`); CUDA two-stage split-KV [cuda_mla_attn.cu](../src/vt/cuda/cuda_mla_attn.cu) — `MlaDecodeStage1` <- `_fwd_grouped_kernel_stage1` (`triton_decode_attention.py:278-458`, IS_MLA `v = tl.trans(k)` branch `:424-431`), `MlaDecodeStage2` <- `_fwd_kernel_stage2` (`:575-639`), `ComputeNumKvSplits` <- `_compute_num_kv_splits` (`triton_mla.py:40-47`), split workspace via the house grow-only per-stream scratch (upstream's `_reserve_attn_logits_workspace` `:57-78`). Deterministic by construction: fixed ASCENDING split merge, NO atomicAdd. `TritonMLABackend::get_impl_cls()` now returns a real `TritonMLAImpl` [backend.h](../include/vllm/v1/attention/backend.h), [backend.cpp](../src/vllm/v1/attention/backend.cpp); PREFILL remains W5 and `TritonMLAImpl::forward` refuses a prefill-shaped batch by name. **W5: `vt::MlaPrefillAttention` + `vt::GatherMlaCache` + `vt::MergeAttnStates` + the chunked-context driver** — `vt::MlaPrefillAttention` [cuda_mla_prefill.cu](../src/vt/cuda/cuda_mla_prefill.cu) / CPU ref [cpu_mla_prefill.cpp](../src/vt/cpu/cpu_mla_prefill.cpp) <- `mla/prefill/flash_attn.py:153-248` `FlashAttnPrefillBackend` (the ONLY MLA prefill backend reachable on sm_121 per `mla/prefill/selector.py:66-76`, and it HARD-RAISES with no fallback at `:191-194`), running over the vendored FA-2 through the NEW launcher entry `LaunchMlaPrefillFA2Bf16` [cuda_flash_attn_fa2.cu](../src/vt/cuda/cuda_flash_attn_fa2.cu) plus two new explicit instantiations of the UNCHANGED generic template (`flash_fwd_split_hdim192_bf16{,_causal}_sm80.cu`). V is ZERO-PADDED 128->192 and the output sliced back, exactly as upstream's `requires_v_padding` path does (`flash_attn.py:88-99,164-168,196-197`) — which is WHY the asymmetric QK 192 / V 128 pair needs no asymmetric kernel. `vt::GatherMlaCache` <- `csrc/libtorch_stable/cache_kernels.cu:992-1064`; `vt::MergeAttnStates` <- `csrc/libtorch_stable/attention/merge_attn_states.cu:18-192` (BOTH `-inf` edge cases ported verbatim). The workspace-bounded loop is [mla_chunked_context.h](../include/vllm/model_executor/layers/attention/mla_chunked_context.h) <- `mla_attention.py:1422-1451,1667-1745,2094-2199,2344-2425`. **The paged launcher `LaunchPrefillFA2Bf16` that every non-MLA prefill calls is textually UNTOUCHED** (211 insertions / 0 deletions in that TU; 2 new vendored files) | [test_attn_backend_registry.cpp:146](../tests/vllm/v1/attention/test_attn_backend_registry.cpp#L146) (GB10 MLA list), [:203](../tests/vllm/v1/attention/test_attn_backend_registry.cpp#L203) (`use_mla=true` -> `TRITON_MLA`, matching the W0 oracle observation), [:230](../tests/vllm/v1/attention/test_attn_backend_registry.cpp#L230) (the DSA seam, proven both directions with a stand-in sparse backend) — ports of `test_attention_backends_selection.py` (MLA cases), `test_mla_prefill_selector.py`, `test_mla_prefill_registry.py`; **W4** [test_ops_mla_attn.cpp](../tests/vt/test_ops_mla_attn.cpp) — port of `tests/kernels/attention/test_mla_decode_cpu.py` (`ref_mla` as a TWO-PASS oracle, its bs=4/mean_seq_len=256/h_q=16/d=576/dv=512/block=16 parametrization, BOTH varlen arms, and its NaN-padding out-of-bounds detector) plus the `test_mla_backends.py` shape sweep: ragged, multi-block, single-block/single-token, EVERY num_kv_splits in {1,2,3,4,5,8,16,17,64,300,512} (incl. splits > seq_len, the empty-split path both stages must skip), 128-head DeepSeek-V3 geometry, head counts 1/3/17 that do not fill a BLOCK_H tile, a 288/256 block-32 non-V2-Lite geometry, bf16 + f32, and run-to-run BIT-exactness over 5 runs. Gated on dgx/sm_121: 11/11 cases, 2,303,193 assertions; `compute-sanitizer` memcheck **0 errors**, racecheck **0 hazards**, synccheck **0 errors**; clean CUDA build 0 warn/0 err; regression set UNCHANGED (27B 235/235, 35B 315/315, Coder 6/6, Qwen3-dense 16/16, OPT 6/6). NO speed number yet — decode perf is W9. **W5** [test_ops_mla_prefill.cpp](../tests/vt/test_ops_mla_prefill.cpp) **4/4 cases / 2,377,052 assertions** and [test_ops_mla_chunked_context.cpp](../tests/vt/test_ops_mla_chunked_context.cpp) **5/5 / 306,037 assertions** on dgx sm_121 — ports of `tests/v1/attention/test_mla_backends.py` and `tests/v1/attention/test_mla_prefill_quant_output.py` (its fp8 arms NOT ported: they need device-capability family 100, unreachable on sm_121 — recorded, not dropped). REAL V2-Lite prefill geometry (QK 192 / V 128 / latent 576, block 16, mscale^2 scale) against an INDEPENDENT double-precision TWO-PASS oracle, plus — for the chunked loop — a SINGLE-SHOT whole-sequence oracle that never chunks: exact / +1 / -1 chunk boundaries, a request with NO context, a chunk in which a request contributes ZERO keys, ragged multi-chunk, 128-head V3, single-token queries, ADVERSARIAL reverse-interleaved block tables, NaN-poisoned outputs, run-to-run BIT-exact over 5 runs. memcheck **0 errors**, racecheck **0 hazards**, synccheck **0 errors** on both binaries; clean CUDA build 0 warn/0 err; regression set UNCHANGED (27B 235/235, 35B 315/315, Coder 138/138, Qwen3-dense 664/664, OPT 36/36). Prefill perf is W9 | [MLA campaign spike](specs/mla-deepseek-campaign.md) | `PARTIAL` | `CLAIM-MLA-DEEPSEEK` | +| `KERNEL-ATTN-MLA-SPARSE` | MLA and sparse attention: CUTLASS, FlashMLA, FlashInfer, FA, Triton, MSA **W6: the MLA attention BLOCK + WEIGHT ABSORPTION — the layer that COMPOSES W3+W4+W5** — [mla_attention.h](../include/vllm/model_executor/models/mla_attention.h) + [mla_attention.cpp](../src/vllm/model_executor/layers/attention/mla_attention.cpp) <- `mla.py:119-181` (`MultiHeadLatentAttentionWrapper.forward`) over `mla_attention.py:553-620` (the cache-update-BEFORE-attention order), `:624-874` (`forward_impl`: the dispatch + the absorbed decode) and `:2344-2425` (`forward_mha`); `AbsorbKvBProjBf16` <- `:875-962 process_weights_after_loading` (split `:892-900`, permutes `:959-962`); `MakeMlaUpProjectFn` <- `:2141-2170` (the `kv_b_proj` callback W5 left open); `BuildDeepseekRopeCosSinCache` <- `deepseek_scaling_rope.py:76-118` over `rotary_embedding/common.py:34-70`; `MlaAttentionScale` <- `deepseek_v2.py:995,1067-1075` (the mscale^2 correction, kept SEPARATE from the rope's own rotation mscale). **Absorption is a LOAD-TIME weight transform plus TWO batched GEMMs, not a fused kernel**, so it needed only two new general primitives: **`vt::BatchedMatmul`** <- `torch.bmm` at `mla_attention.py:789` (q-side W_UK fold) and `:1034` (`_v_up_proj`), CUDA impl = cuBLASLt STRIDED-BATCHED [cuda_matmul.cu](../src/vt/cuda/cuda_matmul.cu) (the cuBLASLt form of the cuBLAS `gemmStridedBatchedEx` torch.bmm resolves to; the only upstream alternatives are ROCm-only aiter fp8/fp4 bmm branches) + CPU ref [cpu_ops.cpp](../src/vt/cpu/cpu_ops.cpp), stride-driven because BOTH call sites pass `.transpose(0,1)` views; and **`vt::ConcatMlaNopeRope`** <- `ConcatMLAQKernel` (`csrc/libtorch_stable/concat_mla_q.cuh`) + wrapper `cache_kernels.cu:1555-1600`, GENERALIZED to arbitrary nope/rope widths and a head-BROADCAST rope operand so one op also serves `_concat_k_nope_k_pe` (`:2063-2092`) — CUDA [cuda_mla_attn.cu](../src/vt/cuda/cuda_mla_attn.cu), CPU [cpu_ops.cpp](../src/vt/cpu/cpu_ops.cpp). Two ADDITIVE relaxations of existing ops, integer-identical for contiguous tensors: `vt::RopeFromCache` stride-driven on q/k (DeepSeek rotates the TRAILING 64-dim slice and its `k_pe` is a column block of the fused kv_a projection) and `vt::MatmulBT` accepting a row-strided ACTIVATION (`kv_b_proj` applied to a 512-column slice of the 576-wide workspace, `:2160`) | CUDA priority `vllm/platforms/cuda.py:84-176` (`_get_backend_priorities`, both branches); MLA classes `vllm/v1/attention/backends/mla/*.py`; MLA prefill selector `mla/prefill/selector.py:47-76`; capability filter `vllm/v1/attention/backend.py:307-360`; CUTLASS build `CMakeLists.txt:1037-1061` **W6** [test_mla_attention_block.cpp](../tests/vllm/model_executor/layers/attention/test_mla_attention_block.cpp) **10/10 cases / 2,372,644 assertions** and [test_ops_mla_absorb.cpp](../tests/vt/test_ops_mla_absorb.cpp) **9/9 / 1,644,807 assertions** on dgx sm_121 — ports of `tests/kernels/test_concat_mla_q.py` (BOTH arms incl. the NON-CONTIGUOUS transposed-nope case, compared bit-exactly since a concat is a pure copy), the MLA-geometry sweep of `tests/v1/attention/test_mla_backends.py`, and the two-pass-oracle discipline of `tests/kernels/attention/test_mla_decode_cpu.py`. **THE ABSORBED-vs-UNABSORBED EQUIVALENCE IS PROVEN NUMERICALLY, THREE WAYS:** an INDEPENDENT double-precision block oracle computing the attention BOTH ways agrees to **< 1e-11** (the identity itself, at both query branches); our absorbed decode reproduces the UNABSORBED oracle to **< 2e-4** (f32); and the SAME batch driven once through the ABSORBED MQA decode kernel and once through the UNABSORBED materialized-MHA prefill path agrees to **< 3e-4** (CPU f32) / **< 4e-2** (CUDA bf16) — two code paths sharing nothing but the weights. Real geometry throughout (V2-Lite 512/128/64/128/16-head, plus V3's 7168 / 128-head / `q_lora_rank=1536` for the lora branch, which has NO e2e coverage and says so). Decode-only / prefill-only-no-context / chunked-prefill-with-context / MIXED (decode packed FIRST) all gated; NaN-poisoned outputs; run-to-run BIT-exact; CUDA cases proven to EXECUTE (124,941 + 290,835 assertions when run alone). memcheck **0 errors**, racecheck **0 hazards**, synccheck **0 errors** (the last requires `--num-cuda-barriers 65536`: the default table OVERFLOWS on a binary driving this many kernel families and the tool then emits a bogus `unspecified launch failure`). Clean CUDA build 0 warn/0 err; regression set UNCHANGED (27B 235/235, 35B 315/315, Coder 138/138, Qwen3-dense 664/664, OPT 36/36). No speed number — W9 owns tuning | **SELECTION (W2) + the DECODE KERNEL (W4) + the PREFILL PATH and CHUNKED-CONTEXT LOOP (W5). What is still absent is the MLA attention BLOCK and MODEL (W6/W7).** Priority TABLE [cuda_attn_priority.h:49](../include/vllm/platforms/cuda_attn_priority.h#L49) (both branches, one row per upstream arch arm) + lookup [cuda_attn_priority.h:86](../include/vllm/platforms/cuda_attn_priority.h#L86); `is_mla()`/`is_sparse()` filter [registry.cpp:63](../src/vllm/v1/attention/registry.cpp#L63); `TritonMLABackend` NAME + 3-D `get_kv_cache_shape` [backend.h:509](../include/vllm/v1/attention/backend.h#L509), [backend.cpp:83](../src/vllm/v1/attention/backend.cpp#L83), registration [backend.cpp:108](../src/vllm/v1/attention/backend.cpp#L108) — **W4: `vt::MlaDecodeAttention`** — `OpId::kMlaDecodeAttention` + args/validation [ops.h](../include/vt/ops.h), [ops.cpp](../src/vt/ops.cpp); CPU single-pass REFERENCE [cpu_mla_attn.cpp](../src/vt/cpu/cpu_mla_attn.cpp) (numerics from `csrc/cpu/mla_decode.cpp`); CUDA two-stage split-KV [cuda_mla_attn.cu](../src/vt/cuda/cuda_mla_attn.cu) — `MlaDecodeStage1` <- `_fwd_grouped_kernel_stage1` (`triton_decode_attention.py:278-458`, IS_MLA `v = tl.trans(k)` branch `:424-431`), `MlaDecodeStage2` <- `_fwd_kernel_stage2` (`:575-639`), `ComputeNumKvSplits` <- `_compute_num_kv_splits` (`triton_mla.py:40-47`), split workspace via the house grow-only per-stream scratch (upstream's `_reserve_attn_logits_workspace` `:57-78`). Deterministic by construction: fixed ASCENDING split merge, NO atomicAdd. `TritonMLABackend::get_impl_cls()` now returns a real `TritonMLAImpl` [backend.h](../include/vllm/v1/attention/backend.h), [backend.cpp](../src/vllm/v1/attention/backend.cpp); PREFILL remains W5 and `TritonMLAImpl::forward` refuses a prefill-shaped batch by name. **W5: `vt::MlaPrefillAttention` + `vt::GatherMlaCache` + `vt::MergeAttnStates` + the chunked-context driver** — `vt::MlaPrefillAttention` [cuda_mla_prefill.cu](../src/vt/cuda/cuda_mla_prefill.cu) / CPU ref [cpu_mla_prefill.cpp](../src/vt/cpu/cpu_mla_prefill.cpp) <- `mla/prefill/flash_attn.py:153-248` `FlashAttnPrefillBackend` (the ONLY MLA prefill backend reachable on sm_121 per `mla/prefill/selector.py:66-76`, and it HARD-RAISES with no fallback at `:191-194`), running over the vendored FA-2 through the NEW launcher entry `LaunchMlaPrefillFA2Bf16` [cuda_flash_attn_fa2.cu](../src/vt/cuda/cuda_flash_attn_fa2.cu) plus two new explicit instantiations of the UNCHANGED generic template (`flash_fwd_split_hdim192_bf16{,_causal}_sm80.cu`). V is ZERO-PADDED 128->192 and the output sliced back, exactly as upstream's `requires_v_padding` path does (`flash_attn.py:88-99,164-168,196-197`) — which is WHY the asymmetric QK 192 / V 128 pair needs no asymmetric kernel. `vt::GatherMlaCache` <- `csrc/libtorch_stable/cache_kernels.cu:992-1064`; `vt::MergeAttnStates` <- `csrc/libtorch_stable/attention/merge_attn_states.cu:18-192` (BOTH `-inf` edge cases ported verbatim). The workspace-bounded loop is [mla_chunked_context.h](../include/vllm/model_executor/layers/attention/mla_chunked_context.h) <- `mla_attention.py:1422-1451,1667-1745,2094-2199,2344-2425`. **The paged launcher `LaunchPrefillFA2Bf16` that every non-MLA prefill calls is textually UNTOUCHED** (211 insertions / 0 deletions in that TU; 2 new vendored files) | [test_attn_backend_registry.cpp:146](../tests/vllm/v1/attention/test_attn_backend_registry.cpp#L146) (GB10 MLA list), [:203](../tests/vllm/v1/attention/test_attn_backend_registry.cpp#L203) (`use_mla=true` -> `TRITON_MLA`, matching the W0 oracle observation), [:230](../tests/vllm/v1/attention/test_attn_backend_registry.cpp#L230) (the DSA seam, proven both directions with a stand-in sparse backend) — ports of `test_attention_backends_selection.py` (MLA cases), `test_mla_prefill_selector.py`, `test_mla_prefill_registry.py`; **W4** [test_ops_mla_attn.cpp](../tests/vt/test_ops_mla_attn.cpp) — port of `tests/kernels/attention/test_mla_decode_cpu.py` (`ref_mla` as a TWO-PASS oracle, its bs=4/mean_seq_len=256/h_q=16/d=576/dv=512/block=16 parametrization, BOTH varlen arms, and its NaN-padding out-of-bounds detector) plus the `test_mla_backends.py` shape sweep: ragged, multi-block, single-block/single-token, EVERY num_kv_splits in {1,2,3,4,5,8,16,17,64,300,512} (incl. splits > seq_len, the empty-split path both stages must skip), 128-head DeepSeek-V3 geometry, head counts 1/3/17 that do not fill a BLOCK_H tile, a 288/256 block-32 non-V2-Lite geometry, bf16 + f32, and run-to-run BIT-exactness over 5 runs. Gated on dgx/sm_121: 11/11 cases, 2,303,193 assertions; `compute-sanitizer` memcheck **0 errors**, racecheck **0 hazards**, synccheck **0 errors**; clean CUDA build 0 warn/0 err; regression set UNCHANGED (27B 235/235, 35B 315/315, Coder 6/6, Qwen3-dense 16/16, OPT 6/6). NO speed number yet — decode perf is W9. **W5** [test_ops_mla_prefill.cpp](../tests/vt/test_ops_mla_prefill.cpp) **4/4 cases / 2,377,052 assertions** and [test_ops_mla_chunked_context.cpp](../tests/vt/test_ops_mla_chunked_context.cpp) **5/5 / 306,037 assertions** on dgx sm_121 — ports of `tests/v1/attention/test_mla_backends.py` and `tests/v1/attention/test_mla_prefill_quant_output.py` (its fp8 arms NOT ported: they need device-capability family 100, unreachable on sm_121 — recorded, not dropped). REAL V2-Lite prefill geometry (QK 192 / V 128 / latent 576, block 16, mscale^2 scale) against an INDEPENDENT double-precision TWO-PASS oracle, plus — for the chunked loop — a SINGLE-SHOT whole-sequence oracle that never chunks: exact / +1 / -1 chunk boundaries, a request with NO context, a chunk in which a request contributes ZERO keys, ragged multi-chunk, 128-head V3, single-token queries, ADVERSARIAL reverse-interleaved block tables, NaN-poisoned outputs, run-to-run BIT-exact over 5 runs. memcheck **0 errors**, racecheck **0 hazards**, synccheck **0 errors** on both binaries; clean CUDA build 0 warn/0 err; regression set UNCHANGED (27B 235/235, 35B 315/315, Coder 138/138, Qwen3-dense 664/664, OPT 36/36). Prefill perf is W9 | [MLA campaign spike](specs/mla-deepseek-campaign.md) | `PARTIAL` | `CLAIM-MLA-DEEPSEEK` | | `KERNEL-ATTN-DFLASH-BLOCK` | **DFlash in-block attention — the project's FIRST non-causal / bidirectional attention primitive** (SPEC-DFLASH D2, DF-DRAFT-MODEL). Per-request uniform (1+k) query block attends within its own block: FULL-attention layers BIDIRECTIONAL (`causal=false`, no mask), SWA layers causal-within-window. f32 online softmax, GQA broadcast. A SEPARATE `vt::` op from the causal `kAttention`/`kPagedAttention` so every other model stays byte-identical | `vllm/model_executor/models/qwen3_dflash.py:86-146` (`_resolve_layer_attention`: full layers default non-causal, SWA causal) + `:149-263` (`DFlashQwen3Attention`); flashinfer non-causal path (vllm#48167 Blackwell non-causal attn, in-pin) | `OpId::kDFlashBlockAttention` + `DFlashBlockAttentionArgs` + decl [ops.h:1713](../include/vt/ops.h#L1713) + wrapper/validation [ops.cpp:2069](../src/vt/ops.cpp#L2069); CPU REFERENCE `DFlashBlockAttentionKernel` [cpu_ops.cpp:1843](../src/vt/cpu/cpu_ops.cpp#L1843) (three-pass block-local softmax, the authoritative impl); CUDA `DFlashBlockAttentionKernelCuda` [cuda_ops.cu:1300](../src/vt/cuda/cuda_ops.cu#L1300) mirroring the causal `AttentionKernel` block-reduction recurrence with per-block bounds + the bidirectional/window mask; the draft model that consumes it [qwen3_dflash.cpp:52](../src/vllm/model_executor/models/qwen3_dflash.cpp#L52) | **CPU GATE GREEN** [test_ops_dflash_block_attn.cpp:79](../tests/vt/test_ops_dflash_block_attn.cpp#L79) 5 cases / 12 assertions — hand-checked non-causal (query 0 sees the future key), the RED causal-vs-non-causal separation (the mask is load-bearing), per-request cu_seqlens block isolation, SWA window bound, GQA; model forward [test_qwen3_dflash_forward.cpp:116](../tests/vllm/models/test_qwen3_dflash_forward.cpp#L116) 5 cases / 95 assertions (RED full-layer-causal-flip); existing causal `test_ops_attention` 9/9·23 UNCHANGED. **GPU GATE GREEN on dgx (2026-07-26, GB10 sm_121a):** CUDA `-Werror=all-warnings` build clean (kernel compiles as-written, no change); CUDA==CPU parity [test_ops_dflash_block_attn CUDA case](../tests/vt/test_ops_dflash_block_attn.cpp#L153) 198412/198412 within the 1e-4 f32-softmax envelope over all 5 corners; `compute-sanitizer --tool memcheck` 0 errors; consumed by the draft-forward parity gate ([test_qwen3_dflash_draft_parity](../tests/parity/test_qwen3_dflash_draft_parity.cpp), fc rel-L2 0.46% / hidden ≤1.3% vs the real vLLM draft). **DONE 2026-07-27 with the DFlash block (`CLAIM-DFLASH-D14`):** the D2 non-causal in-block primitive is the CPU/materialized reference the D12+ paged/warp kernels are gated against; closure [ledger](parity-ledger.md#L722). | [DFlash spec](specs/dflash-spec-decode.md) §1.3/§6 D2 | `DONE` | `489a7544` | | `KERNEL-ATTN-DFLASH-PAGED-BLOCK` | **DFlash PAGED in-block attention — the CAPTURE-SAFE form of `KERNEL-ATTN-DFLASH-BLOCK`** (SPEC-DFLASH D12 Part B, the CUDA-graph draft-attention primitive). The (1+k) block queries attend over `[PAGED context ; their own (1+k) block]`: the growing context enters as DATA (paged K/V cache `[pages,block_size,Hkv,D]` + per-request `seq_lens` + `block_table`, mirroring `PagedAttentionKernel`) instead of a variable-size materialized combined buffer, so the launch grid is STATIC over the fixed `Nq=(1+k)*num_reqs` rows and EVERY metadata input is a persistent DEVICE tensor read in place — NO `cudaMallocAsync`/`cudaMemcpyAsync` of a function-local host `cu_seqlens` (the [[cudagraph-capture-bakes-stack-addresses]] UAF class the eager `LaunchDFlashBlockAttention` had). Same f32 online softmax + D2 in-block mask over the COMBINED index; bit-identical to `DFlashBlockAttention` over the materialized `[context;block]` buffer | vLLM full CG `dflash/cudagraph.py` + `speculator.py:411-458` + `precompute_and_store_context_kv` (`qwen3_dflash.py:548-619`) @ `555967922`; paged read mirrors our `PagedAttentionKernel` [cuda_paged_attn.cu:184](../src/vt/cuda/cuda_paged_attn.cu#L184) | `OpId::kDFlashPagedBlockAttention` + `DFlashPagedBlockAttentionArgs` + decl [ops.h](../include/vt/ops.h) + wrapper/validation [ops.cpp](../src/vt/ops.cpp); CPU REFERENCE `DFlashPagedBlockAttentionKernel` [cpu_ops.cpp](../src/vt/cpu/cpu_ops.cpp); CUDA `DFlashPagedBlockAttentionKernelCuda` [cuda_ops.cu](../src/vt/cuda/cuda_ops.cu#L1452) (static grid, persistent device metadata) + D14 WARP variant [DFlashPagedBlockAttentionWarpKernel](../src/vt/cuda/cuda_ops.cu#L1433) | **GPU GATE GREEN on dgx (2026-07-27, GB10 sm_121a):** CUDA `-Werror` clean (0 warnings); [test_ops_dflash_paged_block_attn.cpp](../tests/vt/test_ops_dflash_paged_block_attn.cpp#L79) cross-checks CPU-paged == materialized `DFlashBlockAttention` across 6 corners (non-causal, causal-SWA, block isolation, GQA, multi-page, zero-context) + CUDA==CPU (f32+bf16) = **795648/795648 assertions**; `compute-sanitizer --tool memcheck` **0 errors**. **D13 (2026-07-27, `CLAIM-DFLASH-D13`): WIRED INTO PRODUCTION** — the single-request DFlash draft forward (`ForwardPagedBody`, `qwen3_dflash.cpp`) now runs the (1+k) block through this kernel reading a fixed-capacity paged `DflashDeviceKVStore`, and the whole draft step is captured into a per-request CUDA graph + replayed (the growing context enters only via the in-place `seq_lens`). Capture-correctness PROVEN: `test_qwen27_dflash_spec_decode` 27/27 with the graph BIT-IDENTICAL to eager (same tokens + acceptance 19/39/29/25); c1 throughput NEAR-PARITY with vLLM-DFlash-ON (ours 0.978×, ~2% below the tight 3-rep band; gap closed 0.917×→0.978× via the paged read, the CG is perf-neutral) — the kernel is landed + wired + gated; STAYS `ACTIVE` with the engine feature (the ~2% ≥vLLM residual is per-step compute for an nsys). **D14 (2026-07-27, `CLAIM-DFLASH-D14`): the residual WAS this kernel → WARP-scoped variant added → SPEED GATE MET, `DONE`.** An nsys (`--cuda-graph-trace=node`) attributed the D13 ~2% residual to THIS kernel: `DFlashPagedBlockAttentionKernel` = 242.9 ms = 1.8% of the graphed step's GPU time, median ~460 us/call (grid `(nq=17,hq=32)` × kBlock=256 threads looping SERIALLY over C~500-640 keys with a 256-wide shared-mem tree reduction + 2 `__syncthreads` PER key — the latency/sync storm the ViT tower fixed with `AttentionDenseFast`), vs vLLM's fused flash draft-attn ~0.15%. Added `DFlashPagedBlockAttentionWarpKernel` ([cuda_ops.cu](../src/vt/cuda/cuda_ops.cu)): ONE WARP per (block-query,head), `__shfl_xor` butterfly head_dim reduction, register accumulator, NO `__syncthreads`; SAME paged/block combined-index read + causal/SWA mask + GQA (copied verbatim from the block kernel), mirroring the shipped `AttentionWarpKernel`. Default ON; `VT_DFLASH_ATTN_BLOCK=1` keeps the bit-identical D12/D13 block kernel. Draft attn **242.9 → 77.9 ms (3.1×)**; our-ON c1 **28.60 → 29.32 tok/s**; FINAL 3-rep A/B our-ON 29.32 ≥ vLLM-ON 29.240 (non-overlapping bands, 1.003×) ⇒ **≥vLLM MET**. Not bit-identical to the block kernel but same f32-online-softmax math within envelope; CUDA==CPU `test_ops_dflash_paged_block_attn` **795648/795648** (f32 1e-4/bf16 3e-2) + **compute-sanitizer 0**; e2e 27/27 graph==eager, acceptance 19/39/29/25 unchanged (1629 accepted identical warp-vs-block); SACRED 235/235 + MTP 9/9 inert; `-Werror` clean; closure [ledger](parity-ledger.md#L738) | [DFlash spec](specs/dflash-spec-decode.md) §0 D12/D13/D14 | `DONE` | `489a7544` | | `KERNEL-ATTN-DSA-SPARSE-INDEX` | **DeepSeek-V4 DSA "Lightning Indexer" sparse-attention SELECTION — the project's FIRST sparse candidate-selection primitive** (DeepSeek-V4-Flash W3). Two ops: (1) the weighted-MQA INDEXER LOGIT `logit[t,s] = Σ_h w[t,h]·ReLU(q[t,h]·k[s])` over the causal candidate window (the per-head **ReLU** is load-bearing — it is what makes the indexer a learned sparse SELECTOR, not a plain attention score), where `w[t,h] = weights_proj[t,h]·index_head_dim^-0.5·index_n_heads^-0.5`; and (2) the per-row **causal top-k** that keeps the `index_topk=512` highest-logit keys (short-context: every candidate, ascending; else top-k with -1 padding). Distinct from every dense/paged/MLA family, which score ALL keys — this one PICKS a sparse key subset the downstream MLA then attends over. W3 also lands the two 512-wide-MLA OUTPUT seams V2/V3 lack (per-head attention-**sink** softmax + **grouped output-LoRA** `wo_a` bmm→`wo_b`) as portable host references beside it | MQA logit `vllm/v1/attention/ops/triton_fp8_mqa_logits.py:120-156` (dot→×kv_scale→ReLU→×weights→Σheads); weight fold `vllm/model_executor/layers/sparse_attn_indexer.py:203-207`; top-k `sparse_attn_indexer.py:488-497` + short-context `vllm/models/deepseek_v4/attention.py:70-86,:813-831`; sinks `deepseek_v4/nvidia/flashinfer_sparse.py:777,:896`; grouped output-LoRA `deepseek_v4/nvidia/ops/o_proj.py:58-73` @ `555967922` | Portable host reference (device kernel is a W7 residual) [deepseek_v4_dsa.cpp](../src/vllm/model_executor/models/deepseek_v4_dsa.cpp) + [deepseek_v4_dsa.h](../include/vllm/model_executor/models/deepseek_v4_dsa.h): `DsaIndexerWeightFold` / `DsaIndexerLogits` / `DsaTopkSelect` / `SoftmaxWithSink` / `GroupedOutputLora` | **CPU UNIT GATE GREEN (2026-07-28, `-Wall -Werror -Wextra` 0-warn):** [test_deepseek_v4_dsa.cpp](../tests/vllm/models/test_deepseek_v4_dsa.cpp) **13/13 cases · 38 assertions** — hand-derived literal cases (the ReLU clip, the weight fold, short-context all-select, full top-k, tie→smaller-index, causal-window offset, sink probability mass, sink numerical stability, grouped-LoRA) + from-first-principles double-precision references on randomized shapes (indexer logits + grouped output-LoRA rel-L2 < 1e-6). Full-model gate is multi-Spark-blocked (156.7 GiB); MHC (W5) + sqrtsoftplus/hash MoE (W6) + device kernel + forward integration (W7) are named residuals | [deepseek-v4-flash spike](specs/deepseek-v4-flash.md) §W3 | `SPIKE` | `CLAIM-DEEPSEEK-V4-W3` | diff --git a/.agents/specs/attn-validate-configuration.md b/.agents/specs/attn-validate-configuration.md index b00f48729..484f7f15e 100644 --- a/.agents/specs/attn-validate-configuration.md +++ b/.agents/specs/attn-validate-configuration.md @@ -41,7 +41,7 @@ own it already exist.** No new roadmap row is opened. 1. **The selector still routes nothing.** `SelectAttentionBackendName`'s result reaches exactly three places in - `src/vllm/v1/worker/gpu/runner.cpp::GpuModelRunner::InitializeKvCache`: + `src/vllm/v1/worker/gpu/runner.cpp::GPUModelRunner::initialize_kv_cache`: `attn_backend_names_` (`:1002`), the `VT_ATTN_SELECT_LOG` debug print (`:1003-1024`), and `vllm::v1::CheckKvCacheShape` (`:1025-1029`). `dense_attn::AttnBlock` calls `vt::PagedAttention` unconditionally @@ -66,7 +66,7 @@ own it already exist.** No new roadmap row is opened. 3. **`AttnSelectorConfig::dtype` is not supplied by the runner.** The production call site fills `head_size`, `block_size` and `kv_cache_dtype` from the resolved per-layer KV geometry, but the model/query dtype is not available at - `InitializeKvCache` (the runner resolves only `ResolveKvCacheDType()` there), + `initialize_kv_cache` (the runner resolves only `ResolveKvCacheDType()` there), so `dtype` keeps its `kBF16` default in production and the `supports_dtype` predicate is exercised by tests alone. **Owner: #1332 M4.** 4. **The non-MLA sm_100 priority arm gained a `use_non_causal` guard upstream** @@ -195,6 +195,61 @@ behavior-preserving control for the existing production call sites. `test_attn_validate_configuration` and `test_attn_backend_registry` cases. No GPU: every predicate is host code and the capability is injected. +## Evidence + +CPU Debug build, `-DVLLM_CPP_CUDA=OFF`. Every run below is a direct binary +invocation, never through a pipe, so the reported status is the binary's. + +**RED before.** `tests/vllm/v1/attention/test_attn_validate_configuration.cpp` +written first against the config fields alone, with no predicate implemented: + +``` +[doctest] test cases: 12 | 2 passed | 10 failed | 0 skipped +[doctest] assertions: 22 | 10 passed | 12 failed | +[doctest] Status: FAILURE! (exit 1) +``` + +The two that passed are the behavior control (`FLASH_ATTN` still selected) and +the case asserting the sm_100 CAUSAL ordering, which was already correct — the +non-causal half of that same case is one of the ten reds. + +**GREEN after**, with the predicate-level cases added: + +| Binary | cases | assertions | status | +|---|---:|---:|---| +| `test_attn_validate_configuration` | 21 | 76 | SUCCESS | +| `test_attn_backend_registry` | 17 | 62 | SUCCESS | +| `test_runner` | 19 | 543 | SUCCESS | + +**Mutations.** Each applied to the worktree, built, run, then restored from a +`tar` snapshot taken before the pass — not `git checkout`, which reads the index +and would have destroyed the untracked new test file. `git diff --stat` was +printed before and after each one and the line counts returned to their +pre-mutation values, and `compile_rc` was printed for each, because a mutation +that fails to build and a mutation that never applied both read as a pass. + +| # | Mutation | `compile_rc` | Result | +|---|---|---:|---| +| M1 | `CandidateInvalidReasons` returns `{}` — the selector stops asking | 0 | RED: `validate_configuration` 12/21, `registry` 15/17 | +| M2 | drop the `capability.present()` precondition | 0 | RED: both files, 1 case each | +| M3 | `supports_block_size` compares equality instead of a multiple | 0 | RED: 4 cases | +| M4 | remove the #1333 causal-only guard | 0 | RED: 1 case | +| M5 | **reachability**: the production geometry stops flowing into the config (`cfg.head_size = 17`) | 0 | RED: `test_runner` **19/19 cases fail** | +| M6 | **reachability**: delete the production config fill entirely | 0 | GREEN: `test_runner` 543/543 | + +**M5 is the reachability proof.** Corrupting one field of the config that +`GPUModelRunner::initialize_kv_cache` builds fails every one of the 19 +`test_runner` cases, because the refusal propagates out of the runner's own +initialization. The capability layer is therefore reached from a production +entry point with production data, not only from its own unit test. + +**M6 is the honest negative, and it is recorded rather than explained away.** +Deleting the fill leaves `test_runner` green, because the defaults still +validate. The layer is reached and executed; no test's failure depends on the +fill being PRESENT, only on its being CORRECT. That is the expected shape while +the selected name still dispatches nothing (`## Owed` item 1), and it is the +second reason this change must not be read as fixing selection. + ## Stop conditions - Stop and report `NEEDS_DECISION` if a ported predicate would refuse a diff --git a/docs/USAGE.md b/docs/USAGE.md index e1f6c665d..9e633ba07 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -734,6 +734,37 @@ block-wise FP8 scale tensor the checkpoint had never contained ([#1256](https://github.com/mudler/vllm.cpp/issues/1256)). A message that blames the wrong side costs more than the failure does. +### A refusal that names the attention backend, and what it cannot tell you + +Starting an engine resolves an attention backend for each KV-cache group, and +that backend is now asked whether it can serve the request before it is chosen. +When none of the backends this build registers can, the engine refuses at +initialization rather than later, and the message names every candidate with +every reason it lost: + +```text +No valid attention backend for device type 1 from +{FLASH_ATTN: [head_size not supported, block_size not supported]} +(use_mla=false, use_sparse=false) +``` + +The reason strings are vLLM's own, so a refusal here and a refusal from the +reference engine read the same. `head_size`, `block_size` and the KV-cache dtype +come from the geometry the engine has just resolved for your checkpoint, so a +refusal is about that checkpoint on this build. + +**What this check cannot tell you.** It reports what a backend *claims*, never +what your binary contains and never whether the kernel will launch. A backend +whose declared floor is compute capability 8.0 is accepted on any newer GPU, even +when the build carries no compiled code for that GPU — which is a real failure +mode, not a hypothetical one, and it surfaces as a launch error rather than as +this refusal. Confirming which architectures a build actually targets is a +separate question, answered under "Confirming which CUDA architecture a build +targets" above. Tracked as +[#1332](https://github.com/mudler/vllm.cpp/issues/1332). + +Selecting a backend by name is not exposed yet; the engine always resolves one. + ### Architectures that resolve but refuse to run A few architectures are registered so their config and weight layout are diff --git a/include/vllm/platforms/cuda_attn_priority.h b/include/vllm/platforms/cuda_attn_priority.h index 28934cdb8..6faf734ca 100644 --- a/include/vllm/platforms/cuda_attn_priority.h +++ b/include/vllm/platforms/cuda_attn_priority.h @@ -1,9 +1,15 @@ // The CUDA attention-backend priority TABLE — a faithful, complete port of -// vllm/platforms/cuda.py:84-176 `_get_backend_priorities` @ pin e24d1b24 (BOTH -// the MLA branch `:93-142` and the non-MLA branch `:143-166`), plus -// vllm/v1/attention/backends/mla/prefill/selector.py:47-76 +// vllm/platforms/cuda.py:83-163 `_get_backend_priorities` @ pin 5559679229 (BOTH +// the MLA branch `:93-143` and the non-MLA branch `:144-163`), plus +// vllm/v1/attention/backends/mla/prefill/selector.py:48-77 // `_get_mla_prefill_backend_priorities`. // +// Re-anchored from `cuda.py:84-176 @ e24d1b24`, the pin retired at W5. The MLA +// rows, the sm_12x two-entry row and the MLA-prefill lists are unchanged in +// CONTENT at the new pin and were re-read line by line to confirm it. The +// non-MLA sm_100 row was NOT: it gained a `use_non_causal` guard, which is +// issue #1333 and is ported below. +// // It lives in a HEADER, not inside the CUDA-only cuda.cpp TU, for two reasons: // * it is pure DATA + a lookup — no CUDA, no device — so the CPU test tier can // assert the REAL table instead of a hand-copied duplicate (the pre-W2 @@ -41,6 +47,11 @@ struct AttnPriorityRow { int major; // exact compute-capability major, or kAnyMajor for upstream `else` std::vector backends; SparseTailOrder sparse_tail; + // cuda.py:148 `and not use_non_causal` — issue #1333. When true, this row + // matches only a CAUSAL request and a non-causal one falls through to the + // `else` row. Exactly one upstream arm carries the guard, so it is a per-row + // field rather than a second key. + bool causal_only = false; }; // Upstream names are the AttentionBackendEnum MEMBER names @@ -70,12 +81,17 @@ inline const std::vector& AttnPriorityTable() { {"FLASH_ATTN_MLA", "FLASHMLA", "FLASHINFER_MLA", "TRITON_MLA", "FLASH_ATTN_MLA_SPARSE", "FLASHMLA_SPARSE"}, SparseTailOrder::kFixed}, - // ── non-MLA (unchanged behavior; previously an inline if-chain) ──────── - // cuda.py:144-152 — sm_100. + // ── non-MLA ─────────────────────────────────────────────────────────── + // cuda.py:145-155 — sm_100, CAUSAL only. The guard is upstream's, and its + // reason is upstream's comment at `:145-147`: SM100f defaults to FlashInfer + // for TRTLLM causal attention, but its non-causal cutlass path — the one + // DFlash attention uses — is known to have problems, so FlashAttention is + // preferred when non-causal on SM100f. Issue #1333. {false, 10, {"FLASHINFER", "FLASH_ATTN", "TRITON_ATTN", "FLEX_ATTENTION", "TURBOQUANT"}, - SparseTailOrder::kFixed}, - // cuda.py:153-166 — everything else, INCLUDING GB10 sm_121 (major 12). + SparseTailOrder::kFixed, /*causal_only=*/true}, + // cuda.py:156-163 — everything else, INCLUDING GB10 sm_121 (major 12), and + // INCLUDING sm_100 for a non-causal request. {false, kAnyMajor, {"FLASH_ATTN", "FLASHINFER", "TRITON_ATTN", "FLEX_ATTENTION", "TURBOQUANT"}, SparseTailOrder::kFixed}, @@ -89,6 +105,9 @@ inline std::vector LookupAttnPriority(int major, const AttnPriorityRow* fallback = nullptr; for (const AttnPriorityRow& row : AttnPriorityTable()) { if (row.use_mla != cfg.use_mla) continue; + // cuda.py:148 — a causal-only row does not match a non-causal request, so + // the walk continues to the `else` row (#1333). + if (row.causal_only && cfg.use_non_causal) continue; if (row.major == major) { match = &row; break; diff --git a/include/vllm/platforms/interface.h b/include/vllm/platforms/interface.h index ebfb15439..fc67ac0d3 100644 --- a/include/vllm/platforms/interface.h +++ b/include/vllm/platforms/interface.h @@ -1,5 +1,8 @@ -// Faithful 1:1 port of vllm/platforms/interface.py:134-229 (class Platform) @ -// pin e24d1b24 — the device-capability / memory-model seam. See +// Faithful 1:1 port of vllm/platforms/interface.py:134-1290 (class Platform) @ +// pin 5559679229 — the device-capability / memory-model seam. (Re-anchored from +// `:134-229 @ e24d1b24`, the pin retired at W5: the class is unchanged in the +// parts we mirror, and the cited `supported_dtypes` / `get_device_capability` +// members moved rather than changing.) See // .agents/porting-inventory.md §9 note 8 (the platforms/ tree is a faithful // mirror of the upstream seam, NOT a deviation) and // .agents/specs/extensibility-platform-seam-2026-07-18.md. @@ -117,15 +120,24 @@ inline ResidencyPolicy CudaResidencyPolicy(size_t device_memory_total_bytes) { return p; } -// The selection inputs of vllm/platforms/cuda.py::_get_backend_priorities @ pin -// e24d1b24 (`use_mla`, `device_capability`, `num_heads`, `kv_cache_dtype`) plus -// the sparse flag that `AttentionBackend.is_sparse()` / -// `vllm/v1/attention/backend.py:307-360 validate_configuration` keys on. The -// capability itself is NOT a field — it is the platform's own -// `get_device_capability()`, exactly as upstream passes it in. +// A 1:1 port of `vllm/v1/attention/selector.py::AttentionSelectorConfig` +// (`:24-39`) @ pin `5559679229` — the ONE request description upstream feeds to +// BOTH `vllm/platforms/cuda.py::_get_backend_priorities` (`:83-163`) and +// `vllm/v1/attention/backend.py::AttentionBackend.validate_configuration` +// (`:320-393`). The capability itself is NOT a field — it is the platform's own +// `get_device_capability()`, exactly as upstream passes it in beside the config +// (`cuda.py:381-384`). // -// Defaults reproduce today's non-MLA dense selection EXACTLY, so every existing -// caller (`get_attn_backend_priority()` with no argument) is unchanged. +// It lives under `platforms/` rather than under `v1/` because +// `Platform::get_attn_backend_priority` takes it and `platforms/` must not depend +// on `v1/`. That is also why `attn_type` below is a STRING: upstream's +// `AttentionType` is a `str` enum (`backend.py:38-46`) and `supports_attn_type` +// takes a `str` (`:292-298`), so the string IS the faithful shape; +// `vllm::v1::AttentionTypeName` converts our enum to it. +// +// Every default is upstream's, so every existing caller +// (`get_attn_backend_priority()` with no argument, and the runner's dense walk) +// keeps its exact selection behavior. struct AttnSelectorConfig { // model_config.use_mla — the MLA branch of _get_backend_priorities:93-142. // On our gate models (Qwen3 dense / GDN) this is false. @@ -139,9 +151,44 @@ struct AttnSelectorConfig { // cuda.py:105 `num_heads is not None and num_heads <= 16` (sm_100 sparse-MLA // ordering only). 0 == unknown, upstream's `None`. int num_heads = 0; - // cuda.py:96 `is_quantized_kv_cache(kv_cache_dtype)` (sm_100 sparse-MLA - // ordering only). Our KV cache is bf16/f32 today, so false. + // cuda.py:98 `is_quantized_kv_cache(kv_cache_dtype)` (sm_100 sparse-MLA + // ordering only). Our KV cache is bf16/f32 today, so false. Upstream DERIVES + // this from `kv_cache_dtype` below; it stays a separate field because the MLA + // priority rows and their tests already read it, and the two are asserted + // consistent nowhere — recorded as such rather than silently unified. bool quantized_kv_cache = false; + + // ─── selector.py:25-28 — the four inputs upstream makes REQUIRED ─────────── + // The attention head dimension. 0 is accepted by every predicate whose + // supported list is empty, which is upstream's answer for an unconstrained + // backend (`backend.py:159-161`). + int head_size = 0; + // The model/query dtype (`supports_dtype`, backend.py:163-165). Upstream has + // no `None` here. The runner does not yet supply it (issue #1332 M4), so on + // the production path it keeps this default. + DType dtype = DType::kBF16; + // The KV-cache quantization name ("auto", "fp8", "fp8_e4m3", ...). EMPTY is + // upstream's `None`, which `supports_kv_cache_dtype` accepts outright + // (`backend.py:167-173`). + std::string kv_cache_dtype; + // The framework block size. 0 is upstream's `None`, accepted outright + // (`backend.py:176-178`). + int block_size = 0; + + // ─── selector.py:30-39 — the request flags, upstream defaults ───────────── + bool has_sink = false; + bool use_mm_prefix = false; + bool use_per_head_quant_scales = false; + // `AttentionType` is a `str` enum upstream; "decoder" is `AttentionType.DECODER`. + std::string attn_type = "decoder"; + bool has_sliding_window = false; + // Non-causal (bidirectional) decoder attention — DFlash's drafter sets it. + // Also a `_get_backend_priorities` input at this pin (`cuda.py:88,148`), which + // is issue #1333. + bool use_non_causal = false; + bool use_batch_invariant = false; + bool use_kv_connector = false; + bool use_pcp = false; }; // Faithful port of vllm/platforms/interface.py:134-229 `class Platform`. Exposes diff --git a/include/vllm/v1/attention/backend.h b/include/vllm/v1/attention/backend.h index d0dfd329a..b53f43d0f 100644 --- a/include/vllm/v1/attention/backend.h +++ b/include/vllm/v1/attention/backend.h @@ -1,4 +1,13 @@ -// Ported from: vllm/v1/attention/backend.py @ e24d1b24 +// Ported from: vllm/v1/attention/backend.py @ pin 5559679229 (re-anchored from +// `e24d1b24`, the pin retired at W5; `validate_configuration` is `:320-393` and +// the `supports_*` predicates `:154-317` there). +// +// ONE anchor is deliberately NOT advanced: FlashAttentionBackend::get_kv_cache_shape +// below still mirrors flash_attn.py @ `e24d1b24`. At `5559679229` upstream returns +// (num_blocks, num_kv_heads, block_size, 2 * head_size) instead. That is the KV +// memory format the whole engine allocates and every paged-attention kernel +// reads, so re-anchoring it is a kernel campaign rather than a comment edit. +// Named here rather than left to be discovered; owner: BACKEND-ATTN-REGISTRY. // // Scope (M1.6 Task 1): the per-step attention metadata + the attention backend // interface the paged-attention path (Task 2/3) and the GDN metadata (Task 4) @@ -23,13 +32,11 @@ // * The deprecated lazy _seq_lens_cpu accessor and the unpadded()/replace() // spec-decode helpers. The host-equivalent num_computed_tokens_cpu array is // ported because chunked-local virtual batching consumes and rewrites it. -// * AttentionBackend: the whole supports_*/validate_configuration capability -// surface, get_kv_cache_stride_order / get_kv_cache_block_dim, cudagraph -// capture/drafting hooks, MLA / sparse-MLA impls. AttentionCGSupport itself -// is ported for the chunked-local wrapper. Ported here: the T0 core -// (CommonAttentionMetadata, the AttentionBackend/AttentionImpl/ -// AttentionMetadataBuilder ABCs, get_name + get_kv_cache_shape, the forward -// signature). +// * AttentionBackend: get_kv_cache_stride_order / get_kv_cache_block_dim, +// cudagraph capture/drafting hooks, MLA / sparse-MLA impls. +// AttentionCGSupport itself is ported for the chunked-local wrapper. +// The supports_*/validate_configuration capability surface is NO LONGER +// deferred — it is ported below (issue #1332 M1). #ifndef VLLM_V1_ATTENTION_BACKEND_H_ #define VLLM_V1_ATTENTION_BACKEND_H_ @@ -39,6 +46,7 @@ #include #include +#include "vllm/platforms/interface.h" #include "vllm/v1/worker/gpu/prepare_inputs.h" #include "vt/ops.h" @@ -53,6 +61,26 @@ enum class AttentionType { kEncoderDecoder // "encoder_decoder" }; +// Upstream's AttentionType IS its string value (backend.py:38-46 is a class of +// `str` constants), and `supports_attn_type` takes that string (`:292-298`). +// `platforms::AttnSelectorConfig::attn_type` therefore carries the string, and +// this is the conversion. Returns the exact upstream spelling. +const char* AttentionTypeName(AttentionType type); + +// The `CacheDType` name (vllm/config/cache.py:19-36) for a KV-cache storage +// dtype, which is what `supports_kv_cache_dtype` matches against. +// +// Upstream's "auto" means "the model dtype, unquantized" (`cache.py:77-78`), and +// an f32/f16/bf16 KV cache is exactly that — so all three map to "auto" rather +// than to their own `CacheDType` spellings, which upstream uses only when the +// user pins the cache to a dtype the model does not have. `vt::DType` carries no +// fp8 member yet, so the quantized names are unreachable from here; when it +// does, this is the one place the mapping goes. +const char* KvCacheDTypeName(vt::DType dtype); + +// vllm/utils/torch_utils.py:75-80 `is_quantized_kv_cache`, over the same name. +bool IsQuantizedKvCacheName(const std::string& kv_cache_dtype); + // Cudagraph support level advertised by an attention metadata builder. // Values preserve upstream ordering; chunked-local explicitly returns kNever. enum class AttentionCGSupport { @@ -233,6 +261,98 @@ class AttentionBackend { // W2 keeps its exact selection behavior. virtual bool is_mla() const { return false; } virtual bool is_sparse() const { return false; } + + // ─── The declared-capability surface (backend.py:154-393) ───────────────── + // + // READ THIS BEFORE TRUSTING A GREEN RESULT FROM IT (issue #1332). Every + // predicate below describes what a backend CLAIMS. None of them can see what + // the shipped binary contains. Upstream's + // FlashAttentionBackend.supports_compute_capability is `>= (8,0)`; it returned + // true on a GB10 (capability 12,1) whose FA2 fatbin holds sm_80 SASS plus + // compute_80 PTX and nothing else, and every launch then failed a driver JIT + // with cudaErrorUnsupportedPtxVersion. `grep -rn get_arch_list vllm/` returns + // zero hits: vLLM never asks what its own fatbins contain, and neither does + // this. The invariant #1332 states is that no backend may be declared valid on + // the strength of a property of the DEVICE alone, and satisfying it needs the + // build-derived compiled-arch manifest (M2) and the launch probe (M3). This + // layer is NECESSARY AND NOT SUFFICIENT, and a reader who takes it for a + // runnability check will reproduce exactly the failure that opened the issue. + // + // Structure mirrors upstream: static declarations returning bool, collected by + // validate_configuration into a list of reason strings; an EMPTY list means + // valid. Ours are virtual member functions because our backends are instances + // rather than classes (MakeAttentionBackend), which is the same adaptation + // get_impl_cls above already records. + + // backend.py:155-156 / :59-64 — the declared value lists. A backend overrides + // the list, not the predicate, wherever upstream does. + virtual std::vector supported_dtypes() const { + return {vt::DType::kF16, vt::DType::kBF16}; // backend.py:59 + } + virtual std::vector supported_kv_cache_dtypes() const { + return {"auto", "float16", "bfloat16"}; // backend.py:60-64 + } + virtual std::vector get_supported_head_sizes() const { return {}; } + // backend.py:69-71 — upstream returns [MultipleOf(1)]. Every entry here IS a + // MultipleOf: supports_block_size accepts any multiple of any entry, which is + // upstream's hybrid_blocks rule at :184-192, and MultipleOf(1) accepts all. + virtual std::vector get_supported_kernel_block_sizes() const { return {1}; } + + // backend.py:158-161. + virtual bool supports_head_size(int head_size) const; + // backend.py:163-165. + virtual bool supports_dtype(vt::DType dtype) const; + // backend.py:167-173 — an EMPTY name is upstream's `None`, accepted outright. + virtual bool supports_kv_cache_dtype(const std::string& kv_cache_dtype) const; + // backend.py:175-192 — 0 is upstream's `None`, accepted outright. + virtual bool supports_block_size(int block_size) const; + + // backend.py:237-298 — the plain feature flags, upstream defaults. + virtual bool supports_sink() const { return false; } // :241-243 + virtual bool supports_mm_prefix() const { return false; } // :249-251 + virtual bool supports_per_head_quant_scales() const { return false; } // :257-259 + virtual bool supports_sliding_window() const { return false; } // :261-263 + virtual bool supports_non_causal() const { return false; } // :265-274 + virtual bool supports_batch_invariance() const { return false; } // :276-278 + virtual bool supports_kv_connector() const { return true; } // :280-282 + virtual bool supports_pcp() const { return false; } // :284-289 + // :291-298 — the base supports DECODER only. + virtual bool supports_attn_type(const std::string& attn_type) const { + return attn_type == AttentionTypeName(AttentionType::kDecoder); + } + // :300-302. + virtual bool supports_compute_capability( + const platforms::DeviceCapability& capability) const { + (void)capability; + return true; + } + + // :304-317 — the cross-field rule a per-field predicate cannot express. + // Returns the reason when the COMBINATION is invalid, nullopt when it is fine. + // + // ADAPTATION: upstream passes nine positional arguments, every one of them a + // field of the AttentionSelectorConfig the caller already holds + // (cuda.py:381-384 splats that config). We pass the config itself, so a new + // upstream field does not change nine signatures. + virtual std::optional supports_combination( + const platforms::AttnSelectorConfig& cfg, + const platforms::DeviceCapability& capability) const { + (void)cfg; + (void)capability; + return std::nullopt; + } + + // backend.py:319-393. Collects one reason per failed predicate and returns + // them ALL: upstream builds a list and prints the whole list (cuda.py:416-420, + // :432-446), because a message that stopped at the first failure would send a + // reader chasing one cause of four. An EMPTY result means the backend is valid + // for this request. + // + // The reason STRINGS are upstream's, byte for byte, so a refusal here and a + // refusal from the oracle read the same. + virtual std::vector validate_configuration( + const platforms::AttnSelectorConfig& cfg, + const platforms::DeviceCapability& capability) const; }; // The T0 concrete full-attention backend. Ports the FlashAttention V1 paged KV @@ -253,6 +373,54 @@ class FlashAttentionBackend final : public AttentionBackend { int64_t num_blocks, int64_t block_size, int64_t num_kv_heads, int64_t head_size, const std::string& cache_dtype_str = "auto") const override; + + // ─── Capability overrides, ported from flash_attn.py:72-239 ────────────── + // flash_attn.py:74-80. + std::vector supported_kv_cache_dtypes() const override { + return {"auto", "float16", "bfloat16", "fp8", "fp8_e4m3"}; + } + // flash_attn.py:82-84 — MultipleOf(16). + std::vector get_supported_kernel_block_sizes() const override { return {16}; } + // flash_attn.py:170-178. Upstream raises the ceiling to 512 when FlashAttention + // v4 resolves (`is_fa_version_supported(4)`); this tree ships FA2 only + // (BACKEND-CUDA-COMP-FA records FA3/FA4 as unported), so the 512 arm is + // unreachable and is deliberately not written as an always-false branch. + bool supports_head_size(int head_size) const override { + if (head_size % 8 != 0) return false; + return head_size <= 256; + } + // flash_attn.py:200-202. SEE THE WARNING ON THE BASE CLASS: this is the exact + // predicate that returned true on a GB10 whose FA2 fatbin had no sm_121 code + // (issue #1332). It is upstream's, and it is not a runnability check. + bool supports_compute_capability( + const platforms::DeviceCapability& capability) const override { + return capability.major > 8 || (capability.major == 8 && capability.minor >= 0); + } + bool supports_sliding_window() const override { return true; } // :98-100 + bool supports_batch_invariance() const override { return true; } // :102-104 + bool supports_non_causal() const override { return true; } // :106-108 + // :110-118 — FlashAttention serves every attention type. + bool supports_attn_type(const std::string& attn_type) const override { + return attn_type == AttentionTypeName(AttentionType::kDecoder) || + attn_type == AttentionTypeName(AttentionType::kEncoder) || + attn_type == AttentionTypeName(AttentionType::kEncoderOnly) || + attn_type == AttentionTypeName(AttentionType::kEncoderDecoder); + } + // :120-123 `get_flash_attn_version() >= 3`. This tree ships FA2, so the answer + // is upstream's own answer for FA2 rather than a divergence — and it is what + // makes the ported test_per_head_quant_scales case assert a refusal. + bool supports_per_head_quant_scales() const override { return false; } + // :204-239. Only the sink rule is expressible here: the fp8-KV and mm_prefix + // rules both key on a resolved FA version >= 3, which this tree never reaches, + // and supported_kv_cache_dtypes / supports_mm_prefix already refuse them. + std::optional supports_combination( + const platforms::AttnSelectorConfig& cfg, + const platforms::DeviceCapability& capability) const override { + if (cfg.has_sink && capability.present() && capability.major < 9) { + return std::string("sink not supported on compute capability < 9.0"); + } + return std::nullopt; + } }; // The dense ROCm paged-attention backend (issue #41 M3). Upstream ROCM_ATTN @@ -348,10 +516,14 @@ class TritonMLABackend final : public AttentionBackend { // makes FLASHINFER_MLA_SPARSE_SM120 lose and TRITON_MLA win on GB10). bool is_mla() const override { return true; } - // triton_mla.py:100-103 supports_block_size — block_size % 16 == 0. Exposed as - // a plain predicate (the full supports_* surface is still deferred, see the - // header note) because get_kv_cache_shape enforces it. - static bool supports_block_size(int64_t block_size) { return block_size % 16 == 0; } + // triton_mla.py:100-103 supports_block_size — block_size % 16 == 0. Declared as + // the SUPPORTED-SIZE LIST rather than as an overridden predicate, which is + // upstream's shape and which the base supports_block_size (backend.py:175-192) + // turns into the same `% 16 == 0` answer. Was a `static` member before #1332; + // a static of that name would have HIDDEN the base virtual, so a caller + // through an AttentionBackend& and a caller through a TritonMLABackend& would + // have got different answers. + std::vector get_supported_kernel_block_sizes() const override { return {16}; } // W4: the MLA decode impl. Upstream `get_impl_cls` returns the CLASS // (triton_mla.py:126-128 -> TritonMLAImpl); here a factory returning an diff --git a/include/vllm/v1/attention/registry.h b/include/vllm/v1/attention/registry.h index 1a1aaebb4..49cb94b3b 100644 --- a/include/vllm/v1/attention/registry.h +++ b/include/vllm/v1/attention/registry.h @@ -1,7 +1,9 @@ // Ported from: vllm/v1/attention/backends/registry.py (AttentionBackendEnum + // @register_backend self-registration) and the platform-driven selection in -// vllm/platforms/cuda.py:361-470 (get_valid_backends / get_attn_backend_cls) @ -// pin e24d1b24 — the attention-backend REGISTRY + platform-priority SELECTION +// vllm/platforms/cuda.py:359-394 (get_valid_backends) and :397-492 +// (get_attn_backend_cls) @ pin 5559679229 — re-anchored from `:361-470 @ +// e24d1b24`, the pin retired at W5 — the attention-backend REGISTRY + +// platform-priority SELECTION // seam (extensibility item 4). This is the ENGINE-level "which AttentionBackend" // seam; the concrete attention KERNEL stays selected at the vt:: op-table level // (vt::PagedAttention -> GetOp(kPagedAttention, device.type)), which is already diff --git a/src/vllm/v1/attention/backend.cpp b/src/vllm/v1/attention/backend.cpp index d1236a99c..7b4eb0b70 100644 --- a/src/vllm/v1/attention/backend.cpp +++ b/src/vllm/v1/attention/backend.cpp @@ -1,5 +1,6 @@ -// Ported from: vllm/v1/attention/backend.py @ e24d1b24 -// (get_kv_cache_shape from vllm/v1/attention/backends/flash_attn.py @ e24d1b24) +// Ported from: vllm/v1/attention/backend.py @ pin 5559679229 +// (get_kv_cache_shape from vllm/v1/attention/backends/flash_attn.py @ e24d1b24 — +// deliberately NOT re-anchored; see the divergence note at the top of backend.h) #include "vllm/v1/attention/backend.h" #include @@ -70,6 +71,159 @@ CommonAttentionMetadata MakeCommonAttentionMetadata( return cam; } +const char* AttentionTypeName(AttentionType type) { + // backend.py:38-46 — the exact upstream string values. + switch (type) { + case AttentionType::kDecoder: + return "decoder"; + case AttentionType::kEncoder: + return "encoder"; + case AttentionType::kEncoderOnly: + return "encoder_only"; + case AttentionType::kEncoderDecoder: + return "encoder_decoder"; + } + return "decoder"; +} + +// vllm/config/cache.py:19-36 CacheDType. See the header for why f32/f16/bf16 all +// map to "auto". +const char* KvCacheDTypeName(vt::DType dtype) { + switch (dtype) { + case vt::DType::kF32: + case vt::DType::kF16: + case vt::DType::kBF16: + return "auto"; + default: + // No other vt::DType is a KV-cache storage dtype in this tree. "auto" is + // the unquantized answer, which is what a caller with an unexpected dtype + // needs the selector to keep doing until the mapping above grows a row. + return "auto"; + } +} + +// vllm/utils/torch_utils.py:75-80. +bool IsQuantizedKvCacheName(const std::string& kv_cache_dtype) { + const auto ends_with = [&](const std::string& suffix) { + return kv_cache_dtype.size() >= suffix.size() && + kv_cache_dtype.compare(kv_cache_dtype.size() - suffix.size(), + suffix.size(), suffix) == 0; + }; + return kv_cache_dtype.rfind("fp8", 0) == 0 || ends_with("per_token_head") || + kv_cache_dtype == "nvfp4"; +} + +// backend.py:158-161 — an EMPTY supported list means "no constraint". +bool AttentionBackend::supports_head_size(int head_size) const { + const std::vector supported = get_supported_head_sizes(); + if (supported.empty()) return true; + return std::find(supported.begin(), supported.end(), head_size) != supported.end(); +} + +// backend.py:163-165. +bool AttentionBackend::supports_dtype(vt::DType dtype) const { + const std::vector supported = supported_dtypes(); + return std::find(supported.begin(), supported.end(), dtype) != supported.end(); +} + +// backend.py:167-173. An EMPTY name is upstream's `None`. +bool AttentionBackend::supports_kv_cache_dtype( + const std::string& kv_cache_dtype) const { + if (kv_cache_dtype.empty()) return true; + const std::vector supported = supported_kv_cache_dtypes(); + if (supported.empty()) return true; + return std::find(supported.begin(), supported.end(), kv_cache_dtype) != + supported.end(); +} + +// backend.py:175-192. Every entry of get_supported_kernel_block_sizes() is an +// upstream MultipleOf, so a framework block size is supported when it is a +// multiple of ANY declared size — upstream's hybrid_blocks rule at :187-191, +// which is why 32 passes a backend declaring MultipleOf(16) and 8 does not. +bool AttentionBackend::supports_block_size(int block_size) const { + if (block_size == 0) return true; // upstream `None` + const std::vector supported = get_supported_kernel_block_sizes(); + if (supported.empty()) return true; + for (const int size : supported) { + if (size != 0 && block_size % size == 0) return true; + } + return false; +} + +// backend.py:319-393, in upstream's order, with upstream's reason strings. +std::vector AttentionBackend::validate_configuration( + const platforms::AttnSelectorConfig& cfg, + const platforms::DeviceCapability& capability) const { + std::vector invalid_reasons; + if (!supports_head_size(cfg.head_size)) { + invalid_reasons.emplace_back("head_size not supported"); + } + if (!supports_dtype(cfg.dtype)) { + invalid_reasons.emplace_back("dtype not supported"); + } + if (!supports_kv_cache_dtype(cfg.kv_cache_dtype)) { + invalid_reasons.emplace_back("kv_cache_dtype not supported"); + } + if (!supports_block_size(cfg.block_size)) { + invalid_reasons.emplace_back("block_size not supported"); + } + if (cfg.use_mm_prefix && !supports_mm_prefix()) { + invalid_reasons.emplace_back( + "partial multimodal token full attention not supported"); + } + if (cfg.use_mla != is_mla()) { + invalid_reasons.emplace_back(cfg.use_mla ? "MLA not supported" + : "non-MLA not supported"); + } + if (cfg.has_sink && !supports_sink()) { + invalid_reasons.emplace_back("attention sinks not supported"); + } + if (cfg.use_sparse != is_sparse()) { + invalid_reasons.emplace_back(cfg.use_sparse ? "sparse not supported" + : "non-sparse not supported"); + } + if (cfg.use_per_head_quant_scales && !supports_per_head_quant_scales()) { + invalid_reasons.emplace_back("per-head quant scales not supported"); + } + // backend.py:366-367, with upstream's own PRECONDITION rather than a new rule: + // CudaPlatform.get_attn_backend_cls asserts `device_capability is not None` + // before it calls this (cuda.py:403-404), and CpuPlatform has a separate + // selector that never reaches it (cpu.py:75-87). Our selector is shared across + // every DeviceType, and DeviceCapability::present() is already false for every + // platform that cannot answer the question, so the predicate applies exactly + // where upstream applies it. Without this, FLASH_ATTN — which this tree also + // registers for kCPU/kMETAL/kVULKAN/kTENSTORRENT — would be refused on every + // one of them by a rule about NVIDIA compute capability. + if (capability.present() && !supports_compute_capability(capability)) { + invalid_reasons.emplace_back("compute capability not supported"); + } + if (!supports_attn_type(cfg.attn_type)) { + invalid_reasons.emplace_back("attention type " + cfg.attn_type + + " not supported"); + } + if (cfg.has_sliding_window && !supports_sliding_window()) { + invalid_reasons.emplace_back("sliding window not supported"); + } + if (cfg.use_non_causal && !supports_non_causal()) { + invalid_reasons.emplace_back("non-causal attention not supported"); + } + if (cfg.use_batch_invariant && !supports_batch_invariance()) { + invalid_reasons.emplace_back("batch invariance not supported"); + } + if (cfg.use_kv_connector && !supports_kv_connector()) { + invalid_reasons.emplace_back("KV connector not supported"); + } + if (cfg.use_pcp && !supports_pcp()) { + invalid_reasons.emplace_back("PCP not supported"); + } + if (const std::optional combination = + supports_combination(cfg, capability); + combination.has_value()) { + invalid_reasons.push_back(*combination); + } + return invalid_reasons; +} + std::vector FlashAttentionBackend::get_kv_cache_shape( int64_t num_blocks, int64_t block_size, int64_t num_kv_heads, int64_t head_size, const std::string& /*cache_dtype_str*/) const { @@ -98,8 +252,9 @@ std::vector RocmAttentionBackend::get_kv_cache_shape( std::vector TritonMLABackend::get_kv_cache_shape( int64_t num_blocks, int64_t block_size, int64_t num_kv_heads, int64_t head_size, const std::string& /*cache_dtype_str*/) const { - // triton_mla.py:100-103 supports_block_size. - if (!supports_block_size(block_size)) { + // triton_mla.py:100-103 supports_block_size — now the base predicate reading + // this class's get_supported_kernel_block_sizes() == {16}. + if (!supports_block_size(static_cast(block_size))) { throw std::invalid_argument("Block size must be a multiple of 16."); } // mla_attention.py:1219 — "num_kv_heads ... assumed to be 1 for MLA". Upstream diff --git a/src/vllm/v1/attention/registry.cpp b/src/vllm/v1/attention/registry.cpp index c068b8f60..4dd4960c0 100644 --- a/src/vllm/v1/attention/registry.cpp +++ b/src/vllm/v1/attention/registry.cpp @@ -1,7 +1,9 @@ // Ported from: vllm/v1/attention/backends/registry.py + vllm/platforms/cuda.py -// :361-470 (get_valid_backends / get_attn_backend_cls) @ pin e24d1b24 — the -// attention-backend registry storage + platform-priority selection. See -// registry.h for the design; this file is the registry table and the selector. +// :359-394 (get_valid_backends) and :397-492 (get_attn_backend_cls) @ pin +// 5559679229 — the attention-backend registry storage + platform-priority +// selection. (Re-anchored from `:361-470 @ e24d1b24`, the pin retired at W5.) +// See registry.h for the design; this file is the registry table and the +// selector. #include "vllm/v1/attention/registry.h" #include @@ -55,15 +57,32 @@ std::unique_ptr MakeAttentionBackend(vt::DeviceType device, namespace { -// The capability half of vllm/v1/attention/backend.py:307-360 -// validate_configuration: a candidate must agree with the request on is_mla() -// and is_sparse(). Backends are stateless descriptors, so constructing one to -// ask is cheap and mirrors upstream querying the CLASS. This is the ONLY place -// sparse/DSA needs to be understood — see registry.h "the DSA seam". -bool CandidateMatchesConfig(vt::DeviceType device, const std::string& name, - const platforms::AttnSelectorConfig& cfg) { +// vllm/v1/attention/backend.py:319-393 `validate_configuration`, asked of one +// candidate. Backends are stateless descriptors, so constructing one to ask is +// cheap and mirrors upstream querying the CLASS. Returns upstream's list of +// reasons; EMPTY means the candidate is valid for this request. +// +// Before #1332 M1 this asked two questions inline (is_mla / is_sparse). Those +// two are still asked, by validate_configuration, in upstream's order and with +// upstream's reason strings — the sparse/DSA seam registry.h describes is +// unchanged, it just no longer has its own private copy of the rule. +std::vector CandidateInvalidReasons( + vt::DeviceType device, const std::string& name, + const platforms::Platform& platform, + const platforms::AttnSelectorConfig& cfg) { const std::unique_ptr backend = MakeAttentionBackend(device, name); - return backend->is_mla() == cfg.use_mla && backend->is_sparse() == cfg.use_sparse; + return backend->validate_configuration(cfg, platform.get_device_capability()); +} + +// `[reason, reason, ...]`, upstream's own rendering (cuda.py:433-438). +std::string JoinReasons(const std::vector& reasons) { + std::string out = "["; + for (size_t i = 0; i < reasons.size(); ++i) { + if (i != 0) out += ", "; + out += reasons[i]; + } + out += "]"; + return out; } } // namespace @@ -84,12 +103,13 @@ std::string SelectAttentionBackendName(const platforms::Platform& platform, "' is not registered for device type " + std::to_string(static_cast(device))); } - if (!CandidateMatchesConfig(device, selected, cfg)) { + const std::vector reasons = + CandidateInvalidReasons(device, selected, platform, cfg); + if (!reasons.empty()) { + // cuda.py:416-420 — the error names every reason, not the first. throw std::invalid_argument( - std::string("selected attention backend '") + selected + - "' does not satisfy the request (use_mla=" + - (cfg.use_mla ? "true" : "false") + - ", use_sparse=" + (cfg.use_sparse ? "true" : "false") + ")"); + std::string("Selected backend ") + selected + + " is not valid for this configuration. Reason: " + JoinReasons(reasons)); } return selected; } @@ -99,17 +119,24 @@ std::string SelectAttentionBackendName(const platforms::Platform& platform, // backend). An unregistered name is skipped exactly as an ImportError-ing // backend is in get_valid_backends; a registered-but-mismatched one is skipped // exactly as validate_configuration's "invalid reasons" reject it. + std::string all_invalid_reasons; for (const std::string& name : platform.get_attn_backend_priority(cfg)) { if (!HasAttentionBackend(device, name)) continue; - if (!CandidateMatchesConfig(device, name, cfg)) continue; - return name; + const std::vector reasons = + CandidateInvalidReasons(device, name, platform, cfg); + if (reasons.empty()) return name; + // cuda.py:387-388 — a rejected candidate is REMEMBERED with its reasons, so + // the exhaustion message can say why each one lost rather than only that + // none won. + if (!all_invalid_reasons.empty()) all_invalid_reasons += ", "; + all_invalid_reasons += name + ": " + JoinReasons(reasons); } + // cuda.py:440-446. throw std::runtime_error( - std::string("no valid attention backend registered for device type ") + - std::to_string(static_cast(device)) + - " (priority list yielded no registered backend; use_mla=" + - (cfg.use_mla ? "true" : "false") + + std::string("No valid attention backend for device type ") + + std::to_string(static_cast(device)) + " from {" + all_invalid_reasons + + "} (use_mla=" + (cfg.use_mla ? "true" : "false") + ", use_sparse=" + (cfg.use_sparse ? "true" : "false") + ")"); } diff --git a/src/vllm/v1/worker/gpu/runner.cpp b/src/vllm/v1/worker/gpu/runner.cpp index 4ac0cebf5..9478c25b3 100644 --- a/src/vllm/v1/worker/gpu/runner.cpp +++ b/src/vllm/v1/worker/gpu/runner.cpp @@ -976,11 +976,31 @@ void GPUModelRunner::initialize_kv_cache(const KVCacheConfig& kv_cache_config) { // A future backend with a different layout fails LOUDLY here at init. const bool is_mla = mla_layer_mask[static_cast(i)] != 0; + // #1332 M1: the selector now applies the full validate_configuration + // capability surface, so the request it is asked has to BE the request. The + // three fields this site can answer come straight from the geometry it just + // resolved. `dtype` (the model/query dtype) is NOT available here — the + // runner resolves only ResolveKvCacheDType() — so it keeps its bf16 default; + // that is owed to #1332 M4 and recorded under `## Owed` in + // .agents/specs/attn-validate-configuration.md. + // + // AND READ THIS BEFORE READING A GREEN SELECTION AS A WORKING BACKEND: the + // name resolved here still reaches only attn_backend_names_, the + // VT_ATTN_SELECT_LOG print below and CheckKvCacheShape. dense_attn::AttnBlock + // calls vt::PagedAttention unconditionally. Nothing DISPATCHES on this. #1332 + // M4 owns that, and until it lands a valid name is a claim, not a route. + vllm::platforms::AttnSelectorConfig cfg; + cfg.head_size = static_cast(fa_dims[i].head_size); + cfg.num_heads = static_cast(fa_dims[i].num_kv_heads); + cfg.block_size = static_cast(fa_block_size); + cfg.kv_cache_dtype = vllm::v1::KvCacheDTypeName(fa_dims[i].dtype); + cfg.quantized_kv_cache = vllm::v1::IsQuantizedKvCacheName(cfg.kv_cache_dtype); + std::string name; if (is_mla) { if (!mla_backend_resolved) { mla_backend_resolved = true; - vllm::platforms::AttnSelectorConfig mla_cfg; + vllm::platforms::AttnSelectorConfig mla_cfg = cfg; mla_cfg.use_mla = true; try { mla_backend = vllm::v1::SelectAttentionBackendName( @@ -995,7 +1015,7 @@ void GPUModelRunner::initialize_kv_cache(const KVCacheConfig& kv_cache_config) { if (!dense_backend_resolved) { dense_backend_resolved = true; dense_backend = vllm::v1::SelectAttentionBackendName( - vllm::platforms::GetPlatform(queue_.device.type)); + vllm::platforms::GetPlatform(queue_.device.type), "", cfg); } name = dense_backend; } diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index bed1f3dd6..a38f3672c 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -1256,6 +1256,8 @@ vllm_cpp_add_test(test_runner vllm/v1/worker/test_runner.cpp) target_include_directories(test_runner PRIVATE ${CMAKE_SOURCE_DIR}/src) vllm_cpp_add_test(test_common_attn_metadata vllm/v1/attention/test_common_attn_metadata.cpp) vllm_cpp_add_test(test_attn_backend_registry vllm/v1/attention/test_attn_backend_registry.cpp) +vllm_cpp_add_test(test_attn_validate_configuration + vllm/v1/attention/test_attn_validate_configuration.cpp) vllm_cpp_add_test(test_attention_window vllm/model_executor/layers/attention/test_attention.cpp) vllm_cpp_add_test(test_chunked_local_attention vllm/v1/attention/test_chunked_local_attention.cpp) diff --git a/tests/vllm/v1/attention/test_attn_validate_configuration.cpp b/tests/vllm/v1/attention/test_attn_validate_configuration.cpp new file mode 100644 index 000000000..9986b1ed4 --- /dev/null +++ b/tests/vllm/v1/attention/test_attn_validate_configuration.cpp @@ -0,0 +1,481 @@ +// Ported from: tests/kernels/attention/test_attention_selector.py @ pin +// 5559679229 — the executable spec of +// `vllm/v1/attention/backend.py::AttentionBackend.validate_configuration` +// (`:320-393`) and the `supports_*` predicates it collects (`:154-317`), as +// applied by `vllm/platforms/cuda.py::get_valid_backends` (`:359-394`) and +// `get_attn_backend_cls` (`:397-492`). +// +// Every upstream case in that file monkeypatches the device capability and then +// asserts a chosen backend NAME, so the whole file is CPU-runnable: the only +// thing it needs from a GPU is a number, and `FakeCudaPlatform` below takes that +// number as a constructor argument. +// +// WHAT THIS FILE DOES NOT MEASURE (issue #1332, and the reason that issue +// exists). These predicates describe what a backend CLAIMS. They cannot see what +// the shipped binary contains, and they cannot see which kernel actually runs: +// `dense_attn::AttnBlock` calls `vt::PagedAttention` unconditionally, so the +// name selected here dispatches nothing. Upstream's +// `FlashAttentionBackend.supports_compute_capability` is `>= (8,0)`; it passed on +// a GB10 (12,1) whose FA2 fatbin holds `sm_80` SASS and `compute_80` PTX alone, +// and every launch then failed a driver JIT. A green run of this file is +// necessary and NOT sufficient. See `.agents/specs/attn-validate-configuration.md`. +#include + +#include +#include +#include +#include + +#include "vllm/platforms/cuda_attn_priority.h" +#include "vllm/platforms/interface.h" +#include "vllm/v1/attention/backend.h" +#include "vllm/v1/attention/registry.h" +#include "vt/backend.h" +#include "vt/device.h" +#include "vt/dtype.h" + +using vllm::platforms::AttnSelectorConfig; +using vllm::platforms::DeviceCapability; +using vllm::platforms::Platform; +using vllm::platforms::ResidencyPolicy; +using vllm::v1::AttentionBackend; +using vllm::v1::MakeAttentionBackend; +using vllm::v1::SelectAttentionBackendName; +using vt::DeviceType; +using vt::DType; + +namespace { + +// The capability-injecting platform of the upstream test's +// `patch("vllm.platforms.current_platform", CudaPlatform())` plus +// `monkeypatch.setattr(torch.cuda, "get_device_capability", ...)`. It delegates +// to the REAL shared priority table so no list is hand-copied here. +class FakeCudaPlatform final : public Platform { + public: + explicit FakeCudaPlatform(int major, int minor) : cap_{major, minor} {} + DeviceType device_type() const override { return DeviceType::kCUDA; } + vt::Backend& backend() const override { return vt::GetBackend(DeviceType::kCPU); } + DeviceCapability get_device_capability() const override { return cap_; } + std::vector supported_dtypes() const override { return {DType::kBF16}; } + ResidencyPolicy residency_policy() const override { return {}; } + std::vector get_attn_backend_priority( + const AttnSelectorConfig& cfg) const override { + return vllm::platforms::LookupAttnPriority(cap_.major, cfg); + } + + private: + DeviceCapability cap_; +}; + +// The request the gate models actually make on GB10: Qwen3-dense geometry, +// bf16, framework block size 16, plain causal decoder attention. Every case +// below starts from this and changes ONE field, so a refusal is attributable. +AttnSelectorConfig GateModelConfig() { + AttnSelectorConfig cfg; + cfg.head_size = 128; + cfg.dtype = DType::kBF16; + cfg.kv_cache_dtype = "auto"; + cfg.block_size = 16; + return cfg; +} + +} // namespace + +TEST_CASE("the gate-model request still selects FLASH_ATTN (behavior control)") { + // THE control for every refusal below: a fully populated, valid config + // resolves exactly what the runner resolves today. If this case ever goes red, + // the capability layer has changed production selection, which is a behavior + // change and not a capability port. + FakeCudaPlatform sm121(12, 1); + CHECK(SelectAttentionBackendName(sm121, "", GateModelConfig()) == "FLASH_ATTN"); + + // And the all-defaults config, which is what every pre-existing call site + // passes, is unchanged too. + CHECK(SelectAttentionBackendName(sm121, "", AttnSelectorConfig{}) == "FLASH_ATTN"); +} + +TEST_CASE("test_flash_attn: an unsupported compute capability refuses FLASH_ATTN") { + // test_attention_selector.py:277-280 — `get_device_capability -> (7, 5)` then + // `assert backend.get_name() != "FLASH_ATTN"`. Upstream's predicate is + // `capability >= DeviceCapability(8, 0)` (flash_attn.py:200-202). sm_75 is + // Turing, the last generation before FlashAttention's floor. + FakeCudaPlatform sm75(7, 5); + CHECK_THROWS_AS(SelectAttentionBackendName(sm75, "", GateModelConfig()), + std::runtime_error); +} + +TEST_CASE("test_flash_attn: an unsupported model dtype refuses FLASH_ATTN") { + // test_attention_selector.py:285-287 (`torch.float8_e4m3fn`) and + // test_fp32_fallback:242-247. `supported_dtypes` is `[float16, bfloat16]` + // (backend.py:59, flash_attn.py:73). Upstream's fp32 request lands on + // FLEX_ATTENTION; this tree registers no FLEX_ATTENTION, so the walk is + // exhausted and refuses — which is the correct answer, not a fallback. + FakeCudaPlatform sm121(12, 1); + AttnSelectorConfig f32 = GateModelConfig(); + f32.dtype = DType::kF32; + CHECK_THROWS_AS(SelectAttentionBackendName(sm121, "", f32), std::runtime_error); + + // HARNESS ADAPTATION: upstream's second parameter here is + // `torch.float8_e4m3fn`, and `vt::DType` carries no fp8 member, so the second + // "outside the supported list" value is `kI8`. The predicate under test is a + // list membership either way, and keeping a second value proves the check is + // the list rather than an `== kF32` special case. + AttnSelectorConfig i8 = GateModelConfig(); + i8.dtype = DType::kI8; + CHECK_THROWS_AS(SelectAttentionBackendName(sm121, "", i8), std::runtime_error); +} + +TEST_CASE("test_flash_attn: an unsupported kv-cache dtype refuses FLASH_ATTN") { + // test_attention_selector.py:289-291. FLASH_ATTN's + // `supported_kv_cache_dtypes` is ["auto","float16","bfloat16","fp8","fp8_e4m3"] + // (flash_attn.py:74-80), so `fp8_e5m2` is outside it. + FakeCudaPlatform sm121(12, 1); + AttnSelectorConfig cfg = GateModelConfig(); + cfg.kv_cache_dtype = "fp8_e5m2"; + CHECK_THROWS_AS(SelectAttentionBackendName(sm121, "", cfg), std::runtime_error); +} + +TEST_CASE("test_flash_attn: an unsupported block size refuses FLASH_ATTN") { + // test_attention_selector.py:293-296 (`block_size = 8`). FLASH_ATTN declares + // `MultipleOf(16)` (flash_attn.py:82-84), and `supports_block_size` accepts any + // MULTIPLE of a declared size (backend.py:184-192), so 32 is fine and 8 is not. + FakeCudaPlatform sm121(12, 1); + AttnSelectorConfig bad = GateModelConfig(); + bad.block_size = 8; + CHECK_THROWS_AS(SelectAttentionBackendName(sm121, "", bad), std::runtime_error); + + AttnSelectorConfig ok = GateModelConfig(); + ok.block_size = 32; + CHECK(SelectAttentionBackendName(sm121, "", ok) == "FLASH_ATTN"); +} + +TEST_CASE("test_flash_attn: an unsupported head size refuses FLASH_ATTN") { + // test_attention_selector.py:313-315 (`head_size = 17`). Upstream: + // `head_size % 8 != 0 -> False`, `<= 256 -> True` (flash_attn.py:170-178). + FakeCudaPlatform sm121(12, 1); + AttnSelectorConfig odd = GateModelConfig(); + odd.head_size = 17; + CHECK_THROWS_AS(SelectAttentionBackendName(sm121, "", odd), std::runtime_error); + + // 264 is a multiple of 8 and ABOVE the 256 ceiling — the second half of the + // predicate, which a `% 8` test alone would not reach. Upstream raises the + // ceiling to 512 only when FlashAttention v4 resolves, and this tree ships FA2. + AttnSelectorConfig big = GateModelConfig(); + big.head_size = 264; + CHECK_THROWS_AS(SelectAttentionBackendName(sm121, "", big), std::runtime_error); + + // 256 itself is inside it. + AttnSelectorConfig edge = GateModelConfig(); + edge.head_size = 256; + CHECK(SelectAttentionBackendName(sm121, "", edge) == "FLASH_ATTN"); +} + +TEST_CASE("test_per_head_quant_scales_backend_selection: FA2 refuses per-head scales") { + // test_attention_selector.py:369-427, the `("FLASH_ATTN", 2, False)` row: + // FA2 does not support per-head quant scales. Upstream's predicate is + // `get_flash_attn_version() >= 3` (flash_attn.py:120-123); this tree ships FA2 + // only, so the answer is upstream's own answer for FA2. + FakeCudaPlatform sm121(12, 1); + AttnSelectorConfig cfg = GateModelConfig(); + cfg.kv_cache_dtype = "fp8"; + cfg.use_per_head_quant_scales = true; + CHECK_THROWS_AS(SelectAttentionBackendName(sm121, "", cfg), std::runtime_error); +} + +TEST_CASE("test_non_causal_backend_selection: FLASH_ATTN serves a non-causal request") { + // test_attention_selector.py:430-487, the two FLASH_ATTN rows: FlashAttention + // supports non-causal (flash_attn.py:106-108), so a drafter's non-causal + // request selects it, and so does an ordinary causal one. + FakeCudaPlatform sm121(12, 1); + AttnSelectorConfig non_causal = GateModelConfig(); + non_causal.use_non_causal = true; + CHECK(SelectAttentionBackendName(sm121, "", non_causal) == "FLASH_ATTN"); +} + +TEST_CASE("a sliding-window request refuses a backend that does not declare it") { + // backend.py:370-371 `has_sliding_window and not supports_sliding_window()`. + // FLASH_ATTN declares it (flash_attn.py:98-100), so the positive control + // selects; GDN_ATTN inherits the base `false`, so a platform preferring GDN + // for a sliding-window request finds nothing valid. + FakeCudaPlatform sm121(12, 1); + AttnSelectorConfig sw = GateModelConfig(); + sw.has_sliding_window = true; + CHECK(SelectAttentionBackendName(sm121, "", sw) == "FLASH_ATTN"); + + class GdnOnlyPlatform final : public Platform { + public: + DeviceType device_type() const override { return DeviceType::kCPU; } + vt::Backend& backend() const override { return vt::GetBackend(DeviceType::kCPU); } + DeviceCapability get_device_capability() const override { return {}; } + std::vector supported_dtypes() const override { return {DType::kBF16}; } + ResidencyPolicy residency_policy() const override { return {}; } + std::vector get_attn_backend_priority( + const AttnSelectorConfig& cfg) const override { + (void)cfg; + return {"GDN_ATTN"}; + } + } gdn; + CHECK(SelectAttentionBackendName(gdn, "", GateModelConfig()) == "GDN_ATTN"); + CHECK_THROWS_AS(SelectAttentionBackendName(gdn, "", sw), std::runtime_error); +} + +TEST_CASE("the refusal names every failed predicate, not only the first") { + // Upstream builds a LIST (`invalid_reasons`) and prints all of it + // (cuda.py:416-420, :432-446). A message that collapsed to the first failure + // would send a reader chasing one cause of four. Three violations at once: + // head size, block size and kv-cache dtype. + FakeCudaPlatform sm121(12, 1); + AttnSelectorConfig bad = GateModelConfig(); + bad.head_size = 17; + bad.block_size = 8; + bad.kv_cache_dtype = "fp8_e5m2"; + + std::string message; + try { + SelectAttentionBackendName(sm121, "", bad); + } catch (const std::exception& e) { + message = e.what(); + } + REQUIRE_FALSE(message.empty()); + CHECK(message.find("head_size not supported") != std::string::npos); + CHECK(message.find("block_size not supported") != std::string::npos); + CHECK(message.find("kv_cache_dtype not supported") != std::string::npos); +} + +TEST_CASE("an explicit override that fails validation is refused by reason") { + // cuda.py:407-420 — a `selected_backend` is validated too, and the error names + // the reasons rather than silently falling through to the priority walk. + FakeCudaPlatform sm121(12, 1); + AttnSelectorConfig odd = GateModelConfig(); + odd.head_size = 17; + + std::string message; + try { + SelectAttentionBackendName(sm121, "FLASH_ATTN", odd); + } catch (const std::exception& e) { + message = e.what(); + } + REQUIRE_FALSE(message.empty()); + CHECK(message.find("FLASH_ATTN") != std::string::npos); + CHECK(message.find("head_size not supported") != std::string::npos); +} + +TEST_CASE("#1333: sm_100 prefers FLASH_ATTN for a non-causal request") { + // cuda.py:144-163 @ pin 5559679229. The sm_100 non-MLA arm is guarded by + // `and not use_non_causal`, because SM100f's non-causal cutlass path — the one + // DFlash attention uses — is known-bad (`:145-147`). Causal keeps FlashInfer + // first; non-causal falls to the `else` arm, FlashAttention first. + FakeCudaPlatform sm100(10, 0); + const std::vector causal{"FLASHINFER", "FLASH_ATTN", "TRITON_ATTN", + "FLEX_ATTENTION", "TURBOQUANT"}; + const std::vector non_causal{"FLASH_ATTN", "FLASHINFER", "TRITON_ATTN", + "FLEX_ATTENTION", "TURBOQUANT"}; + AttnSelectorConfig plain; + CHECK(sm100.get_attn_backend_priority(plain) == causal); + + AttnSelectorConfig nc; + nc.use_non_causal = true; + CHECK(sm100.get_attn_backend_priority(nc) == non_causal); + + // sm_121 is in the `else` arm either way, so the guard moves nothing there. + FakeCudaPlatform sm121(12, 1); + CHECK(sm121.get_attn_backend_priority(plain) == non_causal); + CHECK(sm121.get_attn_backend_priority(nc) == non_causal); +} + +// ─── The predicate-level upstream cases, asked of the backend directly ─────── + +TEST_CASE("test_flash_attn_rejects_unhandled_kv_cache_dtypes") { + // test_attention_selector.py:524-540, all six upstream parameters. The point + // upstream makes in its own docstring: FlashAttentionBackend must not CLAIM + // support for kv_cache dtypes it cannot handle. Our list is upstream's + // (flash_attn.py:74-80), so the same six fall outside it. + const std::unique_ptr fa = + MakeAttentionBackend(DeviceType::kCUDA, "FLASH_ATTN"); + REQUIRE(fa != nullptr); + for (const char* name : {"fp8_e5m2", "fp8_ds_mla", "fp8_inc", "nvfp4", + "fp8_per_token_head", "int8_per_token_head"}) { + CAPTURE(name); + CHECK_FALSE(fa->supports_kv_cache_dtype(name)); + } +} + +TEST_CASE("test_flash_attn_accepts_handled_fp8_variants") { + // test_attention_selector.py:543-556, both upstream parameters: "fp8" (the + // alias for fp8_e4m3fn) and "fp8_e4m3" are the two FlashAttention can handle. + // Upstream needs an `is_xpu` monkeypatch to reach the answer through + // fa_utils; our predicate is the plain list membership it resolves to. + const std::unique_ptr fa = + MakeAttentionBackend(DeviceType::kCUDA, "FLASH_ATTN"); + REQUIRE(fa != nullptr); + CHECK(fa->supports_kv_cache_dtype("fp8")); + CHECK(fa->supports_kv_cache_dtype("fp8_e4m3")); + // backend.py:169-170 — upstream's `None`, which our empty string models. + CHECK(fa->supports_kv_cache_dtype("")); +} + +TEST_CASE("validate_configuration returns an EMPTY list for a valid request") { + // backend.py:339,393 — the empty list IS the "valid" answer, and the selector + // reads emptiness rather than a bool. A predicate that returned a one-element + // list of "" would select nothing while looking correct here, so the size is + // asserted, not just the truthiness. + const std::unique_ptr fa = + MakeAttentionBackend(DeviceType::kCUDA, "FLASH_ATTN"); + REQUIRE(fa != nullptr); + const std::vector reasons = + fa->validate_configuration(GateModelConfig(), DeviceCapability{12, 1}); + CHECK(reasons.empty()); +} + +TEST_CASE("validate_configuration collects ONE reason per failed predicate") { + // backend.py:339-393. Four independent violations produce four reasons, in + // upstream's order, with upstream's strings. A short-circuiting implementation + // returns one; a mis-ordered one fails the sequence. + const std::unique_ptr fa = + MakeAttentionBackend(DeviceType::kCUDA, "FLASH_ATTN"); + REQUIRE(fa != nullptr); + AttnSelectorConfig bad = GateModelConfig(); + bad.head_size = 17; // :340-341 + bad.dtype = DType::kF32; // :342-343 + bad.kv_cache_dtype = "nvfp4"; // :344-345 + bad.block_size = 8; // :346-347 + const std::vector reasons = + fa->validate_configuration(bad, DeviceCapability{12, 1}); + const std::vector expected{ + "head_size not supported", "dtype not supported", + "kv_cache_dtype not supported", "block_size not supported"}; + CHECK(reasons == expected); +} + +TEST_CASE("validate_configuration keys MLA and sparse on equality, both ways") { + // backend.py:352-363 — the check is `use_mla != is_mla()`, not "backend is + // MLA", so a dense backend asked for MLA and an MLA backend asked for dense + // BOTH fail, with different strings. This is the pair the pre-#1332 selector + // asked inline, so it is the behavior that must not have moved. + const std::unique_ptr fa = + MakeAttentionBackend(DeviceType::kCUDA, "FLASH_ATTN"); + const std::unique_ptr mla = + MakeAttentionBackend(DeviceType::kCUDA, "TRITON_MLA"); + REQUIRE(fa != nullptr); + REQUIRE(mla != nullptr); + + AttnSelectorConfig want_mla = GateModelConfig(); + want_mla.use_mla = true; + const std::vector fa_reasons = + fa->validate_configuration(want_mla, DeviceCapability{12, 1}); + CHECK(fa_reasons == std::vector{"MLA not supported"}); + + // TRITON_MLA asked for a DENSE request says the opposite thing. Its head size + // and block size are the MLA geometry, so the request is otherwise valid. + AttnSelectorConfig dense = GateModelConfig(); + dense.head_size = 576; + const std::vector mla_reasons = + mla->validate_configuration(dense, DeviceCapability{12, 1}); + CHECK(mla_reasons == std::vector{"non-MLA not supported"}); + + AttnSelectorConfig sparse = want_mla; + sparse.head_size = 576; + sparse.use_sparse = true; + const std::vector sparse_reasons = + mla->validate_configuration(sparse, DeviceCapability{12, 1}); + CHECK(sparse_reasons == std::vector{"sparse not supported"}); +} + +TEST_CASE("the compute-capability predicate is skipped when there is none") { + // backend.py:366-367 applies `supports_compute_capability` unconditionally, + // and CudaPlatform.get_attn_backend_cls guarantees a capability before calling + // (cuda.py:403-404) while CpuPlatform never calls it at all (cpu.py:75-87). + // Our selector is shared, and this tree registers FLASH_ATTN for kCPU too, so + // a default-constructed (absent) DeviceCapability must NOT refuse it — that + // would refuse the CPU tier on a rule about NVIDIA compute capability. + const std::unique_ptr fa = + MakeAttentionBackend(DeviceType::kCPU, "FLASH_ATTN"); + REQUIRE(fa != nullptr); + CHECK_FALSE(DeviceCapability{}.present()); + CHECK(fa->validate_configuration(GateModelConfig(), DeviceCapability{}).empty()); + + // And a capability that IS present is still enforced. + const std::vector turing = + fa->validate_configuration(GateModelConfig(), DeviceCapability{7, 5}); + CHECK(turing == std::vector{"compute capability not supported"}); + // 8.0 is the floor itself, so it passes (flash_attn.py:200-202 is `>=`). + CHECK(fa->validate_configuration(GateModelConfig(), DeviceCapability{8, 0}).empty()); +} + +TEST_CASE("attn_type is upstream's STRING, and the base serves decoder alone") { + // backend.py:291-298 + :38-46. FLASH_ATTN overrides to all four + // (flash_attn.py:110-118); GDN_ATTN inherits the base, which is decoder-only. + CHECK(std::string(vllm::v1::AttentionTypeName(vllm::v1::AttentionType::kDecoder)) == + "decoder"); + CHECK(std::string(vllm::v1::AttentionTypeName( + vllm::v1::AttentionType::kEncoderOnly)) == "encoder_only"); + + const std::unique_ptr fa = + MakeAttentionBackend(DeviceType::kCUDA, "FLASH_ATTN"); + const std::unique_ptr gdn = + MakeAttentionBackend(DeviceType::kCPU, "GDN_ATTN"); + REQUIRE(fa != nullptr); + REQUIRE(gdn != nullptr); + for (const char* type : {"decoder", "encoder", "encoder_only", "encoder_decoder"}) { + CAPTURE(type); + CHECK(fa->supports_attn_type(type)); + } + CHECK(gdn->supports_attn_type("decoder")); + CHECK_FALSE(gdn->supports_attn_type("encoder_only")); + + AttnSelectorConfig encoder = GateModelConfig(); + encoder.attn_type = "encoder_only"; + const std::vector reasons = + gdn->validate_configuration(encoder, DeviceCapability{}); + CHECK(reasons == std::vector{"attention type encoder_only not supported"}); +} + +TEST_CASE("supports_combination is asked, and FLASH_ATTN refuses a pre-Hopper sink") { + // flash_attn.py:204-218 — the cross-field rule no per-field predicate can + // express: sinks need compute capability >= 9.0. supports_sink() is false in + // this tree (no sink kernel), so the sink flag is refused by the per-field + // predicate first and BOTH reasons appear — which is the list shape, again. + const std::unique_ptr fa = + MakeAttentionBackend(DeviceType::kCUDA, "FLASH_ATTN"); + REQUIRE(fa != nullptr); + AttnSelectorConfig sink = GateModelConfig(); + sink.has_sink = true; + const std::vector reasons = + fa->validate_configuration(sink, DeviceCapability{8, 6}); + const std::vector expected{ + "attention sinks not supported", + "sink not supported on compute capability < 9.0"}; + CHECK(reasons == expected); + + // On Blackwell the combination rule is silent, so only the per-field one + // remains — proving the combination hook is genuinely consulted rather than + // being a constant. + const std::vector blackwell = + fa->validate_configuration(sink, DeviceCapability{10, 0}); + CHECK(blackwell == std::vector{"attention sinks not supported"}); +} + +TEST_CASE("block_size accepts any MULTIPLE of a declared size (hybrid blocks)") { + // backend.py:184-192 — the rule is `block_size % supported == 0`, NOT + // equality, because the framework block size only has to be a multiple of the + // kernel's requirement. A `== 16` implementation passes 16 and fails 32. + const std::unique_ptr fa = + MakeAttentionBackend(DeviceType::kCUDA, "FLASH_ATTN"); + REQUIRE(fa != nullptr); + CHECK(fa->supports_block_size(16)); + CHECK(fa->supports_block_size(32)); + CHECK(fa->supports_block_size(64)); + CHECK_FALSE(fa->supports_block_size(8)); + CHECK_FALSE(fa->supports_block_size(24)); + // backend.py:177-178 — upstream's `None`. + CHECK(fa->supports_block_size(0)); + + // TRITON_MLA declares the same MultipleOf(16) through the LIST, and it is now + // the base virtual rather than a static that hid it: the answer must be the + // same through an AttentionBackend& as it was through the concrete type. + const std::unique_ptr mla = + MakeAttentionBackend(DeviceType::kCUDA, "TRITON_MLA"); + REQUIRE(mla != nullptr); + CHECK(mla->supports_block_size(16)); + CHECK_FALSE(mla->supports_block_size(24)); +} From 63ff582725c083b1ef8087ef0470fdc7f9f6710a Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Wed, 19 Aug 2026 11:41:46 +0000 Subject: [PATCH 3/3] =?UTF-8?q?spec(BACKEND-ATTN-REGISTRY):=20the=20GB10?= =?UTF-8?q?=20pair=20=E2=80=94=20vLLM's=20priority-0=20backend=20is=20unru?= =?UTF-8?q?nnable=20and=20its=20priority-1=20backend=20works=20(#1332)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The argument that a declared capability cannot answer a runnability question was recorded from source alone. It now has its empirical half, and it is sharper than the reading was. Measured through the pinned oracle on a GB10 board (compute capability 12,1), same wheel and same prompt, one variable. Asking for FLASHINFER generates `' Paris. The capital of France is also the capital of the Republic of France.'` and exits 0. The default resolves FLASH_ATTN — upstream's priority 0 for this device — and dies at the FIRST attention call with `cudaErrorUnsupportedPtxVersion`. So the priority-0 choice is unrunnable on the board while the priority-1 choice works, and every predicate this row just ported passes BOTH: each declares a compute-capability floor at or below 12,1, and neither is ever asked what its own fatbin contains. That is the gap stated as a pair of observations rather than as an argument, and it is why M1 must not be reported as fixing selection. It is motivation for M2 and M3, which are NOT in this change. Recorded in the row spec beside the owed debt, and in the warning on the `AttentionBackend` capability surface, which is what an implementer reads before trusting a predicate. No host name, share path or address is written, per #1190. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-4-6 [Claude Code] --- .agents/specs/attn-validate-configuration.md | 19 +++++++++++++++++++ docs/USAGE.md | 20 +++++++++++++++----- include/vllm/v1/attention/backend.h | 8 +++++++- 3 files changed, 41 insertions(+), 6 deletions(-) diff --git a/.agents/specs/attn-validate-configuration.md b/.agents/specs/attn-validate-configuration.md index 484f7f15e..31c74dcf9 100644 --- a/.agents/specs/attn-validate-configuration.md +++ b/.agents/specs/attn-validate-configuration.md @@ -63,6 +63,25 @@ own it already exist.** No new roadmap row is opened. be the defect #1332 exists to correct. The compiled-arch manifest (M2) and the launch probe (M3) are what make the layer sound. **Owner: #1332 M2/M3, row `BACKEND-CUDA-COMP-FA`.** + + **The measurement that settles this, and it is not a thought experiment.** + Run through the pinned oracle on a GB10 board in the fleet (compute capability + 12,1), same wheel, same prompt, one variable: + + | Requested backend | Result | + |---|---| + | `FLASHINFER` | `GENERATED: ' Paris. The capital of France is also the capital of the Republic of France.'`, `GEN_RC=0` | + | default (resolves `FLASH_ATTN`) | dies at the FIRST attention call, `cudaErrorUnsupportedPtxVersion` | + + vLLM's own priority list puts `FLASH_ATTN` at position 0 and `FLASHINFER` at + position 1 for this device (`cuda.py:156-163`, the `else` arm our table row + mirrors). **The priority-0 choice is unrunnable on the board and the + priority-1 choice works, and every filter in this file passes both.** Both + declare a compute-capability floor at or below 12,1; neither is asked what its + own fatbin contains. That is the gap, stated as a pair of observations rather + than as an argument, and it is why this row's M1 must not be reported as + fixing selection. Motivation only — M2 and M3 close it and are NOT in this + change. 3. **`AttnSelectorConfig::dtype` is not supplied by the runner.** The production call site fills `head_size`, `block_size` and `kv_cache_dtype` from the resolved per-layer KV geometry, but the model/query dtype is not available at diff --git a/docs/USAGE.md b/docs/USAGE.md index c9544c428..fa887f3f2 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -756,11 +756,21 @@ refusal is about that checkpoint on this build. **What this check cannot tell you.** It reports what a backend *claims*, never what your binary contains and never whether the kernel will launch. A backend whose declared floor is compute capability 8.0 is accepted on any newer GPU, even -when the build carries no compiled code for that GPU — which is a real failure -mode, not a hypothetical one, and it surfaces as a launch error rather than as -this refusal. Confirming which architectures a build actually targets is a -separate question, answered under "Confirming which CUDA architecture a build -targets" above. Tracked as +when the build carries no compiled code for that GPU. + +That is a real failure mode, not a hypothetical one, and it surfaces as a launch +error rather than as the refusal above. It has been measured on a GB10 board +(compute capability 12,1) against the reference engine, same wheel and same +prompt: asking for its `FLASHINFER` backend generates text and exits cleanly, +while the default — which resolves `FLASH_ATTN`, the reference engine's *first* +preference for that device — dies at the first attention call with +`cudaErrorUnsupportedPtxVersion`. The first preference could not run and the +second could, and no capability check on either side could tell them apart. + +So if a run dies inside attention rather than being refused before it starts, +the backend was accepted on a claim your build does not honour. Confirming which +architectures a build actually targets is a separate question, answered under +"Confirming which CUDA architecture a build targets" above. Tracked as [#1332](https://github.com/mudler/vllm.cpp/issues/1332). Selecting a backend by name is not exposed yet; the engine always resolves one. diff --git a/include/vllm/v1/attention/backend.h b/include/vllm/v1/attention/backend.h index b53f43d0f..1fb3b578b 100644 --- a/include/vllm/v1/attention/backend.h +++ b/include/vllm/v1/attention/backend.h @@ -272,7 +272,13 @@ class AttentionBackend { // compute_80 PTX and nothing else, and every launch then failed a driver JIT // with cudaErrorUnsupportedPtxVersion. `grep -rn get_arch_list vllm/` returns // zero hits: vLLM never asks what its own fatbins contain, and neither does - // this. The invariant #1332 states is that no backend may be declared valid on + // this. Measured on that board through the pinned oracle, same wheel and same + // prompt: asking for FLASHINFER generates text and exits 0, while the default + // — which resolves FLASH_ATTN, upstream's priority 0 for this device — dies at + // the FIRST attention call. The priority-0 choice is unrunnable and the + // priority-1 choice works, and every predicate below passes BOTH. + // + // The invariant #1332 states is that no backend may be declared valid on // the strength of a property of the DEVICE alone, and satisfying it needs the // build-derived compiled-arch manifest (M2) and the launch probe (M3). This // layer is NECESSARY AND NOT SUFFICIENT, and a reader who takes it for a