dense_nvfp4_gemm.h caches the Marlin repack in a process-static map keyed on the weight's ADDRESS, the stale-pointer hazard ResidentSlot exists to fix

[localai-bot]

Claim

include/vllm/model_executor/models/dense_nvfp4_gemm.h:379-384 caches the dense Marlin repack in a process-static map keyed on the weight object's address.

An address is only a valid identity while the object lives. Across two engine builds in one process — the second engine's weights allocated where the first's used to be — a hit on that map returns the previous engine's repacked device buffer. The result is numerically plausible and silently wrong: right shapes, right dtypes, wrong values.

Why this is a known class, not a new theory

#237 exists for precisely this, and ResidentSlot is the fix it produced. qwen3_5.cpp:2429-2431 already uses ResidentSlot, and Nvfp4Weight carries an unused resident_marlin slot — so the correct mechanism is present in the type and simply not used by the shared header.

I could not find this instance filed anywhere; #237 tracks the class and the qwen3_5.cpp call site, not dense_nvfp4_gemm.h.

Why it matters more now than when it was written

The header was, until recently, effectively single-consumer. Two things changed:

1. dense_nvfp4_gemm.h is a shared seam — ResidentNvfp4, MatmulNvfp4MarlinD, MatmulNvfp4W4A16D plus a policy layer at compressed_tensors/schemes/nvfp4.h. It is the seam every NVFP4 consumer is told to route through rather than hand-rolling a parallel path.
2. MODEL-NEMOTRON-H (Run Nemotron-3.5-Lightning-30B-A3B-NVFP4 end to end: NemotronH hybrid (23 Mamba2 / 6 GQA / 23 non-gated relu² MoE) with a MIXED_PRECISION checkpoint #517, The paged runner's RECURRENT allocation is Qwen3.5-shaped: hybrid models are refused by the wrong model's name, and the only correct behaviour reads as a user config error #810) is about to route lm_head and 5935 expert projections through it. That is a second engine class in the same process space, which is exactly the condition the address key cannot survive.

So the hazard moves from latent to reachable as the seam gains its second consumer.

What done looks like

1. Key the cache on ResidentSlot rather than the weight address, mirroring qwen3_5.cpp:2429-2431, and use the resident_marlin slot Nvfp4Weight already carries.
2. A red-before test: build two engines in one process such that the second's weight lands at a recycled address, and show the current code returns the first engine's buffer. That test is the whole value of the fix — without it this is a refactor with no evidence, and the defect is invisible to any single-engine gate.
3. Consider whether a structural checker belongs here, in the shape of scripts/check-fp4-resident-consistency.py, which already enforces a neighbouring obligation on ResidentNvfp4 — an address-keyed static in a shared header is a pattern worth refusing by construction rather than by review.

Found while scoping A2-Q of #810. Not fixed there: it is a behaviour change with its own gate, and folding it into a model row would land it without the two-engine test that gives it meaning.

Related: #237 (the class and ResidentSlot), #940 (the FP8 seam extraction, which deliberately left an adjacent latent defect to #974 for the same reason), #974.

[localai-bot]

Sharper: there are TWO MarlinDenseResidentFor, same name, one safe and one not, selected by translation unit

Verified at origin/main:

include/vllm/model_executor/models/dense_nvfp4_gemm.h:379
    inline MarlinDenseResident& MarlinDenseResidentFor(const Nvfp4Weight* w) {
      static std::mutex mu;
      static std::unordered_map<const Nvfp4Weight*, MarlinDenseResident> cache;   <- ADDRESS-KEYED

src/vllm/model_executor/models/qwen3_5.cpp:2429
    MarlinDenseResident& MarlinDenseResidentFor(const Nvfp4Weight* w) {
      return ResidentIn<MarlinDenseResident>(w->resident_marlin);                 <- SLOT-KEYED, safe
    }

And each is used by the MatmulNvfp4W4A16D in its own TU — the header's at dense_nvfp4_gemm.h:509, qwen3_5's at qwen3_5.cpp:2521.

So which implementation you get depends on which translation unit you compile in, and the one in the SHARED SEAM is the unsafe one. A model routed through the header — which is what the shared-seam rule instructs — silently takes the address-keyed cache. A model with its own local copy, as qwen3_5.cpp has, takes the ResidentSlot that Nvfp4Weight already carries for exactly this purpose.

That inverts the usual incentive: following the seam rule gets you the defect, and hand-rolling gets you the fix. It also means the safe behaviour is not discoverable from the seam — a reader of dense_nvfp4_gemm.h has no reason to suspect a better implementation exists elsewhere under the same name.

Consequence for the row that is about to become the second consumer

MODEL-NEMOTRON-H's A2-Q2 unit (#810 / #517) routes lm_head through MatmulNvfp4W4A16D. Taking the header's entry point unexamined makes that unit the second-consumer condition an address key cannot survive.

So A2-Q2 has a design decision, not an inherited annoyance: route lm_head through the slot as qwen3_5.cpp does, or fix this first. That will be stated in its spec so it is a decision on the record rather than a default nobody noticed.

Note on the fix

Whichever way it goes, the two definitions should not both survive under one name. A same-named function with different safety properties in two TUs is a shadowing hazard independent of the caching bug — it defeats grep, and a reader who finds one has no signal that the other exists.

[localai-bot]

A new consumer, found while auditing the DFlash2 draft path at f44077b51

The DFlash2 draft's logits GEMM takes the address-keyed resident rather than the caller-owned slot A2-Q2b added for exactly this purpose.

src/vllm/model_executor/models/qwen3_dflash.cpp:95 calls

return dense_nvfp4::MatmulNvfp4W4A16D(d, x, weights.lm_head_fp4, DType::kF32);

with no resident argument, so it falls through to MarlinDenseResidentFor(&w) — the static std::unordered_map<const Nvfp4Weight*, MarlinDenseResident> at include/vllm/model_executor/models/dense_nvfp4_gemm.h:404-409. The draft's Qwen3DFlashWeights is heap-allocated per engine, so it is in the affected class: destroy one engine and build another in the same process, and the allocator can hand the new draft weights the old address, which then inherits a ready entry whose device pointers belong to the dead engine.

This is not a new bug class — it is this issue's class with a consumer nobody has checked. Flagging it because #984's red-before is two-engine, and the DFlash2 draft head is a reachable path into it that the issue does not currently name.

Not filing separately. Whoever takes #984 should include this call site in the sweep, and the fix is the one this issue already prescribes: pass the caller-owned slot rather than declining it.