feat(ENG-EXPERT-STREAM-DEVICE): W0f — the dense weights were resident twice, and --device cuda now decodes a 369.97 GiB checkpoint on a 119.631 GiB GB10 (#1299)

[localai-bot]

Replaces #1326. That pull request is correct and reviewed, and its base is gone:
it was stacked on row/ENG-EXPERT-STREAM-DEVICE-W0 at 95883dcae, and #1377
squash-landed the same W0 work onto main as 5f4eb356e. A squash destroys the
merge base, so #1326 now presents product-code conflicts that are two unrelated
histories describing one change rather than a disagreement. This branch is that
same change rebuilt on main, with every one of main's later repairs kept.

What the defect was

With W0's lane on, Qwen3.8-2.4T-A95B UD-Q1_0 (369.97 GiB) LOADED on
--device cuda on a 119.631 GiB GB10 and then exhausted the machine inside its
first forward: zero decode steps, seven attempts, every one identical (#1299).

The measurements name it rather than a reading of the code. A 0.15 GiB slot
arena died exactly where an 18.55 GiB one did, so the arena is not the cost. A
1-token prompt, whose protected set fits with no in-place fallback at all,
behaved identically to a 5-token one, so prefill protection is not it. Growth was
ANONYMOUS (RssAnon 8.1 to 61.4 GB) while file-backed stayed flat, so nothing
was pinning the mapping. Host anon plus swap reached ~65 GB against system used
~119 GB, and the ~42 GB difference is device memory that unified memory does not
charge to RSS.

So every non-expert weight was resident twice: once as the host OwnedTensor,
once as ResidentWeight's device staging copy. On a part where device memory IS
host memory, the second copy buys nothing and costs everything.

The change

ResidentWeight now takes the same branch W0c gave KqExpertSlice, on the same
probed predicate: where Platform::host_memory_is_device_addressable(), it
returns a tensor over w.bytes.data() instead of Alloc + Copy into
w.d_dev. A DISCRETE device answers false, falls through, and is byte-identical
to before, asserted by its own case rather than by inspection.

Safety is by alignment, not by surveying kernels. The staging branch is a
verbatim byte copy returning the same dtype, shape and dropped marker set, so the
only thing a consumer can notice about the substitution is the pointer's
alignment. kDeviceAliasAlignment is 256 because cuBLASLt's
CUBLASLT_MATMUL_PREF_MIN_ALIGNMENT_A_BYTES defaults to 256 and this tree never
sets it, which dominates every explicit pointer gate in the CUDA kernels -- at
least seven across four files, the strictest asking 32. A plain
std::vector<uint8_t> gives 16 and no more, so MakeHostBytesDeviceAliasable
re-homes an OWNED misaligned buffer once into an aligned block: one memcpy that
REPLACES the host-to-device copy it removes. A misaligned BORROW declines and
stages instead, because copying a clean file-backed GGUF mapping into anonymous
memory would create the residency this change exists to remove.

Three fresh-review findings and one defect found in flow ride with it, each with
its own red-first case: MoeBlockBf16Cuda released host mirrors the aliased
device pointer table still pointed at, so the release now asks
HostMirrorIsRedundant; the dense release guarded on d_dev || d_dev_f32, and
d_dev_f32 is an f32 UPCAST that can never stand in for the raw bytes; the
no host bytes refusal fired ABOVE the device-copy memo and turned an
always-served case into a throw; an already-aligned direct-upload borrow re-ran
issue #150's madvise(MADV_DONTNEED) on every forward step; and #1320's
i8mm-repack tripwire was missing on both arms of ResidentWeight.

The result, and the gate that does not pass

It decodes. 32/32 steps, peak RSS 97.75 GiB, decode-phase exhausted delta 0
(G0-LIVE PASS). G0-CORRECT FAILS: the CUDA ids diverge from the CPU arm at step 7
on a near-tie, where the CPU arm's own runner-up is exactly the token CUDA
emitted, 1.4% behind, and one step later the margin is 0.1%. G0-SPEED is
therefore VOID by this row's own stop condition and no rate is claimed
anywhere.

Can the alias move a logit? Measured on the target silicon, and no

The probe compares what a consumer can actually see. rc job
7c7a05e9-be87-48f4-94ae-1bbe0340f063 on dgx:gpu0 -- NVIDIA GB10 sm_121,
driver 580.173.02, cuBLASLt 130101, the predicate re-derived in the job's own
output as pageableMemoryAccess=1 integrated=1 -- ran six checkpoint shapes
crossed with both cuBLASLt formulations the dense path issues, 12 measurements,
PROBE_EXIT=0, PROBE_FAILURES=0. A repeated heuristic call is identical 12/12;
the tree's unset preference equals the documented 256 default 12/12; weakening
the promise to 16 moves nothing 12/12; and cublasLtMatmul output is bit-exact
between a cudaMalloc operand and a 256-aligned host block 12/12, differing=0.
At least five distinct algorithm configurations appear across the twelve and they
differ from the earlier thor:gpu0 leg's, so the heuristic was re-resolved rather
than replayed and the instrument discriminates.

The structural reason needs no lease: cublasLtMatmulAlgoGetHeuristic takes no
operand pointers, so alignment reaches it only through a preference this tree
never sets.

So the alias does not cause the step-7 divergence. Excluding one cause is not
identifying another. What DOES cause it is unmeasured, and it is carried under
## Owed with its next traceable step named: a two-arm dump of the step-7
forward that names the first differing tensor.

What main already had, and what this rebuild kept

Every repair #1377 and #1378 landed survives, unmodified. None of the files
carrying them is in this diff:

• GgufExpertTowersReachSlotLane, the fifth lane term, in
  gguf_device_fit.{h,cpp} and read by the loader.
• PeekRoute promoted into
  include/vllm/model_executor/model_loader/gguf_keep_quant.h.
• HostMemoryIsDeviceAddressableFromAttrs extracted into
  src/vllm/platforms/platform.cpp and gated over all four attribute pairs.

Their gates are green with W0f in the tree, which is the live evidence rather
than an inspection: test_gguf_device_fit 17 cases / 130 assertions,
test_gguf_device_fit_reach 14 / 66, test_platform 14 / 114.

Two more of main's repairs are inside files this branch does touch, and both
are kept: the HostAddressable RAII guard in
test_expert_stream_device_slot.cpp (the one conflicting case now uses it,
including for its discrete arm, instead of writing the flag directly), and
docs/USAGE.md's limit list, which grows from four limits to six rather than
back to two, keeping the model-family bullet and the keep-quant OR keep-f16
residency bullet.

#1414 then landed the W0e measurement while this branch was gating. Its facts are
carried, not overwritten: the CPU arm's 11.05 s/token at 4000 slots on a live
cache is the standing figure and the 4000-slot count is what both recipes set,
and the 8000-slot slowdown keeps main's attribution -- the extra 9.27 GiB of
arena takes the free memory the borrowed 370 GiB mapping is served out of, which
is the page cache rather than an arena that fails to fit.

Records, and one union-merge duplicate

docs/STATUS.md, docs/BENCHMARKS.md and docs/FEATURES.md take one scoped row
edit each; every other key is byte-for-byte unchanged.
.agents/benchmark-record.md is a pure append of 208 lines with 0 deletions, and
it now carries TWO sections rather than one merged one, because #1414's run and
this one are different trees: 95883dcae (loads, generates nothing) and
9c783a8be (32/32 steps). The spec's ## Evidence mirrors that split, and each
side says which figures are its own and which must not be mixed.

.agents/issue-index.md appends one row (#1320). It appended two, and the second
was the union-merge duplicate AGENTS.md warns about: #1414 appended a #1299
row on main and this branch appended its own, and merge=union kept both without
a conflict. scripts/check-agent-record.py caught it. Main's row is kept
byte-for-byte and this branch's is dropped, which is the only resolution the
append-only rule allows.

Two record edits are consequences of this change rather than part of it, named so
a reviewer does not have to work out why they are in the diff. Adding 198 lines to
qwen3_5_weights.h moved struct Fp8Weight from line 342 to line 540, which
scripts/check-agent-record.py reported as a stale-anchor regression (33 against
a baseline of 32); the three citations of it are repointed, in
.agents/quantization-matrix.md (both the text and the #L fragment, which is
the one the checker reads) and in two specs. And appending a measurement to
.agents/benchmark-record.md obliges docs/FEATURES.md under
scripts/check-doc-checkpoint.py, so the routed-expert-streaming row now says the
staging device decodes and that its token gate fails. That is #1387's shape,
caught before the commit was published rather than after.

The fresh review returned FAIL, and this is the repair

Five findings, four of them in the record rather than in the product code, which
survived eleven of twelve mutations and is unchanged apart from one comment.

The alignment constant had no gate (MEDIUM). kDeviceAliasAlignment was 256
and every assertion about it in the tree is written
% vllm::kDeviceAliasAlignment == 0, which is a tautology in the constant. The
review lowered it to 16 and the whole suite reported SUCCESS at exit 0 -- the
promise this change's entire safety argument rests on could be deleted with no
gate saying so. That argument is that cuBLASLt is PROMISED 256:
CUBLASLT_MATMUL_PREF_MIN_ALIGNMENT_A_BYTES defaults to 256 and this tree never
overrides it, with grep -rn MIN_ALIGNMENT src/vt/ empty against a positive
control on CUBLASLT_MATMUL_PREF_MAX_WORKSPACE_BYTES in the same file. A case in
test_resident_weight_host_addressable.cpp now pins the literal.

RED-FIRST, and re-taken on the pushed head ac5e1ad8f. Before the case, the
constant at 16 left every one of the eight suites GREEN at exit 0 with identical
counts -- the reviewer's mutation 5 reproduced exactly. With the case, the same
mutation gives compile rc 0, git diff --stat 1 file / 1 insertion / 1 deletion,
test cases: 13 | 12 passed | 1 failed, Status: FAILURE!, exit 1. Restored
byte-for-byte by sha256 (7f1d10b8...), rebuilt, 13 cases / 72 assertions /
SUCCESS! / exit 0. The reviewer's mutation to 24 still reds with its original
std::bad_alloc and SIGABRT intact beside the new failure -- exit 134, 5 cases
failed while the assertions: line read 1 failed of 16, which is why the
Status: line is what was read. The pin masks nothing, and the existing %
assertions stay because they check the buffers.

The header stated two things that were false (MEDIUM). Its rationale block
said this row's CUDA-versus-CPU token divergence IS the two arms' GEMM
arithmetic, which the spec explicitly forbids asserting, and said the GB10 leg is
owed when the spec and .agents/benchmark-record.md both record it as RAN. The
block now says what the probe established -- the alias is EXCLUDED as the cause,
on the target silicon -- and says plainly that the cause is not identified.

Three public projections had carried that same error outward, so the correction
does not stop at the header. docs/BENCHMARKS.md and docs/STATUS.md both read
"G0-CORRECT FAIL on a near-tie ... not the alias"; both now say the cause is NOT
identified, in fewer characters than before. docs/USAGE.md said it hardest, in
a user-facing document: "the divergence is a measured near-tie rather than a
disagreement about the model" and "the two arms run genuinely different GEMM
kernels". What is measured there is the MARGIN, and that is what it now says.

The disowned ratio is deleted rather than restated (LOW). It was correctly
labelled VOID and not claimed, and no rate reaches any public document. But
digits survive a copy-paste and a disclaimer does not, and this repository has
watched a disowned number become a quoted one. Both medians stay in the table, so
anyone entitled to the quotient can divide; what is gone is the pre-computed
string. docs/FEATURES.md gets back "not throughput", which stops a reader
treating the lane as a speed feature and which survives in docs/USAGE.md.

One anchor went stale inside this pull request. The rewritten rationale block
adds 23 net lines above struct Fp8Weight, and check-agent-record.py caught
qwen3_5_weights.h:540 going stale, 33 against a baseline of 32. The anchor is
repaired, never the baseline: the struct moved 540 -> 563, and
sed -n '540,553p' before and sed -n '563,576p' after hash identically
(9f809a38...), so the shift maps the cited range byte for byte rather than
being re-derived by eye. Two prose citations the checker does not scan are
corrected with it. ANCHOR-ROT is back to 38.

NOT REACHED, and named here as ## Nothing lands dead requires

No CI gate reaches the alias branch through a production entry point.
test_expert_stream_wiring enters Qwen3_5Model::Forward and the reachability
mutation reds it, but it runs on the CPU device, where ResidentWeight returns
at the is_cpu() early return about ninety lines above the alias branch. In CI
the branch is reached only through detail::StageWeightForTest, a test-only
seam.

This is deliberate rather than an oversight. The branch is selected by
needs_weight_staging() && host_memory_is_device_addressable(), and no CPU tier
can register a platform that answers both -- exactly one machine this project can
reach does. The device evidence is real and is the stronger of the two: the W0e
run entered the branch 43,501 times through Qwen3_5Model::Forward on
dgx:gpu0. It is simply not repeatable in CI, and closing it needs either a GPU
CI lane on a probed-capable part or a production entry point a fake staging
platform can drive end to end.

• What is unreached: the MakeHostBytesDeviceAliasable alias branch of
  ResidentWeight, on any CI runner.
• Row that owns the wiring: ENG-EXPERT-STREAM-DEVICE.
• Issue that tracks it: Qwen3.8-2.4T UD-Q1_0 LOADS on --device cuda and then exhausts a GB10 in its first forward: the non-expert weights are resident twice #1299
• Listed under ## Owed in .agents/specs/expert-stream-device-slots.md.

The disclosure was in the spec and in neither the pull request body nor any
commit body. Because this repository sets squash_merge_commit_message = PR_BODY, this body IS the landed commit message, so a disclosure absent from it
is a disclosure that does not land.

origin/main merged, and the two keyed records resolved BY KEY

GitHub reported this pull request CONFLICTING while git merge-tree read clean
locally, which is the forge ignoring the merge=union driver on the append-only
.agents/issue-index.md. Materialising the merge here clears it.

origin/main moved TWICE during this repair, so there are two merges:
63d87805c (#1399) and then 7f9c6802e (#1436, #1428), which landed while the
gate was running. Both are clean, and both had their keyed records verified by
key rather than taken on the driver's word.

.agents/issue-index.md is the exact union at each step, ending at 454 rows with
every one of this branch's 452 and every one of main's 453 present, no extra row
and no duplicate; main's #1396 and #1411 rows arrive and this branch's #1320 row
stays. Eight files were co-edited across the two merges --
include/vllm/model_executor/models/qwen3_5_weights.h,
src/vllm/model_executor/models/qwen3_5_dense_weights.cpp,
.agents/specs/qwen38-27b-quant-arms.md, tests/CMakeLists.txt and the four
public pages -- and each line main added was checked line by line to be present
in the merged file: all are, and all five of this branch's edits in those pages
survive. The one line either merge drops relative to main is this branch's own
W0f change at 9c30dfe7d. struct Fp8Weight is still at
qwen3_5_weights.h:563, so the repaired anchor is still correct after both.

check-agent-record.py was re-run AFTER each merge, not before, because a shared
summary counter byte-identical on both sides merges without a conflict while both
row changes apply.

Conflict markers were checked with git grep under a positive control, because
four record checkers return 0 on a file that contains them (#1417).

Gate

Re-taken on the pushed head f423df992, from a FRESH build directory rather
than incrementally, because the second merge brought main's source in and an
incremental build masks -Werror. Clean-configure plus full build, CMAKE_RC=0
and BUILD_RC=0 read from the unpiped commands, 0 warnings, and
grep -c 'No space left on device' = 0 on every log with 61 GB free -- stated
because this box has hit 100% during a run of this row and an ENOSPC presents as
a code verdict rather than as infrastructure. Eight focused suites, each with a
non-zero case count and an explicit Status: SUCCESS!:

Suite	cases	assertions	exit
test_expert_stream_device_slot	5	45	0
test_resident_weight_host_addressable	13	72	0
test_load_direct_upload	16	203	0
test_expert_stream_wiring	4	882	0
test_gguf_device_fit	17	130	0
test_gguf_device_fit_reach	14	66	0
test_platform	14	114	0
test_gguf_keep_quant	39	6093	0

check-agent-record.py, check-public-doc-tables.py,
check-issue-index-append-only.py and check-doc-checkpoint.py all exit 0, run
from this worktree's own scripts/ copy because a checker resolves its root from
its own path.

A full ctest on the pre-merge head was 561 of 564, with three failures none of
which is this change: test_qwen3_5_decode_graph_seam SEGFAULTs on main itself
(#1403), test_nemotron_h_paged_forward throws
No valid attention backend for device type 0, and test_qwen3_5_moe_vision
failed under parallel load with safetensors: empty file on a fixed /tmp path
while the disk was full, then passed 7 of 7 serially. test_cpu_x86_llamacpp_floor
is not registered in this tree, so there is nothing to re-run for it.

Issue: #1299
Row issue: #1124
Tripwire filed and covered in flow: #1320
Spec: .agents/specs/expert-stream-device-slots.md

#1124, #1299 and #1320 all stay OPEN: the row is not finished, the correctness
gate does not pass, and #1320's loader-policy half belongs to
QUANT-GGUF-KEEPQ-LOADER. Do not merge this before a fresh review, and note that
CI cannot give a verdict on this repository right now -- every completed workflow
run is cancelled (#1285), so treat it as REMOTE_UNVERIFIED and never as green.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [Claude Code]