feat(MODEL-MUSIC-MUSIC3): W2/W3 — the autoregressive half, and the token gate the spec promised does not exist (#672)

.agents/specs/minimax-music3.md

@@ -10,7 +10,7 @@
 [#14456](https://github.com/huggingface/diffusers/pull/14456), head
 `c6da9936e4bda83107943a16eb8682e9a37d8527` — **OPEN, not merged**.
 **Cross-check:** SGLang-Omni `748a0b437e4a8faad44d7bbfd5a0ae55d1fef830`.
-**Status:** **W0 DONE, W1 LANDED.** Spec committed, both oracles pinned, §1.1 resolved and confirmed at runtime, the diffusers oracle gateable against committed goldens, and the modular loader in the tree. W2/W3 dispatched.
+**Status:** **W0 + W1 DONE, W3 DONE, W2 PARTIAL.** Spec committed, both oracles pinned, §1.1 resolved and confirmed at runtime, the diffusers oracle gateable against committed goldens, the modular loader in the tree, and the autoregressive half's compute gated at reduced dimensions and against the real bf16 checkpoint. §5's token-exact gate is WITHDRAWN: upstream's AR stage has no greedy path. The 8.6B language-model forward and W4-W7 are owed.
 **Developer directive (2026-08-13):** "land minimax music 3 support complete, to
 vllm.cpp, wired to the ABI and to the example http server, merge to main, tested
 e2e." That fixes W6's shape (the ABI surface and the example server are in scope,
@@ -292,9 +292,55 @@ LLM→diffusion handoff on *continuous hidden states* rather than discrete token
 
 ## 5. Gates
 
-**LLM half — token-exact.** The global LLM and the depth decoder emit discrete RVQ
-codes. Greedy decode of the code sequence is compared against the oracle
-token-for-token on a fixed prompt. This is a real token gate and it binds.
+**LLM half — token-exact. WITHDRAWN 2026-08-14 by W2/W3; the artifact refuted
+it.** What this paragraph said was: "The global LLM and the depth decoder emit
+discrete RVQ codes. Greedy decode of the code sequence is compared against the
+oracle token-for-token on a fixed prompt. This is a real token gate and it
+binds." It is kept in full, because a withdrawn claim that leaves no trace is how
+the same wrong gate gets re-specified.
+
+**There is no greedy decode of this model to compare against.** `_sample_top_k`
+(`encoders.py:94-103`) is the only sampler either stage uses; `_AR_SAMPLING_TOP_K`
+is a module constant of 50, there is no temperature and no argmax branch, and the
+last line is `torch.multinomial(probs, 1, generator=generator)`. The committed
+`rvq_codes.npy` is therefore a **seeded sample**, and reproducing it
+token-for-token means reproducing torch's CPU Mersenne-Twister and its
+multinomial — a claim about torch's RNG, not about this model.
+
+A second, independent reason the same conclusion holds, and the one that would
+survive even a bit-exact RNG: **both** stages sample from a CFG mix of a
+conditional and an unconditional row (`encoders.py:327-328`, `:134-135`), and the
+goldens store the **conditional row only** (`encoders.py:132,343`, both slice
+`[:1]`). The unconditional branch is not in the golden set, so the guided
+distribution the codes were drawn from cannot be reconstructed from what is
+committed.
+
+**What replaces it.** The codes are consumed as INPUTS and the AR half is gated
+on TENSORS, at two scales:
+
+* reduced dimensions, float32, against goldens produced by *executing* upstream's
+  own `MiniMaxMusic3ConditionEncoder` and `MiniMaxMusic3RVQDepthDecoder`
+  (`scripts/gen-minimax-music3-ar-goldens.py`). This separates an algebra defect
+  from rounding, and it runs in CI with no checkpoint;
+* full scale, bf16, real weights: the condition mix against
+  `condition_chunk0.npy` (176 128 values) and the depth decoder against
+  `frame_hiddens[:, 4096:]` (716 800 values), driven by the golden codes and the
+  golden `last_hidden`.
+
+The full-scale bound is calibrated against a **matched control** rather than
+guessed. torch's own `sdpa_kernel(MATH)` arm, running upstream's own module on
+the identical inputs, reproduces the goldens to 46.34% bit-identical at mean
+absolute error 1.659e-03 — its CPU attention kernel runs a blocked online softmax
+that no closed-form rounding model reproduced. Ours is 43.61% and 1.824e-03,
+inside that spread. Chasing a particular kernel's rounding below the control is
+not "more correct" (AGENTS.md's near-tie discipline).
+
+**Still owed on the LLM half:** the 8.6B `Qwen3ForCausalLM` forward itself.
+`frame_hiddens[:, :4096]` is the language model's own hidden state, and
+reproducing it means running that model teacher-forced on the golden codes
+through our landed Qwen3 path, which needs an `inputs_embeds` entry it does not
+have. That is the remainder of W2 and it is recorded here rather than discovered
+later.
 
 **Acoustic half — per-stage tensor parity.** No logits, no sampler, so no token
 gate exists to have. Each stage is compared against the oracle's own output for
@@ -372,15 +418,41 @@ pinned SHA.
 
 ## Now
 
-**W0 DONE, W1 LANDED, row `ACTIVE`.** The diffusers oracle generates audio and is
-`gateable = yes` against 13 committed per-stage goldens; both oracles are pinned;
-§1.1 is resolved and confirmed at runtime; the modular loader is in the tree with
-the dtype invariant §2.1 enforced and the native arm refused by name.
-
-W2/W3 (the autoregressive half) are dispatched. Then W4/W5 acoustic, W6 the
-speech-family registration plus the `vllm_speech_*` ABI and the example HTTP
-server, W7 the quantized arms.
-
-Two things are owed and neither is W0's to close: **no speed number exists** — the
-oracle capture was CPU-only because `dgx.casa` was down, so nothing here touches
-the speed axis — and SGLang-Omni remains `gateable = no`, read but never executed.
+**W0 + W1 DONE, W3 DONE, W2 PARTIAL; row `ACTIVE`.** The diffusers oracle
+generates audio and is `gateable = yes` against 13 committed per-stage goldens;
+both oracles are pinned; §1.1 is resolved and confirmed at runtime; the modular
+loader is in the tree with the dtype invariant §2.1 enforced and the native arm
+refused by name. W2/W3 add the autoregressive half's compute —
+[`minimax_music3_ar.h`](../../include/vllm/model_executor/models/minimax_music3_ar.h)
+and its two gates. Nothing generates a song yet.
+
+**W3 is complete and gated at both scales.** The learned 8-layer condition mix
+reproduces `condition_chunk0.npy` to 175 989 of 176 128 values **bit-identical**
+(mean absolute error 1.99e-07, no value beyond one bf16 ULP-or-2^-7), and the
+4-layer RVQ depth decoder reproduces `frame_hiddens[:, 4096:]` — 716 800 values
+over 25 frames × 7 depth steps — inside the matched control's spread (§5). The
+16-position window is exercised at its boundary and one past it. The reduced
+dimension gate is 25 cases / 338 assertions and needs no checkpoint.
+
+**W2 is partial, and the split is exact.** Everything the autoregressive loop
+does *around* the language model has landed and is gated: the prompt the
+checkpoint contract fixes (both upstream rewrite passes, string for string, on
+the oracle capture's own prompt), the unconditional CFG row, the frame budget and
+its two refusals, the semantic vocabulary mask, the guided-logit pipeline
+including the re-mask that keeps a NaN from becoming a candidate, `_sample_top_k`
+up to its draw, and the frame feedback embedding. What has NOT landed is the
+8.6B `Qwen3ForCausalLM` forward itself — see §5's "still owed".
+
+**§5's token-exact claim is withdrawn**, and that is this phase's most important
+finding rather than a footnote: upstream's AR stage has no greedy path at all, so
+`rvq_codes.npy` is a seeded sample and is consumed as an input by these gates.
+§5 now records the reasoning and the tensor gates that replace it.
+
+Next: W4/W5 acoustic, W6 the speech-family registration plus the `vllm_speech_*`
+ABI and the example HTTP server, W7 the quantized arms. W4 is unblocked and does
+not depend on W2's remainder.
+
+Two things are owed and neither is this phase's to close: **no speed number
+exists** — every capture so far ran on CPU because `dgx.casa` was down, so
+nothing here touches the speed axis — and SGLang-Omni remains `gateable = no`,
+read but never executed.

CMakeLists.txt

@@ -768,6 +768,7 @@ add_library(vllm STATIC
   src/vllm/model_executor/models/minimax_h3_audio_vae.cpp
   src/vllm/model_executor/models/vocoder1d.cpp
   src/vllm/model_executor/models/minimax_music3_loader.cpp
+  src/vllm/model_executor/models/minimax_music3_ar.cpp
   src/vllm/model_executor/models/gpt2.cpp
   src/vllm/multimodal/speech_engine.cpp
   src/vllm/model_executor/models/indextts2.cpp

docs/FEATURES.md

@@ -163,7 +163,7 @@ in `ltx2_text_encoder.cpp` is the call that would have to change.
 | Whisper audio encoder | openai/whisper-small; whisper-large-v3 (Voxtral cfg) | encoder tower 77/77; large-v3 tower 203/203 | pending |
 | MiniMax-H3 DiT (`MiniMaxH3DiTModel`, vllm-omni lane) | MiniMax-H3 (33.1B video+audio) | portable 79/79; all three modalities COHERENT on Q4_K_M (§8.20); PRUNED ckpts run, Q8_0 seam 0.9941 (§8.21); ref2va grid was NVFP4 quant error, §8.9 REFUTED; GGUF/NVFP4/bf16 shards stream | FP4/Marlin landed; speed pending; no bf16 render yet. Render from the Q4_K_M GGUF, not the NVFP4 arm. Krea 2 text-to-image (roadmap C11) is scoped to reuse these DiT seams |
 | LTX-2.5 DiT (`LTX2VideoTransformer3DModel`, Lightricks lane) | LTX-2.5 (21.00B video+audio) | `SPIKE`. DiT, VAEs+ENCODERS, conditioning, pipeline, quant loaders gated at reduced dims. Prompt-side AdaLN ported, host+device. Typed prompt to Gemma-4 to cross-attn, FIXTURE-gated. A prompted render is OWED | Family `ltx-2.5`, `ltx2-gen`. ~29 GB NVFP4/GB10, FP8 ~44 GB, +~24 GB tower. FP8/torchao/1st-party NVFP4 load; `keyframes_abs_pos_embedding` alone needs `allow_unported`. DiffVAE, LoRA, image cond refused. Speed PENDING |
-| MiniMax-Music3 (`MiniMaxMusic3ForConditionalGeneration`, diffusers lane) | MiniMax-Music3 (8.6B Qwen3 LLM + 0.646B RVQ decoder + 2.4B fp32 DiT + DAC Flow-VAE); diffusers arm, ~28.5 GB | `ACTIVE`. Oracle gateable with 13 per-stage goldens (#708); modular loader gated 1413/1413 on the real 27 GB checkpoint, all 1012 tensors accounted (#714). Output 44100 Hz stereo. No stage runs yet: W2-W7 owed | Not measured. The denominator will be SGLang-Omni in its production configuration (both CUDA graphs, compiled DIT and DAV, batched seeded sampling) |
+| MiniMax-Music3 (`MiniMaxMusic3ForConditionalGeneration`, diffusers lane) | MiniMax-Music3 (8.6B Qwen3 LLM + 0.646B RVQ decoder + 2.4B fp32 DiT + DAC Flow-VAE); diffusers arm, ~28.5 GB | `ACTIVE`. Loader gated 1413/1413 on the real checkpoint (#714); W2/W3 AR half computes and is gated vs the real bf16 weights. No greedy path upstream, so the token-exact gate is withdrawn. W4-W7 owed | Not measured. The denominator will be SGLang-Omni in its production configuration (both CUDA graphs, compiled DIT and DAV, batched seeded sampling) |
 | MTP speculator | Qwen3.6-27B, Qwen3.6-35B-A3B | token-identical to vLLM `mtp` at c1 | ~4% faster c1; +16% output tput (MoE) |
 | DFlash block-diffusion | Qwen3 (DFlash draft) | near-tie e2e 27/27 vs vLLM | 2.9x over spec-off, 1.003x vs vLLM DFlash-on |
 | DeepSeek-V4 MTP | DeepSeek-V4-Flash (nextn head) | lossless 5/5; real-model weight-blocked | pending |
@@ -197,7 +197,7 @@ on the committed fixture); reranking/classify models are not yet registered.
 | Audio | ✅ correctness-gated | ✅ | ◐ | ◐ |
 | Video+audio GENERATION (MiniMax-H3 DiT, LTX-2.5 DiT) | ◐ H3: all three modalities COHERENT on Q4_K_M (t2va, fl2va, ref2va; §8.20); the NVFP4 arm carries the patch grid; GGUF/NVFP4/bf16 loaders, pruned too (§8.21). LTX-2.5: a second lane, `SPIKE`, gated at reduced dims | ✅ H3 (vllm-omni, BF16-only, no quantized arm); LTX-2.5 only through the generic diffusers adapter, no native recipe ([vllm-omni#6066](https://github.com/vllm-project/vllm-omni/issues/6066)) | ☐ | ☐ |
 | Speech / audio GENERATION (TTS, vLLM-Omni lane) | ◐ IndexTTS-2.5 only: the S2Mel DiT is COMPLETE front to tail and gated against upstream; its tail runs on REAL shipped weights. Other stages gated at reduced dims. No render, no route (#634) | ✅ (vllm-omni: MOSS-TTS, Qwen3-TTS, Higgs Audio v3, Voxtral TTS, IndexTTS-2.5) | not assessed | not assessed |
-| MUSIC generation (MiniMax-Music3) | ☐ not generating. The W1 checkpoint LOADER has landed ([spec](../.agents/specs/minimax-music3.md), #672); no stage runs yet. Lyrics plus a structured description in, a multi-minute stereo song out | ☐ absent from the pin, from vLLM `main` and from `vllm-omni` alike | ◐ served by SGLang-Omni, a third repository, which loads the NATIVE checkpoint layout | ☐ |
+| MUSIC generation (MiniMax-Music3) | ☐ not generating. W1 loader + W2/W3 AR half landed ([spec](../.agents/specs/minimax-music3.md), #672); the language-model forward and every acoustic stage are not. Lyrics + a structured description in, a stereo song out | ☐ absent from the pin, from vLLM `main` and from `vllm-omni` alike | ◐ served by SGLang-Omni, a third repository, which loads the NATIVE checkpoint layout | ☐ |
 | Multimodal over the OpenAI server | ◐ image request path wired, forward pending | ✅ | ✅ | ◐ |
 
 Image, video and audio are correct through the CLI and library. Over the HTTP

docs/USAGE.md

@@ -2367,3 +2367,88 @@ python3 scripts/convert-indextts2-checkpoint.py \
 VLLM_CPP_INDEXTTS2_S2MEL=$CHECKPOINT_ROOT/IndexTTS-2.5-safetensors/s2mel.safetensors \
   ./build/tests/test_indextts2_s2mel_loader
 ```
+
+## MiniMax-Music3: the autoregressive half
+
+Phases W2 and W3 of #672.
+`include/vllm/model_executor/models/minimax_music3_ar.h` is what consumes three
+of W1's six components: the prompt the `language_model` is driven with, the
+semantic stage's classifier-free-guidance logit pipeline, the learned 8-layer
+condition mix, and the 4-layer RVQ depth decoder. **It still does not generate a
+song** — the DiT, the scheduler and the vocoder are W4–W5, and the 8.6B
+`Qwen3ForCausalLM` forward itself is the remainder of W2.
+
+### The token gate the spec promised does not exist
+
+Worth stating plainly, because the spec said otherwise until this phase measured
+it. MiniMax-Music3's autoregressive stage has **no greedy path**:
+`_sample_top_k` (`encoders.py:94-103`) is the only sampler either stage uses, it
+has no temperature and no argmax branch, and it ends in
+`torch.multinomial(probs, 1, generator=generator)`. The oracle's
+`rvq_codes.npy` is a *seeded sample*, so matching it token-for-token would be
+reproducing torch's RNG rather than this model. Independently: both stages sample
+from a CFG mix of a conditional and an unconditional row, and the goldens store
+the conditional row only, so the guided distribution is not reconstructible from
+what is committed.
+
+The codes are therefore **inputs** to these gates, and the AR half is gated on
+tensors.
+
+### Running the gates
+
+The reduced-dimension gate needs no checkpoint. Its goldens come from executing
+upstream's own `MiniMaxMusic3ConditionEncoder` and `MiniMaxMusic3RVQDepthDecoder`
+at small dimensions in float32, so it isolates an algebra defect from rounding:
+
+```sh
+cmake -S . -B build -DVLLM_CPP_BUILD_TESTS=ON
+cmake --build build -j 8 --target test_minimax_music3_ar
+./build/tests/test_minimax_music3_ar
+```
+
+The full-scale gate drives the real bf16 weights on the oracle capture's own
+inputs and skips loudly without the checkpoint:
+
+```sh
+VLLM_CPP_MUSIC3_CHECKPOINT=/path/to/minimax-music3 \
+  ./build/tests/test_minimax_music3_ar_real
+```
+
+It compares 176 128 values for the condition mix (against `condition_chunk0.npy`)
+and 716 800 for the depth decoder (against `frame_hiddens[:, 4096:]`, 25 frames ×
+7 depth steps), and it reports the counts rather than only a verdict.
+
+Regenerate the reduced-dimension goldens with the pinned oracle's interpreter
+(see `tools/oracle/README.md`) after an upstream change:
+
+```sh
+~/venvs/music3-oracle/bin/python scripts/gen-minimax-music3-ar-goldens.py \
+  --out tests/vllm/models/minimax_music3_ar_goldens.inc
+```
+
+### Two things that will bite a later phase
+
+**The code rows are offset by one from the frames.** `rvq_codes.npy` is `[26, 8]`
+and `frame_hiddens` is `[25, ...]`: row 0 of the codes is the priming decode step,
+which emits no frame (`encoders.py:342`). `rows[1:]` align with the frames.
+Comparing the unshifted sequences yields two individually plausible tensors and a
+wrong gate.
+
+**`ArCompute` is not a precision knob.** The autoregressive half runs bf16, and a
+bf16 torch module rounds at *every* op boundary, so an fp32 host forward is a
+different computation rather than a more precise one — measured, it leaves
+448 450 of 716 800 values beyond one bf16 ULP. `ArCompute::kBFloat16` mirrors the
+rounding; `kFloat32` is the reduced-dimension goldens' dtype. A caller at
+`kBFloat16` also owes its weights at bf16, *including* the condition encoder,
+whose file is fp32 while its runtime is not.
+
+And bit-exactness against torch is not on offer here, which is worth knowing
+before a later phase spends a day chasing it. torch's bf16 `nn.Linear` on CPU
+reproduces to 32 759 of 32 768 values, but its dispatched attention reproduces to
+only 25 736: the CPU kernel runs a blocked online softmax, and four candidate
+rounding models (pre-scaled q, bf16-rounded scores, bf16-rounded probabilities,
+and their combinations) were all *worse* than the plain form. The full-scale
+bound is therefore
+calibrated against torch's own `sdpa_kernel(MATH)` arm on the identical inputs
+(46.34% bit-identical, mean absolute error 1.659e-03) rather than against a
+bit-exactness that no second implementation can reach.