feat(MODEL-MUSIC-MUSIC3): W2/W3 — the autoregressive half, and the token gate the spec promised does not exist (#672)

[localai-bot]

Phases W2 and W3 of .agents/specs/minimax-music3.md.
W3 is complete; W2 is partial and the split is exact. Nothing here generates a
song — the DiT, scheduler, vocoder, ABI and server are W4–W6.

Issue: #672 · Row: MODEL-MUSIC-minimax-music3-mini-max-music3-for-conditional-generation

The finding that matters, and it is a correction

Spec §5 said the LLM half gets a token-exact gate: "Greedy decode of the code
sequence is compared against the oracle token-for-token … This is a real token
gate and it binds." Measured against the artifact, it is not available.

Upstream's autoregressive stage has no greedy path. _sample_top_k
(encoders.py:94-103) is the only sampler either stage uses, _AR_SAMPLING_TOP_K
is a module constant of 50, there is no temperature and no argmax branch, and the
last line is torch.multinomial(probs, 1, generator=generator). The committed
rvq_codes.npy is a seeded sample, so matching it token-for-token would be
reproducing torch's CPU Mersenne-Twister, not this model.

A second, independent reason that would survive even a bit-exact RNG: both
stages sample from a CFG mix of a conditional and an unconditional row
(encoders.py:327-328, :134-135), and the goldens store the conditional row
only (:132, :343, both slice [:1]). The distribution the codes were
drawn from is not reconstructible from what is committed.

§5 is rewritten rather than quietly edited — the withdrawn paragraph is kept
verbatim, because a withdrawn claim that leaves no trace is how the same wrong
gate gets re-specified. The codes are consumed as inputs, and the AR half is
gated on tensors.

What lands

• ConditionMix — the learned 8-layer mix: a softmax over the layer logits
  (not a normalize), one scalar layer_scale on the mixed result (not per
  layer), a k=3 p=1 Conv1d over time through the shared vocoder1d primitives
  (not a pointwise projection), then nearest interpolation at an
  input-over-output scale. Each has a plausible wrong form that returns the
  right shape.
• DepthDecoderForward / DepthSequenceEmbeds / AudioHeadLogits — the
  4-layer RVQ depth decoder. Attention is causal with no RoPE and no KV cache, so
  one forward over the depth sequence equals upstream's incremental schedule step
  for step; that equivalence is verified bit-exactly, not assumed. The
  16-position window is exercised at its boundary and one past it.
• EmbedAudioFrame / FrameHiddenRow — the frame feedback and the
  conditioning row, with the two differently-based codebook offsets called out
  where they bite.
• W2's deterministic front half — prompt assembly (both upstream rewrite
  passes, string for string; the replacement order inside _normalize_lyrics
  is load-bearing), the unconditional CFG row, the frame budget and its two
  refusals, the semantic vocabulary mask, the guided-logit pipeline including the
  re-mask that keeps a NaN from becoming a candidate, and _sample_top_k up to
  but not including its draw.

Dtype is a parameter, because upstream's is

The AR half runs bf16, and a bf16 torch module rounds at every op boundary,
so an fp32 host forward is a different computation, not a more precise one.
Measured: the fp32 forward left 448 450 of 716 800 values beyond one bf16 ULP at
mean|d| 2.649e-03. ArCompute threads the store width through every function
with an op boundary. This is AGENTS.md's too-wide rule in its awkward direction —
nothing is numerically wrong at fp32, no token gate could ever see it, and only
a tensor gate against the real weights can.

Two gates

test_minimax_music3_ar — 25 cases / 338 assertions, no checkpoint. Goldens
come from executing upstream's own classes at reduced dimensions in float32
(scripts/gen-minimax-music3-ar-goldens.py), so it separates an algebra defect
from rounding. Worst deviation observed 4.8e-07 against a 1e-5 relative bound.

test_minimax_music3_ar_real — 4 cases / 894 assertions on the real bf16
checkpoint, skipping loudly without it:

stage	reference	result
condition mix	condition_chunk0.npy	175 989 of 176 128 bit-identical, mean|d| 1.99e-07, zero beyond 1 bf16 ULP-or-2⁻⁷
depth decoder	frame_hiddens[:, 4096:]	716 800 values (25 frames × 7 depth steps), 43.61% bit-identical, mean|d| 1.824e-03, max|d| 0.125

The depth-decoder bound is calibrated against a matched control, not guessed.
With the input layernorm bit-identical, torch's dispatched attention on identical
bf16 q/k/v reproduces only 25 736 of 32 768 values (its bf16 Linear reproduces
32 759) — the CPU kernel runs a blocked online softmax, and four candidate
rounding models were all worse than the plain form. So the control is torch
against itself: upstream's own module under sdpa_kernel(MATH) on these exact
inputs reproduces the goldens to 46.34% bit-identical at mean|d| 1.659e-03.
Ours sits inside that spread.

The bit-identical fraction is the discriminator with four orders of margin
(an fp32 forward scores 0.0004%), and it is asserted alongside absolute mean and
max bounds rather than instead of them — a count alone bounds nothing, and Pearson
would see none of it because it is scale-invariant.

npy.h grows an opt-in allow_fortran_order: condition_chunk0.npy is
stored column-major, and read as C-order it has the right value count and is the
wrong tensor. The default still throws, so every existing caller is unchanged.

RED before GREEN, and mutations that moved

Against a shape-correct stub: 25 cases, 2 passed, 23 failed; 336 assertions,
101 passed, 235 failed. With the implementation: 25/25 and 338/338.

Each mutation was applied to the head, rebuilt, run, and reverted; the tree was
restored byte-for-byte and verified by sha256.

mutation	reduced-dim gate	full-scale gate
softmax → plain normalize	3 cases / 3 assertions RED	1 case / 3 RED
interpolation scale inverted	3 / 11 RED	—
codebook offset dropped	1 / 1 RED	—
attention made non-causal	4 / 65 RED	—
frame row keeps depth position 0	1 / 24 RED	—
bf16 rounding removed	green (by design — fp32 there)	2 cases / 4 assertions RED
RMSNorm collapsed to one rounding	green (by design)	1 case / 3 assertions RED
code-row alignment off by one	green (not reachable there)	1 case / 4 assertions RED

The last three are the point of having two gates: they are invisible at reduced
dimensions in fp32 and caught at full scale in bf16.

One mutation is recorded because it did not move: mutating only the score
loop of the causal attention is a numerical no-op, since softmax is
shift-invariant and the later loops still stop at i. That is a bad mutation,
not a coverage hole, and it is noted in the mutation script so the next reviewer
does not read it as one.

Still owed on W2

The 8.6B Qwen3ForCausalLM forward itself. frame_hiddens[:, :4096] is the
language model's own hidden state; reproducing it means running that model
teacher-forced on the golden codes through our landed Qwen3 dense path, which
needs an inputs_embeds entry it does not have. Recorded in §5 and in the spec's
## Now rather than discovered later.

Gate + environment

scripts/agent-preflight.sh --no-require-role is green except
test_cpu_x86_llamacpp_floor, which fails with NO_QUIET_WINDOW after 30s (busy=109% load=98.35) — the environmental signature, on a box loaded by this
lane's own builds, and unrelated to any path this PR touches
(#631). audit-live-rows.py --check also exits 1, verified RED on
origin/main in a clean worktree with the identical command, flagging this
same row as abandoned-ACTIVE because no commit on main named the full row ID;
this PR's commits now do.

Every measurement here was taken on CPU. Nothing in this PR touches the speed
axis, and SGLang-Omni remains gateable = no.

Note: windows-msvc-cpu, windows-msvc-vulkan and sanitize-cpu (address,undefined) are red on main from other lanes (#645, #674) and will show
here; they are not this change.

🤖 Generated with Claude Code