diff --git a/.agents/benchmark-record.md b/.agents/benchmark-record.md index 342b283d9..72283f556 100644 --- a/.agents/benchmark-record.md +++ b/.agents/benchmark-record.md @@ -22379,3 +22379,129 @@ oracle that never produced a file. **No parity number is claimed and the token gate is still unclaimed.** `our-ON == our-OFF` remains FALSE and unattributed. Owed detail lives under `## Owed` in [`mtp-k-gt-1.md`](specs/mtp-k-gt-1.md). +## MUSIC3-DIT-DEVICE — the 2.4B fp32 DiT on `thor:gpu0`, per-forward A/B against the host reference (2026-08-17, `row/MUSIC3-DIT-DEVICE`, #672) + +**Not a parity ratio.** There is no reference leg: SGLang-Omni is `gateable = no` +and serves the native layout, so this is an INTERNAL two-arm number about our own +host reference vs our own device arm. Every axis in `docs/BENCHMARKS.md` against +the reference stays `PENDING`. + +### Device, named because a number without one is meaningless here + +**`thor:gpu0` — NVIDIA Thor, sm_110, aarch64, 14 cores, ~122 GB UNIFIED, driver +595.78.** Nothing below is compared to a `dgx:gpu0` (GB10) or `orin:gpu0` number; +the three boxes are different machines. Image `vllmcpp-thor:cuda13.0.1`, nvcc +13.0.88, `-DVLLM_CPP_CUDA=ON -DVLLM_CPP_CUDA_ARCHITECTURES=110 +-DVLLM_CPP_TRITON=OFF -DVLLM_CPP_SERVER=ON`, no cutlass. Checkpoint mounted +read-only from the NAS. Same binary, same weights, same committed inputs on both +arms; the arms never overlapped. + +**Lease discipline, recorded because it was imperfect.** The CUDA build, the +two-arm correctness series and the first timing runs were driven over `ssh` under +`flock $HOME/gpu.lock`, which is what this row's brief specified and which the +`rc` lease system has since superseded — during that window the fleet reported +`thor:gpu0` as FREE while it was in use. The device-arm timing series that the +speed claim rests on was run under a real `rc hold` on `thor:gpu0` +(`a91d21dc`, taken 10:32Z, released 10:54Z on completion). The hold carried no +`--reason` string and a 75 m TTL for ~22 m of work; both are recorded as errors, +and single commands should go through `rc run --max-runtime` instead. + +### What is timed + +`VLLM_CPP_MUSIC3_DIT_REPEAT=R` runs the guided velocity R times per timestep in +`tests/parity/test_minimax_music3_acoustic_real.cpp`. The timer brackets ONLY +that loop: the 9.7 GB checkpoint load and the weight staging are outside it, and +staging is timed separately. NOTE, corrected in fresh review: the golden reads +(4x `LoadF32Npy`, 2x `Compare`, 2x `ReportInto`) are INSIDE the bracket +(`test_minimax_music3_acoustic_real.cpp:592-606`), which inflates the intercept +and makes the reported per-forward number conservative, not inflated. One guided velocity is TWO +DiT forwards (the conditional and the unconditional CFG branch). + +| arm | repeats | forwards | loop | per forward | staging | box load | +|---|---|---|---|---|---|---| +| CPU (`VLLM_CPP_MUSIC3_DEVICE=0`) | 1 | 4 | 819.818584 s | **204.954646 s** | 0 (no-op) | 3.42 | +| CPU (`=0`) | 1 | 4 | 819.992 s | **204.998 s** | 0 (no-op) | 10.37 | +| CUDA (`=1`) | 1 | 4 | 0.749077 s | **0.187269 s** | 0.603561 s | 4.79 | +| CUDA (`=1`) | 3 | 12 | 2.110301 s | **0.175858 s** | 0.660600 s | 5.32 | +| CUDA (`=1`) | 1 | 4 | 0.743367 s | **0.185842 s** | 0.609463 s | 5.1 | +| CUDA (`=1`) | 1 | 4 | 0.743881 s | **0.185970 s** | 0.612787 s | 4.44 | + +Two-point fit over the device arm's 4- and 12-forward runs: + + slope = 0.170607 s per forward intercept = 0.063012 s + +**Per DiT forward at the capture's geometry (latent length 86, sequence 87): +204.955 s host vs 0.1706-0.1873 s device — 1102x on the matched R=1 pair, 1201x +on the fitted slope.** The device R=1 point was taken THREE times across two +sessions, bracketing R=3, at 0.749077 / 0.743367 / 0.743881 s: a 0.77 % spread. + +**The contention asymmetry was measured away, not argued away.** The first CPU +point sat at box load 10.37 against the device arm's 4.4-5.3, which would have +inflated the ratio if it mattered. Re-taken on an idle box (load 3.42) with the +fixed instrument it reads **204.954646 s against 204.998 s — 0.021 %**. The host +DiT forward is single-threaded on a 14-core box, so a load of 10 still leaves it +a core. Both points are in the table rather than the convenient one. + +### The weights are staged ONCE, as a measurement + +One staging costs 0.60-0.66 s. The entire FOUR-forward loop costs 0.745 s and the +TWELVE-forward loop 2.110 s; twelve stagings would be 7.35 s by themselves. The +loop's fitted intercept is 0.063 s, a tenth of a single staging. A per-forward +or per-window upload is excluded arithmetically, not by reading the code. + +### The whole-process ratios — lower, and the honest ceiling on what a user sees + +Same gate binary end to end, including the identical 9.7 GB NAS load on both +arms: 1054-1071 s (CPU) vs 238-298 s (CUDA), **3.5-4.5x** — the spread is NAS +cache state, not compute. The earlier full two-arm +correctness series, identical scripts throughout: 49 min 17 s vs 15 min 49 s, +**3.12x**. Load averages across the series 4.0-5.1, box otherwise idle apart from +`k3s`; `uptime` recorded on both sides of every run and the host never rebooted. + +The distance between 1100x on the DiT and 4x on the process IS the owed list: the +checkpoint load, the DAC vocoder and the RVQ depth decoder are unchanged and now +dominate. + +### Extrapolation, labelled as one + +The only geometry measured is the capture's single 86-frame window. Applying the +fit to the 660 forwards a 45 s clip runs at the shipped defaults (30 steps x 2 CFG +x 11 windows) gives **~37.6 h of DiT on the host against ~113 s on the device**, +with the one-time staging 0.53 % of the device total. That is an extrapolation +from one window geometry and is not a measured clip-level result. + +**No end-to-end song pair is offered, and that is a limit rather than an +omission.** At 30 steps the host arm's DiT alone is ~37.6 h, so an e2e pair is +not runnable on the CPU arm at a realistic setting; at a setting short enough to +run, the DiT is a small enough share that the pair would be measuring the +vocoder. + +### Correctness taken in the same series, at bounds that did not move + +Full scale, the real 2.4B fp32 checkpoint against the committed oracle capture, +11 008 values per step, `kDitRelTol` 1e-4 / `kDitAbsFloor` 5e-5 / +`kDitMeanAbsTol` 5e-6 — all unchanged: + +| arm | step | bit-identical | mean\|d\| | max\|d\| | outside | +|---|---|---|---|---|---| +| Thor CPU | first | 423 (3.843 %) | 1.71434e-06 | 2.38419e-05 | 0 | +| Thor CPU | last | 235 (2.135 %) | 2.22396e-06 | 2.83718e-05 | 0 | +| Thor CUDA | first | 473 (4.297 %) | 1.64344e-06 | 2.47955e-05 | 0 | +| Thor CUDA | last | 222 (2.017 %) | 2.44677e-06 | 2.59876e-05 | 0 | +| CONTROL torch-vs-torch | first | 15.416 % | 7.526e-07 | 7.153e-06 | — | +| CONTROL torch-vs-torch | last | 5.596 % | 1.424e-06 | 1.335e-05 | — | + +The Thor CPU arm reproduces the x86-64 numbers already recorded for this gate +VALUE FOR VALUE, so the CPU path is unchanged across two architectures. + +### Instrument defect found and fixed inside this series + +The first timing line printed `DIT_TIMING arm=1` on the CPU run: a `const char*` +in a doctest `MESSAGE` chain takes the bool conversion. It is the SAME defect +§11.5 recorded for this row's arm banner, reintroduced by a fresh `<<` chain. +Both lines are now assembled as one `std::string`. EVERY number in the table was +then re-taken with the fixed instrument, and the CPU arm's pre-fix point is kept +beside its post-fix twin rather than replaced by it, because the pair is what +proves the label defect never touched the values: 819.992 s against +819.818584 s. + diff --git a/.agents/specs/minimax-music3.md b/.agents/specs/minimax-music3.md index e3062f923..e5898e495 100644 --- a/.agents/specs/minimax-music3.md +++ b/.agents/specs/minimax-music3.md @@ -1094,6 +1094,15 @@ language model is slow", and the language model is not the part that is slow — the LM's own weight load is 180 s of I/O and its forward is 12-14% of the AR profile. +**One sentence above is now out of date on the device arm, and is corrected here +rather than left to mislead.** *"the depth decoder and the DiT do not go through +`vt` at all"* was true when it was written; it is still true of the DEPTH DECODER +on both arms and of the DiT under `--speech-device 0`. It is NOT true of the DiT +under `--speech-device 1`, which §13 routes through `vt::MatmulBT`, +`vt::LayerNorm`, `vt::AttentionCross`, `vt::RopeFromCache`, `vt::SiluAndMul` and +`vt::Add` with device-resident weights. Every profile number quoted above is the +CPU arm's and still describes it exactly. + --- ## 10. The parity sweep, the music-only server, and the weights record (#672) @@ -1454,10 +1463,27 @@ own measurement and this change's evidence budget went to the device seam. **That last paragraph is now DONE — §12.** It landed with its own measurement and its own gate, and it took `vocoder1d::Conv1d` with it. -**And the paragraph above it is now DONE too — §13.** `vt::Conv1d` and -`vt::ConvTranspose1d` exist, with a CPU provider and a CUDA provider, and all -four `vocoder1d` consumers route through them. The two REMAINING device rows are -the depth decoder and the DiT, which are unchanged and owned elsewhere. +**TWO of the three device rows are now DONE.** The vocoder row closed in §13: +`vt::Conv1d` and `vt::ConvTranspose1d` exist with a CPU provider and a CUDA +provider, and every `vocoder1d` consumer routes through them. The DiT row closed +in §14. + +**A coverage hole in this row, found by fresh review and filed as #1131.** The +DiT device arm's kernels and staging are gated — eight mutations against them go +RED — but its **production switch is not**. Setting `on_device = false` in +`Music3DenoiseChunks`, or disabling the half-set refusal, leaves every suite +GREEN. A change that silently stopped the DiT reaching the device would be +invisible, and the arm would run on the host with every number still looking +right. The gate that closes it must assert the device path was TAKEN (invocation +count or resident dtype), not merely that outputs agree — the two arms agree +numerically by design. + +**Only the depth decoder remains, and its "blocked on" entry above is now known +to be WRONG in one respect**, corrected in §14.5: it is not "nothing but the +work". The shipped depth decoder runs `ArCompute::kBFloat16`, which rounds the +RESULT of every op to bf16 (`minimax_music3_ar.cpp:36-40`), so routing it through +an f32 `vt::MatmulBT` would silently drop that rounding. Mirroring it needs bf16 +STORAGE — a dtype decision with its own numeric evidence, not a transcription. ### 11.5 Evidence — Jetson Thor, sm_110, in the container @@ -2181,3 +2207,308 @@ all six runs (`8250.57898`, `-633.539342`, `903.742105`, `657.314583`). The device arm shipping OFF by default was already the right call for numerics reasons (§13.6); this measurement says it would also have been the right call for speed. + +--- + +## 14. The 2.4B fp32 DiT reaches the device (#672) — §11.4's second owed row + +§11.4 recorded three device rows as owed. §12 closed the arm-independent one. +This closes the **DiT**, which is the one that mattered most, and it says up +front which of the other two it does not close and why. + +### 14.1 Why this row and not another + +The DiT is not one stage among six; at a real duration it is the request. + +A 45 s clip at the shipped defaults (`num_inference_steps` 30) runs `DitForward` +**660 times** — 30 steps x 2 CFG branches x 11 windows — and each call is 36 +blocks over `length + 1` tokens at inner dim 2048, ff 8192. That is on the order +of **634 TFLOP in the DiT against ~29 TFLOP for the entire autoregressive half**: +the DiT is roughly **20x everything else in the model put together**. On the +scalar host loops it is measured in hours; one run was killed at 8 h 11 m having +averaged 4.3 of 20 cores. + +That is also why §11.5's device arm reached only 0.946x. It moved the 8.6B +language model, which is real work, and left the stage that is twenty times +larger on the host. A device arm that does not include the DiT is a device arm +for the minority of the profile. + +### 14.2 What moved, onto which shared op, and what did NOT + +**No new kernel.** Every op below already existed with a CUDA provider; this row +adds a forward that composes them, not a kernel that competes with them. + +| reference helper (`minimax_music3_acoustic.cpp`) | shared op | +|---|---| +| `Linear` | `vt::MatmulBT` (+ `vt::Add` for the rank-1 bias) | +| `LayerNorm` | `vt::LayerNorm` | +| `ApplyPartialRotary` | `vt::RopeFromCache` over a `[seq, rotary_dim]` cache | +| `Attention` (NON-causal) | `vt::AttentionCross`, `bias = nullptr` | +| `value * silu(gate)` | `vt::SiluAndMul`, over a stage-time half swap | +| residual adds | `vt::Add` | +| `PointwiseConv` (both 1x1 convolutions) | `vt::MatmulBT` on the transposed activation | + +The stage table, after: + +| stage | `--speech-device 1` runs it | +|---|---| +| 8.6B `Qwen3ForCausalLM`, prefill + decode + paged KV | **device** (§11) | +| guided logits, top-k draw, frame feedback | host | +| 0.646B RVQ depth decoder | **host** — OWED, and §14.5 corrects why | +| condition mix (once per WINDOW, not per step) | **host** — OWED | +| **2.4B fp32 DiT, every step, both CFG branches** | **device — THIS ROW** | +| scheduler, CFG mix, Euler step, overlap blend, carry | host (elementwise on `[128, length]`; not the cost) | +| DAC Flow-VAE vocoder | **host — BLOCKED on a missing op** (§11.4) | + +### 14.3 Four things this had to get right + +**The 1x1 convolutions are GEMMs, and that is what unblocked the row.** `vt` has +no CUDA 1-D convolution provider at all — the finding §11.4 recorded against the +vocoder applies here too, because the DiT's `preprocess_conv` and +`postprocess_conv` are `nn.Conv1d(kernel=1)`. But a kernel-1 convolution over +`[C, L]` is a GEMM once the activation is transposed: + + conv(x)[co][t] = SUM_ci W[co][ci] * x[ci][t] + transposed: conv(x)^T[t][co] = SUM_ci x^T[t][ci] * W[co][ci] = MatmulBT(x^T, W) + +So the forward works FRAME-MAJOR `[length, channels]` throughout and transposes +once on the host at each end, where the tensors are `[128, length]`. No +convolution op is needed, nothing is hand-rolled outside the seam, and the +vocoder's blocker does not transfer. + +**The half swap is an identity applied exactly once.** Upstream computes +`ff_out(gate_states * silu(gate))` where `gate_states, gate = ff_in(x).chunk(2, +-1)` — the FIRST half is the value, the SECOND is what SiLU runs on +(`transformer_minimax_music3.py:142-143`). `vt::SiluAndMul` computes +`silu(x[:, :D]) * x[:, D:]`: the opposite assignment. Exchanging the two ROW +BLOCKS of the projection and the two halves of its bias — **once, at stage +time** — makes the shared op compute upstream's expression exactly, with no +per-step permutation. 660 forwards x 36 layers would otherwise permute a +`[seq, 16384]` tensor 23 760 times per clip. The gate for this is a mutation, not +an assertion: §14.4. + +**The rotary is the LEADING slice, and `vt::RopeFromCache` already rotates +exactly that.** Music3 ships `rotary_dim` 32 of `head_dim` 64 and rotates only +the leading window, leaving the tail copied through +(`minimax_music3_acoustic.cpp:500-514`). `RopeFromCacheKernel` indexes +`row + pair` and `row + pair + half` within each head and computes +`x*c - y*s, x*s + y*c` — the same rotation over the same slice. `BuildDitRotaryTables` +returns cos/sin already duplicated across both halves of the window, so the cache +this forward builds is the FIRST half of each, packed `cos | sin`. + +**The attention is NON-causal and `vt::Attention` is not it.** Upstream +dispatches with no mask (`:97-103`), so every token attends to every token +INCLUDING the prepended timestep one. `vt::Attention` is the causal op; using it +would have silently masked the future and still produced a finite, plausible +tensor. `vt::AttentionCross` with a null bias is the op that means this. + +### 14.4 Correctness — same goldens, same bounds, nothing widened + +**The CPU arm is bit-identical, and structurally rather than by measurement.** +`minimax_music3_acoustic.cpp`, `minimax_music3_ar.cpp`, `minimax_music3_llm.cpp` +and `vocoder1d.cpp` have a **zero diff** in this change. `--speech-device 0` +takes the same `DitForward`, source byte for source byte, so there is no number +to move. The device forward is an ADDITIONAL entry point in a new file +(`minimax_music3_device.cpp`), which is the shape `minimax_h3_device.cpp` and +`ltx2_device.cpp` already use. + +**Reduced dimensions, against upstream's own goldens, at the EXISTING bound.** +`DitForwardDevice` is checked through the SAME `ExpectClose` at the SAME +`kRelTol` 1e-5 / `kAbsFloor` 1e-6 as `DitForward`, and each case reports BOTH +arms' distance to the golden — because the question is not whether the two arms +agree with each other (a shared-helper comparison proves consistency, not +correctness) but whether the device arm is as close to UPSTREAM as the host arm +already is: + +| arm | worst \|arm - upstream\| | +|---|---| +| host `DitForward` (the accepted control) | 1.565e-07 | +| device forward, CPU backend | 1.192e-07 | +| device forward, **CUDA sm_110** | 2.980e-07 | + +All three are inside the 1e-6 absolute floor with room to spare, and **no +tolerance was relaxed**. The CPU-backend arm is closer to upstream than the host +loops are; the CUDA arm is about 1.9x the host arm's distance and about a fifth +of the bound. + +**Two mutations, because a bound that nothing violates has not been shown to +discriminate.** + +* **The half swap.** Pre-swapping the host weights makes the stage-time swap undo + the test's, so the forward computes `silu(value) * gate` — the wrong network, + same shapes, same finiteness. **20 of 20 values outside the bound, worst + \|diff\| 1.538e-03**, four orders above the noise. The pair pins the DIRECTION, + not just the magnitude: routing it the other way round would fail the right + case and pass this one. +* **The condition.** Conditional and unconditional forwards must be different + tensors — a DiT that dropped its conditioning would match both goldens + identically. 20 of 20 differ, on both backends. + +Two more cases guard the staging contract itself: every mis-sized weight is +refused **at stage time** naming the tensor (before 9.7 GB moves at real +dimensions), and `release_host` is asserted to leave the source vectors empty +AND at zero capacity while the staged copy still reproduces the golden — which is +also the check that would catch a released host buffer uploaded without a +synchronize. + +**FULL SCALE — the real 2.4B fp32 checkpoint against the oracle capture, on +sm_110.** `tests/parity/test_minimax_music3_acoustic_real.cpp` now takes +`VLLM_CPP_MUSIC3_DEVICE` (default 0 = CPU, so an unset environment reproduces +every number this file ever printed) resolved through the SAME +`multimodal::SpeechEngineDeviceType` the engine calls. Both arms, same box, same +binary, same goldens, **same bounds** — `kDitRelTol` 1e-4 / `kDitAbsFloor` 5e-5 / +`kDitMeanAbsTol` 5e-6, all unchanged. 11 008 values per step: + +| arm | step | bit-identical | mean\|d\| | max\|d\| | outside | +|---|---|---|---|---|---| +| **Thor CPU** (`device 0`) | first | 423 (3.843 %) | 1.71434e-06 | 2.38419e-05 | **0** | +| **Thor CPU** | last | 235 (2.135 %) | 2.22396e-06 | 2.83718e-05 | **0** | +| **Thor CUDA** (`device 1`) | first | 473 (4.297 %) | **1.64344e-06** | 2.47955e-05 | **0** | +| **Thor CUDA** | last | 222 (2.017 %) | 2.44677e-06 | **2.59876e-05** | **0** | +| CONTROL (torch vs torch, `set_num_threads(1)`) | first | 15.416 % | 7.526e-07 | 7.153e-06 | — | +| CONTROL | last | 5.596 % | 1.424e-06 | 1.335e-05 | — | + +**Three things this table shows that one arm could not.** The device arm sits +ON TOP of the host arm rather than beside it — better on two of the four figures +(more bit-identical and a lower mean at the first step, a lower max at the last) +and marginally worse on the other two, which is what two correct float32 +implementations of the same graph look like. Both arms sit at the same +multiple of the recorded torch-vs-torch control (about 1.2-1.7x its mean, 2-3.5x +its max), so the device arm did not move the row's relationship to the control. +And **the Thor CPU arm reproduces the x86-64 numbers this spec already recorded — +3.843 %, 1.714e-06, 2.384e-05; 2.135 %, 2.224e-06, 2.837e-05 — VALUE FOR VALUE**, +so the CPU path is unchanged across two architectures, not merely unchanged on +the box that measured it. + +The four ARM=1 cases that print those numbers are 464 assertions against the +CPU arm's 461; the three extra are this row's staging `CHECK` and the two +`REQUIRE`s that refuse a device arm with no staged weights. + +### 14.6 Speed — MEASURED, on one named device, per DiT forward + +**Device: `thor:gpu0` — NVIDIA Thor, sm_110, aarch64, 14 cores, ~122 GB UNIFIED, +driver 595.78.** Every number below is from that one box. No number here is +compared to one from `dgx:gpu0` (GB10) or `orin:gpu0`, because those are +different machines and a ratio across them would mean nothing. + +Image `vllmcpp-thor:cuda13.0.1`, nvcc 13.0.88, configured `-DVLLM_CPP_CUDA=ON +-DVLLM_CPP_CUDA_ARCHITECTURES=110 -DVLLM_CPP_TRITON=OFF -DVLLM_CPP_SERVER=ON`, +no cutlass. Checkpoint read-only from the NAS. Same binary, same weights, same +committed inputs on both arms; the arms never overlapped. + +**What is timed is the DiT and only the DiT.** `VLLM_CPP_MUSIC3_DIT_REPEAT=R` +makes the gate run its guided velocity R times per timestep instead of once, and +the timer brackets that loop — the 9.7 GB checkpoint load and the weight staging +are outside it, and the staging is timed separately. + +**Corrected in fresh review:** an earlier revision of this paragraph also claimed +the GOLDEN READS were outside the bracket. They are not — four `LoadF32Npy` +calls, two `Compare` and two `ReportInto` sit INSIDE the `t0`/`loop_s` bracket +(`test_minimax_music3_acoustic_real.cpp:592-606`). That inflates the intercept and +makes the per-forward number SLOWER than the pure forward, so the headline ratio +is conservative rather than inflated — but the sentence was wrong as written, and +a reader checking the intercept against the fit would have been misled. + +| arm | repeats | forwards | loop | per forward | staging | box load | +|---|---|---|---|---|---|---| +| CPU (`device 0`) | 1 | 4 | **819.818584 s** | **204.954646 s** | 0 (host, no-op) | 3.42 | +| CPU (`device 0`) | 1 | 4 | **819.992 s** | **204.998 s** | 0 (host, no-op) | 10.37 | +| CUDA (`device 1`) | 1 | 4 | **0.749077 s** | **0.187269 s** | 0.603561 s | 4.79 | +| CUDA (`device 1`) | 3 | 12 | **2.110301 s** | **0.175858 s** | 0.660600 s | 5.32 | +| CUDA (`device 1`) | 1 | 4 | **0.743367 s** | **0.185842 s** | 0.609463 s | 5.1 | +| CUDA (`device 1`) | 1 | 4 | **0.743881 s** | **0.185970 s** | 0.612787 s | 4.44 | + +**Per DiT forward at the capture's geometry (latent length 86, seq 87): +204.955 s on the host, 0.1706-0.1873 s on the device — between 1094x and +1201x.** (An earlier revision wrote the low end as 1102x. That is the ratio at +the FASTEST R=1 point, 0.185970 s; the range's slow end is 0.187269 s, and +204.954646 / 0.187269 = 1094x. Caught in fresh review.) The two-point fit over the device arm's 4- and 12-forward runs gives + + slope = 0.170607 s per forward intercept = 0.063012 s + +so the ratio is 1102x taken on the matched R=1 pair and 1201x taken on the +fitted per-forward slope. The device R=1 point was taken THREE times across two +sessions, bracketing the R=3 point, at 0.749077 / 0.743367 / 0.743881 s — a +0.77 % spread. + +**The contention asymmetry was checked rather than assumed, and it is nil.** The +first CPU point was taken at box load 10.37 while the device points sat at +4.4-5.3, which would have inflated the ratio if it mattered. It was re-taken on +an idle box (load 3.42) with the fixed instrument: **204.954646 s vs 204.998 s, +agreeing to 0.021 %**. The host DiT forward is single-threaded and this box has +14 cores, so a load of 10 still leaves it a core. Both CPU points are reported +above rather than the convenient one. + +**The weights are staged ONCE, and this is the measurement that says so rather +than the code comment.** One staging costs 0.60-0.66 s. The ENTIRE four-forward +loop costs 0.745 s and the twelve-forward loop 2.110 s; twelve stagings would be +7.35 s on their own. The loop's fitted intercept is 0.063 s — a tenth of one +staging. A per-forward upload is arithmetically excluded by the numbers, not +argued away. + +Extrapolated to a full clip — and it is an EXTRAPOLATION, labelled as one, +because the only geometry measured is the capture's single 86-frame window — the +660 forwards of a 45 s clip at the shipped defaults are **~37.6 h of DiT on the +host against ~113 s on the device, with the one-time staging 0.54 % of the +latter**. + +**The whole-process ratios, which are lower and are the honest ceiling on what a +user sees today.** The same gate binary end to end, including the identical +9.7 GB NAS load on both arms, ran 1054-1071 s (CPU) vs 238-298 s (CUDA) — +**3.5-4.5x**, the spread being NAS cache state rather than compute; +and the earlier full two-arm correctness series, identical scripts throughout, +ran 49 min 17 s vs 15 min 49 s — **3.12x**. The gap between 1100x on the DiT and +4x on the process is the point of §14.5: the load, the host vocoder and the depth +decoder are unchanged, and they now dominate. + +**No end-to-end song pair is offered.** At the shipped 30 steps the host arm's +DiT alone is ~37.6 h, so an e2e pair at a realistic setting is not runnable on +the CPU arm; at a setting short enough to run, the DiT is a small enough share +that the pair would measure the vocoder. The per-forward A/B above is the +measurement that isolates what this row changed, and no clip-level speed claim is +made from it. + +**No parity claim.** SGLang-Omni is still `gateable = no`; every reference axis +in `docs/BENCHMARKS.md` stays `PENDING`. + +### 14.7 One instrument defect, found inside this change + +The first revision of the timing line printed **`DIT_TIMING arm=1`** on the CPU +run. `vt::DeviceTypeName` returns `const char*`, and a `const char*` fed to +doctest's `MESSAGE` chain takes the **bool** conversion and prints `1`. The +staging line had it too, printing `(1)` where it meant `(host, no-op)`. + +**This is #672's own §11.5 defect reappearing in a new line**, which is the +reason it is recorded here rather than quietly fixed: the lesson from the first +occurrence was written down, and a fresh `<<` chain reintroduced it anyway. Both +lines are now assembled as one `std::string` and printed, which is what the arm +banner beside them already did — and the banner is why the numbers survived, +because it said `ran on 'cpu' (VLLM_CPP_MUSIC3_DEVICE=0)` correctly while the +line below it said `arm=1`. + +**Every number in §14.6 was re-taken with the fixed instrument**, and the CPU +arm's pre-fix point is kept beside its post-fix twin rather than replaced by it, +because the pair is what proves the label defect never touched the values: +819.992 s (pre-fix, `arm=1` printed, load 10.37) against 819.818584 s (post-fix, +`arm=cpu` printed, load 3.42). The correct banner sat above both, the process +wall clocks corroborate both, and the correctness numbers both runs printed match +the x86-64 values this spec already recorded value for value. + +### 14.5 What is still OWED, and a correction to §11.4 + +§11.4 said the depth decoder was blocked on "nothing but the work". **That is +wrong, and this row is where it was found out.** The shipped depth decoder and +condition mix run at `ArCompute::kBFloat16`, which rounds the RESULT of every op +to bf16 (`minimax_music3_ar.cpp:36-40`) because that is what torch stores. +Routing them through an f32 `vt::MatmulBT` would silently drop that rounding — a +change to the numbers wearing a refactor's clothes. Mirroring them needs bf16 +STORAGE so the shared op rounds where the reference's `Store` rounds, which is a +dtype decision with its own numeric evidence, not a transcription. It is also +worth ~15 TFLOP against the DiT's 634, so it is second in size as well as second +in order. + +The condition mix has a further reason to be second: it runs **once per window**, +not once per step, so it is outside the 660-forward loop entirely. + +The vocoder row is unchanged: `vt` still has no `ConvTranspose1d` with any +provider, that op has three consumers, and it is its own row. diff --git a/CMakeLists.txt b/CMakeLists.txt index d21e491f2..9798db800 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -778,6 +778,7 @@ add_library(vllm STATIC src/vllm/model_executor/models/minimax_music3_quant.cpp src/vllm/model_executor/models/minimax_music3_ar.cpp src/vllm/model_executor/models/minimax_music3_acoustic.cpp + src/vllm/model_executor/models/minimax_music3_device.cpp src/vllm/model_executor/models/minimax_music3_llm.cpp src/vllm/model_executor/models/minimax_music3_speech.cpp src/vllm/model_executor/models/gpt2.cpp diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index ea50a632b..1665a7414 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -486,8 +486,9 @@ built on it rather than keeping the flattering one. | vLLM 0.26 re-benchmark | Pending | Re-run the binding grids on the advanced pin | | MiniMax-H3 FP4 speed (W-FP4a) | **Measured GB10 (`row/H3-FP4-GPU-E2E`).** Marlin W4A16 byte-exact vs bf16; fp4 a memory win, 0.8x bf16/forward. Real-ckpt fp4-resident e2e RUNS (mp4/wav) | fp4 speed CLOSED. bf16-vs-quant A/B: ENCODER half MEASURED (§8.15), DiT half NOT (no bf16 render exists). Detail: benchmark-record + spec §8 | | LTX-2.5 axes | Speed `PENDING` (vllm-omni#6066 has no native 2.5), binding oracle too. **SIZE: 704x448/25f and 448x256/25f both COMPLETE on GB10 (4231 s, 3085 s)**; one run each, contended box, no oracle, no ceiling (#1088) | Wall is NOT the VAE decode after #1041/#1009: a ~1731 s serial phase FLAT in resolution is 57-66% (#1087). ~59 GiB cliff did NOT recur (floor 38.9 GiB). 2 baselines UNRESOLVED (lock). PROMPTED real-ckpt render OWED | -| MiniMax-Music3 (`MiniMaxMusic3ForConditionalGeneration`) | **Every axis vs the reference stays `PENDING`.** A PARTIAL device arm now exists (#672): only the 8.6B LM runs on the accelerator, so this is an internal two-arm number and NOT a parity ratio | Denominator: SGLang-Omni `748a0b43` in its production configuration (both CUDA graphs, compiled DIT and DAV, batched seeded sampling) | +| MiniMax-Music3 (`MiniMaxMusic3ForConditionalGeneration`) | **Every axis vs the reference stays `PENDING`.** A PARTIAL device arm now exists (#672): the 8.6B LM and the 2.4B fp32 DiT run on the accelerator, so the rows below are internal two-arm numbers and NOT parity ratios | Denominator: SGLang-Omni `748a0b43` in its production configuration (both CUDA graphs, compiled DIT and DAV, batched seeded sampling) | | MiniMax-Music3 device arm, Jetson Thor sm_110 (#672) | `--device 1` vs `--device 0`, same request/seed, idle box: 2 AR frames **846.6 vs 835.1 s (1.014x SLOWER)**; 10 frames **1430.4 vs 1512.1 s (0.946x)**. Fit: **-11.65 s/frame, +34.8 s fixed** | A third duration (the fit has no residual), and moving the depth decoder + DiT + vocoder, which are 5 of 6 stages and still host scalar loops | +| MiniMax-Music3 DiT device arm, `thor:gpu0` sm_110 (#672) | Per DiT forward at the capture's geometry, same binary/weights/inputs, idle box: **204.955 s host vs 0.186-0.187 s device, 1094-1102x** (1201x fitted). Staged ONCE (0.61 s; loop intercept 0.063 s). Whole process 3.5-4.5x | e2e song pair NOT runnable (host DiT alone ~37.6 h at 30 steps). Depth decoder/condition mix (bf16-storage), vocoder (no `ConvTranspose1d`) still host. Detail: benchmark-record | | MiniMax-Music3 CPU host kernels, x86-64 20-core (#672) | KERNEL A/B at the vocoder's real geometry, min of 5 interleaved rounds: convolution chain **13.36 -> 1.25 s, 10.7x**; `Conv1d` 12.03x, `LinearNoBias` 10.88x. Output fingerprints IDENTICAL on both arms | e2e pair VOID (cold CIFS cache; a foreign `ctest` at load 76.6) and re-running. Stages 0/1 only ~2x: the pivot trades WEIGHT locality for accumulator locality. Detail: benchmark-record | | MiniMax-H3 render coherence (`row/H3-RENDER-CLOSE` #77) | **CLOSED: a COHERENT scene on GB10.** #70/#74 white was wrong-PARTITION usage (t2va on the ref2va ckpt); t2va on the FL2VA GGUF renders a prompt-matched orange cat (adj-cos 0.95 vs 0.06, no patch-grid) | Verified first: t2va inputs byte-exact vs upstream; CUDA device==host at seq 1920. Follow-up `H3-TASK-PARTITION-GUARD`: the task/partition mismatch now RAISES 1:1 with `_resolve_task` (spec §8.6-8.7) | | MiniMax-H3 image conditioning (`row/H3-CONDITIONED-E2E`, `row/H3-VISION-SCATTER`, `row/H3-REF2VA-ASSEMBLY`) | **fl2va COHERENT; ref2va assembly bug FIXED+gated.** vision→cond scatter gated; ref2va block-dim double-division fixed + RED-first gated (128 vs 512) + a permanent ref2va DiT-forward rung (§8.10) | grid RE-ATTRIBUTED: with the fix ref2va grids in fp4 AND bf16, and t2va with no refs on the ref2va NVFP4 also grids while FL2VA-GGUF renders, so it is the **NVFP4 checkpoint/loader**, NOT assembly/fp4 (§8.10) | diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 1f9974f04..8cdf9c5f8 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -167,7 +167,7 @@ in `ltx2_text_encoder.cpp` is the call that would have to change. | Whisper audio encoder | openai/whisper-small; whisper-large-v3 (Voxtral cfg) | encoder tower 77/77; large-v3 tower 203/203 | pending | | MiniMax-H3 DiT (`MiniMaxH3DiTModel`, vllm-omni lane) | MiniMax-H3 (33.1B video+audio) | portable 79/79; all three modalities COHERENT on Q4_K_M (§8.20); PRUNED ckpts run, Q8_0 seam 0.9941 (§8.21); ref2va grid was NVFP4 quant error, §8.9 REFUTED; GGUF/NVFP4/bf16 shards stream | FP4/Marlin landed; speed pending; no bf16 render yet. Render from the Q4_K_M GGUF, not the NVFP4 arm. Krea 2 text-to-image (roadmap C11) is scoped to reuse these DiT seams | | LTX-2.5 DiT (`LTX2VideoTransformer3DModel`, Lightricks lane) | LTX-2.5 (21.00B video+audio) | `SPIKE`. DiT, VAEs+ENCs, cond, pipeline, quant loaders gated, reduced dims. Prompt AdaLN host+dev; Gemma-4->xattn FIXTURE-gated. Img chain PPM->resize->encode->place->noise. Temporal x2 ups gated, UNDRIVEN. Render OWED | `ltx-2.5`/`ltx2-gen`. ~29 GB NVFP4/GB10, FP8 ~44 GB, +24 GB tower. FP8/torchao/NVFP4; kf abs-pos ported; BOTH DiTs load, NO `allow_unported`. IMG+LAST kf SERVED `crf=0`, A2V WAV+LoRA; DiffVAE/ref refused. PENDING | -| MiniMax-Music3 (`MiniMaxMusic3ForConditionalGeneration`, diffusers lane) | MiniMax-Music3 (8.6B Qwen3 LLM + 0.646B RVQ decoder + 2.4B fp32 DiT + DAC Flow-VAE); diffusers arm, ~28.5 GB | `ACTIVE`. Loader 1413/1413; AR, acoustic and the 8.6B LM forward all gated vs real weights; `SpeechRegistry` + `vllm_speech_*` v21 + `/v1/audio/speech`; GGUF Q4_K depth decoder value-gated. HTTP request OBSERVED (#852) | Not compared to a reference. Host kernels multi-core, same song bytes (§12). PARTIAL device arm (`--speech-device 1`, #672): 8.6B LM only. Denominator: SGLang-Omni, production config | +| MiniMax-Music3 (`MiniMaxMusic3ForConditionalGeneration`, diffusers lane) | MiniMax-Music3 (8.6B Qwen3 LLM + 0.646B RVQ decoder + 2.4B fp32 DiT + DAC Flow-VAE); diffusers arm, ~28.5 GB | `ACTIVE`. Loader 1413/1413; AR, acoustic and the 8.6B LM forward all gated vs real weights; `SpeechRegistry` + `vllm_speech_*` v21 + `/v1/audio/speech`; GGUF Q4_K depth decoder value-gated. HTTP request OBSERVED (#852) | No reference number. Host kernels multi-core, same song bytes (§12). PARTIAL device arm (#672): 8.6B LM + 2.4B fp32 DiT staged once (§14); rest host. Denominator SGLang-Omni production | | LTX-2.5 DFR base + generated keyframe slots | LTX-2.5 (21.00B video+audio) | gated vs EXECUTED upstream `dfr_layout` + 3 `dfr_pipeline` helpers @ `fd4ded7f` (`test_ltx2_dfr` 11/11, 652 assertions); canvas, tiles, stitch, carry-forward as EXACT index vectors, since each defect is plausible| `--pipeline-kind dfr`. Canvas PADS 9 to 25 then trims back; slots on the x8 grid, MARKED, read back BEFORE the trim. `num_generated_keyframes` SERVED elsewhere. Temporal ROUNDS refused (#986); detail LoRA refused (#975)| | LTX-2.5 tiled + streaming Conv VAE decode | LTX-2.5 video VAE | gated vs executed upstream `ltx_core` @ `fd4ded7f` (`test_ltx2_tiling` 10/10, 915 assertions); one-tile and untiled-spatial controls BIT-EXACT vs untiled on both causality arms; an untiled frames axis is REFUSED | Streams temporal chunks through upstream's AUTO layout (768/64 px, 80/24 frames); above one tile the pixel volume is never materialized. NO-OP below 768px and 81 frames; 81-120 IS tiled, differing 6.70% of range | | LTX-2.5 Conv VAE decode arithmetic width | LTX-2.5 video VAE | `test_ltx2_vae` "the decode's convolution accumulates in f32", entering through `Ltx2VideoDecodeStreaming`; widening the accumulator to `double`, or deleting the production call site, each turns it RED | **f32**, the width `F.conv3d` uses at f32 AND bf16 (MEASURED). Was f64 at 8 sites ([#1008](https://github.com/mudler/vllm.cpp/issues/1008)). Conv sums BLOCKED per input channel, as torch's. STORAGE stays f32; bf16 owed | diff --git a/docs/STATUS.md b/docs/STATUS.md index 451c34eae..bbea61041 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -144,7 +144,7 @@ token-for-token correctness against the pinned oracle. | InternLM2 dense (fused-`wqkv` interleaved split) | Correctness-complete, speed-pending | Token-exact 16/16 (internlm2-chat-1_8b): 12/16 strict + 4/16 bf16 near-tie (max gap 0.0 nats), 0 divergent; first InternLM model; ZERO new compute kernel (reuses the Llama dense forward; the only delta is a loader-side de-interleave of the fused `wqkv`, which packs q/k/v interleaved by KV-group) | | MiniMax-H3 (`MiniMaxH3DiTModel`, video+audio DIFFUSION) | **ABI v12 ONE SURFACE; device selector uses generic `DeviceType`; DSR 32.** t2va+fl2va COHERENT; bf16 shards STREAM | ref2va ckpt fidelity §8.12; encoder A/B §8.15; GB10 re-verify residual; CPU fold 6/137 (one queue + device provenance mutation-gated) | | LTX-2.5 (`LTX2VideoTransformer3DModel`, video+audio DIFFUSION) | **L1-L9c landed (#435).** 21.00B / 48 blocks. `VideoEngine` seam + ABI **v18**, DiT forward (CPU f32 parity, bf16 device-resident), Gemma-4 TE, both VAEs, connector, pipeline, NVFP4/FP8, keyframe bias (#658) | BOTH shipped DiTs now load inside the contract, no `allow_unported`. One runs device-resident on GB10; those 320x192/25f frames ARE a scene. A prompted render is OWED; speed and oracle parity `PENDING` | -| MiniMax-Music3 (`MiniMaxMusic3ForConditionalGeneration`, text-to-MUSIC) | **`ACTIVE`: W0-W7 landed; every stage including the 8.6B LM forward is implemented and gated (#672).** Oracle is the OPEN diffusers PR #14456 `c6da9936` | GGUF arms for 4 components owed. LM forward gated in a control; HTTP OBSERVED (#852). PARTIAL device arm, Thor sm_110 (#672): 8.6B LM only. CPU kernels 10.7x on the vocoder chain, same song BYTES. No reference number | +| MiniMax-Music3 (`MiniMaxMusic3ForConditionalGeneration`, text-to-MUSIC) | **`ACTIVE`: W0-W7 landed; every stage including the 8.6B LM forward is implemented and gated (#672).** Oracle is the OPEN diffusers PR #14456 `c6da9936` | GGUF arms for 4 components owed. LM forward gated in a control; HTTP OBSERVED (#852). PARTIAL device arm, Thor sm_110 (#672): 8.6B LM + 2.4B fp32 DiT (§14). CPU kernels 10.7x on the vocoder chain. No reference number | | Command-R / Cohere dense (`CohereForCausalLM`) | Implemented, gate-blocked | ZERO-new-kernel port grounded in vLLM `commandr.py`: weight-only Cohere LayerNorm + GPT-J full-width RoPE + PARALLEL residual + `logit_scale` + tied embeddings, all reuse; compiles, links, self-registers. No SACRED gate yet (real checkpoints HF-gated, ungated ones tiny-random, GPU box disk-full); oracle run-verified at W0. See docs/BENCHMARKS.md | | Phi-1 / Phi-2 dense (`PhiForCausalLM`, parallel residual) | Correctness-complete, speed-pending | Token-exact 16/16 (microsoft/phi-2): 9/16 strict + 7/16 bf16 near-ties (max gap 0.25 nats), 0 forward-divergent; the OLDER Microsoft Phi arch, DISTINCT from Phi-3/Phi-4; ZERO new compute kernel (GPT-J parallel residual, LayerNorm-with-bias, biased qkv/dense, partial NeoX rope 32/80, non-gated NewGELU MLP reusing `vt::GeluTanh`, untied biased lm_head); F16 dtype-aware loader | | MiniCPM dense (`MiniCPMForCausalLM`, three scalars) | Correctness-complete, speed-pending | Token-exact 16/16 (openbmb/MiniCPM-2B-sft-bf16): 10/16 strict + 6/16 bf16 near-ties (max gap 0.0 nats), 0 forward-divergent; first OpenBMB MiniCPM model; ZERO new compute kernel (the Llama/Granite dense forward plus three scalars: scale_emb, scale_depth/sqrt(layers) residual, dim_model_base logit scaling), tied lm_head; `.bin`-only weights converted to safetensors via trusted torch | diff --git a/docs/USAGE.md b/docs/USAGE.md index 230b9c445..57d0b8beb 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -1988,11 +1988,16 @@ claimed. Ask for a short duration and few `num_inference_steps` while you are checking that it works. The part that dominates is *not* the one you would guess. The 8.6B language -model goes through `vt` and uses the CPU threadpool; the RVQ depth decoder and -the DiT do not — they are scalar host loops with a double accumulator, written -that way in W2-W5 so their reduction order is reproducible against torch, and -they run single-threaded. In one 0.1 s request the depth decoder alone is the -majority of the wall clock. +model goes through `vt` and uses the CPU threadpool; the RVQ depth decoder does +not — it is a scalar host loop with a double accumulator, written that way in +W2/W3 so its reduction order is reproducible against torch. In one 0.1 s request +the depth decoder alone is the majority of the wall clock. + +At a *real* duration the DiT is the whole story instead, which is why it is the +stage that moved first: a 45 s clip at the default 30 inference steps runs the +DiT 660 times (30 steps x 2 CFG branches x 11 windows) for roughly 634 TFLOP +against about 29 TFLOP for the entire autoregressive half. On the host loops +that is measured in hours. `--speech-device 1` puts it on the accelerator. #### What runs on the device, and what does not @@ -2005,20 +2010,34 @@ direction that matters. |---|---| | 8.6B `Qwen3ForCausalLM` (prefill + every decode step, its paged KV) | **device** | | guided-logit pipeline, top-k draw, frame feedback embedding | host (two 200 000-wide rows per step; not the cost) | +| **2.4B fp32 flow-matching DiT** (every denoise step, both CFG branches) | **device**, weights staged ONCE | | 0.646B RVQ depth decoder (7 steps per frame) | **host**, scalar loops | -| condition mix, 2.4B fp32 flow-matching DiT, scheduler | **host**, scalar loops | +| condition mix (once per window), scheduler, CFG mix, Euler step | **host** | | DAC Flow-VAE vocoder (`Conv1d` / `ConvTranspose1d`) | **host**, scalar loops | The language model reaches the device because it is already routed through the shared `Qwen3DenseModel` forward that five text registrations ride — nothing was forked for it, and the only thing this option changes is which queue that -forward is handed and where its KV cache is allocated. The other stages do not, -for two different reasons, and both are owed rather than hidden: - -* the depth decoder and the DiT are host `std::vector` reference loops - under `-ffp-contract=off`, kept that way so their reduction order stays - reproducible against torch. Moving them means routing them through the shared - `vt` GEMM seam with device-resident weights, not adding a flag; +forward is handed and where its KV cache is allocated. + +The DiT reaches it the same way: through shared `vt` ops only +(`MatmulBT`, `LayerNorm`, `AttentionCross`, `RopeFromCache`, `SiluAndMul`, +`Add`), with **no new kernel**. Its 9.7 GB of fp32 weights are uploaded once per +request, before the window loop, and the host copy is released as each tensor +lands — a 45 s clip runs that forward 660 times, so a per-step or even +per-window upload would cost more than the compute it enables. `fp32 stays +fp32`: the acoustic half is float32 because upstream chose float32 for it, and +this arm mirrors that rather than buying speed with a narrower dtype. + +The remaining stages do not move, for two different reasons, and both are owed +rather than hidden: + +* the depth decoder and the condition mix are host `std::vector` + reference loops under `-ffp-contract=off`, and they run at + `ArCompute::kBFloat16` — every op's *result* is rounded to bf16, which is what + upstream stores. Routing them through an f32 GEMM would silently drop that + rounding, so mirroring them needs bf16 storage, which is a dtype decision with + its own numeric evidence rather than a transcription; * the vocoder needs `ConvTranspose1d`, and **`vt` has no such op at all** — the 1-D convolutions it does have (`vt::DepthwiseConv1d`, `vt::CausalConv1dFwd`) are depthwise or causal-with-state, and `vt::Conv2d` and `vt::DepthwiseConv1d` @@ -2026,10 +2045,16 @@ for two different reasons, and both are owed rather than hidden: this stage would need, so it is named here rather than hand-rolled outside the seam. -Because the host stages are unchanged, the CPU arm is **bit-identical** to the -one every Music3 correctness gate was taken on, and the device arm's output -differs from it exactly where the language model's own arithmetic differs — one -stage, not five. +Because the host stages are unchanged — and because `--speech-device 0` takes +the same `DitForward` it always did, source byte for source byte — the CPU arm +is **bit-identical** to the one every Music3 correctness gate was taken on. The +device arm's output differs from it exactly where the language model's and the +DiT's own arithmetic differ: two stages, not six, and neither difference is a +shape or an ordering defect. The DiT's device forward is gated against the same +upstream goldens at the same tolerance as the host one; nothing was widened for +it, and `VLLM_CPP_MUSIC3_DEVICE=1` runs that comparison on either arm +(`tests/parity/test_minimax_music3_acoustic_real.cpp`, with +`VLLM_CPP_MUSIC3_DIT=1`). **The two arms do not produce the same song, and that is structural.** The autoregressive stage has no greedy path upstream: it ends every draw in a seeded diff --git a/include/vllm/model_executor/models/minimax_music3_device.h b/include/vllm/model_executor/models/minimax_music3_device.h new file mode 100644 index 000000000..291e28b17 --- /dev/null +++ b/include/vllm/model_executor/models/minimax_music3_device.h @@ -0,0 +1,151 @@ +// MiniMax-Music3 — the DEVICE-RESIDENT acoustic forward (#672, spec §11.4). +// +// `DitForward` (minimax_music3_acoustic.cpp) is the portable reference: host +// `std::vector` throughout, one scalar loop per op. It is what every +// Music3 gate was taken on and it is NOT changed by this header — the device arm +// is an additional entry point, not a rewrite of the existing one. +// +// WHY THIS EXISTS. The 2.4B fp32 DiT is ~20x the whole autoregressive half: a +// 45 s clip at the default 30 inference steps runs `DitForward` 660 times (30 +// steps x 2 CFG branches x 11 windows) at ~0.96 TFLOP each, and on the scalar +// host loops that is measured in hours. `d9441ef3` put the 8.6B language model +// on the device and reached 0.946x precisely because this half did not move. +// +// ─── WHAT IS PORTED, AND ONTO WHICH SHARED OP ──────────────────────────────── +// Every step below names the reference helper it replaces. NO new kernel is +// added: each one is a shared `vt::` op that already carries a CUDA provider. +// +// Linear -> vt::MatmulBT (+ vt::Add for the rank-1 bias) +// LayerNorm -> vt::LayerNorm +// ApplyPartialRotary -> vt::RopeFromCache over a [seq, rotary_dim] cache +// Attention (NON-causal) -> vt::AttentionCross (bias = nullptr) +// `x * silu(gate)` -> vt::SiluAndMul, over a STAGE-TIME half swap +// residual adds -> vt::Add +// PointwiseConv (1x1) -> vt::MatmulBT on the transposed activation +// +// ─── fp32 STAYS fp32 ───────────────────────────────────────────────────────── +// Spec §2.1: the acoustic half is float32 because upstream chose float32 for it, +// and this arm mirrors that. Every staged weight and every device activation +// below is `vt::DType::kF32`. Narrowing to bf16 would be a different change with +// its own evidence, not a free speedup taken in passing. +// +// ─── NUMERICS: CLOSE, NOT BIT-IDENTICAL, AND SAID SO ───────────────────────── +// This arm does NOT reproduce the host reference bit for bit, and does not claim +// to. Three named differences, none of them a shape or an order-of-operations +// defect: +// +// 1. The reference accumulates every reduction in `double` and stores float32 +// (see the dtype note at the top of minimax_music3_acoustic.cpp). The +// shared ops accumulate in float32 — which is what torch itself does, so on +// this axis the device arm is the CLOSER mirror of upstream, not the looser +// one. +// 2. A `nn.Linear` bias enters the reference INSIDE the accumulator +// (`double acc = bias`), and here it is a separate `vt::Add` afterwards. +// 3. `vt::AttentionCross`'s CUDA kernel uses the online-softmax recurrence +// where the reference uses an explicit three-pass max/exp/normalize. +// +// It is gated against the SAME upstream goldens at the SAME tolerance as the +// host forward (tests/vllm/models/test_minimax_music3_acoustic.cpp), and no +// tolerance was widened to admit it. +#pragma once + +#include +#include + +#include "vllm/model_executor/models/minimax_music3_acoustic.h" +#include "vt/device.h" +#include "vt/ops.h" + +namespace vllm { +namespace models { +namespace music3 { + +// One transformer block's weights, resident on the queue's device. +// +// `ff_in_weight` / `ff_in_bias` are the ONLY tensors whose CONTENT differs from +// the host struct, and the reason is mechanical rather than a choice. +// `transformer_minimax_music3.py:142-143` computes `ff_out(gate_states * +// silu(gate))` where `gate_states, gate = ff_in(x).chunk(2, -1)` — the FIRST +// half is the value and the SECOND is what SiLU runs on. `vt::SiluAndMul` +// computes `silu(x[:, :D]) * x[:, D:]`, i.e. the opposite assignment. Swapping +// the two ROW BLOCKS of the projection once, at stage time, makes the shared op +// compute exactly upstream's expression with no per-step permutation and no +// bespoke kernel. The multiply is commutative, so this is an identity, not an +// approximation. +struct Music3DitDeviceLayer { + vt::Tensor norm1_weight; // [inner] + vt::Tensor norm1_bias; // [inner] + vt::Tensor to_q; // [attn_inner, inner] + vt::Tensor to_k; // [attn_inner, inner] + vt::Tensor to_v; // [attn_inner, inner] + vt::Tensor to_out; // [inner, attn_inner] + vt::Tensor norm2_weight; // [inner] + vt::Tensor norm2_bias; // [inner] + vt::Tensor ff_in_weight; // [2 * ff, inner], HALVES SWAPPED (see above) + vt::Tensor ff_in_bias; // [2 * ff], HALVES SWAPPED + vt::Tensor ff_out_weight; // [inner, ff] + vt::Tensor ff_out_bias; // [inner] +}; + +// The DiT staged ONCE onto a device, with the storage that owns it. +// +// STAGED ONCE IS THE WHOLE POINT. 660 forwards per 45 s clip means a per-step +// upload of 9.7 GB would cost more than the compute it enables; the fixed +// +34.8 s `d9441ef3` measured for the language model is what a one-time upload +// looks like, and this arm keeps that shape. `storage` holds one +// `vt::Backend::Alloc` block per tensor, freed with this object. +struct Music3DitDeviceWeights { + vt::Tensor preprocess_conv_weight; // [concat, concat] (the 1x1 kernel axis is dropped) + vt::Tensor proj_in_weight; // [inner, concat] + std::vector layers; + vt::Tensor proj_out_weight; // [in_channels, inner] + vt::Tensor postprocess_conv_weight; // [in_channels, in_channels] + + // The timestep embedder stays on the HOST, deliberately and cheaply. + // `time_proj` + `time_embed` is ONE row through a [inner, fourier_dim] and an + // [inner, inner] projection — 4.7M MACs against the 2.4G MACs PER TOKEN the + // block stack runs, i.e. under a millionth of the forward at any real window + // length. Running it through the existing `DitTimestepEmbedding` keeps that + // piece BIT-IDENTICAL to the CPU arm for free; moving it would have needed an + // ungated SiLU op that `vt` does not carry. Only these five tensors are + // populated in this struct. + DitWeights host_time_embed; + + std::vector> storage; +}; + +// Upload the DiT to `queue`'s device, once. +// +// `release_host` EMPTIES each source vector as it is uploaded. The shipped DiT +// is 9.7 GB of fp32 and Jetson Thor's ~122 GB is UNIFIED — host and device draw +// on one pool — so holding both copies is a real 19.4 GB peak on the box this +// arm was written for, which `vm.overcommit_memory=1` and zero swap turn into a +// REBOOT rather than an OOM kill (.agents/environment.md). Pass true from a +// serving path, false from a gate that compares the two arms. +// +// Throws (naming the tensor) if a weight is mis-sized for `config`, or if the +// device has no provider for one of the ops the forward needs — a refusal at +// stage time rather than 36 layers into the first step. +Music3DitDeviceWeights StageMusic3DitWeights(vt::Queue& queue, + const MiniMaxMusic3TransformerConfig& config, + DitWeights& weights, bool release_host); + +// `DitForward`'s device twin: same inputs, same outputs, same layouts. +// +// `latents` [in_channels, length], CHANNEL-major (host) +// `condition` [length, condition_dim], FRAME-major (host) +// returns [in_channels, length], the flow-matching VELOCITY (host) +// +// The host<->device boundary is the ARGUMENTS ONLY: one upload of +// [length, concat_channels] on the way in and one download of +// [length, in_channels] on the way out, per call. Everything between — all 36 +// blocks, both 1x1 convolutions, both projections — stays in device memory. +std::vector DitForwardDevice(vt::Queue& queue, const std::vector& latents, + int64_t length, const std::vector& condition, + double timestep, + const MiniMaxMusic3TransformerConfig& config, + const Music3DitDeviceWeights& weights); + +} // namespace music3 +} // namespace models +} // namespace vllm diff --git a/include/vllm/model_executor/models/minimax_music3_speech.h b/include/vllm/model_executor/models/minimax_music3_speech.h index 29ae19ae2..0cad95313 100644 --- a/include/vllm/model_executor/models/minimax_music3_speech.h +++ b/include/vllm/model_executor/models/minimax_music3_speech.h @@ -69,6 +69,7 @@ #include "vllm/model_executor/models/minimax_music3_acoustic.h" #include "vllm/model_executor/models/minimax_music3_ar.h" +#include "vllm/model_executor/models/minimax_music3_device.h" #include "vllm/model_executor/models/minimax_music3_loader.h" #include "vllm/multimodal/speech_engine.h" @@ -198,10 +199,35 @@ struct Music3DenoiseOptions { // * the carry span is taken from the RESTORED latents (denoise.py:252-256); // * the scheduler is RESET per window (denoise.py:152-156), so step 0's sigma // is the first of a fresh schedule and not a continuation. +// +// THE DEVICE ARM (#672, spec §11.4) is the optional trailing parameter and +// NOTHING ELSE. Left default-constructed — which every caller written before it +// does — the loop runs `DitForward`, the host reference every Music3 gate was +// taken on, unchanged. Given a queue and the DiT staged onto that queue's +// device, the two `DitForward` calls per step become `DitForwardDevice` and +// nothing else in this function moves: the condition mix, the overlap blend, the +// CFG mix, the Euler step and the carry stay on the host, in the same order, +// computing the same numbers. +// +// The weights are staged by the CALLER, once, and handed in — not staged here. +// A 45 s clip runs this loop's inner body 660 times over 11 windows, so staging +// per window would upload 9.7 GB eleven times for one clip; the fixed cost has +// to sit outside every loop in this function, and putting the parameter here +// rather than a path inside is what makes that structural instead of careful. +struct Music3DenoiseDeviceArm { + vt::Queue* queue = nullptr; + const Music3DitDeviceWeights* dit = nullptr; + // Both or neither. One alone is a caller that thinks it asked for the device + // arm and did not, so it is REFUSED rather than silently ignored. + bool engaged() const { return queue != nullptr && dit != nullptr; } + bool half_set() const { return (queue != nullptr) != (dit != nullptr); } +}; + std::vector> Music3DenoiseChunks( const std::vector& frame_hiddens, int64_t num_frames, const MiniMaxMusic3Config& config, const Music3AcousticWeights& weights, - const Music3DenoiseOptions& options, const Music3NoiseSource& noise); + const Music3DenoiseOptions& options, const Music3NoiseSource& noise, + const Music3DenoiseDeviceArm& device_arm = {}); // The decode + stitch (decoders.py:75-92): each window's latents through the // vocoder, cropped by `VocoderCropSpan`, concatenated, and CLAMPED to [-1, 1] diff --git a/scripts/merged-gemm-consistency-allowlist.txt b/scripts/merged-gemm-consistency-allowlist.txt index 7d2dcd34f..ea5770573 100644 --- a/scripts/merged-gemm-consistency-allowlist.txt +++ b/scripts/merged-gemm-consistency-allowlist.txt @@ -48,3 +48,5 @@ minimax_h3_device # known-drift pending fold. The DTYPE half of this blocker is minimax_h3_video_vae_device # deliberately-not-merged: the video VAE's mid-block feed-forward already ships w1 as a MERGED [2*ff_inner, dim] operand and already issues ONE MatmulBT, so nothing here is an unmerged gate/up pair -- the epilogue is the only hand-call. It cannot take layers::UnquantizedMlpGateUpMethod for two reasons the seam does not carry: the VAE runs the whole block in f32 (DType::kF32 activations end to end, deliberately, because it is a decoder whose output is pixels rather than logits) while the method's DBufs are kBF16, and every VAE Linear carries a rank-1 bias the bias-free method has no slot for. Same weight-RESIDENCY blocker as minimax_h3_device besides: the DiT stages device tensors up front and binds plain views rather than OwnedTensor/ResidentWeight. Adopting the seam means giving it an f32 arm and a bias arm, which is a shared-layer change, not a model fold. minimax_h3_encoder_device # deliberately-not-merged: the H3-Encoder's gate/up are ggml BLOCK-QUANT weights (the 32B tower is kept in Q4_K/Q6_K so it fits a 122 GB box), and layers::UnquantizedMlpGateUpMethod is by name and by contract the UNQUANTIZED arm -- it takes an OwnedTensor of plain bf16 and has no block-quant path. Worse, the shipped Q4_K_M checkpoint keeps gate/up UNFUSED whenever the group mixes encodings, so there is not always a single merged [2I,H] operand to hand a merged-GEMM seam at all. Adopting the seam here means giving it a keep-quant arm, which is a shared-layer change, not a model fold. + +minimax_music3_device # deliberately-not-merged: there is NO unmerged gate/up pair here. Upstream's `ff.net.0.proj` is ONE nn.Linear producing [2*ff_inner, inner] which `.chunk(2,-1)` splits (transformer_minimax_music3.py:142-143), so the DiT already ships the merged operand and already issues ONE vt::MatmulBT -- the epilogue is the only hand-call, and a merged-GEMM seam has nothing left to merge. It cannot take layers::UnquantizedMlpGateUpMethod for three reasons the seam does not carry, each of which is a shared-layer change rather than a model fold. DTYPE: the whole acoustic half is f32 by upstream's own choice (spec §2.1) and the method's DBufs are kBF16, so adopting it would narrow a tensor this row is required to keep wide. BIAS: `ff.net.0.proj` and `ff.net.2` both carry a rank-1 bias and the bias-free method has no slot for one. RESIDENCY: the method takes an OwnedTensor staged on demand via ResidentWeight, whereas StageMusic3DitWeights uploads device tensors UP FRONT and binds plain views -- the same ownership blocker minimax_h3_device and minimax_h3_video_vae_device already record. There is a fourth, smaller mismatch worth naming so it is not rediscovered: upstream computes `value * silu(gate)` with the VALUE half first, the opposite assignment to vt::SiluAndMul's `silu(first) * second`, which this file resolves by exchanging the two row blocks of the projection ONCE at stage time. diff --git a/src/vllm/model_executor/models/minimax_music3_device.cpp b/src/vllm/model_executor/models/minimax_music3_device.cpp new file mode 100644 index 000000000..0070f382f --- /dev/null +++ b/src/vllm/model_executor/models/minimax_music3_device.cpp @@ -0,0 +1,397 @@ +// MiniMax-Music3 — the DEVICE-RESIDENT acoustic forward. See +// minimax_music3_device.h for what is ported onto which shared op, why fp32 +// stays fp32, and the three named reasons this arm is close to but not +// bit-identical to the host reference. +// +// ─── THE ONE STRUCTURAL DECISION IN THIS FILE ──────────────────────────────── +// +// The reference works in CHANNEL-MAJOR [channels, length] at the two ends and +// FRAME-MAJOR [length, channels] in the middle, and transposes between them +// (minimax_music3_acoustic.cpp:558-564, :625-631). This file transposes ONCE on +// the host at each boundary and stays FRAME-MAJOR everywhere in between, +// because both 1x1 convolutions are then plain GEMMs: +// +// conv1d(kernel=1): out[co][t] = Σ_ci W[co][ci] * in[ci][t] +// transposed: out^T[t][co] = Σ_ci in^T[t][ci] * W[co][ci] +// = MatmulBT(in^T, W) +// +// So `preprocess_conv` and `postprocess_conv` need no convolution op at all — +// which matters, because `vt` has no CUDA 1-D convolution provider (spec §11.4) +// and hand-rolling one outside the seam is what AGENTS.md forbids. The two +// host-side transposes it costs are [in_channels, length] tensors: 128 x length +// floats, against the 2.4G MACs per token the block stack runs. +#include "vllm/model_executor/models/minimax_music3_device.h" + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "vllm/model_executor/models/dense_device_glue.h" +#include "vt/backend.h" +#include "vt/dtype.h" +#include "vt/op_provider.h" // GetOp — the stage-time provider refusal +#include "vt/ops.h" + +namespace vllm { +namespace models { +namespace music3 { + +namespace { + +using dense_attn::DBuf; +using dense_attn::Dev; +using dense_attn::MakeTensor; +using dense_attn::Reshape; +using vt::DType; +using vt::Tensor; + +[[noreturn]] void Fail(const std::string& message) { throw std::runtime_error(message); } + +void RequireStageSize(const std::vector& values, int64_t expected, const char* what) { + if (static_cast(values.size()) != expected) { + Fail(std::string("MiniMax-Music3 DiT stage: ") + what + " is " + + std::to_string(values.size()) + " values, expected " + std::to_string(expected)); + } +} + +// A rank-N f32 view over a fresh device block, owned by `storage`. +// +// The host source is copied and then, when `release` is set, DROPPED — the +// vector is swapped with an empty one rather than merely cleared, because +// `clear()` keeps the capacity and the whole point is to return the 9.7 GB to +// the allocator before the next tensor asks for its device twin. +Tensor UploadF32(vt::Backend& backend, vt::Queue& queue, std::vector& src, + const std::vector& shape, bool release, + std::vector>* storage) { + int64_t numel = 1; + for (int64_t s : shape) numel *= s; + const size_t bytes = static_cast(numel) * sizeof(float); + void* p = backend.Alloc(bytes); + std::shared_ptr owner(p, [&backend](void* q) { backend.Free(q); }); + backend.Copy(queue, p, src.data(), bytes); + // The copy must have LANDED before the host buffer goes away. On a CPU queue + // this is a memcpy and the sync is free; on CUDA the source is pageable host + // memory, so releasing it under an unsynchronized async copy is exactly the + // use-after-free that reads as a plausible-looking wrong tensor. + if (release) { + backend.Synchronize(queue); + std::vector().swap(src); + } + storage->push_back(std::move(owner)); + return MakeTensor(p, DType::kF32, queue.device, shape); +} + +// `vt::MatmulBT` + an optional rank-1 row-broadcast bias — the device twin of +// the reference `Linear`. The bias is a SEPARATE add here where the reference +// seeds the accumulator with it; that is difference (2) in the header's +// numerics note, and it is a float32 rounding, not a reordering of the sum. +void LinearDev(vt::Queue& q, Tensor& out, const Tensor& in, const Tensor& weight, + const Tensor* bias) { + vt::MatmulBT(q, out, in, weight); + if (bias != nullptr && bias->data != nullptr) vt::Add(q, out, out, *bias); +} + +} // namespace + +Music3DitDeviceWeights StageMusic3DitWeights(vt::Queue& queue, + const MiniMaxMusic3TransformerConfig& config, + DitWeights& weights, bool release_host) { + const int64_t in_channels = config.in_channels; + const int64_t concat = config.concat_channels(); + const int64_t inner = config.inner_dim(); + const int64_t attn_inner = config.num_attention_heads * config.attention_head_dim; + const int64_t ff = config.ff_inner_dim; + const int64_t fourier = config.fourier_embedding_dim; + if (static_cast(weights.layers.size()) != config.num_layers) { + Fail("MiniMax-Music3 DiT stage: the weights carry " + + std::to_string(weights.layers.size()) + " blocks, the config declares " + + std::to_string(config.num_layers)); + } + + // REFUSE UP FRONT, before 9.7 GB moves. `vt::GetOp` throws naming the op and + // the device, so a backend without (say) a cross-attention provider is a + // one-line refusal at stage time rather than a failure 36 layers into the + // first of 660 forwards. Every op the forward below calls is listed. + for (vt::OpId op : {vt::OpId::kMatmulBT, vt::OpId::kAdd, vt::OpId::kLayerNorm, + vt::OpId::kSiluAndMul, vt::OpId::kRopeFromCache, + vt::OpId::kAttentionCross}) { + (void)vt::GetOp(op, queue.device.type); + } + + // The timestep embedder is validated FIRST, before anything is uploaded and + // therefore before `release_host` destroys anything. A size error found after + // the 36 blocks had been released would be a correct refusal that had already + // consumed the caller's weights. + const int64_t fourier_half = fourier / 2; + RequireStageSize(weights.time_proj_weight, fourier_half, "time_proj.weight"); + RequireStageSize(weights.time_embed_linear_1_weight, inner * fourier, + "time_embed.linear_1.weight"); + RequireStageSize(weights.time_embed_linear_1_bias, inner, "time_embed.linear_1.bias"); + RequireStageSize(weights.time_embed_linear_2_weight, inner * inner, + "time_embed.linear_2.weight"); + RequireStageSize(weights.time_embed_linear_2_bias, inner, "time_embed.linear_2.bias"); + + vt::Backend& backend = vt::GetBackend(queue.device.type); + Music3DitDeviceWeights staged; + staged.layers.resize(static_cast(config.num_layers)); + const bool rel = release_host; + auto up = [&](std::vector& src, const std::vector& shape, const char* what) { + int64_t numel = 1; + for (int64_t s : shape) numel *= s; + RequireStageSize(src, numel, what); + return UploadF32(backend, queue, src, shape, rel, &staged.storage); + }; + + // The 1x1 convolutions ship as [out, in, 1]; the kernel axis is dropped here + // because the GEMM form above consumes them as [out, in]. The element count + // is identical, so this is a reinterpretation of the same bytes, not a slice. + staged.preprocess_conv_weight = + up(weights.preprocess_conv_weight, {concat, concat}, "preprocess_conv.weight"); + staged.proj_in_weight = up(weights.proj_in_weight, {inner, concat}, "proj_in.weight"); + + for (int64_t l = 0; l < config.num_layers; ++l) { + DitLayerWeights& src = weights.layers[static_cast(l)]; + Music3DitDeviceLayer& dst = staged.layers[static_cast(l)]; + dst.norm1_weight = up(src.norm1_weight, {inner}, "norm1.weight"); + dst.norm1_bias = up(src.norm1_bias, {inner}, "norm1.bias"); + dst.to_q = up(src.to_q, {attn_inner, inner}, "attn.to_q.weight"); + dst.to_k = up(src.to_k, {attn_inner, inner}, "attn.to_k.weight"); + dst.to_v = up(src.to_v, {attn_inner, inner}, "attn.to_v.weight"); + dst.to_out = up(src.to_out, {inner, attn_inner}, "attn.to_out.0.weight"); + dst.norm2_weight = up(src.norm2_weight, {inner}, "norm2.weight"); + dst.norm2_bias = up(src.norm2_bias, {inner}, "norm2.bias"); + + // THE HALF SWAP (minimax_music3_device.h documents why). Upstream computes + // `value * silu(gate)` with value FIRST; `vt::SiluAndMul` computes + // `silu(first) * second`. Exchanging the two row blocks of the projection + // and the two halves of its bias — once, here — makes the shared op compute + // upstream's expression exactly. Doing it at stage time rather than per step + // is what keeps it free: 660 forwards x 36 layers would otherwise permute a + // [seq, 16384] tensor 23 760 times. + RequireStageSize(src.ff_in_weight, 2 * ff * inner, "ff.net.0.proj.weight"); + RequireStageSize(src.ff_in_bias, 2 * ff, "ff.net.0.proj.bias"); + { + std::vector swapped(static_cast(2 * ff * inner)); + const size_t half = static_cast(ff * inner); + std::memcpy(swapped.data(), src.ff_in_weight.data() + half, half * sizeof(float)); + std::memcpy(swapped.data() + half, src.ff_in_weight.data(), half * sizeof(float)); + if (rel) std::vector().swap(src.ff_in_weight); + dst.ff_in_weight = up(swapped, {2 * ff, inner}, "ff.net.0.proj.weight (swapped)"); + } + { + std::vector swapped(static_cast(2 * ff)); + const size_t half = static_cast(ff); + std::memcpy(swapped.data(), src.ff_in_bias.data() + half, half * sizeof(float)); + std::memcpy(swapped.data() + half, src.ff_in_bias.data(), half * sizeof(float)); + if (rel) std::vector().swap(src.ff_in_bias); + dst.ff_in_bias = up(swapped, {2 * ff}, "ff.net.0.proj.bias (swapped)"); + } + dst.ff_out_weight = up(src.ff_out_weight, {inner, ff}, "ff.net.2.weight"); + dst.ff_out_bias = up(src.ff_out_bias, {inner}, "ff.net.2.bias"); + } + + staged.proj_out_weight = up(weights.proj_out_weight, {in_channels, inner}, "proj_out.weight"); + staged.postprocess_conv_weight = + up(weights.postprocess_conv_weight, {in_channels, in_channels}, "postprocess_conv.weight"); + + // The timestep embedder stays on the host — see the header. These are COPIES, + // so `release_host` does not take them: they are 18 MB at the shipped + // dimensions and they are what keeps `temb` bit-identical to the CPU arm. + // Their sizes were checked at the top of this function. + staged.host_time_embed.time_proj_weight = weights.time_proj_weight; + staged.host_time_embed.time_embed_linear_1_weight = weights.time_embed_linear_1_weight; + staged.host_time_embed.time_embed_linear_1_bias = weights.time_embed_linear_1_bias; + staged.host_time_embed.time_embed_linear_2_weight = weights.time_embed_linear_2_weight; + staged.host_time_embed.time_embed_linear_2_bias = weights.time_embed_linear_2_bias; + + backend.Synchronize(queue); + return staged; +} + +std::vector DitForwardDevice(vt::Queue& queue, const std::vector& latents, + int64_t length, const std::vector& condition, + double timestep, + const MiniMaxMusic3TransformerConfig& config, + const Music3DitDeviceWeights& weights) { + if (length <= 0) { + Fail("MiniMax-Music3 DiT: a window of " + std::to_string(length) + + " latent frames has nothing to denoise"); + } + const int64_t in_channels = config.in_channels; + const int64_t condition_dim = config.condition_dim; + const int64_t concat = config.concat_channels(); + const int64_t inner = config.inner_dim(); + const int64_t heads = config.num_attention_heads; + const int64_t head_dim = config.attention_head_dim; + const int64_t attn_inner = heads * head_dim; + const int64_t ff = config.ff_inner_dim; + const int64_t seq = length + 1; + if (static_cast(latents.size()) != in_channels * length) { + Fail("MiniMax-Music3 DiT: latents [in_channels, length] is " + + std::to_string(latents.size()) + " values, expected " + + std::to_string(in_channels * length)); + } + if (static_cast(condition.size()) != length * condition_dim) { + Fail("MiniMax-Music3 DiT: condition [length, condition_dim] is " + + std::to_string(condition.size()) + " values, expected " + + std::to_string(length * condition_dim)); + } + if (static_cast(weights.layers.size()) != config.num_layers) { + Fail("MiniMax-Music3 DiT: the staged weights carry " + + std::to_string(weights.layers.size()) + " blocks, the config declares " + + std::to_string(config.num_layers)); + } + + vt::Backend& backend = vt::GetBackend(queue.device.type); + Dev d{backend, queue}; + + // `cat((hidden_states, zeros_like(hidden_states), encoder_hidden_states.T))` + // (:218-219), built directly in the TRANSPOSED [length, concat] orientation. + // The middle block is a genuine ZERO PAD and not a second copy of the latents. + std::vector stacked_t(static_cast(length * concat), 0.0f); + for (int64_t t = 0; t < length; ++t) { + float* row = stacked_t.data() + t * concat; + for (int64_t c = 0; c < in_channels; ++c) { + row[c] = latents[static_cast(c * length + t)]; + } + for (int64_t c = 0; c < condition_dim; ++c) { + row[2 * in_channels + c] = condition[static_cast(t * condition_dim + c)]; + } + } + + DBuf stacked(d, DType::kF32, {length, concat}, stacked_t.data()); + // RESIDUAL 1x1 convolution (:220), as the transposed GEMM this file's header + // note derives. `pre` then holds conv(x) and the add makes it conv(x) + x. + DBuf pre(d, DType::kF32, {length, concat}); + vt::MatmulBT(queue, pre.t(), stacked.t(), weights.preprocess_conv_weight); + vt::Add(queue, pre.t(), pre.t(), stacked.t()); + + // The timestep embedding is PREPENDED as one extra token (:227) that the + // rotary sees and `proj_out` then drops (:236). `hidden` is allocated at the + // full [seq, inner] and `proj_in` writes STRAIGHT into rows 1..seq — a view, + // not a copy, so the projection lands where the block stack wants it. + DBuf hidden(d, DType::kF32, {seq, inner}); + Tensor hidden_tail = MakeTensor(static_cast(hidden.t().data) + inner, DType::kF32, + queue.device, {length, inner}); + vt::MatmulBT(queue, hidden_tail, pre.t(), weights.proj_in_weight); + + // Row 0: the timestep embedding, computed on the host through the reference's + // own helpers so it is bit-identical to the CPU arm (header rationale). + const std::vector temb = DitTimestepEmbedding( + FourierTimeEmbedding(timestep, weights.host_time_embed.time_proj_weight, + config.fourier_embedding_dim), + config, weights.host_time_embed); + if (static_cast(temb.size()) != inner) { + Fail("MiniMax-Music3 DiT: the timestep embedding is " + std::to_string(temb.size()) + + " values, expected inner_dim = " + std::to_string(inner)); + } + backend.Copy(queue, hidden.t().data, temb.data(), static_cast(inner) * sizeof(float)); + + // ── the rotary cache, in the layout vt::RopeFromCache reads ──────────────── + // That op indexes `cache[position * rotary_dim + pair]` for the cosine and + // `+ half + pair` for the sine, then computes x' = x*c - y*s, y' = x*s + y*c + // over the LEADING rotary_dim of each head — which is `ApplyPartialRotary` + // exactly (minimax_music3_acoustic.cpp:500-514), including the untouched tail. + // `BuildDitRotaryTables` returns cos/sin already duplicated across the two + // halves of the rotary window; the cache wants ONE half of each, so the first + // `half` columns of each table are what is packed here. + const int64_t rotary_dim = config.rotary_dim; + const int64_t half = rotary_dim / 2; + const DitRotaryTables tables = BuildDitRotaryTables(seq, rotary_dim, kDitRotaryTheta); + std::vector cache(static_cast(seq * rotary_dim)); + for (int64_t s = 0; s < seq; ++s) { + for (int64_t j = 0; j < half; ++j) { + cache[static_cast(s * rotary_dim + j)] = + tables.cos[static_cast(s * rotary_dim + j)]; + cache[static_cast(s * rotary_dim + half + j)] = + tables.sin[static_cast(s * rotary_dim + j)]; + } + } + std::vector positions_host(static_cast(seq)); + for (int64_t s = 0; s < seq; ++s) positions_host[static_cast(s)] = static_cast(s); + DBuf rope_cache(d, DType::kF32, {seq, rotary_dim}, cache.data()); + DBuf positions(d, DType::kI32, {seq}, positions_host.data()); + + vt::RopeArgs rope_args; + rope_args.rotary_dim = static_cast(rotary_dim); + rope_args.is_neox_style = true; // rotate_half over the rotary window + vt::LayerNormArgs norm_args; + norm_args.eps = static_cast(kDitLayerNormEps); + vt::AttentionCrossArgs attn_args; + attn_args.scale = static_cast(1.0 / std::sqrt(static_cast(head_dim))); + + // The reference's `Attention` helper takes NO mask (:97-103 dispatches with + // none), so every token attends to every token including the prepended + // timestep one. `vt::AttentionCross` with a null bias is that op; `vt:: + // Attention` is the CAUSAL one and would silently mask the future here. + for (int64_t l = 0; l < config.num_layers; ++l) { + const Music3DitDeviceLayer& layer = weights.layers[static_cast(l)]; + + DBuf normed(d, DType::kF32, {seq, inner}); + vt::LayerNorm(queue, normed.t(), hidden.t(), &layer.norm1_weight, &layer.norm1_bias, + norm_args); + DBuf qb(d, DType::kF32, {seq, attn_inner}); + DBuf kb(d, DType::kF32, {seq, attn_inner}); + DBuf vb(d, DType::kF32, {seq, attn_inner}); + LinearDev(queue, qb.t(), normed.t(), layer.to_q, nullptr); + LinearDev(queue, kb.t(), normed.t(), layer.to_k, nullptr); + LinearDev(queue, vb.t(), normed.t(), layer.to_v, nullptr); + + Tensor q3 = Reshape(qb.t(), {seq, heads, head_dim}); + Tensor k3 = Reshape(kb.t(), {seq, heads, head_dim}); + Tensor v3 = Reshape(vb.t(), {seq, heads, head_dim}); + vt::RopeFromCache(queue, q3, &k3, positions.t(), rope_cache.t(), rope_args); + + DBuf attended(d, DType::kF32, {seq, heads, head_dim}); + vt::AttentionCross(queue, attended.t(), q3, k3, v3, nullptr, attn_args); + Tensor attended2 = Reshape(attended.t(), {seq, attn_inner}); + DBuf projected(d, DType::kF32, {seq, inner}); + LinearDev(queue, projected.t(), attended2, layer.to_out, nullptr); + vt::Add(queue, hidden.t(), hidden.t(), projected.t()); + + DBuf normed2(d, DType::kF32, {seq, inner}); + vt::LayerNorm(queue, normed2.t(), hidden.t(), &layer.norm2_weight, &layer.norm2_bias, + norm_args); + DBuf gated(d, DType::kF32, {seq, 2 * ff}); + LinearDev(queue, gated.t(), normed2.t(), layer.ff_in_weight, &layer.ff_in_bias); + // `silu(first) * second` on the SWAPPED projection == upstream's + // `value * silu(gate)`. See StageMusic3DitWeights. + DBuf activated(d, DType::kF32, {seq, ff}); + vt::SiluAndMul(queue, activated.t(), gated.t()); + DBuf ff_out(d, DType::kF32, {seq, inner}); + LinearDev(queue, ff_out.t(), activated.t(), layer.ff_out_weight, &layer.ff_out_bias); + vt::Add(queue, hidden.t(), hidden.t(), ff_out.t()); + } + + // Drop the timestep token (the same [length, inner] view `proj_in` wrote), + // project, then the RESIDUAL 1x1 convolution (:238) as a GEMM again. + DBuf out_rows(d, DType::kF32, {length, in_channels}); + vt::MatmulBT(queue, out_rows.t(), hidden_tail, weights.proj_out_weight); + DBuf post(d, DType::kF32, {length, in_channels}); + vt::MatmulBT(queue, post.t(), out_rows.t(), weights.postprocess_conv_weight); + vt::Add(queue, out_rows.t(), out_rows.t(), post.t()); + + std::vector rows(static_cast(length * in_channels)); + backend.Copy(queue, rows.data(), out_rows.t().data, rows.size() * sizeof(float)); + backend.Synchronize(queue); + + // Back to the CHANNEL-MAJOR [in_channels, length] the caller and the vocoder + // both expect. + std::vector out(static_cast(in_channels * length)); + for (int64_t c = 0; c < in_channels; ++c) { + for (int64_t t = 0; t < length; ++t) { + out[static_cast(c * length + t)] = rows[static_cast(t * in_channels + c)]; + } + } + return out; +} + +} // namespace music3 +} // namespace models +} // namespace vllm diff --git a/src/vllm/model_executor/models/minimax_music3_speech.cpp b/src/vllm/model_executor/models/minimax_music3_speech.cpp index 0863a6353..8cc21cd84 100644 --- a/src/vllm/model_executor/models/minimax_music3_speech.cpp +++ b/src/vllm/model_executor/models/minimax_music3_speech.cpp @@ -187,12 +187,21 @@ std::vector> Music3DenoiseChunks(const std::vector& fr const MiniMaxMusic3Config& config, const Music3AcousticWeights& weights, const Music3DenoiseOptions& options, - const Music3NoiseSource& noise) { + const Music3NoiseSource& noise, + const Music3DenoiseDeviceArm& device_arm) { if (!noise) Fail("MiniMax-Music3: the denoise loop needs a noise source"); if (options.num_inference_steps <= 0) { Fail("MiniMax-Music3: `num_inference_steps` must be positive, got " + std::to_string(options.num_inference_steps)); } + // Half a device arm is a caller that believes it asked for the GPU and got the + // host loops. Refused by name rather than ignored, because the failure it + // otherwise produces is a correct song delivered thirty hours late. + if (device_arm.half_set()) { + Fail("MiniMax-Music3: the denoise device arm needs BOTH a queue and staged DiT weights; " + "got only the " + std::string(device_arm.queue != nullptr ? "queue" : "weights")); + } + const bool on_device = device_arm.engaged(); ConditionMixConfig mix; mix.condition_hidden_dim = config.condition_encoder.condition_hidden_dim; @@ -275,10 +284,19 @@ std::vector> Music3DenoiseChunks(const std::vector& fr BlendOverlap(latents, channels, length, noise_prompt, previous_latent, previous_length, overlap, time_value); } + // The ONLY line the device arm changes. Both CFG branches take the same + // arm — running one on each would make the guidance mix a comparison + // between two different numerics rather than between two conditionings. const std::vector conditional = - DitForward(latents, length, condition, time_value, config.transformer, weights.dit); + on_device ? DitForwardDevice(*device_arm.queue, latents, length, condition, time_value, + config.transformer, *device_arm.dit) + : DitForward(latents, length, condition, time_value, config.transformer, + weights.dit); const std::vector unconditional = - DitForward(latents, length, zero_condition, time_value, config.transformer, weights.dit); + on_device ? DitForwardDevice(*device_arm.queue, latents, length, zero_condition, + time_value, config.transformer, *device_arm.dit) + : DitForward(latents, length, zero_condition, time_value, config.transformer, + weights.dit); const std::vector velocity = ClassifierFreeGuidanceMix(conditional, unconditional, options.guidance_scale); latents = FlowMatchStep(latents, velocity, step, schedule); @@ -534,13 +552,40 @@ class Music3SpeechEngine final : public multimodal::SpeechEngine { (void)calls; // ── the ACOUSTIC half (before_denoise.py / denoise.py / decoders.py) ───── - const Music3AcousticWeights acoustic = Music3LoadAcousticWeights(paths_, config_); + // + // NON-const, because the device arm below STAGES OUT OF IT. The DiT is + // 9.7 GB of fp32 and Jetson Thor's ~122 GB is UNIFIED — host and device draw + // on one pool — so uploading while the host copy is still held is a real + // 19.4 GB peak on the only box this arm runs on, and that box reboots + // instead of OOM-killing (.agents/environment.md). `StageMusic3DitWeights` + // drops each host tensor as it lands, so the peak is one tensor over the + // 9.7 GB, not twice it. + Music3AcousticWeights acoustic = Music3LoadAcousticWeights(paths_, config_); Music3DenoiseOptions options; options.num_inference_steps = request.num_inference_steps; options.guidance_scale = request.guidance_scale; + + // Staged ONCE per request, outside every loop in `Music3DenoiseChunks`. A + // 45 s clip runs the DiT 660 times over 11 windows; a per-window upload + // would move 9.7 GB eleven times and a per-step one 660 times, either of + // which costs more than the compute it enables. + // + // Not staged in the CONSTRUCTOR, unlike the queue: the acoustic weights are + // deliberately loaded per request and released with the request, so that the + // 18.5 GB autoregressive half and the 10 GB acoustic half are never + // co-resident (upstream drives the same split by hand, encoders.py:302-309). + // Staging follows the weights, not the engine. + Music3DitDeviceWeights staged_dit; + Music3DenoiseDeviceArm arm; + if (queue_.device.type != vt::DeviceType::kCPU) { + staged_dit = StageMusic3DitWeights(queue_, config_.transformer, acoustic.dit, + /*release_host=*/true); + arm.queue = &queue_; + arm.dit = &staged_dit; + } const std::vector> chunks = Music3DenoiseChunks(frame_hiddens, frames, config_, acoustic, options, - Music3SeededNoise(request.seed)); + Music3SeededNoise(request.seed), arm); int64_t samples = 0; multimodal::SpeechResult out; diff --git a/tests/parity/test_minimax_music3_acoustic_real.cpp b/tests/parity/test_minimax_music3_acoustic_real.cpp index 2b5a55875..1253daa7f 100644 --- a/tests/parity/test_minimax_music3_acoustic_real.cpp +++ b/tests/parity/test_minimax_music3_acoustic_real.cpp @@ -86,6 +86,8 @@ // the checkpoint is present. #include +#include +#include #include #include #include @@ -99,7 +101,12 @@ #include "npy.h" #include "vllm/model_executor/model_loader/safetensors_reader.h" #include "vllm/model_executor/models/minimax_music3_acoustic.h" +#include "vllm/model_executor/models/minimax_music3_device.h" #include "vllm/model_executor/models/minimax_music3_loader.h" +#include "vllm/model_executor/models/minimax_music3_speech.h" // kMusic3SpeechFamily +#include "vllm/multimodal/speech_engine.h" // SpeechEngineDeviceType +#include "vt/backend.h" +#include "vt/device.h" namespace fs = std::filesystem; namespace m3 = vllm::models::music3; @@ -353,16 +360,61 @@ m3::DitWeights LoadDit(const vllm::MiniMaxMusic3TransformerConfig& config) { return m3::DitWeightsFromTensors(config, tensors); } +// WHERE this gate runs the 2.4B DiT. Default 0 = CPU, so an unset environment +// reproduces every number this file has ever printed. `VLLM_CPP_MUSIC3_DEVICE=1` +// runs the SAME comparison against the SAME goldens at the SAME bounds through +// `DitForwardDevice` (#672, spec §11.4) — no tolerance is widened for it, which +// is the claim that matters. +// +// Resolved through the SHARED `multimodal::SpeechEngineDeviceType` the engine +// itself calls, not a private copy: a gate that resolved the device its own way +// could pass while the engine bound a different one. +struct DitArm { + vt::Queue queue{}; + bool on_device = false; + std::string banner; +}; + +DitArm ResolveDitArm() { + DitArm arm; + const char* env = std::getenv("VLLM_CPP_MUSIC3_DEVICE"); + const int32_t sel = (env != nullptr && env[0] == '1') ? 1 : 0; + const vt::DeviceType type = + vllm::multimodal::SpeechEngineDeviceType(sel, m3::kMusic3SpeechFamily); + arm.on_device = type != vt::DeviceType::kCPU; + arm.queue = arm.on_device ? vt::GetBackend(type).CreateQueue() + : vt::Queue{vt::Device{vt::DeviceType::kCPU, 0}, nullptr}; + // ONE std::string. Built as a MESSAGE chain, both fields collapse inside + // doctest and a CPU run prints the device arm's banner — the instrument defect + // #672 already hit once, where the CPU numbers would have been recorded as the + // device arm's had the line not been read. + arm.banner = std::string("music3 acoustic real: the 2.4B DiT ran on '") + + vt::DeviceTypeName(arm.queue.device.type) + "' (VLLM_CPP_MUSIC3_DEVICE=" + + (env == nullptr ? std::string("unset") : std::string(env)) + ")"; + return arm; +} + // One guided velocity: the conditional and the zero-conditioned forward, mixed. +// Both branches take the SAME arm — running one on each would make the guidance +// mix a comparison between two numerics rather than between two conditionings. std::vector GuidedVelocity(const std::vector& latents, const std::vector& condition, double timestep, const vllm::MiniMaxMusic3TransformerConfig& config, - const m3::DitWeights& weights) { + const m3::DitWeights& weights, DitArm* arm, + const m3::Music3DitDeviceWeights* staged) { + const std::vector zeros(condition.size(), 0.0f); + if (arm != nullptr && arm->on_device) { + REQUIRE(staged != nullptr); + const std::vector conditional = m3::DitForwardDevice( + arm->queue, latents, kLatentLength, condition, timestep, config, *staged); + const std::vector unconditional = m3::DitForwardDevice( + arm->queue, latents, kLatentLength, zeros, timestep, config, *staged); + return m3::ClassifierFreeGuidanceMix(conditional, unconditional, m3::kDitGuidanceScale); + } const std::vector conditional = m3::DitForward(latents, kLatentLength, condition, timestep, config, weights); const std::vector unconditional = - m3::DitForward(latents, kLatentLength, std::vector(condition.size(), 0.0f), - timestep, config, weights); + m3::DitForward(latents, kLatentLength, zeros, timestep, config, weights); return m3::ClassifierFreeGuidanceMix(conditional, unconditional, m3::kDitGuidanceScale); } @@ -492,17 +544,59 @@ TEST_CASE("music3 acoustic real: the DiT reproduces the capture's guided velocit std::vector shape; const std::vector condition = LoadF32Npy("condition_chunk0.npy", &shape); - const m3::DitWeights weights = LoadDit(config.transformer); + m3::DitWeights weights = LoadDit(config.transformer); + + DitArm arm = ResolveDitArm(); + MESSAGE(arm.banner); + // `release_host` FALSE: this is a gate, and both arms must remain runnable in + // one process. The SERVING path is what passes true. + // + // TIMED, because this staging is the thing the speed claim is ABOUT. If the + // weights were re-uploaded per forward, the repeat sweep below would show it + // as slope rather than as intercept. + m3::Music3DitDeviceWeights staged; + const auto stage_t0 = std::chrono::steady_clock::now(); + if (arm.on_device) { + staged = m3::StageMusic3DitWeights(arm.queue, config.transformer, weights, + /*release_host=*/false); + CHECK(static_cast(staged.layers.size()) == config.transformer.num_layers); + } + const double stage_s = + std::chrono::duration(std::chrono::steady_clock::now() - stage_t0).count(); + // Same one-string rule as DIT_TIMING below: the first revision printed + // `dit staging: 9.3e-08 s (1)` because the `const char*` arm tag went to + // doctest's bool overload. + const std::string staging_line = std::string("dit staging: ") + std::to_string(stage_s) + + " s (" + + (arm.on_device ? "device upload" : "host, no-op") + ")"; + MESSAGE(staging_line); + + // `VLLM_CPP_MUSIC3_DIT_REPEAT=R` runs the guided velocity R times per timestep + // instead of once, so a run at R and a run at R' give TWO POINTS on the same + // binary and the same weights. The slope is the per-forward cost and the + // intercept is everything paid once — which is the only way to state "the + // weights are staged once" as a MEASUREMENT rather than as a claim about the + // code. Default 1, so an unset environment runs exactly what it always did. + int64_t repeats = 1; + if (const char* r = std::getenv("VLLM_CPP_MUSIC3_DIT_REPEAT")) { + repeats = std::max(1, std::atoll(r)); + } int64_t total_outside = 0; + int64_t forwards = 0; + const auto t0 = std::chrono::steady_clock::now(); for (int64_t index : {static_cast(0), kDenoiseSteps - 1}) { const std::string tag = index == 0 ? "first" : "last"; const double timestep = index == 0 ? kFirstTimestep : kLastTimestep; CAPTURE(tag); const std::vector latents = LoadF32Npy("denoise_" + tag + "_sample_in.npy", &shape); const std::vector want = LoadF32Npy("denoise_" + tag + "_velocity.npy", &shape); - const std::vector got = - GuidedVelocity(latents, condition, timestep, config.transformer, weights); + std::vector got; + for (int64_t r = 0; r < repeats; ++r) { + got = GuidedVelocity(latents, condition, timestep, config.transformer, weights, &arm, + &staged); + forwards += 2; // one guided velocity is the conditional AND the unconditional forward + } const Report report = Compare(got, want, kDitRelTol, kDitAbsFloor); ReportInto("dit guided velocity " + tag, report); CHECK(report.compared == kLatentChannels * kLatentLength); @@ -510,6 +604,21 @@ TEST_CASE("music3 acoustic real: the DiT reproduces the capture's guided velocit CHECK(report.mean_abs < kDitMeanAbsTol); total_outside += report.outside; } + const double loop_s = std::chrono::duration(std::chrono::steady_clock::now() - t0).count(); + // ONE line, ONE std::string — and this is not a style preference, it is the + // defect this row hit twice. A `const char*` handed to doctest's MESSAGE chain + // converts to BOOL and prints `1`: the first revision of this line reported + // `arm=1` on the CPU run, next to numbers that were themselves correct. That + // is #672's own recorded instrument defect (§11.5) reappearing in a new line, + // and the fix is the one that worked there: assemble the string, then print it. + std::string timing = "DIT_TIMING arm="; + timing += vt::DeviceTypeName(arm.queue.device.type); + timing += " repeats=" + std::to_string(repeats); + timing += " forwards=" + std::to_string(forwards); + timing += " loop_s=" + std::to_string(loop_s); + timing += " stage_s=" + std::to_string(stage_s); + timing += " per_forward_s=" + std::to_string(loop_s / static_cast(forwards)); + MESSAGE(timing); CHECK(total_outside == 0); } @@ -520,13 +629,26 @@ TEST_CASE("music3 acoustic real: the DiT's two guidance branches are different t std::vector shape; const std::vector condition = LoadF32Npy("condition_chunk0.npy", &shape); const std::vector latents = LoadF32Npy("denoise_first_sample_in.npy", &shape); - const m3::DitWeights weights = LoadDit(config.transformer); - - const std::vector conditional = m3::DitForward( - latents, kLatentLength, condition, kFirstTimestep, config.transformer, weights); + m3::DitWeights weights = LoadDit(config.transformer); + + DitArm arm = ResolveDitArm(); + MESSAGE(arm.banner); + m3::Music3DitDeviceWeights staged; + if (arm.on_device) { + staged = m3::StageMusic3DitWeights(arm.queue, config.transformer, weights, + /*release_host=*/false); + } + const std::vector zeros(condition.size(), 0.0f); + const std::vector conditional = + arm.on_device ? m3::DitForwardDevice(arm.queue, latents, kLatentLength, condition, + kFirstTimestep, config.transformer, staged) + : m3::DitForward(latents, kLatentLength, condition, kFirstTimestep, + config.transformer, weights); const std::vector unconditional = - m3::DitForward(latents, kLatentLength, std::vector(condition.size(), 0.0f), - kFirstTimestep, config.transformer, weights); + arm.on_device ? m3::DitForwardDevice(arm.queue, latents, kLatentLength, zeros, + kFirstTimestep, config.transformer, staged) + : m3::DitForward(latents, kLatentLength, zeros, kFirstTimestep, + config.transformer, weights); const int64_t identical = CountIdentical(conditional, unconditional); // A DiT that dropped its conditioning would still pass the velocity gate for // any guidance scale if the two branches were equal, because the mix would diff --git a/tests/vllm/models/test_minimax_music3_acoustic.cpp b/tests/vllm/models/test_minimax_music3_acoustic.cpp index 51bc8a54f..dc5d4b1ca 100644 --- a/tests/vllm/models/test_minimax_music3_acoustic.cpp +++ b/tests/vllm/models/test_minimax_music3_acoustic.cpp @@ -29,7 +29,9 @@ // (AGENTS.md; spec §5). #include +#include #include +#include #include #include #include @@ -37,6 +39,9 @@ #include "minimax_music3_acoustic_goldens.inc" #include "vllm/model_executor/models/minimax_music3_acoustic.h" +#include "vllm/model_executor/models/minimax_music3_device.h" +#include "vt/backend.h" +#include "vt/device.h" namespace { @@ -715,6 +720,239 @@ TEST_CASE("music3 acoustic: the DiT refuses every wrong-shaped input by name") { std::runtime_error); } +// --------------------------------------------------------------------------- +// The DEVICE-RESIDENT DiT (#672, spec §11.4) +// +// THE TOLERANCE, AND THE CONTROL THAT JUSTIFIES IT. Nothing below is a new +// bound. `DitForwardDevice` is checked against the SAME upstream float32 +// goldens, through the SAME `ExpectClose`, at the SAME kRelTol/kAbsFloor as +// `DitForward` — because the question that matters is not "do the two arms +// agree with each other" (a shared-helper comparison proves consistency, not +// correctness) but "is the device arm as close to UPSTREAM as the host arm is". +// +// Each case therefore reports BOTH distances to the golden, host and device, on +// the identical input. The host arm's distance is the measured control: it was +// accepted with these goldens when the bound was set, so a device arm whose +// distance is at or below it is inside a spread that already exists rather than +// inside one this row widened. No tolerance is relaxed here, and the two +// mutation cases below prove the bound still discriminates. +// --------------------------------------------------------------------------- + +namespace { + +// Both arms, same inputs, both against upstream. Returns nothing; every number +// is asserted or printed, and the CASE count is what the suite reports. +void CheckDeviceDit(vt::Queue& q, const char* arm) { + const vllm::MiniMaxMusic3TransformerConfig config = DitConfig(); + const size_t latent_count = + static_cast(config.in_channels * vllm_test::kMusic3DitLength); + const size_t condition_count = + static_cast(vllm_test::kMusic3DitLength * config.condition_dim); + const std::vector latents = ToVector(vllm_test::kMusic3DitLatents, latent_count); + const std::vector condition = ToVector(vllm_test::kMusic3DitCondition, condition_count); + const std::vector zeros(condition_count, 0.0f); + + // `release_host` FALSE here on purpose: this gate needs the host arm too, and + // the serving path is the caller that passes true. + m3::DitWeights host = DitWeights(); + const m3::Music3DitDeviceWeights staged = + m3::StageMusic3DitWeights(q, config, host, /*release_host=*/false); + REQUIRE(staged.layers.size() == static_cast(config.num_layers)); + + const std::vector dev_cond = m3::DitForwardDevice( + q, latents, vllm_test::kMusic3DitLength, condition, vllm_test::kMusic3DitTimestep, config, + staged); + const std::vector host_cond = + m3::DitForward(latents, vllm_test::kMusic3DitLength, condition, + vllm_test::kMusic3DitTimestep, config, host); + const double dev_worst = + ExpectClose(dev_cond, vllm_test::kMusic3DitOut, latent_count, + (std::string(arm) + " dit conditional (device)").c_str()); + const double host_worst = + ExpectClose(host_cond, vllm_test::kMusic3DitOut, latent_count, + (std::string(arm) + " dit conditional (host control)").c_str()); + MESSAGE(std::string(arm) << " dit conditional: " << latent_count + << " values; worst |device-upstream| = " << dev_worst + << ", worst |host-upstream| = " << host_worst + << " (bound " << kRelTol << " rel / " << kAbsFloor << " abs)"); + + const std::vector dev_uncond = + m3::DitForwardDevice(q, latents, vllm_test::kMusic3DitLength, zeros, + vllm_test::kMusic3DitTimestep, config, staged); + const double dev_worst_u = + ExpectClose(dev_uncond, vllm_test::kMusic3DitOutUncond, latent_count, + (std::string(arm) + " dit unconditional (device)").c_str()); + MESSAGE(std::string(arm) << " dit unconditional: " << latent_count + << " values; worst |device-upstream| = " << dev_worst_u); + + // The two branches must be DIFFERENT tensors on the device arm too: a forward + // that dropped its condition would match the conditional golden and this one + // identically, and both ExpectClose calls above would still be green. + size_t differing = 0; + for (size_t i = 0; i < latent_count; ++i) { + if (dev_cond[i] != dev_uncond[i]) ++differing; + } + MESSAGE(std::string(arm) << " dit branches: " << differing << " of " << latent_count + << " values differ between conditional and unconditional"); + CHECK(differing == latent_count); +} + +} // namespace + +TEST_CASE("music3 acoustic: the DEVICE-resident DiT matches upstream (CPU backend)") { + vt::Queue q{vt::Device{}, nullptr}; + CheckDeviceDit(q, "cpu-backend"); +} + +TEST_CASE("music3 acoustic: the DEVICE-resident DiT matches upstream on CUDA") { + vt::Backend* cuda = nullptr; + try { + cuda = &vt::GetBackend(vt::DeviceType::kCUDA); + } catch (...) { + MESSAGE("SKIP: no CUDA backend registered (this is a CPU-only build)"); + return; + } + vt::Queue q = cuda->CreateQueue(); + CheckDeviceDit(q, "cuda"); +} + +TEST_CASE("music3 acoustic: the ff_in HALF SWAP is load-bearing, and the gate sees it") { + // The device arm computes `value * silu(gate)` by handing vt::SiluAndMul — which + // computes `silu(first) * second` — a projection whose two ROW BLOCKS were + // exchanged at stage time. That exchange is an identity ONLY if it is applied + // exactly once. Pre-swapping the host weights makes the stage-time swap undo + // the test's, so the forward computes `silu(value) * gate` instead: the wrong + // network, same shapes, same finiteness. + // + // This is the mutation that proves the bound above discriminates. If the + // forward were routing `silu`/`mul` the other way round the RIGHT case would + // fail and this one would pass, so the pair pins the direction rather than + // just the magnitude. + const vllm::MiniMaxMusic3TransformerConfig config = DitConfig(); + const size_t latent_count = + static_cast(config.in_channels * vllm_test::kMusic3DitLength); + const std::vector latents = ToVector(vllm_test::kMusic3DitLatents, latent_count); + const std::vector condition = ToVector( + vllm_test::kMusic3DitCondition, + static_cast(vllm_test::kMusic3DitLength * config.condition_dim)); + + m3::DitWeights mutated = DitWeights(); + const size_t ff = static_cast(config.ff_inner_dim); + const size_t inner = static_cast(config.inner_dim()); + for (m3::DitLayerWeights& layer : mutated.layers) { + std::vector w(layer.ff_in_weight.size()); + std::copy(layer.ff_in_weight.begin() + static_cast(ff * inner), + layer.ff_in_weight.end(), w.begin()); + std::copy(layer.ff_in_weight.begin(), + layer.ff_in_weight.begin() + static_cast(ff * inner), + w.begin() + static_cast(ff * inner)); + layer.ff_in_weight = w; + std::vector b(layer.ff_in_bias.size()); + std::copy(layer.ff_in_bias.begin() + static_cast(ff), layer.ff_in_bias.end(), + b.begin()); + std::copy(layer.ff_in_bias.begin(), layer.ff_in_bias.begin() + static_cast(ff), + b.begin() + static_cast(ff)); + layer.ff_in_bias = b; + } + + vt::Queue q{vt::Device{}, nullptr}; + const m3::Music3DitDeviceWeights staged = + m3::StageMusic3DitWeights(q, config, mutated, /*release_host=*/false); + const std::vector out = m3::DitForwardDevice( + q, latents, vllm_test::kMusic3DitLength, condition, vllm_test::kMusic3DitTimestep, config, + staged); + + size_t outside = 0; + double worst = 0.0; + for (size_t i = 0; i < latent_count; ++i) { + const double a = out[i], b = vllm_test::kMusic3DitOut[i]; + const double bound = std::max(kAbsFloor, kRelTol * std::max(std::abs(a), std::abs(b))); + if (!(std::abs(a - b) <= bound)) ++outside; + worst = std::max(worst, std::abs(a - b)); + } + MESSAGE("half-swap mutation: " << outside << " of " << latent_count + << " values outside the bound, worst |diff| = " << worst); + // A defect that moves values by O(1) must move essentially all of them. This + // is the negative control for every ExpectClose above. + CHECK(outside > latent_count / 2); +} + +TEST_CASE("music3 acoustic: the DEVICE DiT refuses every wrong-shaped input by name") { + const vllm::MiniMaxMusic3TransformerConfig config = DitConfig(); + const size_t latent_count = + static_cast(config.in_channels * vllm_test::kMusic3DitLength); + const std::vector latents = ToVector(vllm_test::kMusic3DitLatents, latent_count); + const std::vector condition = ToVector( + vllm_test::kMusic3DitCondition, + static_cast(vllm_test::kMusic3DitLength * config.condition_dim)); + vt::Queue q{vt::Device{}, nullptr}; + + m3::DitWeights host = DitWeights(); + const m3::Music3DitDeviceWeights staged = + m3::StageMusic3DitWeights(q, config, host, /*release_host=*/false); + CHECK_THROWS_AS(m3::DitForwardDevice(q, {1.0f}, vllm_test::kMusic3DitLength, condition, 0.25, + config, staged), + std::runtime_error); + CHECK_THROWS_AS(m3::DitForwardDevice(q, latents, vllm_test::kMusic3DitLength, {1.0f}, 0.25, + config, staged), + std::runtime_error); + CHECK_THROWS_AS(m3::DitForwardDevice(q, latents, 0, condition, 0.25, config, staged), + std::runtime_error); + + // A mis-sized weight is refused at STAGE time — before 9.7 GB moves at real + // dimensions — rather than 36 layers into the first of 660 forwards. + m3::DitWeights broken = DitWeights(); + broken.layers.pop_back(); + CHECK_THROWS_AS(m3::StageMusic3DitWeights(q, config, broken, /*release_host=*/false), + std::runtime_error); + m3::DitWeights short_proj = DitWeights(); + short_proj.proj_in_weight.pop_back(); + CHECK_THROWS_AS(m3::StageMusic3DitWeights(q, config, short_proj, /*release_host=*/false), + std::runtime_error); +} + +TEST_CASE("music3 acoustic: release_host EMPTIES the source, and the staged copy still runs") { + // "Device-resident" has to mean the host copy is GONE, not that a second copy + // exists. On Jetson Thor the two pools are one pool: holding both is a real + // 19.4 GB peak on a box that reboots instead of OOM-killing. + const vllm::MiniMaxMusic3TransformerConfig config = DitConfig(); + const size_t latent_count = + static_cast(config.in_channels * vllm_test::kMusic3DitLength); + const std::vector latents = ToVector(vllm_test::kMusic3DitLatents, latent_count); + const std::vector condition = ToVector( + vllm_test::kMusic3DitCondition, + static_cast(vllm_test::kMusic3DitLength * config.condition_dim)); + + vt::Queue q{vt::Device{}, nullptr}; + m3::DitWeights host = DitWeights(); + const m3::Music3DitDeviceWeights staged = + m3::StageMusic3DitWeights(q, config, host, /*release_host=*/true); + + size_t emptied = 0, total = 0; + for (const m3::DitLayerWeights& layer : host.layers) { + for (const std::vector* v : + {&layer.to_q, &layer.to_k, &layer.to_v, &layer.to_out, &layer.ff_in_weight, + &layer.ff_out_weight}) { + ++total; + if (v->empty() && v->capacity() == 0) ++emptied; + } + } + MESSAGE("release_host: " << emptied << " of " << total + << " per-layer host projections released (empty AND zero capacity)"); + CHECK(emptied == total); + // The time embedder is the ONE thing deliberately kept — it runs on the host + // so that `temb` stays bit-identical to the CPU arm. + CHECK(staged.host_time_embed.time_embed_linear_2_weight.size() == + static_cast(config.inner_dim() * config.inner_dim())); + + // And the staged copy is intact: a released host buffer that had been uploaded + // without a synchronize would read as garbage here rather than as the golden. + const std::vector out = m3::DitForwardDevice( + q, latents, vllm_test::kMusic3DitLength, condition, vllm_test::kMusic3DitTimestep, config, + staged); + ExpectClose(out, vllm_test::kMusic3DitOut, latent_count, "dit after release_host"); +} + // --------------------------------------------------------------------------- // W5 — the vocoder // ---------------------------------------------------------------------------