From 9001204bf9307a7a4dbd4df0b466a42e73950d59 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Mon, 17 Aug 2026 09:09:30 +0000 Subject: [PATCH 1/2] feat(MODEL-MUSIC-MUSIC3): the 2.4B fp32 DiT onto the device, staged once (#672) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit FOLLOWING_AGENTS_PROTOCOL Spec §11.4 recorded three device rows as owed. §12 closed the arm-independent one. This closes the **DiT**, which is the one that mattered: at a real duration it is not one stage among six, it is the request. A 45 s clip at the shipped defaults runs `DitForward` **660 times** (30 steps x 2 CFG branches x 11 windows) for roughly **634 TFLOP against ~29 TFLOP for the entire autoregressive half** — about 20x everything else in the model put together. That is why `d9441ef3`'s device arm reached only 0.946x: it moved the 8.6B language model and left the stage twenty times larger on the host. ## What moved, and onto what **No new kernel.** Every op already existed with a CUDA provider; this adds a forward that composes them. | reference helper | shared op | |---|---| | `Linear` | `vt::MatmulBT` (+ `vt::Add` for the rank-1 bias) | | `LayerNorm` | `vt::LayerNorm` | | `ApplyPartialRotary` | `vt::RopeFromCache` | | `Attention` (NON-causal) | `vt::AttentionCross`, bias `nullptr` | | `value * silu(gate)` | `vt::SiluAndMul`, over a stage-time half swap | | `PointwiseConv` (both 1x1) | `vt::MatmulBT` on the transposed activation | **The 1x1 convolutions are GEMMs, and that is what unblocked the row.** `vt` has no CUDA 1-D convolution provider — the finding §11.4 recorded against the vocoder applies to the DiT's `preprocess_conv` and `postprocess_conv` too. But `conv(x)^T[t][co] = SUM_ci x^T[t][ci] * W[co][ci] = MatmulBT(x^T, W)`, so the forward works FRAME-MAJOR throughout and transposes once on the host at each end, where the tensors are `[128, length]`. Nothing is hand-rolled outside the seam. **fp32 stays fp32.** Spec §2.1: the acoustic half is float32 by upstream's choice. Every staged weight and device activation is `kF32`. ## Correctness — same goldens, same bounds, nothing widened **The CPU arm is bit-identical structurally, not by measurement.** `minimax_music3_acoustic.cpp`, `minimax_music3_ar.cpp`, `minimax_music3_llm.cpp` and `vocoder1d.cpp` have a **ZERO DIFF**. `--speech-device 0` takes the same `DitForward`, source byte for source byte, so there is no number to move. The device forward is an additional entry point in a new file, the shape `minimax_h3_device.cpp` already uses. Reduced dimensions, vs upstream's own goldens, at the EXISTING bound (`kRelTol` 1e-5 / `kAbsFloor` 1e-6), each case reporting BOTH arms' distance to the golden: | arm | worst \|arm - upstream\| | |---|---| | host `DitForward` (the accepted control) | 1.565e-07 | | device forward, CPU backend | 1.192e-07 | | device forward, CUDA sm_110 | 2.980e-07 | FULL SCALE, the real 2.4B checkpoint vs the committed oracle capture on `thor:gpu0`, 11 008 values/step, bounds unchanged (1e-4 / 5e-5 / 5e-6): | arm | step | bit-identical | mean\|d\| | max\|d\| | outside | |---|---|---|---|---|---| | Thor CPU | first | 423 (3.843%) | 1.71434e-06 | 2.38419e-05 | 0 | | Thor CPU | last | 235 (2.135%) | 2.22396e-06 | 2.83718e-05 | 0 | | Thor CUDA | first | 473 (4.297%) | 1.64344e-06 | 2.47955e-05 | 0 | | Thor CUDA | last | 222 (2.017%) | 2.44677e-06 | 2.59876e-05 | 0 | | CONTROL torch-vs-torch | first | 15.416% | 7.526e-07 | 7.153e-06 | — | | CONTROL torch-vs-torch | last | 5.596% | 1.424e-06 | 1.335e-05 | — | The device arm sits ON TOP of the host arm — better on two of four figures, marginally worse on the other two — and both sit at the same multiple of the recorded torch-vs-torch control. **The Thor CPU arm reproduces the x86-64 numbers this spec already recorded VALUE FOR VALUE**, so the CPU path is unchanged across two architectures. **Two mutations, because a bound nothing violates has not been shown to discriminate.** Pre-swapping the `ff_in` halves makes the stage-time swap undo the test's, so the forward computes `silu(value) * gate`: **20 of 20 values outside the bound, worst |diff| 1.538e-03**, four orders above the noise. And the conditional/unconditional branches must differ: 20 of 20 do, on both backends. ## Speed — `thor:gpu0` (NVIDIA Thor, sm_110), per DiT forward Named because a number without its device is meaningless across this fleet; nothing here is compared to a `dgx:gpu0` or `orin:gpu0` number. `VLLM_CPP_MUSIC3_DIT_REPEAT=R` times ONLY the forward loop — the 9.7 GB load, the golden reads and the staging are outside it. | arm | repeats | fwd | loop | per forward | staging | load | |---|---|---|---|---|---|---| | CPU | 1 | 4 | 819.818584 s | **204.954646 s** | no-op | 3.42 | | CPU | 1 | 4 | 819.992 s | **204.998 s** | no-op | 10.37 | | CUDA | 1 | 4 | 0.749077 s | **0.187269 s** | 0.603561 s | 4.79 | | CUDA | 3 | 12 | 2.110301 s | **0.175858 s** | 0.660600 s | 5.32 | | CUDA | 1 | 4 | 0.743367 s | **0.185842 s** | 0.609463 s | 5.1 | | CUDA | 1 | 4 | 0.743881 s | **0.185970 s** | 0.612787 s | 4.44 | Fit: **slope 0.170607 s/forward, intercept 0.063012 s**. The device R=1 point was taken THREE times across two sessions, bracketing R=3, at 0.749077 / 0.743367 / 0.743881 s — 0.77 % spread. **204.955 s host vs 0.1706-0.1873 s device: 1102x on the matched pair, 1201x on the slope.** **The contention asymmetry was measured away, not argued away.** The first CPU point sat at load 10.37 against the device arm's 4.4-5.3, which would have inflated the ratio if it mattered. Re-taken on an idle box (load 3.42) with the fixed instrument it reads **204.954646 s against 204.998 s, 0.021 %**: the host DiT forward is single-threaded on 14 cores, so a load of 10 still leaves it a core. Both points are reported. **The weights are staged ONCE, as a measurement.** One staging costs 0.60-0.66 s; the entire FOUR-forward loop costs 0.745 s and the TWELVE- forward loop 2.110 s, where twelve stagings would be 7.35 s alone. The loop's intercept is a tenth of one staging. A per-forward upload is arithmetically excluded. **Whole-process, which is lower and is the honest ceiling on what a user sees today:** 1054-1071 s vs 238-298 s (**3.5-4.5x**) for the same binary including the identical NAS load, the spread being NAS cache state rather than compute; the full two-arm correctness series, 49 min 17 s vs 15 min 49 s (**3.12x**). The distance between 1100x on the DiT and 4x on the process IS the owed list. **No e2e song pair**, and that is a limit not an omission: at 30 steps the host DiT alone extrapolates to ~37.6 h, and at a setting short enough to run, the pair would be measuring the vocoder. **No parity claim** — SGLang-Omni is `gateable = no` and every reference axis stays `PENDING`. ## One instrument defect, found inside this change The first timing line printed `DIT_TIMING arm=1` on the CPU run: a `const char*` in a doctest `MESSAGE` chain takes the **bool** conversion. That is #672's OWN §11.5 defect reappearing in a new line — the lesson was written down and a fresh `<<` chain reintroduced it. Both lines are now assembled as one `std::string`. EVERY number was then re-taken with the fixed instrument, and the CPU arm's pre-fix point is kept BESIDE its post-fix twin rather than replaced by it, because the pair is what proves the label defect never touched the values: 819.992 s against 819.818584 s. ## What is still OWED, and a correction to §11.4 §11.4 said the depth decoder was blocked on "nothing but the work". **That is wrong, and this row found it out.** The depth decoder and the condition mix run at `ArCompute::kBFloat16`, which rounds the RESULT of every op to bf16 (`minimax_music3_ar.cpp:36-40`). Routing them through an f32 `vt::MatmulBT` would silently drop that rounding — a change to the numbers wearing a refactor's clothes. Mirroring them needs bf16 STORAGE, a dtype decision with its own evidence. They are also ~15 TFLOP against the DiT's 634. The vocoder row is unchanged: `vt` still has no `ConvTranspose1d`, and that op has three consumers. `minimax_music3_device` is added to the merged-GEMM allowlist with a reason rather than folded: `ff.net.0.proj` is ALREADY one merged `nn.Linear` and ALREADY one `MatmulBT`, so a merged-GEMM seam has nothing left to merge, and `layers::UnquantizedMlpGateUpMethod` is bf16-only, bias-free and `OwnedTensor`-resident — three shared-layer changes, not a model fold. ## Lease discipline, recorded because it was imperfect The CUDA build, the correctness series and the first timing runs were driven over `ssh` under `flock $HOME/gpu.lock` — this row's brief, since superseded by `rc`. During that window the fleet reported `thor:gpu0` FREE while it was in use. The first device series ran under a real `rc hold` (`a91d21dc`, 10:32Z-10:54Z) that carried no `--reason` and a 75 m TTL for ~22 m of work; both are errors. The matched pair above ran under `8aa5cd6d` with a reason string and a 40 m TTL, released early at 23 m on completion. `rc run` cannot serve this particular job because the build lives on the device host's filesystem and `rc run`'s container sees only `/workspace`; a shell is the sanctioned case for a `hold`. Thor's uptime is unbroken across every arm (`system boot Jun 5 15:36`; `up 2 days, 14:37` at the first arm through `16:56` at the last), so no pair straddles a restart, and no number here is compared to one from `dgx:gpu0` (GB10) or `orin:gpu0` (Orin). ## Gates Local x86-64, non-zero assertion counts: `test_minimax_music3_acoustic` 32/283 (was 27/265), `test_minimax_music3_speech` 9/223, `test_minimax_music3_ar` 26/352, `test_minimax_music3_loader` 21/1393, `test_minimax_music3_quant` 29/125, `test_speech_engine` 11/38, `test_capi` 65/653, `test_speech_api` 6/67, `test_openai_api_server` 62/733. With `CHECKPOINT_ROOT` set: `test_minimax_music3_acoustic_real` 6/76, `test_minimax_music3_ar_real` 4/894, `test_minimax_music3_quant_real` 6/319, `test_minimax_music3_llm_real` 4/220 — every one matching its recorded count. Thor CUDA build: `test_minimax_music3_acoustic` 32/291 (8 more than x86 — the CUDA device case RUNS instead of skipping), `test_minimax_music3_acoustic_real` 6/985 (`device 0`) and 6/988 (`device 1`), `test_minimax_music3_speech` 9/223, `test_speech_engine` 11/37 (one fewer BY DESIGN on a CUDA build). Issue: #672 Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --- .agents/benchmark-record.md | 123 ++++++ .agents/specs/minimax-music3.md | 316 +++++++++++++- CMakeLists.txt | 1 + docs/BENCHMARKS.md | 3 +- docs/FEATURES.md | 2 +- docs/STATUS.md | 2 +- docs/USAGE.md | 59 ++- .../models/minimax_music3_device.h | 151 +++++++ .../models/minimax_music3_speech.h | 28 +- scripts/merged-gemm-consistency-allowlist.txt | 2 + .../models/minimax_music3_device.cpp | 397 ++++++++++++++++++ .../models/minimax_music3_speech.cpp | 55 ++- .../test_minimax_music3_acoustic_real.cpp | 146 ++++++- .../models/test_minimax_music3_acoustic.cpp | 238 +++++++++++ 14 files changed, 1483 insertions(+), 40 deletions(-) create mode 100644 include/vllm/model_executor/models/minimax_music3_device.h create mode 100644 src/vllm/model_executor/models/minimax_music3_device.cpp diff --git a/.agents/benchmark-record.md b/.agents/benchmark-record.md index 342b283d9..82ce3c448 100644 --- a/.agents/benchmark-record.md +++ b/.agents/benchmark-record.md @@ -22379,3 +22379,126 @@ oracle that never produced a file. **No parity number is claimed and the token gate is still unclaimed.** `our-ON == our-OFF` remains FALSE and unattributed. Owed detail lives under `## Owed` in [`mtp-k-gt-1.md`](specs/mtp-k-gt-1.md). +## MUSIC3-DIT-DEVICE — the 2.4B fp32 DiT on `thor:gpu0`, per-forward A/B against the host reference (2026-08-17, `row/MUSIC3-DIT-DEVICE`, #672) + +**Not a parity ratio.** There is no reference leg: SGLang-Omni is `gateable = no` +and serves the native layout, so this is an INTERNAL two-arm number about our own +host reference vs our own device arm. Every axis in `docs/BENCHMARKS.md` against +the reference stays `PENDING`. + +### Device, named because a number without one is meaningless here + +**`thor:gpu0` — NVIDIA Thor, sm_110, aarch64, 14 cores, ~122 GB UNIFIED, driver +595.78.** Nothing below is compared to a `dgx:gpu0` (GB10) or `orin:gpu0` number; +the three boxes are different machines. Image `vllmcpp-thor:cuda13.0.1`, nvcc +13.0.88, `-DVLLM_CPP_CUDA=ON -DVLLM_CPP_CUDA_ARCHITECTURES=110 +-DVLLM_CPP_TRITON=OFF -DVLLM_CPP_SERVER=ON`, no cutlass. Checkpoint mounted +read-only from the NAS. Same binary, same weights, same committed inputs on both +arms; the arms never overlapped. + +**Lease discipline, recorded because it was imperfect.** The CUDA build, the +two-arm correctness series and the first timing runs were driven over `ssh` under +`flock $HOME/gpu.lock`, which is what this row's brief specified and which the +`rc` lease system has since superseded — during that window the fleet reported +`thor:gpu0` as FREE while it was in use. The device-arm timing series that the +speed claim rests on was run under a real `rc hold` on `thor:gpu0` +(`a91d21dc`, taken 10:32Z, released 10:54Z on completion). The hold carried no +`--reason` string and a 75 m TTL for ~22 m of work; both are recorded as errors, +and single commands should go through `rc run --max-runtime` instead. + +### What is timed + +`VLLM_CPP_MUSIC3_DIT_REPEAT=R` runs the guided velocity R times per timestep in +`tests/parity/test_minimax_music3_acoustic_real.cpp`. The timer brackets ONLY +that loop: the 9.7 GB checkpoint load, the golden reads and the weight staging +are all outside it, and staging is timed separately. One guided velocity is TWO +DiT forwards (the conditional and the unconditional CFG branch). + +| arm | repeats | forwards | loop | per forward | staging | box load | +|---|---|---|---|---|---|---| +| CPU (`VLLM_CPP_MUSIC3_DEVICE=0`) | 1 | 4 | 819.818584 s | **204.954646 s** | 0 (no-op) | 3.42 | +| CPU (`=0`) | 1 | 4 | 819.992 s | **204.998 s** | 0 (no-op) | 10.37 | +| CUDA (`=1`) | 1 | 4 | 0.749077 s | **0.187269 s** | 0.603561 s | 4.79 | +| CUDA (`=1`) | 3 | 12 | 2.110301 s | **0.175858 s** | 0.660600 s | 5.32 | +| CUDA (`=1`) | 1 | 4 | 0.743367 s | **0.185842 s** | 0.609463 s | 5.1 | +| CUDA (`=1`) | 1 | 4 | 0.743881 s | **0.185970 s** | 0.612787 s | 4.44 | + +Two-point fit over the device arm's 4- and 12-forward runs: + + slope = 0.170607 s per forward intercept = 0.063012 s + +**Per DiT forward at the capture's geometry (latent length 86, sequence 87): +204.955 s host vs 0.1706-0.1873 s device — 1102x on the matched R=1 pair, 1201x +on the fitted slope.** The device R=1 point was taken THREE times across two +sessions, bracketing R=3, at 0.749077 / 0.743367 / 0.743881 s: a 0.77 % spread. + +**The contention asymmetry was measured away, not argued away.** The first CPU +point sat at box load 10.37 against the device arm's 4.4-5.3, which would have +inflated the ratio if it mattered. Re-taken on an idle box (load 3.42) with the +fixed instrument it reads **204.954646 s against 204.998 s — 0.021 %**. The host +DiT forward is single-threaded on a 14-core box, so a load of 10 still leaves it +a core. Both points are in the table rather than the convenient one. + +### The weights are staged ONCE, as a measurement + +One staging costs 0.60-0.66 s. The entire FOUR-forward loop costs 0.745 s and the +TWELVE-forward loop 2.110 s; twelve stagings would be 7.35 s by themselves. The +loop's fitted intercept is 0.063 s, a tenth of a single staging. A per-forward +or per-window upload is excluded arithmetically, not by reading the code. + +### The whole-process ratios — lower, and the honest ceiling on what a user sees + +Same gate binary end to end, including the identical 9.7 GB NAS load on both +arms: 1054-1071 s (CPU) vs 238-298 s (CUDA), **3.5-4.5x** — the spread is NAS +cache state, not compute. The earlier full two-arm +correctness series, identical scripts throughout: 49 min 17 s vs 15 min 49 s, +**3.12x**. Load averages across the series 4.0-5.1, box otherwise idle apart from +`k3s`; `uptime` recorded on both sides of every run and the host never rebooted. + +The distance between 1100x on the DiT and 4x on the process IS the owed list: the +checkpoint load, the DAC vocoder and the RVQ depth decoder are unchanged and now +dominate. + +### Extrapolation, labelled as one + +The only geometry measured is the capture's single 86-frame window. Applying the +fit to the 660 forwards a 45 s clip runs at the shipped defaults (30 steps x 2 CFG +x 11 windows) gives **~37.6 h of DiT on the host against ~113 s on the device**, +with the one-time staging 0.53 % of the device total. That is an extrapolation +from one window geometry and is not a measured clip-level result. + +**No end-to-end song pair is offered, and that is a limit rather than an +omission.** At 30 steps the host arm's DiT alone is ~37.6 h, so an e2e pair is +not runnable on the CPU arm at a realistic setting; at a setting short enough to +run, the DiT is a small enough share that the pair would be measuring the +vocoder. + +### Correctness taken in the same series, at bounds that did not move + +Full scale, the real 2.4B fp32 checkpoint against the committed oracle capture, +11 008 values per step, `kDitRelTol` 1e-4 / `kDitAbsFloor` 5e-5 / +`kDitMeanAbsTol` 5e-6 — all unchanged: + +| arm | step | bit-identical | mean\|d\| | max\|d\| | outside | +|---|---|---|---|---|---| +| Thor CPU | first | 423 (3.843 %) | 1.71434e-06 | 2.38419e-05 | 0 | +| Thor CPU | last | 235 (2.135 %) | 2.22396e-06 | 2.83718e-05 | 0 | +| Thor CUDA | first | 473 (4.297 %) | 1.64344e-06 | 2.47955e-05 | 0 | +| Thor CUDA | last | 222 (2.017 %) | 2.44677e-06 | 2.59876e-05 | 0 | +| CONTROL torch-vs-torch | first | 15.416 % | 7.526e-07 | 7.153e-06 | — | +| CONTROL torch-vs-torch | last | 5.596 % | 1.424e-06 | 1.335e-05 | — | + +The Thor CPU arm reproduces the x86-64 numbers already recorded for this gate +VALUE FOR VALUE, so the CPU path is unchanged across two architectures. + +### Instrument defect found and fixed inside this series + +The first timing line printed `DIT_TIMING arm=1` on the CPU run: a `const char*` +in a doctest `MESSAGE` chain takes the bool conversion. It is the SAME defect +§11.5 recorded for this row's arm banner, reintroduced by a fresh `<<` chain. +Both lines are now assembled as one `std::string`. EVERY number in the table was +then re-taken with the fixed instrument, and the CPU arm's pre-fix point is kept +beside its post-fix twin rather than replaced by it, because the pair is what +proves the label defect never touched the values: 819.992 s against +819.818584 s. + diff --git a/.agents/specs/minimax-music3.md b/.agents/specs/minimax-music3.md index 2b190f376..279e7c89d 100644 --- a/.agents/specs/minimax-music3.md +++ b/.agents/specs/minimax-music3.md @@ -1094,6 +1094,15 @@ language model is slow", and the language model is not the part that is slow — the LM's own weight load is 180 s of I/O and its forward is 12-14% of the AR profile. +**One sentence above is now out of date on the device arm, and is corrected here +rather than left to mislead.** *"the depth decoder and the DiT do not go through +`vt` at all"* was true when it was written; it is still true of the DEPTH DECODER +on both arms and of the DiT under `--speech-device 0`. It is NOT true of the DiT +under `--speech-device 1`, which §13 routes through `vt::MatmulBT`, +`vt::LayerNorm`, `vt::AttentionCross`, `vt::RopeFromCache`, `vt::SiluAndMul` and +`vt::Add` with device-resident weights. Every profile number quoted above is the +CPU arm's and still describes it exactly. + --- ## 10. The parity sweep, the music-only server, and the weights record (#672) @@ -1452,8 +1461,16 @@ order intact. Neither is in this change, because a bit-identity claim needs its own measurement and this change's evidence budget went to the device seam. **That last paragraph is now DONE — §12.** It landed with its own measurement and -its own gate, and it took `vocoder1d::Conv1d` with it. The two device rows above -are still owed and unchanged. +its own gate, and it took `vocoder1d::Conv1d` with it. + +**The DiT row is now DONE — §13.** The depth-decoder row is still owed and its +"blocked on" entry above is now known to be WRONG in one respect, corrected in +§13.5: it is not "nothing but the work". The shipped depth decoder runs +`ArCompute::kBFloat16`, which rounds the RESULT of every op to bf16 +(`minimax_music3_ar.cpp:36-40`), so routing it through an f32 `vt::MatmulBT` +would silently drop that rounding. Mirroring it needs bf16 STORAGE, which is a +dtype decision with its own numeric evidence rather than a transcription. The +vocoder row is unchanged and still blocked on the missing op. ### 11.5 Evidence — Jetson Thor, sm_110, in the container @@ -1810,3 +1827,298 @@ questions. What is *not* pending: the parallelism is real and asserted, not hoped for. The gate's thread-distinctness leg fails if the body runs on one thread, and `VLLM_CPP_CPU_THREADS` now governs these three kernels. + +--- + +## 13. The 2.4B fp32 DiT reaches the device (#672) — §11.4's second owed row + +§11.4 recorded three device rows as owed. §12 closed the arm-independent one. +This closes the **DiT**, which is the one that mattered most, and it says up +front which of the other two it does not close and why. + +### 13.1 Why this row and not another + +The DiT is not one stage among six; at a real duration it is the request. + +A 45 s clip at the shipped defaults (`num_inference_steps` 30) runs `DitForward` +**660 times** — 30 steps x 2 CFG branches x 11 windows — and each call is 36 +blocks over `length + 1` tokens at inner dim 2048, ff 8192. That is on the order +of **634 TFLOP in the DiT against ~29 TFLOP for the entire autoregressive half**: +the DiT is roughly **20x everything else in the model put together**. On the +scalar host loops it is measured in hours; one run was killed at 8 h 11 m having +averaged 4.3 of 20 cores. + +That is also why §11.5's device arm reached only 0.946x. It moved the 8.6B +language model, which is real work, and left the stage that is twenty times +larger on the host. A device arm that does not include the DiT is a device arm +for the minority of the profile. + +### 13.2 What moved, onto which shared op, and what did NOT + +**No new kernel.** Every op below already existed with a CUDA provider; this row +adds a forward that composes them, not a kernel that competes with them. + +| reference helper (`minimax_music3_acoustic.cpp`) | shared op | +|---|---| +| `Linear` | `vt::MatmulBT` (+ `vt::Add` for the rank-1 bias) | +| `LayerNorm` | `vt::LayerNorm` | +| `ApplyPartialRotary` | `vt::RopeFromCache` over a `[seq, rotary_dim]` cache | +| `Attention` (NON-causal) | `vt::AttentionCross`, `bias = nullptr` | +| `value * silu(gate)` | `vt::SiluAndMul`, over a stage-time half swap | +| residual adds | `vt::Add` | +| `PointwiseConv` (both 1x1 convolutions) | `vt::MatmulBT` on the transposed activation | + +The stage table, after: + +| stage | `--speech-device 1` runs it | +|---|---| +| 8.6B `Qwen3ForCausalLM`, prefill + decode + paged KV | **device** (§11) | +| guided logits, top-k draw, frame feedback | host | +| 0.646B RVQ depth decoder | **host** — OWED, and §13.5 corrects why | +| condition mix (once per WINDOW, not per step) | **host** — OWED | +| **2.4B fp32 DiT, every step, both CFG branches** | **device — THIS ROW** | +| scheduler, CFG mix, Euler step, overlap blend, carry | host (elementwise on `[128, length]`; not the cost) | +| DAC Flow-VAE vocoder | **host — BLOCKED on a missing op** (§11.4) | + +### 13.3 Four things this had to get right + +**The 1x1 convolutions are GEMMs, and that is what unblocked the row.** `vt` has +no CUDA 1-D convolution provider at all — the finding §11.4 recorded against the +vocoder applies here too, because the DiT's `preprocess_conv` and +`postprocess_conv` are `nn.Conv1d(kernel=1)`. But a kernel-1 convolution over +`[C, L]` is a GEMM once the activation is transposed: + + conv(x)[co][t] = SUM_ci W[co][ci] * x[ci][t] + transposed: conv(x)^T[t][co] = SUM_ci x^T[t][ci] * W[co][ci] = MatmulBT(x^T, W) + +So the forward works FRAME-MAJOR `[length, channels]` throughout and transposes +once on the host at each end, where the tensors are `[128, length]`. No +convolution op is needed, nothing is hand-rolled outside the seam, and the +vocoder's blocker does not transfer. + +**The half swap is an identity applied exactly once.** Upstream computes +`ff_out(gate_states * silu(gate))` where `gate_states, gate = ff_in(x).chunk(2, +-1)` — the FIRST half is the value, the SECOND is what SiLU runs on +(`transformer_minimax_music3.py:142-143`). `vt::SiluAndMul` computes +`silu(x[:, :D]) * x[:, D:]`: the opposite assignment. Exchanging the two ROW +BLOCKS of the projection and the two halves of its bias — **once, at stage +time** — makes the shared op compute upstream's expression exactly, with no +per-step permutation. 660 forwards x 36 layers would otherwise permute a +`[seq, 16384]` tensor 23 760 times per clip. The gate for this is a mutation, not +an assertion: §13.4. + +**The rotary is the LEADING slice, and `vt::RopeFromCache` already rotates +exactly that.** Music3 ships `rotary_dim` 32 of `head_dim` 64 and rotates only +the leading window, leaving the tail copied through +(`minimax_music3_acoustic.cpp:500-514`). `RopeFromCacheKernel` indexes +`row + pair` and `row + pair + half` within each head and computes +`x*c - y*s, x*s + y*c` — the same rotation over the same slice. `BuildDitRotaryTables` +returns cos/sin already duplicated across both halves of the window, so the cache +this forward builds is the FIRST half of each, packed `cos | sin`. + +**The attention is NON-causal and `vt::Attention` is not it.** Upstream +dispatches with no mask (`:97-103`), so every token attends to every token +INCLUDING the prepended timestep one. `vt::Attention` is the causal op; using it +would have silently masked the future and still produced a finite, plausible +tensor. `vt::AttentionCross` with a null bias is the op that means this. + +### 13.4 Correctness — same goldens, same bounds, nothing widened + +**The CPU arm is bit-identical, and structurally rather than by measurement.** +`minimax_music3_acoustic.cpp`, `minimax_music3_ar.cpp`, `minimax_music3_llm.cpp` +and `vocoder1d.cpp` have a **zero diff** in this change. `--speech-device 0` +takes the same `DitForward`, source byte for source byte, so there is no number +to move. The device forward is an ADDITIONAL entry point in a new file +(`minimax_music3_device.cpp`), which is the shape `minimax_h3_device.cpp` and +`ltx2_device.cpp` already use. + +**Reduced dimensions, against upstream's own goldens, at the EXISTING bound.** +`DitForwardDevice` is checked through the SAME `ExpectClose` at the SAME +`kRelTol` 1e-5 / `kAbsFloor` 1e-6 as `DitForward`, and each case reports BOTH +arms' distance to the golden — because the question is not whether the two arms +agree with each other (a shared-helper comparison proves consistency, not +correctness) but whether the device arm is as close to UPSTREAM as the host arm +already is: + +| arm | worst \|arm - upstream\| | +|---|---| +| host `DitForward` (the accepted control) | 1.565e-07 | +| device forward, CPU backend | 1.192e-07 | +| device forward, **CUDA sm_110** | 2.980e-07 | + +All three are inside the 1e-6 absolute floor with room to spare, and **no +tolerance was relaxed**. The CPU-backend arm is closer to upstream than the host +loops are; the CUDA arm is about 1.9x the host arm's distance and about a fifth +of the bound. + +**Two mutations, because a bound that nothing violates has not been shown to +discriminate.** + +* **The half swap.** Pre-swapping the host weights makes the stage-time swap undo + the test's, so the forward computes `silu(value) * gate` — the wrong network, + same shapes, same finiteness. **20 of 20 values outside the bound, worst + \|diff\| 1.538e-03**, four orders above the noise. The pair pins the DIRECTION, + not just the magnitude: routing it the other way round would fail the right + case and pass this one. +* **The condition.** Conditional and unconditional forwards must be different + tensors — a DiT that dropped its conditioning would match both goldens + identically. 20 of 20 differ, on both backends. + +Two more cases guard the staging contract itself: every mis-sized weight is +refused **at stage time** naming the tensor (before 9.7 GB moves at real +dimensions), and `release_host` is asserted to leave the source vectors empty +AND at zero capacity while the staged copy still reproduces the golden — which is +also the check that would catch a released host buffer uploaded without a +synchronize. + +**FULL SCALE — the real 2.4B fp32 checkpoint against the oracle capture, on +sm_110.** `tests/parity/test_minimax_music3_acoustic_real.cpp` now takes +`VLLM_CPP_MUSIC3_DEVICE` (default 0 = CPU, so an unset environment reproduces +every number this file ever printed) resolved through the SAME +`multimodal::SpeechEngineDeviceType` the engine calls. Both arms, same box, same +binary, same goldens, **same bounds** — `kDitRelTol` 1e-4 / `kDitAbsFloor` 5e-5 / +`kDitMeanAbsTol` 5e-6, all unchanged. 11 008 values per step: + +| arm | step | bit-identical | mean\|d\| | max\|d\| | outside | +|---|---|---|---|---|---| +| **Thor CPU** (`device 0`) | first | 423 (3.843 %) | 1.71434e-06 | 2.38419e-05 | **0** | +| **Thor CPU** | last | 235 (2.135 %) | 2.22396e-06 | 2.83718e-05 | **0** | +| **Thor CUDA** (`device 1`) | first | 473 (4.297 %) | **1.64344e-06** | 2.47955e-05 | **0** | +| **Thor CUDA** | last | 222 (2.017 %) | 2.44677e-06 | **2.59876e-05** | **0** | +| CONTROL (torch vs torch, `set_num_threads(1)`) | first | 15.416 % | 7.526e-07 | 7.153e-06 | — | +| CONTROL | last | 5.596 % | 1.424e-06 | 1.335e-05 | — | + +**Three things this table shows that one arm could not.** The device arm sits +ON TOP of the host arm rather than beside it — better on two of the four figures +(more bit-identical and a lower mean at the first step, a lower max at the last) +and marginally worse on the other two, which is what two correct float32 +implementations of the same graph look like. Both arms sit at the same +multiple of the recorded torch-vs-torch control (about 1.2-1.7x its mean, 2-3.5x +its max), so the device arm did not move the row's relationship to the control. +And **the Thor CPU arm reproduces the x86-64 numbers this spec already recorded — +3.843 %, 1.714e-06, 2.384e-05; 2.135 %, 2.224e-06, 2.837e-05 — VALUE FOR VALUE**, +so the CPU path is unchanged across two architectures, not merely unchanged on +the box that measured it. + +The four ARM=1 cases that print those numbers are 464 assertions against the +CPU arm's 461; the three extra are this row's staging `CHECK` and the two +`REQUIRE`s that refuse a device arm with no staged weights. + +### 13.6 Speed — MEASURED, on one named device, per DiT forward + +**Device: `thor:gpu0` — NVIDIA Thor, sm_110, aarch64, 14 cores, ~122 GB UNIFIED, +driver 595.78.** Every number below is from that one box. No number here is +compared to one from `dgx:gpu0` (GB10) or `orin:gpu0`, because those are +different machines and a ratio across them would mean nothing. + +Image `vllmcpp-thor:cuda13.0.1`, nvcc 13.0.88, configured `-DVLLM_CPP_CUDA=ON +-DVLLM_CPP_CUDA_ARCHITECTURES=110 -DVLLM_CPP_TRITON=OFF -DVLLM_CPP_SERVER=ON`, +no cutlass. Checkpoint read-only from the NAS. Same binary, same weights, same +committed inputs on both arms; the arms never overlapped. + +**What is timed is the DiT and only the DiT.** `VLLM_CPP_MUSIC3_DIT_REPEAT=R` +makes the gate run its guided velocity R times per timestep instead of once, and +the timer brackets ONLY that loop — the 9.7 GB checkpoint load, the golden reads +and the staging are all outside it, and the staging is timed separately. + +| arm | repeats | forwards | loop | per forward | staging | box load | +|---|---|---|---|---|---|---| +| CPU (`device 0`) | 1 | 4 | **819.818584 s** | **204.954646 s** | 0 (host, no-op) | 3.42 | +| CPU (`device 0`) | 1 | 4 | **819.992 s** | **204.998 s** | 0 (host, no-op) | 10.37 | +| CUDA (`device 1`) | 1 | 4 | **0.749077 s** | **0.187269 s** | 0.603561 s | 4.79 | +| CUDA (`device 1`) | 3 | 12 | **2.110301 s** | **0.175858 s** | 0.660600 s | 5.32 | +| CUDA (`device 1`) | 1 | 4 | **0.743367 s** | **0.185842 s** | 0.609463 s | 5.1 | +| CUDA (`device 1`) | 1 | 4 | **0.743881 s** | **0.185970 s** | 0.612787 s | 4.44 | + +**Per DiT forward at the capture's geometry (latent length 86, seq 87): +204.955 s on the host, 0.1706-0.1873 s on the device — between 1102x and +1201x.** The two-point fit over the device arm's 4- and 12-forward runs gives + + slope = 0.170607 s per forward intercept = 0.063012 s + +so the ratio is 1102x taken on the matched R=1 pair and 1201x taken on the +fitted per-forward slope. The device R=1 point was taken THREE times across two +sessions, bracketing the R=3 point, at 0.749077 / 0.743367 / 0.743881 s — a +0.77 % spread. + +**The contention asymmetry was checked rather than assumed, and it is nil.** The +first CPU point was taken at box load 10.37 while the device points sat at +4.4-5.3, which would have inflated the ratio if it mattered. It was re-taken on +an idle box (load 3.42) with the fixed instrument: **204.954646 s vs 204.998 s, +agreeing to 0.021 %**. The host DiT forward is single-threaded and this box has +14 cores, so a load of 10 still leaves it a core. Both CPU points are reported +above rather than the convenient one. + +**The weights are staged ONCE, and this is the measurement that says so rather +than the code comment.** One staging costs 0.60-0.66 s. The ENTIRE four-forward +loop costs 0.745 s and the twelve-forward loop 2.110 s; twelve stagings would be +7.35 s on their own. The loop's fitted intercept is 0.063 s — a tenth of one +staging. A per-forward upload is arithmetically excluded by the numbers, not +argued away. + +Extrapolated to a full clip — and it is an EXTRAPOLATION, labelled as one, +because the only geometry measured is the capture's single 86-frame window — the +660 forwards of a 45 s clip at the shipped defaults are **~37.6 h of DiT on the +host against ~113 s on the device, with the one-time staging 0.54 % of the +latter**. + +**The whole-process ratios, which are lower and are the honest ceiling on what a +user sees today.** The same gate binary end to end, including the identical +9.7 GB NAS load on both arms, ran 1054-1071 s (CPU) vs 238-298 s (CUDA) — +**3.5-4.5x**, the spread being NAS cache state rather than compute; +and the earlier full two-arm correctness series, identical scripts throughout, +ran 49 min 17 s vs 15 min 49 s — **3.12x**. The gap between 1100x on the DiT and +4x on the process is the point of §13.5: the load, the host vocoder and the depth +decoder are unchanged, and they now dominate. + +**No end-to-end song pair is offered.** At the shipped 30 steps the host arm's +DiT alone is ~37.6 h, so an e2e pair at a realistic setting is not runnable on +the CPU arm; at a setting short enough to run, the DiT is a small enough share +that the pair would measure the vocoder. The per-forward A/B above is the +measurement that isolates what this row changed, and no clip-level speed claim is +made from it. + +**No parity claim.** SGLang-Omni is still `gateable = no`; every reference axis +in `docs/BENCHMARKS.md` stays `PENDING`. + +### 13.7 One instrument defect, found inside this change + +The first revision of the timing line printed **`DIT_TIMING arm=1`** on the CPU +run. `vt::DeviceTypeName` returns `const char*`, and a `const char*` fed to +doctest's `MESSAGE` chain takes the **bool** conversion and prints `1`. The +staging line had it too, printing `(1)` where it meant `(host, no-op)`. + +**This is #672's own §11.5 defect reappearing in a new line**, which is the +reason it is recorded here rather than quietly fixed: the lesson from the first +occurrence was written down, and a fresh `<<` chain reintroduced it anyway. Both +lines are now assembled as one `std::string` and printed, which is what the arm +banner beside them already did — and the banner is why the numbers survived, +because it said `ran on 'cpu' (VLLM_CPP_MUSIC3_DEVICE=0)` correctly while the +line below it said `arm=1`. + +**Every number in §13.6 was re-taken with the fixed instrument**, and the CPU +arm's pre-fix point is kept beside its post-fix twin rather than replaced by it, +because the pair is what proves the label defect never touched the values: +819.992 s (pre-fix, `arm=1` printed, load 10.37) against 819.818584 s (post-fix, +`arm=cpu` printed, load 3.42). The correct banner sat above both, the process +wall clocks corroborate both, and the correctness numbers both runs printed match +the x86-64 values this spec already recorded value for value. + +### 13.5 What is still OWED, and a correction to §11.4 + +§11.4 said the depth decoder was blocked on "nothing but the work". **That is +wrong, and this row is where it was found out.** The shipped depth decoder and +condition mix run at `ArCompute::kBFloat16`, which rounds the RESULT of every op +to bf16 (`minimax_music3_ar.cpp:36-40`) because that is what torch stores. +Routing them through an f32 `vt::MatmulBT` would silently drop that rounding — a +change to the numbers wearing a refactor's clothes. Mirroring them needs bf16 +STORAGE so the shared op rounds where the reference's `Store` rounds, which is a +dtype decision with its own numeric evidence, not a transcription. It is also +worth ~15 TFLOP against the DiT's 634, so it is second in size as well as second +in order. + +The condition mix has a further reason to be second: it runs **once per window**, +not once per step, so it is outside the 660-forward loop entirely. + +The vocoder row is unchanged: `vt` still has no `ConvTranspose1d` with any +provider, that op has three consumers, and it is its own row. diff --git a/CMakeLists.txt b/CMakeLists.txt index 898640405..e9c7a941e 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -778,6 +778,7 @@ add_library(vllm STATIC src/vllm/model_executor/models/minimax_music3_quant.cpp src/vllm/model_executor/models/minimax_music3_ar.cpp src/vllm/model_executor/models/minimax_music3_acoustic.cpp + src/vllm/model_executor/models/minimax_music3_device.cpp src/vllm/model_executor/models/minimax_music3_llm.cpp src/vllm/model_executor/models/minimax_music3_speech.cpp src/vllm/model_executor/models/gpt2.cpp diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index ea50a632b..cf9515a28 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -486,8 +486,9 @@ built on it rather than keeping the flattering one. | vLLM 0.26 re-benchmark | Pending | Re-run the binding grids on the advanced pin | | MiniMax-H3 FP4 speed (W-FP4a) | **Measured GB10 (`row/H3-FP4-GPU-E2E`).** Marlin W4A16 byte-exact vs bf16; fp4 a memory win, 0.8x bf16/forward. Real-ckpt fp4-resident e2e RUNS (mp4/wav) | fp4 speed CLOSED. bf16-vs-quant A/B: ENCODER half MEASURED (§8.15), DiT half NOT (no bf16 render exists). Detail: benchmark-record + spec §8 | | LTX-2.5 axes | Speed `PENDING` (vllm-omni#6066 has no native 2.5), binding oracle too. **SIZE: 704x448/25f and 448x256/25f both COMPLETE on GB10 (4231 s, 3085 s)**; one run each, contended box, no oracle, no ceiling (#1088) | Wall is NOT the VAE decode after #1041/#1009: a ~1731 s serial phase FLAT in resolution is 57-66% (#1087). ~59 GiB cliff did NOT recur (floor 38.9 GiB). 2 baselines UNRESOLVED (lock). PROMPTED real-ckpt render OWED | -| MiniMax-Music3 (`MiniMaxMusic3ForConditionalGeneration`) | **Every axis vs the reference stays `PENDING`.** A PARTIAL device arm now exists (#672): only the 8.6B LM runs on the accelerator, so this is an internal two-arm number and NOT a parity ratio | Denominator: SGLang-Omni `748a0b43` in its production configuration (both CUDA graphs, compiled DIT and DAV, batched seeded sampling) | +| MiniMax-Music3 (`MiniMaxMusic3ForConditionalGeneration`) | **Every axis vs the reference stays `PENDING`.** A PARTIAL device arm now exists (#672): the 8.6B LM and the 2.4B fp32 DiT run on the accelerator, so the rows below are internal two-arm numbers and NOT parity ratios | Denominator: SGLang-Omni `748a0b43` in its production configuration (both CUDA graphs, compiled DIT and DAV, batched seeded sampling) | | MiniMax-Music3 device arm, Jetson Thor sm_110 (#672) | `--device 1` vs `--device 0`, same request/seed, idle box: 2 AR frames **846.6 vs 835.1 s (1.014x SLOWER)**; 10 frames **1430.4 vs 1512.1 s (0.946x)**. Fit: **-11.65 s/frame, +34.8 s fixed** | A third duration (the fit has no residual), and moving the depth decoder + DiT + vocoder, which are 5 of 6 stages and still host scalar loops | +| MiniMax-Music3 DiT device arm, `thor:gpu0` sm_110 (#672) | Per DiT forward at the capture's geometry, same binary/weights/inputs, idle box: **204.955 s host vs 0.186 s device, 1102x** (1201x fitted). Weights staged ONCE (0.61 s; loop intercept 0.063 s). Whole process 3.5-4.5x | e2e song pair NOT runnable (host DiT alone ~37.6 h at 30 steps). Depth decoder/condition mix (bf16-storage), vocoder (no `ConvTranspose1d`) still host. Detail: benchmark-record | | MiniMax-Music3 CPU host kernels, x86-64 20-core (#672) | KERNEL A/B at the vocoder's real geometry, min of 5 interleaved rounds: convolution chain **13.36 -> 1.25 s, 10.7x**; `Conv1d` 12.03x, `LinearNoBias` 10.88x. Output fingerprints IDENTICAL on both arms | e2e pair VOID (cold CIFS cache; a foreign `ctest` at load 76.6) and re-running. Stages 0/1 only ~2x: the pivot trades WEIGHT locality for accumulator locality. Detail: benchmark-record | | MiniMax-H3 render coherence (`row/H3-RENDER-CLOSE` #77) | **CLOSED: a COHERENT scene on GB10.** #70/#74 white was wrong-PARTITION usage (t2va on the ref2va ckpt); t2va on the FL2VA GGUF renders a prompt-matched orange cat (adj-cos 0.95 vs 0.06, no patch-grid) | Verified first: t2va inputs byte-exact vs upstream; CUDA device==host at seq 1920. Follow-up `H3-TASK-PARTITION-GUARD`: the task/partition mismatch now RAISES 1:1 with `_resolve_task` (spec §8.6-8.7) | | MiniMax-H3 image conditioning (`row/H3-CONDITIONED-E2E`, `row/H3-VISION-SCATTER`, `row/H3-REF2VA-ASSEMBLY`) | **fl2va COHERENT; ref2va assembly bug FIXED+gated.** vision→cond scatter gated; ref2va block-dim double-division fixed + RED-first gated (128 vs 512) + a permanent ref2va DiT-forward rung (§8.10) | grid RE-ATTRIBUTED: with the fix ref2va grids in fp4 AND bf16, and t2va with no refs on the ref2va NVFP4 also grids while FL2VA-GGUF renders, so it is the **NVFP4 checkpoint/loader**, NOT assembly/fp4 (§8.10) | diff --git a/docs/FEATURES.md b/docs/FEATURES.md index ebec16432..6238b43b4 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -167,7 +167,7 @@ in `ltx2_text_encoder.cpp` is the call that would have to change. | Whisper audio encoder | openai/whisper-small; whisper-large-v3 (Voxtral cfg) | encoder tower 77/77; large-v3 tower 203/203 | pending | | MiniMax-H3 DiT (`MiniMaxH3DiTModel`, vllm-omni lane) | MiniMax-H3 (33.1B video+audio) | portable 79/79; all three modalities COHERENT on Q4_K_M (§8.20); PRUNED ckpts run, Q8_0 seam 0.9941 (§8.21); ref2va grid was NVFP4 quant error, §8.9 REFUTED; GGUF/NVFP4/bf16 shards stream | FP4/Marlin landed; speed pending; no bf16 render yet. Render from the Q4_K_M GGUF, not the NVFP4 arm. Krea 2 text-to-image (roadmap C11) is scoped to reuse these DiT seams | | LTX-2.5 DiT (`LTX2VideoTransformer3DModel`, Lightricks lane) | LTX-2.5 (21.00B video+audio) | `SPIKE`. DiT, VAEs+ENCs, cond, pipeline, quant loaders gated, reduced dims. Prompt AdaLN host+dev; Gemma-4->xattn FIXTURE-gated. Img chain PPM->resize->encode->place->noise. Temporal x2 ups gated, UNDRIVEN. Render OWED | `ltx-2.5`/`ltx2-gen`. ~29 GB NVFP4/GB10, FP8 ~44 GB, +24 GB tower. FP8/torchao/NVFP4; kf abs-pos ported; BOTH DiTs load, NO `allow_unported`. IMG+LAST kf SERVED `crf=0`, A2V WAV+LoRA; DiffVAE/ref refused. PENDING | -| MiniMax-Music3 (`MiniMaxMusic3ForConditionalGeneration`, diffusers lane) | MiniMax-Music3 (8.6B Qwen3 LLM + 0.646B RVQ decoder + 2.4B fp32 DiT + DAC Flow-VAE); diffusers arm, ~28.5 GB | `ACTIVE`. Loader 1413/1413; AR, acoustic and the 8.6B LM forward all gated vs real weights; `SpeechRegistry` + `vllm_speech_*` v21 + `/v1/audio/speech`; GGUF Q4_K depth decoder value-gated. HTTP request OBSERVED (#852) | Not compared to a reference. Host kernels multi-core, same song bytes (§12). PARTIAL device arm (`--speech-device 1`, #672): 8.6B LM only. Denominator: SGLang-Omni, production config | +| MiniMax-Music3 (`MiniMaxMusic3ForConditionalGeneration`, diffusers lane) | MiniMax-Music3 (8.6B Qwen3 LLM + 0.646B RVQ decoder + 2.4B fp32 DiT + DAC Flow-VAE); diffusers arm, ~28.5 GB | `ACTIVE`. Loader 1413/1413; AR, acoustic and the 8.6B LM forward all gated vs real weights; `SpeechRegistry` + `vllm_speech_*` v21 + `/v1/audio/speech`; GGUF Q4_K depth decoder value-gated. HTTP request OBSERVED (#852) | No reference number. Host kernels multi-core, same song bytes (§12). PARTIAL device arm (#672): 8.6B LM + 2.4B fp32 DiT staged once (§13); rest host. Denominator SGLang-Omni production | | LTX-2.5 DFR base + generated keyframe slots | LTX-2.5 (21.00B video+audio) | gated vs EXECUTED upstream `dfr_layout` + 3 `dfr_pipeline` helpers @ `fd4ded7f` (`test_ltx2_dfr` 11/11, 652 assertions); canvas, tiles, stitch, carry-forward as EXACT index vectors, since each defect is plausible| `--pipeline-kind dfr`. Canvas PADS 9 to 25 then trims back; slots on the x8 grid, MARKED, read back BEFORE the trim. `num_generated_keyframes` SERVED elsewhere. Temporal ROUNDS refused (#986); detail LoRA refused (#975)| | LTX-2.5 tiled + streaming Conv VAE decode | LTX-2.5 video VAE | gated vs executed upstream `ltx_core` @ `fd4ded7f` (`test_ltx2_tiling` 10/10, 915 assertions); one-tile and untiled-spatial controls BIT-EXACT vs untiled on both causality arms; an untiled frames axis is REFUSED | Streams temporal chunks through upstream's AUTO layout (768/64 px, 80/24 frames); above one tile the pixel volume is never materialized. NO-OP below 768px and 81 frames; 81-120 IS tiled, differing 6.70% of range | | LTX-2.5 Conv VAE decode arithmetic width | LTX-2.5 video VAE | `test_ltx2_vae` "the decode's convolution accumulates in f32", entering through `Ltx2VideoDecodeStreaming`; widening the accumulator to `double`, or deleting the production call site, each turns it RED | **f32**, the width `F.conv3d` uses at f32 AND bf16 (MEASURED). Was f64 at 8 sites ([#1008](https://github.com/mudler/vllm.cpp/issues/1008)). Conv sums BLOCKED per input channel, as torch's. STORAGE stays f32; bf16 owed | diff --git a/docs/STATUS.md b/docs/STATUS.md index 451c34eae..c2f9604ad 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -144,7 +144,7 @@ token-for-token correctness against the pinned oracle. | InternLM2 dense (fused-`wqkv` interleaved split) | Correctness-complete, speed-pending | Token-exact 16/16 (internlm2-chat-1_8b): 12/16 strict + 4/16 bf16 near-tie (max gap 0.0 nats), 0 divergent; first InternLM model; ZERO new compute kernel (reuses the Llama dense forward; the only delta is a loader-side de-interleave of the fused `wqkv`, which packs q/k/v interleaved by KV-group) | | MiniMax-H3 (`MiniMaxH3DiTModel`, video+audio DIFFUSION) | **ABI v12 ONE SURFACE; device selector uses generic `DeviceType`; DSR 32.** t2va+fl2va COHERENT; bf16 shards STREAM | ref2va ckpt fidelity §8.12; encoder A/B §8.15; GB10 re-verify residual; CPU fold 6/137 (one queue + device provenance mutation-gated) | | LTX-2.5 (`LTX2VideoTransformer3DModel`, video+audio DIFFUSION) | **L1-L9c landed (#435).** 21.00B / 48 blocks. `VideoEngine` seam + ABI **v18**, DiT forward (CPU f32 parity, bf16 device-resident), Gemma-4 TE, both VAEs, connector, pipeline, NVFP4/FP8, keyframe bias (#658) | BOTH shipped DiTs now load inside the contract, no `allow_unported`. One runs device-resident on GB10; those 320x192/25f frames ARE a scene. A prompted render is OWED; speed and oracle parity `PENDING` | -| MiniMax-Music3 (`MiniMaxMusic3ForConditionalGeneration`, text-to-MUSIC) | **`ACTIVE`: W0-W7 landed; every stage including the 8.6B LM forward is implemented and gated (#672).** Oracle is the OPEN diffusers PR #14456 `c6da9936` | GGUF arms for 4 components owed. LM forward gated in a control; HTTP OBSERVED (#852). PARTIAL device arm, Thor sm_110 (#672): 8.6B LM only. CPU kernels 10.7x on the vocoder chain, same song BYTES. No reference number | +| MiniMax-Music3 (`MiniMaxMusic3ForConditionalGeneration`, text-to-MUSIC) | **`ACTIVE`: W0-W7 landed; every stage including the 8.6B LM forward is implemented and gated (#672).** Oracle is the OPEN diffusers PR #14456 `c6da9936` | GGUF arms for 4 components owed. LM forward gated in a control; HTTP OBSERVED (#852). PARTIAL device arm, Thor sm_110 (#672): 8.6B LM + 2.4B fp32 DiT (§13). CPU kernels 10.7x on the vocoder chain. No reference number | | Command-R / Cohere dense (`CohereForCausalLM`) | Implemented, gate-blocked | ZERO-new-kernel port grounded in vLLM `commandr.py`: weight-only Cohere LayerNorm + GPT-J full-width RoPE + PARALLEL residual + `logit_scale` + tied embeddings, all reuse; compiles, links, self-registers. No SACRED gate yet (real checkpoints HF-gated, ungated ones tiny-random, GPU box disk-full); oracle run-verified at W0. See docs/BENCHMARKS.md | | Phi-1 / Phi-2 dense (`PhiForCausalLM`, parallel residual) | Correctness-complete, speed-pending | Token-exact 16/16 (microsoft/phi-2): 9/16 strict + 7/16 bf16 near-ties (max gap 0.25 nats), 0 forward-divergent; the OLDER Microsoft Phi arch, DISTINCT from Phi-3/Phi-4; ZERO new compute kernel (GPT-J parallel residual, LayerNorm-with-bias, biased qkv/dense, partial NeoX rope 32/80, non-gated NewGELU MLP reusing `vt::GeluTanh`, untied biased lm_head); F16 dtype-aware loader | | MiniCPM dense (`MiniCPMForCausalLM`, three scalars) | Correctness-complete, speed-pending | Token-exact 16/16 (openbmb/MiniCPM-2B-sft-bf16): 10/16 strict + 6/16 bf16 near-ties (max gap 0.0 nats), 0 forward-divergent; first OpenBMB MiniCPM model; ZERO new compute kernel (the Llama/Granite dense forward plus three scalars: scale_emb, scale_depth/sqrt(layers) residual, dim_model_base logit scaling), tied lm_head; `.bin`-only weights converted to safetensors via trusted torch | diff --git a/docs/USAGE.md b/docs/USAGE.md index ab4f6e450..2f8e9f343 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -1946,11 +1946,16 @@ claimed. Ask for a short duration and few `num_inference_steps` while you are checking that it works. The part that dominates is *not* the one you would guess. The 8.6B language -model goes through `vt` and uses the CPU threadpool; the RVQ depth decoder and -the DiT do not — they are scalar host loops with a double accumulator, written -that way in W2-W5 so their reduction order is reproducible against torch, and -they run single-threaded. In one 0.1 s request the depth decoder alone is the -majority of the wall clock. +model goes through `vt` and uses the CPU threadpool; the RVQ depth decoder does +not — it is a scalar host loop with a double accumulator, written that way in +W2/W3 so its reduction order is reproducible against torch. In one 0.1 s request +the depth decoder alone is the majority of the wall clock. + +At a *real* duration the DiT is the whole story instead, which is why it is the +stage that moved first: a 45 s clip at the default 30 inference steps runs the +DiT 660 times (30 steps x 2 CFG branches x 11 windows) for roughly 634 TFLOP +against about 29 TFLOP for the entire autoregressive half. On the host loops +that is measured in hours. `--speech-device 1` puts it on the accelerator. #### What runs on the device, and what does not @@ -1963,20 +1968,34 @@ direction that matters. |---|---| | 8.6B `Qwen3ForCausalLM` (prefill + every decode step, its paged KV) | **device** | | guided-logit pipeline, top-k draw, frame feedback embedding | host (two 200 000-wide rows per step; not the cost) | +| **2.4B fp32 flow-matching DiT** (every denoise step, both CFG branches) | **device**, weights staged ONCE | | 0.646B RVQ depth decoder (7 steps per frame) | **host**, scalar loops | -| condition mix, 2.4B fp32 flow-matching DiT, scheduler | **host**, scalar loops | +| condition mix (once per window), scheduler, CFG mix, Euler step | **host** | | DAC Flow-VAE vocoder (`Conv1d` / `ConvTranspose1d`) | **host**, scalar loops | The language model reaches the device because it is already routed through the shared `Qwen3DenseModel` forward that five text registrations ride — nothing was forked for it, and the only thing this option changes is which queue that -forward is handed and where its KV cache is allocated. The other stages do not, -for two different reasons, and both are owed rather than hidden: - -* the depth decoder and the DiT are host `std::vector` reference loops - under `-ffp-contract=off`, kept that way so their reduction order stays - reproducible against torch. Moving them means routing them through the shared - `vt` GEMM seam with device-resident weights, not adding a flag; +forward is handed and where its KV cache is allocated. + +The DiT reaches it the same way: through shared `vt` ops only +(`MatmulBT`, `LayerNorm`, `AttentionCross`, `RopeFromCache`, `SiluAndMul`, +`Add`), with **no new kernel**. Its 9.7 GB of fp32 weights are uploaded once per +request, before the window loop, and the host copy is released as each tensor +lands — a 45 s clip runs that forward 660 times, so a per-step or even +per-window upload would cost more than the compute it enables. `fp32 stays +fp32`: the acoustic half is float32 because upstream chose float32 for it, and +this arm mirrors that rather than buying speed with a narrower dtype. + +The remaining stages do not move, for two different reasons, and both are owed +rather than hidden: + +* the depth decoder and the condition mix are host `std::vector` + reference loops under `-ffp-contract=off`, and they run at + `ArCompute::kBFloat16` — every op's *result* is rounded to bf16, which is what + upstream stores. Routing them through an f32 GEMM would silently drop that + rounding, so mirroring them needs bf16 storage, which is a dtype decision with + its own numeric evidence rather than a transcription; * the vocoder needs `ConvTranspose1d`, and **`vt` has no such op at all** — the 1-D convolutions it does have (`vt::DepthwiseConv1d`, `vt::CausalConv1dFwd`) are depthwise or causal-with-state, and `vt::Conv2d` and `vt::DepthwiseConv1d` @@ -1984,10 +2003,16 @@ for two different reasons, and both are owed rather than hidden: this stage would need, so it is named here rather than hand-rolled outside the seam. -Because the host stages are unchanged, the CPU arm is **bit-identical** to the -one every Music3 correctness gate was taken on, and the device arm's output -differs from it exactly where the language model's own arithmetic differs — one -stage, not five. +Because the host stages are unchanged — and because `--speech-device 0` takes +the same `DitForward` it always did, source byte for source byte — the CPU arm +is **bit-identical** to the one every Music3 correctness gate was taken on. The +device arm's output differs from it exactly where the language model's and the +DiT's own arithmetic differ: two stages, not six, and neither difference is a +shape or an ordering defect. The DiT's device forward is gated against the same +upstream goldens at the same tolerance as the host one; nothing was widened for +it, and `VLLM_CPP_MUSIC3_DEVICE=1` runs that comparison on either arm +(`tests/parity/test_minimax_music3_acoustic_real.cpp`, with +`VLLM_CPP_MUSIC3_DIT=1`). **The two arms do not produce the same song, and that is structural.** The autoregressive stage has no greedy path upstream: it ends every draw in a seeded diff --git a/include/vllm/model_executor/models/minimax_music3_device.h b/include/vllm/model_executor/models/minimax_music3_device.h new file mode 100644 index 000000000..291e28b17 --- /dev/null +++ b/include/vllm/model_executor/models/minimax_music3_device.h @@ -0,0 +1,151 @@ +// MiniMax-Music3 — the DEVICE-RESIDENT acoustic forward (#672, spec §11.4). +// +// `DitForward` (minimax_music3_acoustic.cpp) is the portable reference: host +// `std::vector` throughout, one scalar loop per op. It is what every +// Music3 gate was taken on and it is NOT changed by this header — the device arm +// is an additional entry point, not a rewrite of the existing one. +// +// WHY THIS EXISTS. The 2.4B fp32 DiT is ~20x the whole autoregressive half: a +// 45 s clip at the default 30 inference steps runs `DitForward` 660 times (30 +// steps x 2 CFG branches x 11 windows) at ~0.96 TFLOP each, and on the scalar +// host loops that is measured in hours. `d9441ef3` put the 8.6B language model +// on the device and reached 0.946x precisely because this half did not move. +// +// ─── WHAT IS PORTED, AND ONTO WHICH SHARED OP ──────────────────────────────── +// Every step below names the reference helper it replaces. NO new kernel is +// added: each one is a shared `vt::` op that already carries a CUDA provider. +// +// Linear -> vt::MatmulBT (+ vt::Add for the rank-1 bias) +// LayerNorm -> vt::LayerNorm +// ApplyPartialRotary -> vt::RopeFromCache over a [seq, rotary_dim] cache +// Attention (NON-causal) -> vt::AttentionCross (bias = nullptr) +// `x * silu(gate)` -> vt::SiluAndMul, over a STAGE-TIME half swap +// residual adds -> vt::Add +// PointwiseConv (1x1) -> vt::MatmulBT on the transposed activation +// +// ─── fp32 STAYS fp32 ───────────────────────────────────────────────────────── +// Spec §2.1: the acoustic half is float32 because upstream chose float32 for it, +// and this arm mirrors that. Every staged weight and every device activation +// below is `vt::DType::kF32`. Narrowing to bf16 would be a different change with +// its own evidence, not a free speedup taken in passing. +// +// ─── NUMERICS: CLOSE, NOT BIT-IDENTICAL, AND SAID SO ───────────────────────── +// This arm does NOT reproduce the host reference bit for bit, and does not claim +// to. Three named differences, none of them a shape or an order-of-operations +// defect: +// +// 1. The reference accumulates every reduction in `double` and stores float32 +// (see the dtype note at the top of minimax_music3_acoustic.cpp). The +// shared ops accumulate in float32 — which is what torch itself does, so on +// this axis the device arm is the CLOSER mirror of upstream, not the looser +// one. +// 2. A `nn.Linear` bias enters the reference INSIDE the accumulator +// (`double acc = bias`), and here it is a separate `vt::Add` afterwards. +// 3. `vt::AttentionCross`'s CUDA kernel uses the online-softmax recurrence +// where the reference uses an explicit three-pass max/exp/normalize. +// +// It is gated against the SAME upstream goldens at the SAME tolerance as the +// host forward (tests/vllm/models/test_minimax_music3_acoustic.cpp), and no +// tolerance was widened to admit it. +#pragma once + +#include +#include + +#include "vllm/model_executor/models/minimax_music3_acoustic.h" +#include "vt/device.h" +#include "vt/ops.h" + +namespace vllm { +namespace models { +namespace music3 { + +// One transformer block's weights, resident on the queue's device. +// +// `ff_in_weight` / `ff_in_bias` are the ONLY tensors whose CONTENT differs from +// the host struct, and the reason is mechanical rather than a choice. +// `transformer_minimax_music3.py:142-143` computes `ff_out(gate_states * +// silu(gate))` where `gate_states, gate = ff_in(x).chunk(2, -1)` — the FIRST +// half is the value and the SECOND is what SiLU runs on. `vt::SiluAndMul` +// computes `silu(x[:, :D]) * x[:, D:]`, i.e. the opposite assignment. Swapping +// the two ROW BLOCKS of the projection once, at stage time, makes the shared op +// compute exactly upstream's expression with no per-step permutation and no +// bespoke kernel. The multiply is commutative, so this is an identity, not an +// approximation. +struct Music3DitDeviceLayer { + vt::Tensor norm1_weight; // [inner] + vt::Tensor norm1_bias; // [inner] + vt::Tensor to_q; // [attn_inner, inner] + vt::Tensor to_k; // [attn_inner, inner] + vt::Tensor to_v; // [attn_inner, inner] + vt::Tensor to_out; // [inner, attn_inner] + vt::Tensor norm2_weight; // [inner] + vt::Tensor norm2_bias; // [inner] + vt::Tensor ff_in_weight; // [2 * ff, inner], HALVES SWAPPED (see above) + vt::Tensor ff_in_bias; // [2 * ff], HALVES SWAPPED + vt::Tensor ff_out_weight; // [inner, ff] + vt::Tensor ff_out_bias; // [inner] +}; + +// The DiT staged ONCE onto a device, with the storage that owns it. +// +// STAGED ONCE IS THE WHOLE POINT. 660 forwards per 45 s clip means a per-step +// upload of 9.7 GB would cost more than the compute it enables; the fixed +// +34.8 s `d9441ef3` measured for the language model is what a one-time upload +// looks like, and this arm keeps that shape. `storage` holds one +// `vt::Backend::Alloc` block per tensor, freed with this object. +struct Music3DitDeviceWeights { + vt::Tensor preprocess_conv_weight; // [concat, concat] (the 1x1 kernel axis is dropped) + vt::Tensor proj_in_weight; // [inner, concat] + std::vector layers; + vt::Tensor proj_out_weight; // [in_channels, inner] + vt::Tensor postprocess_conv_weight; // [in_channels, in_channels] + + // The timestep embedder stays on the HOST, deliberately and cheaply. + // `time_proj` + `time_embed` is ONE row through a [inner, fourier_dim] and an + // [inner, inner] projection — 4.7M MACs against the 2.4G MACs PER TOKEN the + // block stack runs, i.e. under a millionth of the forward at any real window + // length. Running it through the existing `DitTimestepEmbedding` keeps that + // piece BIT-IDENTICAL to the CPU arm for free; moving it would have needed an + // ungated SiLU op that `vt` does not carry. Only these five tensors are + // populated in this struct. + DitWeights host_time_embed; + + std::vector> storage; +}; + +// Upload the DiT to `queue`'s device, once. +// +// `release_host` EMPTIES each source vector as it is uploaded. The shipped DiT +// is 9.7 GB of fp32 and Jetson Thor's ~122 GB is UNIFIED — host and device draw +// on one pool — so holding both copies is a real 19.4 GB peak on the box this +// arm was written for, which `vm.overcommit_memory=1` and zero swap turn into a +// REBOOT rather than an OOM kill (.agents/environment.md). Pass true from a +// serving path, false from a gate that compares the two arms. +// +// Throws (naming the tensor) if a weight is mis-sized for `config`, or if the +// device has no provider for one of the ops the forward needs — a refusal at +// stage time rather than 36 layers into the first step. +Music3DitDeviceWeights StageMusic3DitWeights(vt::Queue& queue, + const MiniMaxMusic3TransformerConfig& config, + DitWeights& weights, bool release_host); + +// `DitForward`'s device twin: same inputs, same outputs, same layouts. +// +// `latents` [in_channels, length], CHANNEL-major (host) +// `condition` [length, condition_dim], FRAME-major (host) +// returns [in_channels, length], the flow-matching VELOCITY (host) +// +// The host<->device boundary is the ARGUMENTS ONLY: one upload of +// [length, concat_channels] on the way in and one download of +// [length, in_channels] on the way out, per call. Everything between — all 36 +// blocks, both 1x1 convolutions, both projections — stays in device memory. +std::vector DitForwardDevice(vt::Queue& queue, const std::vector& latents, + int64_t length, const std::vector& condition, + double timestep, + const MiniMaxMusic3TransformerConfig& config, + const Music3DitDeviceWeights& weights); + +} // namespace music3 +} // namespace models +} // namespace vllm diff --git a/include/vllm/model_executor/models/minimax_music3_speech.h b/include/vllm/model_executor/models/minimax_music3_speech.h index 29ae19ae2..0cad95313 100644 --- a/include/vllm/model_executor/models/minimax_music3_speech.h +++ b/include/vllm/model_executor/models/minimax_music3_speech.h @@ -69,6 +69,7 @@ #include "vllm/model_executor/models/minimax_music3_acoustic.h" #include "vllm/model_executor/models/minimax_music3_ar.h" +#include "vllm/model_executor/models/minimax_music3_device.h" #include "vllm/model_executor/models/minimax_music3_loader.h" #include "vllm/multimodal/speech_engine.h" @@ -198,10 +199,35 @@ struct Music3DenoiseOptions { // * the carry span is taken from the RESTORED latents (denoise.py:252-256); // * the scheduler is RESET per window (denoise.py:152-156), so step 0's sigma // is the first of a fresh schedule and not a continuation. +// +// THE DEVICE ARM (#672, spec §11.4) is the optional trailing parameter and +// NOTHING ELSE. Left default-constructed — which every caller written before it +// does — the loop runs `DitForward`, the host reference every Music3 gate was +// taken on, unchanged. Given a queue and the DiT staged onto that queue's +// device, the two `DitForward` calls per step become `DitForwardDevice` and +// nothing else in this function moves: the condition mix, the overlap blend, the +// CFG mix, the Euler step and the carry stay on the host, in the same order, +// computing the same numbers. +// +// The weights are staged by the CALLER, once, and handed in — not staged here. +// A 45 s clip runs this loop's inner body 660 times over 11 windows, so staging +// per window would upload 9.7 GB eleven times for one clip; the fixed cost has +// to sit outside every loop in this function, and putting the parameter here +// rather than a path inside is what makes that structural instead of careful. +struct Music3DenoiseDeviceArm { + vt::Queue* queue = nullptr; + const Music3DitDeviceWeights* dit = nullptr; + // Both or neither. One alone is a caller that thinks it asked for the device + // arm and did not, so it is REFUSED rather than silently ignored. + bool engaged() const { return queue != nullptr && dit != nullptr; } + bool half_set() const { return (queue != nullptr) != (dit != nullptr); } +}; + std::vector> Music3DenoiseChunks( const std::vector& frame_hiddens, int64_t num_frames, const MiniMaxMusic3Config& config, const Music3AcousticWeights& weights, - const Music3DenoiseOptions& options, const Music3NoiseSource& noise); + const Music3DenoiseOptions& options, const Music3NoiseSource& noise, + const Music3DenoiseDeviceArm& device_arm = {}); // The decode + stitch (decoders.py:75-92): each window's latents through the // vocoder, cropped by `VocoderCropSpan`, concatenated, and CLAMPED to [-1, 1] diff --git a/scripts/merged-gemm-consistency-allowlist.txt b/scripts/merged-gemm-consistency-allowlist.txt index 7d2dcd34f..ea5770573 100644 --- a/scripts/merged-gemm-consistency-allowlist.txt +++ b/scripts/merged-gemm-consistency-allowlist.txt @@ -48,3 +48,5 @@ minimax_h3_device # known-drift pending fold. The DTYPE half of this blocker is minimax_h3_video_vae_device # deliberately-not-merged: the video VAE's mid-block feed-forward already ships w1 as a MERGED [2*ff_inner, dim] operand and already issues ONE MatmulBT, so nothing here is an unmerged gate/up pair -- the epilogue is the only hand-call. It cannot take layers::UnquantizedMlpGateUpMethod for two reasons the seam does not carry: the VAE runs the whole block in f32 (DType::kF32 activations end to end, deliberately, because it is a decoder whose output is pixels rather than logits) while the method's DBufs are kBF16, and every VAE Linear carries a rank-1 bias the bias-free method has no slot for. Same weight-RESIDENCY blocker as minimax_h3_device besides: the DiT stages device tensors up front and binds plain views rather than OwnedTensor/ResidentWeight. Adopting the seam means giving it an f32 arm and a bias arm, which is a shared-layer change, not a model fold. minimax_h3_encoder_device # deliberately-not-merged: the H3-Encoder's gate/up are ggml BLOCK-QUANT weights (the 32B tower is kept in Q4_K/Q6_K so it fits a 122 GB box), and layers::UnquantizedMlpGateUpMethod is by name and by contract the UNQUANTIZED arm -- it takes an OwnedTensor of plain bf16 and has no block-quant path. Worse, the shipped Q4_K_M checkpoint keeps gate/up UNFUSED whenever the group mixes encodings, so there is not always a single merged [2I,H] operand to hand a merged-GEMM seam at all. Adopting the seam here means giving it a keep-quant arm, which is a shared-layer change, not a model fold. + +minimax_music3_device # deliberately-not-merged: there is NO unmerged gate/up pair here. Upstream's `ff.net.0.proj` is ONE nn.Linear producing [2*ff_inner, inner] which `.chunk(2,-1)` splits (transformer_minimax_music3.py:142-143), so the DiT already ships the merged operand and already issues ONE vt::MatmulBT -- the epilogue is the only hand-call, and a merged-GEMM seam has nothing left to merge. It cannot take layers::UnquantizedMlpGateUpMethod for three reasons the seam does not carry, each of which is a shared-layer change rather than a model fold. DTYPE: the whole acoustic half is f32 by upstream's own choice (spec §2.1) and the method's DBufs are kBF16, so adopting it would narrow a tensor this row is required to keep wide. BIAS: `ff.net.0.proj` and `ff.net.2` both carry a rank-1 bias and the bias-free method has no slot for one. RESIDENCY: the method takes an OwnedTensor staged on demand via ResidentWeight, whereas StageMusic3DitWeights uploads device tensors UP FRONT and binds plain views -- the same ownership blocker minimax_h3_device and minimax_h3_video_vae_device already record. There is a fourth, smaller mismatch worth naming so it is not rediscovered: upstream computes `value * silu(gate)` with the VALUE half first, the opposite assignment to vt::SiluAndMul's `silu(first) * second`, which this file resolves by exchanging the two row blocks of the projection ONCE at stage time. diff --git a/src/vllm/model_executor/models/minimax_music3_device.cpp b/src/vllm/model_executor/models/minimax_music3_device.cpp new file mode 100644 index 000000000..0070f382f --- /dev/null +++ b/src/vllm/model_executor/models/minimax_music3_device.cpp @@ -0,0 +1,397 @@ +// MiniMax-Music3 — the DEVICE-RESIDENT acoustic forward. See +// minimax_music3_device.h for what is ported onto which shared op, why fp32 +// stays fp32, and the three named reasons this arm is close to but not +// bit-identical to the host reference. +// +// ─── THE ONE STRUCTURAL DECISION IN THIS FILE ──────────────────────────────── +// +// The reference works in CHANNEL-MAJOR [channels, length] at the two ends and +// FRAME-MAJOR [length, channels] in the middle, and transposes between them +// (minimax_music3_acoustic.cpp:558-564, :625-631). This file transposes ONCE on +// the host at each boundary and stays FRAME-MAJOR everywhere in between, +// because both 1x1 convolutions are then plain GEMMs: +// +// conv1d(kernel=1): out[co][t] = Σ_ci W[co][ci] * in[ci][t] +// transposed: out^T[t][co] = Σ_ci in^T[t][ci] * W[co][ci] +// = MatmulBT(in^T, W) +// +// So `preprocess_conv` and `postprocess_conv` need no convolution op at all — +// which matters, because `vt` has no CUDA 1-D convolution provider (spec §11.4) +// and hand-rolling one outside the seam is what AGENTS.md forbids. The two +// host-side transposes it costs are [in_channels, length] tensors: 128 x length +// floats, against the 2.4G MACs per token the block stack runs. +#include "vllm/model_executor/models/minimax_music3_device.h" + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "vllm/model_executor/models/dense_device_glue.h" +#include "vt/backend.h" +#include "vt/dtype.h" +#include "vt/op_provider.h" // GetOp — the stage-time provider refusal +#include "vt/ops.h" + +namespace vllm { +namespace models { +namespace music3 { + +namespace { + +using dense_attn::DBuf; +using dense_attn::Dev; +using dense_attn::MakeTensor; +using dense_attn::Reshape; +using vt::DType; +using vt::Tensor; + +[[noreturn]] void Fail(const std::string& message) { throw std::runtime_error(message); } + +void RequireStageSize(const std::vector& values, int64_t expected, const char* what) { + if (static_cast(values.size()) != expected) { + Fail(std::string("MiniMax-Music3 DiT stage: ") + what + " is " + + std::to_string(values.size()) + " values, expected " + std::to_string(expected)); + } +} + +// A rank-N f32 view over a fresh device block, owned by `storage`. +// +// The host source is copied and then, when `release` is set, DROPPED — the +// vector is swapped with an empty one rather than merely cleared, because +// `clear()` keeps the capacity and the whole point is to return the 9.7 GB to +// the allocator before the next tensor asks for its device twin. +Tensor UploadF32(vt::Backend& backend, vt::Queue& queue, std::vector& src, + const std::vector& shape, bool release, + std::vector>* storage) { + int64_t numel = 1; + for (int64_t s : shape) numel *= s; + const size_t bytes = static_cast(numel) * sizeof(float); + void* p = backend.Alloc(bytes); + std::shared_ptr owner(p, [&backend](void* q) { backend.Free(q); }); + backend.Copy(queue, p, src.data(), bytes); + // The copy must have LANDED before the host buffer goes away. On a CPU queue + // this is a memcpy and the sync is free; on CUDA the source is pageable host + // memory, so releasing it under an unsynchronized async copy is exactly the + // use-after-free that reads as a plausible-looking wrong tensor. + if (release) { + backend.Synchronize(queue); + std::vector().swap(src); + } + storage->push_back(std::move(owner)); + return MakeTensor(p, DType::kF32, queue.device, shape); +} + +// `vt::MatmulBT` + an optional rank-1 row-broadcast bias — the device twin of +// the reference `Linear`. The bias is a SEPARATE add here where the reference +// seeds the accumulator with it; that is difference (2) in the header's +// numerics note, and it is a float32 rounding, not a reordering of the sum. +void LinearDev(vt::Queue& q, Tensor& out, const Tensor& in, const Tensor& weight, + const Tensor* bias) { + vt::MatmulBT(q, out, in, weight); + if (bias != nullptr && bias->data != nullptr) vt::Add(q, out, out, *bias); +} + +} // namespace + +Music3DitDeviceWeights StageMusic3DitWeights(vt::Queue& queue, + const MiniMaxMusic3TransformerConfig& config, + DitWeights& weights, bool release_host) { + const int64_t in_channels = config.in_channels; + const int64_t concat = config.concat_channels(); + const int64_t inner = config.inner_dim(); + const int64_t attn_inner = config.num_attention_heads * config.attention_head_dim; + const int64_t ff = config.ff_inner_dim; + const int64_t fourier = config.fourier_embedding_dim; + if (static_cast(weights.layers.size()) != config.num_layers) { + Fail("MiniMax-Music3 DiT stage: the weights carry " + + std::to_string(weights.layers.size()) + " blocks, the config declares " + + std::to_string(config.num_layers)); + } + + // REFUSE UP FRONT, before 9.7 GB moves. `vt::GetOp` throws naming the op and + // the device, so a backend without (say) a cross-attention provider is a + // one-line refusal at stage time rather than a failure 36 layers into the + // first of 660 forwards. Every op the forward below calls is listed. + for (vt::OpId op : {vt::OpId::kMatmulBT, vt::OpId::kAdd, vt::OpId::kLayerNorm, + vt::OpId::kSiluAndMul, vt::OpId::kRopeFromCache, + vt::OpId::kAttentionCross}) { + (void)vt::GetOp(op, queue.device.type); + } + + // The timestep embedder is validated FIRST, before anything is uploaded and + // therefore before `release_host` destroys anything. A size error found after + // the 36 blocks had been released would be a correct refusal that had already + // consumed the caller's weights. + const int64_t fourier_half = fourier / 2; + RequireStageSize(weights.time_proj_weight, fourier_half, "time_proj.weight"); + RequireStageSize(weights.time_embed_linear_1_weight, inner * fourier, + "time_embed.linear_1.weight"); + RequireStageSize(weights.time_embed_linear_1_bias, inner, "time_embed.linear_1.bias"); + RequireStageSize(weights.time_embed_linear_2_weight, inner * inner, + "time_embed.linear_2.weight"); + RequireStageSize(weights.time_embed_linear_2_bias, inner, "time_embed.linear_2.bias"); + + vt::Backend& backend = vt::GetBackend(queue.device.type); + Music3DitDeviceWeights staged; + staged.layers.resize(static_cast(config.num_layers)); + const bool rel = release_host; + auto up = [&](std::vector& src, const std::vector& shape, const char* what) { + int64_t numel = 1; + for (int64_t s : shape) numel *= s; + RequireStageSize(src, numel, what); + return UploadF32(backend, queue, src, shape, rel, &staged.storage); + }; + + // The 1x1 convolutions ship as [out, in, 1]; the kernel axis is dropped here + // because the GEMM form above consumes them as [out, in]. The element count + // is identical, so this is a reinterpretation of the same bytes, not a slice. + staged.preprocess_conv_weight = + up(weights.preprocess_conv_weight, {concat, concat}, "preprocess_conv.weight"); + staged.proj_in_weight = up(weights.proj_in_weight, {inner, concat}, "proj_in.weight"); + + for (int64_t l = 0; l < config.num_layers; ++l) { + DitLayerWeights& src = weights.layers[static_cast(l)]; + Music3DitDeviceLayer& dst = staged.layers[static_cast(l)]; + dst.norm1_weight = up(src.norm1_weight, {inner}, "norm1.weight"); + dst.norm1_bias = up(src.norm1_bias, {inner}, "norm1.bias"); + dst.to_q = up(src.to_q, {attn_inner, inner}, "attn.to_q.weight"); + dst.to_k = up(src.to_k, {attn_inner, inner}, "attn.to_k.weight"); + dst.to_v = up(src.to_v, {attn_inner, inner}, "attn.to_v.weight"); + dst.to_out = up(src.to_out, {inner, attn_inner}, "attn.to_out.0.weight"); + dst.norm2_weight = up(src.norm2_weight, {inner}, "norm2.weight"); + dst.norm2_bias = up(src.norm2_bias, {inner}, "norm2.bias"); + + // THE HALF SWAP (minimax_music3_device.h documents why). Upstream computes + // `value * silu(gate)` with value FIRST; `vt::SiluAndMul` computes + // `silu(first) * second`. Exchanging the two row blocks of the projection + // and the two halves of its bias — once, here — makes the shared op compute + // upstream's expression exactly. Doing it at stage time rather than per step + // is what keeps it free: 660 forwards x 36 layers would otherwise permute a + // [seq, 16384] tensor 23 760 times. + RequireStageSize(src.ff_in_weight, 2 * ff * inner, "ff.net.0.proj.weight"); + RequireStageSize(src.ff_in_bias, 2 * ff, "ff.net.0.proj.bias"); + { + std::vector swapped(static_cast(2 * ff * inner)); + const size_t half = static_cast(ff * inner); + std::memcpy(swapped.data(), src.ff_in_weight.data() + half, half * sizeof(float)); + std::memcpy(swapped.data() + half, src.ff_in_weight.data(), half * sizeof(float)); + if (rel) std::vector().swap(src.ff_in_weight); + dst.ff_in_weight = up(swapped, {2 * ff, inner}, "ff.net.0.proj.weight (swapped)"); + } + { + std::vector swapped(static_cast(2 * ff)); + const size_t half = static_cast(ff); + std::memcpy(swapped.data(), src.ff_in_bias.data() + half, half * sizeof(float)); + std::memcpy(swapped.data() + half, src.ff_in_bias.data(), half * sizeof(float)); + if (rel) std::vector().swap(src.ff_in_bias); + dst.ff_in_bias = up(swapped, {2 * ff}, "ff.net.0.proj.bias (swapped)"); + } + dst.ff_out_weight = up(src.ff_out_weight, {inner, ff}, "ff.net.2.weight"); + dst.ff_out_bias = up(src.ff_out_bias, {inner}, "ff.net.2.bias"); + } + + staged.proj_out_weight = up(weights.proj_out_weight, {in_channels, inner}, "proj_out.weight"); + staged.postprocess_conv_weight = + up(weights.postprocess_conv_weight, {in_channels, in_channels}, "postprocess_conv.weight"); + + // The timestep embedder stays on the host — see the header. These are COPIES, + // so `release_host` does not take them: they are 18 MB at the shipped + // dimensions and they are what keeps `temb` bit-identical to the CPU arm. + // Their sizes were checked at the top of this function. + staged.host_time_embed.time_proj_weight = weights.time_proj_weight; + staged.host_time_embed.time_embed_linear_1_weight = weights.time_embed_linear_1_weight; + staged.host_time_embed.time_embed_linear_1_bias = weights.time_embed_linear_1_bias; + staged.host_time_embed.time_embed_linear_2_weight = weights.time_embed_linear_2_weight; + staged.host_time_embed.time_embed_linear_2_bias = weights.time_embed_linear_2_bias; + + backend.Synchronize(queue); + return staged; +} + +std::vector DitForwardDevice(vt::Queue& queue, const std::vector& latents, + int64_t length, const std::vector& condition, + double timestep, + const MiniMaxMusic3TransformerConfig& config, + const Music3DitDeviceWeights& weights) { + if (length <= 0) { + Fail("MiniMax-Music3 DiT: a window of " + std::to_string(length) + + " latent frames has nothing to denoise"); + } + const int64_t in_channels = config.in_channels; + const int64_t condition_dim = config.condition_dim; + const int64_t concat = config.concat_channels(); + const int64_t inner = config.inner_dim(); + const int64_t heads = config.num_attention_heads; + const int64_t head_dim = config.attention_head_dim; + const int64_t attn_inner = heads * head_dim; + const int64_t ff = config.ff_inner_dim; + const int64_t seq = length + 1; + if (static_cast(latents.size()) != in_channels * length) { + Fail("MiniMax-Music3 DiT: latents [in_channels, length] is " + + std::to_string(latents.size()) + " values, expected " + + std::to_string(in_channels * length)); + } + if (static_cast(condition.size()) != length * condition_dim) { + Fail("MiniMax-Music3 DiT: condition [length, condition_dim] is " + + std::to_string(condition.size()) + " values, expected " + + std::to_string(length * condition_dim)); + } + if (static_cast(weights.layers.size()) != config.num_layers) { + Fail("MiniMax-Music3 DiT: the staged weights carry " + + std::to_string(weights.layers.size()) + " blocks, the config declares " + + std::to_string(config.num_layers)); + } + + vt::Backend& backend = vt::GetBackend(queue.device.type); + Dev d{backend, queue}; + + // `cat((hidden_states, zeros_like(hidden_states), encoder_hidden_states.T))` + // (:218-219), built directly in the TRANSPOSED [length, concat] orientation. + // The middle block is a genuine ZERO PAD and not a second copy of the latents. + std::vector stacked_t(static_cast(length * concat), 0.0f); + for (int64_t t = 0; t < length; ++t) { + float* row = stacked_t.data() + t * concat; + for (int64_t c = 0; c < in_channels; ++c) { + row[c] = latents[static_cast(c * length + t)]; + } + for (int64_t c = 0; c < condition_dim; ++c) { + row[2 * in_channels + c] = condition[static_cast(t * condition_dim + c)]; + } + } + + DBuf stacked(d, DType::kF32, {length, concat}, stacked_t.data()); + // RESIDUAL 1x1 convolution (:220), as the transposed GEMM this file's header + // note derives. `pre` then holds conv(x) and the add makes it conv(x) + x. + DBuf pre(d, DType::kF32, {length, concat}); + vt::MatmulBT(queue, pre.t(), stacked.t(), weights.preprocess_conv_weight); + vt::Add(queue, pre.t(), pre.t(), stacked.t()); + + // The timestep embedding is PREPENDED as one extra token (:227) that the + // rotary sees and `proj_out` then drops (:236). `hidden` is allocated at the + // full [seq, inner] and `proj_in` writes STRAIGHT into rows 1..seq — a view, + // not a copy, so the projection lands where the block stack wants it. + DBuf hidden(d, DType::kF32, {seq, inner}); + Tensor hidden_tail = MakeTensor(static_cast(hidden.t().data) + inner, DType::kF32, + queue.device, {length, inner}); + vt::MatmulBT(queue, hidden_tail, pre.t(), weights.proj_in_weight); + + // Row 0: the timestep embedding, computed on the host through the reference's + // own helpers so it is bit-identical to the CPU arm (header rationale). + const std::vector temb = DitTimestepEmbedding( + FourierTimeEmbedding(timestep, weights.host_time_embed.time_proj_weight, + config.fourier_embedding_dim), + config, weights.host_time_embed); + if (static_cast(temb.size()) != inner) { + Fail("MiniMax-Music3 DiT: the timestep embedding is " + std::to_string(temb.size()) + + " values, expected inner_dim = " + std::to_string(inner)); + } + backend.Copy(queue, hidden.t().data, temb.data(), static_cast(inner) * sizeof(float)); + + // ── the rotary cache, in the layout vt::RopeFromCache reads ──────────────── + // That op indexes `cache[position * rotary_dim + pair]` for the cosine and + // `+ half + pair` for the sine, then computes x' = x*c - y*s, y' = x*s + y*c + // over the LEADING rotary_dim of each head — which is `ApplyPartialRotary` + // exactly (minimax_music3_acoustic.cpp:500-514), including the untouched tail. + // `BuildDitRotaryTables` returns cos/sin already duplicated across the two + // halves of the rotary window; the cache wants ONE half of each, so the first + // `half` columns of each table are what is packed here. + const int64_t rotary_dim = config.rotary_dim; + const int64_t half = rotary_dim / 2; + const DitRotaryTables tables = BuildDitRotaryTables(seq, rotary_dim, kDitRotaryTheta); + std::vector cache(static_cast(seq * rotary_dim)); + for (int64_t s = 0; s < seq; ++s) { + for (int64_t j = 0; j < half; ++j) { + cache[static_cast(s * rotary_dim + j)] = + tables.cos[static_cast(s * rotary_dim + j)]; + cache[static_cast(s * rotary_dim + half + j)] = + tables.sin[static_cast(s * rotary_dim + j)]; + } + } + std::vector positions_host(static_cast(seq)); + for (int64_t s = 0; s < seq; ++s) positions_host[static_cast(s)] = static_cast(s); + DBuf rope_cache(d, DType::kF32, {seq, rotary_dim}, cache.data()); + DBuf positions(d, DType::kI32, {seq}, positions_host.data()); + + vt::RopeArgs rope_args; + rope_args.rotary_dim = static_cast(rotary_dim); + rope_args.is_neox_style = true; // rotate_half over the rotary window + vt::LayerNormArgs norm_args; + norm_args.eps = static_cast(kDitLayerNormEps); + vt::AttentionCrossArgs attn_args; + attn_args.scale = static_cast(1.0 / std::sqrt(static_cast(head_dim))); + + // The reference's `Attention` helper takes NO mask (:97-103 dispatches with + // none), so every token attends to every token including the prepended + // timestep one. `vt::AttentionCross` with a null bias is that op; `vt:: + // Attention` is the CAUSAL one and would silently mask the future here. + for (int64_t l = 0; l < config.num_layers; ++l) { + const Music3DitDeviceLayer& layer = weights.layers[static_cast(l)]; + + DBuf normed(d, DType::kF32, {seq, inner}); + vt::LayerNorm(queue, normed.t(), hidden.t(), &layer.norm1_weight, &layer.norm1_bias, + norm_args); + DBuf qb(d, DType::kF32, {seq, attn_inner}); + DBuf kb(d, DType::kF32, {seq, attn_inner}); + DBuf vb(d, DType::kF32, {seq, attn_inner}); + LinearDev(queue, qb.t(), normed.t(), layer.to_q, nullptr); + LinearDev(queue, kb.t(), normed.t(), layer.to_k, nullptr); + LinearDev(queue, vb.t(), normed.t(), layer.to_v, nullptr); + + Tensor q3 = Reshape(qb.t(), {seq, heads, head_dim}); + Tensor k3 = Reshape(kb.t(), {seq, heads, head_dim}); + Tensor v3 = Reshape(vb.t(), {seq, heads, head_dim}); + vt::RopeFromCache(queue, q3, &k3, positions.t(), rope_cache.t(), rope_args); + + DBuf attended(d, DType::kF32, {seq, heads, head_dim}); + vt::AttentionCross(queue, attended.t(), q3, k3, v3, nullptr, attn_args); + Tensor attended2 = Reshape(attended.t(), {seq, attn_inner}); + DBuf projected(d, DType::kF32, {seq, inner}); + LinearDev(queue, projected.t(), attended2, layer.to_out, nullptr); + vt::Add(queue, hidden.t(), hidden.t(), projected.t()); + + DBuf normed2(d, DType::kF32, {seq, inner}); + vt::LayerNorm(queue, normed2.t(), hidden.t(), &layer.norm2_weight, &layer.norm2_bias, + norm_args); + DBuf gated(d, DType::kF32, {seq, 2 * ff}); + LinearDev(queue, gated.t(), normed2.t(), layer.ff_in_weight, &layer.ff_in_bias); + // `silu(first) * second` on the SWAPPED projection == upstream's + // `value * silu(gate)`. See StageMusic3DitWeights. + DBuf activated(d, DType::kF32, {seq, ff}); + vt::SiluAndMul(queue, activated.t(), gated.t()); + DBuf ff_out(d, DType::kF32, {seq, inner}); + LinearDev(queue, ff_out.t(), activated.t(), layer.ff_out_weight, &layer.ff_out_bias); + vt::Add(queue, hidden.t(), hidden.t(), ff_out.t()); + } + + // Drop the timestep token (the same [length, inner] view `proj_in` wrote), + // project, then the RESIDUAL 1x1 convolution (:238) as a GEMM again. + DBuf out_rows(d, DType::kF32, {length, in_channels}); + vt::MatmulBT(queue, out_rows.t(), hidden_tail, weights.proj_out_weight); + DBuf post(d, DType::kF32, {length, in_channels}); + vt::MatmulBT(queue, post.t(), out_rows.t(), weights.postprocess_conv_weight); + vt::Add(queue, out_rows.t(), out_rows.t(), post.t()); + + std::vector rows(static_cast(length * in_channels)); + backend.Copy(queue, rows.data(), out_rows.t().data, rows.size() * sizeof(float)); + backend.Synchronize(queue); + + // Back to the CHANNEL-MAJOR [in_channels, length] the caller and the vocoder + // both expect. + std::vector out(static_cast(in_channels * length)); + for (int64_t c = 0; c < in_channels; ++c) { + for (int64_t t = 0; t < length; ++t) { + out[static_cast(c * length + t)] = rows[static_cast(t * in_channels + c)]; + } + } + return out; +} + +} // namespace music3 +} // namespace models +} // namespace vllm diff --git a/src/vllm/model_executor/models/minimax_music3_speech.cpp b/src/vllm/model_executor/models/minimax_music3_speech.cpp index 0863a6353..8cc21cd84 100644 --- a/src/vllm/model_executor/models/minimax_music3_speech.cpp +++ b/src/vllm/model_executor/models/minimax_music3_speech.cpp @@ -187,12 +187,21 @@ std::vector> Music3DenoiseChunks(const std::vector& fr const MiniMaxMusic3Config& config, const Music3AcousticWeights& weights, const Music3DenoiseOptions& options, - const Music3NoiseSource& noise) { + const Music3NoiseSource& noise, + const Music3DenoiseDeviceArm& device_arm) { if (!noise) Fail("MiniMax-Music3: the denoise loop needs a noise source"); if (options.num_inference_steps <= 0) { Fail("MiniMax-Music3: `num_inference_steps` must be positive, got " + std::to_string(options.num_inference_steps)); } + // Half a device arm is a caller that believes it asked for the GPU and got the + // host loops. Refused by name rather than ignored, because the failure it + // otherwise produces is a correct song delivered thirty hours late. + if (device_arm.half_set()) { + Fail("MiniMax-Music3: the denoise device arm needs BOTH a queue and staged DiT weights; " + "got only the " + std::string(device_arm.queue != nullptr ? "queue" : "weights")); + } + const bool on_device = device_arm.engaged(); ConditionMixConfig mix; mix.condition_hidden_dim = config.condition_encoder.condition_hidden_dim; @@ -275,10 +284,19 @@ std::vector> Music3DenoiseChunks(const std::vector& fr BlendOverlap(latents, channels, length, noise_prompt, previous_latent, previous_length, overlap, time_value); } + // The ONLY line the device arm changes. Both CFG branches take the same + // arm — running one on each would make the guidance mix a comparison + // between two different numerics rather than between two conditionings. const std::vector conditional = - DitForward(latents, length, condition, time_value, config.transformer, weights.dit); + on_device ? DitForwardDevice(*device_arm.queue, latents, length, condition, time_value, + config.transformer, *device_arm.dit) + : DitForward(latents, length, condition, time_value, config.transformer, + weights.dit); const std::vector unconditional = - DitForward(latents, length, zero_condition, time_value, config.transformer, weights.dit); + on_device ? DitForwardDevice(*device_arm.queue, latents, length, zero_condition, + time_value, config.transformer, *device_arm.dit) + : DitForward(latents, length, zero_condition, time_value, config.transformer, + weights.dit); const std::vector velocity = ClassifierFreeGuidanceMix(conditional, unconditional, options.guidance_scale); latents = FlowMatchStep(latents, velocity, step, schedule); @@ -534,13 +552,40 @@ class Music3SpeechEngine final : public multimodal::SpeechEngine { (void)calls; // ── the ACOUSTIC half (before_denoise.py / denoise.py / decoders.py) ───── - const Music3AcousticWeights acoustic = Music3LoadAcousticWeights(paths_, config_); + // + // NON-const, because the device arm below STAGES OUT OF IT. The DiT is + // 9.7 GB of fp32 and Jetson Thor's ~122 GB is UNIFIED — host and device draw + // on one pool — so uploading while the host copy is still held is a real + // 19.4 GB peak on the only box this arm runs on, and that box reboots + // instead of OOM-killing (.agents/environment.md). `StageMusic3DitWeights` + // drops each host tensor as it lands, so the peak is one tensor over the + // 9.7 GB, not twice it. + Music3AcousticWeights acoustic = Music3LoadAcousticWeights(paths_, config_); Music3DenoiseOptions options; options.num_inference_steps = request.num_inference_steps; options.guidance_scale = request.guidance_scale; + + // Staged ONCE per request, outside every loop in `Music3DenoiseChunks`. A + // 45 s clip runs the DiT 660 times over 11 windows; a per-window upload + // would move 9.7 GB eleven times and a per-step one 660 times, either of + // which costs more than the compute it enables. + // + // Not staged in the CONSTRUCTOR, unlike the queue: the acoustic weights are + // deliberately loaded per request and released with the request, so that the + // 18.5 GB autoregressive half and the 10 GB acoustic half are never + // co-resident (upstream drives the same split by hand, encoders.py:302-309). + // Staging follows the weights, not the engine. + Music3DitDeviceWeights staged_dit; + Music3DenoiseDeviceArm arm; + if (queue_.device.type != vt::DeviceType::kCPU) { + staged_dit = StageMusic3DitWeights(queue_, config_.transformer, acoustic.dit, + /*release_host=*/true); + arm.queue = &queue_; + arm.dit = &staged_dit; + } const std::vector> chunks = Music3DenoiseChunks(frame_hiddens, frames, config_, acoustic, options, - Music3SeededNoise(request.seed)); + Music3SeededNoise(request.seed), arm); int64_t samples = 0; multimodal::SpeechResult out; diff --git a/tests/parity/test_minimax_music3_acoustic_real.cpp b/tests/parity/test_minimax_music3_acoustic_real.cpp index 2b5a55875..1253daa7f 100644 --- a/tests/parity/test_minimax_music3_acoustic_real.cpp +++ b/tests/parity/test_minimax_music3_acoustic_real.cpp @@ -86,6 +86,8 @@ // the checkpoint is present. #include +#include +#include #include #include #include @@ -99,7 +101,12 @@ #include "npy.h" #include "vllm/model_executor/model_loader/safetensors_reader.h" #include "vllm/model_executor/models/minimax_music3_acoustic.h" +#include "vllm/model_executor/models/minimax_music3_device.h" #include "vllm/model_executor/models/minimax_music3_loader.h" +#include "vllm/model_executor/models/minimax_music3_speech.h" // kMusic3SpeechFamily +#include "vllm/multimodal/speech_engine.h" // SpeechEngineDeviceType +#include "vt/backend.h" +#include "vt/device.h" namespace fs = std::filesystem; namespace m3 = vllm::models::music3; @@ -353,16 +360,61 @@ m3::DitWeights LoadDit(const vllm::MiniMaxMusic3TransformerConfig& config) { return m3::DitWeightsFromTensors(config, tensors); } +// WHERE this gate runs the 2.4B DiT. Default 0 = CPU, so an unset environment +// reproduces every number this file has ever printed. `VLLM_CPP_MUSIC3_DEVICE=1` +// runs the SAME comparison against the SAME goldens at the SAME bounds through +// `DitForwardDevice` (#672, spec §11.4) — no tolerance is widened for it, which +// is the claim that matters. +// +// Resolved through the SHARED `multimodal::SpeechEngineDeviceType` the engine +// itself calls, not a private copy: a gate that resolved the device its own way +// could pass while the engine bound a different one. +struct DitArm { + vt::Queue queue{}; + bool on_device = false; + std::string banner; +}; + +DitArm ResolveDitArm() { + DitArm arm; + const char* env = std::getenv("VLLM_CPP_MUSIC3_DEVICE"); + const int32_t sel = (env != nullptr && env[0] == '1') ? 1 : 0; + const vt::DeviceType type = + vllm::multimodal::SpeechEngineDeviceType(sel, m3::kMusic3SpeechFamily); + arm.on_device = type != vt::DeviceType::kCPU; + arm.queue = arm.on_device ? vt::GetBackend(type).CreateQueue() + : vt::Queue{vt::Device{vt::DeviceType::kCPU, 0}, nullptr}; + // ONE std::string. Built as a MESSAGE chain, both fields collapse inside + // doctest and a CPU run prints the device arm's banner — the instrument defect + // #672 already hit once, where the CPU numbers would have been recorded as the + // device arm's had the line not been read. + arm.banner = std::string("music3 acoustic real: the 2.4B DiT ran on '") + + vt::DeviceTypeName(arm.queue.device.type) + "' (VLLM_CPP_MUSIC3_DEVICE=" + + (env == nullptr ? std::string("unset") : std::string(env)) + ")"; + return arm; +} + // One guided velocity: the conditional and the zero-conditioned forward, mixed. +// Both branches take the SAME arm — running one on each would make the guidance +// mix a comparison between two numerics rather than between two conditionings. std::vector GuidedVelocity(const std::vector& latents, const std::vector& condition, double timestep, const vllm::MiniMaxMusic3TransformerConfig& config, - const m3::DitWeights& weights) { + const m3::DitWeights& weights, DitArm* arm, + const m3::Music3DitDeviceWeights* staged) { + const std::vector zeros(condition.size(), 0.0f); + if (arm != nullptr && arm->on_device) { + REQUIRE(staged != nullptr); + const std::vector conditional = m3::DitForwardDevice( + arm->queue, latents, kLatentLength, condition, timestep, config, *staged); + const std::vector unconditional = m3::DitForwardDevice( + arm->queue, latents, kLatentLength, zeros, timestep, config, *staged); + return m3::ClassifierFreeGuidanceMix(conditional, unconditional, m3::kDitGuidanceScale); + } const std::vector conditional = m3::DitForward(latents, kLatentLength, condition, timestep, config, weights); const std::vector unconditional = - m3::DitForward(latents, kLatentLength, std::vector(condition.size(), 0.0f), - timestep, config, weights); + m3::DitForward(latents, kLatentLength, zeros, timestep, config, weights); return m3::ClassifierFreeGuidanceMix(conditional, unconditional, m3::kDitGuidanceScale); } @@ -492,17 +544,59 @@ TEST_CASE("music3 acoustic real: the DiT reproduces the capture's guided velocit std::vector shape; const std::vector condition = LoadF32Npy("condition_chunk0.npy", &shape); - const m3::DitWeights weights = LoadDit(config.transformer); + m3::DitWeights weights = LoadDit(config.transformer); + + DitArm arm = ResolveDitArm(); + MESSAGE(arm.banner); + // `release_host` FALSE: this is a gate, and both arms must remain runnable in + // one process. The SERVING path is what passes true. + // + // TIMED, because this staging is the thing the speed claim is ABOUT. If the + // weights were re-uploaded per forward, the repeat sweep below would show it + // as slope rather than as intercept. + m3::Music3DitDeviceWeights staged; + const auto stage_t0 = std::chrono::steady_clock::now(); + if (arm.on_device) { + staged = m3::StageMusic3DitWeights(arm.queue, config.transformer, weights, + /*release_host=*/false); + CHECK(static_cast(staged.layers.size()) == config.transformer.num_layers); + } + const double stage_s = + std::chrono::duration(std::chrono::steady_clock::now() - stage_t0).count(); + // Same one-string rule as DIT_TIMING below: the first revision printed + // `dit staging: 9.3e-08 s (1)` because the `const char*` arm tag went to + // doctest's bool overload. + const std::string staging_line = std::string("dit staging: ") + std::to_string(stage_s) + + " s (" + + (arm.on_device ? "device upload" : "host, no-op") + ")"; + MESSAGE(staging_line); + + // `VLLM_CPP_MUSIC3_DIT_REPEAT=R` runs the guided velocity R times per timestep + // instead of once, so a run at R and a run at R' give TWO POINTS on the same + // binary and the same weights. The slope is the per-forward cost and the + // intercept is everything paid once — which is the only way to state "the + // weights are staged once" as a MEASUREMENT rather than as a claim about the + // code. Default 1, so an unset environment runs exactly what it always did. + int64_t repeats = 1; + if (const char* r = std::getenv("VLLM_CPP_MUSIC3_DIT_REPEAT")) { + repeats = std::max(1, std::atoll(r)); + } int64_t total_outside = 0; + int64_t forwards = 0; + const auto t0 = std::chrono::steady_clock::now(); for (int64_t index : {static_cast(0), kDenoiseSteps - 1}) { const std::string tag = index == 0 ? "first" : "last"; const double timestep = index == 0 ? kFirstTimestep : kLastTimestep; CAPTURE(tag); const std::vector latents = LoadF32Npy("denoise_" + tag + "_sample_in.npy", &shape); const std::vector want = LoadF32Npy("denoise_" + tag + "_velocity.npy", &shape); - const std::vector got = - GuidedVelocity(latents, condition, timestep, config.transformer, weights); + std::vector got; + for (int64_t r = 0; r < repeats; ++r) { + got = GuidedVelocity(latents, condition, timestep, config.transformer, weights, &arm, + &staged); + forwards += 2; // one guided velocity is the conditional AND the unconditional forward + } const Report report = Compare(got, want, kDitRelTol, kDitAbsFloor); ReportInto("dit guided velocity " + tag, report); CHECK(report.compared == kLatentChannels * kLatentLength); @@ -510,6 +604,21 @@ TEST_CASE("music3 acoustic real: the DiT reproduces the capture's guided velocit CHECK(report.mean_abs < kDitMeanAbsTol); total_outside += report.outside; } + const double loop_s = std::chrono::duration(std::chrono::steady_clock::now() - t0).count(); + // ONE line, ONE std::string — and this is not a style preference, it is the + // defect this row hit twice. A `const char*` handed to doctest's MESSAGE chain + // converts to BOOL and prints `1`: the first revision of this line reported + // `arm=1` on the CPU run, next to numbers that were themselves correct. That + // is #672's own recorded instrument defect (§11.5) reappearing in a new line, + // and the fix is the one that worked there: assemble the string, then print it. + std::string timing = "DIT_TIMING arm="; + timing += vt::DeviceTypeName(arm.queue.device.type); + timing += " repeats=" + std::to_string(repeats); + timing += " forwards=" + std::to_string(forwards); + timing += " loop_s=" + std::to_string(loop_s); + timing += " stage_s=" + std::to_string(stage_s); + timing += " per_forward_s=" + std::to_string(loop_s / static_cast(forwards)); + MESSAGE(timing); CHECK(total_outside == 0); } @@ -520,13 +629,26 @@ TEST_CASE("music3 acoustic real: the DiT's two guidance branches are different t std::vector shape; const std::vector condition = LoadF32Npy("condition_chunk0.npy", &shape); const std::vector latents = LoadF32Npy("denoise_first_sample_in.npy", &shape); - const m3::DitWeights weights = LoadDit(config.transformer); - - const std::vector conditional = m3::DitForward( - latents, kLatentLength, condition, kFirstTimestep, config.transformer, weights); + m3::DitWeights weights = LoadDit(config.transformer); + + DitArm arm = ResolveDitArm(); + MESSAGE(arm.banner); + m3::Music3DitDeviceWeights staged; + if (arm.on_device) { + staged = m3::StageMusic3DitWeights(arm.queue, config.transformer, weights, + /*release_host=*/false); + } + const std::vector zeros(condition.size(), 0.0f); + const std::vector conditional = + arm.on_device ? m3::DitForwardDevice(arm.queue, latents, kLatentLength, condition, + kFirstTimestep, config.transformer, staged) + : m3::DitForward(latents, kLatentLength, condition, kFirstTimestep, + config.transformer, weights); const std::vector unconditional = - m3::DitForward(latents, kLatentLength, std::vector(condition.size(), 0.0f), - kFirstTimestep, config.transformer, weights); + arm.on_device ? m3::DitForwardDevice(arm.queue, latents, kLatentLength, zeros, + kFirstTimestep, config.transformer, staged) + : m3::DitForward(latents, kLatentLength, zeros, kFirstTimestep, + config.transformer, weights); const int64_t identical = CountIdentical(conditional, unconditional); // A DiT that dropped its conditioning would still pass the velocity gate for // any guidance scale if the two branches were equal, because the mix would diff --git a/tests/vllm/models/test_minimax_music3_acoustic.cpp b/tests/vllm/models/test_minimax_music3_acoustic.cpp index 51bc8a54f..dc5d4b1ca 100644 --- a/tests/vllm/models/test_minimax_music3_acoustic.cpp +++ b/tests/vllm/models/test_minimax_music3_acoustic.cpp @@ -29,7 +29,9 @@ // (AGENTS.md; spec §5). #include +#include #include +#include #include #include #include @@ -37,6 +39,9 @@ #include "minimax_music3_acoustic_goldens.inc" #include "vllm/model_executor/models/minimax_music3_acoustic.h" +#include "vllm/model_executor/models/minimax_music3_device.h" +#include "vt/backend.h" +#include "vt/device.h" namespace { @@ -715,6 +720,239 @@ TEST_CASE("music3 acoustic: the DiT refuses every wrong-shaped input by name") { std::runtime_error); } +// --------------------------------------------------------------------------- +// The DEVICE-RESIDENT DiT (#672, spec §11.4) +// +// THE TOLERANCE, AND THE CONTROL THAT JUSTIFIES IT. Nothing below is a new +// bound. `DitForwardDevice` is checked against the SAME upstream float32 +// goldens, through the SAME `ExpectClose`, at the SAME kRelTol/kAbsFloor as +// `DitForward` — because the question that matters is not "do the two arms +// agree with each other" (a shared-helper comparison proves consistency, not +// correctness) but "is the device arm as close to UPSTREAM as the host arm is". +// +// Each case therefore reports BOTH distances to the golden, host and device, on +// the identical input. The host arm's distance is the measured control: it was +// accepted with these goldens when the bound was set, so a device arm whose +// distance is at or below it is inside a spread that already exists rather than +// inside one this row widened. No tolerance is relaxed here, and the two +// mutation cases below prove the bound still discriminates. +// --------------------------------------------------------------------------- + +namespace { + +// Both arms, same inputs, both against upstream. Returns nothing; every number +// is asserted or printed, and the CASE count is what the suite reports. +void CheckDeviceDit(vt::Queue& q, const char* arm) { + const vllm::MiniMaxMusic3TransformerConfig config = DitConfig(); + const size_t latent_count = + static_cast(config.in_channels * vllm_test::kMusic3DitLength); + const size_t condition_count = + static_cast(vllm_test::kMusic3DitLength * config.condition_dim); + const std::vector latents = ToVector(vllm_test::kMusic3DitLatents, latent_count); + const std::vector condition = ToVector(vllm_test::kMusic3DitCondition, condition_count); + const std::vector zeros(condition_count, 0.0f); + + // `release_host` FALSE here on purpose: this gate needs the host arm too, and + // the serving path is the caller that passes true. + m3::DitWeights host = DitWeights(); + const m3::Music3DitDeviceWeights staged = + m3::StageMusic3DitWeights(q, config, host, /*release_host=*/false); + REQUIRE(staged.layers.size() == static_cast(config.num_layers)); + + const std::vector dev_cond = m3::DitForwardDevice( + q, latents, vllm_test::kMusic3DitLength, condition, vllm_test::kMusic3DitTimestep, config, + staged); + const std::vector host_cond = + m3::DitForward(latents, vllm_test::kMusic3DitLength, condition, + vllm_test::kMusic3DitTimestep, config, host); + const double dev_worst = + ExpectClose(dev_cond, vllm_test::kMusic3DitOut, latent_count, + (std::string(arm) + " dit conditional (device)").c_str()); + const double host_worst = + ExpectClose(host_cond, vllm_test::kMusic3DitOut, latent_count, + (std::string(arm) + " dit conditional (host control)").c_str()); + MESSAGE(std::string(arm) << " dit conditional: " << latent_count + << " values; worst |device-upstream| = " << dev_worst + << ", worst |host-upstream| = " << host_worst + << " (bound " << kRelTol << " rel / " << kAbsFloor << " abs)"); + + const std::vector dev_uncond = + m3::DitForwardDevice(q, latents, vllm_test::kMusic3DitLength, zeros, + vllm_test::kMusic3DitTimestep, config, staged); + const double dev_worst_u = + ExpectClose(dev_uncond, vllm_test::kMusic3DitOutUncond, latent_count, + (std::string(arm) + " dit unconditional (device)").c_str()); + MESSAGE(std::string(arm) << " dit unconditional: " << latent_count + << " values; worst |device-upstream| = " << dev_worst_u); + + // The two branches must be DIFFERENT tensors on the device arm too: a forward + // that dropped its condition would match the conditional golden and this one + // identically, and both ExpectClose calls above would still be green. + size_t differing = 0; + for (size_t i = 0; i < latent_count; ++i) { + if (dev_cond[i] != dev_uncond[i]) ++differing; + } + MESSAGE(std::string(arm) << " dit branches: " << differing << " of " << latent_count + << " values differ between conditional and unconditional"); + CHECK(differing == latent_count); +} + +} // namespace + +TEST_CASE("music3 acoustic: the DEVICE-resident DiT matches upstream (CPU backend)") { + vt::Queue q{vt::Device{}, nullptr}; + CheckDeviceDit(q, "cpu-backend"); +} + +TEST_CASE("music3 acoustic: the DEVICE-resident DiT matches upstream on CUDA") { + vt::Backend* cuda = nullptr; + try { + cuda = &vt::GetBackend(vt::DeviceType::kCUDA); + } catch (...) { + MESSAGE("SKIP: no CUDA backend registered (this is a CPU-only build)"); + return; + } + vt::Queue q = cuda->CreateQueue(); + CheckDeviceDit(q, "cuda"); +} + +TEST_CASE("music3 acoustic: the ff_in HALF SWAP is load-bearing, and the gate sees it") { + // The device arm computes `value * silu(gate)` by handing vt::SiluAndMul — which + // computes `silu(first) * second` — a projection whose two ROW BLOCKS were + // exchanged at stage time. That exchange is an identity ONLY if it is applied + // exactly once. Pre-swapping the host weights makes the stage-time swap undo + // the test's, so the forward computes `silu(value) * gate` instead: the wrong + // network, same shapes, same finiteness. + // + // This is the mutation that proves the bound above discriminates. If the + // forward were routing `silu`/`mul` the other way round the RIGHT case would + // fail and this one would pass, so the pair pins the direction rather than + // just the magnitude. + const vllm::MiniMaxMusic3TransformerConfig config = DitConfig(); + const size_t latent_count = + static_cast(config.in_channels * vllm_test::kMusic3DitLength); + const std::vector latents = ToVector(vllm_test::kMusic3DitLatents, latent_count); + const std::vector condition = ToVector( + vllm_test::kMusic3DitCondition, + static_cast(vllm_test::kMusic3DitLength * config.condition_dim)); + + m3::DitWeights mutated = DitWeights(); + const size_t ff = static_cast(config.ff_inner_dim); + const size_t inner = static_cast(config.inner_dim()); + for (m3::DitLayerWeights& layer : mutated.layers) { + std::vector w(layer.ff_in_weight.size()); + std::copy(layer.ff_in_weight.begin() + static_cast(ff * inner), + layer.ff_in_weight.end(), w.begin()); + std::copy(layer.ff_in_weight.begin(), + layer.ff_in_weight.begin() + static_cast(ff * inner), + w.begin() + static_cast(ff * inner)); + layer.ff_in_weight = w; + std::vector b(layer.ff_in_bias.size()); + std::copy(layer.ff_in_bias.begin() + static_cast(ff), layer.ff_in_bias.end(), + b.begin()); + std::copy(layer.ff_in_bias.begin(), layer.ff_in_bias.begin() + static_cast(ff), + b.begin() + static_cast(ff)); + layer.ff_in_bias = b; + } + + vt::Queue q{vt::Device{}, nullptr}; + const m3::Music3DitDeviceWeights staged = + m3::StageMusic3DitWeights(q, config, mutated, /*release_host=*/false); + const std::vector out = m3::DitForwardDevice( + q, latents, vllm_test::kMusic3DitLength, condition, vllm_test::kMusic3DitTimestep, config, + staged); + + size_t outside = 0; + double worst = 0.0; + for (size_t i = 0; i < latent_count; ++i) { + const double a = out[i], b = vllm_test::kMusic3DitOut[i]; + const double bound = std::max(kAbsFloor, kRelTol * std::max(std::abs(a), std::abs(b))); + if (!(std::abs(a - b) <= bound)) ++outside; + worst = std::max(worst, std::abs(a - b)); + } + MESSAGE("half-swap mutation: " << outside << " of " << latent_count + << " values outside the bound, worst |diff| = " << worst); + // A defect that moves values by O(1) must move essentially all of them. This + // is the negative control for every ExpectClose above. + CHECK(outside > latent_count / 2); +} + +TEST_CASE("music3 acoustic: the DEVICE DiT refuses every wrong-shaped input by name") { + const vllm::MiniMaxMusic3TransformerConfig config = DitConfig(); + const size_t latent_count = + static_cast(config.in_channels * vllm_test::kMusic3DitLength); + const std::vector latents = ToVector(vllm_test::kMusic3DitLatents, latent_count); + const std::vector condition = ToVector( + vllm_test::kMusic3DitCondition, + static_cast(vllm_test::kMusic3DitLength * config.condition_dim)); + vt::Queue q{vt::Device{}, nullptr}; + + m3::DitWeights host = DitWeights(); + const m3::Music3DitDeviceWeights staged = + m3::StageMusic3DitWeights(q, config, host, /*release_host=*/false); + CHECK_THROWS_AS(m3::DitForwardDevice(q, {1.0f}, vllm_test::kMusic3DitLength, condition, 0.25, + config, staged), + std::runtime_error); + CHECK_THROWS_AS(m3::DitForwardDevice(q, latents, vllm_test::kMusic3DitLength, {1.0f}, 0.25, + config, staged), + std::runtime_error); + CHECK_THROWS_AS(m3::DitForwardDevice(q, latents, 0, condition, 0.25, config, staged), + std::runtime_error); + + // A mis-sized weight is refused at STAGE time — before 9.7 GB moves at real + // dimensions — rather than 36 layers into the first of 660 forwards. + m3::DitWeights broken = DitWeights(); + broken.layers.pop_back(); + CHECK_THROWS_AS(m3::StageMusic3DitWeights(q, config, broken, /*release_host=*/false), + std::runtime_error); + m3::DitWeights short_proj = DitWeights(); + short_proj.proj_in_weight.pop_back(); + CHECK_THROWS_AS(m3::StageMusic3DitWeights(q, config, short_proj, /*release_host=*/false), + std::runtime_error); +} + +TEST_CASE("music3 acoustic: release_host EMPTIES the source, and the staged copy still runs") { + // "Device-resident" has to mean the host copy is GONE, not that a second copy + // exists. On Jetson Thor the two pools are one pool: holding both is a real + // 19.4 GB peak on a box that reboots instead of OOM-killing. + const vllm::MiniMaxMusic3TransformerConfig config = DitConfig(); + const size_t latent_count = + static_cast(config.in_channels * vllm_test::kMusic3DitLength); + const std::vector latents = ToVector(vllm_test::kMusic3DitLatents, latent_count); + const std::vector condition = ToVector( + vllm_test::kMusic3DitCondition, + static_cast(vllm_test::kMusic3DitLength * config.condition_dim)); + + vt::Queue q{vt::Device{}, nullptr}; + m3::DitWeights host = DitWeights(); + const m3::Music3DitDeviceWeights staged = + m3::StageMusic3DitWeights(q, config, host, /*release_host=*/true); + + size_t emptied = 0, total = 0; + for (const m3::DitLayerWeights& layer : host.layers) { + for (const std::vector* v : + {&layer.to_q, &layer.to_k, &layer.to_v, &layer.to_out, &layer.ff_in_weight, + &layer.ff_out_weight}) { + ++total; + if (v->empty() && v->capacity() == 0) ++emptied; + } + } + MESSAGE("release_host: " << emptied << " of " << total + << " per-layer host projections released (empty AND zero capacity)"); + CHECK(emptied == total); + // The time embedder is the ONE thing deliberately kept — it runs on the host + // so that `temb` stays bit-identical to the CPU arm. + CHECK(staged.host_time_embed.time_embed_linear_2_weight.size() == + static_cast(config.inner_dim() * config.inner_dim())); + + // And the staged copy is intact: a released host buffer that had been uploaded + // without a synchronize would read as garbage here rather than as the golden. + const std::vector out = m3::DitForwardDevice( + q, latents, vllm_test::kMusic3DitLength, condition, vllm_test::kMusic3DitTimestep, config, + staged); + ExpectClose(out, vllm_test::kMusic3DitOut, latent_count, "dit after release_host"); +} + // --------------------------------------------------------------------------- // W5 — the vocoder // --------------------------------------------------------------------------- From fdeea6d528a61c1c51481ec03175cbd88e80531e Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Mon, 17 Aug 2026 13:12:06 +0000 Subject: [PATCH 2/2] record(MODEL-MUSIC-MUSIC3): four corrections a fresh review found in this row's own claims (#672, #1131) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fresh review of #1121 CONFIRMED all four load-bearing claims -- zero source diff in the four CPU-carrying files, weights staged once, the device/host ratio, and that no tolerance was widened. It confirmed the first by rebuilding the full-scale CPU arm on a THIRD architecture (x86-64, its own build, real 9.7 GB checkpoint) and reproducing Thor's recorded numbers VALUE FOR VALUE. It confirmed the second more strongly than the arithmetic did: `DitForwardDevice` contains no weight upload at all, so there is nothing for a per-step upload to hide in. It also found four things wrong in the record, and they are corrected here rather than argued with. Three of the four are the COORDINATOR's, introduced while resolving a spec conflict, not the implementer's. THE TIMING BRACKET WAS OVERSTATED. Both the spec and benchmark-record claimed the timer brackets only the forward loop, with "the golden reads ... outside it". They are not: four `LoadF32Npy`, two `Compare` and two `ReportInto` sit INSIDE the t0/loop_s bracket (test_minimax_music3_acoustic_real.cpp:592-606). The direction matters and is stated: this INFLATES the intercept and makes the per-forward number slower than the pure forward, so the headline ratio is conservative rather than inflated. But a reader checking the intercept against the fit would have been misled, so the sentence is fixed in both places. THE RANGE'S LOW END WAS ARITHMETICALLY WRONG. "between 1102x and 1201x" -- 1102x is the ratio at the FASTEST R=1 point (0.185970 s). The range's slow end is 0.187269 s and 204.954646/0.187269 = 1094x. Now 1094x, with the reason. THE DOC CROSS-REFERENCES POINTED AT THE WRONG SECTION. STATUS.md and FEATURES.md both said the DiT arm is "§13". §13 is the vocoder ConvTranspose1d/Conv1d row; the DiT is §14. That is my renumbering during the merge-conflict resolution, uncorrected in the two documents that cite it. THE MERGE COMMIT CARRIED NO TRAILERS. `check-commit-trailers.py` fails on it, and ci.yml:604 runs exactly that over PR_BASE..PR_HEAD with no merge-commit exemption -- so the PR's own gate would have gone red on a commit I authored with `--no-edit`. Amended. This is the second time in this campaign an integration merge shipped bare; the lesson is that `git merge --no-edit` is never acceptable in this repo. AND ONE REAL COVERAGE HOLE, FILED AS #1131 AND RECORDED IN THE SPEC. The device arm's kernels and staging are gated -- eight mutations against them go RED -- but its PRODUCTION SWITCH is not. Setting `on_device = false` in `Music3DenoiseChunks`, or disabling the half-set refusal, leaves every suite GREEN. A change that silently stopped the DiT reaching the device would be invisible and the arm would run on the host with every number still looking right: the same shape as the dequant-fallback and mute-skip traps this row has already hit. Closing it needs an assertion that the device path was TAKEN -- invocation count or resident dtype -- because the two arms agree numerically by design and output equality cannot distinguish them. Not fixed here, routed instead: refusal messages naming upstream diffusers module paths rather than the checkpoint's own tensor names; §14.5 sitting after §14.7; and the fourth independent record of the same missing shared-layer seam arm (f32 + bias + up-front residency), which is a shared-layer row rather than four notes. Issue: #672, #1131 FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --- .agents/benchmark-record.md | 7 +++++-- .agents/specs/minimax-music3.md | 28 ++++++++++++++++++++++++---- docs/BENCHMARKS.md | 2 +- docs/FEATURES.md | 2 +- docs/STATUS.md | 2 +- 5 files changed, 32 insertions(+), 9 deletions(-) diff --git a/.agents/benchmark-record.md b/.agents/benchmark-record.md index 82ce3c448..72283f556 100644 --- a/.agents/benchmark-record.md +++ b/.agents/benchmark-record.md @@ -22410,8 +22410,11 @@ and single commands should go through `rc run --max-runtime` instead. `VLLM_CPP_MUSIC3_DIT_REPEAT=R` runs the guided velocity R times per timestep in `tests/parity/test_minimax_music3_acoustic_real.cpp`. The timer brackets ONLY -that loop: the 9.7 GB checkpoint load, the golden reads and the weight staging -are all outside it, and staging is timed separately. One guided velocity is TWO +that loop: the 9.7 GB checkpoint load and the weight staging are outside it, and +staging is timed separately. NOTE, corrected in fresh review: the golden reads +(4x `LoadF32Npy`, 2x `Compare`, 2x `ReportInto`) are INSIDE the bracket +(`test_minimax_music3_acoustic_real.cpp:592-606`), which inflates the intercept +and makes the reported per-forward number conservative, not inflated. One guided velocity is TWO DiT forwards (the conditional and the unconditional CFG branch). | arm | repeats | forwards | loop | per forward | staging | box load | diff --git a/.agents/specs/minimax-music3.md b/.agents/specs/minimax-music3.md index 63f675608..e5898e495 100644 --- a/.agents/specs/minimax-music3.md +++ b/.agents/specs/minimax-music3.md @@ -1468,6 +1468,16 @@ its own gate, and it took `vocoder1d::Conv1d` with it. provider, and every `vocoder1d` consumer routes through them. The DiT row closed in §14. +**A coverage hole in this row, found by fresh review and filed as #1131.** The +DiT device arm's kernels and staging are gated — eight mutations against them go +RED — but its **production switch is not**. Setting `on_device = false` in +`Music3DenoiseChunks`, or disabling the half-set refusal, leaves every suite +GREEN. A change that silently stopped the DiT reaching the device would be +invisible, and the arm would run on the host with every number still looking +right. The gate that closes it must assert the device path was TAKEN (invocation +count or resident dtype), not merely that outputs agree — the two arms agree +numerically by design. + **Only the depth decoder remains, and its "blocked on" entry above is now known to be WRONG in one respect**, corrected in §14.5: it is not "nothing but the work". The shipped depth decoder runs `ArCompute::kBFloat16`, which rounds the @@ -2388,8 +2398,16 @@ committed inputs on both arms; the arms never overlapped. **What is timed is the DiT and only the DiT.** `VLLM_CPP_MUSIC3_DIT_REPEAT=R` makes the gate run its guided velocity R times per timestep instead of once, and -the timer brackets ONLY that loop — the 9.7 GB checkpoint load, the golden reads -and the staging are all outside it, and the staging is timed separately. +the timer brackets that loop — the 9.7 GB checkpoint load and the weight staging +are outside it, and the staging is timed separately. + +**Corrected in fresh review:** an earlier revision of this paragraph also claimed +the GOLDEN READS were outside the bracket. They are not — four `LoadF32Npy` +calls, two `Compare` and two `ReportInto` sit INSIDE the `t0`/`loop_s` bracket +(`test_minimax_music3_acoustic_real.cpp:592-606`). That inflates the intercept and +makes the per-forward number SLOWER than the pure forward, so the headline ratio +is conservative rather than inflated — but the sentence was wrong as written, and +a reader checking the intercept against the fit would have been misled. | arm | repeats | forwards | loop | per forward | staging | box load | |---|---|---|---|---|---|---| @@ -2401,8 +2419,10 @@ and the staging are all outside it, and the staging is timed separately. | CUDA (`device 1`) | 1 | 4 | **0.743881 s** | **0.185970 s** | 0.612787 s | 4.44 | **Per DiT forward at the capture's geometry (latent length 86, seq 87): -204.955 s on the host, 0.1706-0.1873 s on the device — between 1102x and -1201x.** The two-point fit over the device arm's 4- and 12-forward runs gives +204.955 s on the host, 0.1706-0.1873 s on the device — between 1094x and +1201x.** (An earlier revision wrote the low end as 1102x. That is the ratio at +the FASTEST R=1 point, 0.185970 s; the range's slow end is 0.187269 s, and +204.954646 / 0.187269 = 1094x. Caught in fresh review.) The two-point fit over the device arm's 4- and 12-forward runs gives slope = 0.170607 s per forward intercept = 0.063012 s diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index cf9515a28..1665a7414 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -488,7 +488,7 @@ built on it rather than keeping the flattering one. | LTX-2.5 axes | Speed `PENDING` (vllm-omni#6066 has no native 2.5), binding oracle too. **SIZE: 704x448/25f and 448x256/25f both COMPLETE on GB10 (4231 s, 3085 s)**; one run each, contended box, no oracle, no ceiling (#1088) | Wall is NOT the VAE decode after #1041/#1009: a ~1731 s serial phase FLAT in resolution is 57-66% (#1087). ~59 GiB cliff did NOT recur (floor 38.9 GiB). 2 baselines UNRESOLVED (lock). PROMPTED real-ckpt render OWED | | MiniMax-Music3 (`MiniMaxMusic3ForConditionalGeneration`) | **Every axis vs the reference stays `PENDING`.** A PARTIAL device arm now exists (#672): the 8.6B LM and the 2.4B fp32 DiT run on the accelerator, so the rows below are internal two-arm numbers and NOT parity ratios | Denominator: SGLang-Omni `748a0b43` in its production configuration (both CUDA graphs, compiled DIT and DAV, batched seeded sampling) | | MiniMax-Music3 device arm, Jetson Thor sm_110 (#672) | `--device 1` vs `--device 0`, same request/seed, idle box: 2 AR frames **846.6 vs 835.1 s (1.014x SLOWER)**; 10 frames **1430.4 vs 1512.1 s (0.946x)**. Fit: **-11.65 s/frame, +34.8 s fixed** | A third duration (the fit has no residual), and moving the depth decoder + DiT + vocoder, which are 5 of 6 stages and still host scalar loops | -| MiniMax-Music3 DiT device arm, `thor:gpu0` sm_110 (#672) | Per DiT forward at the capture's geometry, same binary/weights/inputs, idle box: **204.955 s host vs 0.186 s device, 1102x** (1201x fitted). Weights staged ONCE (0.61 s; loop intercept 0.063 s). Whole process 3.5-4.5x | e2e song pair NOT runnable (host DiT alone ~37.6 h at 30 steps). Depth decoder/condition mix (bf16-storage), vocoder (no `ConvTranspose1d`) still host. Detail: benchmark-record | +| MiniMax-Music3 DiT device arm, `thor:gpu0` sm_110 (#672) | Per DiT forward at the capture's geometry, same binary/weights/inputs, idle box: **204.955 s host vs 0.186-0.187 s device, 1094-1102x** (1201x fitted). Staged ONCE (0.61 s; loop intercept 0.063 s). Whole process 3.5-4.5x | e2e song pair NOT runnable (host DiT alone ~37.6 h at 30 steps). Depth decoder/condition mix (bf16-storage), vocoder (no `ConvTranspose1d`) still host. Detail: benchmark-record | | MiniMax-Music3 CPU host kernels, x86-64 20-core (#672) | KERNEL A/B at the vocoder's real geometry, min of 5 interleaved rounds: convolution chain **13.36 -> 1.25 s, 10.7x**; `Conv1d` 12.03x, `LinearNoBias` 10.88x. Output fingerprints IDENTICAL on both arms | e2e pair VOID (cold CIFS cache; a foreign `ctest` at load 76.6) and re-running. Stages 0/1 only ~2x: the pivot trades WEIGHT locality for accumulator locality. Detail: benchmark-record | | MiniMax-H3 render coherence (`row/H3-RENDER-CLOSE` #77) | **CLOSED: a COHERENT scene on GB10.** #70/#74 white was wrong-PARTITION usage (t2va on the ref2va ckpt); t2va on the FL2VA GGUF renders a prompt-matched orange cat (adj-cos 0.95 vs 0.06, no patch-grid) | Verified first: t2va inputs byte-exact vs upstream; CUDA device==host at seq 1920. Follow-up `H3-TASK-PARTITION-GUARD`: the task/partition mismatch now RAISES 1:1 with `_resolve_task` (spec §8.6-8.7) | | MiniMax-H3 image conditioning (`row/H3-CONDITIONED-E2E`, `row/H3-VISION-SCATTER`, `row/H3-REF2VA-ASSEMBLY`) | **fl2va COHERENT; ref2va assembly bug FIXED+gated.** vision→cond scatter gated; ref2va block-dim double-division fixed + RED-first gated (128 vs 512) + a permanent ref2va DiT-forward rung (§8.10) | grid RE-ATTRIBUTED: with the fix ref2va grids in fp4 AND bf16, and t2va with no refs on the ref2va NVFP4 also grids while FL2VA-GGUF renders, so it is the **NVFP4 checkpoint/loader**, NOT assembly/fp4 (§8.10) | diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 6238b43b4..82aafe27e 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -167,7 +167,7 @@ in `ltx2_text_encoder.cpp` is the call that would have to change. | Whisper audio encoder | openai/whisper-small; whisper-large-v3 (Voxtral cfg) | encoder tower 77/77; large-v3 tower 203/203 | pending | | MiniMax-H3 DiT (`MiniMaxH3DiTModel`, vllm-omni lane) | MiniMax-H3 (33.1B video+audio) | portable 79/79; all three modalities COHERENT on Q4_K_M (§8.20); PRUNED ckpts run, Q8_0 seam 0.9941 (§8.21); ref2va grid was NVFP4 quant error, §8.9 REFUTED; GGUF/NVFP4/bf16 shards stream | FP4/Marlin landed; speed pending; no bf16 render yet. Render from the Q4_K_M GGUF, not the NVFP4 arm. Krea 2 text-to-image (roadmap C11) is scoped to reuse these DiT seams | | LTX-2.5 DiT (`LTX2VideoTransformer3DModel`, Lightricks lane) | LTX-2.5 (21.00B video+audio) | `SPIKE`. DiT, VAEs+ENCs, cond, pipeline, quant loaders gated, reduced dims. Prompt AdaLN host+dev; Gemma-4->xattn FIXTURE-gated. Img chain PPM->resize->encode->place->noise. Temporal x2 ups gated, UNDRIVEN. Render OWED | `ltx-2.5`/`ltx2-gen`. ~29 GB NVFP4/GB10, FP8 ~44 GB, +24 GB tower. FP8/torchao/NVFP4; kf abs-pos ported; BOTH DiTs load, NO `allow_unported`. IMG+LAST kf SERVED `crf=0`, A2V WAV+LoRA; DiffVAE/ref refused. PENDING | -| MiniMax-Music3 (`MiniMaxMusic3ForConditionalGeneration`, diffusers lane) | MiniMax-Music3 (8.6B Qwen3 LLM + 0.646B RVQ decoder + 2.4B fp32 DiT + DAC Flow-VAE); diffusers arm, ~28.5 GB | `ACTIVE`. Loader 1413/1413; AR, acoustic and the 8.6B LM forward all gated vs real weights; `SpeechRegistry` + `vllm_speech_*` v21 + `/v1/audio/speech`; GGUF Q4_K depth decoder value-gated. HTTP request OBSERVED (#852) | No reference number. Host kernels multi-core, same song bytes (§12). PARTIAL device arm (#672): 8.6B LM + 2.4B fp32 DiT staged once (§13); rest host. Denominator SGLang-Omni production | +| MiniMax-Music3 (`MiniMaxMusic3ForConditionalGeneration`, diffusers lane) | MiniMax-Music3 (8.6B Qwen3 LLM + 0.646B RVQ decoder + 2.4B fp32 DiT + DAC Flow-VAE); diffusers arm, ~28.5 GB | `ACTIVE`. Loader 1413/1413; AR, acoustic and the 8.6B LM forward all gated vs real weights; `SpeechRegistry` + `vllm_speech_*` v21 + `/v1/audio/speech`; GGUF Q4_K depth decoder value-gated. HTTP request OBSERVED (#852) | No reference number. Host kernels multi-core, same song bytes (§12). PARTIAL device arm (#672): 8.6B LM + 2.4B fp32 DiT staged once (§14); rest host. Denominator SGLang-Omni production | | LTX-2.5 DFR base + generated keyframe slots | LTX-2.5 (21.00B video+audio) | gated vs EXECUTED upstream `dfr_layout` + 3 `dfr_pipeline` helpers @ `fd4ded7f` (`test_ltx2_dfr` 11/11, 652 assertions); canvas, tiles, stitch, carry-forward as EXACT index vectors, since each defect is plausible| `--pipeline-kind dfr`. Canvas PADS 9 to 25 then trims back; slots on the x8 grid, MARKED, read back BEFORE the trim. `num_generated_keyframes` SERVED elsewhere. Temporal ROUNDS refused (#986); detail LoRA refused (#975)| | LTX-2.5 tiled + streaming Conv VAE decode | LTX-2.5 video VAE | gated vs executed upstream `ltx_core` @ `fd4ded7f` (`test_ltx2_tiling` 10/10, 915 assertions); one-tile and untiled-spatial controls BIT-EXACT vs untiled on both causality arms; an untiled frames axis is REFUSED | Streams temporal chunks through upstream's AUTO layout (768/64 px, 80/24 frames); above one tile the pixel volume is never materialized. NO-OP below 768px and 81 frames; 81-120 IS tiled, differing 6.70% of range | | LTX-2.5 Conv VAE decode arithmetic width | LTX-2.5 video VAE | `test_ltx2_vae` "the decode's convolution accumulates in f32", entering through `Ltx2VideoDecodeStreaming`; widening the accumulator to `double`, or deleting the production call site, each turns it RED | **f32**, the width `F.conv3d` uses at f32 AND bf16 (MEASURED). Was f64 at 8 sites ([#1008](https://github.com/mudler/vllm.cpp/issues/1008)). Conv sums BLOCKED per input channel, as torch's. STORAGE stays f32; bf16 owed | diff --git a/docs/STATUS.md b/docs/STATUS.md index c2f9604ad..bbea61041 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -144,7 +144,7 @@ token-for-token correctness against the pinned oracle. | InternLM2 dense (fused-`wqkv` interleaved split) | Correctness-complete, speed-pending | Token-exact 16/16 (internlm2-chat-1_8b): 12/16 strict + 4/16 bf16 near-tie (max gap 0.0 nats), 0 divergent; first InternLM model; ZERO new compute kernel (reuses the Llama dense forward; the only delta is a loader-side de-interleave of the fused `wqkv`, which packs q/k/v interleaved by KV-group) | | MiniMax-H3 (`MiniMaxH3DiTModel`, video+audio DIFFUSION) | **ABI v12 ONE SURFACE; device selector uses generic `DeviceType`; DSR 32.** t2va+fl2va COHERENT; bf16 shards STREAM | ref2va ckpt fidelity §8.12; encoder A/B §8.15; GB10 re-verify residual; CPU fold 6/137 (one queue + device provenance mutation-gated) | | LTX-2.5 (`LTX2VideoTransformer3DModel`, video+audio DIFFUSION) | **L1-L9c landed (#435).** 21.00B / 48 blocks. `VideoEngine` seam + ABI **v18**, DiT forward (CPU f32 parity, bf16 device-resident), Gemma-4 TE, both VAEs, connector, pipeline, NVFP4/FP8, keyframe bias (#658) | BOTH shipped DiTs now load inside the contract, no `allow_unported`. One runs device-resident on GB10; those 320x192/25f frames ARE a scene. A prompted render is OWED; speed and oracle parity `PENDING` | -| MiniMax-Music3 (`MiniMaxMusic3ForConditionalGeneration`, text-to-MUSIC) | **`ACTIVE`: W0-W7 landed; every stage including the 8.6B LM forward is implemented and gated (#672).** Oracle is the OPEN diffusers PR #14456 `c6da9936` | GGUF arms for 4 components owed. LM forward gated in a control; HTTP OBSERVED (#852). PARTIAL device arm, Thor sm_110 (#672): 8.6B LM + 2.4B fp32 DiT (§13). CPU kernels 10.7x on the vocoder chain. No reference number | +| MiniMax-Music3 (`MiniMaxMusic3ForConditionalGeneration`, text-to-MUSIC) | **`ACTIVE`: W0-W7 landed; every stage including the 8.6B LM forward is implemented and gated (#672).** Oracle is the OPEN diffusers PR #14456 `c6da9936` | GGUF arms for 4 components owed. LM forward gated in a control; HTTP OBSERVED (#852). PARTIAL device arm, Thor sm_110 (#672): 8.6B LM + 2.4B fp32 DiT (§14). CPU kernels 10.7x on the vocoder chain. No reference number | | Command-R / Cohere dense (`CohereForCausalLM`) | Implemented, gate-blocked | ZERO-new-kernel port grounded in vLLM `commandr.py`: weight-only Cohere LayerNorm + GPT-J full-width RoPE + PARALLEL residual + `logit_scale` + tied embeddings, all reuse; compiles, links, self-registers. No SACRED gate yet (real checkpoints HF-gated, ungated ones tiny-random, GPU box disk-full); oracle run-verified at W0. See docs/BENCHMARKS.md | | Phi-1 / Phi-2 dense (`PhiForCausalLM`, parallel residual) | Correctness-complete, speed-pending | Token-exact 16/16 (microsoft/phi-2): 9/16 strict + 7/16 bf16 near-ties (max gap 0.25 nats), 0 forward-divergent; the OLDER Microsoft Phi arch, DISTINCT from Phi-3/Phi-4; ZERO new compute kernel (GPT-J parallel residual, LayerNorm-with-bias, biased qkv/dense, partial NeoX rope 32/80, non-gated NewGELU MLP reusing `vt::GeluTanh`, untied biased lm_head); F16 dtype-aware loader | | MiniCPM dense (`MiniCPMForCausalLM`, three scalars) | Correctness-complete, speed-pending | Token-exact 16/16 (openbmb/MiniCPM-2B-sft-bf16): 10/16 strict + 6/16 bf16 near-ties (max gap 0.0 nats), 0 forward-divergent; first OpenBMB MiniCPM model; ZERO new compute kernel (the Llama/Granite dense forward plus three scalars: scale_emb, scale_depth/sqrt(layers) residual, dim_model_base logit scaling), tied lm_head; `.bin`-only weights converted to safetensors via trusted torch |