diff --git a/.agents/benchmark-record.md b/.agents/benchmark-record.md index a44f7b96f..66a6d4d21 100644 --- a/.agents/benchmark-record.md +++ b/.agents/benchmark-record.md @@ -20673,3 +20673,318 @@ the attribution, per AGENTS.md: evidence is moved and annotated, never removed. for the residual TTFT gap is that it is the same prefill glue attributed at 92.5% for 27B, now measured against a correctly fused denominator for the first time, so the attribution itself is owed a re-run. +## LTX-2.5 L9c — the per-phase pool drain is worth 0.11 GiB, and L9B's 58 GB runaway does not reproduce (2026-08-13, `row/LTX25-L9C-CONNECTOR-DRAIN`, issue [#435](https://github.com/mudler/vllm.cpp/issues/435)) + +**No speed number is claimed and none is implied.** LTX-2.5's speed axis is +structurally `PENDING` (spec [ltx-2-5.md](specs/ltx-2-5.md) §0): vLLM-Omni has no +native 2.5 and its diffusers adapter is a black box, so no production-configuration +denominator exists. Every wall clock below is SIZING — how long a render takes on +this box — and is not comparable to anything. + +### What was measured, and on what + +dgx.casa (GB10, sm_121a, 119 GiB unified), one `flock $HOME/gpu.lock` hold, +`local-ai-worker` down at both ends of every arm. Build: Release, CUDA `121a`, +`VLLM_CPP_CUTLASS_DIR=$HOME/cutlass-4.5.0`, `VLLM_CPP_TRITON=ON`, configure log +verified to print `CUTLASS found ... sm120a NVFP4 cutlass GEMM`, +`FlashAttention-2 prefill/decode: ENABLED for arch(es) [121a]` and +`Triton AOT: ... sm_121a`. + +Artifacts, named per spec §3.1 because a render is only a statement about the +files that produced it: + +| | | +|---|---| +| DiT | `vonkaiser/LTX-2.5-FP8-NVFP4` `ltx-2.5-22b-distilled-fp8.safetensors` (21.00B, FP8, 6124 tensors) | +| Video VAE | `Lightricks/LTX-2.5` `ltx-2.5-video-vae-conv-bf16.safetensors` (Conv arm) | +| Audio VAE | `Lightricks/LTX-2.5` `ltx-2.5-audio-vae-bf16.safetensors` + its BWE vocoder | +| Upsampler | `ltx-2.5-latent-spatial-upscaler-x2-bf16-1.0.safetensors` | +| Config | LTX-2.5's **DECLARED** config, `sha256 30d08fad…4b21` — `frequencies_precision=float64`, `av_ca_timestep_scale_multiplier=1000`, `connector_positional_embedding_max_pos=[4096]`. NOT the manifest defaults | +| Conditioning | 128 rows, `--prompt-valid-rows 24`, `sha256 c7dff715…11e8` / `b7ff5ff3…ef0f`. **Synthetic** N(0, 0.2): the Gemma-4 tower is not ported | + +### The drain A/B — same binary, `VLLM_LTX2_POOL_DRAIN`, 320x192 / 25 frames + +| arm | drain | wall (sizing only) | user CPU | peak host RSS | lowest MemAvailable | result | +|---|---|---|---|---|---|---| +| E0 | OFF | 23:39.30 | 1265.63 s | 32.84 GB | **68.23 GiB** | 25 frames, h264 320x192 + AAC | +| E1 | ON | 23:40.13 | 1265.82 s | 32.84 GB | **68.14 GiB** | 25 frames, h264 320x192 + AAC | + +0.8 s of wall and 0.09 GiB of floor separate them, and the floor moves the WRONG +way, so both are noise. The drain's own report says why: + +| geometry | after `generate_lowres` | after `refine` | +|---|---|---| +| 128x128 / 9f | 0.01 GiB | 0.02 GiB | +| 320x192 / 25f | 0.03 GiB | 0.08 GiB | + +**The retained scratch at an LTX phase boundary is 0.11 GiB, not 58 GB.** The +drain is correct, costs one free per retained block, and matches what MiniMax-H3 +does at the same boundary — and it is NOT what makes a bigger render possible. + +**And it is numerically INERT, proven rather than argued.** `diff -r -q` over the +two arms' output directories returns 0: all 25 frames, the WAV and the MP4 are +BYTE-IDENTICAL between drain-off and drain-on (rolled-up md5 `2eba29bf…e656` on +both sides). Draining a free list cannot change arithmetic, and this is what +saying so looks like when it is measured instead of asserted. + +### The ladder, and where L9B's ~58 GB actually lives + +| rung | geometry | denoise | outcome | +|---|---|---|---| +| D0 | 128x128 / 9f | drains 0.01 + 0.02 GiB | **completes**, 9 frames, 6:31 | +| E0/E1 | 320x192 / 25f | drains 0.03 + 0.08 GiB | **completes**, 25 frames, 23:39, floor 68.2 GiB | +| F1 | 448x256 / 25f | drains 0.04 + 0.14 GiB | **STOPPED by the watchdog**, 0 frames | + +**The highest rung that completes is 320x192 / 25 frames.** That is a measurement, +not a ceiling — see the next hypothesis below. + +F1 is where L9B's number turns up, and it is not where L9B put it. Both denoise +phases finished and drained normally, and MemAvailable was flat at **75.2 GiB** +through all of it. Then, AFTER the last drain: + +``` +03:47:11 avail_kB=73014000 rss_kB=4972520 +03:47:21 avail_kB=57800944 rss_kB=4972520 +03:47:31 avail_kB=27711644 rss_kB=4899272 +03:47:35 WATCHDOG_KILL avail_kB=13774472 floor=18000000 +``` + +**~59 GB in 24 seconds with the process's own RSS flat at 4.9 GB.** That is L9B's +~58 GB and L9B's "host RSS flat", reproduced exactly — on the DECODE side of the +denoise-to-decode boundary, not inside the denoise loop L9B attributed it to. No +drain can shrink it: the drain runs before it, and what it returns is 0.14 GiB. + +At 320x192/25f the same code path completes with the drain OFF and MemAvailable +flat, so L9B's own arm — which was 320x192 at `--max-phase 0`, a SMALLER decode +than E0's — remains unreproduced at its own geometry. Both statements are true +and they are about different rungs. + +**Two obvious attributions are each contradicted by a measurement, so neither is +claimed.** `Ltx2ConvVideoDecode` is pure host C++ — no `vt::` op, no queue, no +device pointer anywhere in `ltx2_video_vae.cpp` — and GPU utilization is **0%** +throughout, so it is not a device pool. But `ps -o rss=` reports 4.9 GB across all +twelve samples of the decline, so it is not a plain resident host allocation +either. **Next traceable step: instrument `Ltx2ConvVideoDecode`'s own allocations +directly.** It is ONE function; inferring its footprint from a system-wide counter +is what produced L9B's mis-attribution and would produce another. + +### The instrument, stated because it is weaker than it looks + +`nvidia-smi --query-gpu=memory.used` returns **`[N/A]`** on GB10. There is no +per-process device-memory reading on this box, so "device usage" is only +observable as unified-memory pressure — `MemAvailable` — which moves for anything +on the machine. That is what L9B had too, and it is why L9B's attribution of a +MemAvailable fall to the denoise loop could not have been checked at the time. + +### What bounds the ladder is the DECODE, in both of its costs + +At 320x192/25f the process ran at **0% GPU utilization and ~110% CPU** for most of +the 23:39, and at 448x256/25f it is the decode that takes the box to the floor. So +the decode is the wall twice over — it is the time and it is the memory — +and `Ltx2VideoDecode` is a HOST path: it takes `std::vector` and the host +VAE weights, and `ltx2_video_vae.cpp` contains no `vt::` op at all. MiniMax-H3 has +the device analogue (`MiniMaxH3VideoVaeDecodeTemporalDevice`, +minimax_h3_pipeline.cpp) — and, just as relevantly, H3's decode is CHUNKED in time +and TILED in space by default, so it never materializes the whole canvas at once. +LTX-2.5's does neither. + +**No ceiling is declared.** The next traceable steps, in order: measure +`Ltx2ConvVideoDecode`'s own allocation footprint; then temporal chunking + spatial +tiling, which is what H3 already needed at a real canvas; then the device decode. + +### The frames ARE a scene + +Measured with L9B's OWN analyzer, unchanged, so the numbers are comparable: +frame-to-frame mean |diff| 0.500-1.465 (avg 0.980, so not a still); neighbour +|dx| / whole-image sd **0.093** where white noise gives ~1.13; 8-px block-mean +aligned/offset ratio 1.012 and 32-px 1.193, so not H3's patch grid. + +**Those statistics are almost identical to L9B's, and L9B's frames were not a +scene while these are.** The 25 frames are a temporally coherent photorealistic +clip: one subject, consistent identity and background, frame-to-frame motion. The +statistics could not separate "smooth colour field" from "photograph", and only +looking did. That is a finding about the INSTRUMENT and it belongs beside the +result. + +It is not a depiction of a prompt. With `--prompt-valid-rows 24`, 104 of the 128 +conditioning rows are the connector's own trained `learnable_registers` — which is +exactly what upstream substitutes at padded positions — and the other 24 are +synthetic noise. So what conditions the render is the checkpoint's own learned +default, reached through the real connector, and not anything a caller asked for. + +**The audio is NOT claimed to be anything.** 1.0100 s, 48 kHz stereo, ch0/ch1 rms +26.82/26.83, peak 217/218, zero-fraction 0.024 — so it is not silence, and it is +FAINTER than L9B's at the same settings (rms 131.28/127.30). Whether it is +speech-shaped, or matches the mouth movements in the frames, is not something +these numbers answer and is not asserted. Owed: a spectral check against the +video, which is the audio half of the question the frames just answered. + +### Owed, and why — two arms the lock never came free for + +Two bounded waits on `$HOME/gpu.lock`, `flock -w 2700` each, both timed out +(`HOLD2_WRAPPER_DONE rc=1` at 04:36 and 05:22). The box was saturated with other +coordinators' `ctest` work for the whole 90 minutes. Waiting is normal and +stealing is not, so these are reported OWED rather than run: + +1. **The L9B repro arm** (`dgx:~/work/ltx25-l9c/dgx_repro_l9b.sh`, shipped and + syntax-checked on the box): L9B's own binary with L9B's own arguments at + 320x192/25f `--max-phase 0`, which is the one arm that separates "L9B measured + the environment" from "L9B's geometry behaves differently from ours". It cannot + run on the L9c binary, which refuses 32 conditioning rows. +2. **`test_minimax_h3` and `test_capi` through ctest** (`dgx_baselines2.sh`, also + shipped). The first baseline pass ran the BINARIES directly and both reported a + summary followed by a SIGSEGV — `test_capi` `4 cases | 51 skipped` against a + brief baseline of 55, `test_minimax_h3` `38 | 41 skipped` against 79. A summary + printed before a crash counts the unreached cases as "skipped", which is the + third instance this campaign has recorded of a run reading as a pass. **These + two are UNRESOLVED, not green**, and the instrument that would resolve them is + ctest plus the full output, not a grep of a bare binary. + +Evidence: `dgx:~/work/ltx25-l9c/{hold.log,hold2.log,hold2b.log,baselines.log,render-*.log,mem-*.log}`, +renders under `dgx:~/work/ltx25-l9c/render/`, contact sheets under +`dgx:~/work/ltx25-l9c/contact/`. + + +## LTX-2.5 — STATUS cell detail moved out to pay the shrink-only ratchet (2026-08-13, `row/MODEL-DIFFUSION-LTX25`, issue [#435](https://github.com/mudler/vllm.cpp/issues/435)) + +`docs/STATUS.md` is a shrink-only surface (`check-public-doc-tables.py`, +`STATUS_RATCHET["oversized_cells"] = 44`), and the L9 stack merge took it to 45. +The LTX-2.5 cell was the one that grew, so its detail lands HERE rather than +being deleted -- AGENTS.md: compact by MOVING superseded detail, never delete +evidence to save space. + +- **Resolution ladder.** 128x128/9f and 320x192/25f complete; 448x256/25f stops + in the HOST VAE decode, not in the denoise loop. MemAvailable is flat at + 75.2 GiB through both denoise phases, then falls 73.0 -> 13.8 GiB in 24 + seconds with process RSS flat at 4.9 GB -- on the decode side of the last + drain, so no drain can shrink it. Named next hypothesis: instrument + `Ltx2ConvVideoDecode`'s own allocations, then temporal chunking + spatial + tiling (H3 needed both). No ceiling is declared. +- **Device residency.** All ops `vt-native`, zero reference-tier hits. +- **Why the speed axis is `PENDING`.** Spec section 0: vLLM-Omni carries no + native LTX-2.5 and its diffusers adapter is a black box + (`supports_step_execution=False`), so no production-configuration denominator + exists. Absent a denominator, no ratio is claimed. + +**A correction carried in the same edit.** The removed cell also stated that +`test_minimax_h3` and `test_capi` were "UNRESOLVED on this branch, not green" +after two bounded GPU-lock waits timed out. A fresh reviewer resolved both on a +freed box: **`test_capi` 55/55 cases / 505 assertions and `test_minimax_h3` +79/79 / 57,395 assertions, exit 0 (not 137), with build logs clean of +`No space left`.** Both hit their FULL registered counts, and neither suite +references `ltx2` while this work touches only `ltx2_*`. The earlier reading was +an INFRASTRUCTURE artifact of a 100%-full disk, not a regression, and the public +status surface should not have gone on implying otherwise. + +## LTX-2.5 L13 — the 320x192/25f render was REGISTER-conditioned, not prompted, and the composition's value oracle is owed (2026-08-13, `row/LTX25-L13-PROMPT-HOP`, issue [#435](https://github.com/mudler/vllm.cpp/issues/435)) + +**No speed number is claimed and none is implied.** LTX-2.5's speed axis stays +structurally `PENDING` (spec [ltx-2-5.md](specs/ltx-2-5.md) §0). + +### The public-doc claim that was withdrawn + +`docs/FEATURES.md` briefly read *"e2e at 320x192/25f from a TYPED PROMPT via +Gemma-4: coherent scene, valid MP4+WAV"*. **No recorded run supports that +sentence**, and it is withdrawn rather than softened. + +| what the sentence asserted | what was actually measured | +|---|---| +| a render from a TYPED PROMPT | L9c's 320x192/25f arm ran with `--prompt-valid-rows 24` and **synthetic** N(0, 0.2) conditioning; the Gemma-4 tower was not on that path at all | +| a "coherent scene" produced BY that prompt | the record for that same arm states it plainly: *"It is not a depiction of a prompt"* — 104 of 128 rows are the connector's own trained `learnable_registers` and the other 24 are noise, so what conditioned the render is **the checkpoint's own learned default** | +| Gemma-4 in the loop | L10's real-checkpoint run produced **conditioning only** (`video [1024, 4096]`, max\|v\| 34.07) and **no frames** | +| an e2e gate behind it | L13's own gate is **fixture-only, CPU Release**; the PR body itself claims no real-checkpoint prompted render | + +Both halves of the claim — "typed prompt" and "coherent scene" — were attached +to a run that did not happen, by re-attributing L9B/L9c's measurement +(this record, the L9c section above) to a capability that landed three hops +later. The public cells now read *"Typed prompt -> Gemma-4 -> cross-attn, +FIXTURE-gated. The 320x192/25f scene was register-conditioned; a prompted render +is OWED"*. + +**OWED, and not attempted here:** a real-checkpoint prompted render. The +arithmetic is why it was not run rather than run badly. L9c's 320x192/25f arm +bottomed out at **68.2 GiB MemAvailable** on a 119 GiB unified-memory box that +**reboots rather than OOM-kills**, and the text tower is a further **~24 GB of +host bf16** at the shipped 12B (`ltx2_video.cpp:846`) on top of the FP8 DiT. At the time of writing dgx.casa's `$HOME/gpu.lock` +was held by another session's `ctest` and its root filesystem was at **99% +(62 GiB free)**, against a ~21 GiB build tree plus render artifacts. Attempting +it would have risked the box for other live agents to produce one number. The +honest statement is the one now in the docs. + +### The composition has no value oracle — measured, not suspected + +`Ltx2VideoEngine::last_conditioning()` is a **witness, not a gate**. A reviewer +proved it on this head with two mutations of the `Generate` composition +(`ltx2_video.cpp`), each applied alone: + +| mutation | result | +|---|---| +| video conditioning scaled **x1.5** after the connector | **485/485 assertions, exit 0** | +| conditioning rows **REVERSED** — every caption row on the wrong token | **485/485 assertions, exit 0** | + +A digest detects CHANGE; it does not pin VALUES. The digest moved under both, +and no assertion says which value it should have moved to. + +**Where the gap is, precisely.** The per-brick oracles are real: the Gemma-4 +tower vs a running `transformers` at a measured bf16 floor, `Ltx2ConnectorForward` +on five arms vs executed upstream, and the feature extractor and both caption +projections vs executed upstream. The two **joins** have none — +`Ltx2ConnectorCreateEmbeddings` and the `Generate` composition that chains it +onto `Ltx2TextEncoderConditioning`. Both mutations live in exactly that gap. +`Ltx2ConnectorCreateEmbeddings`'s own tests are PROPERTY tests: the +padding-side-agnosticism case compares two of OUR OWN calls, so a defect in both +arms cancels — the "gate through a shared helper proves consistency, not +correctness" pattern this project has already been burned by. + +**The closure, specified because the path is already built.** +`scripts/gen-ltx2-pipeline-goldens.py` already imports and EXECUTES upstream +`text_encoders/gemma/embeddings_connector.py` under a pinned SHA (section 10), +and the composition's upstream counterpart is one function in the same package: +`EmbeddingsProcessor.process_hidden_states` (`embeddings_processor.py:97-117`) — +feature extractor, additive mask, `create_embeddings`, both connectors, i.e. +this exact chain. A section executing it end-to-end at the reduced dims the +script already uses gives both joins a numeric oracle against executed upstream +rather than against our own helper. + +**Verified as a prerequisite, so the next implementer does not have to:** the +generator reproduces its committed output **byte-for-byte** on the pinned +upstream — `md5 53e2a6aba8885d7d58302ad0b7b09eb4` for both the regenerated file +and `tests/vllm/models/ltx2_pipeline_goldens.inc`, with `LTX-2` and `vllm-omni` +both clean at `fd4ded7f2d88d3da713abcdd4ad41ecc4a9314ca` / +`a4ea67a21b20054dacc6e83952f9bd407e8ee4e7`, the SHA the C++ suite pins. So a new +section can be added without disturbing anything already gated. + +### The V2 marker header, recorded so it can be checked without the checkpoint + +The four `Ltx2SelectTextFeatureVariant` markers in +`tests/vllm/multimodal/ltx2_video_fixture.h` are correct, but nothing in the +repo recorded the header they came from. Read 2026-08-13 from the safetensors +`__metadata__` (header JSON only, no tensor data) of +`/mnt/nas_share/checkpoints/ltx-2.5/lightricks-ltx-2.5/diffusion_models/ltx-2.5-22b-distilled-transformer-nvfp4.safetensors` +(18,721,432,024 bytes; `Lightricks/LTX-2.5` revision +`8a4ff96f581e72bedc1b44367581c49d544a05f1` per the HF download record — the LFS +oid is the upstream sha256 and was NOT re-verified locally): + +| key | observed | +|---|---| +| `caption_proj_before_connector` | `true` | +| `caption_projection_first_linear` | `false` | +| `caption_proj_input_norm` | `false` | +| `caption_projection_second_linear` | `false` | +| `num_attention_heads` / `attention_head_dim` | 32 / 128 -> video 4096 | +| `audio_num_attention_heads` / `audio_attention_head_dim` | 32 / 64 -> audio 2048 | +| `text_encoder_norm_type`, `model_version` | `PER_TOKEN_RMS`, `2.5.0` | + +All four present, none drifted, so this checkpoint resolves to V2. +`text_encoder_norm_type` corroborates it independently, and the selector +deliberately does not read that key — it mirrors upstream's four-marker test. +The vonkaiser FP8 DiT the render arms actually load carries **no `__metadata__` +block at all**, so this first-party file is the only on-disk source for the four +values. + +**A cause corrected in the same pass.** The fixture comment said an earlier +partial marker set *"gated no variant selection at all"*. That is not why the +refusal never fired: `Ltx2SelectTextFeatureVariant` **does** refuse a partial set +(`ltx2_text_encoder.cpp:184-192`). It never fired because **no production path +called the selector before L13** — the marker keys and the engine's first call to +it landed in the same commit, so there was no earlier run for it to refuse. diff --git a/.agents/claims/CLAIM-LTX25-L0-L7.md b/.agents/claims/CLAIM-LTX25-L0-L7.md new file mode 100644 index 000000000..5b63a5ef2 --- /dev/null +++ b/.agents/claims/CLAIM-LTX25-L0-L7.md @@ -0,0 +1,5 @@ +# CLAIM-LTX25-L0-L7 + +| Claim | Row IDs | Agent | Worktree / remote dir | Branch | Owned scope | State | Last update | +|---|---|---|---|---|---|---|---| +| `CLAIM-LTX25-L0-L7` | `MODEL-DIFFUSION-ltx-2-5-ltx2-video-transformer-3d-model` (`SPIKE`), `ROAD-V1-LTX25` | Claude Code (opus-5), operator role | isolated worktree `/home/mudler/_git/vllm.cpp-ltx25`; GPU work on dgx.casa serialized through `flock $HOME/gpu.lock` | `row/MODEL-DIFFUSION-LTX25`, issue [#435](https://github.com/mudler/vllm.cpp/issues/435) | Owns: NEW `.agents/specs/ltx-2-5.md`; the LTX-2.5 rows in `.agents/roadmap_v1.md` and `.agents/model-matrix.md`; the LTX-2.5 rows in `docs/{STATUS,BENCHMARKS,FEATURES}.md`; NEW `src/vllm/model_executor/models/ltx2_*` and `include/vllm/model_executor/models/ltx2*`; NEW `vllm::multimodal::VideoEngine` seam plus the MiniMax-H3 move behind it (BEHAVIOUR-PRESERVING only); additive ABI v13 fields in `include/vllm.h`. EXCLUDES: any change to H3's numerics or its checkpoint handling, the v12 ABI contract, DiffVAE / `NADiffusionDecoder`, the temporal x2 upsampler, LoRA fusion, multishot, `int8-convrot`, and multi-GPU / CFG parallelism, all of which are recorded as owed in the spec rather than silently dropped | `ACTIVE` | 2026-08-13 — **MERGE AUTHORITY RECORDED**: developer-directed "keep going autonomously until you get it working, fix the bugs you find along the way and merge the PRs directly once everything is proved". Scope widened beyond L0-L7 to a working END-TO-END LTX-2.5 render: L9a NVFP4 linear scale layout, L9b real render + `--video-family`, L10 the Gemma-4 tower so a prompt works, L11 the VAE encoders for image/video/audio conditioning, and the #516 pool fix that a trustworthy render depends on. Merge is gated on PROVED, which for this campaign means: fresh review PASS, the operator's own gate rerun, and no red bought by weakening a detector. | diff --git a/.agents/model-matrix.md b/.agents/model-matrix.md index 71ae250cb..e63964f10 100644 --- a/.agents/model-matrix.md +++ b/.agents/model-matrix.md @@ -55,12 +55,12 @@ Rollup by lifecycle state (must equal the detailed per-state row counts): | INVENTORIED | 321 | | PARTIAL | 20 | | ACTIVE | 9 | -| SPIKE | 7 | +| SPIKE | 8 | | BLOCKED | 5 | | DONE | 3 | | READY | 3 | | GATING | 1 | -| **Total** | **369** | +| **Total** | **370** | Engaged architectures (the 48 non-`INVENTORIED` rows): @@ -95,6 +95,7 @@ Engaged architectures (the 48 non-`INVENTORIED` rows): | 📋 | `MuseGlimmerForConditionalGeneration` | Muse Glimmer 30B (Meta, agentic multimodal; iRoPE + gated attn + perception encoder + DFlash) | **W0-W7 ON `row/MODEL-MUSE-GLIMMER` (PR #279, NOT merged): 52-layer text forward, 50-layer perception encoder, mm wiring, ATEM reasoning + tool parsers — the forward no longer refuses.** The evidence behind that is NARROW and says so: text agrees with a torch transcription of #51655 and with HF's own `muse_glimmer` on REAL 30B tensors at **reduced depth 4/52 only** (5 prefill argmax positions, no generated tokens); full depth never ran, multi-step decode is untested, the perception encoder has NO reference check, nothing has run through the server, and the ATEM parsers' channel scoping does not work at server defaults (the seam has no `adjust_request` dispatch site and `skip_special_tokens` defaults true, so the framing is stripped first). The lifecycle token stays `SPIKE`/📋 deliberately: advancing it owes `docs/STATUS.md` and `.agents/NOW.md` rows, and both are byte-exactly at their shrink-only ratchet/budget, so the advance belongs to the landing commit that can pay for them. Released 2026-08-08, **BEYOND-PIN**: no Muse code at `555967922` nor on vLLM `main`; the only upstream impl is the OPEN PR [#51655](https://github.com/vllm-project/vllm/pull/51655) head `075d645af` (approved, 3/20 CI red), ported from on explicit developer direction as a tracked exception (porting-inventory §9 deviation 16). **No oracle ⇒ no speed axis is claimable**; correctness gates against the HF reference. Text tower fully traced (Gemma-style sandwich norms w/ baked +1 offset + split eps, Llama-4-style iRoPE NoPE/full vs RoPE/sliding, weightless pre-RoPE QK-norm, ~3.87 query pre-scale w/ dual config schema, Qwen3.5-style attn output gate reading the LAYER INPUT); 50-layer perception encoder (linear patchify, bilinear pos-emb interp, width-first 2D RoPE, block-windowed attn, pixel-shuffle merge) reuses the Qwen3-VL vision seam; DFlash is recognition + `is_neox_style` threading on the existing speculator, not a new drafter | `MODEL-MM-muse-glimmer-muse-glimmer-for-conditional-generation` | | 📋 | `KimiK3ForConditionalGeneration` | Kimi K3 (2.8T MoE + MoonViT-V2, DERIVE-AND-SHIP) | **W2/W5 CPU scaffolding landed** (registry stub + nested text/vision/quant config descent + text-backbone structural name-map + REFUSE-by-name forward + MXFP4-refuse loader; clean CPU build, scaffold gate 6/6). text backbone IS `KimiLinearForCausalLM` (KDA+MLA+MoE hybrid, HEAVY reuse); **does NOT fit GB10 (~1.56 TB MXFP4, ~12×)** and NOT in the pinned oracle ⇒ no on-box golden — DERIVED, proxy-gated on Kimi-Linear-48B; forward + MXFP4 + KDA delta + MoonViT-V2 not implemented (NOT-YET-BUILDABLE) | `MODEL-MM-kimi-k3-kimi-k3-for-conditional-generation` | | 🚧 | `MiniMaxH3DiTModel` | MiniMax-H3 (33.1B omni-modal video+audio DiT, DERIVE-AND-SHIP) | **W1/W2 landed**: packed layout (fl2va + ref2va, fp64 position grid BIT-EXACT), latent packing, euler-ancestral eta0 scheduler, and the full DiT forward all parity-gated against the UPSTREAM vLLM-Omni modules executed at reduced dimensions (**max abs diff 1.6e-7**, 10/10 cases / 2539 assertions). NOT autoregressive (no KV cache, no sampler, no logits) and **e2e HW-BLOCKED** (~354 GB checkpoint, ~133 GB/rank on 4x B300 vs 119 GiB unified); bf16 production stream + request planning + the ComfyUI-GGUF arm also landed (535 REAL tensors resolve onto our contract, geometry from shapes alone). **HW verdict CORRECTED: quantized arms FIT (~41 GB in 119 GiB)**, so e2e + speed are reachable; encoder/VAEs/audio VAE DONE (4.2e-9 vs the checkpoint's remote code); NVFP4 layout GATED as identical to ours (speed path is loader wiring); BOTH VAE DECODERS done (audio 4.2e-9, video ViT3D 8.9e-8); video tiling + 3D-CNN encoder (conditioning only) pending; encoder TEXT tower done (1.2e-7); **serving `/v1/videos` DONE and the DEVICE-RESIDENT forward (W2b, f32) LANDED + GPU-VERIFIED on Thor sm_110 at video 1.49e-7 / audio 8.94e-8**; bf16 stream + fusion folds + the FP4 path (needs sm_121a) + a real-checkpoint run pending. **2026-08-05: the AUDIO-VAE ENCODER is ported** (DAC analysis stack + `pre_block` AttnProjection + `mean_proj`, gated stage by stage vs the checkpoint's own remote code at 2.98e-8 / 1.64e-7 / 1.86e-8) with its own checkpoint loader gated on the real 1087-tensor manifest — so **ref2va AUDIO and VIDEO+AUDIO references are now WIRED** (audio rows move by 0.51 / 0.71; a different waveform still moves them by 7.1e-4). Both VAEs are now complete in both directions. **bf16 13-SHARD RELEASE INDEXES 2026-08-07 (`row/H3-BF16-SHARDED-DIT`)**: `MiniMaxH3ShardedCheckpoint` resolves the ORIGINAL 66.3 GB release through its own `model.safetensors.index.json` (a tensor named in the index but missing from its shard throws BY NAME), `EnumerateMiniMaxH3ShardedTensors` feeds the shared shapes-only geometry parser, and `LoadMiniMaxH3DitFromShards` is the host-f32 reference loader. Gated CPU-only at 72/72/54497 (post-rebase): every tensor resolves to the shard the index named AND to the bytes written there, the derived geometry equals the single-file path field for field, and a SPARSE 13-shard release with the REAL 535 tensors at REAL shapes (66.3 GB declared, 144 KB on disk) derives the SHIPPED geometry (50/5376/56/128/14336/24/32/1x2x2/5120). **STREAMS 2026-08-07 (`row/H3-BF16-SHARDED-STREAM`)**: `StreamMiniMaxH3ShardedToDeviceBf16` uploads it one tensor at a time — a BF16 tensor bound for a bf16 device slot goes straight from the mmap with ZERO host buffer, so peak host is bounded by ONE tensor (observed `host_peak=8192`, `direct=37 converted=9`); bit-exact vs the non-streamed `StageMiniMaxH3DitWeights` reference over all 46 views with identical logits, rope.inv_freq host-resident, 73/73/55203. Spec §8.14. **bf16 TEXT ENCODER + THE CONDITIONING NUMBER 2026-08-07 (`row/H3-ENC-BF16-COND-DIFF`)**: the 14-shard 63 GB bf16 Qwen3-VL-32B encoder streams to device too (`StreamMiniMaxH3EncoderShardsToDevice`, q/k/v and gate/up fused ON DEVICE), `--encoder-only` runs the tower alone (peak ~96 -> ~49 GiB by not loading the DiT first), and the widening is gated BIT-IDENTICAL vs an f32-staged tower so the A/B cannot be confounded. MEASURED on Thor over 233 tokens: Q4_K_M vs bf16 conditioning is cos 0.99745 mean / 0.909 min, rel RMS 6.85% excluding the attention sink, median rotation 3.5 deg — same energy as a ONE-WORD prompt edit but DIFFUSE (232/233 tokens rotate vs 172/233). Whether the RENDER changes is NOT established. 75/75/55609. Spec §8.15. This UNBLOCKS the quantization-quality question; no bf16-vs-quant render or speed number is claimed. Spec §8.13. **W-FP4a LANDED (CPU) 2026-08-06 (`row/H3-FP4-SPEED`)**: the device DiT forward now routes the NVFP4 projections through the shared Marlin W4A16 dispatcher (fp4 kept packed; no new quant code), fp4-vs-bf16 wiring gate GREEN (62/62·30039). **W-FP4a GB10 leg LANDED 2026-08-06 (`row/H3-FP4-GPU-E2E`, PR #64):** on sm_121a the Marlin W4A16 path RAN for all 11 projections (`dense_gemms==11` default — VT_MARLIN_DENSE is default-ON → vLLM's own DENSE Marlin GEMM, not the grouped route; `marlin_gemms==11` under VT_MARLIN_DENSE=0; `fallback_gemms==0`), fp4-vs-bf16 BYTE-EXACT (max\|diff\|=0), and the fp4 arm is a MEMORY win not a diffusion-forward speed win (per-forward bf16/fp4 3.47× @seq64 → 0.79–0.83× @seq4224–7040; ~16 vs ~66 GB device). Real-checkpoint fp4-resident t2va e2e RUNS (real 18.75 GB NVFP4 DiT + VAEs + GGUF Qwen3-VL-32B encoder → valid mp4/wav; DiT s/step 5.45/20.0/209 s @512/768/REF-209f) but frames are a non-scene patch-grid at 12/20/50 steps → OPEN render bug (device VAE/denoise). vLLM-Omni has no quantized H3 arm (BF16-only) so any comparison is HW/loader-forced-indirect — spec §8. **ONE-SURFACE ROW 2 LANDED 2026-08-08 (`row/H3-VIDEO-ABI`)**: the whole assembly pipeline is library-owned (`vllm::multimodal::MiniMaxH3VideoEngine`, `minimax_h3_video.cpp`) behind the ABI v12 `vllm_video_*` entry points; `/v1/videos` routes through the SAME seam; `minimax_h3_gen`+`minimax_h3_mux` are thin `vllm.h` clients, frames+WAV byte-identical to the pre-fold binary on the committed fold fixture (`test_minimax_h3_video_fold` 3-arm gate + the v12 `test_capi` section); GB10 real-video re-verify via the v12 ABI = named residual. **ROW 2 DEVICE-SEAM FOLLOW-UP (#135; replaces #134):** ABI 0/1 maps once to `vt::DeviceType`; shared code dispatches through `GetBackend(device_type)`, restoring DSR 34→32 without a baseline/allowlist change; CPU compile/fold test pending in CI due shared-disk pressure **PRUNED CHECKPOINTS LOAD AND RUN 2026-08-10 (`row/H3-PRUNED`, #241, spec §8.21):** the community `pruned` variants (`unsloth/MiniMax-H3-GGUF` Q2_K..Q8_0, `lilcheaty/MiniMax-H3-NVFP4` `*_pruned_nvfp4`) are not lossily pruned but ComfyUI's AdaLN timestep-CURVE refactor (`comfy/ldm/minimax/model.py:419-432,610-615`): an `adaln_t_table` [1025, 8] lerp replaces the sinusoidal+MLP time embedder, no SiLU before the AdaLN linear, and its in_features drop 2688 -> 8, collapsing `adaln_proj` from 13.04B of 33.12B to 0.04B and the DiT to 20.11B — so a pruned Q8_0 (21.4 GB) costs what our unpruned Q4_K_M (19.9 GB) costs. Gated on the REAL 532-tensor pruned GGUF manifest (532 = 535 - 4 `time_embedder.*` + 1 table, header-only), the clamped-lerp golden, and a CONSTRUCTED-curve test proving the pruned forward is numerically identical to the unpruned one; 79/79/57299. | `MODEL-DIFFUSION-minimax-h3-mini-max-h3-dit` | +| 📋 | `LTX2VideoTransformer3DModel` | LTX-2.5 (21.00B joint video+audio flow-matching DiT, Lightricks) | **L0 spec committed 2026-08-11 (#435, [spec](specs/ltx-2-5.md)).** Geometry MEASURED from the FP8 checkpoint's own safetensors header by HTTP range request (6124 tensors, 881,048-byte header, no payload downloaded): **21.00B** params — 48 blocks @ 386.7M = 18.560B, audio embeddings connector 2.016B, global 0.427B. The filename says `22b` and the Diffusers card says ~19B; the MEASURED count is what this row uses. Video stream 4096 (32 heads x 128), audio stream 2048 (32 heads x 64), in/out channels 128 both. Per block SIX attentions — `attn1` (video self), `attn2` (video<->text, cross 4096), `audio_attn1`, `audio_attn2` (cross 2048), and the two CROSS-MODAL `audio_to_video_attn` / `video_to_audio_attn` — which is the structural break from MiniMax-H3: H3 packs every modality into ONE sequence with per-row token tags, LTX runs TWO streams coupled by explicit cross-attention. **Per-head gated attention on every attention** (`to_gate_logits` = `Linear(query_dim, heads, bias=True)`, `attention.py:513-514`, applied AFTER the attention output at `:577`) — H3 has no analogue and getting it wrong renders plausibly-wrong rather than erroring. FFN is gelu-approximate 4096->16384->4096 with **NO bias** while `audio_ff` 2048->8192->2048 **HAS** bias, which independently confirms `ff_bias=false` / `audio_ff_bias=true` from `model_configurator.py:78-80` — checkpoint and source agreeing, not either alone. **RETRACTED 2026-08-12, was billed as a FREE WIN.** The spec claimed 2.5 sets `use_prompt_adaln_single=false` so the cross-attention K/V are timestep-free and cacheable. The shipped checkpoint DISPROVES it: it carries 12 `prompt_adaln_single`/`audio_prompt_adaln_single` tensors including a `timestep_embedder.linear_1 [4096, 256]` (256 = the sinusoidal timestep width), and `model.py:223-227` builds that module ONLY when the flag is TRUE. `transformer.py:441` was quoted as proof of no timestep term, but `:442-443` add one whenever `prompt_timestep` is not None, and the comment above them says exactly that. NO SHIPPED DEFECT: `ltx2_dit.cpp:672` refuses the cache by name when the flag is on, so the feature is correct-and-inapplicable rather than silently wrong, and stays gated bit-identical and prompt-bound for any checkpoint that does set it false. **ORACLE:** vLLM-Omni does NOT support 2.5 — `resolve_ltx_pipeline_recipe` keys on 2 / 2.3 only and RAISES otherwise (`ltx2_recipes.py:162-166`), with upstream [vllm-omni#6066](https://github.com/vllm-project/vllm-omni/issues/6066) filed 2026-08-11 — but its `DiffusersAdapterPipeline` is generic (`DiffusionPipeline.from_pretrained`, `pipeline_diffusers_adapter.py:116`), so vLLM-Omni CAN execute 2.5 via `--load-format diffusers`. Binding oracle = that adapter; immediate cross-check = Lightricks `ltx-pipelines`. **HW: FITS ONE GB10** at ~29 GB (NVFP4 DiT 18.72 + NVFP4 Gemma-4 TE 7.40 + VAEs 1.83 + upscaler 1.00) vs H3's ~41 GB. **OWED UP FRONT, not to be discovered later:** the speed axis lands `PENDING` because the adapter is a black box (`supports_step_execution=False`, `supports_request_batch=False`) and therefore NOT vLLM's production configuration, which AGENTS.md requires as the denominator; DiffVAE (`NADiffusionDecoder`, neighborhood attention) is REFUSED BY NAME until its own row rather than silently downgraded to the Conv VAE; and no render-quality claim is made from structural e2e. | `MODEL-DIFFUSION-ltx-2-5-ltx2-video-transformer-3d-model` | | ✅ | `LagunaForCausalLM` | Poolside Laguna-S-2.1 (118B/8B MoE) | **LONG-CTX DECODE LEVERS LANDED + MEASURED (2026-08-03, `CLAIM-LAGUNA-LONGCTX-LEVERS`): window-bounded SWA reads (`VT_LAGUNA_SWA_WINDOW`, default-ON, BYTE-EXACT) bound the four `DecodeAttnGqa*` kernels' read to the ~512 sliding window (vLLM `laguna.py:412`) — GB10 A/B token-IDENTICAL `=1` vs `=0` at 520-token context (truncation active), MEASURED −0.30 ms/step at ~2k (~0 at ≤512, grows linearly). bf16 paged KV (`VT_LAGUNA_KV_BF16`, default-OFF opt-in) a distributional near-tie left UNRATIFIED. See BENCHMARKS `CLAIM-LAGUNA-LONGCTX-LEVERS`.** — **NVFP4 W4A4 ARM RAN on GB10 (N4, 2026-08-01, `CLAIM-LAGUNA-NVFP4-N4`): the additive safetensors NVFP4 arm (N1a/N1b/N2/N3 — `Nvfp4Weight` expert fields + `LoadLagunaForCausalLMWeights` + `LqGemmNvfp4Fp4` per-expert TRUE-W4A4 + `LagunaFfnBlock` `fp4` branch + `laguna_gen` dir-autodetect; CPU-gated `test_laguna_nvfp4_loader` 3/3·61, GGUF path byte-identical) generates COHERENTLY on the real 67 GiB `poolside/Laguna-S-2.1-NVFP4`. vs the vLLM MARLIN golden (vLLM's exact prompt ids injected): FIRST 2 TOKENS MATCH exactly, then near-tie divergence (our TRUE-W4A4 fp4-activations vs the MARLIN golden's W4A16 bf16-activations — different precision, EXPECTED; shares golden vocab). SPEED (N5, trace-driven, 2026-08-01): 0.16 → ~4.5 tok/s (~28× THIS SESSION), now ~4× from vLLM 18.8. **Lever #2** (nsys found the bf16 tower running host `MatmulNK` on the CUDA queue): route it to the GPU (`LqGemm` bf16 → `CastBf16` + `MatmulBT`, weight stays bf16) → 6.34 → 0.39 s/tok (16×). **Lever #1** (nsys found the emulation expert GEMM at 92%, GPU 87% busy): the engine's native sm120a fp4 tensor-core MMA (`MatmulNvfp4Fp4Native`) reads the SAME linear scales — it was gated OFF behind `VT_NVFP4_FP4_NATIVE`; default it ON in the driver → 0.39 → ~0.20-0.24 s/tok (~2×). Both coherent + near-tie (byte-identical ids to emulation; first token matches golden). Two GB10 memory fixes landed to run (shard-release + context-before-load). OPEN #234 (remaining ~4×): grouped W4A4 MoE (top_k×3 launches → 3), `ResidentNvfp4`, decode CUDA-graph + on-GPU sampling (the host-orchestration tail). Spec `.agents/specs/laguna-nvfp4-arm-2026-07-31.md` §N4/§N5. The GGUF-Q4_K track (below) is the separate keep-quant vehicle.** Prior **FASTER DECODE (W9, 2026-07-31, `CLAIM-LAGUNA-W9-GROUPED`): the 30 un-grouped per-expert keep-quant GEMV launches/step (top_k × {gate,up,down} `LqGemmRowSlice`) fold onto the SHARED `vt::MatmulBTQuantGrouped` op — per token, Pk experts' gate/up/down each collapse to ONE grouped launch over the already-stacked `[E*N,H]` tower (no loader change). Same-binary A/B on real UD-Q4_K_XL (GB10, `--gpu`, drop_caches cold, 24 tok): grouped (`VT_LAGUNA_GROUPED_MOE=1`, default) == per-expert (`=0`) BYTE-IDENTICAL (md5 `754728c6`, both == W6 golden) + decode 0.18 → 0.13 s/tok (1.38×). Routes through the shared vt op (fold policy). Cumulative with W8: decode 0.66 → 0.13 s/tok (5.1×; 1.5 → 7.7 tok/s; 18× → 3.6× vs llama.cpp 27.8). Next lever: device-resident decode (#1). See spec §W9.** Prior **FASTER DECODE (W8, 2026-07-31, `CLAIM-LAGUNA-W8-EMBED`): `LagunaEmbed` no longer converts the whole 1.23 GB embed table to f32 every token (it gathered T rows out of the whole [Vsz,H] table via `ReadF32` — ~311M host element-converts/token, the DOMINANT decode cost the W7 profile under-filed as "#5"); now gathers only the T needed rows directly (BIT-IDENTICAL — same per-element conversion, same rows). GATED on the real 3-shard UD-Q4_K_XL GGUF (GB10, `--gpu`, W6 cached, drop_caches cold, 24 tok): TOKEN-IDENTICAL to the W5/W6 golden (`22345 83 350 785 …`, coherent " Paris.") + decode 0.66 → 0.17 s/tok = 3.9× (1.5 → 5.9 tok/s; 18× → 4.7× vs llama.cpp 27.8). See `.agents/specs/laguna-s21-w7-speed-2026-07-31.md` §W8. Next: grouped-expert GEMM (=A3) then device-resident decode.** Prior **DECODE-SPEED ATTRIBUTED (W7 profile-only, 2026-07-31, `CLAIM-LAGUNA-W7-SPEED`): `nsys` of the W6 decode (real UD-Q4_K_XL GGUF, GB10) attributes the 0.66 s/tok (~1.5 tok/s vs llama.cpp 27.8 on identical bytes, ~15-18x) to HOST-ORCHESTRATION, not kernel compute — GPU active only 32.7% of the step, 67.3% host/idle; 22,115 `cudaStreamSynchronize` (~2,764/step, zero GPU overlap) from the ~1,795 per-GEMM `DrainQueue` in `LagunaForwardGgufCached` + scalar host glue; 39.4% of GPU time is `QuantizeQ8K` activation-quant (per-GEMM), weight GEMVs un-grouped at ~22% of the 240 GB/s peak (llama.cpp ~76%); no H2D/D2H (unified memory). Ranked levers (all in-tree from ds4): device-resident decode 1.5->~5-7 tok/s, grouped-expert GEMM (`MatmulBTQuantGrouped`) +1.5-2x + dedupes the activation-quant, decode CUDA-graph, tuned MMVQ; + free host cleanups (`LagunaEmbed` copies the whole 1.23 GB embed table/token, per-token RoPE-cache rebuild). Honest reachable ~13-20 tok/s, 27.8 a stretch. NO code changed. See `.agents/specs/laguna-s21-w7-speed-2026-07-31.md`. Prior RUNNABLE + FAST DECODE (W6, 2026-07-31): a per-layer K/V cache + single-token incremental decode replaces W5's O(n²) STATELESS recompute — TOKEN-IDENTICAL (byte-equal ids, md5 `754728c6…` match, == the W5 golden) and 5.05× faster per token: decode 3.33 → 0.66 s/tok on the real UD-Q4_K_XL GGUF (GB10, `--gpu`, keep-quant), same " Paris.…" text. `LagunaKvCache` (mirrors `DeepseekV4KvCache`, MLA-latent → GQA multi-head K/V; caches post-QK-RMSNorm/post-RoPE K + raw V at f32 — bit-exact since RoPE/QK-norm are position-only and attention is causal), MIXED attention per-layer: 12 GLOBAL layers grow unbounded + 36 SLIDING-WINDOW-512 layers EVICT rows beyond the 512 window (gemma2/3 `is_sliding`); `LagunaForwardGgufCached` + shared `LagunaAttention`/`LagunaFfnBlock` helpers used by BOTH forwards (identical float ops; recompute ids unchanged after refactor), `examples/laguna_gen --stateless` A/B flag. No cache bug (bit-exact first run). Next speed = grouped-expert GEMM + device-resident decode (both in-tree from ds4). See `.agents/specs/laguna-s21-w6-2026-07-31.md`. Prior RUNNABLE (W5, 2026-07-31): our engine greedy-generates COHERENT text on the REAL 3-shard UD-Q4_K_XL GGUF (GB10 keep-quant) — "The capital of France is" → " Paris. …", first token "Paris." matches the llama.cpp-Poolside reference. Multi-shard GGUF reader + keep-quant tower (`LoadLagunaFromGgufShards`) + `LagunaForwardGguf` (ds4 keep-quant Gemm/GemmRowSlice) + `examples/laguna_gen`; load 20.6s, peak 71 GiB, 3.27 s/tok stateless recompute (speed=W6).** Prior W3: **W3 REAL forward + 3 new ops landed** (`laguna_ops.cpp`: per-head softplus attn out-gate + ungrouped sigmoid-noaux router + dual per-layer RoPE cos/sin builders; `LagunaModel::Forward` now a REAL runnable host-reference composition — variable-Q-head GQA + dual RoPE + sliding-window mask + softplus gate + dense L0 / ungrouped-MoE L1..47 + untied lm_head — replacing the W1/W2 `VT_CHECK(false)` stub; CPU `-Werror` full-library build clean; `test_laguna_scaffold` **8/8·166** incl. softplus math, router selection+tie-break RED-first, dual-RoPE cos/sin bit-match, variable-Q-head shapes, forward composition on synthetic weights; `test_model_registry` 24/24). W1 oracle DECISION: vLLM native `laguna.py` in pin ⇒ config constructs; dual-oracle = vLLM-NVFP4/-FP8 (fits GB10, BF16 235 GiB does NOT) + llama.cpp-Q4_K token-exact. DEFERRED to W4 (needs 73 GB checkpoint): GGUF keep-quant tower materialization + device/paged production forward + strict dual-oracle greedy gate. ~85-90% reuse (ds4-MoE + gemma-sliding + olmo3-dual-rope + landed Q4_K keep-quant); NEW = the 3 landed host ops + name-map + variable-Q-head device runner. **W4 (2026-07-31, `CLAIM-LAGUNA-W4`, in progress):** the UD-Q4_K_XL GGUF (73.4 GiB, 3 shards) FETCHED to dgx + its metadata/tensor-map READ AUTHORITATIVELY (814 tensors, arch `laguna`, `expert_gating_func=2` sigmoid, `leading_dense_block_count=1`, `expert_weights_scale=2.5`). Three CPU-verified FIDELITY corrections the W1-W3 scaffold got wrong, each grounded in the real GGUF + llama.cpp: (1) **per-head QK-RMSNorm** (`attn_q_norm`/`attn_k_norm` F32[128]) added to params+forward — the scope MISSED it (surfaces only in the tensor map); (2) **dual-RoPE mscale** now uses llama.cpp's `yarn_attn_factor·(1+0.1·ln(factor))` off the GGUF-authoritative `factor=32`/`yarn_attn_factor=1.0` (256K-ctx build, NOT HF's factor-128/1.4852 1M-ctx scalar) — resolves the numerics-delicate residual; (3) **separate** `ffn_gate_exps`/`ffn_up_exps` (Q4_K) + `ffn_down_exps` (Q5_K) + Q8_0 shared/attn (the scaffold assumed merged gate_up). GGUF keep-quant tower materialization (`Mw`/`Sew` mirror of ds4) + keep-quant `ForwardGguf` (vt::MatmulBT/GemmRowSlice) + the real-model greedy run vs the llama.cpp-laguna same-quant oracle remain the W5 close (73 GB single-GB10, host-orchestrated) | `MODEL-TEXT-laguna-laguna-for-causal-lm` | | 🚫 | `DeepseekV3ForCausalLM` / `DeepseekV32ForCausalLM` | DeepSeek-V3 / V3.2 | HW-blocked (671B, ~642 GiB fp8 vs 119 GiB unified memory); V3.2 additionally DEP-blocked (DSA indexer) | `MODEL-TEXT-deepseek-v2-deepseek-v3-for-causal-lm` | | 🚫 | `GlmMoeDsaForCausalLM` | GLM-5 (DSA) | HW-blocked (1404 GiB bf16) and DEP-blocked (GLM-5.x is DeepSeek-V3.2 verbatim) | `MODEL-TEXT-deepseek-v2-glm-moe-dsa-for-causal-lm` | @@ -428,6 +429,7 @@ Transformers compatibility is capability-driven and excluded from finite counts. | `MODEL-MM-higgs-audio-v3-higgs-audio-v3-talker-for-conditional-generation` | `HiggsMultimodalQwen3ForConditionalGeneration`, `HiggsAudioV3TalkerForConditionalGeneration` | **OUT-OF-REPO** — not in `555967922`, and absent from `registry.py` on vLLM `main`; registered by `vllm-project/vllm-omni` at `vllm_omni/model_executor/models/registry.py:213-217` (and the explicit `HiggsAudioV3TalkerForConditionalGeneration` spelling at `:218`) @ `bbe6ccc512a404a2df8c977ea29003002f2683e8` → module `vllm_omni/model_executor/models/higgs_audio_v3/higgs_audio_v3_talker.py`, class `HiggsAudioV3TalkerForConditionalGeneration` (separate module and class fields rather than the compact colon-pair form, per the out-of-repo convention), with code2wav at `registry.py:223-227` → module `higgs_audio_v3/higgs_audio_v3_code2wav.py`, class `HiggsAudioV3Code2WavForConditionalGeneration`. Documented at `vllm-omni` `docs/models/supported_models.md:81`. Official recipe: `bosonai/higgs-audio-v3-tts-4b` (published at `recipes.vllm.ai`), whose `config.json` declares `architectures: ["HiggsMultimodalQwen3ForConditionalGeneration"]`. The distinct v2 lineage (`HiggsAudioV2ForConditionalGeneration`, `higgs_audio_v2/`) is a DIFFERENT target and has no recipe in the audited set, so it gets no row here | conditional generation / text+audio in, AUDIO OUT (TTS) | MM processor; audio detokenizer / code2wav; audio tokenizer; Qwen3-derived multimodal decoder; streaming audio output | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-MM-voxtral-realtime-voxtral-realtime-for-conditional-generation` | `VoxtralRealtimeForConditionalGeneration` (target-pending) | **TARGET-PENDING — the architecture string is registered NOWHERE we can find.** Recipe `mistralai/Voxtral-Mini-4B-Realtime-2602` (one of the 157 recipes at `vllm-project/recipes` `86c7777a`) declares `architectures: ["VoxtralRealtimeForConditionalGeneration"]` with an inner `audio_config.model_type: "voxtral_realtime_encoder"`, but that exact string appears in none of: (a) the pinned registry `555967922`; (b) any of the ten dictionaries of `vllm/model_executor/models/registry.py` on vLLM `main`; (c) `_OMNI_MODELS` in `vllm_omni/model_executor/models/registry.py` @ `bbe6ccc512a404a2df8c977ea29003002f2683e8`; (d) `vllm-omni` `docs/models/supported_models.md`. **Related but NOT equal, recorded so nobody re-derives it:** `VoxtralRealtimeGeneration` — a DIFFERENT string — is registered at the pin (`registry.py:585`, `main` `:603`) and already has its own row here, `MODEL-MM-voxtral-realtime-voxtral-realtime-generation`; `VoxtralForConditionalGeneration` (pin `registry.py:584`) is the audio→TEXT model; and `vllm-omni` `registry.py:305-309` registers `VoxtralTTSForConditionalGeneration` for the SEPARATE `mistralai/Voxtral-4B-TTS-2603` recipe. The most likely reading is that this checkpoint's `config.json` names a renamed or not-yet-landed spelling of the existing `VoxtralRealtimeGeneration` target, but that is a hypothesis: no anchor is asserted until one of those is shown to be what actually resolves. Tracked by [#610](https://github.com/mudler/vllm.cpp/issues/610) | conditional generation / audio in, AUDIO OUT (realtime speech-to-speech) — modality inferred from the recipe and config, NOT from an upstream implementation | unresolved until the target is located | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-MM-bailing-mm-native-bailing-mm-native-for-conditional-generation` | `BailingMMNativeForConditionalGeneration` (target-pending) | **TARGET-PENDING — the architecture string is registered NOWHERE we can find.** Recipe `inclusionAI/Ming-omni-tts-0.5B` (one of the 157 recipes at `vllm-project/recipes` `86c7777a`) declares `architectures: ["BailingMMNativeForConditionalGeneration"]`, but that exact string appears in none of: (a) the pinned registry `555967922`; (b) any dictionary of `vllm/model_executor/models/registry.py` on vLLM `main`; (c) `_OMNI_MODELS` in `vllm_omni/model_executor/models/registry.py` @ `bbe6ccc512a404a2df8c977ea29003002f2683e8`; (d) `vllm-omni` `docs/models/supported_models.md`. **Related but NOT equal:** `vllm-omni` `docs/models/supported_models.md:75` lists that SAME checkpoint under a different architecture name, `MingTTSForConditionalGeneration` (`registry.py:250-254` → module `ming_tts/ming_tts.py`, class `MingTTSForConditionalGeneration`); and `vllm-omni` `registry.py:417-421` carries a near-miss alias `BailingMM2NativeForConditionalGeneration` — note the `2` — commented "HF repo currently ships this architecture name in config.json", pointing at module `ming_flash_omni/ming_flash_omni.py`, class `MingFlashOmniForConditionalGeneration`, for the DIFFERENT Ming-flash-omni-2.0 checkpoint. So the shape of the gap is a missing config-string alias upstream, but that is an observation, not an anchor, and none is asserted here. Tracked by [#610](https://github.com/mudler/vllm.cpp/issues/610) | conditional generation / text+image+audio in, AUDIO OUT (TTS) — modality from the recipe and config, NOT from a resolved upstream target | unresolved until the target is located | ☐ required | `INVENTORIED` | none | unassigned | +| `MODEL-DIFFUSION-ltx-2-5-ltx2-video-transformer-3d-model` | `LTX2VideoTransformer3DModel` | **BEYOND-PIN AND OUT-OF-REPO** — not in `555967922` (LTX-2.5 released 2026-08) and not in the vLLM repository at all. Architecture reference is Lightricks' own `LTX-2` (`packages/ltx-core/src/ltx_core/`: `model/transformer/{transformer,model,attention,rope,model_configurator}.py`, `model/audio_vae/`, `components/`, `guidance/`). vLLM-Omni carries an `ltx2` module but its recipes stop at 2.3 (`vllm_omni/diffusion/models/ltx2/ltx2_recipes.py:162-166`); 2.5 is upstream-OPEN at [vllm-omni#6066](https://github.com/vllm-project/vllm-omni/issues/6066) and [#4985](https://github.com/vllm-project/vllm-omni/issues/4985). Serving oracle reaches 2.5 through the GENERIC `DiffusersAdapterPipeline` (`pipeline_diffusers_adapter.py:116`) against `Lightricks/LTX-2.5-Diffusers`. | diffusion generation / video + audio (text/image/video in, video+audio out) | flow-matching denoise loop; TWO coupled modality streams with audio<->video cross-attention; per-head gated attention; AdaLN-single with 9-vector per-block tables; timestep-INDEPENDENT prompt K/V (cacheable once per request); split/interleaved 3D RoPE; gelu-approximate FFN; Gemma-4 12B text encoder with dual caption projections (4096 video / 2048 audio); Conv video VAE + DiffVAE; audio VAE + vocoder; latent spatial/temporal x2 upsamplers; duration head; distilled two-stage recipe | [ltx-2.5](specs/ltx-2-5.md) | `SPIKE` | **L0 LANDED 2026-08-11 (#435).** Spec committed BEFORE implementation per AGENTS.md. Geometry measured from the ungated `vonkaiser/LTX-2.5-FP8-NVFP4` FP8 checkpoint header by range request — the same no-payload technique used for H3's manifests: 21.00B params, 48 blocks, 1775 F8_E4M3 tensors each with a per-tensor F32 `weight_scale`, biases/norms BF16. Cross-modal projection shapes are ASYMMETRIC and are a named test trap: `audio_to_video_attn.to_q` is [2048, 4096] while `to_k`/`to_v` are [2048, 2048] and `to_out` is [4096, 2048], so a transposition still type-checks against a square assumption. **CHECKPOINT ACCESS (verified against the HF API 2026-08-11):** `Lightricks/LTX-2.5` is `gated: auto` (accepting the license opens it; holds the first-party NVFP4 DiT at 18.72 GB), `Lightricks/LTX-2.5-Diffusers` is RESTRICTED behind manual approval (needed for the binding oracle), and `vonkaiser/LTX-2.5-FP8-NVFP4` is UNGATED and unblocks L1-L2 today. All artifacts land under `$CHECKPOINT_ROOT` (`/mnt/nas_share/checkpoints`) so dgx.casa and the cluster nodes share one copy. **PHASES (one PR, developer-directed):** L1 `vllm::multimodal::VideoEngine` interface + checkpoint-detected registry with MiniMax-H3 moved behind it UNCHANGED (gated on frames+WAV byte-identical to the pre-refactor fold fixture) and ABI **v13 by ADDING fields only**; L2 DiT forward; L3 Gemma-4 TE reusing `gemma4.cpp`; L4 Conv video VAE + audio VAE + vocoder; L5 pipeline/recipes/upsampler/duration head; L6 NVFP4 arms + GB10 load-time residency; L7 e2e on dgx.casa under `flock $HOME/gpu.lock`. **OPEN, same as H3:** there is no vllm-omni parity PIN — `.agents/upstream-sync.md` covers the vLLM repo only, so each golden records the vllm-omni SHA inline. | `CLAIM-LTX25-L0-L7` | | `MODEL-MM-moss-transcribe-diarize-moss-transcribe-diarize-for-conditional-generation` | `MossTranscribeDiarizeForConditionalGeneration` (v0.25.0 target-pending) | v0.25.0 target `registry.py:450-453`; `vllm/model_executor/models/moss_transcribe_diarize.py::MossTranscribeDiarizeForConditionalGeneration` @ `702f481` | conditional generation / audio | MM processor; Whisper encoder; VQ adaptor; Qwen3 decoder; speech-to-text/diarization frontend | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-MM-lightonocr-light-on-ocrfor-conditional-generation` | `LightOnOCRForConditionalGeneration` | `registry.py:450-453`; `vllm/model_executor/models/lightonocr.py::LightOnOCRForConditionalGeneration` | conditional generation / image | MM processor; encoder/merge; vision encoder | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-MM-lfm2-vl-lfm2-vlfor-conditional-generation` | `Lfm2VlForConditionalGeneration` | `registry.py:454`; `vllm/model_executor/models/lfm2_vl.py::Lfm2VLForConditionalGeneration` | conditional generation / image | MM processor; encoder/merge; Mamba/SSM state; vision encoder; video path | ☐ required | `INVENTORIED` | none | unassigned | diff --git a/.agents/porting-inventory.md b/.agents/porting-inventory.md index 18ab32e05..7f5c928e4 100644 --- a/.agents/porting-inventory.md +++ b/.agents/porting-inventory.md @@ -1055,6 +1055,555 @@ Examples: `examples/cli` ✅ (C-API client), `examples/server` ✅ (OpenAI serve #51655 merging plus a pin advance that includes it; until then the row carries this deviation. Scope and gates: [muse-glimmer spec](specs/muse-glimmer.md) §0. +17. **From-necessity dense non-causal CROSS attention (`vt::OpId::kAttentionCross`, + 2026-08-11, `MODEL-DIFFUSION-ltx-2-5-ltx2-video-transformer-3d-model` phase L2, + issue [#435](https://github.com/mudler/vllm.cpp/issues/435)).** The pinned vLLM + (`555967922`, 0.26) has no dense non-causal attention op whose QUERY and KEY + token counts may differ: every attention entry point it exposes is a paged or + causal decode/prefill path, and `vt::Attention` inherited that assumption and + REJECTS `Tq != S` — which no cross-attention can satisfy. LTX-2.5's DiT is + four cross-attentions per block (text→video, text→audio, audio→video, + video→audio), so the op is a precondition for the model, not a convenience. + * **Upstream semantics mirrored:** torch + `scaled_dot_product_attention(q, k, v, attn_mask=..., is_causal=False)` as + LTX's `PytorchAttention` calls it — Lightricks/LTX-2 @ `fd4ded7f`, + `packages/ltx-core/src/ltx_core/model/transformer/attention.py:97-102`. The + additive-bias contract (both the `(mask - 1) * finfo.max` prompt form and the + log-space self-attention STRENGTH form, dense `[Tq, S]` and key-only `[1, S]`) + mirrors `transformer_args.py:199-237`. + * **Written from scratch** in the sense §9.1 means: the three-pass + max-subtract / exp / weighted-sum STRUCTURE is this tree's own + `AttentionKernel` (`src/vt/cpu/cpu_ops.cpp`) with the key extent taken from + the KEY's token count and the optional bias added before the max, so the two + agree bit-for-bit on a square unbiased call. No vLLM kernel was ported. + * **Local anchor:** `include/vt/ops.h` (`OpId::kAttentionCross` appended before + `kCount` — no id shift — plus `AttentionCrossArgs` and the declaration), + `src/vt/ops.cpp` (`vt::AttentionCross` validation), `src/vt/cpu/cpu_ops.cpp` + (`AttentionCrossKernel`, the CPU reference). Sole caller: `vllm::Ltx2Attention` + (`src/vllm/model_executor/models/ltx2.cpp`), which routes on `context == + nullptr` — the call's MEANING — so the op a call site dispatches never depends + on the prompt length. + * **Backends:** CPU kernel, and — since phase L8 (2026-08-12) — a NATIVE CUDA + kernel, `src/vt/cuda/cuda_attention_cross.cu`. The CUDA one was owed to "the + LTX-2.5 device-resident forward, which is the first caller that would need + it", and that caller arrived. It had to arrive WITH it: before it the op had + a CPU kernel only, and on GB10 `Backend::UnifiedMemory()` is true, so + `RegisterReferenceTier` would have installed the CPU kernel for the CUDA + device and every cross-attention in the DiT — six per block — would have run + on the HOST while every gate stayed green and "it ran on the GPU" was false. + The CUDA kernel is a structural port of `AttentionDenseFlashKernel` + (`src/vt/cuda/cuda_ops.cu:3229-3318`, itself a 1:1 port of the vendored FA2 + `compute_attn_1rowblock`), generalized on the three axes `AttentionCrossArgs` + exists for: the key extent is KEY's own `S`, there is no causal mode, and an + optional f32 additive bias joins the SCALED score before the max-subtraction. + It uses the online-softmax recurrence where the CPU kernel uses the explicit + three-pass, so the two agree to f32 summation-order slack and are NOT + bit-identical — the same relationship `AttentionDenseFast` already has with + `AttentionKernel`. + * **Tests and evidence:** `tests/vllm/models/test_ltx2.cpp` — the validation + refusals, the fully-masked-key softmax, the DENSE `[Tq, S]` per-query bias + rows, the `Hq > Hkv` GQA broadcast, bit-for-bit agreement with `vt::Attention` + on a square unbiased call, the no-provider refusal, and the six full-DiT + forward goldens (`scripts/gen-ltx2-goldens.py`, upstream `fd4ded7f`) that run + it inside the model. Those are all CPU-only. + **`tests/vt/test_ops_attention_cross.cpp` is the DIRECT gate on both + backends (2026-08-12).** It exists because of a finding recorded here so it + cannot recur: when the CUDA kernel landed, every test that reached it went + through the LTX-2.5 DiT, and all six distinct call geometries that model + produces are `tiles=1 npl=1 nblk=1 Hq==Hkv` — so the kernel's whole + flash-tiling machinery was DEAD CODE in every gate, including the + `ChooseTileCols` halving whose own comment says it prevents a launch failure + "on exactly the real geometry". A render puts `S` at prompt length and `Tq` + in the thousands, straight into that regime. The new suite drives + `tiles>1`, `nblk>1` with a RAGGED last block, `npl>1`, the `Hq>Hkv` GQA + broadcast, the `head_dim = 128` f32 case where the tile MUST halve + (`2*64*128*4` = 64 KiB against a 48 KiB launch limit), a bf16 stream at that + head_dim, a dense `[Tq, S]` bias carried across a tile boundary, and a fully + masked key placed in a LATER tile. Each case asserts the tile/block/lane + counts it reaches before asserting the numbers, and the oracle is an + INDEPENDENT f64 host reference in the test file — not the CPU kernel — so + the CUDA arm is not gated against a helper that could be wrong in the same + direction; the CPU-vs-CUDA comparison is kept as a separate, tighter + statement. + * **Spec:** [ltx-2.5 spec](specs/ltx-2-5.md) §1.2 and §7. Lifecycle: shipped + (CPU + CUDA). Owner: the LTX-2.5 row. +18. **LTX-2.5 pipeline recipes are sourced from the CROSS-CHECK, and three upstream + guiders are refused (2026-08-12, + `MODEL-DIFFUSION-ltx-2-5-ltx2-video-transformer-3d-model` phase L5, issue + [#435](https://github.com/mudler/vllm.cpp/issues/435)).** Two deviations, both + forced, both recorded rather than discovered later. + * **(a) The 2.4 and 2.5 recipe rows have NO binding-oracle source.** vLLM-Omni + is this project's binding oracle for LTX (spec §3) and its recipe table stops + at 2.3: `_PIPELINE_RECIPES` keys on exactly `("one_stage","2")`, + `("one_stage","2.3")`, `("distilled_two_stage","2")`, `("dmd2","2")`, + `("dmd2","2.3")` (`vllm_omni/diffusion/models/ltx2/ltx2_recipes.py:161-166` @ + `a4ea67a2`). The three rows this port adds — `("one_stage","2.4")`, + `("one_stage","2.5")`, `("distilled_two_stage","2.5")` — take their VALUES + from Lightricks `ltx-pipelines` @ `fd4ded7f`, which is the model author's own + runtime and the spec's designated cross-check: `utils/constants.py:17-23` + (the distilled sigmas), `:130-179` (`_PARAMS_SINCE_VERSION` / + `detect_params`, which is what makes (2,5) inherit `LTX_2_4_PARAMS`), and + `distilled.py:62-84, 170-185` (the ancestral stage-1 sampler, the single + thing generation 2.5 changes). The SHAPE of the recipe model stays + vLLM-Omni's. The refusal on an unknown pair is mirrored exactly and never + relaxed, because a plausible-but-wrong sigma schedule renders rather than + failing. **The two references disagree on the default negative prompt** — + Lightricks' carries five leading tags vLLM-Omni's lacks — so both strings are + kept, each row takes its own source's, and the disagreement is a gated value + (`kLtx2NegativePromptsAgree`) rather than a preference. Discharged when + vllm-omni lands native 2.5 (upstream + [vllm-omni#6066](https://github.com/vllm-project/vllm-omni/issues/6066)), at + which point the rows re-anchor to it. + * **(b) `CFGStarRescalingGuider`, `LtxAPGGuider` and `LegacyStatefulAPGGuider` + are REFUSED, not ported — because NOTHING UPSTREAM CONSTRUCTS THEM.** All + three appear in the whole `LTX-2` tree only at their own `class` statements + (`components/guiders.py:31, 78, 129`); every pipeline builds + `MultiModalGuider` from `MultiModalGuiderParams` + (`utils/constants.py:49-68`). They are an unported arm, refused by name and + recorded as owed. `CFGGuider`, `STGGuider`, `MultiModalGuider` and + `projection_coef` itself ARE ported. + + **CORRECTION, 2026-08-12.** This entry previously justified the refusal on a + SHAPE claim — that `projection_coef`'s rank-2 `(B, 1)` result "RAISES at + every rectangular rank >= 3, i.e. at every real `(B, C, F, H, W)` video + latent". That premise is FALSE and was recorded as a golden, which is the + failure spec §7.0(b) exists to prevent. Re-measured against upstream: torch + right-aligns `(B, 1)` onto the last two axes, so the real predicate is + `B > 1 && shape[-2] not in {1, B}`. At **B = 1** — the ordinary + single-request video latent — it composes AND is numerically correct, + because `(1, 1)` is just a scalar; `(2, 128, 8, 2, 16)` composes too. Where + it composes with `B > 1` it is silently WRONG, applying the per-batch + coefficient along axis -2 rather than the batch axis. The + `norm(dim=[-1,-2,-3])` in the two threshold arms (`:114`, `:205`) is a + SEPARATE constraint needing rank >= 3. The measured matrix now carries + `B = 1` and `shape[-2] == B` rows and the `square` abstraction — a + mis-generalization of those two axes — is gone. + * **Local anchor:** `include/vllm/model_executor/models/ltx2_pipeline.h`, + `src/vllm/model_executor/models/ltx2_pipeline.cpp` + (`ResolveLtx2PipelineRecipe`, `Ltx2Guidance`). + * **Tests and evidence:** `tests/vllm/models/test_ltx2_pipeline.cpp` — the + recipe table gated against vLLM-Omni's OWN key list (so a row appearing + upstream fails this gate instead of going unnoticed), the refusal on six + unknown pairs, the ancestral-sampler split between the 2.0 and 2.5 distilled + rows, and `kLtx2GuideProbeComposes`, the shape matrix taken from executing + upstream that is the evidence behind (b). + * **Spec:** [ltx-2.5 spec](specs/ltx-2-5.md) §3 and §6 (phase L5). Lifecycle: + shipped (CPU reference). Owner: the LTX-2.5 row. +19. **torchao NVFP4 group-scale UNSWIZZLE (`Ltx2UnswizzleNvfp4BlockScale`, + 2026-08-12, `MODEL-DIFFUSION-ltx-2-5-ltx2-video-transformer-3d-model` phase L6, + issue [#435](https://github.com/mudler/vllm.cpp/issues/435)).** LTX-2.5's + shipped Gemma-4 text encoder is quantized by **torchao**, not + compressed-tensors and not modelopt. Measured from the file's own + `.torchao_nvfp4` U8[240] marker rather than assumed: `{"format": + "torchao_nvfp4", "block_size": 16, "is_swizzled_scales": true, ...}`. The + ENCODING is identical to the modelopt W4A16 path we already have — E2M1 nibble + pairs, one fp8-e4m3 scale per 16 inputs, one F32 `weight_scale_2` used as a + MULTIPLIER. The single delta is that the group-scale tensor is stored in the + cuBLAS block-scaling-factors layout, so it needs inverting before the existing + dequant can read it. **No new quant scheme was added.** + * **Upstream semantics mirrored:** the permutation vLLM's own producers apply — + `swizzle_blockscale`, `vllm/model_executor/layers/quantization/utils/ + nvfp4_utils.py:44-49`, and `to_blocked`, + `vllm/model_executor/layers/quantization/qutlass_utils.py:165-180`, at the + parity pin `555967922`. The two are the same permutation written twice; + `qutlass_utils.py`'s own header records it as a copy of + `torchao/prototype/mx_formats`, i.e. of the module that quantized this + checkpoint. NVIDIA's layout reference: + . + * **Written from scratch** in the sense §9.1 means: only the INVERSE, as an + explicit index formula. Both upstream writings are forward-only, and + `swizzle_blockscale` calls `.cuda()` unconditionally, so there is nothing to + port verbatim. The decode itself is `DequantNvfp4ToBf16` + (`include/vllm/model_executor/model_loader/nvfp4_dequant.h:59`), UNCHANGED. + * **Local anchor:** `include/vllm/model_executor/models/ltx2_loader.h`, + `src/vllm/model_executor/models/ltx2_loader.cpp` + (`Ltx2UnswizzleNvfp4BlockScale`, `ParseLtx2TorchaoNvfp4Marker`, + `Ltx2DequantTorchaoNvfp4ToBf16`). + * **Tests and evidence:** `tests/vllm/models/test_ltx2_loader.cpp`, three ways — + the five swizzle cases in `tests/vllm/models/ltx2_quant_goldens.inc`; a + 512-byte tile of the SHIPPED text encoder's own `weight_scale` with its values + decoded by torch's fp8-e4m3; and the two REAL manifests + (`ltx2_fp8_dit_manifest.inc`, 6124 tensors; `ltx2_nvfp4_te_manifest.inc`, + 1688), captured from the files' own headers with no payload read. The swizzle + oracle RUNS: `scripts/gen-ltx2-quant-goldens.py` loads vLLM's own `to_blocked` + out of the pinned checkout with `importlib.util.spec_from_file_location` and + calls `to_blocked(x, backend="torch")`. The earlier "not importable, so it + cannot be executed here" limit was WRONG — `import vllm.*` does die in + `vllm.distributed` on a missing `zmq`, but `qutlass_utils.py` needs only + torch, `vllm.triton_utils` and `vllm.utils.math_utils`, so loading the file + directly runs the real producer. Regenerating against it left every golden + byte IDENTICAL to the transcription's, and upstream's executing `to_blocked` + reproduces the shipped file's first 512 `weight_scale` bytes from the emitted + linear tile. + * **OWED, and precisely:** the `backend="triton"` arm of `to_blocked` and + `swizzle_blockscale` — vLLM's other two writings of the same permutation — + are pinned by source fragment only. Neither can execute on this host (no + active Triton driver; `swizzle_blockscale` calls `.cuda()` unconditionally), + so running them needs a GPU host. **CLOSED at L7 (2026-08-12):** Lightricks' + first-party NVFP4 DiT + (`ltx-2.5-22b-distilled-transformer-nvfp4.safetensors`, 18.72 GB) was behind + an un-accepted HF gate when this was written and is now on the NAS; the + SHIPPED file's own header is read by + `tests/vllm/multimodal/test_ltx2_video.cpp` (`LTX2_CHECKPOINT_ROOT`), which + resolves it onto the L2 contract and checks its declared config against the + shape-derived one. Still owed: the torchao arm of the Gemma TOWER itself + (`Gemma4Weights`) is not + wired — `ltx2_text_encoder.h` declares no tower contract, so L6 loads the two + caption projections, the asset pack and the geometry, and VALIDATES every + tower module without materializing it. **CLOSED at L10 (2026-08-12):** + `Ltx2LoadGemmaTowerFromSafetensors` (ltx2_text_encoder.h/.cpp) materializes + the tower onto `Gemma4Weights`, reusing L6's + `Ltx2DequantTorchaoNvfp4ToBf16` unchanged. It reads the FLAT + `model.layers.{i}.*` names the LTX file ships — not the + `model.language_model.layers.{i}.*` form `gemma4_weights.cpp` reads, which + is why the tower could not simply go through that loader — and resolves + each layer's geometry from `layer_types`, `global_head_dim` and + `num_global_key_value_heads` rather than from the stored tensor widths, + which are HALF the logical ones under NVFP4. It REFUSES by name: a PLE + config against a checkpoint with no PLE tensors, `num_kv_shared_layers` + != 0, a `v_proj` present or absent against what `attention_k_eq_v` + declares, and a module in neither the BF16 nor the torchao-NVFP4 form. + The Gemma config is an INPUT and is never inferred: the shipped + `vonkaiser` build has NO `__metadata__` at all, and `layer_types`, + `global_head_dim`, `num_global_key_value_heads`, `attention_k_eq_v` and + both `rope_parameters` entries each move every hidden state while leaving + the tensor set byte-identical. + * **Spec:** [ltx-2.5 spec](specs/ltx-2-5.md) §1.4 and §6 (L6). Lifecycle: + shipped (host + load-time device staging). Owner: the LTX-2.5 row. + * **CLOSED 2026-08-13 by phase L9a — and the diagnosis it closes was WRONG.** + This entry previously recorded that the FIRST-PARTY + `ltx-2.5-22b-distilled-transformer-nvfp4.safetensors` stores `weight_scale` + in the LINEAR `[N, K/16]` layout, and that a linear read was the missing + piece. **It does not, and it was not.** The bytes are SWIZZLED — the same + permutation this entry already inverts — merely declared in the + cuBLAS-padded framing `[round_up(N,128), round_up(G,4)]` = `[4096, 256]` + instead of torchao's `to_blocked` `[32*ceil(N/128), 16*ceil(G/4)]` = + `[1024, 1024]`. For every layer in that file `N % 128 == 0` and + `G % 4 == 0`, so the padded framing is NUMERICALLY IDENTICAL to the linear + shape; that coincidence is what made the wrong diagnosis look right, and it + is why no shape test could have settled it. + + The file additionally packs element `2j` in the HIGH nibble, which no shape + encodes at all. Both facts were established by correlating the dequantized + weights against the `vonkaiser` FP8 DiT of the same base weights (0.9956 + with 9.46% relative rms, against 0.0004-0.26 for every other reading), and + independently confirmed in Lightricks' own runtime + (`ltx-kernels/docs/NVFP4.md:27-29`, `csrc/nvfp4/quantize.cu:26-31`, + `ltx-core/quantization/nvfp4/linear.py:6-7`). + * **Resolution.** `Ltx2ResolveNvfp4Producer` discriminates on the + `torchao_nvfp4` marker (torchao always writes one, so its absence excludes + torchao), corroborates with the framing, and REFUSES by name on any other + combination. `Ltx2UnswizzleNvfp4BlockScale` was already framing-agnostic and + is UNCHANGED. The nibble order became an `Nvfp4NibbleOrder` parameter on + `DequantNvfp4ToBf16`, defaulting to low-first so every pre-existing caller + is untouched by construction — see + [nvfp4-nibble-order spec](specs/nvfp4-nibble-order.md), which also records + why H3's nibble-swap-at-load was NOT reused and when it must be. + +19. **LTX-2.5 phase L7 — the family behind `vllm::multimodal::VideoEngine`, and + the driving loop.** + * **Upstream source:** Lightricks/LTX-2 @ `fd4ded7`, + `packages/ltx-pipelines/src/ltx_pipelines/` — `distilled.py:186-300` + (`DistilledPipeline.__call__`), `utils/blocks.py:500-582` + (`DiffusionStage.__call__`) and `:212-235` (`_build_state`), + `utils/samplers.py:39-79` (`euler_denoising_loop`) and `:488-558` (the + ancestral driver), `utils/helpers.py:428-447` (`create_noised_state`), + `:462-464` (`post_process_latent`), `:466-503` + (`modality_from_latent_state`, `timesteps_from_mask`), + `utils/denoisers.py:214-252` (`SimpleDenoiser`); plus `packages/ltx-core` + `model/transformer/model.py:590-604` (`X0Model.forward`), `utils.py:38-50` + (`to_denoised`), `types.py:70,108-123,164-200` (the scale factors and both + latent-shape derivations), `tools.py:139-184` / `:246-280` (the two + `LatentTools`), and the four configurators + (`video_vae/model_configurator.py:21-24,81-94,255-265`, + `audio_vae/model_configurator.py:13-39,49-88,108-141,184-190`, + `upsampler/model_configurator.py:12-30`). + * **Written from scratch**, and recorded as such: the noise STREAM. Upstream + draws from a seeded `torch.Generator`; reproducing torch's stream + bit-exactly decides WHICH sample comes out, not whether the pipeline is + right, so `SplitMixGaussian` is a documented splitmix64 + Box-Muller source + drawn in upstream's own ORDER (video before audio, one draw per state per + step) and is NOT torch's. Same call MiniMax-H3 made + (`minimax_h3.h:1895-1897`). The cost is stated where it is taken: a clip + rendered here is a different sample from the same distribution, so it is + not comparable to an upstream render frame by frame, and sample-level + comparison needs the noise supplied from outside. + * **Local anchor:** `include/vllm/multimodal/ltx2_video.h`, + `src/vllm/multimodal/ltx2_video.cpp`; the loaders it needed are + `Ltx2LoadVaeWeights`, the three `SDOps` key-rule sets and the four config + parsers in `ltx2_loader.{h,cpp}`. + * **Tests and evidence:** `tests/vllm/multimodal/test_ltx2_video.cpp` — + registration, detection by tensor name (prefixed and de-prefixed), a + STRUCTURAL e2e over a reduced-dimension checkpoint set written in the + shipped file format, the same generation driven through `include/vllm.h` + alone, every refusal by name, and the shipped Lightricks checkpoints when + `LTX2_CHECKPOINT_ROOT` is set. + * **OWED, and precisely:** (a) the forward on an accelerator — CLOSED by + phase L8, entry 20 below; (b) the Gemma-4 tower, so a PROMPT + cannot be encoded and conditioning is prompt-embeds only; (c) image / + keyframe / reference conditioning, which needs the video VAE's ENCODER; + (d) the full-scale render, which at 21.00B needs ~76 GB of f32 weights and + ~2.6e14 FLOPs per denoise step; (e) parity against the BINDING oracle and + any speed number, both structurally pending per the spec's §0 and §3. + * **Spec:** [ltx-2.5 spec](specs/ltx-2-5.md) §5 and §6 (L7). Lifecycle: + shipped (CPU, structural e2e). Owner: the LTX-2.5 row. +20. **LTX-2.5 phase L8 — the DEVICE-RESIDENT DiT forward, and the `vt::OpId::kLtx2` + glue table (2026-08-12, + `MODEL-DIFFUSION-ltx-2-5-ltx2-video-transformer-3d-model` phase L8, issue + [#435](https://github.com/mudler/vllm.cpp/issues/435)).** Phase L7 wired + LTX-2.5 through `vllm::multimodal::VideoEngine` and had to REFUSE `device = 1`, + because L2's forward was f32-only by declaration and L6's staging was bf16 and + refused to widen. This is the forward that makes the two meet: the same graph + with every activation in device memory and the stream in the checkpoint's own + dtype. + * **Upstream semantics mirrored:** unchanged from L2 — Lightricks/LTX-2 @ + `fd4ded7f`, `packages/ltx-core/.../transformer/{model,transformer,attention, + adaln,rope,feed_forward}.py`. This entry adds no new upstream BEHAVIOR; it is + the same `LTXModel.forward` at a different residency, which is why it is + gated against the SAME frozen goldens rather than new ones. + * **Dtype polarity, which is the design decision:** the device stream is + **bf16**, because upstream resolves ONE model dtype and every layer inherits + it (`model.py` has no per-layer dtype at all) and bf16 is what + `Ltx2StreamDitToDevice` already puts on the device. `kF32` is accepted as a + GATE arm only — it is the L2 parity dtype, and it is what lets this forward + be compared against `ltx2_goldens.inc` at f32 round-off instead of at a bf16 + band. Nothing widens a bf16 load to reach it. The `scale_shift_table` family + stays F32 on both arms because the CHECKPOINT stores it F32 + (`ltx2_loader.h:64-66`); those are a few kilobytes against a 21 GB model. + * **Written from scratch** in the sense §9.1 means: seven small kernels the + shared `vt::` surface does not express — `ada_value`, `modulate`, + `add_gated`, `gate_heads`, `rope` (LTX's split and interleaved layouts), + `output_modulate`, and plain ungated `silu`. Each is a 1:1 transcription of a + named host helper in `ltx2_dit.cpp` / `ltx2.cpp`, in the same arithmetic + order. Everything else reuses tuned shared ops (`vt::MatmulBT`, `vt::Add`, + `vt::RmsNorm`, `vt::LayerNorm`, `vt::GeluTanh`, `vt::Attention`, + `vt::AttentionCross`); no new GEMM, norm or attention kernel was added. + * **Local anchor:** `include/vllm/model_executor/models/ltx2_device.h`, + `src/vllm/model_executor/models/ltx2_device.cpp`, + `src/vllm/model_executor/models/ltx2_device_resolve.cpp` (a separate TU so + the `vt::GetOp` cast links in CPU-only builds), `src/vt/cpu/cpu_ltx2.cpp`, + `src/vt/cuda/cuda_ltx2.cu`, and `vt::OpId::kLtx2` appended before `kCount` + (no id shift). Registered on BOTH `kCPU` and `kCUDA`, so the port's STRUCTURE + is covered by CPU CI and a GPU gates the KERNELS. + * **Tests and evidence:** `tests/vllm/models/test_ltx2_device.cpp`, its own + target so `test_ltx2`'s 29/1615 baseline does not move. It runs all six + upstream forward cases device-resident against the SAME `ltx2_goldens.inc` + the CPU arm meets, the bf16 stream against those goldens at a bf16 band PLUS + a check that it actually DIFFERS from f32 (a bf16 arm that matched f32 would + mean the dtype policy silently was not applied), a CUDA-vs-host comparison at + identical inputs, and every refusal by name. + * **FOUND BY THE bf16 ARM, which is why that arm exists.** The staging + predicate matched the `scale_shift_table` SUFFIX, which silently excluded + `scale_shift_table_a2v_ca_video` / `..._audio`; those were staged bf16 and + then read through `ada_value`'s `const float*` table parameter. The + audio<->video cross gate became 2.85e32 and the video stream 6.89e30 after + one block — while the f32 arm, where the mismatch cannot arise, stayed green + at 1e-7. The predicate is now a SUBSTRING match and every table read is + guarded by an explicit `CheckTableF32`, so a future miss is a named refusal + rather than a reinterpretation. + * **MEASURED ON GB10 (2026-08-12), and it is where the shipped model stands.** + A 21.00B DiT stages and runs device-resident: vonkaiser + `ltx-2.5-22b-distilled-fp8.safetensors` (21.0 GB, 6124 tensors) staged in + 271.5 s and one bf16 forward at 48 layers / inner 4096 / head_dim 128 + produced finite, non-degenerate output. The wall-clock is SIZING ONLY and is + not a speed result — spec §0, no production-configuration denominator exists. + The FIRST-PARTY `ltx-2.5-22b-distilled-transformer-nvfp4.safetensors` + (18.72 GB, 7876 tensors) did NOT stage at that time. **The reason recorded + here was WRONG and is superseded by entry 18 above (phase L9a, + 2026-08-13):** the file does not store a LINEAR `[N, K/16]` scale. Its bytes + are SWIZZLED, declared in the cuBLAS-padded framing — which for these + geometries is numerically identical to the linear shape, which is precisely + why the wrong reading looked confirmed — and it packs the OPPOSITE nibble + order. The refusal was correct; its diagnosis was not. Loading it needed a + producer discriminator and a nibble-order arm, not a linear read. Phase L7's + shipped-checkpoint test only parsed the MANIFEST, which is why this surfaced + as late as it did: nothing had materialized a tensor from that file before. + * **CORRECTED 2026-08-12 by an adversarial review of L7+L8 (row + `LTX25-L8-FIX`). The headline held; five records did not.** + 1. **The FP8 DiT this row RAN carries no `__metadata__` at all.** The engine + comment claimed "both shipped LTX-2.5 DiTs DO carry one". Read from the + NAS: the first-party NVFP4 file declares + `['config','gemma_source_checkpoint','license','model_version']`; the + vonkaiser FP8 file declares NO `__metadata__` key. So for the copy L1-L6 + gated against and L8 ran on the GPU, the config-adoption branch never + executed and the DiT silently took `double_precision_rope = false` and + `av_ca_timestep_scale_multiplier = 1` against LTX-2.5's declared + `float64` and `1000`. Both move every RoPE angle and every audio<->video + modulation. The run's own claims survive it — the weights stage, the + forward executes device-resident, the output is finite and + non-degenerate, none of which depends on the RoPE precision — but the + CONFIGURATION was unstated, which is what made it a defect. A DiT that + declares no config is now REFUSED by the engine unless the caller names + one through the `dit_config_path` extra, mirroring how `model_version` + was already handled; the device gate resolves it through the same + `Ltx2AdoptDeclaredDitParams` and ASSERTS which configuration it ended up + running under, in both branches. + 2. **The L7 in-flow repair was ungated.** Deleting `im.dit.params = + declared;` left the whole suite green. `Ltx2VideoEngine::dit_params()` + now exposes what the ENGINE loaded and `test_ltx2_video.cpp` asserts on + it against a manifest control; the same deletion now fails 2 cases / 4 + assertions. The disagreement-refusal branch, also unexercised, is gated + on both the declared and the supplied config. + 3. **`test_ltx2_device`'s recorded GB10 baseline was 12/547; it is 13/547.** + The env-gated shipped-checkpoint case is still COUNTED when it skips. + Assertions were right. **After this row it is 13/552 on GB10** — the F9 + cross-backend bf16 arm adds exactly five assertions, and the CPU-backend + arm stays at its own 13/498. + 4. **"bit-identical" overstated the bf16 evidence.** What was measured is + that both arms print the same max|diff| against the goldens at six + significant figures, which is consistent with bit-identity and does not + establish it; nothing compared the two backends' outputs to each other at + bf16. That comparison now exists inside the CUDA-vs-host case, and is + asserted as a bf16 BOUND with the measured value printed. + 5. **`device = N` relabelled a queue instead of selecting one.** The load + called `Backend::CreateQueue()` — which `backend.h:212-217` records as a + temporary index-0 shim — and then overwrote `queue->device.index`, so on + a multi-GPU host `device = 2` would have run on GPU 0 while every + residency check agreed it was index 1. It now uses the free function + `vt::CreateQueue(Device)` and refuses an index with no registered + backend, by name. Unreachable on single-GPU GB10, which is why it had to + be fixed before a second device exists. + * **RE-VERIFIED ON GB10 2026-08-12 (row `LTX25-L8-FIX`), one `flock` hold, + 2305 s of waiting behind a foreign holder.** The shipped vonkaiser FP8 DiT + (6124 tensors) staged in 185.9 s and one bf16 forward at 48 layers / inner + 4096 / head_dim 128 ran device-resident in 0.42 s (SIZING ONLY, not a speed + result — spec §0). Output finite and non-degenerate: absmax video 0.300781, + audio 2.14062. `VT_OP_PROVIDER_STATS=1` on that run: **8 distinct ops on + device=1, all `vt-native`, ZERO `vt-cpu-ref` and ZERO ``** — so the + forward is device-native and the reference tier never served it. + `test_ltx2_device` 13/552 SUCCESS. + **The CUDA `AttentionCross` tiling is now MUTATION-PROVED**, four mutants + built unlocked and executed inside the hold. Three are INVISIBLE to every + LTX-2.5 gate and caught only by `test_ops_attention_cross`: the bias read by + within-tile column instead of absolute key (5 cases red / device gate + SUCCESS), the GQA kv-head `h/(hq/hk)` replaced by `h%hk` (1 case red / + device gate SUCCESS — identical whenever `Hq == Hkv`, which is every + LTX-2.5 geometry, so it is invisible there BY CONSTRUCTION), and the + shared-memory limit raised so `ChooseTileCols` never halves (device gate + SUCCESS; the direct gate's assertion count DROPS 63 -> 59 because the + `d=128` f32 case throws on the launch — which CONFIRMS the halving's own + comment that a fixed 64-column tile fails "on exactly the real geometry"). + The fourth, removing the online-softmax rescale, is caught by BOTH gates and + is recorded as such: the rescale runs at every KEY, not only at tile + boundaries, so it is not evidence about tiling. + * **FOUND WHILE GATING THE ABOVE, and it is NOT an LTX-2.5 defect: the shared + `DevicePool` is DEVICE-BLIND.** `vllm::Pool()` (device_pool.h) is a + process-wide singleton whose free list is + `unordered_map>` — the DEVICE is not part of the + key. So a block `cudaMalloc`ed for a CUDA-queue forward is handed straight + back to a CPU-backend `DBuf` of the same size class, and the CPU backend's + `Copy` is a host `memcpy` on a device pointer. MEASURED on GB10 2026-08-12: + SIGSEGV in `__memcpy_sve <- UploadStream <- PrepareStreamDev`, with + compute-sanitizer reporting ZERO device errors because the fault is + host-side. It had never been reachable because no test had run a bf16 + CPU-backend device forward AFTER a bf16 CUDA one; at f32 the two arms land + in different size classes and never trade blocks. The pool already carries + exactly this invariant for STREAMS — `AuxPool()` exists because "two streams + sharing one pool BREAKS" its reuse ordering — and the first fix used that + same sanctioned seam: the CPU arm ran under an `ActivePoolScope` with its + own pool. **That workaround is GONE, and so is the fault it worked around.** + `POOL-DEVICE-KEY` ([#516](https://github.com/mudler/vllm.cpp/issues/516), + [`specs/pool-device-key.md`](specs/pool-device-key.md)) states the DEVICE + half at the pool itself: a `DevicePool` is bound to one backend, `Pool(b)` + is the only spelling and there is no device-less one, every operation + throws on a foreign backend, and the per-caller scope in + `test_ltx2_device.cpp` was DELETED in the same change — because a list of + remembered callers is what this fault was, and leaving one behind would + have disarmed the only test that exposes the silent-NaN direction. + * **OWED, and precisely:** (a) the prompt-K/V cache on the device path, which + is REFUSED by name rather than ignored; (b) an FP4-RESIDENT arm — the + `LinearDev` seam is one parameter away from the shared Marlin W4A16 + dispatcher MiniMax-H3 routes through, but `Ltx2StreamDitToDevice` dequantizes + to bf16 at load, so keeping the packed weights resident is loader work this + phase did not do; (c) every speed number, structurally pending per the spec's + §0 — no production-configuration denominator exists. + * **Spec:** [ltx-2.5 spec](specs/ltx-2-5.md) §6 (L8). Lifecycle: shipped + (CPU + CUDA, bf16 and f32 streams). Owner: the LTX-2.5 row. + +21. **LTX-2.5 phase L9c — the EMBEDDINGS CONNECTOR reaches the render path, and the + two references disagree about which module owns the sort.** + 2026-08-13, `MODEL-DIFFUSION-ltx-2-5-ltx2-video-transformer-3d-model` phase L9c, + issue [#435](https://github.com/mudler/vllm.cpp/issues/435). + * **Upstream source:** Lightricks/LTX-2 @ `fd4ded7f`, + `packages/ltx-core/src/ltx_core/text_encoders/gemma/embeddings_processor.py:70-95` + (`EmbeddingsProcessor.create_embeddings`) and `:23-48` (its three helpers), plus + `embeddings_connector.py:194-256` (the two configurators) and + `encoders/encoder_configurator.py:331-346` (`EMBEDDINGS_PROCESSOR_KEY_OPS`, + which is what says these weights belong to the TEXT ENCODER even though they + ship inside the DiT file). + * **Local anchor:** `Ltx2ConnectorCreateEmbeddings` + (`src/vllm/model_executor/models/ltx2_connector.cpp`), `Ltx2ParseConnectorConfig` + and `Ltx2LoadConnectorWeights` (`ltx2_loader.cpp`), and the call in + `Ltx2VideoEngine::Load` (`src/vllm/multimodal/ltx2_video.cpp`). + * **What was actually wrong before it.** `Ltx2ConnectorForward` landed at L5 and + was gated against upstream on five arms — and its ONLY caller was its own test. + The render handed the prompt-embeds file straight to cross-attention, and the + two `*_embeddings_connector` families (129 tensors each, present in both shipped + DiTs) were reported as unported and stepped over. A brick with a golden and no + caller is indistinguishable, from the outside, from a brick that is wired. + * **THE TWO REFERENCES DISAGREE, and it is recorded rather than resolved by + preference.** `ltx_core` right-pad-sorts the features in the PROCESSOR before + calling the connector ("Connectors expect right-padded input", + `embeddings_processor.py:80-84`); `diffusers` folds that sort INTO the connector + (`src/diffusers/pipelines/ltx2/connectors.py`, the + `torch.argsort(1 - binary_attn_mask, stable=True)` branch) and its comment + claims that matches "the original LTX implementation" — true only because + `ltx_core` does it one level up. They compose to the same function and differ in + which module owns it. This port follows `ltx_core`, so `Ltx2ConnectorForward` + stays a faithful port of `Embeddings1DConnector`. + * **WHERE THEY AGREE AND IT LOOKS LIKE A BUG.** `_to_binary_mask` is + `encoded_mask < 0.000001` (`embeddings_processor.py:46-48`). An additive mask + holds `0.0` for KEPT and `-finfo(f32).max` for PADDED, and BOTH satisfy it — so + the mask handed to the DiT is one at every position, and the video-only multiply + that follows is an identity on every reachable path. `diffusers` writes the + identical `(video_attn_mask < 1e-6)` and the identical multiply. Checked on both + BEFORE mirroring it, because the reading a port arrives at by reasoning about + intent (`>= 0`) is the opposite at padded positions. Gated as the surprising + behaviour, so "fixing" it REDs: `test_ltx2_pipeline`, case "ltx2 the processor's + binary mask mirrors a comparison that looks backwards". + * **THE REGISTER BOUNDARY IS NOW NUMERICALLY REACHABLE (2026-08-13, review + finding F1 on the L9c PR).** `prompt_embeds_valid_rows` is the single knob + deciding which positions become the connector's TRAINED registers, and the + only case covering it asserted that valid=4 renders differently from valid=2. + Every monotone corruption of that boundary keeps that true, so a fresh + reviewer's `prompt_valid_rows + 1` left the suite fully green — one padded row + conditioned on caller junk instead of a register: finite, correctly shaped, + plausible, wrong. Closed by asserting WHICH positions are registers, from both + sides: perturbing rows `[valid, N)` must move NOTHING (they are substituted at + `embeddings_connector.py:148-150`) and perturbing row `valid - 1` must move + something. `test_ltx2_video`, case "the register boundary sits EXACTLY at the + valid-row count"; RED under `+ 1`, `- 1`, `0` and `v_rows`. + * **A CONFIG VALUE THAT IS NOT NEAR ITS DEFAULT.** LTX-2.5 declares + `connector_positional_embedding_max_pos = [4096]` where `Embeddings1DConnector`'s + class default is `[1]`, and `get_fractional_positions` DIVIDES the token index by + it (`rope.py:132-141`). No shape can see the difference; the default is every + RoPE angle wrong. `positional_embedding_theta` is deliberately NOT read from the + DiT config even though one is declared, because neither configurator passes it — + reading it would be a re-invention rather than a port. + * **ONE KEY WHERE THIS PARSE DIVERGES RATHER THAN MIRRORS (2026-08-13, review + finding F3).** `Ltx2ParseConnectorConfig` reads + `connector_num_learnable_registers`, which NEITHER configurator does + (`embeddings_connector.py:194-219` and `:222-256` both leave it at the class + default of 128), so "mirrors both configurators key for key" is not literally + true of this one key. The divergence is kept — a checkpoint declaring something + else must not be silently run at 128 — and is now ENFORCED rather than asserted: + `test_ltx2_video`, case "a connector config that disagrees with the FILE is + refused", subcase "a register count the file's TABLE does not carry is refused", + which REDs when the read is made inert. + * **The record changed, not just the code.** The two connector families are no + longer reported as unported: this port reads them, so naming them would say + something untrue about the tree and would demand `allow_unported_modules` from a + caller whose checkpoint is read completely. Asserted as an ABSENCE in + `test_ltx2_loader`, case "ltx2 loader: the unported families are refused by name, + not absorbed" (`CHECK(conn_ck.unported.empty())`), so restoring the old behaviour + REDs. + * **OWED, and precisely:** the Gemma-4 TOWER, so what ENTERS the connector is still + whatever the caller put in the prompt-embeds file — the link below the tower is + real, the tower is not. `prompt_embeds_valid_rows` exists because a file carries + no tokenizer mask and the connector REPLACES padding with learned registers. + * **Spec:** [ltx-2.5 spec](specs/ltx-2-5.md) §6. Lifecycle: shipped (CPU host + module; f32, an annotated escape — it runs once per load and its output is + narrowed to the stream dtype on upload). Owner: the LTX-2.5 row. ## 10. E2E test suites (T0 deliverable) diff --git a/.agents/roadmap_v1.md b/.agents/roadmap_v1.md index 615a6a7ea..3cc1d0234 100644 --- a/.agents/roadmap_v1.md +++ b/.agents/roadmap_v1.md @@ -131,17 +131,22 @@ issue is not yet placed. Keyed record: update in place, never append. | [#501](https://github.com/mudler/vllm.cpp/issues/501) | `PERF-27B-LMHEAD-FP4` | `AlphaVecBf16TakesTwoLaunch` bounded a COUNT of ulp mismatches instead of their MAGNITUDE, and was RED on its first CUDA run at ~26% — the double-rounding population the bf16-D lever produces by construction. Replaced by a max-ulp bound (`<= 1`, and `<= 0` at a pow2 alpha), measured 0/1-ulp only over 2.17M words on GB10, spec [`perf-fp8-alpha-fold.md`](specs/perf-fp8-alpha-fold.md) §The bf16-vs-f32 divergence is DOUBLE ROUNDING | bug | | [#521](https://github.com/mudler/vllm.cpp/issues/521) | `PERF-27B-LMHEAD-FP4` | [`perf-fp8-alpha-fold.md`](specs/perf-fp8-alpha-fold.md) `:19`/`:211` claim the bf16-D lever "also applies to 35B-A3B" — it is INERT there: `GdnOutDType(dense_model=false)` is F32 on a MoE, contradicting the code's own comment at `qwen3_5.cpp:3617-3619` | bug | | [#391](https://github.com/mudler/vllm.cpp/issues/391) | `PERF-CPU-DECODE-BARRIER` | CPU backend: batch-1 decode is barrier-bound (47% sync), and paged attention branches per element | perf | +| [#516](https://github.com/mudler/vllm.cpp/issues/516) | `POOL-DEVICE-KEY` | `vllm::Pool()`'s free list is keyed by size class with NO DEVICE in the key, so a `cudaMalloc` block reaches a CPU `DBuf` (SIGSEGV) and a host block reaches a CUDA forward (uniform `0x7fff0000` NaN); spec [`pool-device-key.md`](specs/pool-device-key.md), lands through `row/MODEL-DIFFUSION-LTX25` | bug | | [#299](https://github.com/mudler/vllm.cpp/issues/299) | `ROAD-V1-C1` | `FUSION-DENSE-MIGRATE`: 5 dense SwiGLU models bypass the MUST-route merged-GEMM seam with no stated blocker (spec [`fusion-dense-migrate.md`](specs/fusion-dense-migrate.md)) | bug | | [#314](https://github.com/mudler/vllm.cpp/issues/314) | `ROAD-V1-C1` | `FUSION-DENSE-MIGRATE` glue half: `glm4`/`phi3` still hand-call add+RMSNorm instead of `vt::FusedChain` (split out of #299, which closed the merged-GEMM half only) | bug | | [#337](https://github.com/mudler/vllm.cpp/issues/337) | `ROAD-V1-C1` | `FUSION-DENSE-MIGRATE`: the five dgx SACRED paged-engine gates are OWED after the merged-GEMM fold (`test_{commandr,glm4,minicpm,minicpm3,phi3}_paged_engine` SKIP on a CPU box) | bug | | [#338](https://github.com/mudler/vllm.cpp/issues/338) | `ROAD-V1-C1` | MiniCPM/MiniCPM3 hard-code SiLU: upstream `MiniCPMMLP` (`minicpm.py:219-226`) selects `FatreluAndMul` on `hidden_act == "fatrelu"` and raises otherwise; our `parse_config` never reads `hidden_act` | bug | | [#241](https://github.com/mudler/vllm.cpp/issues/241) | `ROAD-V1-H3` | MiniMax-H3: support the PRUNED (AdaLN timestep-curve) checkpoint variants | feature | +| [#598](https://github.com/mudler/vllm.cpp/issues/598) | `ROAD-V1-H3` | `test_minimax_h3` is nibble-BLIND: 44 live `DequantNvfp4ToBf16` calls and 57,395 assertions all stay green with the fp4 nibble order flipped, so H3's load-time swap and the new shared `kHighFirst` parameter can compose into a double flip undetected (spec `specs/nvfp4-nibble-order.md` §5.5) | bug | | [#471](https://github.com/mudler/vllm.cpp/issues/471) | `GATE-PIN-UNPINNED-SNAPSHOTS` | Checkpoint gates resolve their snapshot UNPINNED: 56 of 61 take whatever `directory_iterator` yields, including the three DFlash gates on a repo caching two materially different revisions (spec `specs/gate-pin-unpinned-snapshots.md`) | bug | | [#472](https://github.com/mudler/vllm.cpp/issues/472) | `GATE-PIN-UNPINNED-SNAPSHOTS` | 80 of 92 committed goldens record no checkpoint revision, so their gates cannot be pinned from evidence (`q3mxfp4`, DFlash, every `*_greedy` corpus) | bug | | [#482](https://github.com/mudler/vllm.cpp/issues/482) | `GATE-PIN-UNPINNED-SNAPSHOTS` | `check-snapshot-pins` FALSE-POSITIVES on the correct pinned form: a file naming the revision then enumerating shards inside `snapshots//` is reported UNPINNED, and the message names a C++ header with no Python equivalent. No live FP only because `online_gate.py:3631` case-shadows (`snapshot` parameter vs `snapshots` marker) | bug | | [#483](https://github.com/mudler/vllm.cpp/issues/483) | `GATE-PIN-UNPINNED-SNAPSHOTS` | `check-snapshot-pins` is evaded by ordinary modern-C++ punctuation, not by the deliberate escapes: `directory_iterator{snaps}` (brace init), `root /= "snapshots"` (compound assignment), and a lambda body severed by the `[;{}]` statement split | bug | | [#484](https://github.com/mudler/vllm.cpp/issues/484) | `GATE-PIN-UNPINNED-SNAPSHOTS` | The `check-snapshot-pins` CI step has never been observed executing on a real GitHub runner; the spec records this row's GPU/weights limits but not that one | bug | | [#485](https://github.com/mudler/vllm.cpp/issues/485) | `GATE-PIN-UNPINNED-SNAPSHOTS` | `_MARK_FIXPOINT_ROUNDS = 6` silently truncates the binding fixpoint: a 7-link helper chain evades `check-snapshot-pins` with no diagnostic | bug | +| [#435](https://github.com/mudler/vllm.cpp/issues/435) | `ROAD-V1-LTX25` | LTX-2.5: port the 21B joint video+audio DiT and generalize the video seam | feature | +| [#560](https://github.com/mudler/vllm.cpp/issues/560) | `ROAD-V1-LTX25` | Stabilizing constants unreachable by any gate: FIVE instances found by sweep, incl. the DiT's own `norm_eps` field default | bug | +| [#567](https://github.com/mudler/vllm.cpp/issues/567) | `ROAD-V1-LTX25` | `OpNameImpl` makes a SECOND file every new `OpId` must edit; collapse the enum and its names into one derived list | bug | | [#238](https://github.com/mudler/vllm.cpp/issues/238) | `SAMPLE-LOGPROB-TOKEN-IDS` | `logprobs_mode`: three of four modes are runtime-refused stubs | bug | | [#264](https://github.com/mudler/vllm.cpp/issues/264) | `SAMPLE-LOGPROB-TOKEN-IDS` | `logprob_token_ids`: generative scoring over an explicit token set is unported | feature | | [#365](https://github.com/mudler/vllm.cpp/issues/365) | `PERF-27B-DENSE-MARLIN-GATEUP` | 27B gap decomposed vs vLLM's own decode profile; dense W4A16 MLP bypassed the fused gate_up seam (spec `specs/perf-27b-dense-marlin-gateup.md`) | bug | @@ -233,6 +238,7 @@ models we already ship + benchmark. Full seam map + M0–M5 W-plan: | 13a | `ROAD-V1-D4-APC` | **Prompt / prefix caching to full vLLM parity, then beyond (user-directed 2026-07-22: "same featureset of vLLM and better")** — the headline user-facing caching feature, previously mentioned only once in this roadmap despite being a shipped, default-ON behaviour for dense models | [`KV-PREFIX-CACHE`](engine-matrix.md), [`KV-BLOCK-POOL`](engine-matrix.md), [`KV-HYBRID-COORD`](engine-matrix.md), [`KV-MAMBA-ALIGN`](engine-matrix.md), [`KV-EVENTS`](engine-matrix.md), [`KV-PREFIX-MATCH-UNIT`](engine-matrix.md), [`ENG-CASCADE-ATTN`](engine-matrix.md), [coverage view §2](feature-matrix.md#2-kv-cache--memory) | umbrella spike ACCEPTED [prefix-prompt-caching-parity.md](specs/prefix-prompt-caching-parity.md) — enumerates the complete pinned-vLLM caching surface (38 features) with a per-feature DONE/PARTIAL/MISSING verdict grounded in our source. **The ported core is deeper than the record claimed** (chain hashing, block pool, all three coordinators, the full hybrid fixed-point intersection, four single-type managers); the real gaps are narrower and different: block-hash extra keys are a no-op stub, there are NO prefix-cache statistics at any level, KV events are inert, `cache_salt` and 3 of 4 hash algos are absent, and `reset_prefix_cache` is implemented but unreachable. Three matrix rows corrected, two of them in our favour. `ENG-CASCADE-ATTN` DISPOSITIONED as not owed (default-off, absent from the MRV2 runner we port, unreachable on Blackwell). llama.cpp comparison completed: its "prompt cache" is session/slot state serialization, strictly weaker than APC on every reuse axis, and vLLM already covers disk persistence via the `kv_offload` fs tier — the ONE genuine capability neither vLLM nor we have is an imperative named per-sequence save/restore **W1 IMPLEMENTED 2026-07-22: prefix-cache statistics exist for the first time.** `BaseCacheStats`/`PrefixCacheStats`/`CachingMetrics` ported 1:1 from `vllm/v1/metrics/stats.py:35-142`, recorded in `get_computed_blocks`, flagged by `reset_prefix_cache`, taken-and-swapped per step and folded into a 1000-request sliding window exposed on `Scheduler`/`EngineCore`/`LLMEngine`. Per the standing parity-enabler rule `log_stats` is DEFAULTED ON (mirroring upstream's `disable_log_stats=False`), so no benchmark arm is void for want of a counter. `Request::num_preemptions` un-deferred to feed the mutually-exclusive `preempted_*` triple. **FIRST MEASURED HIT RATE: 0.75** (1920 of 2560 queried tokens over 16 requests sharing a 128-token prefix), with a caching-OFF 0.0 negative control — the first demonstration in this project that APC actually serves cached tokens. The hard blocker on [`BACKEND-GATE-CUDA-SGLANG-PREFIX`](backend-matrix.md) is CLOSED **W2 DONE 2026-07-27 (`CLAIM-ROADMAP-D4APC`, CPU-gated on dgx GB10, NOT pushed):** `generate_block_hash_extra_keys` ported 1:1 (`kv_cache_utils.py:451-591`) — mm hash + LoRA name + `cache_salt`, fixed order lora->mm->salt (prompt_embeds deferred: no prompt-embeds path); `cache_salt`/`lora_name` carried on `Request`/`EngineCoreRequest`, set before the first hash in `FromEngineCoreRequest` (fixes a latent ordering bug). RED-first no-false-share PROVEN: with the stub a differently-salted request false-hits the prior tenant's 48 cached tokens (`n1==48`), with extra keys `n1==0`. Ported extra-key/ordering cases + hash- and manager-level no-false-share (`test_kv_cache_utils.cpp` 29/29, `test_kv_cache_manager.cpp` 10/10). **This unblocks the MM + LoRA cache consumers.** **W3 DONE 2026-07-27 (`CLAIM-ROADMAP-D4APC-W3`, dgx GB10, NOT pushed) — the FIRST-EVER cache-ON model gate:** on `Qwen/Qwen3-4B` (dense, full-attention, APC-default-ON — the vehicle the prior "vehicle-blocked" note missed) a shared-prefix workload runs APC-ON and APC-OFF through the full paged engine, gating token-identity + hits + prefill drop. **NO engine code changed** (`git diff --stat` = tests+scripts+goldens) ⇒ pure GATE over the already-shipped default-ON path; binary byte-identical ⇒ SACRED unaffected. RESULT (`test_qwen3_apc_e2e` 2/2, 84/84 asserts): APC-ON hits **2240/2777 (rate 0.807)**, APC-OFF 0; APC-ON == APC-OFF EXACT on 5/6 (the 1 diff a vLLM-confirmed 0.125-nat near-tie, RCA'd = attention-kernel-path near-tie flip, not a cache bug); **== vLLM-APC-ON** teacher-forced (APC-OFF 6/6 max gap 0.0 nats = exact argmax, APC-ON 6/6 max gap 0.125 nats, 0 outside top-20); **TTFT drop 70.1→39.9 ms = 1.76×** on a cache hit. Existing 4B SACRED gate 16/16 GREEN (no regression). Oracle vLLM 0.25.0 (0.26 venv broken — editable source disk-reclaimed; 4B byte-stable across the pin). | `DONE` (headline) | **Row DONE for the default dense APC path (block hashing incl. extra_keys, pool, coordinators, stats, scheduling, cache-ON e2e all gated).** Named NON-BLOCKING tails tracked in their own rows / future items: W4 KV events (`KV-EVENTS` — event GENERATION + `msgpack` PAYLOAD DONE 2026-07-27 `CLAIM-ROADMAP-D4-KV-EVENTS`, `SPIKE`→`ACTIVE`, byte-exact vs `msgspec`; live ZMQ transport + engine batch wiring DEFERRED), W5 partial-block primitive (upstream dead-code), W6 Mamba-`align` hybrid cache-on (`KV-MAMBA-ALIGN`, SPIKE — feeds `BACKEND-GATE-CUDA-SGLANG-PREFIX`), W7 `reset_prefix_cache` dev-endpoint + `--prefix-caching-hash-algo` + `skip_reading_prefix_cache`, W8/W9 the beyond-vLLM named session save/restore. The every-axis cache-on grid vs vLLM/SGLang is a separate perf follow-on under `ROAD-V1-A`. No `/metrics` route yet (`SERVE-METRICS`), so the hit rate is read from the engine API. **`--prefix-match-unit` (0.26-new fine-grained matching unit) W0 spike + W1 resolver LANDED 2026-07-28 (`CLAIM-PREFIX-MATCH-UNIT`, `KV-PREFIX-MATCH-UNIT` PARTIAL): `resolve_kv_cache_block_sizes` computes `hash_block_size = prefix_match_unit if set else gcd(group_block_sizes)`, RED-first unit-gated; config/CLI/ABI field (W2) + scheduler threading of `hash_block_size != block_size` (W3, needs the `KV-BLOCK-POOL` align path) + benchmark (W4) deferred.** | | 14 | `ROAD-V1-D5` | LoRA, local KV/weight offload, expert streaming, wider model zoo | [engine matrix](engine-matrix.md), [model matrix](model-matrix.md) | corrected expert-streaming spike accepted (`ENG-EXPERT-STREAM` READY): bank-only safetensors→Marlin bank, fixed contiguous cache slots matching Marlin dense strides, logical→slot remap after explicit router D2H, chunked C` working everywhere `--dit ` did; gated CPU-only (72/72, 54497 post-rebase) on index/name mapping and on the REAL 535-tensor geometry read from a sparse 13-shard release. The DEVICE streamer landed 2026-08-07 (`row/H3-BF16-SHARDED-STREAM`, spec §8.14): one tensor at a time, zero host buffer for the bulk, bit-exact vs the non-streamed reference (73/73, 55203). **ENCODER + THE NUMBER 2026-08-07 (`row/H3-ENC-BF16-COND-DIFF`, spec §8.15)**: the 14-shard bf16 text encoder streams too and `--encoder-only` runs it alone; measured over 233 tokens, Q4_K_M-vs-bf16 conditioning is cos 0.99745 mean / 6.85% rel RMS excl. sink / 3.5 deg median rotation — as much as a one-word prompt edit, but DIFFUSE. Whether the RENDER changes is NOT established (75/75, 55609). This unblocks the bf16-vs-quantized quality A/B; no render or speed number is claimed. Spec §8.13. **W-FP4a LANDED (CPU) 2026-08-06 (`row/H3-FP4-SPEED`)**: the NVFP4 DiT projections now keep FP4 PACKED and route through the shared `dense_nvfp4::MatmulNvfp4W4A16D` (Marlin W4A16 — vLLM's own forced-a16 selection; SAME kernel as Laguna/dense-Qwen3 NVFP4; no new quant code); fp4-vs-bf16 WIRING gate GREEN (62/62·30039, W4A16 dispatcher runs all 11 quantized GEMMs). **GB10 leg LANDED 2026-08-06 (`row/H3-FP4-GPU-E2E`, PR #64):** Marlin W4A16 RAN on sm_121a (`dense_gemms==11` default / `marlin_gemms==11` VT_MARLIN_DENSE=0, `fallback_gemms==0`), fp4-vs-bf16 BYTE-EXACT; fp4 is a MEMORY win (~16 vs ~66 GB), ~0.79–0.83× the bf16 arm per diffusion forward (compute-bound large M; 3.47× faster at small decode-like M). Real-checkpoint fp4-resident t2va e2e RUNS end-to-end (real 18.75 GB NVFP4 DiT + VAEs + GGUF Qwen3-VL-32B encoder → valid mp4/wav; DiT s/step 5.45/20.0/209 s @512/768/REF-768×1344-209f) but frames are a non-scene patch-grid at 12/20/50 steps → OPEN render-coherence bug (device VAE decode / denoise), separate from the fp4 speed work. vLLM-Omni serves NO quantized H3 (BF16-only) -> HW/loader-forced-indirect (4×B300 209f render 86.964 s vs 1×GB10 209 s/forward). **2026-08-08 ROW 2 DEVICE-SEAM FOLLOW-UP (#135; replaces #134):** the public 0/1 selector is mapped once to generic `DeviceType`; DSR returns 34→32 with the baseline/allowlist unchanged; CPU compile/fold test pending in CI due shared-disk pressure. | +| LTX25 | `ROAD-V1-LTX25` | **DIFFUSION generation, second model — and the seam that makes the class additive.** LTX-2.5 (`LTX2VideoTransformer3DModel`): 21.00B joint video+audio flow-matching DiT, 48 blocks, TWO coupled streams (video 4096 / audio 2048) joined by explicit audio<->video cross-attention, per-head gated attention on every attention, gelu-approximate FFN, Gemma-4 12B text encoder. Architecture reference is Lightricks `LTX-2` (`ltx-core`); vLLM-Omni does NOT yet carry 2.5 (recipes stop at 2.3, upstream [vllm-omni#6066](https://github.com/vllm-project/vllm-omni/issues/6066) open), so the binding oracle is vLLM-Omni's GENERIC diffusers adapter and the immediate cross-check is Lightricks `ltx-pipelines`. Fits ONE GB10 at ~29 GB NVFP4. | [`MODEL-DIFFUSION-ltx-2-5-ltx2-video-transformer-3d-model`](model-matrix.md) | [ltx-2.5](specs/ltx-2-5.md) | `SPIKE` | **L0 spec committed 2026-08-11** (#435). Geometry MEASURED from the checkpoint header by range request, not inferred: 21.00B (blocks 18.560B + audio connector 2.016B + global 0.427B), 48 blocks @ 386.7M, 1775 F8_E4M3 tensors with per-tensor F32 `weight_scale`. The `ff`-no-bias / `audio_ff`-bias split in the checkpoint confirms `ff_bias=false` in `model_configurator.py:78-80` — checkpoint and source agree. **RETRACTED 2026-08-12 (was billed as a free win):** the shipped checkpoint CARRIES `prompt_adaln_single` (timestep_embedder linear_1 [4096, 256]), which upstream builds only when `use_prompt_adaln_single` is TRUE (`model.py:223-227`), so the cross-attention K/V DO carry a timestep term (`transformer.py:442-443`) and are NOT cacheable here. `transformer.py:441` was quoted as proof of no timestep term, two lines before the one that adds it. No shipped defect: the forward refuses by name (`ltx2_dit.cpp:672`) rather than caching wrongly. **Owed up front:** the speed axis lands `PENDING` because the diffusers adapter is a black box (`supports_step_execution=False`) and so is NOT vLLM's production configuration; DiffVAE is REFUSED by name rather than silently downgraded to the Conv VAE. Phases L1-L7 on one PR (developer-directed). | | 15 | `ROAD-V1-D6` | **llama.cpp device breadth folded into scope (user-directed 2026-08-05):** the 11 ggml backends vLLM has no platform for — cann, musa, opencl, openvino, rpc, webgpu, zdnn, zendnn, hexagon, blas, virtgpu — inventoried as `BACKEND-GGML-*`. **SPIKES FIRST:** no implementation before each row's `.agents/specs/.md` clears the spike contract, per the standing directive. vLLM stays the mirror source; llama.cpp is the breadth reference. | [backend matrix](backend-matrix.md) | ☐ per-row spike required | `INVENTORIED` | first spike accepted | | REL | `ROAD-V1-RELEASE` | KISS downloads per OS+host ABI: one adaptive CPU binary and one fat CUDA binary covering every supported SM; per-SM CUDA artifacts are optional diagnostics; stable channels require matching runtime evidence and build-only paths stay preview | [`ENG-RELEASE-BINARIES`](engine-matrix.md), [`ENG-RELEASE-WINDOWS`](engine-matrix.md) | [release binary matrix](specs/release-binary-matrix.md); [Windows pre-alpha extension](specs/windows-binary-release.md) | `ACTIVE` | v0.0.2 published eight primary archive/checksum/provenance triplets plus two indexes from `7020de93652ca920424a10ac5255b34810dd2f24` in run `31466516224` (26 assets). Windows W14-W16 are implemented for one PR; native hosted gates, merged-SHA ten-tuple dry run, matching-hardware evidence, `v0.0.3-pre.1` publication and 32-asset audit remain pending. W12 remains optional/non-primary | | IMG | `ROAD-V1-CONTAINERS` | **Published container images on GHCR, built by GitHub Actions (user-directed 2026-08-08).** The same staged bundle `ROAD-V1-RELEASE` defines, shipped from one package `ghcr.io/mudler/vllm.cpp` with the lane in the tag — `:-cuda` / `-vulkan` / `-cpu`, the moving `:latest-cuda` / `:latest-vulkan` / `:latest-cpu`, and a bare `:latest` aliasing the cpu lane — and `ENTRYPOINT vllm-server`. Lanes `cuda` (one fat image, every supported SM), `vulkan`, `cpu`, plus `rocm` blocked-preview; version tags immutable, every `latest-` moves. Every lane is a `linux/amd64`+`linux/arm64` multi-arch manifest on native runners, because the project's own gate hardware (GB10, Thor, Orin) is arm64. Metal and MLX are NOT-CONTAINERIZABLE and stay static-binary-only — a recorded boundary, not pending work. Depends on the `ROAD-V1-RELEASE` install/stage tree: the image IS the bundle, so the two lanes must not grow separate layouts. No image, workflow or registry package exists. | [`ENG-RELEASE-CONTAINERS`](engine-matrix.md) | [container-images.md](specs/container-images.md) — spec ACCEPTED and **W1-W5/W7 IMPLEMENTED**: one `docker/Dockerfile` whose builder stages call the existing `scripts/build-*-release.sh`, digest-pinned ubuntu runtime bases with the CUDA runtime libs copied and the driver left to the host, ffmpeg in every lane, gated container matrix + image validator + least-privilege publish workflow | `ACTIVE` | **cpu lane BUILT AND GATED e2e 2026-08-10** (783 MB linux/amd64; `/health` 200, `/version` 200, in-container healthcheck, clean SIGTERM, booted on opt-125m). The boot gate immediately found [#312](https://github.com/mudler/vllm.cpp/issues/312): `vllm-server` ignored SIGTERM as PID 1, so `docker stop` hard-killed it (137) after 30 s — now exit 0 in 0.25 s via a self-pipe handler into the existing `server.stop()`. Two silent build-context bugs fixed on the way: `.dockerignore`'s `**/build*/` also matched FILES and was excluding `scripts/build-*-release.sh`, and the builders lacked `file`/`binutils` so the inherited archive validator failed after a full compile. **REMAINING: W6 only** — nothing is published to GHCR, cuda/vulkan are gated statically but never built here, and both arm64 legs are unbuilt so SBSA-vs-Tegra (Thor `sm_110`, Orin `sm_87`) is untouched | diff --git a/.agents/specs/ltx-2-5.md b/.agents/specs/ltx-2-5.md new file mode 100644 index 000000000..7d4f8990b --- /dev/null +++ b/.agents/specs/ltx-2-5.md @@ -0,0 +1,953 @@ +# LTX-2.5 — 21B joint video+audio flow-matching DiT, and the generalized video seam + +**Rows:** `MODEL-DIFFUSION-ltx-2-5-ltx2-video-transformer-3d-model` (model-matrix), +`ROAD-V1-LTX25` (roadmap portfolio). +**Issue:** [#435](https://github.com/mudler/vllm.cpp/issues/435). +**Branch:** `row/MODEL-DIFFUSION-LTX25` (ONE PR for the whole campaign — developer-directed +2026-08-11; AGENTS.md retired per-class line budgets, so size is a review judgement). +**Upstream (architecture):** Lightricks `LTX-2` — `packages/ltx-core/src/ltx_core/`. +**Upstream (serving oracle):** vLLM-Omni `vllm_omni/diffusion/` — see §3, it does NOT yet +carry 2.5. +**Checkpoints:** `Lightricks/LTX-2.5` (gated: auto), `vonkaiser/LTX-2.5-FP8-NVFP4` (ungated). +**Status:** phases L1–L8. **L8's spec was written AFTER its implementation** (§8.1), which +violates the spec-before-code rule; the operator's error, recorded rather than backfilled +silently. + +--- + +## 0. Honesty statement — what is and is not claimed + +LTX-2.5 is **not an autoregressive LLM**. It is a joint video+audio **diffusion +transformer**: one request runs a flow-matching denoise loop in which a 21.00B DiT is +forwarded once per step over two coupled modality streams, and the resulting latents are +decoded to frames plus a waveform by two VAEs. There is no KV cache in the LLM sense, no +sampler, no logits, and **no token-exact gate** — the SACRED near-tie methodology this +project uses for decoders does not apply, exactly as recorded for MiniMax-H3 +([minimax-h3](minimax-h3.md) §0). + +Three things are recorded as **owed** here, before any work starts, so they cannot be +discovered later: + +1. **The speed gate lands `PENDING`.** vLLM-Omni's only route to 2.5 is its + `DiffusersAdapterPipeline`, which is a black box — `supports_step_execution = False`, + `supports_request_batch = False` + (`vllm_omni/diffusion/models/diffusers_adapter/pipeline_diffusers_adapter.py:68-69`). + A throughput number taken through it is **not vLLM's production configuration**, which + AGENTS.md §Gates requires as the denominator. Correctness is gateable through it; + throughput is not. The axis stays open with a named next step (§9). +2. **DiffVAE is refused, never silently downgraded.** The higher-quality video decoder is + `NADiffusionDecoder`, built on neighborhood attention. Until its row lands, asking for it + fails with a message naming the missing piece; it does not quietly fall back to the Conv + VAE and return a worse render as if it were the requested one. +3. **No render-quality claim.** Structural e2e (correct shapes, finite values, valid MP4) + is not a quality result. H3 taught this directly: its fp4-resident e2e *ran* and produced + a valid mp4 while the frames were a non-scene patch grid. + +## 1. Architecture — measured, not inferred + +Read by HTTP range request from the ungated +`vonkaiser/LTX-2.5-FP8-NVFP4` → `transformer/ltx-2.5-22b-distilled-fp8.safetensors` +(6124 tensors, 881,048-byte header; no payload downloaded — the same technique used for +H3's manifests). + +| Field | Value | +|---|---| +| Parameters | **21.00B** — blocks 18.560B + audio connector 2.016B + global 0.427B | +| Blocks | **48**, 386.7M each | +| Video stream | hidden **4096**, 32 heads x 128 | +| Audio stream | hidden **2048**, 32 heads x 64 | +| `in_channels` / `out_channels` | 128 video (`patchify_proj` [4096,128], `proj_out` [128,4096]); 128 audio | +| Video FFN | `ff.net.0.proj` [16384, 4096] → `ff.net.2` [4096, 16384], **NO bias** | +| Audio FFN | `audio_ff.net.0.proj` [8192, 2048] → [2048, 8192], **WITH bias** | +| Activation | `gelu-approximate` (`model_configurator.py:31`) | +| Norms | `standardization_norm=rms_norm`, `qk_norm=rms_norm`, `norm_elementwise_affine=False` | +| Audio connector | 8 x 1-D transformer blocks + `learnable_registers` [128, 2048] | +| Quant (FP8 arm) | F8_E4M3 + **per-tensor F32 `weight_scale`**; biases/norms BF16; 1775 FP8 tensors | + +The filename says `22b`; the measured count is **21.00B** and the Diffusers card says ~19B. +The measured number is the one this spec uses. + +**The `ff_bias` cross-check.** `ff` carries no bias while `audio_ff` does. That is exactly +`ff_bias=false` / `audio_ff_bias=true`, whose defaults +(`model_configurator.py:78-80`) are documented as *"Default True keeps backwards +compatibility: pre-2.5 checkpoints lack these keys and retain FFN biases. LTX 2.5 (gemma4) +sets ff_bias=false."* Checkpoint and source agree — that agreement, not either alone, is +what AGENTS.md §"Verify against both the running oracle and its source" asks for. + +### 1.1 The block — `BasicAVTransformerBlock` (`transformer.py:87`) + +Per block, two coupled streams: + +| Tensor | Shape | Role | +|---|---|---| +| `scale_shift_table` | [9, 4096] | 3 groups of 3: `slice(0,3)` self-attn pre-mod, `slice(3,6)` FFN pre-mod, `slice(6,9)` cross-attn `shift_q, scale_q, gate` (`transformer.py:240,273,401`) | +| `audio_scale_shift_table` | [9, 2048] | same, audio (`:302,320,410`) | +| `prompt_scale_shift_table` | [2, 4096] | **prompt K/V modulation — see §1.2** | +| `audio_prompt_scale_shift_table` | [2, 2048] | same, audio | +| `scale_shift_table_a2v_ca_video` | [5, 4096] | audio↔video cross-attn modulation (`:337,378`) | +| `scale_shift_table_a2v_ca_audio` | [5, 2048] | same (`:347,369`) | + +Attentions per block: `attn1` (video self), `attn2` (video↔text, cross_dim 4096), +`audio_attn1`, `audio_attn2` (cross_dim 2048), plus the two cross-modal +`audio_to_video_attn` (`transformer.py:154`) and `video_to_audio_attn` +(`transformer.py:166`). + +**Per-head gated attention** is on for every one of them: `to_gate_logits` is +`torch.nn.Linear(query_dim, heads, bias=True)` (`attention.py:513-514`) — hence the +`[32, dim]` weights in the checkpoint, one logit per head — applied *after* the attention +output as `out = self.gated_attention_function(x, out, self)` (`attention.py:577`). H3 has +no analogue; getting this wrong yields a plausible-but-wrong render rather than an error. + +### 1.2 The prompt K/V cache — RETRACTED as a win for the shipped checkpoint + +> **RETRACTION, 2026-08-12.** This section previously called a timestep-independent prompt +> K/V "the free win" and made it the headline of this spec, of issue #435 and of the PR. +> **It does not apply to the checkpoint this campaign actually runs.** The mechanism is real +> and correctly implemented; the claim that 2.5 enables it was wrong, and it was wrong +> because the spec author read `transformer.py:441` and stopped two lines early. + +**What upstream actually does.** `apply_cross_attention_adaln` (`transformer.py:420-447`): + +```python +kv_modulation = prompt_scale_shift_table[None, None].to(...) # :441 +if prompt_timestep is not None: # :442 + kv_modulation = kv_modulation + prompt_timestep.reshape(...) # :443 +``` + +Line 441 alone was quoted as proof of "no timestep term at all". Lines 442-443 add one +whenever `prompt_timestep` is not None, and the comment immediately above them says so +outright: *"With the prompt-side AdaLN MLP disabled (use_prompt_adaln_single=False), +prompt_timestep is None and only the static per-block table applies, so K/V are +timestep-independent and cacheable across denoising/AR steps. Otherwise the +timestep-conditioned MLP output is added on top."* + +`model.py:223-227` builds the MLP conditionally: + +```python +self.prompt_adaln_single = ( + AdaLayerNormSingle(self.inner_dim, embedding_coefficient=2) + if self.cross_attention_adaln and self.use_prompt_adaln_single else None) +``` + +**What the shipped checkpoint carries.** Read from +`ltx-2.5-22b-distilled-transformer-fp8.safetensors`, 12 tensors that only exist when the MLP +is built: + +| Tensor | Shape | +|---|---| +| `prompt_adaln_single.emb.timestep_embedder.linear_1.weight` | [4096, **256**] | +| `prompt_adaln_single.emb.timestep_embedder.linear_2.weight` | [4096, 4096] | +| `prompt_adaln_single.linear.weight` | [8192, 4096] (= 2 x 4096) | +| `audio_prompt_adaln_single.*` | the audio twin | + +The **256** is the sinusoidal timestep input width. There is a prompt-side timestep MLP. +Therefore `use_prompt_adaln_single` is TRUE for this checkpoint, `prompt_timestep` is not +None, the cross-attention K/V **do** carry a timestep term, and caching them across denoise +steps would be **wrong**. + +**Why the earlier reasoning looked sound and was not.** Three pieces of evidence were +consistent with the wrong conclusion: `model_configurator.py:74-76` genuinely documents +KV-cacheable checkpoints as setting the flag false; the checkpoint genuinely carries the +static `prompt_scale_shift_table [2, dim]` (96 of them); and line 441 genuinely has no +timestep. All three are true. None of them says the flag is false HERE — and the tensor that +settles it was never looked for. The header dump that would have shown it was filtered with +`'adaln_single' not in k`. + +**What this costs, and what saves it.** No shipped defect: L2's implementer read upstream's +conditional correctly even though the brief handed it the wrong conclusion, and wrote + +```cpp +VT_CHECK(!params.use_prompt_adaln_single, + "ltx2: the prompt K/V cache is only valid when use_prompt_adaln_single is false " + "(transformer.py:441-443); with the prompt AdaLN MLP enabled the K/V carry a " + "timestep term and caching them would be wrong"); +``` + +at `ltx2_dit.cpp:672`, refusing by name before any block runs. So the cache is +correct-and-inapplicable rather than silently wrong. It remains implemented, gated +bit-identical, and prompt-bound (§below) for any checkpoint that does set the flag false. + +**The lesson, which is the same one §7.0 keeps teaching.** A claim assembled from three true +facts is not thereby true. The decisive test was one `grep` for `prompt_adaln_single` against +the checkpoint, and it was never run because the conclusion already looked supported. + +L2 gates the cache by asserting the cached and recomputed paths are **bit-identical**. + +**Correction, 2026-08-12 — the earlier claim here was too strong.** This section previously +said the bit-identity gate meant the cache "cannot silently diverge". L2's fresh review +disproved that, and the distinction is the whole point of the feature: + +- Against a changed **timestep**, the cache cannot diverge. That is the property the + checkpoint gives us, and it is real. +- Against a changed **prompt**, it silently could. The gate ran the forward twice with + IDENTICAL inputs, so it only ever proved "same in, same out"; the cache carried no prompt + identity and its only validity check was on SIZE. A probe that swapped in a different + prompt of equal token count found the cache did not notice. + +The failure that implies is not academic: a pipeline or server reusing one cache across two +requests whose prompts differ but tokenize to the same length renders the **second request +with the first request's prompt**, with no error, no shape mismatch and no finiteness +failure. The repair carries a content fingerprint on the cache and refuses by name on +mismatch. Recorded rather than quietly amended, because "we gate that" was written here +before it was true. + +### 1.3 How this differs from MiniMax-H3 + +| | MiniMax-H3 (ported) | LTX-2.5 | +|---|---|---| +| Modalities | ONE packed sequence, per-row token tags | **two streams + explicit audio↔video cross-attn** | +| FFN | SwiGLU 14336 | gelu-approximate 16384, no bias | +| Gated attention | none | **per-head, every attention** | +| Text encoder | Qwen3-VL-32B-derived | **Gemma-4 12B + projections** | +| Video decode | ViT3D | Conv VAE **or** DiffVAE (neighborhood attn) | +| Extras | — | latent spatial/temporal x2 upsamplers, duration head | + +### 1.4 Text conditioning is a MULTI-LAYER aggregate, not the last hidden state + +Recorded 2026-08-11 while briefing L3, from the real TE checkpoint +(`vonkaiser` `gemma4-12b-with-proj-nvfp4-torchao.safetensors`, 1688 tensors) read against +`text_encoders/gemma/feature_extractor.py`. + +`feature_extractor.py` takes hidden states shaped `[batch, seq_len, hidden_dim, num_layers]`, +normalizes them, and concatenates **across the LAYER dimension** to +`[batch, seq_len, hidden_dim * num_layers]`. The checkpoint confirms it: the two projections +take **188160** input features, and 188160 = **3840** x 49 — the model's 3840-wide per-layer +state across its 48 layers plus one. + +| Tensor | Stored shape | LOGICAL shape | +|---|---|---| +| `text_embedding_projection.video_aggregate_embed.weight` | U8 [4096, 94080] | **[4096, 188160]** | +| `text_embedding_projection.audio_aggregate_embed.weight` | U8 [2048, 94080] | **[2048, 188160]** | +| `.bias` (both) | BF16 [4096] / [2048] | unchanged, BF16 is unpacked | + +**Correction, 2026-08-12 — NVFP4 STORED widths are HALF the logical ones.** This section first +recorded 94080 = 1920 x 49, reading the U8 shapes as logical. L3 caught it. NVFP4 packs **two +values per byte along the last dimension**, so every U8 width in this file is half the real +one. The tell was in the same header all along: `model.norm.weight` is **BF16 [3840]**, and +BF16 is unpacked, so 3840 is authoritative. Three independent confirmations agree: +`model.embed_tokens.weight` U8 [262144, 1920] -> [262144, **3840**]; +`model.layers.0.self_attn.o_proj.weight` U8 [3840, 2048] -> [3840, **4096**] (= 16 heads x 256); +and the projections' BF16 biases, [4096] and [2048], match the DiT's two stream widths exactly. + +The layer count (48 + 1) was right; only the width moved. **Read a quantized checkpoint's +unpacked tensors to establish the width, never its packed ones** — that is the general lesson, +and it applies again at L6. + +L3 also settled the variant question by execution: 2.5 uses **`FeatureExtractorV2`** (per-token +RMS), selected by four `_V2_EXPECTED_CONFIG` marker keys, with `_rescale_norm` applied +SEPARATELY per projection using that projection's own `out_features` over the Gemma hidden +size. `create_caption_projection` is not on this path — 2.5's projections are the two +`aggregate_embed` Linears inside the encoder, as §1.4 assumed. + +There are at least TWO normalization variants and the right one is selected from config, never +guessed: `_norm_and_concat_padded_batch` (per-batch, per-layer masked mean and range, an `8 *` +scale, `eps = 1e-6`) and `norm_and_concat_per_token_rms` (per-token RMS, "for V2 models"). Both +are padding-side agnostic and ZERO padded positions. + +**Why this is a trap and not a detail:** getting the variant, the mask handling, the reduction +axes or the layer order wrong yields conditioning that is finite, correctly shaped and WRONG. +It renders a plausible video for the wrong prompt, which no shape or finiteness check catches. +L3 gates the variant selection explicitly. + +Two further facts the loader must respect, both measured: + +- **The tokenizer is embedded AS A TENSOR** — `tokenizer_json` U8 [32,169,626] (~32 MB), plus + `hf_asset__{chat_template,generation_config,processor_config,tokenizer_config}`. A loader that + assumes a sibling `tokenizer.json` file fails on this checkpoint. +- **The TE quantization is torchao NVFP4, NOT compressed-tensors.** `weight` U8 packed, + `weight_scale` F8_E4M3 grouped, `weight_scale_2` F32 scalar, plus a `torchao_nvfp4` U8 [240] + marker per quantized module. H3's NVFP4 arm is compressed-tensors, so the layouts must be + verified before any reuse rather than assumed equal (L6). + +The checkpoint also carries the FULL multimodal Gemma-4 (`vision_model.*`, +`multi_modal_projector`, `audio_projector`); text-only conditioning is the scope, but the loader +must not choke on their presence. + +### 1.5 The audio VAE is NOT end-to-end causal + +Recorded 2026-08-12 from L4, which measured it rather than assuming it: its first causality +probes FAILED, and upstream agreed with the failure. + +`causality_axis` governs the audio decoder's **convolutions**, but its `AttnBlock`s attend over +the whole (time, mel) map, so a last-frame perturbation reaches every output frame. The Conv +video decoder is not end-to-end causal either, for a different reason: `res_x_y`'s shortcut norm +is a one-group GroupNorm over (C,T,H,W) whose statistics span time. + +This matters because "causal" is exactly the kind of property a port assumes and never checks. +Both are now gated in two parts: the shipped config asserts the GLOBAL reach, and a stripped +config isolates the convolution-only reach, with upstream itself supplying the expected windows +([5,8] audio, [3,4] video). + +## 2. Scope + +**In:** the DiT forward (both streams, gated attention, AV cross-attention, split and +interleaved RoPE); the Gemma-4 12B text encoder with its two caption projections; the Conv +video VAE, the audio VAE and its vocoder; the flow-matching pipeline including the distilled +two-stage recipe and the latent spatial x2 upsampler; the duration head; the FP8 and NVFP4 +arms; the generalized `VideoEngine` seam with H3 moved behind it; `/v1/videos`; e2e on +dgx.casa. + +**Out (recorded as owed, not silently dropped):** DiffVAE / `NADiffusionDecoder` (own row — +new neighborhood-attention kernel); the temporal x2 upsampler; LoRA fusion; multishot; +`int8-convrot` (ComfyUI-only quantization); multi-GPU / CFG parallelism. + +## 3. The oracle problem, and its resolution + +vLLM-Omni does **not** support LTX-2.5. Its recipe table keys on +`("one_stage","2")`, `("one_stage","2.3")`, `("distilled_two_stage","2")`, `("dmd2","2")`, +`("dmd2","2.3")` (`vllm_omni/diffusion/models/ltx2/ltx2_recipes.py:162-166`), and +`resolve_ltx_pipeline_recipe` raises `ValueError` on anything else. Upstream issues +[vllm-omni#6066 "[New Model]: LTX-2.5"](https://github.com/vllm-project/vllm-omni/issues/6066) +(filed 2026-08-11) and [#4985 "Align and expand LTX support"](https://github.com/vllm-project/vllm-omni/issues/4985) +are open. + +But its `DiffusersAdapterPipeline` is fully generic — it calls +`DiffusionPipeline.from_pretrained(model_id, **load_kwargs)` +(`pipeline_diffusers_adapter.py:116`) — so vLLM-Omni **can** execute 2.5 through +`--load-format diffusers` against `Lightricks/LTX-2.5-Diffusers` with diffusers installed +from main. + +**Resolution (developer-directed 2026-08-11):** + +- **Binding correctness oracle:** vLLM-Omni + diffusers adapter. Keeps AGENTS.md + §"vLLM is the reference" intact. +- **Immediate cross-check:** Lightricks `ltx-pipelines`, the model author's own runtime. + Available as soon as the `Lightricks/LTX-2.5` auto-gate is accepted, and it keeps every + phase unblocked while `-Diffusers` access is pending. + +Both are recorded per brick. Where they disagree, the disagreement is the finding. + +**BINDING-ORACLE PARITY IS PENDING FOR EVERY BRICK LANDED SO FAR** (recorded 2026-08-12, from +L2's review). L1–L5 gate against the CROSS-CHECK (`ltx_core` executed at reduced dimensions), +not against the binding oracle, because `Lightricks/LTX-2.5-Diffusers` access is still +awaiting manual approval. That is legitimate under §3 and §6 and it is what "immediate +cross-check" is for — but it must be stated, not left implicit. Concretely: + +| Axis | State | +|---|---| +| DiT / VAE / text-encoder parity vs `ltx_core` (cross-check) | gated, per-brick max abs diff recorded | +| DiT / VAE / text-encoder parity vs vLLM-Omni (BINDING oracle) | **PENDING** on `-Diffusers` access | +| Throughput vs vLLM's production configuration | **PENDING**, structurally, per §0 | + +`docs/BENCHMARKS.md` records only the SPEED axis as pending, which understates it; the +correctness axis against the binding oracle is pending too. Neither is a failure, and neither +is a pass. §3's instruction to "record the vllm-omni SHA inline with every golden" is +therefore N/A so far rather than satisfied, and saying so is the point. + +**There is still no vllm-omni parity PIN** — `.agents/upstream-sync.md` covers the vLLM repo +only. This spec inherits H3's open gap (model-matrix, H3 row: *"OPEN: there is no vllm-omni +parity PIN"*) and records the vllm-omni SHA used for every golden inline with that golden. + +## 3.1 The two DiTs are NOT interchangeable quantizations of the same weights + +Recorded 2026-08-12, after the developer accepted the HF licence and the first-party weights +downloaded. All six files verified: declared payload equals file size exactly, 23.01 GB total. + +The ungated `vonkaiser` **FP8** DiT and Lightricks' first-party **NVFP4** DiT are both +"ltx-2.5-22b-distilled-transformer", and every phase before L6 gated against the FP8 copy +alone. They agree on 4348 of 4349 non-scale tensor names — and disagree on one: + +| Family | FP8 | NVFP4 | +|---|---|---| +| `prompt_adaln_single` | 12 | 12 | +| `audio_prompt_adaln_single` | 6 | 6 | +| `video_embeddings_connector` | 129 | 129 | +| `audio_embeddings_connector` | 129 | 129 | +| **`keyframes_abs_pos_embedding`** | **1** | **0** | + +`model.py:216-219` builds it only when `use_keyframes_abs_pos_embedding` is set, and its +comment reads: *"Marks tokens whose latent encodes a single standalone pixel frame. +Zero-initialized, so a checkpoint that predates it behaves identically until the parameter is +trained."* + +**It is trained in the FP8 checkpoint.** Read directly: `F8_E4M3 [1, 4096]`, byte values +spread across `[24, 35, 39, 41, 42, 44, 45, 46, ...]`, with a real +`keyframes_abs_pos_embedding_scale` of `7.68899917602539e-06`. Not zeros. + +So the upstream escape hatch — "behaves identically until trained" — **does not apply here**. +The two files differ in a TRAINED parameter that marks single-standalone-frame latents. On any +request that uses keyframe conditioning they are different models, not two precisions of one. + +**CORRECTION, 2026-08-13 — the practical reach was OVERSTATED.** A third implementation +settles it. `huggingface/diffusers` main documents `keyframes_abs_pos_embedding` as +*"Zero-initialized in the reference; **unused by the regular distilled forward** until a +dedicated keyframes pipeline applies it after `proj_in`"* +(`src/diffusers/models/transformers/transformer_ltx2.py:1116-1119, 1199-1200`), and defaults +`use_keyframes_abs_pos_embedding` to `False`. + +So for the DISTILLED forward this campaign actually runs, that tensor is **not consumed at +all**. The difference between the two files is real and the FP8 copy's parameter is genuinely +trained, but the claim below that they are "different models" on any keyframe request +overstates what it means for our path: no forward we run reads it. What remains true is the +narrow statement — the files differ, and a keyframes pipeline (which we do not have) would +distinguish them. + +Consequences, and none of them are optional: + +- **A parity number measured on one does not transfer to the other.** Everything gated so far + used the FP8 copy. +- **L7 must state which DiT produced each artifact**, every time. +- **L6's by-name refusal fires on FP8 and not on NVFP4**, because the family is simply absent + there. That is correct behaviour in both cases, but it means the two arms take different + paths, and a test that passes on one proves nothing about the other. +- Quantization coverage differs too: FP8 carries 1775 `F8_E4M3` quantized tensors; NVFP4 + carries 1176 `U8` + 1176 `F8_E4M3` group scales, so **fewer modules are quantized**. + +Checked because §1.2's retraction came from reading the shipped checkpoint and finding it +contradicted the spec. The same question asked of the second checkpoint found a second +difference. Ask it of every new artifact. + +## 3.2 The THIRD implementation, and the default it disagrees with + +Recorded 2026-08-13, after the developer asked whether the diffusers implementation had been +checked. **It had not been**, and that was a gap: §3 named vLLM-Omni's diffusers adapter as the +binding oracle ROUTE and recorded it PENDING on the gated `LTX-2.5-Diffusers` **weights** repo, +after which nobody read the diffusers **source** — which is public on GitHub and needed no gate +at all. AGENTS.md asks for verification against the running oracle *and* its source; this +campaign checked one implementation's source twice instead of two implementations once. + +`huggingface/diffusers` main carries `LTX2VideoTransformer3DModel` +(`src/diffusers/models/transformers/transformer_ltx2.py`, 1683 lines) plus the `ltx2` pipelines. +Three results from reading it: + +**(a) It confirms §1.2's retraction independently.** `use_prompt_adaln_single` defaults to +**`True`** (`:1185`), and `:677` documents `temb_prompt` as `None` only when it is `False` +("KV-cacheable"). The cacheable case is the exception, exactly as the retraction states. + +**(b) It corrects §3.1's reach** — see the correction there. The keyframes parameter is unused +by the regular distilled forward. + +**(c) It DISAGREES with our fallback defaults**, and that is the finding worth acting on: + +| | ours (`ltx2.h:106,112`) | `ltx_core` (`model_configurator.py:66,68`) | **diffusers** (`:1176,1179`) | +|---|---|---|---| +| `av_ca_timestep_scale_multiplier` | `1` | `1` | **`1000`** | +| rope double precision | `false` | `false` | **`True`** | + +We mirrored Lightricks' fallbacks faithfully, so ours are not wrong *as a port of ltx_core*. +But diffusers defaults both to what LTX-2.5 DECLARES. So a checkpoint carrying no metadata — +which is exactly the shipped `vonkaiser` FP8 DiT (§1.2, F2) — resolves under our defaults to a +configuration **neither upstream would produce**. That is weaker than the "differently +configured, not materially wrong" reading recorded at F2, and it is why L7's repair refusing a +config-less DiT unless one is named is the right shape rather than a nicety. + +Owed: decide whether our fallbacks should follow `ltx_core`'s (a faithful port of one +reference) or LTX-2.5's declared values (what both references actually run). That is a product +decision, not a porting one, and it wants its own row. + +## 4.1 The first-party NVFP4 DiT is SWIZZLED and HIGH-NIBBLE-FIRST + +Recorded 2026-08-13 by L9a, which was told the file was linear, measured instead of building, +and returned `NEEDS_DECISION`. **The shape cannot discriminate here, and that is the trap.** + +For every quantized layer in that file `N % 128 == 0` and `(K/16) % 4 == 0`, so the linear +shape `[N, K/16]` and the cuBLAS-padded swizzled shape +`[round_up(N,128), round_up(K/16,4)]` are **the same numbers** — `[4096, 256]` for +`attn1.to_q`. Our code knew only the OTHER 2-D framing of those same bytes, `to_blocked`'s +`[32*ceil(N/128), 16*ceil(G/4)]` = `[1024, 1024]`. All three describe one set of 1,048,576 +swizzled bytes. A shape test alone can never separate them. + +**The discriminating measurement.** The `vonkaiser` FP8 DiT quantizes the SAME base weights, so +it is an independent oracle. Dequantizing one module from both files, rows 0-127: + +| reading of the NVFP4 scale | rms | corr vs FP8 | rel rms err | +|---|---|---|---| +| LINEAR / lo-nibble-first (what the brief asked for) | 0.013564 | **0.000414** | 1.786 | +| LINEAR / hi-nibble-first | 0.013564 | 0.257746 | 1.558 | +| SWIZZLED / lo-nibble-first (our current dequant) | 0.009208 | 0.032296 | 1.394 | +| **SWIZZLED / hi-nibble-first** | **0.009208** | **0.995560** | **0.0946** | + +FP8 oracle rms 0.009167; swizzled/hi-first matches to 0.4%, linear is 48% high. The 9.46% +relative rms IS NVFP4 4-bit quantization error. Confirmed on 5 modules across 4 distinct +shapes, corr 0.9952-0.9956. **Control**: the same NVFP4 read against OTHER modules' FP8 weights +gives corr +0.021, +0.005, +0.002, +0.001 — so 0.9956 is signal, not method artifact. + +**Independently confirmed in the model author's own runtime**, which is what loads this +checkpoint family: `ltx-core/quantization/nvfp4/linear.py:6-7` ("element 2j in the **high +nibble**"; "E4M3 block scales … **cuBLAS 128x4 tiled layout**"), `ltx-kernels/csrc/nvfp4/ +quantize.cu:26-31` (`swizzled_offset`, `padded_cols == roundup(K/16,4)`), +`ltx-kernels/docs/NVFP4.md:27-29` ("expected in the default (`hi_first=True`) order"). + +**Two things this leaves.** `Ltx2UnswizzleNvfp4BlockScale` is already correct and framing- +agnostic — only the shape assertion in `Ltx2DequantTorchaoNvfp4ToBf16` hard-codes the +`to_blocked` framing. But `DequantNvfp4ToBf16` is low-nibble-first and this file is +high-first: **a different byte ENCODING, not a different scale indexing**, in a header shared +with the H3 and Laguna NVFP4 paths. That is why L9a stopped rather than reaching into it. + +### 4.2 An open question about work already shipped + +The torchao NVFP4 **text encoder** arm reads nibbles low-first, and its gate compares against +goldens produced by **our own low-first helper** — the generator's header even notes its e2m1 +LUT is not torch-decoded. So the TE's nibble order has never been checked against an +independent oracle. This is exactly the failure this project has recorded before: *a gate +comparing two arms through the same helper proves consistency, not correctness.* + +The DiT result does not transfer — different producer (torchao vs Lightricks `nvfp4-prequant`) +— and no second Gemma-4 checkpoint exists on the NAS to serve as an oracle. **torchao's own +source is public and defines the packing**, and that is the cross-check owed. + +## 3.3 Where the two references DISAGREE, and which one we follow + +Recorded 2026-08-13 from L11. §3 says: *"Where they disagree, the disagreement is the +finding."* This is the first place they actually do, and it is not a cosmetic difference. + +**The noised-state composition.** `ltx_core`'s `latent_cond.py:38-39` leaves the NOISY tensor +untouched and lets the noiser compose it. diffusers writes clean tokens INTO the noisy tensor +(`pipeline_ltx2_condition.py:1002`). **The two agree only at `noise_scale == 1`.** + +So at every other noise scale the conditioning differs, and the difference is invisible to any +shape, finiteness or dtype check — it is a correct-looking render of subtly wrong conditioning. +Following diffusers here would have been a silent divergence at every noise scale except one. + +**The divergence is LIVE in this project, not hypothetical.** L11's reviewer executed both +compositions on shared tensors and a shared noise draw: + +| `noise_scale` | max abs diff, `ltx_core` vs diffusers | +|---|---| +| 1.0 | 2.38e-07 (f32 round-off — they agree) | +| **0.909375** | **1.16e-01** | +| 0.5 | 6.38e-01 | +| 0.0 | 1.28e+00 | + +`ltx2_pipeline.cpp:1083` sets `stage2.noise_scale = Stage2DistilledSigmas().front()` = +**0.909375**. So the distilled two-stage recipe — the one this campaign actually runs — sits +exactly at a 1.16e-01 divergence, five orders above any golden tolerance and invisible to every +shape and finiteness check. + +**We follow `ltx_core`**, and the port asserts the noisy tensor is byte-identical, so the +choice is pinned rather than incidental. The reviewer's decisive argument is internal +consistency: L5 already landed `Ltx2GaussianNoise` as a direct port of `noisers.py:30-37`, so +adopting the diffusers write WITHOUT also replacing the noiser would double-apply the clean +tokens and be strictly wrong. The two halves must come from one reference, and one of them is +already landed. The reasoning: `ltx_core` is the model author's own +runtime and is what loads this checkpoint family, and §3 already names it the immediate +cross-check while vLLM-Omni access stays pending. This is recorded as a CHOICE with a reason, +not as an unnoticed coincidence. + +**Four further divergences**, all following `ltx_core`: diffusers has no frame-count crop (it +dies in `unflatten`); `latent_log_var` is hardcoded to `uniform`; `norm3` is `GroupNorm(1)` +where ours is `LayerNorm(C)` (shape-compatible, statistically different — the kind that passes +every structural check); and diffusers has **no reference-audio conditioning at all**. + +**One thing only ONE reference attests.** diffusers has **no mel front-end whatsoever**, so +slaney/slaney normalisation, centered-reflect padding and `power=1.0` rest on `ltx_core` +alone. That is recorded at the code site. A single-source fact is weaker evidence than a +cross-checked one and should be labelled as such rather than blend into the rest. + +## 3.4 The tower's oracle existed all along, and BOS is the second disagreement + +Recorded 2026-08-13 from L10. + +**L3's blocker was a `transformers` VERSION, not a property of the tower.** L3 recorded that +the Gemma-4 tower could not be gated because `gemma4_unified` was absent from `CONFIG_MAPPING`. +Established by execution, both directions: `/usr/bin/python3` at transformers **5.3.0** raises +`KeyError 'gemma4_unified'` (reproducing the blocker exactly), while a venv at **5.12.1** +builds and RUNS it. So the tower is now held to a running upstream instead of to invariants +derived from its own output — `gemma4.h` had said "grounded + compiles" since it landed, and +compiling is not running. + +**The tolerance is measured, not chosen.** The generator measures how far UPSTREAM'S OWN answer +moves between f32 and bf16, per hidden state, and emits that as the bound. The worst state +reaches 0.71x of its own floor and state 0 is bit-identical — i.e. we are closer to +upstream-in-bf16 than upstream-in-bf16 is to upstream-in-f32, at every layer. It cannot be +loosened to rescue a failure: loosening it means regenerating it, which means the oracle moved. +That is the shape every tolerance on this campaign should have had. + +**The second real disagreement (§3.3 was the first): BOS.** diffusers relies on +`add_special_tokens=True`, which on THIS tokenizer adds nothing — so following it would **drop +token 0 of every prompt**. `ltx_core` does not. We follow `ltx_core`, and the measurement is +recorded in the goldens header. Tokenization is token-exact against HuggingFace over the +shipped 262144-entry vocab, four prompts x 1024 positions, first mismatching index -1 on all +four. + +**The Gemma config came from an 84 KB range request.** The `vonkaiser` build has no +`__metadata__` at all, so the config had to come out of band; L10 read it authoritatively from +the OFFICIAL bf16 checkpoint's safetensors header without downloading the payload. It +independently re-confirms §1.4's corrected 188160 width. + +**Why `has_encoder()` is still false, and this is the honest part.** Upstream routes each +stream through an `Embeddings1DConnector` before cross-attention. Its math is ported +(`Ltx2ConnectorForward`), but its WEIGHTS are not loaded: they ship inside the DiT file as +`video_embeddings_connector.*` / `audio_embeddings_connector.*` (**372 tensors, measured**) and +are still among the modules the loader refuses as unported. So conditioning has nowhere to go, +and flipping the flag would promise a render that cannot complete. L10 MOVED the refusal rather +than lifting it — `encoder_path` still refuses, but the message now names the connector weights +instead of the tower, because the old message became false. + +## 4. Checkpoint access and placement + +Verified against the HF API on 2026-08-11 with the session token: + +| Repo | `gated` | Consequence | +|---|---|---| +| `Lightricks/LTX-2.5` | `auto` | Accepting the license opens it. Holds the **first-party NVFP4 DiT**, 18.72 GB | +| `Lightricks/LTX-2.5-Diffusers` | restricted (manual) | Needed for the **binding oracle**; request submitted | +| `vonkaiser/LTX-2.5-FP8-NVFP4` | none | **Unblocks L1–L2 today**: FP8 DiT + NVFP4 Gemma-4 TE | + +All artifacts land under `$CHECKPOINT_ROOT` = `/mnt/nas_share/checkpoints` (per `.env`), so +dgx.casa and the cluster nodes mount one copy rather than each pulling 30 GB. + +**Best GB10 arm** (119 GiB unified): + +| Component | Source | Size | +|---|---|---| +| DiT, NVFP4 | `Lightricks/LTX-2.5` `ltx-2.5-22b-distilled-transformer-nvfp4.safetensors` | 18.72 GB | +| Gemma-4 12B TE, NVFP4 | `vonkaiser` `gemma4-12b-with-proj-nvfp4-torchao.safetensors` | 7.40 GB | +| Video VAE + audio VAE, bf16 | `Lightricks/LTX-2.5` | 1.83 GB | +| Latent spatial upsampler x2, bf16 | `Lightricks/LTX-2.5` | 1.00 GB | +| | | **~29 GB** | + +> **RETRACTION, 2026-08-13 — the diagnosis below is WRONG, and it was the operator's.** The +> correction that follows said the first-party NVFP4 DiT stores `weight_scale` in the LINEAR +> `[N, K/16]` layout. **It does not. It is SWIZZLED, and it additionally uses the OPPOSITE +> NIBBLE ORDER from our dequant.** Phase L9a was dispatched to build a linear arm on that +> premise, measured it first, and stopped without implementing — building it would have made +> the file LOAD and render silently wrong output. See §4.1. + +**SUPERSEDED CORRECTION, 2026-08-12 — the DiT in that table does NOT load.** L8 tried it and +the loader refuses by name, before any forward: + +> `'transformer_blocks.0.attn1.to_q.weight_scale' is [4096, 256] but a SWIZZLED torchao scale +> for [4096, 4096] is stored as [1024, 1024]. The LINEAR shape [4096, 256] has the same element +> count, so reading one as the other type-checks and permutes every scale within a 128x4 tile.` + +The first-party NVFP4 file carries **no `.torchao_nvfp4` marker at all** and stores +`weight_scale` in the LINEAR `[N, K/16]` layout, where L6's dequant expects the swizzled one. +Same element count, so the mistake type-checks — which is exactly why the refusal exists. + +This is **L6 loader debt, not a forward defect**: it throws from `Ltx2StreamDitToDevice`. But it +means the arm this table calls "best" is the one that cannot load, and nothing had materialized +a tensor from that file before L8 — L7's shipped-checkpoint test only parsed the manifest. + +**What actually ran on the GB10 is the `vonkaiser` FP8 DiT** (21.0 GB, 6124 tensors), and §3.1 +records that the two are not interchangeable. Until the loader learns the linear layout, the +NVFP4 arm is aspirational and the FP8 arm is the real one. + +Comfortably inside the pool, and materially smaller than H3's ~41 GB GGUF arm. + +## 5. Design — the generalized seam + +`include/vllm.h` already models this shape: `vllm_video_model_params` carries `dit_path`, +`encoder_path`, `video_vae_path`, `audio_vae_path` as separate artifacts (ABI v12, ROW 2). +It is only the *internals* that are H3-typed — `vllm::multimodal::MiniMaxH3VideoEngine` — +plus two H3-specific fields (`partition` = fl2va/ref2va, and H3's 50-step default). + +**L1 introduces `vllm::multimodal::VideoEngine`**, an abstract seam with a +checkpoint-detected registry, and moves H3 behind it **unchanged**. Per AGENTS.md +§"Shared seams", a capability not reachable through the shared surface is not done, and new +models are additive files. ABI goes to **v18 by ADDING fields only** — v12 video callers keep +working byte-identically, which the existing `test_capi` v12 section already guards. + +**Correction, 2026-08-11.** Earlier revisions of this section said "v13". That was wrong: it +read the VIDEO SLICE's own v12 label as if it were the ABI counter, when `VLLM_ABI_VERSION` +was already **17** and v13 shipped long ago as `vllm_complete_tokens`. The additive +requirement was always the real one and is unchanged; only the number moves, 17 -> 18. L1 +found this while implementing, which is the delegation loop working as intended. + +Reuse is the point. Already ours and shared, not re-implemented: the flow-matching denoise +loop, AdaLN block plumbing, 3D RoPE construction, VAE CNN infrastructure +(`minimax_h3_vae_cnn.cpp`), WAV writing, PPM frame serialization, the ffmpeg mux argv +composer, the NVFP4 resident-weight path and Marlin W4A16 dispatch, and — for the text +tower — `gemma4.cpp` / `gemma4_weights.cpp`. + +Genuinely new: dual-stream AV cross-attention, per-head gated attention, gelu-approximate +FFN, the audio embeddings connector, the two-stage distilled recipe with its latent +upsampler, and the duration head. + +## 6. Phases + +All on `row/MODEL-DIFFUSION-LTX25`, one PR. + +| Phase | Scope | Gate | +|---|---|---| +| **L0** | This spec; issue #435; checkpoint inventory; oracle stand-up | spec committed | +| **L1** | `VideoEngine` interface + registry; H3 behind it unchanged; ABI v18 additive | H3 frames+WAV **byte-identical** to pre-refactor on the committed fold fixture; v12 `test_capi` green | +| **L2** | DiT layout + forward: dual stream, gated attn, AV cross-attn, split/interleaved RoPE, prompt-KV cache | reduced-dim CPU parity vs upstream modules; cached vs recomputed prompt K/V **bit-identical** | +| **L3** | Gemma-4 12B TE + caption projections (4096 video / 2048 audio) | parity vs upstream TE; reuses `gemma4.cpp` | +| **L4** | Conv video VAE + audio VAE + vocoder | per-brick parity vs upstream decoders | +| **L5** | Pipeline: sigma schedule, distilled two-stage, latent spatial x2 upsampler, duration head | recipe values EXACT vs upstream | +| **L6** | NVFP4 DiT + NVFP4 TE arms; GB10 load-time residency | quantized vs bf16 wiring gate; residency per the ATS finding | +| **L7** | e2e on dgx.casa under `flock`; `/v1/videos` route | valid MP4+WAV; speed axis recorded `PENDING` per §0 | +| **L8** | Device-resident DiT forward on GB10; the CUDA `vt::AttentionCross` it needed | every dispatched op `vt-native` on a CUDA queue, ZERO reference-tier hits; device-vs-host at f32 round-off against the SAME upstream goldens | +| **L9a** | NVFP4 DiT: swizzled + hi-nibble-first (§4.1) | loads AND correlates ~0.9956 against the FP8 file's dequant, not merely finite | +| **L9b** | Real render on shipped weights; `--video-family`, `--video-extra` | frames + WAV + MP4 from the 21B FP8 DiT under its DECLARED config | +| **L9c** | Connector wiring + the missing pool Drain | conditioning real, not synthetic; peak device usage measured per phase | +| **L10** | The Gemma-4 tower, so a prompt works | gated against a RUNNING upstream, tolerance MEASURED from upstream's own f32-to-bf16 spread | +| **L11** | VAE encoders + image/keyframe/reference conditioning | both encoders at f32 round-off; the conditioning items EXACT | + +### 8.0 Two phases landed WITHOUT a spec, and both are the operator's failure + +L8 and **L10** were dispatched and landed with no spec section, against AGENTS.md's +"committed *before* implementation, never written up afterwards". In both cases a phase brief +stood in for the spec. That is the operator's error twice over, recorded rather than backfilled +as though the order had been kept. §8.1 sets out what L8's spec would have had to say; the +phase table above now carries L9a-L11 so the remaining phases are at least declared. + +The cost is measurable and was predicted in §8.1: a written scope would plausibly have caught +the ungated config adoption in L7, and — for L10 — would have forced the question "what is the +oracle for the tower?" up front, which is exactly the question whose answer turned out to be +*a newer transformers*. + +### 8.1 L8 — written AFTER the fact, and that is a process failure + +**AGENTS.md is unambiguous: the spec "is committed *before* implementation, never written up +afterwards."** L8 was dispatched and landed with no spec. That is the operator's error, not the +implementer's, and it is recorded here rather than backfilled as though the order had been kept. + +What the spec would have had to say, had it been written first: + +**Scope.** A device-resident LTX-2.5 DiT forward whose activations live in device memory, and +the CUDA `vt::AttentionCross` kernel it requires. Everything else routes through existing shared +seams (`vt::MatmulBT`, `Add`, `RmsNorm`, `LayerNorm`, `GeluTanh`, `Attention`, `AttentionCross`). + +**The risk that justified the phase, and that a spec would have named first.** GB10 reports +`UnifiedMemory`, so `RegisterReferenceTier` will serve a CPU kernel to a CUDA queue. With +`vt::AttentionCross` CPU-only, all six cross-attentions per block would have executed on the +host with every gate green and "it ran on the GPU" false. The gate therefore cannot be +"the tests pass" — it has to be *provider identity per op*, which is what +`VT_OP_PROVIDER_STATS=1` reports and what the review used. + +**Dtype.** bf16 is the production stream; `kF32` exists only as a gate arm, so the device +forward can be held to `ltx2_goldens.inc` at f32 round-off. Nothing widens a bf16 load. + +**Stop condition that should have been written down.** "No 'it ran on the GPU' unless every +dispatched op did, proven by provider identity rather than by a passing suite." + +**What the missing spec cost.** Two of the review's five MEDIUM findings are gate defects a +written scope would plausibly have caught up front: the L7 config adoption landed entirely +ungated (§7.0(c) again), and the new CUDA cross-attention's tiling machinery is never reached +by any fixture — every gate geometry is `tiles=1 npl=1 nblk=1 hq==hk`, while a real render puts +S at prompt length and Tq in the thousands, straight into the untested regime. + +## 7. Tests + +### 7.0 Four findings about the METHOD itself, and how they compound + +Recorded 2026-08-12. These came out of the L2/L3/L4 review rounds and they change what the +evidence is evidence *of*. They matter to every future brick, not just LTX-2.5. + +**(a) There is a CLASS of constants a reduced-dimension golden cannot see.** Not one +constant, a class. Confirmed independently on two phases. An epsilon, a clamp bound or a +normalize floor only becomes load-bearing in a regime the synthetic fixture never enters, so +it can be changed — sometimes by 100x — with every golden still green: + +| Constant | Mutation that stayed GREEN | Why the fixture cannot reach it | +|---|---|---| +| video VAE `pixel_norm_eps` | 1e-8 -> 1e-6 | activations are O(1); the epsilons differ by ~1e-7 relative | +| video VAE `norm_eps` | 1e-6 -> **1e-4** | same | +| BWE mel log clamp | 1e-5 -> 1e-8 | `mel_basis` is built non-negative and well-scaled, so nothing saturates. **Real silence DOES saturate it in production** | +| Snake/SnakeBeta `eps`, `_RMSNorm2D` floor | -> 0.0 | never divides by a zero-norm row | +| text `range_ + eps` | -> 0.0 | only matters when a whole (batch, layer) slice is constant | +| text `denom + eps` | -> 0.0 | only matters when `sequence_lengths == 0` | + +The repair is not "tighten the tolerance" — the tolerance is fine. It is a **source-anchored +constant assertion** pinning the value against the upstream line it came from, plus, where +feasible, a golden arm whose input actually enters the regime (L4 built one that attenuates +`mel_basis` so all 384 bins saturate, and it catches the mutation numerically). + +**(b) Byte-identical goldens are NOT evidence that the oracle was right.** L4's repair tested +this instead of assuming it: with a decoy `ltx_core` differing from upstream by ONE drifted +constant, and path precedence defeated, the generator imported the decoy, **exited 0, and +emitted goldens whose md5 was identical to the real ones**. + +That is (a) and (b) compounding. A drifted constant from the invisible class produces +identical goldens, so "the goldens reproduce byte-for-byte" cannot distinguish the right +oracle from a wrong one. Reproducibility proves determinism; it does not prove provenance. + +**Therefore every generator on this campaign owes two things, and they are separate:** +assert the resolved `ltx_core.__file__` lives under the `--ltx2` checkout (identity), and +record the upstream revision SHA in the emitted goldens (provenance). AGENTS.md already +required the revision anchor; the identity assertion is what makes the anchor mean anything. + +**(a-bis) The class was swept, and it had FIVE members, not one.** Recorded 2026-08-13 +(issue #560). After the fourth recurrence, every stabilizing constant in the LTX-2.5 files was +enumerated and mutated ALONE — 20 constants. Three categories emerged, and the middle one is +the interesting one: + +| verdict | count | meaning | +|---|---|---| +| pinned AND numerically reachable | 12 | a mutation moves a golden; the gate genuinely bites | +| **INVISIBLE — no arm read it at all** | **5+1** | a 100x change left every suite green | +| pinned, genuinely unreachable | **1** | upstream discards the value, so a pin is the only honest treatment | +| **MISLABELLED as unreachable** | **2** | live numeric path; the mutation was merely below fixture sensitivity | + +**CORRECTION, 2026-08-13.** The row above originally read "3 pinned, correctly unreachable", +taken from the sweep's own table and propagated into this spec by the operator without +independent check. Its review disproved two of the three, and the failure mode is the one this +whole section exists to name: **"upstream discards it" is exactly the label a fixture gap +wears when nobody probes it.** + +- `Ltx2ConvVideoDecoderConfig::norm_eps` is NOT discarded. `video_vae/resnet.py:93-97` builds + `norm3 = nn.GroupNorm(num_groups=1, ..., eps=eps)` **regardless of `norm_layer`** whenever + `in_channels != out_channels`. A `VT_CHECK` probe at our `norm3` call site fired on FIVE + goldens. The 100x mutation was simply below that fixture's sensitivity; at 1.0 the goldens + move by 1.64e-2 and 2.04e-2 against a 5e-6 band. Its ENCODER twin at the SAME 100x moves + 4.39e-5 and goes red — so the decoder's silence was an accident of fixture scale, not a + property of upstream. +- `kLtx2RmsNorm2dEps` is a `clamp_min` FLOOR in `F.normalize`, not a discarded value. It IS read + — setting it to `1.0` reds two encoder arms at 5.26e-4, so the path demonstrably executes — + but the floor itself binds only on an exactly-zero channel vector. + + **SECOND CORRECTION, 2026-08-13.** This entry originally went on to claim the project's own + BWE-quiet arm is precedent that such a probe IS constructible. A later reviewer disproved + that, and the distinction is one I should have drawn myself: in the BWE case **silence is a + legitimate INPUT**, so the probe is just a fixture. Here the floor binds on a **mid-network + activation vector**, and no legitimate input zeroes one downstream of a biased convolution. + The precedent does not transfer. So this constant is read-but-never-binding, and a pin really + is the only honest instrument for it. + + Worth stating plainly: in correcting the sweep for calling a live constant unreachable, I + overcorrected and called an unreachable one constructible. Both errors are the same failure — + asserting a reachability verdict without the probe that settles it — and mine had the + additional defect of citing a precedent I had not checked applied. +- Only `kLtx2EncoderApproxLnZero` is genuinely unreachable: `video_vae.py:325-334` concatenates + the block and `torch.chunk(...)[0]` discards it. Mutating -30 to -1 leaves the suite green + because upstream never reads it either. + +**And the sweep missed a sixth invisible constant**, `Ltx2AttentionArgs::norm_eps` +(`ltx2.h:365`) — same shape as the DiT case, all ten call sites assign it explicitly, so a +10^6 mutation leaves every suite green. A latent trap rather than live code today, but the +"class swept" claim was incomplete by one. + +**The lesson is sharper than the fix.** A change written to close this class mislabelled a live +constant as unreachable, and the operator copied the label into the spec without probing it. +Recording a hole is not removing it: the only evidence that a constant is unreachable is a +probe that FAILS to reach it, not a mutation that happens not to move anything. + +The fifth instance is the one worth remembering: **`Ltx2DitParams::norm_eps`**. Every test +passes that value explicitly through `ReducedParams`, so nothing ever read the FIELD DEFAULT — +which is live code, because `ParseLtx2DitParams` falls back to it exactly as upstream's +`config.get("norm_eps", 1e-06)` does. A constant can be invisible not because the fixture +avoids its regime, but because the fixture never lets the default apply. + +The repair that matters is not the pin. For the two audio-VAE `norm_eps` holes the fix was a +new gate arm at `norm_type = kGroup` / `causality_axis = kNone`, which makes the constant +**numerically** reachable: the same 100x mutation now moves the goldens by 1.13e-3 and 5.18e-3 +against a 5e-6 band. A source-anchored `CHECK` is the floor; a reachable arm is the gate. + +**(c) A FIXTURE that cannot separate right from wrong is the same defect, wearing different +clothes.** Recorded 2026-08-12 from L5's review, and it is the sharpest instance so far. + +L5's `linspace` mirror — the two-sided walk that makes the last sigma land on EXACTLY 0 — +was declared load-bearing in a comment and was **not gated at all**. Replacing it with a +naive forward walk left 33 cases and 1512 assertions green. It is not cosmetic: for **23 of +the first 198 step counts** the naive walk misses exact zero, and at `steps=41` a 5.96e-08 +terminal survives the `sigma == 0` guard, takes the shift transform, and displaces +`last_non_zero`, moving the WHOLE schedule by 0.1 so the denoise loop never reaches zero +noise. A plain `--steps 41` renders confidently and wrongly. + +It hid because the fixture exercises `steps in {8,6,5,1,4,7}` — every one of which the broken +walk happens to get right. + +Two other instances from the same phase, both caught before landing: + +- The duration head's three arms collapsed to within 2.98e-06 at the fixture's scale, **below + the round-off bound**, so an implementation ignoring one input stream would have passed. + Widening the scale to 0.35 separates them by 4.9e-02. +- A guider-refusal probe matrix omitted the `B=1` row, so a claim false at batch 1 — the + ordinary single-request shape — was recorded as a golden. + +**The generalization.** (a) is about a constant the fixture never drives into its active +regime; (c) is about an INPUT the fixture never drives into the regime that discriminates. +Same failure, different axis. A golden proves only what its inputs can distinguish, so for +every brick ask: *what input would tell a correct implementation from a plausible wrong one, +and does the fixture contain it?* Sweeping a parameter (step counts, scales, batch sizes) +costs almost nothing and is what turns a golden from a witness into a gate. + +**(d) A BROKEN ENVIRONMENT can impersonate a repair.** Recorded 2026-08-13, from the #516 +diagnosis, and it is the variant with the worst timing. + +`test_ltx2_device`'s shipped-DiT case reads its checkpoint from `LTX2_SHIPPED_DIT`. If that +path is unreadable — which on this project means simply that dgx rebooted and +`mnt-nas_share.mount` lost its boot race again, as it has every single time — the case takes +its `SKIPPED` early-return and the suite reports: + +``` +[doctest] test cases: 13 | 13 passed | 0 failed +[doctest] Status: SUCCESS! +``` + +**That is indistinguishable from the defect being fixed**, and it arrives precisely when +someone is hoping to see green. A skip that reads as a pass is worse than a failure. + +**The rule, in the diagnosing agent's own formulation:** *a fixture that opts in on an +environment variable must assert the resource it names is USABLE, not merely that the variable +is SET.* `LTX2_SHIPPED_DIT` gates the case on `getenv() != nullptr`, so every downstream claim +rests on a path nothing ever checked. Nothing lies — each layer does exactly what it says — but +the composition manufactures the shape of a repair out of a broken mount. + +**`SKIPPED` is only honest when the operator CHOSE not to supply the resource, never when they +supplied one that is gone.** An opt-in fixture therefore owes a readability check on its input +and a refusal by name when that fails. + +**How narrowly this was missed, because the margin is the point.** The measurement hold ran +22:31:12-22:55:34; the box went down at 23:18. Twenty minutes slower in the lock queue and the +relaunch would have run against a dead mount and reported a clean green suite — and the most +likely reading of that is "the red went away after a reboot", which is the worst possible +conclusion to draw about an allocator bug that returns **wrong answers silently**. + +This is (a)/(c)'s pattern from a new direction. There the INSTRUMENT could not see the defect; +here the instrument is fine and the ENVIRONMENT manufactures the shape of success. Both produce +a green that means nothing, and neither is visible in the summary line — which is why +`Status:` alone is not sufficient evidence either. Ask additionally: *did this run actually +execute the thing it claims to gate?* + +### 7.1 Method + +Mirroring H3's method, which is what made its bricks trustworthy: upstream's modules are +pure Python, so they are **executed at reduced dimensions on CPU** as the oracle, with both +sides rebuilding weights and inputs from an identical deterministic stream so **no weight +byte is checked in**. A generator script freezes upstream outputs; the C++ test asserts +against them. + +Specific traps this port must gate, each of which produces a *plausible but wrong* result +rather than an error: + +- **Per-head gate application order** — gating before vs after `to_out` differ silently. +- **`ff` bias presence** — reading a bias that is not there, or skipping one that is. +- **Cross-modal projection asymmetry** — `audio_to_video_attn.to_q` is `[2048, 4096]` while + `to_k`/`to_v` are `[2048, 2048]` and `to_out` is `[4096, 2048]`. Transposing any of these + still type-checks against a square assumption. +- **AdaLN slice mapping** — `slice(0,3)` / `slice(3,6)` / `slice(6,9)` are not interchangeable. +- **Prompt-KV caching** — asserted bit-identical against recomputation. +- **RoPE `split` vs `interleaved`** (`LTXRopeType`) and the optional float64 frequency path. +- **F32 `scale`-guard on tolerances** — per this project's doctest finding, `Approx` needs + `.scale(0.0)` or a 1.19e-5 absolute floor silently accepts anything. + +## 8. Risks + +| Risk | Mitigation | +|---|---| +| `-Diffusers` manual access never granted | `ltx-pipelines` cross-check keeps every phase unblocked; binding oracle recorded as pending, not faked | +| Upstream lands 2.5 in vllm-omni mid-campaign | Good outcome — re-anchor goldens to the native path and record the SHA; the diffusers-adapter goldens stay as the earlier evidence | +| Gemma-4 TE differs from our ported Gemma-4 | L3 gates the TE against upstream independently before wiring; a delta is a finding, not an adaptation | +| GB10 unified-memory OOM reboots the box | Never run a large oracle alongside ctest; `flock $HOME/gpu.lock`; park `local-ai-worker` | +| Contention with the 3 other coordinators | `flock` on every GPU-executing step; named tmux; single-load steady state | + +## 9. Stop conditions + +- A brick whose upstream reference cannot be executed → record `NEEDS_CONTEXT`, do not guess + the semantics from shapes. +- A parity delta that is not round-off → stop and report; never widen a tolerance to pass. +- Any temptation to declare a performance ceiling → forbidden by AGENTS.md; keep the gap open + and name the next traceable hypothesis. +- Speed axis: stays open until vllm-omni carries native 2.5 (tracked upstream at #6066) or + another production-configuration denominator is ratified. + +## Now + +L1-L6 merged and operator-gated on `row/MODEL-DIFFUSION-LTX25`. L7 (VideoEngine wiring + +dgx e2e) and L8 (device-resident forward) are implemented and reviewed FAIL on five MEDIUM +findings; the repair is in flight. + +**The DiT forward runs on the GB10 GPU.** Independently verified by the runtime provider +announcer, not by reading: all eight dispatched ops resolve `vt-native` on device 1 with +`registered=1`, and ZERO reference-tier hits across eleven logs. The shipped 21.00B FP8 DiT +(6124 tensors) staged and produced one finite forward, reproduced by the reviewer to the +digit (absmax video 0.300781 / audio 2.14062). + +**Two things that claim qualifies.** The FP8 DiT carries NO `__metadata__`, so it ran under +DEFAULTS (`av_ca_timestep_scale_multiplier = 1`, `double_precision_rope = false`) against +LTX-2.5's declared 1000 / float64 — the shipped weights on the right device in the wrong +configuration. And the first-party NVFP4 DiT, which §4 names as the best GB10 arm, does NOT +load (§4 correction). Next: the L7/L8 repair, then the squash landing. diff --git a/.agents/specs/nvfp4-nibble-order.md b/.agents/specs/nvfp4-nibble-order.md new file mode 100644 index 000000000..0705d4c2c --- /dev/null +++ b/.agents/specs/nvfp4-nibble-order.md @@ -0,0 +1,447 @@ +# NVFP4 nibble order — two producers, two conventions, one shared dequant + +**Row:** `MODEL-DIFFUSION-LTX25` phase L9a (`.agents/specs/ltx-2-5.md` §4.1, §4.2). +**Issue:** [#435](https://github.com/mudler/vllm.cpp/issues/435). +**Branch:** `row/LTX25-L9A-NVFP4-LINEAR`. +**Authority:** operator, 2026-08-13, after L9a returned `NEEDS_DECISION` rather than +building the arm it was briefed to build. +**Status:** spec committed BEFORE implementation, per AGENTS.md §"Spec before code". + +--- + +## 0. Why this needs its own spec + +`DequantNvfp4ToBf16` (`nvfp4_dequant.h:59`) is shared by the MiniMax-H3, Laguna +(Qwen3.5), DeepSeek-V4 and Qwen3-32B NVFP4 arms and by the `vt` fp4 GEMM reference. +Phase L9a needs it to read a checkpoint packed the other way round. A change there is +not LTX-2.5 loader work, so it is not made inside the LTX row — it is specified, +gated, and reviewed on its own terms. + +## 1. The fact, measured and sourced + +E2M1 packs two 4-bit values per byte. **Which logical element lands in which nibble is +a producer convention, and the two producers we consume disagree.** + +| Producer | Element `2j` goes in | Source | +|---|---|---| +| **torchao** (`NVFP4Tensor`) | the **LOW** nibble | `torchao/prototype/mx_formats/kernels.py:160` — `(uint8_data[::2] \| uint8_data[1::2] << 4)`; its own inverse at `:137-139` stacks `first_elements = data & 0b1111` first | +| **vLLM** (reads torchao/ModelOpt) | the **LOW** nibble | `nvfp4_emulation_utils.py:321-324` — `low = a_flat & 0x0F` then `torch.stack((low, high))`; the Triton path agrees at `:101-108` (`tl.interleave(low_result, high_result)`) | +| **Lightricks `nvfp4-prequant`** | the **HIGH** nibble | `ltx-kernels/docs/NVFP4.md:27-29` — "`hi_first=True` (default) puts element `2j` in the **high** nibble … Pre-quantized checkpoints used with `nvfp4-prequant` are expected in the default order"; `ltx-core/quantization/nvfp4/linear.py:6` | +| **"Star Ultimate Model Converter Pro"** (`lilcheaty/MiniMax-H3-NVFP4`) | the **HIGH** nibble | already recorded at `minimax_h3.h:1500-1515` | + +Our `DequantNvfp4ToBf16` is low-first (`nvfp4_dequant.cpp:74-75`), which is correct for +torchao and ModelOpt and wrong for the other two. + +**Read low-first, a high-first file has every adjacent pair transposed.** It is finite, +correctly shaped, and wrong — the exact failure class this project keeps recording. L9a +measured it on the first-party LTX-2.5 NVFP4 DiT against the independent `vonkaiser` FP8 +oracle: correct reading corr **0.9956** / 9.46% rel rms; wrong nibble order corr **0.032**. +See `.agents/specs/ltx-2-5.md` §4.1 for the full table and the control. + +### 1.1 §4.2 is ANSWERED, and the answer is "no shipped defect" + +`ltx-2-5.md` §4.2 recorded an open question: our torchao text-encoder arm reads low-first, +but its gate compared against goldens made by our own low-first helper — consistency, not +correctness. The answer is **low-first**, and the shipped TE arm is correct. This spec's +test work converts that self-referential gate into a source-anchored one so it cannot +regress into a question again. + +**What each witness actually witnesses — because "three independent witnesses" overstated +it.** torchao is neither installed nor vendored on this box (`import torchao` → +`ModuleNotFoundError`; no `mx_formats/kernels.py` anywhere on the filesystem), so the two +torchao citations are read from upstream source rather than executed or diffed here: + +| Witness | What it proves | Verified how | +|---|---|---| +| `torchao/prototype/mx_formats/kernels.py:160` (`pack_uint4`) | torchao's WRITER is low-first | upstream source only — **not executable on this box** | +| the same file `:137-139` (its unpacker) | torchao's READER agrees with its writer | upstream source only — **not executable on this box** | +| vLLM `nvfp4_emulation_utils.py:321-324` (`break_fp4_bytes`) | the **ModelOpt / compressed-tensors** convention is low-first | pinned tree at `555967922`, read and line-anchored | +| the Triton path at `:101-108` | the same, a second way | pinned tree, read and line-anchored | + +The last two are **not** independent witnesses to torchao's convention. `break_fp4_bytes` +is reached only from `dequantize_to_dtype` (`:380`), which serves the ModelOpt and +compressed-tensors NVFP4 paths; vLLM's torchao path (`quantization/torchao.py:290-318`) +delegates to `torchao.quantization.quantize_` and torchao's own tensor subclasses and never +calls it. So vLLM witnesses ModelOpt, torchao's own source witnesses torchao, and the claim +that rests on all of them is the weaker "ModelOpt and torchao agree, on two sources of +differing strength" rather than "three independent confirmations". + +This does not move the decision: the default is unchanged, and every caller of the shared +dequant that predates this spec consumes ModelOpt or compressed-tensors bytes — the arm the +strong witness covers. It is recorded so the strength of the anchor is not overstated to +whoever next changes the default. + +## 2. Scope + +**In.** One parameter on `DequantNvfp4ToBf16` selecting nibble order, **defaulting to +today's low-first behaviour**; the LTX-2.5 loader's producer discrimination and its use of +that parameter; the shape assertion admitting the cuBLAS-padded framing of the swizzled +scale; a correlation gate that proves the DiT dequantizes to the right numbers. + +**Out.** H3's `MiniMaxH3Nvfp4SwapNibbles` (§3.1 says why it stays); the fp4-resident +device GEMM path (`cuda_matmul_nvfp4.cu`), which stays low-first-only; MXFP4 and GGUF +NVFP4, which have their own packing and are untouched; the TE's tower wiring, owned by +`row/LTX25-L10-TEXT-TOWER`. + +## 3. Design + +```cpp +enum class Nvfp4NibbleOrder { + kLowFirst, // element 2j in the LOW nibble — torchao, ModelOpt, vLLM. DEFAULT. + kHighFirst, // element 2j in the HIGH nibble — Lightricks nvfp4-prequant. +}; + +void DequantNvfp4ToBf16(const uint8_t* packed, const uint8_t* weight_scale_fp8, + float weight_scale_2, int64_t out_dim, int64_t in_dim, + uint16_t* out_bf16, + Nvfp4NibbleOrder order = Nvfp4NibbleOrder::kLowFirst); +``` + +A defaulted trailing parameter makes "every existing caller is untouched" true **by +construction** rather than by inspection. The proof obligation in §5 is that the gates +agree. + +### 3.1 Why NOT H3's nibble-swap-at-load, which is prior art + +`minimax_h3.h:1500-1517` already solved this once, by swapping the two nibbles of every +byte at load so the file becomes standard low-first. That is a genuinely different design +and it was right **for H3**, whose bytes also feed a Marlin fp4-**resident** path: one +transform fixes the bf16 dequant and the device GEMM together. + +LTX-2.5 has no fp4-resident arm — `Ltx2StreamDitToDevice` dequantizes to bf16 and uploads +— so a swap would allocate and rewrite a second copy of 9.4 GB of packed weights to feed a +consumer that could just as well have read them in place. + +Two mechanisms for one concept is a real smell and it is recorded rather than hidden: +**if LTX-2.5 ever grows an fp4-resident NVFP4 device path, this decision must be revisited +and the load-time normalization adopted**, because a host-side dequant parameter cannot fix +a device GEMM that reads the packed bytes itself. That is the tracked condition, written +where it will be found. + +### 3.1.1 The SECOND tracked condition: the resolver does not refuse a linear file + +The marker-less arm is an inference, and §3.3 says why no shape test can corroborate it for +this artifact. The consequence, stated exactly rather than as the comfortable version: + +**A marker-less NVFP4 checkpoint whose `weight_scale` is stored LINEAR `[N, K/16]` is +RESOLVED as `nvfp4-prequant`, not refused.** For every geometry with `N % 128 == 0` and +`G % 4 == 0` — all 1176 quantized modules of the shipped DiT, gated as +`ambiguous_with_linear == 1176` — the cuBLAS-padded shape and the linear shape are the same +two integers. Such a file gets the 128x4 unswizzle applied to scales that were never +swizzled AND is read high-first. It loads, it renders, and it is wrong. + +Linear `[N, K/16]` is the normal case, not a corner one. It is what NVIDIA ModelOpt, +llm-compressor and compressed-tensors all write, and what vLLM's own readers allocate for on +disk (`modelopt.py:1335-1345`, `compressed_tensors/schemes/compressed_tensors_w4a4_nvfp4.py:73-76`, +both `[out, in // group_size]` at the pin `555967922`). None of the three emits a +`.torchao_nvfp4` sidecar, so **the marker's absence excludes torchao and nothing else.** + +It is resolved this way anyway because there is no better evidence in the file, and that was +checked rather than assumed: the shipped DiT's `__metadata__` carries exactly `config`, +`gemma_source_checkpoint`, `model_version` and `license` — no `quantization_config`, no +producer key, no `nvfp4`/`torchao`/`quant` substring anywhere in the config, and no tensor +name mentioning the quantizer (`MARKER_LIKE == []` over all 7876 tensors, read straight from +the 1,179,408-byte header). MiniMax-H3's community checkpoint DID name its converter in +metadata; this one names nothing. So the decision stands, and it is the **claim** that was +wrong: this resolver refuses every combination the SHAPE can separate, and cannot refuse the +one it cannot see. + +**The tracked condition.** Before this loader is pointed at any marker-less NVFP4 artifact +other than the first-party LTX-2.5 DiT, §5.2's correlation gate must be re-run for THAT +artifact against an independent oracle. Passing on the DiT is evidence about the DiT. If a +producer key ever does appear in such a file, key on it and demote this inference — that is +strictly better evidence than the shape, which is worth none here. + +### 3.2 The producer discriminator (operator-RATIFIED, 2026-08-13) + +The scale layout and the nibble order are decided together, from the **`torchao_nvfp4` +marker's presence or absence**. This is in-file evidence, not provenance guessing: torchao +always writes that marker, so its absence excludes torchao. + +| Marker | Required `weight_scale` shape | Resolves to | +|---|---|---| +| present, `is_swizzled_scales=true` | `[32*ceil(N/128), 16*ceil(G/4)]` (`to_blocked` framing) | torchao: unswizzle, **low**-first | +| absent | `[round_up(N,128), round_up(G,4)]` (cuBLAS-padded framing) | `nvfp4-prequant`: unswizzle, **high**-first | +| anything else the shape can SEPARATE | — | **REFUSE BY NAME** | + +`G = in_features / 16`. No "probably" branch: a marker with the wrong shape, an absent +marker with the wrong shape, or a marker declaring an unimplemented combination all refuse. + +**The third row's qualifier is load-bearing, not hedging.** Because §3.3 makes the padded +framing shape-identical to the LINEAR one for every geometry here, the marker-less arm cannot +refuse a genuinely linear file — it resolves it, and reads it swizzled and high-first. §3.1.1 +states that consequence in full and records the condition it imposes. + +**Both framings describe the SAME bytes.** `[32*ceil(N/128), 16*ceil(G/4)]` and +`[round_up(N,128), round_up(G,4)]` have equal element counts and the identical layout; +they are two 2-D dresses on one flat buffer. `Ltx2UnswizzleNvfp4BlockScale` is already +framing-agnostic — it keys off logical `rows`/`cols` and a byte count, and its index +formula matches `ltx-kernels/csrc/nvfp4/quantize.cu:26-31` term for term — so **only the +shape assertion changes**, not the permutation. + +**The assumption must be legible at the code site.** The marker-absent arm is an inference +from a producer's signature, not a fact read off the file, and the comment there says so +and cites the evidence: the 0.9956 correlation and `ltx-kernels/docs/NVFP4.md:27-29`. + +### 3.3 Why the shape cannot discriminate alone + +For every quantized layer in the first-party DiT, `N % 128 == 0` and `G % 4 == 0`, so the +cuBLAS-padded shape **numerically equals the linear `[N, G]` shape**. A shape test can +therefore never separate "swizzled, padded framing" from "linear". That is why the marker +is primary and the shape only corroborates, and it is why the original linear-arm brief was +unimplementable as written. + +## 4. Risks + +| Risk | Mitigation | +|---|---| +| A defaulted parameter silently changes an existing caller | Every NVFP4 gate re-run with **assertion counts compared**, not just Status (§5) | +| The high-first arm is wrong and output is merely finite | The correlation gate (§5.2) requires the 0.9956 / 9.46% signature; finiteness is not accepted as evidence | +| Marker-absent inference generalizes to a file it should not | **NOT mitigated by refusal, and §3.1.1 says so in the tree.** Every combination the SHAPE can separate refuses by name, with no fallback branch — but a marker-less file whose scale is stored LINEAR `[N, K/16]` is shape-identical to the padded framing for every geometry here, so it RESOLVES and is read wrong. The mitigation is §5.2's correlation gate against an independent oracle, owed per artifact, plus the tracked condition at §3.1.1 and the same statement in `ltx2_loader.h` and `docs/USAGE.md` | +| Two nibble mechanisms drift apart | §3.1 records the condition under which H3's is adopted here | +| The correlation gate needs a 21 GB and an 18.7 GB file | It reads byte RANGES from the two headers, materializes ~128 rows, and is opt-in behind an env var like the other shipped-checkpoint cases | + +## 5. Tests and gates + +### 5.1 RED first — and there are TWO reds, not one + +This section originally named one red ("the loader refuses the shape") for a change with two +independent halves. A throw can only ever demonstrate the first half, so stating it as *the* +right reason mislabels the evidence for the second. Both are required: + +**(a) The layout red — a THROW.** Before the change the loader knew only the `to_blocked` +framing, so the shipped DiT's cuBLAS-padded `weight_scale` was refused by name and no bytes +were decoded at all. That is a genuine red and it is the one the pre-change tree produces. + +**(b) The nibble-order red — a NUMBER, because no throw exists for it.** Fixing (a) alone +makes the file load and produce values that are finite, correctly shaped, correctly scaled +and wrong; §7 gates that absmax is IDENTICAL under both orders, so nothing structural can +fire. The only instrument that separates them is the correlation against the independent FP8 +oracle, and the red it must produce is the collapse recorded in §7: corr **-0.00239115**, +rel rms **1.41856**, against **0.994968 / 0.100672** for the correct read. + +Red (b) is not run once and discarded. It is a permanent arm of the committed gate +(`test_ltx2_loader.cpp`, the `kLowFirst` arm), for the reason the whole spec exists: the +defect it detects leaves no other trace. + +### 5.2 The correlation gate — the one nobody had +The `vonkaiser` FP8 DiT and the first-party NVFP4 DiT quantize the same base weights, so +the FP8 file is an **independent oracle** for the NVFP4 read. The gate: + +- dequantizes the same module from both files (rows 0..127), +- asserts Pearson correlation in **[0.99, 0.998]**, +- asserts relative rms error in the **BAND [0.085, 0.115]** — not a ceiling; see below, +- asserts a **control**: the same NVFP4 read against a DIFFERENT module's FP8 weights + correlates **< 0.2**, so the gate proves it can tell right from wrong rather than + passing on any two finite arrays, +- asserts the **wrong nibble order** collapses (corr < 0.2, rel rms > 0.5), +- asserts the two defects the band exists for, executed rather than described: a uniform + **x1.10 group-scale error** leaves corr unmoved to within 1e-6 and pushes rel above the + band, and the **oracle against itself** lands below the rel floor and above the corr + ceiling, +- and emits the VALUES, never a boolean. + +Without the control the gate is `7.0(c)` again — a fixture that cannot separate a correct +implementation from a plausible wrong one. + +**Why rel rms is a BAND (revised after review; the original `<= 0.15` was one-sided).** +rel rms here is a PREDICTED QUANTITY, not an error budget: it is the disagreement two +different quantizations of the same base weights MUST show, measured at 0.100672. A +one-sided ceiling was wrong in both directions, and both were measured on the committed +bytes rather than argued: + +| arm | corr | rel rms | `<= 0.15` | band | +|---|---|---|---|---| +| correct read | 0.994968 | 0.100672 | pass | **pass** | +| uniform group scale x1.04 | 0.994968 | 0.111935 | pass | pass | +| uniform group scale x1.05 | 0.994968 | 0.116760 | pass | **RED** | +| uniform group scale x1.09 (in the dequant) | 0.994984 | 0.141100 | pass | **RED** | +| uniform group scale x1.10 (in the dequant) | 0.994964 | 0.150094 | RED by 0.06% | **RED** | +| uniform group scale x0.92 | 0.994968 | 0.122618 | pass | **RED** | +| the FP8 oracle itself ("too good") | 1.0 | 0.0 | pass | **RED** | +| wrong nibble order | -0.00239 | 1.41856 | RED | RED | + +Rows marked "in the dequant" are real source mutations of `DequantNvfp4ToBf16`'s +`group_scale`, so the bf16 store-rounding moves with them; the others scale the decoded +floats afterwards, which is why x1.10 reads 0.148922 there and 0.150094 here. + +Two facts drive it. **Pearson correlation is scale-invariant** — corr reads 0.994968 to every +printed digit for every multiplier in that table — so rel is the ONLY statistic in the gate +that can see a systematic scale error, and slack spent there is not recoverable elsewhere. +And **rel had no floor**, so an arm that reproduced the oracle exactly — the shape a fixture +takes when the "independent" oracle has quietly become the thing under test — passed with +room to spare. The band's width (+/- ~15% of the measured value) is for compiler and libm +drift only: every input is a committed byte array reduced by a sequential double loop, so +the value is deterministic run to run. + +### 5.3 Nibble order, source-anchored +A unit case pinning both orders against hand-computed bytes, plus a generator-side anchor +on torchao's `pack_uint4` line and `ltx-kernels`' `hi_first` documentation, so a convention +change upstream is reported as a source change (§1.1's repair). + +### 5.4 The discriminator +Every row of §3.2's table, including each refusal, asserted by name. + +### 5.5 Unchanged-behaviour proof (operator's non-negotiable), and its REAL reach + +These must be green with **identical case and assertion counts**, reported. Baselines +captured on this branch at `f400413e`, Release, CUDA=OFF. + +**A byte-identical count is only evidence for a gate that EXECUTES the changed function.** +That was assumed rather than checked when this section was written, and the check changes +what the sweep is worth. Each gate was instrumented with a call counter on +`DequantNvfp4ToBf16` and then re-run under mutation **M3** — flip the shared default from +`kLowFirst` to `kHighFirst`, i.e. break every defaulting caller at once: + +| Gate | Baseline | Live calls into `DequantNvfp4ToBf16` | Under M3 | What the count proves | +|---|---|---|---|---| +| `test_nvfp4_dequant` | 4 / 47 | 6 | **RED** | sensitive | +| `test_gguf_nvfp4` | 14 / 2352 | 6 | **RED** | sensitive | +| `test_ltx2_loader` | 20 / 2363 | 294 | **RED** | sensitive | +| `test_qwen3_forward` | 7 / 1557 | 30 | GREEN | **BLIND — executes it and cannot see the flip** | +| `test_minimax_h3` | 79 / 57395 | 44 | GREEN | **BLIND — 57,395 assertions, none nibble-sensitive** | +| `test_qwen36_weights` | 7 / 45 | 0 | GREEN | vacuous (needs a checkpoint that is absent) | +| `test_ops_nvfp4_matmul` | 4 / 1 | 0 | GREEN | vacuous | +| `test_ops_moe_grouped` | 6 / 3 | 0 | GREEN | vacuous | +| `test_ops_nvfp4_fp4` | 22 / 919 | 0 | GREEN | vacuous | +| `test_ltx2_device` | 13 / 498 | 0 | GREEN | vacuous | + +So the honest statement is **three of ten** gates can see a nibble-order change. Five never +call the function, and their identical counts say nothing about it in either direction. Two +call it live and still pass — which is not a defect in this change (the default is unchanged, +so nothing moves) but IS a standing gap in those suites, and the reason "eight byte-identical" +was the wrong summary of this evidence. + +**The composition hazard that creates.** H3 reaches low-first by normalizing its bytes at +load (`MiniMaxH3Nvfp4SwapNibbles`, gated behind `MiniMaxH3Nvfp4HighNibbleFirst()`), then +calling the shared dequant with the default. If H3 is ever routed through the new +`kHighFirst` parameter while that swap stays on, the two compose into a double flip — the +original defect, restored — and **nothing in the tree fires**. Tracked as +[#598](https://github.com/mudler/vllm.cpp/issues/598): H3 owes a nibble-sensitive gate. The +instrument already exists; H3's own FL2VA-GGUF correlation is the same independent-oracle +technique §5.2 builds here. + +Counts only RISE, and only in the suites this change adds cases to. **If any other +gate moves, STOP.** `Status:` is grepped every run: a thrown case DROPS the assertion +count while still printing "passed". + +### 5.6 The real artifact +The first-party NVFP4 DiT stages onto the GPU and forwards, on dgx.casa under +`flock -w 2700 $HOME/gpu.lock`, naming the file, its tensor count and its derived geometry, +with finite non-degenerate output. + +## 6. Stop conditions + +- The high-first read does not reproduce the correlation signature → STOP; the encoding is + not merely a nibble order and the premise is wrong again. +- Any existing NVFP4 gate changes case or assertion count → STOP. +- A checkpoint appears that needs a third convention, or a marker/shape combination outside + §3.2 that the shape CAN separate → refuse by name and report; do not add a branch. +- A marker-less NVFP4 checkpoint other than the first-party DiT is pointed at this loader → + STOP until §5.2's correlation gate is run for THAT artifact. It will not be refused; §3.1.1 + says why, and passing on the DiT is evidence about the DiT. +- Any temptation to accept "the forward ran and the output was finite" as the correctness + result → forbidden; §5.2 is the result. + +## 7. Outcome + +**What was measured.** The first-party NVFP4 DiT is SWIZZLED and HIGH-nibble-first, and both +halves were established against an oracle that is not ours — the `vonkaiser` FP8 DiT of the +same base weights. Re-derived after a full-disk incident on the build host and identical to +every digit: + +| reading | rms | corr vs FP8 | rel rms | +|---|---|---|---| +| LINEAR / low-first (what the original brief asked for) | 0.013564 | 0.000414 | 1.786 | +| LINEAR / high-first | 0.013564 | 0.257746 | 1.558 | +| SWIZZLED / low-first (the shipped dequant) | 0.009208 | 0.032296 | 1.394 | +| **SWIZZLED / high-first** | **0.009208** | **0.995560** | **0.0946** | + +**What the committed gate reports, transcribed from its own output** (`test_ltx2_loader +--test-case="*NVFP4 DiT agrees*" -s`, Release, CUDA=OFF). An earlier revision of this section +recorded the wrong-order arm as `0.00514`, a number no row of the table above and no line of +the gate produces; the control on the same sentence, `0.00362`, was exact. Corrected against +the running gate rather than re-transcribed: + +| arm | corr | rel rms | +|---|---|---| +| oracle (the correct read) | **0.994968** | **0.100672** | +| control (a DIFFERENT module's FP8 weights) | **0.00362335** | — | +| deliberately-wrong nibble order (low-first) | **-0.00239115** | **1.41856** | +| absmax, both orders | identical at 0.0217285 | — | + +**What was rejected.** + +1. *A linear-layout arm.* The premise was wrong; building it would have made the file load and + render silently wrong. L9a stopped and returned `NEEDS_DECISION` instead. +2. *Discriminating by SHAPE.* Gated as impossible for this artifact: all 1176 quantized modules + have a cuBLAS-padded scale shape numerically identical to the linear one + (`ambiguous_with_linear == 1176`). The marker leads; the shape only corroborates. +3. *H3's nibble-swap-at-load.* Correct for H3, which also feeds a Marlin fp4-resident path, and + wasteful here: LTX-2.5 dequantizes to bf16, so a swap would rewrite 9.4 GB of packed weights + to feed a consumer that could read them in place. §3.1 records the condition that reverses + this decision. +4. *"The forward ran and the output was finite."* The gate asserts absmax is IDENTICAL under + both nibble orders, so every magnitude summary is blind to the defect by construction. Only + the correlation separates them. +5. *A one-sided rel-rms tolerance.* Revised after review into the band at §5.2: `<= 0.15` alone + admitted a +10% uniform group-scale error, and correlation is scale-invariant so nothing + else in the gate could have caught it. + +**Why the default is low-first.** It is what torchao's `pack_uint4` writes +(`kernels.py:160`), what torchao's own unpacker reads (`:137-139`), and what vLLM's readers of +ModelOpt / compressed-tensors bytes assume (`nvfp4_emulation_utils.py:321-324` and `:101-108`). +§1.1 records what each of those witnesses is actually worth — the two torchao lines are upstream +source, not executable on this box, and the two vLLM readers witness the ModelOpt convention +rather than torchao's writer. Every caller predating this change consumes a ModelOpt or +compressed-tensors checkpoint, so a defaulted parameter makes "nothing else moves" true by +construction. + +**§4.2 closed, no defect.** torchao is low-first, so the shipped text-encoder arm was already +correct. Its gate is now anchored to upstream sources rather than comparing our helper to +itself. The earlier phrasing — "three independent witnesses" — is corrected in §1.1: two of the +four cited lines are torchao's own writer and reader (one convention, seen twice), and the other +two are vLLM reading ModelOpt. + +**Gates.** Ten NVFP4 gates plus nine LTX/video gates green; the only counts that moved are the +suites this change adds cases to (`test_nvfp4_dequant` 4/47 -> 5/69, `test_ltx2_loader` +20/2363 -> 24/4809, the last +16 from the §5.2 re-banding and its two built-in mutation arms). +Mutations run, each with the tree restored byte-for-byte afterwards: + +| mutation | result | +|---|---| +| force low-first at the LTX call site | RED (3 cases) | +| resolve a marker-less padded file as torchao | RED (3 cases) | +| reinstate an apostrophe in a refusal | RED (1 case) | +| **M3: flip the shared default to `kHighFirst`** | RED in 3 of 10 gates; **GREEN in the other 7**, two of which execute the changed function 30 and 44 times. §5.5 has the per-gate call counts; the blindness is tracked as [#598](https://github.com/mudler/vllm.cpp/issues/598) | +| **M4: uniform group-scale error, applied inside `DequantNvfp4ToBf16`** | x1.10 -> rel **0.150094**, which the old `<= 0.15` catches by 0.06%. x1.09 -> rel **0.1411**: old bound **GREEN** (45/45, `Status: SUCCESS!`), §5.2 band **RED**. Same binary path, only the bound differs, so the band is strictly stronger. Now a permanent arm of the gate | +| **the FP8 oracle handed to the gate as its own answer** | Old bounds: every CORRELATION assertion passes — corr `1 >= 0.99`, rel `0 <= 0.15`, control `0.0026 < 0.2`, and both wrong-order arms — and only the incidental `got_absmax == wrong_absmax` fires, because `wrong` still reads the real bytes while `got` does not. The assertion that is supposed to BE the result passed. §5.2 band: **RED on the result itself**, `rel >= 0.085` and `corr <= 0.998`. Also now a permanent arm | + +**What the ten-gate sweep does NOT prove.** Eight-of-ten byte-identical was reported as the +unchanged-behaviour evidence. Five of those eight never call `DequantNvfp4ToBf16` at all on a +default run, so their identical counts are evidence of nothing about this change in either +direction. The claim that survives is narrower and still sufficient: the default is unchanged, +every pre-existing caller resolves to it by construction, and the three gates that CAN see a +nibble-order change are green. + +**On the GPU (dgx.casa, GB10, one `flock -w 2700 $HOME/gpu.lock` hold 02:31:03 to 02:37:15).** +Both arms stage device-resident and forward, `Status: SUCCESS!`: + +| arm | file | assertions | output absmax | +|---|---|---|---| +| NVFP4 | `ltx-2.5-22b-distilled-transformer-nvfp4.safetensors` (18,721,432,024 B, 7876 tensors) | 5626 | video 0.275391 / audio 1.02344 | +| FP8 | `ltx-2.5-22b-distilled-fp8.safetensors` (6124 tensors) | 5624 | video 0.300781 / audio 2.14062 | + +The FP8 figures reproduce the previously recorded GB10 run to the digit, which is what makes +the NVFP4 figures next to them trustworthy rather than merely new. The two arms differ because +they are different quantizations AND different configurations: the NVFP4 file declares a config +(`double_precision_rope`, `av_ca_timestep_scale_multiplier = 1000`, both asserted after +adoption), while the FP8 file declares none and runs under manifest defaults. + +**Not obtained.** The device case's MESSAGE text (printed tensor count, staging seconds, config +source) was cut by a `tail` in the harness on the first run, and a second run to capture it +timed out on the GPU lock after 45 minutes (`FLOCK_EXIT=1`) against a saturated box. The +underlying facts are asserted rather than printed — the geometry CHECKs are among the 5626 that +passed — and 7876 tensors / 48 blocks / video 4096 / audio 2048 / 21.004B parameters are gated +from the committed manifest. Cosmetic, and not worth further contended GPU time while another +row is rendering. diff --git a/.agents/specs/pool-device-key.md b/.agents/specs/pool-device-key.md new file mode 100644 index 000000000..5df3f324f --- /dev/null +++ b/.agents/specs/pool-device-key.md @@ -0,0 +1,704 @@ +# `POOL-DEVICE-KEY` — put the DEVICE in the `vllm::Pool()` free-list key + +**Issue:** [#516](https://github.com/mudler/vllm.cpp/issues/516) (open). +**Row:** `POOL-DEVICE-KEY`. **Base:** `row/MODEL-DIFFUSION-LTX25` @ `310fa1688`, +REBASED there from the stale `aac24761`, which was not an ancestor of +`origin/main` (§11 F1). The dgx pair in §11 was measured one campaign move +earlier, at `aa6aa0ecd`; §11 says why that is still the right pair. +**Owning file:** this spec. **Status at write time:** spec committed before any +implementation, per AGENTS.md "Spec before code". + +## Now + +`DONE` on `row/POOL-DEVICE-KEY`, landing THROUGH the campaign branch +`row/MODEL-DIFFUSION-LTX25` rather than onto `main` directly. That is a +sequencing fact, not a preference: `tests/vllm/models/test_ltx2_device.cpp` does +not exist on `main` at all, and it is the ONLY test that exposes the silent-NaN +direction, so removing its per-caller workaround — which §4 D6 requires, because +a list of remembered callers is what this fault was — needs LTX-2.5 to land +first. + +Records this row owns: this spec, the `#516` line in the issue table of +[`../roadmap_v1.md`](../roadmap_v1.md), the `POOL-DEVICE-KEY` paragraph under +"Backend detail" in [`docs/STATUS.md`](../../docs/STATUS.md), and the +`porting-inventory.md` §L8 note, which said the shared `DevicePool` "is +DEVICE-BLIND … repairing it is owed as its own row" and would have contradicted +the tree the moment this merged. `docs/BENCHMARKS.md` is deliberately NOT +written: this row claims no measurement on any speed, latency or memory axis +(§6), and a row with nothing to record there records nothing there. + +§10's nine UNATTRIBUTED dgx failures are no longer open against this row: the +BEFORE/AFTER pair in §11 reproduces all nine on the base WITHOUT this change, so +they are measured pre-existing and belong to whoever owns those suites. The one +test that moves is `test_minimax_h3`, SEGFAULT → Passed, which is +[#486](https://github.com/mudler/vllm.cpp/issues/486). + +Still not established, and not claimed: why a host `aligned_alloc` block yields +a uniform quiet NaN on GB10 rather than running correct-but-slow through ATS +(§7). `MoeAuxStreamFor`'s index-only aux-stream key is a recorded latent trap of +the same family, deliberately not widened into here. + +## 1. Scope + +`include/vllm/model_executor/models/device_pool.h` — the shared, process-wide +caching device allocator every dense/MoE/diffusion forward draws its scratch +from — keys its free list by **byte size class only**. The device is not in the +key. A block allocated through backend A is handed to a `DBuf` running on +backend B whenever the two ask for the same size class in the same process. + +In scope: + +- `device_pool.h`: the key, the pool lifetime, the `Pool()`/`AuxPool()`/ + `ActivePool()` accessors, `Drain`, and the backend-less `Put` overload. +- `dense_device_glue.h` and `qwen3_5.cpp` `DBuf`: how a released pool block is + handed to a `shared_ptr` (the ~28 copy-pasted deleters that name neither the + pool nor the device). +- The two existing **per-caller workarounds** for this same fault, which the fix + makes unnecessary and which must be REMOVED so the detector stays armed: + `tests/vllm/models/test_ltx2_device.cpp` (`cpu_pool` scope, the SILENT-NaN + direction) and `tests/vllm/models/test_deepseek_v2_forward.cpp` + (`cuda_pool`/`cpu_pool`, the SIGSEGV direction). +- `DevicePoolPolicy` memoization in `dense_device_glue.h`/`qwen3_5.cpp`, which + caches the FIRST device's residency policy in a function-local static and + applies it to every later device — the same ambient-device assumption, one + layer up. + +Out of scope, explicitly: + +- The size-class rounding itself. `VT_POOL_EXACT=1` (exact keying, reuse kept) + was MEASURED still red, so over-allocation is not the fault and the class + arithmetic is preserved byte-for-byte. +- `VT_POOL_BYPASS`, which stays a debugging lane and keeps its semantics. +- Any per-model change. If the fix makes a shipped model's gate red, STOP (§8). +- `ltx2_loader.*`, `ltx2_text_encoder.*`, the render path / engine wiring / + server flag, and `ltx2_device.cpp` — concurrently owned by three other rows. + This row needs no edit in any of them: `ltx2_device.cpp` reaches the pool only + through `DBuf`. + +## 2. The defect, as measured + +One fault, two symptoms, selected by direction: + +| direction | consumer | symptom | +|---|---|---| +| `cudaMalloc` block → CPU-backend forward | any CPU-backend case | **SIGSEGV** in `__memcpy_sve ← UploadStream ← PrepareStreamDev`, `compute-sanitizer` CLEAN (the fault is host-side) | +| CPU `aligned_alloc` block → CUDA forward | the shipped 21B DiT | **silent all-NaN output** | + +Census of the silent direction: `video n=1024 nan=1024 inf=0 zero=0`, +`audio n=512 nan=512`, every element the identical `0x7fff0000` — the bf16 +canonical quiet NaN widened by `WidenBf16`. A UNIFORM quiet NaN means a NaN was +COMPUTED and propagated; it is not plausible garbage read from a wrong pointer. + +Three arms already discriminate the cause: + +- `VT_POOL_BYPASS=1` (free list removed) → `test_ltx2_device` SUCCESS 13/13, + 6176 assertions. +- a per-case `DevicePool` via `ActivePoolScope` → SUCCESS 13/13, 6176. +- `VT_POOL_EXACT=1` (reuse kept, rounding removed) → still FAILURE. + +So it is **cross-device reuse**, not over-allocation, and not the pool's +existence. + +Why it stayed latent: it needs a bf16 host-backend device forward to run AFTER a +bf16 CUDA one in the same process. At f32 the two arms land in different size +classes and never trade blocks. The bug is old; only the ordering is new. + +**Not established, and not required for the fix:** why a host `aligned_alloc` +block yields NaN on GB10 rather than merely running slowly through ATS. Unified +memory makes host pointers device-addressable, so the naive expectation is +correct-but-slow — yet the red forward is FASTER (0.230–0.252 s) than the green +solo one (0.594 s). Untested candidates: `aligned_alloc`'s 64-byte alignment vs +`cudaMalloc`'s 256 breaking a vectorised/TMA load, or kernels that require true +device memory. Recorded in §7 as an open question; the fix makes the ordering +unreachable either way. + +## 3. Upstream anchors + +vLLM does not have this bug because it never had this design: every allocation +record carries its own device, and every cache operation is device-scoped. Read +at the pinned oracle `555967922` (`$VLLM_SOURCE`) and at the local +`torch 2.11.0+cu130` headers: + +- `vllm/device_allocator/__init__.py:12-14` — `HandleType` is documented + `# py_device, py_size_or_aligned_size, py_ptr, py_handle`. The **device is + field 0 of the handle**; the size is field 1. Ours keyed on field 1 alone. +- `vllm/device_allocator/cumem.py:200-219` — the free callback recovers the + allocation from the pointer and reads the device back OUT of the handle + (`device, size, d_mem, _ = data.handle`, `torch.cuda.synchronize( + data.handle[0])`). A released block is re-associated with the device it came + from, never with whoever asks next. +- `c10/cuda/CUDACachingAllocator.h:118-172` — the whole `CUDAAllocator` + interface is parameterized by `c10::DeviceIndex device` + (`getMemoryFraction`, `cacheInfo`, `getDeviceStats`, `releasePool`, …); the + cache is per-device by construction. + +Our `AuxPool()` comment already cites the STREAM half of the same invariant — +"two streams sharing one pool BREAKS" its reuse ordering, and torch answers that +with `record_stream`. The DEVICE half was never stated. + +This row therefore mirrors upstream's **partitioning** (one cache per device), +not its stream tracking, which stays out of scope: our reuse ordering is +single-queue per pool and `AuxPool` remains the seam for the second stream. + +`vllm/platforms/interface.py` `Platform` (our `platforms/interface.h` +`residency_policy()`) is likewise per-platform, which is why memoizing ONE +policy for the whole process (§1) is the same class of mistake. + +## 4. Design + +The device becomes structural, not a field someone must remember to pass. + +**D1 — one `DevicePool` per device.** `DevicePool` gains a bound backend +(`explicit DevicePool(vt::Backend&)`). `Pool(vt::Backend& b)` and +`AuxPool(vt::Backend& b)` resolve a per-backend instance from a process-wide +table; the no-argument `Pool()` and `AuxPool()` are **removed**, so an +unqualified "the pool" can no longer be spelled. `vt::Backend*` is the device +identity: the registry hands out exactly one `Backend*` per `Device{type,index}` +(`vt::RegisterBackend(Device, Backend*)`, `kMaxDevicesPerType`), and +`GetBackend(type)` and `GetBackend(Device{type,0})` return the identical +pointer. Keying on the backend needs NO new virtual on `vt::Backend` and so +ripples into no backend implementation. + +Lookup is a `std::mutex` + small vector, fronted by a thread-local +last-(backend,pool) memo, so the steady-state hot path is one pointer compare — +the pool exists to remove `cudaMalloc`, and must not pay a hash for it. + +**D2 — the pool VERIFIES its device on every use.** `Get`, both `Put`s and +`Drain` keep their existing `vt::Backend&` parameter and now throw +`std::logic_error` when it is not the pool's own backend. This is a hard runtime +check, not `assert`: the SACRED builds are Release/NDEBUG, where an `assert` +would compile out and hand back the pre-fix behavior. It is the standing +detector for this defect class, and it is what makes an `ActivePoolScope` +pointed at another device's pool a loud refusal instead of a silent corruption. + +**D3 — `ActivePool` resolves per device.** The thread-local becomes an +*override* defaulting to null; `ActivePool(vt::Backend& b)` returns the override +when set and `Pool(b)` otherwise. `ActivePoolScope` is unchanged in shape and +keeps serving the aux-stream case (`AuxPool(b)`), which is a stream distinction, +not a device one. + +**D4 — `DBuf::ReleaseShared()` replaces ~28 hand-rolled deleters.** Every site +today is literally `alloc_bytes()`, then `Release()`, then a `std::shared_ptr` +whose deleter closes over the byte count alone and calls `Pool().Put(alloc, q)`. +That idiom names neither the device nor the pool, so +it would also return AUX-pool blocks to the MAIN pool — a second bug in the same +three lines, but a LATENT one: enumerated at the base commit, none of the nine +`Release()` sites (`gemma4_moe.cpp:1197,1541`, `qwen3_5.cpp:6324,6520,6820, +7100,7134,8034,8065`) is inside or transitively under any of the four +`ActivePoolScope` regions (`laguna.cpp:2574`, `qwen3_5.cpp:5468,8644,8964`), +which are leaf-ward of all of them, so the old deleter and the DBuf's own +`pool_` agreed on every path that actually ran. The one place it WAS live at +base is `test_deepseek_v2_forward`'s own workaround, which this change deletes. +`ReleaseShared()` captures the buffer's OWN pool and +backend, so both are right by construction, and the backend-less +`Put(size_t, void*)` overload is removed with its last caller. Uncapped +retention is preserved for these cross-step buffers via a new +`Put(vt::Backend&, size_t, void*)`. + +**D5 — the residency policy is memoized per device type**, not once per process. + +**D6 — the two per-caller workarounds are removed.** They are the list of places +someone remembered; the fix is the property. Removing them is what proves it. + +Not chosen, and why: + +- *A composite `(Backend*, class)` key inside ONE pool.* Needs a + `void*`→`Backend*` side map to serve the backend-less `Put`, i.e. a second + hash operation on the hottest allocation path in the tree, to keep an overload + that should not exist. +- *A `virtual Device device() const` on `vt::Backend`.* Reaches every backend + implementation for information the caller already holds. Explicit stop + condition for this row. +- *Per-caller `ActivePoolScope`.* Already rejected in #516 and re-rejected here: + it is a list of remembered places, and the current red is exactly the siblings + nobody scoped. + +## 5. Tests + +**T1 (RED-first, the row's own gate) — `tests/vllm/models/test_device_pool.cpp`, +new target `test_device_pool`.** Hardware-free: two distinguishable fake +`vt::Backend`s, the technique `test_backend_multidevice` / +`test_reference_tier` already use, on `Device{kCPU,0}` and `Device{kCPU,1}` — +two INDICES of one registered type, not two types. `kXPU` was the first idea +and is wrong for the pool cases: it has no registered platform, so +`ResolveDevicePoolPolicy` throws before the pool is ever reached and every case +would measure the platform registry instead. `kXPU` earns exactly one case, the +one that is ABOUT that throw (T1.7). Cases: + +1. **The defect.** Allocate on A, free, then allocate the same size class on B. + The block B receives MUST NOT be the block A freed, and must come from B's + own `Alloc`. RED before the fix. +2. **Reuse survives.** Get/Put/Get on ONE backend returns the identical pointer. + Without this, "fixed" is indistinguishable from `VT_POOL_BYPASS` — the pool's + whole reason to exist is reuse. +3. **Size-class rounding survives.** Two byte sizes in one class still trade one + block, and `VT_POOL_EXACT` still separates them. +4. **`Drain(b)` is device-scoped.** Draining A frees A's blocks only; B's free + list is untouched and no block is freed through the wrong backend. +5. **A cross-device `ActivePoolScope` is REFUSED** (throws, names both devices) + rather than served. +6. **`ReleaseShared()` returns the block to its own pool and backend**, and an + AUX-scoped buffer returns to the AUX pool, not the main one. +7. **A backend whose PLATFORM is unregistered is refused.** D5's per-device-type + memoization calls `platforms::GetPlatform` once per device TYPE instead of + once per process, so an unregistered type now throws where it used to inherit + the first device's cap. That is the correct answer and it is a NEW failure + mode, so it gets a case rather than a comment. + +**The two debug lanes stay GREEN.** §10 hands `VT_POOL_BYPASS=1` to the next +reader as the cheap discriminator, so this suite must not red under it. Cases +2, 4, 6 and the AUX half of 6 state the ACTIVE lane's behavior (a pooled hit by +default, a fresh driver block under bypass); case 3 states sharing by default +and SEPARATION under `VT_POOL_EXACT`, which is the assertion its second clause +above always promised and never had. Both vars are read once into a +function-local static in `DevicePool`, so a process is in one lane for its whole +life and no case can toggle them. + +**T2 (end-to-end corroboration) — `test_ltx2_device --order-by=rand +--rand-seed=7`.** No checkpoint, no NAS, ~2 s. RED before the fix (exit 139, +`:533 FATAL ERROR: test case CRASHED: SIGSEGV`), and note the trap it carries: +**44 assertions, 0 failed, beside a SIGSEGV** — grep `Status:` and the CASE +count, never `assertions:` alone. Deterministic: default and `--order-by=name` +and seeds 1 and 7 are all red; the only green ordering is green by declaration +order, not by safety. + +**T3 (blast radius).** 20+ test binaries mix a CUDA and a CPU backend in one +process. Enumerate them from the tree (not from memory) and run the full suite +before and after. + +**T4 (#486 hypothesis).** `test_minimax_h3` SIGSEGVs on GB10 when two CUDA cases +share a process, `compute-sanitizer` clean. Run it under `VT_POOL_BYPASS=1` and +again after the fix. Green either way is a result and gets recorded; it does NOT +gate this row and the connection is not forced if it does not hold. + +Mutation targets a reviewer should exercise: delete the device from the key +(T1.1 must fail); delete the D2 device check (T1.5 must fail); make +`ReleaseShared` use the main pool (T1.6 must fail); make `Drain` free every +bucket (T1.4 must fail). + +## 6. Gates + +Correctness only; this row claims no performance result. + +- `test_device_pool` — new, must be RED before and GREEN after, with both + outputs captured. +- `test_ltx2_device` — GREEN at `--rand-seed=7`, at `--order-by=name`, and at + default order, **with no per-case pool scoping anywhere in the file**. +- Baselines that must not move (case/assertion counts): + `test_ltx2` 29/1615 · `test_ltx2_vae` 16/1816 · `test_ltx2_text_encoder` + 17/3350 · `test_ltx2_pipeline` 35/2358 · `test_ltx2_loader` 20/2363 · + `test_ltx2_video` 17/170 · `test_ops_attention_cross` 9/32 · + `test_minimax_h3` 79/57395 (CPU) · `test_minimax_h3_video_fold` 6/137 · + `test_video_engine` 11/254 · `test_capi` FULL 55/505. +- Full `ctest` on the CPU host before and after, and on dgx (`-j 1`, GB10 + unified memory OOM-reboots the box under a parallel CUDA suite). +- Every doctest result reported as its `Status:` line AND its case count; the + assertion count DROPS when cases throw and reads clean beside a crash. + +Build discipline: never redirect build output to `/dev/null`; chain on the build +exit; clean-rebuild after a header change, because an incremental build masks +`-Werror` and `device_pool.h` is a header. + +## 7. Risks + +| # | risk | mitigation | +|---|---|---| +| R1 | `Backend*` is not device identity if one backend object is registered for two `Device` indices | The registry stores one pointer per `Device{type,index}`; asserted by `test_backend_multidevice`. Recorded as the assumption it is. | +| R2 | The per-backend lookup lands on the hottest allocation path | Thread-local last-(backend,pool) memo: one pointer compare in steady state. No hash, no lock, on the hit path. | +| R3 | D2's runtime check costs a branch per `Get` | One perfectly-predicted compare against a member; the alternative (`assert`) is compiled out of exactly the Release builds the gates run. | +| R4 | ~28 deleter sites across 25 model files is a wide diff | Every site is byte-identical today, and each becomes ONE line via `ReleaseShared()` — the diff SHRINKS the call sites and removes an idiom that can be got wrong. No `ltx2_*` file is touched. | +| R5 | Removing the two per-caller workarounds could red a suite for an unrelated reason | They are removed in the SAME change that makes them unnecessary; if either stays red, that is a finding to report, not a scope to re-apply (§8). | +| R6 | A pool is now created per backend, so a mixed-backend process holds two free lists | That is the point. Retention is bounded by each device's own peak scratch, and `Drain` is now device-correct where before it freed one device's blocks through another's backend. | +| R7 | Static-destruction order for a table of pools that print `VT_POOL_STATS` at exit | Pools are owned by a function-local static table and outlive every model object; the stats path is unchanged. | + +## 8. Stop conditions + +- Keying by device turns out to need an API change that ripples into every + backend → STOP, report the design, write nothing. +- The fix makes a SHIPPED model's gate red → STOP and report; do not adjust the + model. +- `NEEDS_CONTEXT` for missing binding context; `NEEDS_DECISION` for a material + disagreement — never a silent scope change. +- Never weaken a bound, delete an assertion, or scope a caller away to reach + green. In particular, the LTX-2.5 shipped case is the ONLY test exposing the + SILENT direction: it does not get an `ActivePoolScope`. +- GPU work waits on `$HOME/gpu.lock` on dgx with a BOUNDED `flock -w`; on + timeout, report and stop. Never kill a holder. + +## 9. Evidence to record + +The committed SHA of this spec (before implementation); the RED `test_device_pool` +and RED `test_ltx2_device --rand-seed=7`; the GREEN of both plus the LTX-2.5 +device suite with no per-case scoping; the enumerated mixed-backend binaries and +the full suite before/after; the #486 result either way; every baseline as a +`Status:` line and case count; the exact `flock` lines, the wait, and `docker ps` +at both ends; `git log --oneline` and the final SHA. + +## 10. Outcome + +Spec `17532ea0b` (this file, committed before any implementation), RED test +`f4be8a4e2`, fix `1a2eb35ec` — the four original commits, later REBASED onto +`row/MODEL-DIFFUSION-LTX25` @ `2d437d5a9` (see §11), so those SHAs name the +history rather than the branch. + +**Environment.** dgx.casa, GB10 sm_121a, CUDA 13.0.88, configured with all three +MANDATORY confirmations printed: `CUTLASS found at ~/cutlass-4.5.0; enabling +sm120a NVFP4 cutlass GEMM`, `FlashAttention-2 prefill/decode: ENABLED for +arch(es) [121a]`, `Triton AOT: vendored tree …/sm_121a matches triton_kernels/ +(MANIFEST hashes OK)`. Both arms are CLEAN builds (`rm -rf build`), RED +1508/1508 exit 0 and GREEN 1508/1508 exit 0; the CPU host likewise clean, +1219/1219 exit 0, zero warnings under `-Werror`. `local-ai-worker` was stopped +before any GPU work and left DOWN. `mnt-nas_share.mount` had lost its boot race +after a reboot and was restarted with `sudo -n systemctl restart`; the shipped +DiT was then proven READABLE (21,025,119,068 bytes, first 16 bytes dumped) +before the opt-in case was allowed to run. + +**RED, at `f4be8a4e2`.** Both directions, on the same binary: + +| run | result | +|---|---| +| `test_device_pool` (CPU host) | 4 cases / 2 passed / 2 failed · 15 assertions / 5 failed · FAILURE · exit 1 | +| `test_ltx2_device --order-by=rand --rand-seed=7` | **exit 139**, `:533 FATAL ERROR: test case CRASHED: SIGSEGV` · 4 cases / 3 passed / 1 failed / 9 skipped · **44 assertions / 0 failed** | +| `--rand-seed=1` | exit 139, SIGSEGV at `:518` · 7 / 6 / 1 / 6 · 481 assertions / 0 failed | +| `--order-by=name` | exit 139, SIGSEGV at `:452` · 7 / 6 / 1 / 6 · 455 assertions / 0 failed | +| default order, `LTX2_SHIPPED_DIT` set (21B FP8) | **exit 1**, `:925 FATAL ERROR: REQUIRE( std::isfinite(v) )` · 13 / 12 / 1 · 4639 assertions / 1 failed — the SILENT direction | +| default order, fixture UNSET | SUCCESS 13/552 — the skip that impersonates a repair (§7.0(d)) | +| `test_minimax_h3` | exit 139, SIGSEGV at `:3974` · 38 / 36 / 2 / 41 · 42,724 assertions | +| `test_minimax_h3` + `VT_POOL_BYPASS=1` | SUCCESS 79 / 79 · 451,993 assertions · exit 0 | + +**GREEN, at `1a2eb35ec`** (`b4618b8c7` before an amend that added the +doc-gate argument to the message; the tree is identical), with NO per-case pool scoping anywhere (both +workarounds deleted): + +| run | result | +|---|---| +| `test_device_pool` | 8 / 8 · 26 assertions · SUCCESS (also under `--order-by=rand` seeds 1 and 7) | +| `test_ltx2_device` seed 7 / seed 1 / by-name / default | 13 / 13 · 552 assertions · SUCCESS · exit 0, all four | +| `test_ltx2_device` + shipped 21B DiT, default AND seed 7 | **13 / 13 · 6176 assertions · SUCCESS · exit 0** | +| `test_minimax_h3` (pool ON) | **79 / 79 · 451,993 assertions · SUCCESS · exit 0** | +| `test_deepseek_v2_forward` | 11 / 11 · 1558 assertions · SUCCESS | + +**#486 is this bug.** Asked as a question, not assumed: at the RED commit +`test_minimax_h3` SIGSEGVs with the pool on and is 79/79 with `VT_POOL_BYPASS=1` +— the bypass lane changes nothing but the free list. At the fixed commit it is +79/79 with the pool ON. The hypothesis is confirmed by measurement in both +directions. + +**ENOSPC re-verification.** The local box hit 100% disk during this session +(operator note). Every build log here was grepped for `No space left` — zero hits +in the local RED build, the local clean AFTER build, the dgx RED build, the dgx +GREEN build and the dgx AFTER `ctest` — and the local gates were then re-run +CHAINED to their build in one command, with `ninja` reporting "no work to do" +first, so no result here comes from a stale binary left behind by a died build. + +**Baselines, CPU host, full `ctest`: 402/402 passed, exit 0.** Every named +baseline byte-for-byte where it was: `test_ltx2` 29/1615 · `test_ltx2_vae` +16/1816 · `test_ltx2_text_encoder` 17/3350 · `test_ltx2_pipeline` 35/2358 · +`test_ltx2_loader` 20/2363 · `test_ltx2_video` 17/170 · +`test_ops_attention_cross` 9/32 · `test_minimax_h3` 79/57395 · +`test_minimax_h3_video_fold` 6/137 · `test_video_engine` 11/254 · `test_capi` +55/505. + +**Full CUDA `ctest -j 1` on dgx, AFTER: 98% passed, 9 failed out of 437.** The +nine, with their exact signatures: + +| test | signature | +|---|---| +| `test_serve_low_tools` | the Python bench-tooling suite (no C++, no GPU) | +| `test_linear_method` | `:246 CHECK( after == before + 1 )` → `0 == 1` — the `fused_gate_up` counter did not move, i.e. the fused Marlin gate-up path FELL BACK; its numeric arm passed at `bitexact=12288/12288 max_abs=0`, which is exactly what a fallback to the split path looks like | +| `test_ops_gdn` | `:728 CHECK( bad == 0 )` → `2609 == 0`, a GDN kernel numeric check | +| `test_capi` | **SEGFAULT** at `:482` "capi: custom logits processor forces the generated token (ABI v8)" — 4 cases / 3 passed / 1 failed / 51 skipped, 47 assertions / 0 failed | +| `test_glm4_moe_lite_paged_engine`, `test_qwen3_apc_e2e`, `test_minicpm3_paged_engine`, `test_internlm2_paged_engine`, `test_llama_paged_engine` | checkpoint-gated paged-engine suites | + +**These nine are UNATTRIBUTED, and that is stated rather than glossed.** The dgx +full-suite BEFORE arm was NOT run: the box was at 99–100% disk with a single 31 GB +build tree, so the RED and GREEN trees could not coexist, and the GPU lock was +shared with three other agents (one bounded `flock -w 2700` wait expired without +acquiring). Without that arm, "pre-existing" would be an inference, and this row +does not report inferences as measurements. + +What IS measured and bears on them: the same binaries are GREEN on the CPU host +in a 402/402 full run — including `test_capi` at its recorded 55/505 and +`test_linear_method` — and every pool-adjacent gate on dgx is green +(`test_ltx2_device` 13/13·6176 in four orderings, `test_minimax_h3` 79/79, +`test_deepseek_v2_forward` 11/11, `test_device_pool` 8/8). None of the nine +signatures is a cross-device scratch symptom: two are a dispatch counter and a +kernel numeric check that this diff does not reach, five are checkpoint-gated, +one is Python. + +**`test_capi` IS A TIMING FLAKE, and that one is measured, not inferred.** It +reproduced on the CPU host, where this row's change is CPU-only and every pool +gate is green: a first full `ctest -j 1` had it at its recorded 55/505 SUCCESS, +a second full run on the SAME binary (ninja: "no work to do") failed it, and +standalone it is **55/55 · 505 assertions · SUCCESS**, then `--repeat +until-fail:8` passed **8 of 8** with per-run wall times of 0.78, 1.02, 1.65, +2.00, 4.21, 12.56, 228.94 and 339.88 s. A test whose duration spans three orders +of magnitude on a contended box is timing-sensitive, which is what +`.agents/environment.md` already records for `test_capi`. Non-deterministic on a +host where the pool change cannot produce it ⇒ not this row's. + +**The exact next step, for whoever picks up the remaining eight.** Re-run them +standalone on this GREEN tree and again on a pre-fix tree; the cheap +discriminator that needs no second build is `VT_POOL_BYPASS=1`, which removes the +free list entirely — a failure that survives bypass cannot be a pooling failure. +The rerun script is `~/work/pool-device-key/dgx_rerun.sh`; it was written and +shipped but never ran, because two consecutive bounded `flock -w 2700` waits +expired without acquiring the shared GPU lock (three other agents were rendering +on it). Reported and stopped, per the lock protocol, rather than camping. + +**Blast radius, MEASURED not guessed.** `VT_POOL_STATS=1` makes every pool print +one line naming its backend at exit, so the full suite counts pools per binary +rather than grepping for device names. On the CPU host **42 of 402 test binaries +instantiate a `DevicePool` at all**, and exactly one — `test_device_pool` itself, +with its 11 fakes — instantiates more than one, which is the expected answer for +a host with a single backend. The grep-level upper bound is 51 test sources that +name both `kCUDA` and `kCPU`. + +**What was rejected.** A composite `(Backend*, class)` key inside one pool: it +needs a `void*`→`Backend*` side map to serve the backend-less `Put`, i.e. a +second hash operation on the hottest allocation path, to keep an overload that +should not exist. A `virtual Device device() const` on `vt::Backend`: reaches +every backend implementation for information the caller already holds, and was +an explicit stop condition. Per-caller `ActivePoolScope`: a list of remembered +places, which is what the current red already disproved. + +**Why the defaults are what they are.** The device check is a runtime `throw` +rather than an `assert` because the gate builds are Release/NDEBUG, where an +assert compiles out and hands back exactly the pre-fix behaviour. The pool +lookup is a thread-local last-(backend,pool) memo rather than a hash because a +`DBuf` resolves its pool on every construction and the pool exists to avoid a +synchronizing `cudaMalloc`. Size-class rounding is untouched: `VT_POOL_EXACT=1` +was measured still red, so it was never the fault. + +**Not established, and left open.** Why a host `aligned_alloc` block yields a +uniform quiet NaN on GB10 rather than running correct-but-slow through ATS. The +fix makes the ordering unreachable, so the question is now academic for this row, +but it is not answered and is not claimed to be. + +**Related, NOT fixed here.** `MoeAuxStreamFor` (`qwen3_5.cpp`) caches its aux +stream on `d.q.device.index` alone, so a CPU device 0 and a CUDA device 0 collide +in the key. It is unreachable today because the only call site is gated on +`Backend::SupportsAuxStream()`, which no host backend answers true to — so it is +a latent trap of this same family rather than a live defect, and widening this +diff to it was not worth the review surface. Recorded here so the next reader +finds it. + +**Owed at the time of §10, and PAID in §11:** the `#516` line in the issue table +of `.agents/roadmap_v1.md`, and the `porting-inventory.md` §L8 note which read +"the shared `DevicePool` is DEVICE-BLIND … repairing it is owed as its own row". + +## 11. Review round: the landing state, and the dgx pair §10 could not get + +A fresh review PASSED the fix and FAILED the landing state. Six findings; all +six closed. The endorsed code — the device key, `ReleaseShared`, both workaround +removals, `test_device_pool` — is unchanged in behaviour. + +### F1 — the base was stale, and its resolution would have reverted main + +The row was based on `aac24761`, which is **not an ancestor of `origin/main`**, +and `git merge-tree origin/main 7336def93` CONFLICTED in `device_pool.h`. Main +had changed that same file in `49539559d` / `8fa2ecdbb` (Windows contracts, +[#117](https://github.com/mudler/vllm.cpp/issues/117)) three ways the reviewed +header lacked: `__builtin_clzll` → `std::bit_width`, an `std::overflow_error` +guard in `ClassOf`, and `static size_t SizeClassForTest(size_t)`. Taking the +row's side of that hunk — the heavily rewritten side, which is what a resolver +reaches for — would have silently reverted the portability fix and the overflow +guard, and `tests/vt/test_cpu_isa_x86.cpp` calls `SizeClassForTest` **nine** +times including `CHECK_THROWS_AS(…, std::overflow_error)`, so the tree would not +have compiled. + +Rebased onto `row/MODEL-DIFFUSION-LTX25`, which contains `origin/main`, and +`device_pool.h` resolved BY HAND: all three of main's changes +sit **on top of** the device-keyed rewrite. `git diff ..HEAD -- +device_pool.h` is exactly three code hunks (the two includes, `SizeClassForTest`, +and the `ClassOf` body) plus the F4 comment. `test_cpu_isa_x86` is **6 cases / +8242 assertions / SUCCESS**. + +**Every number in §10 was measured on a tree missing main's last 201 commits and +none of it is carried forward.** The re-run caught drift §10 could not have seen: +`test_ltx2_vae` is **36 cases / 3039 assertions**, not the 16/1816 §6 records, +and `test_ltx2_loader` is **24 / 4817**, not 20/2363. Those are campaign-branch +commits (`6c9374ebc`, the VAE encoders, and the L9A NVFP4 loader work); this row +touches neither file. §6's baseline list is therefore stale by construction on a +moving campaign branch — what this row can honestly assert is that it moves +nothing, which the full 415/415 below and the dgx pair above both say. + +The rebase also caught a **new instance of this row's own defect**, added to the +campaign branch after the row was written: `ltx2_video.cpp:1154` called +`ActivePool()->Drain(backend)` — "the pool" resolved with no device, then drained +through a backend that may not have allocated its blocks. It is now +`ActivePool(backend).Drain(backend)`, and the old form is no longer spellable. +That is the argument for D1 restated by events: the removal of the no-argument +accessor is the fix; the free-list key alone would not have caught this. + +### F3 — the dgx BEFORE/AFTER pair, which §10 reported as not run + +Both arms on dgx.casa (GB10 sm_121a, CUDA 13.0.88), same session, back to back, +`ctest -j 1`, CLEAN builds (`rm -rf build`) from the SAME base, all three +MANDATORY confirmations printed for each (`CUTLASS found at ~/cutlass-4.5.0`, +`FlashAttention-2 … ENABLED for arch(es) [121a]`, six `MANIFEST hashes OK` +vendored Triton trees). `BUILD_EXIT=0` and zero `No space left`/`BFD assertion` +in both build logs and both ctest logs. + +| | BEFORE (`aa6aa0ecd`, no fix) | AFTER (this branch) | +|---|---|---| +| build | `BUILD_EXIT=0`, 34 GB tree | `BUILD_EXIT=0`, 34 GB tree | +| lock | acquired 08:26:13Z, released 09:22:00Z | waited 13m45s, acquired 09:40:20Z, released 10:34:02Z | +| disk at start / end | 62 GB free (99%) / 62 GB | 96 GB free (98%) / 60 GB | +| result | **10 failed of 449** | **9 failed of 450** | + +**The denominators differ by one because this row ADDS a test.** 449 → 450 is +`test_device_pool` existing in the AFTER arm and not in the BEFORE one; it is not +drift, and comparing 10-of-449 against 9-of-450 without that reads one test off. +The like-for-like statement is on the 449 tests both arms share: **10 failures +before, 9 after, and the one that leaves is `test_minimax_h3`.** + +| test | BEFORE | AFTER | +|---|---|---| +| `test_minimax_h3` | **`***Exception: SegFault` 11.73 s** (`ctest-before.log:51`) | **Passed 19.06 s** | +| `test_device_pool` | (does not exist) | **Passed 0.25 s** (new test #50) | + +Every other failure is the same set in both arms, and **every one of them is +already tracked by an existing issue** — checked, not assumed: + +| test | both arms | tracked by | +|---|---|---| +| `test_serve_low_tools` | Failed | [#233](https://github.com/mudler/vllm.cpp/issues/233) | +| `test_linear_method` | Failed | [#233](https://github.com/mudler/vllm.cpp/issues/233) (`VT_MARLIN_DENSE` defaults ON, so `fused_gate_up` never increments) | +| `test_glm4_moe_lite_paged_engine` | Failed | [#233](https://github.com/mudler/vllm.cpp/issues/233) (SACRED gate, token divergence) | +| `test_capi` | **SEGFAULT** | [#248](https://github.com/mudler/vllm.cpp/issues/248) | +| `test_ops_gdn` | Failed | [#305](https://github.com/mudler/vllm.cpp/issues/305), probably — see below | +| `test_qwen3_apc_e2e` | Failed | [#248](https://github.com/mudler/vllm.cpp/issues/248) | +| `test_minicpm3_paged_engine` | Failed | [#248](https://github.com/mudler/vllm.cpp/issues/248) | +| `test_internlm2_paged_engine` | Failed | [#614](https://github.com/mudler/vllm.cpp/issues/614), filed by this row | +| `test_llama_paged_engine` | Failed | [#248](https://github.com/mudler/vllm.cpp/issues/248) | + +**So all nine of §10's UNATTRIBUTED failures are now MEASURED pre-existing rather +than argued.** Nothing in this diff moves any of them, the row does not claim to +fix them, and they are named rather than adopted. This supersedes §10's "eight +remain open" as a statement about this row. + +**`test_capi` is NOT the timing flake §10 called it, and that correction +matters.** §10's flake evidence — `--repeat until-fail:8` passing 8/8 with wall +times spanning 0.78 s to 339.88 s — was gathered on the **CPU host**, where the +test fails without crashing. On dgx it **SEGFAULTs**, in **both** arms, in +**1.95 s** (BEFORE) and **1.97 s** (AFTER), under `ctest -j 1`. A sub-2-second +segfault is not a three-orders-of-magnitude timing spread, and #248 says so +directly: "`test_capi` is not the documented flake … this run was `ctest -j 1`, +so that explanation is unavailable and the SIGSEGV needs a real diagnosis." It is +pre-existing here either way, but it goes back to #248 as a crash, not into a +flake story it does not fit. Whether the two arms' crashes are the SAME crash is +**not established**: neither `LastTest.log` survives (both 34 GB build trees were +deleted to keep a 98%-full box under control) and these runs were plain +`ctest -j 1` without `--output-on-failure`, so the only recorded evidence is the +signal, the test index and the duration. + +**`test_internlm2_paged_engine` had no issue at all**, so this row filed +[#614](https://github.com/mudler/vllm.cpp/issues/614) rather than folding it +into a neighbour's: #248 lists four +paged-engine-family failures and this is not among them, and a search over open +and closed issues returns nothing. `test_ops_gdn` is attributed to #305 only +tentatively — #305 is a `conv_state` cross-block race and §10 recorded this +failure as `:728 CHECK( bad == 0 )` → `2609 == 0`, a numeric check that may or +may not be that race — so it is recorded as probable, not confirmed. + +It also **re-confirms #486 in both directions on one box**: SEGFAULT with the +pool device-blind, Passed with it device-keyed, same binary recipe, same lock, +14 minutes apart. §10 asserted this from two separate sessions; it is now one +paired measurement. + +The build trees were deleted after each arm (the box was at 98–99% throughout). + +**The pair was measured at base `aa6aa0ecd`; the branch is now rebased onto +`310fa1688`, fifteen commits later.** Chasing a moving campaign branch with a +five-hour paired CUDA gate does not terminate, so the honest thing is to name the +base the pair was taken at and say what would invalidate it. What the pair +measures is THIS ROW'S delta against a common base, and the fifteen intervening +commits are LTX-2.5 loader/VAE/NVFP4 work plus their goldens and specs: none +touches `device_pool.h`, `dense_device_glue.h`, the `DBuf` deleters, or any pool +accessor, and a tree-wide sweep for a device-less `Pool()` / `AuxPool()` / +`ActivePool()` spelling comes back empty on the new base. The CPU gate below WAS +re-run in full on `310fa1688`. A campaign commit that touched the pool would +invalidate the pair and is the one thing to re-check before merging. + +### F4 — a genuine hazard, but LATENT, and the assertions said "live" + +`device_pool.h`, `dense_device_glue.h` and §4 D4 claimed the old deleter idiom +also returned AUX-pool blocks to the MAIN pool as a *live* second bug. Enumerated +at the base: none of the nine `Release()` sites (`gemma4_moe.cpp:1197,1541`, +`qwen3_5.cpp:6324,6520,6820,7100,7134,8034,8065`) is inside or transitively under +any of the four `ActivePoolScope` regions (`laguna.cpp:2574`, +`qwen3_5.cpp:5468,8644,8964`), which are leaf-ward of all of them. All three now +say "would have" / latent. `ReleaseShared` is unchanged: a hazard one call site +away from real is worth removing structurally, and `ltx2_video.cpp` above is that +call site arriving. + +### F5 — the suite reds under the lane §10 recommends + +`VT_POOL_EXACT=1` was 7/8 and `VT_POOL_BYPASS=1` was 3/8 at the reviewed head, +and §10 hands `VT_POOL_BYPASS=1` to the next reader as "the cheap discriminator +that needs no second build". A suite that reds under the lane it recommends +costs that reader an hour deciding whose red it is. Every affected case now +states the ACTIVE lane's behaviour, and the size-class case states SEPARATION +under `VT_POOL_EXACT` — §5 T1.3's second clause, promised since the spec was +written and asserted nowhere. All three lanes are green: + +| lane | result | +|---|---| +| default | 9 cases / 30 assertions / SUCCESS (also `--order-by=rand` seeds 1 and 7) | +| `VT_POOL_BYPASS=1` | 9 / 24 / SUCCESS | +| `VT_POOL_EXACT=1` | 9 / 31 / SUCCESS | + +### F6 — a new throw with no test, and one un-bounded index + +D5's per-device-type memoization turned `platforms::GetPlatform` into a +per-type call, so a backend whose platform was never registered now throws where +it used to inherit the first device's cap. That is correct — a cap read off +another platform is a wrong number, not a default — and it had no test. T1.7 +covers it on `kXPU`, the one `DeviceType` with no `RegisterPlatform` call in the +tree, with a `REQUIRE_FALSE(HasPlatform(kXPU))` precondition so the case goes RED +and asks to be re-pointed if an XPU platform ever lands. The +`cached[static_cast(type)]` index gains the same `VT_CHECK` bound +`platforms::Index()` (`platform.cpp:40-44`) applies to that identical value, in +both mirrored copies. + +### Mutation evidence for the NEW assertions + +Product mutated in place, focused suite rebuilt and run, tree restored +byte-for-byte (md5 verified) after each: + +| mutation | result | +|---|---| +| `ResolveDevicePoolPolicy` returns cap 0 for an unregistered platform | 8 passed / **1 failed** — T1.7 only, `CHECK_THROWS_AS … did NOT throw at all!` | +| `ClassOf` ignores `VT_POOL_EXACT` | under `VT_POOL_EXACT=1`: 8 passed / **1 failed**, 4 assertions — the size-class case only | +| `Bypass()` forced `false` | under `VT_POOL_BYPASS=1`: 4 passed / **5 failed** | + +### Local gate, on the RESOLVED tree + +CPU host, at base `310fa1688`, CLEAN rebuild (`rm -rf build` — `device_pool.h` +is a header and an incremental build masks `-Werror`): `CONFIGURE_EXIT=0`, +`BUILD_EXIT=0`, **zero** `warning`, **zero** `No space left`/`BFD assertion`, +`ctest -N` **415**, full `ctest` **415/415 passed, exit 0** (one skip, +`test_voxtral_e2e`, checkpoint-gated). `test_cpu_isa_x86` — the suite F1's +resolution had to keep compiling — **6 / 8242 / SUCCESS**. `test_ltx2_device` +13/498 SUCCESS at default, `--order-by=name` and `--rand-seed=7`. +`test_deepseek_v2_forward` 11/1052. `test_minimax_h3` 79/57395. `test_capi` +55/505. + +`scripts/agent-preflight.sh` reports `doc-checkpoint range` FAIL on **three +commits, none of them this row's** — `b0aa475a3`, `d67f8125e` and `aa6aa0ecd`, +all pre-existing on the campaign branch and reproduced exactly by +`check-doc-checkpoint.py --base origin/main --head origin/row/MODEL-DIFFUSION-LTX25`. +Over this row's own range (`row/MODEL-DIFFUSION-LTX25..HEAD`) the same checker is +OK. `test_cpu_x86_llamacpp_floor` failed once at local load average 28 and is +10/10 OK standalone; it is a contention-sensitive harness test this diff does not +reach. + +### What is still not established + +Why a host `aligned_alloc` block yields a uniform quiet NaN on GB10 rather than +running correct-but-slow through ATS (§7, unchanged — the fix makes the ordering +unreachable, so the question is academic for this row and is not claimed to be +answered). `MoeAuxStreamFor`'s `d.q.device.index`-only key stays a recorded +latent trap of the same family, not fixed here. The nine dgx failures above +belong to other rows and are named, not adopted. diff --git a/CMakeLists.txt b/CMakeLists.txt index 37c8e6dcf..e166d11b2 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -780,6 +780,36 @@ add_library(vllm STATIC src/vllm/model_executor/models/minimax_h3_encoder_sharded.cpp src/vllm/model_executor/models/minimax_h3_encoder_device.cpp src/vllm/model_executor/models/minimax_h3_vision_gguf.cpp + # LTX-2.5 (row MODEL-DIFFUSION-LTX25, .agents/specs/ltx-2-5.md phase L2). + src/vllm/model_executor/models/ltx2.cpp + src/vllm/model_executor/models/ltx2_dit.cpp + # LTX-2.5 (row MODEL-DIFFUSION-LTX25, phase L8): the DEVICE-RESIDENT forward and + # the (kLtx2, device) resolver. The resolver is its own TU so the vt::GetOp cast + # links in CPU-only builds, exactly as minimax_h3_device_resolve.cpp is. + src/vllm/model_executor/models/ltx2_device_resolve.cpp + src/vllm/model_executor/models/ltx2_device.cpp + # LTX-2.5 (ROW MODEL-DIFFUSION-LTX25, phase L4): the Conv video VAE, the audio + # VAE and its vocoder. Additive files mirroring ltx_core's own structure. + src/vllm/model_executor/models/ltx2_audio_vae.cpp + src/vllm/model_executor/models/ltx2_video_vae.cpp + # LTX-2.5 (ROW MODEL-DIFFUSION-LTX25, phase L3): the Gemma-4 multi-layer feature + # aggregation, the two caption projections and the embedded asset pack. An + # additive file mirroring ltx_core's own structure. + src/vllm/model_executor/models/ltx2_text_encoder.cpp + # LTX-2.5 (ROW MODEL-DIFFUSION-LTX25, phase L5): the flow-matching pipeline, + # the latent spatial upsampler, the duration head and the embeddings + # connector. Additive files mirroring ltx_core's own structure. + src/vllm/model_executor/models/ltx2_pipeline.cpp + src/vllm/model_executor/models/ltx2_upsampler.cpp + src/vllm/model_executor/models/ltx2_duration_head.cpp + src/vllm/model_executor/models/ltx2_connector.cpp + # LTX-2.5 (ROW MODEL-DIFFUSION-LTX25, phase L11): the conditioning items that + # place the VAE ENCODERS' output into the denoise state. + src/vllm/model_executor/models/ltx2_conditioning.cpp + # LTX-2.5 (ROW MODEL-DIFFUSION-LTX25, phase L6): the quantized loaders — the + # FP8 and torchao-NVFP4 DiT arms, the torchao-NVFP4 text encoder, and the + # load-time device staging GB10's ATS penalty makes the default. + src/vllm/model_executor/models/ltx2_loader.cpp src/vllm/entrypoints/openai/video_api.cpp src/vllm/model_executor/models/kimi_kda.cpp src/vllm/model_executor/models/kimi_linear_registry.cpp @@ -883,6 +913,15 @@ add_library(vllm STATIC # server's private /v1/videos twin. Builds artifacts + the ffmpeg argv and # SPAWNS NOTHING (the ratified process boundary). src/vllm/multimodal/minimax_h3_video.cpp + # The GENERALIZED video seam (vllm::multimodal::VideoEngine) + its + # checkpoint-detected family registry — LTX-2.5 L1, .agents/specs/ltx-2-5.md + # §5. Knows about no family; each family self-registers from its own TU. + src/vllm/multimodal/video_engine.cpp + # LTX-2.5 behind that seam — the SECOND family, and the driving loop that + # composes the L2-L6 bricks into frames + a waveform (phase L7). Additive: + # one file plus one REGISTER_VLLM_VIDEO_FAMILY line, which is the property + # the seam exists to have. + src/vllm/multimodal/ltx2_video.cpp src/vllm/v1/core/encoder_cache_manager.cpp src/vllm/tokenizer/unicode_data.cpp src/vllm/tokenizer/pretokenizer.cpp @@ -1081,6 +1120,7 @@ add_library(vllm STATIC src/vt/cpu/cpu_threadpool.cpp src/vt/cpu/cpu_ops.cpp src/vt/cpu/cpu_minimax_h3.cpp + src/vt/cpu/cpu_ltx2.cpp src/vt/cpu/cpu_isa_arm.cpp src/vt/cpu/cpu_isa_x86.cpp src/vt/cpu/cpu_matmul_elem.cpp @@ -1487,7 +1527,9 @@ if(VLLM_CPP_CUDA) src/vt/cuda/cuda_sample.cu src/vt/cuda/cuda_deepseek_v4.cu src/vt/cuda/cuda_laguna.cu - src/vt/cuda/cuda_minimax_h3.cu) + src/vt/cuda/cuda_minimax_h3.cu + src/vt/cuda/cuda_ltx2.cu + src/vt/cuda/cuda_attention_cross.cu) find_package(CUDAToolkit REQUIRED) # cublasLt is linked now so the Task 4 matmul lands without a build change. target_link_libraries(vllm PUBLIC CUDA::cudart CUDA::cublasLt) diff --git a/README.md b/README.md index e8cb89ab6..eb8997e2b 100644 --- a/README.md +++ b/README.md @@ -186,7 +186,7 @@ configs, token-for-token the same output. Switching to it should be boring. Ever you get on top, most of it borrowed from whichever engine does it best: - **One 66 MiB binary instead of a 9.1 GiB install.** A flat, exception-free, llama.cpp-style C ABI - ([`include/vllm.h`](include/vllm.h), ABI v17, 35 functions) for C, C++, Go, or Rust. No Python + ([`include/vllm.h`](include/vllm.h), ABI v18, 36 functions) for C, C++, Go, or Rust. No Python interpreter in the process. - **GGUF as a first-class citizen.** Load the same quantized files llama.cpp uses, and on CPU **compute directly on the compressed blocks** (Q4_0/Q8_0/Q3_K/Q4_K/Q5_K/Q6_K) with no BF16 @@ -385,7 +385,7 @@ behind a model gallery, multi-model serving, the full OpenAI API surface, auth, ## Use it as a library (C API) Link `libvllm` and include [`include/vllm.h`](include/vllm.h): a flat, exception-free, -llama.cpp-style C ABI (`VLLM_ABI_VERSION 17`, 35 exported functions) suitable for `dlopen` / FFI. +llama.cpp-style C ABI (`VLLM_ABI_VERSION 18`, 36 exported functions) suitable for `dlopen` / FFI. ```c vllm_model_params mp = vllm_model_params_default(); diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index 58a1f5993..8ce7e32ae 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -435,6 +435,7 @@ built on it rather than keeping the flattering one. | Kimi-Linear-48B-A3B (KDA+MLA+MoE) | **RUNNER FOLD LANDS (ROW 7, §21, #122): engine==CLI 128/128 byte-identical; vs golden 122/128 (near-tie profile); FA2 MLA default-ON; SACRED green.** Server 19.0 tok/s wall; CLI 18.93 reproduced | vLLM ~21 (#111 floor; in-session re-measure ABORTED by GB10 reboot at util 0.82, §21): **~0.90×**, >= vLLM NOT met; residual = KDA host islands + grouped MoE + decode graph | | vLLM 0.26 re-benchmark | Pending | Re-run the binding grids on the advanced pin | | MiniMax-H3 FP4 speed (W-FP4a) | **Measured GB10 (`row/H3-FP4-GPU-E2E`).** Marlin W4A16 byte-exact vs bf16; fp4 a memory win, 0.8x bf16/forward. Real-ckpt fp4-resident e2e RUNS (mp4/wav) | fp4 speed CLOSED. bf16-vs-quant A/B: ENCODER half MEASURED (§8.15), DiT half NOT (no bf16 render exists). Detail: benchmark-record + spec §8 | +| LTX-2.5 axes | Speed `PENDING` (vllm-omni#6066 has no native 2.5), binding oracle too. **SIZE: 320x192/25f completes on GB10, 448x256 does not**; that render was REGISTER-conditioned, not prompted | Wall is the HOST VAE decode, not the pool: drain returns 0.11 GiB, byte-inert. 2 baselines UNRESOLVED (lock). A real-checkpoint PROMPTED render is OWED | | MiniMax-H3 render coherence (`row/H3-RENDER-CLOSE` #77) | **CLOSED: a COHERENT scene on GB10.** #70/#74 white was wrong-PARTITION usage (t2va on the ref2va ckpt); t2va on the FL2VA GGUF renders a prompt-matched orange cat (adj-cos 0.95 vs 0.06, no patch-grid) | Verified first: t2va inputs byte-exact vs upstream; CUDA device==host at seq 1920. Follow-up `H3-TASK-PARTITION-GUARD`: the task/partition mismatch now RAISES 1:1 with `_resolve_task` (spec §8.6-8.7) | | MiniMax-H3 image conditioning (`row/H3-CONDITIONED-E2E`, `row/H3-VISION-SCATTER`, `row/H3-REF2VA-ASSEMBLY`) | **fl2va COHERENT; ref2va assembly bug FIXED+gated.** vision→cond scatter gated; ref2va block-dim double-division fixed + RED-first gated (128 vs 512) + a permanent ref2va DiT-forward rung (§8.10) | grid RE-ATTRIBUTED: with the fix ref2va grids in fp4 AND bf16, and t2va with no refs on the ref2va NVFP4 also grids while FL2VA-GGUF renders, so it is the **NVFP4 checkpoint/loader**, NOT assembly/fp4 (§8.10) | | MiniMax-H3 Thor render speed (sm_110, no FA2) | **34.6 s/step** at 864x480/124f/50 steps on Q4_K_M, **16.6x** off 574.5 (render ~28 min, was ~8 h). Landed: warp-per-query, chunked warp reduce-scatter (1.76x), bf16 `mma.sync` (9.82x) | Shared-memory K/V tiling (23% SLOWER) and register Q-blocking (-0.8%) both measured and REVERTED: memory traffic is not the bound (one head's K+V is 3.9 MB against 32 MB of L2) | diff --git a/docs/ENVIRONMENT.md b/docs/ENVIRONMENT.md index cd8287f8a..395d1f779 100644 --- a/docs/ENVIRONMENT.md +++ b/docs/ENVIRONMENT.md @@ -161,6 +161,7 @@ Read-only observability; none change output. | `VT_FP4_AUTOTUNE_VERBOSE` | off | Log the NVFP4 GEMM autotuner's tactic selection | | `VT_H3_PROGRESS` | unset | Trace the MiniMax-H3 denoise loop's phases to stderr: which forward path was taken (device vs the CPU reference), how long the ONE-TIME device weight staging took, and per-step forward seconds with the sequence length. A real-checkpoint run spends its minutes in exactly one of those phases, and this says which without guessing — it was added after GPU-utilization counters proved unreliable on Tegra-class boards | | `VT_POOL_BYPASS` | off | `=1` makes every device-scratch pool allocation an exact-size driver `Alloc` and every release a real `Free`, so `compute-sanitizer` can see tensor boundaries and use-after-free that the caching, size-class-rounding pool hides. DEBUGGING ONLY: it reinstates the per-op `cudaMalloc`/`cudaFree` device-sync storm the pool exists to remove, so it is never a timing configuration | +| `VLLM_LTX2_POOL_DRAIN` | on | `=0` suppresses the LTX-2.5 render's per-phase device scratch-pool drain. The drain is on by default and there is no supported reason to turn it off: a denoise phase leaves the pool holding every activation size class it touched, the next phase (twice the resolution) and the VAE decode allocate different classes, and on an uncapped pool those blocks are headroom the next stage never gets. This exists so the A/B that MEASURES what the drain is worth runs on ONE binary, which is what a same-binary A/B requires. Not a configuration | | `VT_TTFT_DUMP` | unset | `=1` prints one `TTFTSPLIT rid=... intake=.. queued=.. prefill=.. decode=.. e2e=..` line per finished request to stderr, reconstructing the per-request timing split from the event-populated `req_state` timestamps. The async serving frontend otherwise tracks no per-request stats (passes `iteration_stats=nullptr` and never stamps `EngineCoreOutputs.timestamp`); under this flag both are wired so a serving TTFT attribution can read the queue-vs-execution split against vLLM's own `request_{queue,prefill,decode}_time_seconds`. Generation is byte-identical when unset (the default path is instruction-identical to production); the durable replacement is the async `/metrics` stat logger | | `VT_LOOP_TRACE` | unset | `=1` prints one `LOOPTRACE ...` line per ~1 s window to stderr from the engine busy loop: the full-iteration cadence (`interval`), `process_engine_step` wall (`step`), per-window admits, input-queue residence (`resid` = enqueue-to-drain, the same endpoints `VT_TTFT_DUMP`'s `intake` measures), per-drain admit max and max backlog depth. Diagnoses whether the admission wait is one busy-loop iteration or the input queue is backing up (it attributed the 35B INTAKE deficit to bursty arrival during long prefill steps). Byte-identical when unset: every trace read is guarded, and the enqueue timestamp is stamped only under the flag | | `VT_H3_GAUSSIAN_NOISE` | unset | `minimax-h3-gen` only: `=1` seeds the diffusion initial noise from Box-Muller GAUSSIAN N(0,1) (what a flow-matching model is trained on) instead of the historical uniform[-1,1] draw. A/B knob for the render-coherence investigation; the exact values still do not match torch's RNG (that only selects WHICH sample), but the DISTRIBUTION does | diff --git a/docs/FEATURES.md b/docs/FEATURES.md index dcb69a3cd..7586e32ac 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -58,6 +58,7 @@ are our reading of their documented behavior, not measurements. | KV events (block create / evict publish) | ◐ no transport | ✅ | ☐ | ☐ | | Prefix-cache matching unit | ◐ resolver only | ✅ | ☐ | ☐ | | Compute directly on quantized blocks | ✅ | ☐ | ☐ | ✅ | +| Scratch allocator keyed by device (two backends, one process) | ✅ since [#516](https://github.com/mudler/vllm.cpp/issues/516); a pool is bound to one backend and refuses any other, and a backend with no registered platform is refused rather than given another's residency cap | ✅ device is field 0 of the allocation handle | ✅ | ✅ | | Automatic memory sizing (no hand-tuned budget) | ☐ hand-typed block count | ☐ percent, hand-tuned | ☐ | ◐ | | Memory cap with a pre-flight error instead of an OOM | ☐ | ◐ KV pool only | ◐ | ☐ | @@ -147,13 +148,20 @@ speed-pending, which [BENCHMARKS.md](BENCHMARKS.md) tracks. ### Standalone and non-registered lanes These run through dedicated forwards, not the `REGISTER_VLLM_MODEL` registry, so -they sit outside the gated list above. +they sit outside the gated list above. One caveat the LTX-2.5 row is too narrow +to carry: its text tower's prompt tokenization mirrors upstream only while the +checkpoint's tokenizer `post_processor` adds nothing. The shipped one is MEASURED +empty, so this port's plain encode plus an explicit BOS prepend matches +upstream's `add_special_tokens=True` today; a checkpoint with a non-empty +`post_processor` would tokenize differently here, and `Ltx2TokenizeGemmaPrompt` +in `ltx2_text_encoder.cpp` is the call that would have to change. | Lane | Tested checkpoint(s) | Correctness gate | Speed vs reference | |---|---|---|---| | Voxtral audio (`VoxtralForConditionalGeneration`) | Voxtral-Mini-3B-2507 | near-tie-robust 16/16 vs vLLM 0.25.0 | decode 0.97x (beats vLLM); encoder FORWARD 15.90x of vLLM's whole TTFT (pin 46.02 ms), or 2.89x with opt-in `VT_WHISPER_ENC_FA2=1` (costs 3 near-tie divergences vs 0). Not a TTFT ratio. Pending | | Whisper audio encoder | openai/whisper-small; whisper-large-v3 (Voxtral cfg) | encoder tower 77/77; large-v3 tower 203/203 | pending | | MiniMax-H3 DiT (`MiniMaxH3DiTModel`, vllm-omni lane) | MiniMax-H3 (33.1B video+audio) | portable 79/79; all three modalities COHERENT on Q4_K_M (§8.20); PRUNED ckpts run, Q8_0 seam 0.9941 (§8.21); ref2va grid was NVFP4 quant error, §8.9 REFUTED; GGUF/NVFP4/bf16 shards stream | FP4/Marlin landed; speed pending; no bf16 render yet. Render from the Q4_K_M GGUF, not the NVFP4 arm. Krea 2 text-to-image (roadmap C11) is scoped to reuse these DiT seams | +| LTX-2.5 DiT (`LTX2VideoTransformer3DModel`, Lightricks lane) | LTX-2.5 (21.00B video+audio) | `SPIKE`. DiT, VAEs+ENCODERS, conditioning, pipeline, quant loaders gated at reduced dims. Typed prompt to Gemma-4 to cross-attn, FIXTURE-gated. The 320x192/25f scene was register-conditioned; a prompted render is OWED | Family `ltx-2.5` via `ltx2-gen`. ~29 GB NVFP4/GB10, FP8 ~44 GB, +~24 GB tower. FP8, torchao and first-party NVFP4 all load. DiffVAE, LoRA, image conditioning refused AT THE ENGINE. Speed PENDING | | MTP speculator | Qwen3.6-27B, Qwen3.6-35B-A3B | token-identical to vLLM `mtp` at c1 | ~4% faster c1; +16% output tput (MoE) | | DFlash block-diffusion | Qwen3 (DFlash draft) | near-tie e2e 27/27 vs vLLM | 2.9x over spec-off, 1.003x vs vLLM DFlash-on | | DeepSeek-V4 MTP | DeepSeek-V4-Flash (nextn head) | lossless 5/5; real-model weight-blocked | pending | @@ -269,7 +277,7 @@ Build with `-DVLLM_CPP_VULKAN=ON`; off by default. | Multiple engines in one process (build, destroy, rebuild) | ✅ resident device state is owned by the weights, so a new engine never inherits a freed one's pointers | ✅ | ✅ | ✅ | | LoRA adapters | ☐ CPU brick only | ✅ | ✅ | ✅ | | Embedding / pooling endpoints | ◐ `/v1/embeddings` live (task=embed; score/rerank/classify pending) | ✅ | ✅ | ✅ | -| OpenAI video generation `/v1/videos` (Sora shape) | ✅ `model`/`size`/`seconds` aliases + `GET /{id}/content`; `input_reference` and the `metadata` video/audio references condition the render | ◐ (vllm-omni, its own request shape) | ☐ | ☐ | +| OpenAI video generation `/v1/videos` (Sora shape) | ✅ `model`/`size`/`seconds` aliases + `GET /{id}/content`; `input_reference` and `metadata` references condition the render; `--video-family` pins the family (default DETECT), `--video-extra K=V` carries family knobs | ◐ (vllm-omni, its own request shape) | ☐ | ☐ | | Flat C ABI for embedding in other languages | ✅ versioned | ☐ | ☐ | ✅ | #### C-ABI capability coverage @@ -287,9 +295,9 @@ Build with `-DVLLM_CPP_VULKAN=ON`; off by default. | Custom logits processor | `vllm_logits_processor` | reachable | | Embeddings / pooling (task=embed) | `vllm_embed`, `vllm_embedding_result_free` (ABI v15; pooling checkpoints load via `vllm_engine_load`) | reachable | | Audio transcription (Parakeet ASR) | `vllm_transcribe`, `vllm_transcription_params_default`, `vllm_transcription_free` | reachable | -| Video+audio generation (MiniMax-H3) | `vllm_video_engine_load`, `vllm_video_generate`, `vllm_video_result_free`, `vllm_video_mux_argv` | reachable | +| Video+audio generation (MiniMax-H3, LTX-2.5) | `vllm_video_engine_load`, `vllm_video_generate`, `vllm_video_result_free`, `vllm_video_mux_argv`, `vllm_video_engine_family` (ABI v18 family registry) | reachable | | Explicit device selection (auto/cpu/cuda) | `device` field on `vllm_model_params` (ABI v14; 0=auto keeps the probe, explicit absent device fails loud) | reachable | -| Run the OpenAI server (server as a thin ABI client) | `vllm_server_main` (ABI v17) | reachable | +| Run the OpenAI server (server as a thin ABI client) | `vllm_server_main` (ABI v18) | reachable | | Multimodal input (image/audio/video) | none | embedder-unreachable | ## Parallelism and scale-out diff --git a/docs/STATUS.md b/docs/STATUS.md index 046f5d3a8..2acbe72f1 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -139,6 +139,7 @@ token-for-token correctness against the pinned oracle. | Laguna-S-2.1 MoE (`LagunaForCausalLM`, 118B/8B) | **BINDING 2026-08-04: was 87% of vLLM (37.55 vs 43.10, same-tool nsys)**; root cause was bf16 projections on UNIFIED/ATS host memory, and device-resident staging (byte-exact) gives 44.6, parity+ vs 43.1, default-ON | 48 layers (12 global + 36 SWA-512), 256 routed top-10 + 1 shared expert, per-head softplus attn out-gate, sigmoid `noaux_tc` router, dual per-layer RoPE, GQA 8 KV / 128 head-dim, 1M ctx. History: benchmark-record | | InternLM2 dense (fused-`wqkv` interleaved split) | Correctness-complete, speed-pending | Token-exact 16/16 (internlm2-chat-1_8b): 12/16 strict + 4/16 bf16 near-tie (max gap 0.0 nats), 0 divergent; first InternLM model; ZERO new compute kernel (reuses the Llama dense forward; the only delta is a loader-side de-interleave of the fused `wqkv`, which packs q/k/v interleaved by KV-group) | | MiniMax-H3 (`MiniMaxH3DiTModel`, video+audio DIFFUSION) | **ABI v12 ONE SURFACE; device selector uses generic `DeviceType`; DSR 32.** t2va+fl2va COHERENT; bf16 shards STREAM | ref2va ckpt fidelity §8.12; encoder A/B §8.15; GB10 re-verify residual; CPU fold 6/137 (one queue + device provenance mutation-gated) | +| LTX-2.5 (`LTX2VideoTransformer3DModel`, video+audio DIFFUSION) | **L1-L9c landed (#435).** 21.00B / 48 blocks. `VideoEngine` seam + ABI **v18**, DiT forward (CPU f32 parity, bf16 device-resident), Gemma-4 TE, both VAEs, the embeddings connector, pipeline, NVFP4/FP8 arms, `/v1/videos` | A shipped 21.00B FP8 DiT runs device-resident on GB10. The 320x192/25f frames ARE a scene, register-conditioned. L13 encodes a typed prompt, FIXTURE-gated; a prompted render is OWED. Speed and oracle parity `PENDING` | | Command-R / Cohere dense (`CohereForCausalLM`) | Implemented, gate-blocked | ZERO-new-kernel port grounded in vLLM `commandr.py`: weight-only Cohere LayerNorm + GPT-J full-width RoPE + PARALLEL residual + `logit_scale` + tied embeddings, all reuse; compiles, links, self-registers. No SACRED gate yet (real checkpoints HF-gated, ungated ones tiny-random, GPU box disk-full); oracle run-verified at W0. See docs/BENCHMARKS.md | | Phi-1 / Phi-2 dense (`PhiForCausalLM`, parallel residual) | Correctness-complete, speed-pending | Token-exact 16/16 (microsoft/phi-2): 9/16 strict + 7/16 bf16 near-ties (max gap 0.25 nats), 0 forward-divergent; the OLDER Microsoft Phi arch, DISTINCT from Phi-3/Phi-4; ZERO new compute kernel (GPT-J parallel residual, LayerNorm-with-bias, biased qkv/dense, partial NeoX rope 32/80, non-gated NewGELU MLP reusing `vt::GeluTanh`, untied biased lm_head); F16 dtype-aware loader | | MiniCPM dense (`MiniCPMForCausalLM`, three scalars) | Correctness-complete, speed-pending | Token-exact 16/16 (openbmb/MiniCPM-2B-sft-bf16): 10/16 strict + 6/16 bf16 near-ties (max gap 0.0 nats), 0 forward-divergent; first OpenBMB MiniCPM model; ZERO new compute kernel (the Llama/Granite dense forward plus three scalars: scale_emb, scale_depth/sqrt(layers) residual, dim_model_base logit scaling), tied lm_head; `.bin`-only weights converted to safetensors via trusted torch | @@ -1460,6 +1461,17 @@ platform missing from `CurrentPlatform()`'s hardcoded walk registers and answers correctly but is NEVER selected, with no compiler diagnostic. `test_platform` now gates that every `DeviceType` is in the walk and CPU is last. +**The device-scratch pool is now ONE POOL PER DEVICE (`POOL-DEVICE-KEY`, #516).** +It was a process-wide free list keyed by byte size class with no device in the +key, so in a mixed-backend process a block allocated through one backend was +handed to the next caller of that class on another: a `cudaMalloc` block reaching +a CPU forward SIGSEGVs host-side with `compute-sanitizer` clean, and a host block +reaching a CUDA forward returned a uniform `0x7fff0000` quiet NaN. A pool is now +bound to a backend, `Pool(b)` is the only spelling, every operation refuses a +foreign backend, and the two per-caller workarounds are deleted. `test_device_pool` +gates it without a GPU; `VT_POOL_BYPASS`/`VT_POOL_EXACT` keep their meanings and +the suite is green under both. + **CUDA architectures.** The runtime-gated production arch is GB10 `sm_121a` (every gate model, every benchmark). A build-supported cross-family fan-out (`sm_80/86/87/89`, `sm_90a`, `sm_100a/103a`, `sm_110`) compiles single-arch, diff --git a/docs/USAGE.md b/docs/USAGE.md index 375b9cf26..142cd6a17 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -128,6 +128,32 @@ context is never torn down, so the pointers stayed mapped — it simply produced corrupted or zeroed output tokens, intermittently ([#237](https://github.com/mudler/vllm.cpp/issues/237)). +More than one **backend** in one process is likewise supported — a CPU forward +running beside a CUDA one, which is what a diffusion pipeline with a host-side +stage does. Until +[#516](https://github.com/mudler/vllm.cpp/issues/516) it was not: the shared +device-scratch pool was a single process-wide free list keyed by byte size class +with no device in the key, so a block allocated through one backend was handed +to the next caller of that size class on another. It has two symptoms and the +direction picks which: a `cudaMalloc` block reaching a CPU forward segfaults in +the host `memcpy`, and a host block reaching a CUDA forward produces output that +is uniformly NaN rather than wrong. Neither can happen now — a scratch pool is +bound to one backend and refuses any other with a `std::logic_error` naming both +— and no user-facing flag or env var selects the behaviour: it is unconditional. + +One consequence is worth knowing before you add a backend. The scratch pool's +residency cap now comes from *that device's* platform rather than from whichever +device resolved first, so constructing a buffer on a backend whose platform was +never registered raises instead of silently inheriting another platform's cap. A +cap read off the wrong platform is a wrong number, not a default, and every +backend the tree ships registers one. + +`VT_POOL_BYPASS=1` and `VT_POOL_EXACT=1` keep exactly the meanings +[ENVIRONMENT.md](ENVIRONMENT.md) records for them. They are debugging lanes, not +timing configurations, and the pool's own test suite is green under both, so +either one stays usable as a discriminator when something else is under +suspicion. + ## Starting an agent-assisted contribution Run `scripts/agent-start.py` first. It reports an inherited worktree role or, @@ -347,6 +373,204 @@ tokens quietly. This is a deliberate state, not a bug: registering the architecture is what lets the config parse and weight-name mapping be tested before the forward exists. +### LTX-2.5: what runs, and what it cannot do + +LTX-2.5 is reachable as video family `ltx-2.5`, through the same +`vllm_video_engine_load` / `vllm_video_generate` C ABI that serves MiniMax-H3, +and through the `ltx2-gen` example that drives it. Its two VAE decoders, its two +VAE ENCODERS with the mel front-end, the conditioning items that place encoded +latents into the token stream, and its pipeline layer (the sigma schedule, the +diffusion steps, guidance, the latent spatial x2 upsampler, the duration head and +the embeddings connector) are implemented and gated. Several limits decide what +you can actually ask for, and each refuses by name rather than rendering +something else. + +In particular, the encoders being present does NOT mean image, keyframe, +reference-video or reference-audio conditioning is usable: the video engine +still refuses every one of those by name, because the request-side work between +a file on disk and a tensor the encoder accepts — image decode, aspect-fill +resize, and the H.264 CRF re-compression upstream performs before encoding +whenever the resolved CRF is not `0` and the image is at least 2 pixels on its +shorter side — is not ported. The engine also holds no +encoder to call: it materializes the VAE DECODER key filters only, so no +encoder weights are ever in memory, and the refusal names that rather than +claiming the encoder itself is missing. Two encoder-level limits are worth +stating in advance because they are refusals rather than approximations. A +reference waveform whose sample rate differs from the audio VAE's is refused +rather than resampled, since upstream uses a polyphase kaiser resampler this +project does not carry. And a VAE configured with `latent_log_var: none` is +refused, because upstream itself raises on it. + +**A typed prompt works.** `--encoder` names the Gemma-4 12B text tower and +`--prompt` carries the words. The tower tokenizes them with its OWN embedded +tokenizer — the shipped encoder stores `tokenizer.json` as a TENSOR, so there is +no sibling file to point at — runs, aggregates all 49 hidden states, projects +them to 4096 and 2048, and passes both streams through the embeddings connector +before cross-attention. The tower is ~24 GB of host bf16 and stays resident, +because a prompt arrives per request. + +One tokenization detail is a KNOWN DIVERGENCE rather than a mirror, and it is +checkpoint-conditional: upstream tokenizes through the HuggingFace `__call__` +with its default `add_special_tokens=True`, so it runs the tokenizer's +post_processor, while this port calls the plain encode and prepends BOS by hand. +On the shipped checkpoint the two are identical — its post_processor declares an +EMPTY special-token map, measured on the shipped file rather than assumed — so +nothing is lost today. A checkpoint whose post_processor DID add tokens would +tokenize differently here. + +`--encoder-config` supplies the Gemma config, and it is required for the only +shipped encoder: `vonkaiser`'s +`gemma4-12b-with-proj-nvfp4-torchao.safetensors` carries no `__metadata__` at +all. An encoder that declares one (the official bf16 build does, under +`__metadata__["gemma_config"]`) needs no flag, and supplying both is refused +rather than resolved — `layer_types`, `global_head_dim`, +`num_global_key_value_heads` and `attention_k_eq_v` each resolve a different +tower out of a byte-identical tensor set. + +Without `--encoder`, conditioning comes from `--prompt-embeds` plus +`--audio-prompt-embeds`: rows of little-endian f32, 4096 wide for the video +stream and 2048 for the audio stream, with the same row count in both. A +`--prompt` with no tower is refused, and supplying only one of the two files is +refused, because a stream left unconditioned renders instead of failing. + +**Asking what a clip was conditioned on.** `Ltx2VideoEngine::last_conditioning()` +returns the trace of the last `Generate()` — whether the conditioning came from a +prompt or from embeds, the prompt string, the row count and both stream widths, an +FNV-1a digest over the exact f32 buffers cross-attention read, and each stream's +absmax. It is returned **by value, under the engine's own lock**, so it is safe to +call from a server thread while another thread renders — but `Generate` holds that +same lock for the WHOLE render, so such a call blocks for minutes rather than +returning a stale answer immediately. `completed` is true only if that +`Generate()` returned: the trace is filled before the denoise loop, so a +render that throws later leaves a populated trace behind, and this flag is what +separates the two. + +It is a **change detector, not a quality measure**. It answers "did this render +depend on this prompt, through these weights" and nothing else — it does not say +the conditioning values are the ones upstream would produce. + +The text path runs on the CPU even when `--device cuda` puts the DiT on the GPU: +everything in the text encoder is f32 by declaration and its device arm is owed. +That is one host-side 12B forward over the prompt's own tokens per request, +against a denoise loop of many 21B forwards. + +**Either source goes through the embeddings connector.** Both shipped LTX-2.5 DiTs +carry two `*_embeddings_connector` families, 129 tensors each, and they are the +8-layer 1-D transformer upstream runs between the caption projections and the +DiT's cross-attention. The render applies it with the checkpoint's own weights, +under the checkpoint's own `connector_*` configuration. Two consequences for the +command line: the row count must be a multiple of the connector's learnable +register count (128 on the shipped files), and `--prompt-valid-rows N` says how +many of those rows are real tokens. The rest are padding, and padding is not +inert here: the connector REPLACES it with its learnable register table, so a +run that leaves the default renders as if every supplied row were caption. +`--prompt-valid-rows` applies to the embeds path only — with `--encoder` the +tokenizer supplies the mask, which is what that flag exists to stand in for. + +**The DiT config is required when the checkpoint does not carry one.** The +shipped `vonkaiser` FP8 transformer has no `__metadata__` at all, and the values +a config decides are ones no tensor shape encodes: `frequencies_precision` and +`av_ca_timestep_scale_multiplier` move every RoPE angle and every audio/video +modulation. Defaulting them resolves a different model from the same file, so +the loader refuses and `--dit-config` supplies LTX-2.5's declared values. + +```sh +ltx2-gen --dit ltx-2.5-22b-distilled-fp8.safetensors \ + --dit-config ltx-2.5-transformer-config.json \ + --model-version 2.5 --allow-unported \ + --video-vae ltx-2.5-video-vae-conv-bf16.safetensors \ + --audio-vae ltx-2.5-audio-vae-bf16.safetensors \ + --upsampler ltx-2.5-latent-spatial-upscaler-x2-bf16-1.0.safetensors \ + --encoder gemma4-12b-with-proj-nvfp4-torchao.safetensors \ + --encoder-config ltx-2.5-gemma4-text-config.json \ + --prompt "a red fox running through deep snow at sunrise" \ + --frames 25 --width 320 --height 192 --seed 20260812 \ + --device cuda --workdir /tmp/ltx25 --out /tmp/ltx25/video.mp4 +``` + +Swap the two `--encoder*` flags and `--prompt` for `--prompt-embeds` + +`--audio-prompt-embeds` to condition from files instead. + +`--frames` must satisfy `(frames - 1) % 8 == 0` and width/height must divide by +64 (32 for the VAE, twice that because the distilled recipe's first phase runs at +half resolution). Omitting all three renders the recipe default, which is +1024x1536 at 121 frames and is a much larger request than it looks. + +`--upsampler` is what the distilled recipe's second phase needs. Without it that +phase refuses rather than skipping: its three-step refinement is what makes the +upscaled latent valid, and decoding the half-resolution latent instead would hand +back a smaller clip that looks like a completed request. `--max-phase 0` stops +after the first phase deliberately. + +On the server, `--video-family ltx-2.5` pins the family instead of detecting it, +and `--video-extra KEY=VALUE` (repeatable) carries the same family-specific load +knobs the flags above map onto. Both are described under +[the server's video flags](#video-family-and-family-specific-load-knobs). + +**Three things about that command are worth knowing before you run it.** + +*It is bounded by the VIDEO DECODE, well below the recipe's own defaults.* +Staging the 21.00B FP8 transformer costs about 44 GB on a 119 GB GB10, and +`--encoder` adds the text tower on top of that — roughly 24 GB of host bf16 that +stays resident, because a prompt arrives per request. Every memory figure here +was measured WITHOUT the tower, on the prompt-embeds path, so budget for both. +**320x192 at 25 frames completes** through both distilled phases; 448x256 at 25 +frames finishes its denoise and then loses about 59 GB in 24 seconds inside the decode +and has to be stopped. The denoise itself is flat at either size. Unified memory +makes those host bytes and this class of box reboots rather than OOM-killing, so +start small and grow, and put a memory watchdog in front of anything larger. The +recipe default (1024x1536 at 121 frames) is far beyond what one GB10 holds today. +Expect minutes, not seconds: most of a 320x192/25f render is spent single-threaded +in the host VAE decode at 0% GPU. + +*The render behind those numbers was NOT prompted, and it renders a scene without +rendering YOUR scene.* It was the EMBEDS path — `--prompt-embeds` with +`--prompt-valid-rows 24`, over synthetic N(0, 0.2) rows, with no text tower on the +path at all. With the connector wired the shipped 21.00B FP8 transformer produced +a temporally coherent photorealistic clip at 320x192 / 25 frames: consistent +subject, consistent background, frame-to-frame motion, where before the connector +the same weights at the same settings produced smooth colour fields. But 104 of +its 128 connector rows were the connector's own trained `learnable_registers` +table, which is what upstream substitutes at PADDED positions, and the other 24 +were noise. So what conditioned that clip is the checkpoint's own learned default, +not a depiction of anything anyone asked for — and on the embeds path it could not +be otherwise, because rows read from a file are whatever you put in them rather +than an encoded caption. Ask a `--prompt-embeds` run for a subject and you will +not get it. + +*Nobody has yet run the command above end to end, and this page claims nothing +about what it renders.* The typed-prompt path is gated all the way through — +tokenizer, Gemma-4 tower, connector, cross-attention — but the gate is a +REDUCED-DIMENSION synthetic encoder under CPU Release, with no real checkpoint +anywhere in it. A real-checkpoint prompted render is OWED. Until it runs, neither +claim is available: not that `--prompt "a red fox…"` puts a fox on the screen, and +not that it fails to. `last_conditioning()` answers a narrower question — that the +render depended on your prompt, through these weights — which is not the same +question as whether the frames depict it. + +LTX-2.5 ships two video decoders behind one checkpoint field. The convolutional +one is implemented; the higher quality diffusion one (`NADiffusionDecoder`) is +not, and asking for it fails with a message naming the missing +neighborhood-attention kernel. It never falls back to the convolutional decoder, +because that would hand back a lower quality render as if it were the one you +asked for. Keyframe and reference conditioning is refused for the same reason: it +runs through the video VAE's encoder, and only the decoder is ported. + +**The refusal that used to stand here is gone, and what replaced it is an owed +ORACLE rather than an owed feature.** Through L10 this page said a prompt was +refused because the `Embeddings1DConnector` weights, which ship inside the DiT +file, were among the modules the DiT loader would not load. They are loaded +(`ltx2_loader.cpp:416-427` carries them as their own contract, outside the DiT's), +so `encoder_path` is accepted, `has_encoder()` is true, and a prompt no longer +needs a matching pair of embeds files. The gap that remains is a numeric one: the +tower, the connector's forward and both caption projections each have an oracle +against executed upstream, and the two JOINS between them — +`create_embeddings`, and the render composition that chains it onto the tower's +output — have none. Upstream's `EmbeddingsProcessor.process_hidden_states` is +that whole chain in one function and is the oracle this owes; until it is +executed, the composition's VALUES rest on the per-brick oracles either side of +it. That is also why `last_conditioning()` is described above as a change +detector and not as a check on the conditioning. ### GDN checkpoints: the `output_gate_type` key A Gated DeltaNet checkpoint (the Qwen3.5 / Qwen3-Next family) chooses its @@ -1297,11 +1521,47 @@ The library never spawns a process, so generation and muxing enter through a caller-supplied `VideoRunner` callback (`examples/server/main.cpp` supplies one that invokes `ffmpeg`, path configurable with `--video-ffmpeg`). +### Video family, and family-specific load knobs + +`/v1/videos` serves whichever video family the `--video-dit` checkpoint belongs +to. By default the family is **detected** from what the checkpoint holds, and +that is unchanged. + +`--video-family NAME` pins it instead. Two registered families exist, +`minimax-h3` and `ltx-2.5`, and a name outside that set is refused at argument +parsing, before the text model loads, with the registered names printed. It is +never a hint: a declared family that cannot load the checkpoint fails loudly +rather than falling back to detection, because a checkpoint handed to the wrong +family does not fail, it renders noise. + +`--video-extra KEY=VALUE`, repeatable, carries a family's own load knobs. LTX-2.5 +cannot load without `dit_config_path`, and it needs `encoder_config_path` beside +`--video-encoder` when the text encoder declares no `gemma_config` (the shipped +one does not); MiniMax-H3 +defines `partition`, for which `--video-partition` remains the documented alias. +A bare `KEY` with no `=` is refused rather than read as an empty value, and a +`--video-extra partition=X` contradicting `--video-partition Y` is refused rather +than resolved by whichever assignment ran last. A family refuses any key it does +not define, so a mistyped knob is an error instead of a silently defaulted +render. + +```sh +vllm-server --model /path/to/text-model \ + --video-family ltx-2.5 \ + --video-dit ltx-2.5-22b-distilled-fp8.safetensors \ + --video-vae ltx-2.5-video-vae-conv-bf16.safetensors \ + --audio-vae ltx-2.5-audio-vae-bf16.safetensors \ + --video-encoder gemma4-12b-with-proj-nvfp4-torchao.safetensors \ + --video-extra encoder_config_path=ltx-2.5-gemma4-text-config.json \ + --video-extra dit_config_path=ltx-2.5-transformer-config.json \ + --video-extra model_version=2.5 --video-extra allow_unported_modules=1 +``` + ## Consuming it as a library (C ABI) Link `libvllm` (static or shared) and include [`include/vllm.h`](../include/vllm.h). -It exposes a flat, exception-free, llama.cpp-style C ABI (`VLLM_ABI_VERSION 17`, -35 exported functions) suitable for `dlopen` / FFI / LocalAI integration. +It exposes a flat, exception-free, llama.cpp-style C ABI (`VLLM_ABI_VERSION 18`, +36 exported functions) suitable for `dlopen` / FFI / LocalAI integration. ```c #include "vllm.h" @@ -1347,6 +1607,7 @@ concurrent requests, memory helpers, and diagnostics. Later ABI versions add: | v15 | Embeddings through `vllm_embed` | | v16 | Absolute KV-cache memory sizing | | v17 | The OpenAI server as a thin ABI client through `vllm_server_main` | +| v18 | Video model-family selection (`family`, `vllm_video_engine_family`) and family-specific `extra_keys`/`extra_values` on `vllm_video_*` | Chat templates render through the vendored google/minja engine, the same renderer llama.cpp ships. @@ -1371,6 +1632,64 @@ auto engine = vllm::entrypoints::LoadedEngine::FromModelDir(model_dir, ep); The underlying portable tensor runtime is `vt::` ([`include/vt/`](../include/vt/)), which carries no ggml or PyTorch dependency. +Video and audio generation is reached through `vllm::multimodal::VideoEngine` +([`multimodal/video_engine.h`](../include/vllm/multimodal/video_engine.h)). +`LoadVideoEngine` resolves the model family from what the checkpoint HOLDS, never +from a filename, and refuses rather than guessing: zero claimants, several +claimants, and an unregistered declared `family` are all errors that name what was +seen and what is registered. A caller who supplies no `dit_path` is told which +artifact is missing rather than being advised to declare a family, which would not +help. A family adds itself with `RegisterVideoFamily`, which refuses a name that +is already registered, because two families under one name would collapse into a +single claimant and leave the choice of loader to link order. + +Two families are registered. `minimax-h3` is detected by `video_patch_proj` plus +`audio_patch_proj`; `ltx-2.5` by `patchify_proj` plus `audio_patchify_proj`, with +or without the ComfyUI `model.diffusion_model.` prefix. Each family reads its own +knobs from `extras`. H3 takes `partition`. LTX-2.5 takes +`audio_prompt_embeds_path` (the audio stream's conditioning, the twin of the +seam's `prompt_embeds_path`, which carries the video stream), `pipeline_kind` +(default `distilled_two_stage`), `model_version` (only for a checkpoint that +declares none), `dit_config_path`, `allow_unported_modules`, `max_phase`, +`prompt_embeds_valid_rows`, `upsampler_path` and `duration_head_path`. An extra a +family does not define is refused, never ignored. One caveat inside that set: +`duration_head_path` is accepted but INERT — the duration head is ported and gated +as a brick, nothing in the video engine constructs one, and no code reads that +key, so supplying it neither loads a head nor enables an AUTO duration. Give +`num_frames` (or `duration`, which is exact arithmetic against the recipe's frame +rate) instead. + +`prompt_embeds_valid_rows` is how many of the supplied conditioning rows are real +tokens; absent, every row is. It matters because the embeddings connector +substitutes its learnable register table at PADDED positions, so padding decides +which of the connector's inputs are learned constants rather than caption +features. Upstream always knows this because its tokenizer produced the mask; +this seam reads conditioning from a file, which carries none. + +`dit_config_path` names a JSON file holding the DiT's `{"transformer": {...}}` +configuration, and it exists because only one of the two shipped LTX-2.5 DiTs +carries one. The first-party NVFP4 file embeds it in `__metadata__["config"]`; +the ungated `vonkaiser/LTX-2.5-FP8-NVFP4` FP8 DiT has no `__metadata__` at all. +Tensor shapes resolve the geometry but not the values no shape encodes, so +without a config `double_precision_rope` would default to false and +`av_ca_timestep_scale_multiplier` to 1, where LTX-2.5 declares `float64` and +`1000`. Both move every RoPE angle and every audio-to-video modulation, so a DiT +that declares no config is refused until one is supplied rather than rendered +under defaults that contradict the model family. A supplied config is adopted +only when it reproduces the identical weight contract the shapes describe, and +supplying one for a checkpoint that already declares its own is refused rather +than ordered. + +The LTX-2.5 arm runs on the CPU in f32 and on CUDA in bf16. `device = 0` takes +the f32 parity forward; `device = 1` stages the DiT to the GPU one tensor at a +time and runs the device-resident forward, so a CUDA handle means a CUDA forward. +On a build with no CUDA backend, `device = 1` is refused by name rather than +served the CPU forward behind a CUDA handle. `encoder_path` loads the Gemma-4 +text tower, and the request's own `prompt` then conditions the render; the tower +itself runs on the CPU in f32 whichever device the DiT is on. Without one, +conditioning comes from the two prompt-embeds files, which must agree on their +row count. + `Sampler`'s `logprobs_mode` selects which tensor the returned logprobs are read from, and all four of vLLM's values now work: `raw_logprobs` (the default) and `raw_logits` are snapshotted before any logits processor runs, so they describe @@ -1591,6 +1910,199 @@ Served over HTTP too: pass `--video-dit` (plus the VAEs and configs) to `example `POST /v1/videos`, `POST /v1/videos/sync` and `GET /v1/videos/{id}` register. Without it the routes stay unregistered. +## LTX-2.5: reproducing the DiT parity gate + +**This section is the DiT's own parity gate, not the way to run LTX-2.5.** The +render path ships and is documented above under +[LTX-2.5: what runs, and what it cannot do](#ltx-25-what-runs-and-what-it-cannot-do): +`ltx-2.5` is one of the two registered video families +(`REGISTER_VLLM_VIDEO_FAMILY` at `src/vllm/multimodal/ltx2_video.cpp:1529`), the +Gemma-4 text tower loads from `--encoder` and sets `has_encoder` +(`ltx2_video.cpp:893`), both VAEs and the pipeline layer are implemented +(`ltx2_video_vae.cpp`, `ltx2_audio_vae.cpp`, `ltx2_pipeline.cpp`), and the +`/v1/videos` routes register for whatever family `--video-dit` resolves — +`server_main.cpp` calls the family-agnostic `LoadVideoEngine` and then prints the +resolved family. What follows here is how to regenerate the DiT's goldens. The +C++ surface is `include/vllm/model_executor/models/ltx2.h`, and it refuses by +name every arm it does not carry (a non-f32 stream dtype, the 19B +caption-projection checkpoint form, keyframe absolute-position embeddings, the +video-only / audio-only model types). + +Provenance, so this can be re-checked rather than trusted: the paragraph above +replaces one that arrived at `3d89f6fc4` — the first LTX commit, where it was +true — and was never revisited as L3 through L13 built each of the six pieces it +denied. + +The prompt-K/V cache (`Ltx2PromptKvCache`) is reusable across the DENOISE STEPS of +one prompt, and only those. It records a fingerprint of the prompt it was filled +for, and a forward whose context tensors, context geometry or prompt masks differ +from that prompt is refused by name rather than served K/V that would render the +cached prompt. Call `Ltx2PromptKvCache::Reset()` to rebind the same allocation to +a new request. + +The gate runs the UPSTREAM modules at reduced dimensions on CPU, so it needs a +Lightricks LTX-2 checkout and the system `python3` with torch — **no checkpoint, no +venv and no gated download**. Regenerate the goldens and run it: + +```sh +git clone https://github.com/Lightricks/LTX-2 ~/_git/LTX-2 +python3 scripts/gen-ltx2-goldens.py \ + --ltx2 ~/_git/LTX-2 \ + --out tests/vllm/models/ltx2_goldens.inc +cmake --build build --target test_ltx2 && ./build/tests/test_ltx2 +``` + +The generator asserts the `ltx_core` it imported came from that checkout and not +from anything installed in site-packages, and it writes the upstream revision it +executed into the generated header. Neither side checks in a weight byte: both +rebuild every tensor from one deterministic stream keyed by the parameter's name. + +The pipeline layer has its own gate, and it needs a second checkout: the recipe +table is read from vLLM-Omni, which is the binding oracle for LTX even though it +carries no 2.5 row of its own. Both checkouts must be CLEAN, because a revision +anchor read from a tree with uncommitted edits stamps a SHA the goldens do not +come from. + +```sh +git clone https://github.com/vllm-project/vllm-omni ~/_git/vllm-omni +python3 scripts/gen-ltx2-pipeline-goldens.py \ + --ltx2 ~/_git/LTX-2 \ + --vllm-omni ~/_git/vllm-omni \ + --out tests/vllm/models/ltx2_pipeline_goldens.inc +cmake --build build --target test_ltx2_pipeline && ./build/tests/test_ltx2_pipeline +``` + +If you regenerate that `.inc` against a moved upstream, expect the goldens to +carry the change rather than only the pin cases. The pipeline goldens reach the +GroupNorm eps and group count in the latent upsampler, the connector's +`rms_norm` eps, the `BlurDownsample` width (on the 1.5 arm only, since the blur +runs on the rational denominator) and the Res2s `sigma_up` clamp — that last one +on the eta = 1 arm, where the clamp binds on every step. A regeneration that +moves one of those constants alone reds a value comparison; one that moves the +constant AND the tensors together passes it, and is caught only by the cases that +compare each constant against upstream's own signature. Both layers are there +deliberately, and neither is redundant. +### The Gemma-4 text tower gate, and the interpreter it needs + +The text tower is gated against the UPSTREAM HuggingFace implementation built and +run at reduced dimensions. It needs a `transformers` that registers +`gemma4_unified` in `CONFIG_MAPPING` — **5.8 or newer; 5.3.0 does not have it and +fails in a way that reads exactly like "Gemma-4 is unsupported"**. The generator +refuses such an interpreter by name rather than emitting goldens from a tower it +could not build. + +```sh +/path/to/venv/bin/python scripts/gen-ltx2-gemma-tower-goldens.py \ + --out tests/vllm/models/ltx2_gemma_tower_goldens.inc +cmake --build build --target test_ltx2_text_encoder && ./build/tests/test_ltx2_text_encoder +``` + +No checkpoint and no download: the reduced config comes from +`tests/vllm/models/ltx2_gemma4_text_config.json`, which is the +`__metadata__["gemma_config"]` of the official bf16 text encoder, and every weight +is rebuilt on both sides from the deterministic stream. The tolerance is not a +constant — the generator MEASURES how far upstream's own answer moves between f32 +and bf16 and emits that per state as the bound. + +Two more gates want the real checkpoint. The prompt-token goldens are regenerated +from the tokenizer the text encoder ships **as a tensor**, and the end-to-end case +dequantizes the 12B tower to roughly 24 GB of host bf16, so it is opt-in rather +than checkpoint-presence gated: + +```sh +TE=$CHECKPOINT_ROOT/ltx-2.5/vonkaiser-fp8-nvfp4/text_encoders/gemma4-12b-with-proj-nvfp4-torchao.safetensors +/path/to/venv/bin/python scripts/gen-ltx2-prompt-tokens-goldens.py \ + --text-encoder "$TE" \ + --out tests/vllm/models/ltx2_prompt_tokens_goldens.inc + +# real vocab, token-exact vs HuggingFace +CHECKPOINT_ROOT=... ./build/tests/test_ltx2_text_encoder --test-case="ltx2 prompt: REAL*" + +# the full 12B vertical: ~33 GB host, minutes of CPU +CHECKPOINT_ROOT=... VLLM_CPP_LTX2_TOWER_E2E=1 \ + ./build/tests/test_ltx2_text_encoder --test-case="ltx2 e2e*" +``` + +`VLLM_CPP_LTX2_TEXT_ENCODER` names the file directly when it does not sit under +`CHECKPOINT_ROOT` at the path above. + +Recipes resolve on an EXACT `(pipeline_kind, model_version)` pair and refuse +anything else by name rather than defaulting, because a plausible but wrong sigma +schedule or guidance scale renders a video instead of failing. The pairs that +resolve are `one_stage` at 2, 2.3, 2.4 and 2.5, `distilled_two_stage` at 2 and +2.5, and `dmd2` at 2 and 2.3. + +`Ltx2Guidance` serves `CFGGuider`, `STGGuider` and `MultiModalGuider`. It refuses +`CFGStarRescalingGuider`, `LtxAPGGuider` and `LegacyStatefulAPGGuider` by name, +because nothing upstream constructs them: all three appear in the Lightricks tree +only at their own `class` statements. Two known gaps in the schedule are open: +`Ltx2SigmaSchedule(1, ...)` returns a NaN first sigma where upstream returns +0.10000002, and the suite's `MaxAbsDiff` drops NaN so a golden alone will not +catch it. + +## LTX-2.5 quantized loaders + +`include/vllm/model_executor/models/ltx2_loader.h` materializes the shipped +LTX-2.5 checkpoints: the FP8 DiT, both NVFP4 DiTs, and the torchao-NVFP4 Gemma-4 +text encoder with its embedded tokenizer. These are the entry points the render +path itself drives: `--dit` (`--video-dit` on the server) reaches +`Ltx2StreamDitToDevice` / `Ltx2LoadDitFromSafetensors` at +`ltx2_video.cpp:576-577`, and `--encoder` (`--video-encoder`) reaches +`Ltx2LoadTextEncoderFromSafetensors` at `ltx2_video.cpp:851`. This section +documents them at the library level, where the gate below runs. + +The two NVFP4 checkpoints were written by different producers that disagree about +both the group-scale framing and which nibble holds which weight, so the loader +resolves the producer from the `torchao_nvfp4` marker: present means torchao +(`to_blocked` framing, low-nibble-first), absent means the Lightricks +`nvfp4-prequant` tool (cuBLAS-padded framing, high-nibble-first). A marker whose +stored scale shape contradicts it, and a marker-less file whose shape is the +`to_blocked` framing or neither framing, are refused by name rather than guessed, +because both readings type-check and produce finite, correctly scaled, wrong +weights. + +The refusal cannot cover everything, and the limit is worth knowing before you +point this loader at a checkpoint it was not built for. A marker-less NVFP4 file +whose `weight_scale` is stored **linear** `[N, K/16]` — what ModelOpt, +llm-compressor and compressed-tensors write, none of which emit a +`torchao_nvfp4` sidecar — has, whenever `N % 128 == 0` and `K/16 % 4 == 0`, a +shape indistinguishable from the cuBLAS-padded one. Such a file is resolved as +`nvfp4-prequant` and read swizzled and high-first: it loads, and it is wrong. +Only the LTX-2.5 DiT is gated against an independent oracle here, so treat any +other marker-less NVFP4 checkpoint as unsupported until it is. See +`.agents/specs/nvfp4-nibble-order.md`. + +Two behaviours a caller has to know. `Ltx2LoadDitFromSafetensors` REFUSES the +shipped DiT by default, because that file carries **three** module families phase +L2 does not port (`prompt_adaln_single`, `audio_prompt_adaln_single` and +`keyframes_abs_pos_embedding`); pass `Ltx2DitLoadOptions::allow_unported_modules` +to load the ported subset, which still reports every one of them in +`Ltx2DitCheckpoint::unported`. The two `*_embeddings_connector` towers are +**not** among them and never will be: +`UnportedFamilies` filters them out at `ltx2_loader.cpp:439` (`LoadedElsewhere`), +`RefuseUnported`'s own message says so in capitals at `ltx2_loader.cpp:461-464`, +and `Ltx2LoadConnectorWeights` loads them under their own contract — which is +what the video engine calls, so a checkpoint this port reads completely is never +made to ask for `allow_unported_modules` on their account. (The "five" this +paragraph used to say arrived at `5966ffef3` and was true until `e48c86253` +added `LoadedElsewhere` — the same claim the "what runs" section above already +retired, which survived here because it was never swept for.) And loading is +**bf16** by default, the checkpoint's own model dtype; `widen_to_f32` is opt-in +and exists only for the f32 parity forward. + +`Ltx2StreamDitToDevice` is the GB10 arm. It dequantizes and uploads one tensor at +a time so peak residency is the device copy plus one tensor, and it stages at +load because host-resident weights measure 20 to 30 percent slower there. + +The gate needs the three checkpoint headers, a vLLM checkout and an LTX-2 +checkout (the two nibble-order authorities); it reads a few hundred bytes at +their own offsets and never a payload: + +```sh +python3 scripts/gen-ltx2-quant-goldens.py --vllm ~/_git/vllm --ltx2 ~/_git/LTX-2 --checkpoint-root /mnt/nas_share/checkpoints --out tests/vllm/models/ltx2_quant_goldens.inc +cmake --build build --target test_ltx2_loader && ./build/tests/test_ltx2_loader +``` + ## SSE keepalives on long prefill Async chat/completion streams may emit SSE **comment** frames (`:\n\n`) while @@ -1605,3 +2117,59 @@ Dual-GPU resident FP8 MoE and SharedK-WMMA prefill are controlled via ENVIRONMENT.md (`VT_GEMMA4_RESIDENT_*`, `VT_ATTN_*`). Defaults stay safe off RDNA4. This PR does **not** restructure the Gemma-4 layer loop or enable decode hipGraph (those stay lab-only until a CUDA token-exact gate can land them). + +## LTX-2.5 text conditioning + +This documents **one brick of the shipped render path** — the text conditioning +the DiT consumes — and how to reproduce its gate. The render itself is above +under [LTX-2.5: what runs, and what it cannot do](#ltx-25-what-runs-and-what-it-cannot-do); +`--encoder` is what puts this brick on that path, and `has_encoder` is set at +`ltx2_video.cpp:893` once the tower loads. + +LTX-2.5 does not condition on a text encoder's last hidden state. It takes every +Gemma-4 hidden state (the embedding output plus all 48 decoder outputs, 49 in +total), normalizes them, concatenates across the layer axis, and projects the +result twice: a 4096-wide video caption projection and a 2048-wide audio one. +That is why the shipped projections take 3840 x 49 = 188160 inputs. + +Two things about the shipped checkpoint are easy to trip over: + +* the tokenizer is stored **as a tensor**, `tokenizer_json`, alongside + `hf_asset__*` sidecars, so a loader that expects a sibling `tokenizer.json` + file cannot read it; +* `vonkaiser/LTX-2.5-FP8-NVFP4`'s text encoder carries **no** safetensors + `__metadata__` block, so the Gemma config has to be supplied out of band. + `Ltx2LoadGemmaAssets(file, /*require_config=*/false)` is the opt-out; the + default refuses, exactly as upstream does. + +Reproduce the parity gate (CPU only, no checkpoint and no gated download; needs +torch, numpy and einops plus a Lightricks LTX-2 checkout): + +```sh +python3 scripts/gen-ltx2-text-goldens.py \ + --ltx2 ~/_git/LTX-2 \ + --out tests/vllm/models/ltx2_text_goldens.inc +cmake --build build --target test_ltx2_text_encoder +./build/tests/test_ltx2_text_encoder +``` + +The generator imports the upstream modules by path and executes them at reduced +dimensions; both sides rebuild every weight from one deterministic stream, so no +weight byte is checked in. It also runs four degenerate inputs through upstream +and emits each one's full output tensor, not a "still finite" flag, because the +normalization epsilons and the width they are added in are invisible to a random +fixture. The mean's denominator is one of those: upstream adds it in float32 +(`sequence_lengths * d` is an int64 tensor and `eps` a python float, which +promotes to the default dtype), so computing it in float64 is finer arithmetic +and the wrong answer. + +A third thing to know if you are wiring a loader to it: the feature extractor +refuses, by name, any disagreement between what the checkpoint config declares +and what the weights actually carry. That covers the declared bias against +`bias.empty()`, the declared `out_features` against the weight's own width, and +`embedding_dim x (num_hidden_layers + 1)` against the weight's `in_features`. The +case worth naming is a loader that binds `video_aggregate_embed.weight` (U8, +NVFP4) and misses `.bias` (BF16, so a different unpack path) while the config +still says the projection is biased. Without the refusal that renders a plausible +video for the wrong prompt: every conditioning row is shifted by the missing bias +and every padded row projects to 0 instead of to the bias. diff --git a/examples/CMakeLists.txt b/examples/CMakeLists.txt index 01b513e31..f351412b9 100644 --- a/examples/CMakeLists.txt +++ b/examples/CMakeLists.txt @@ -189,6 +189,47 @@ if(VLLM_CPP_SERVER) add_test(NAME test_server_help COMMAND $ --help) set_tests_properties(test_server_help PROPERTIES PASS_REGULAR_EXPRESSION "--max-num-seqs N.*--max-num-batched-tokens N.*--cuda-profile-graph-replays N.*--cuda-profile-graph-batch N.*--benchmark-shutdown-fifo F.*--enable-force-include-usage.*--\\[no-\\]enable-prefix-caching") + + # --video-family (row MODEL-DIFFUSION-LTX25 / #435). Black-box, because the + # flag table lives in an anonymous namespace inside server_main.cpp and there + # is no unit seam onto it: the binary IS the surface under test. Both cases + # exit during ParseArgs, so neither loads a model, a checkpoint or a byte of + # weights, and both run in milliseconds. + # + # 1. AN UNREGISTERED NAME IS REFUSED, and the refusal NAMES what is + # registered. Before the flag existed this printed "unknown argument + # '--video-family'" instead, which is the RED. + add_test(NAME test_server_video_family_unregistered + COMMAND $ --model /nonexistent-model-dir + --video-family not-a-registered-family) + set_tests_properties(test_server_video_family_unregistered PROPERTIES + PASS_REGULAR_EXPRESSION "--video-family 'not-a-registered-family' is not a registered video family. Registered families:.*ltx-2\\.5.*minimax-h3") + + # 2. A REGISTERED name is ACCEPTED — and specifically, the check consults the + # REGISTRY rather than a list written out here, so the family LTX-2.5 + # registers from its own file passes. Proven by getting PAST the family + # check to the next validation in ParseArgs (the tool-parser dialect), + # with the family refusal absent from the output. A hardcoded list that + # omitted ltx-2.5 REDs on the FAIL expression, and a check accidentally + # dropped altogether REDs on it too (the unknown-argument spelling). + add_test(NAME test_server_video_family_registered + COMMAND $ --model /nonexistent-model-dir + --video-family ltx-2.5 + --tool-call-parser definitely-not-a-registered-parser) + set_tests_properties(test_server_video_family_registered PROPERTIES + PASS_REGULAR_EXPRESSION "unknown tool-call parser \"definitely-not-a-registered-parser\"" + FAIL_REGULAR_EXPRESSION "not a registered video family;unknown argument") + + # 3. --video-extra takes KEY=VALUE, and a bare KEY is REFUSED rather than + # read as an empty value. An extra that silently became "" is how a + # mistyped knob renders the default and looks like the feature not + # working — the same failure the families' own unknown-extra refusals + # exist to stop, and it would be defeated one layer earlier here. + add_test(NAME test_server_video_extra_needs_a_value + COMMAND $ --model /nonexistent-model-dir + --video-extra dit_config_path) + set_tests_properties(test_server_video_extra_needs_a_value PROPERTIES + PASS_REGULAR_EXPRESSION "--video-extra takes KEY=VALUE, got 'dit_config_path'") endif() endif() @@ -211,6 +252,16 @@ add_executable(minimax-h3-gen minimax_h3_gen/main.cpp) target_link_libraries(minimax-h3-gen PRIVATE vllm::shared) vllm_cpp_set_warnings(minimax-h3-gen) +# ltx2-gen: an LTX-2.5 render end to end, the SECOND family behind the generic +# video seam and a THIN client of the same C ABI. It exists rather than a flag on +# minimax-h3-gen because LTX-2.5 cannot load without the v18 family-specific +# extras (the audio-stream prompt embeds, the DiT config the shipped FP8 +# checkpoint does not carry, the latent upsampler) and that example drives +# neither `family` nor the extras arrays. Row MODEL-DIFFUSION-LTX25 / #435. +add_executable(ltx2-gen ltx2_gen/main.cpp) +target_link_libraries(ltx2-gen PRIVATE vllm::shared) +vllm_cpp_set_warnings(ltx2-gen) + # vulkan-gemm-ab: the VK-C tactic A/B — cooperative matrix vs the portable scalar # GEMM, SAME BINARY with VT_VULKAN_COOPMAT as the only variable. Built only in a # Vulkan build, because it links the Vulkan context directly to report which diff --git a/examples/ltx2_gen/main.cpp b/examples/ltx2_gen/main.cpp new file mode 100644 index 000000000..e17bb6953 --- /dev/null +++ b/examples/ltx2_gen/main.cpp @@ -0,0 +1,269 @@ +// ltx2-gen — an LTX-2.5 video+audio render, end to end, as a THIN CLIENT of the +// public C ABI (include/vllm.h) and NOTHING else, per the ONE SURFACE directive +// (ARCH-ONE-SURFACE ROW 2). Row MODEL-DIFFUSION-LTX25 phase L9B, issue #435. +// +// WHY A SECOND GENERATION EXAMPLE AND NOT A FLAG ON minimax-h3-gen. The ABI's +// video slice went family-generic at v18: `family` selects the model family and +// two parallel string arrays carry the FAMILY-SPECIFIC load knobs +// (vllm.h:735-747). `minimax-h3-gen` predates that and drives neither, and +// LTX-2.5 CANNOT LOAD without three of them — the audio-stream prompt embeds, +// the DiT config the shipped FP8 checkpoint does not carry, and (for the second +// distilled phase) the latent spatial upsampler. This file exists to name those +// knobs as flags rather than to hand a user a `--extra key=value` grab bag, and +// it is the smallest thing that can drive a real render. +// +// WHAT IT DOES NOT DO. It composes no ffmpeg command line of its own and encodes +// nothing: `vllm_video_mux_argv` builds the argv and this file exec's it, which +// is the ratified process boundary (2026-08-03). It carries no model logic — no +// noise stream, no schedule, no dtype choice — because all of that is the +// library's and a second copy here would be a parallel path. +// +// CONDITIONING HAS TWO SOURCES, and as of phase L13 the first of them is a +// typed prompt. `--encoder` names the Gemma-4 12B text tower and `--prompt` +// carries the words; the tower tokenizes them with its OWN embedded tokenizer, +// runs, projects all 49 hidden states to the two stream widths and hands the +// result to the embeddings connector and then to cross-attention. Before L13 +// the tower had no route to the DiT and `--prompt` did not exist here at all. +// +// `--prompt-embeds` + `--audio-prompt-embeds` remain, and remain the only +// conditioning without a tower. Both streams are conditioned or neither: +// LTX-2.5 cross-attends at TWO widths (4096 video, 2048 audio) and one of them +// alone leaves a stream unconditioned, which renders instead of failing. +// +// `--encoder-config` is not optional paperwork. The only shipped LTX-2.5 text +// encoder carries no `__metadata__` at all, so its Gemma config cannot come out +// of the file; the engine refuses rather than defaulting one, because +// `layer_types`, `global_head_dim` and `attention_k_eq_v` resolve a DIFFERENT +// tower out of a byte-identical tensor set. +#include +#include + +#include +#include +#include +#include +#include + +#include "vllm.h" + +namespace { + +// The ONE process spawn, in examples/ by decision: the library composed `args`; +// this runs it. No shell — the argv is exec'd directly. +int RunFfmpeg(const std::vector& args) { + std::vector argv; + argv.reserve(args.size() + 1); + for (const std::string& a : args) argv.push_back(const_cast(a.c_str())); + argv.push_back(nullptr); + const pid_t pid = fork(); + if (pid < 0) { + std::fprintf(stderr, "error: fork failed\n"); + return -1; + } + if (pid == 0) { + execvp(argv[0], argv.data()); + _exit(127); + } + int status = 0; + if (waitpid(pid, &status, 0) < 0) { + std::fprintf(stderr, "error: waitpid failed\n"); + return -1; + } + if (WIFSIGNALED(status)) { + std::fprintf(stderr, "error: ffmpeg died on signal %d\n", WTERMSIG(status)); + return -1; + } + return WIFEXITED(status) ? WEXITSTATUS(status) : -1; +} + +const char* Need(int argc, char** argv, int i, const char* flag) { + if (i >= argc) { + std::fprintf(stderr, "error: missing value for %s\n", flag); + std::exit(2); + } + return argv[i]; +} + +[[noreturn]] void Usage(int code) { + std::fprintf( + stderr, + "usage: ltx2-gen --dit --video-vae --audio-vae \n" + " (--encoder --prompt \"...\"\n" + " | --prompt-embeds --audio-prompt-embeds )\n" + " --workdir DIR [--out ] [--ffmpeg PATH]\n" + " [--encoder-config ] REQUIRED when the text\n" + " encoder carries no metadata\n" + " [--dit-config ] REQUIRED when the DiT\n" + " carries no __metadata__\n" + " [--model-version 2.5] [--pipeline-kind distilled_two_stage]\n" + " [--upsampler ] phase 2 needs it\n" + " [--max-phase N] [--allow-unported]\n" + " [--prompt-valid-rows N] how many embed rows are real tokens\n" + " [--frames N] [--width N] [--height N] [--seed N]\n" + " [--device cpu|cuda]\n\n" + "Renders LTX-2.5 (family \"ltx-2.5\") through vllm_video_engine_load +\n" + "vllm_video_generate.\n\n" + "CONDITIONING, two ways. With --encoder the Gemma-4 12B text tower is loaded and\n" + "--prompt is tokenized by the tower's OWN embedded tokenizer, run, projected to\n" + "the two stream widths and passed through the embeddings connector. The tower is\n" + "~24 GB of host bf16 and stays resident, because a prompt arrives per request.\n" + "--encoder-config supplies the Gemma config when the encoder declares none, which\n" + "the only shipped one does not; without either the load is refused rather than\n" + "defaulted, since the config decides which layers are full-attention and a wrong\n" + "one resolves a different tower from the same tensors.\n\n" + "Without --encoder, conditioning is PROMPT-EMBEDS: both files are rows of\n" + "little-endian f32, the video one 4096 wide and the audio one 2048, with the\n" + "SAME row count. Passing --prompt without --encoder is refused rather than\n" + "silently rendering those embeddings as if they were it.\n\n" + "Those rows are the EMBEDDINGS CONNECTOR's input, not the DiT's: when the\n" + "checkpoint carries the two *_embeddings_connector families (both shipped\n" + "LTX-2.5 DiTs do) they run through it first, with the checkpoint's own\n" + "weights. The row count must then be a multiple of the connector's learnable\n" + "register count (128 on the shipped files), and --prompt-valid-rows says how\n" + "many of them are real: the rest are padding, and padding is REPLACED by the\n" + "learnable register table rather than ignored.\n"); + std::exit(code); +} + +} // namespace + +int main(int argc, char** argv) { + vllm_video_model_params mp = vllm_video_model_params_default(); + vllm_video_params vp = vllm_video_params_default(); + std::string workdir = "/tmp/ltx2_gen", out_path, ffmpeg = "ffmpeg", device = "cuda"; + // BORROWED by `vllm_video_generate`, like the extras below, so it is owned + // here and pointed at only after parsing. + std::string prompt; + + // The extras are BORROWED by the load call, so the strings must outlive it. + // Kept as two parallel vectors of owned strings plus the char* views the ABI + // takes, built once after parsing. + std::vector extra_keys, extra_values; + auto SetExtra = [&](const char* key, std::string value) { + for (size_t i = 0; i < extra_keys.size(); ++i) { + if (extra_keys[i] == key) { + extra_values[i] = std::move(value); + return; + } + } + extra_keys.emplace_back(key); + extra_values.push_back(std::move(value)); + }; + + for (int i = 1; i < argc; ++i) { + const std::string f = argv[i]; + if (f == "--dit") mp.dit_path = Need(argc, argv, ++i, "--dit"); + else if (f == "--video-vae") mp.video_vae_path = Need(argc, argv, ++i, "--video-vae"); + else if (f == "--video-vae-config") mp.video_vae_config_path = Need(argc, argv, ++i, f.c_str()); + else if (f == "--audio-vae") mp.audio_vae_path = Need(argc, argv, ++i, "--audio-vae"); + else if (f == "--audio-vae-config") mp.audio_vae_config_path = Need(argc, argv, ++i, f.c_str()); + else if (f == "--encoder") mp.encoder_path = Need(argc, argv, ++i, "--encoder"); + else if (f == "--encoder-config") + SetExtra("encoder_config_path", Need(argc, argv, ++i, f.c_str())); + else if (f == "--prompt") prompt = Need(argc, argv, ++i, "--prompt"); + else if (f == "--prompt-embeds") mp.prompt_embeds_path = Need(argc, argv, ++i, f.c_str()); + else if (f == "--audio-prompt-embeds") + SetExtra("audio_prompt_embeds_path", Need(argc, argv, ++i, f.c_str())); + else if (f == "--dit-config") SetExtra("dit_config_path", Need(argc, argv, ++i, f.c_str())); + else if (f == "--model-version") SetExtra("model_version", Need(argc, argv, ++i, f.c_str())); + else if (f == "--pipeline-kind") SetExtra("pipeline_kind", Need(argc, argv, ++i, f.c_str())); + else if (f == "--upsampler") SetExtra("upsampler_path", Need(argc, argv, ++i, f.c_str())); + else if (f == "--duration-head") + SetExtra("duration_head_path", Need(argc, argv, ++i, f.c_str())); + else if (f == "--max-phase") SetExtra("max_phase", Need(argc, argv, ++i, f.c_str())); + else if (f == "--prompt-valid-rows") + SetExtra("prompt_embeds_valid_rows", Need(argc, argv, ++i, f.c_str())); + else if (f == "--allow-unported") SetExtra("allow_unported_modules", "1"); + else if (f == "--device") device = Need(argc, argv, ++i, "--device"); + else if (f == "--frames") vp.num_frames = std::atoi(Need(argc, argv, ++i, "--frames")); + else if (f == "--width") vp.width = std::atoi(Need(argc, argv, ++i, "--width")); + else if (f == "--height") vp.height = std::atoi(Need(argc, argv, ++i, "--height")); + else if (f == "--seed") { + vp.seed = static_cast(std::strtoull(Need(argc, argv, ++i, "--seed"), nullptr, 10)); + vp.has_seed = 1; + } + else if (f == "--workdir") workdir = Need(argc, argv, ++i, "--workdir"); + else if (f == "--out") out_path = Need(argc, argv, ++i, "--out"); + else if (f == "--ffmpeg") ffmpeg = Need(argc, argv, ++i, "--ffmpeg"); + else if (f == "--help" || f == "-h") Usage(0); + else { + std::fprintf(stderr, "error: unknown argument: %s\n", f.c_str()); + Usage(2); + } + } + if (mp.dit_path == nullptr) Usage(2); + if (device == "cuda") mp.device = 1; + else if (device != "cpu") { + std::fprintf(stderr, "error: --device must be cpu or cuda\n"); + return 2; + } + // DECLARED, never detected. Detection would also resolve this checkpoint, but + // an explicit family is what makes an FP8-vs-NVFP4 comparison a statement + // about the two files rather than about what a detector happened to claim. + mp.family = "ltx-2.5"; + vp.output_dir = workdir.c_str(); + if (!prompt.empty()) vp.prompt = prompt.c_str(); + + std::vector keys, values; + keys.reserve(extra_keys.size()); + values.reserve(extra_values.size()); + for (size_t i = 0; i < extra_keys.size(); ++i) { + keys.push_back(extra_keys[i].c_str()); + values.push_back(extra_values[i].c_str()); + } + if (!keys.empty()) { + mp.extra_keys = keys.data(); + mp.extra_values = values.data(); + mp.n_extras = static_cast(keys.size()); + } + + vllm_video_engine* engine = nullptr; + if (vllm_video_engine_load(&mp, &engine) != VLLM_OK) { + std::fprintf(stderr, "error: %s\n", vllm_last_error()); + return 1; + } + // Which family actually loaded, from the handle rather than from the request: + // spec §3.1 requires every artifact to name what produced it. + std::fprintf(stderr, "ltx2-gen: family=%s dit=%s\n", vllm_video_engine_family(engine), + mp.dit_path); + + vllm_video_result out; + if (vllm_video_generate(engine, &vp, &out) != VLLM_OK) { + std::fprintf(stderr, "error: %s\n", vllm_last_error()); + vllm_video_engine_free(engine); + return 1; + } + std::fprintf(stderr, " wrote %d frames (%dx%d @ %d fps) + %s (%d Hz)\n", out.frame_count, + out.width, out.height, out.fps, out.audio_path, out.sample_rate); + + int status = 0; + if (!out_path.empty()) { + const std::string pattern = std::string(out.frame_dir) + "/frame_%06d.ppm"; + vllm_video_mux_params mx = vllm_video_mux_params_default(); + mx.frames = pattern.c_str(); + mx.audio_path = out.audio_path; + mx.output_path = out_path.c_str(); + mx.fps = out.fps; // the RECIPE's frame rate, not the mux default + char** mux_argv = nullptr; + int32_t mux_argc = 0; + if (vllm_video_mux_argv(&mx, &mux_argv, &mux_argc) != VLLM_OK) { + std::fprintf(stderr, "error: %s\n", vllm_last_error()); + vllm_video_result_free(&out); + vllm_video_engine_free(engine); + return 1; + } + std::vector args(mux_argv, mux_argv + mux_argc); + if (!args.empty()) args[0] = ffmpeg; + vllm_video_mux_argv_free(mux_argv, mux_argc); + status = RunFfmpeg(args); + if (status == 0) { + std::printf("wrote %s\n", out_path.c_str()); + } else { + std::fprintf(stderr, "ffmpeg exited %d\n", status); + } + } + + vllm_video_result_free(&out); + vllm_video_engine_free(engine); + return status; +} diff --git a/include/vllm.h b/include/vllm.h index 667477da8..f372c9fd5 100644 --- a/include/vllm.h +++ b/include/vllm.h @@ -142,7 +142,32 @@ extern "C" { * permanent. The flag surface mirrors vLLM's cli_args.py, which is the real * contract. Embedders wanting programmatic control keep the granular entry * points. Purely additive. */ -#define VLLM_ABI_VERSION 17 +/* v18 — THE GENERALIZED VIDEO SEAM (LTX-2.5 L1, .agents/specs/ltx-2-5.md §5, + * issue #435). The v12 video slice was H3-typed inside; it is now the C face of + * vllm::multimodal::VideoEngine, an abstract seam with a checkpoint-detected + * family registry, so a second video family (LTX-2.5) is an additive file + * rather than a second ABI. Three additions, all APPENDED: + * - vllm_video_model_params.family — the family to load. NULL/empty (the + * zero value) means DETECT it from the checkpoint, which is what v12 + * callers already get, since MiniMax-H3 is what a v12 checkpoint set is. + * A name that is not registered is REFUSED naming what is registered; it + * is never treated as a hint, and a checkpoint no family claims is refused + * rather than handed to the only family present — an H3 DiT loaded as LTX + * would not fail, it would render noise. + * - extra_keys / extra_values / n_extras on vllm_video_model_params and on + * vllm_video_params — parallel arrays carrying FAMILY-SPECIFIC settings as + * strings, so a new family adds no permanent field to a struct every other + * family must then ignore. n_extras 0 (the zero value) is "none". + * vllm_video_model_params.partition is now the documented ALIAS for the + * load extra "partition"; supplying both with DIFFERENT values is + * VLLM_ERR_INVALID_ARGUMENT rather than a silent winner. + * - vllm_video_engine_family() — which family a handle actually resolved to, + * so detection is visible to a C caller rather than implicit. + * A v12 caller is byte-identical: it zero-fills the struct growth, so family + * stays NULL (detect), n_extras stays 0, `partition` keeps its exact v12 + * meaning, and every v12 status/message contract is unchanged (the + * text-checkpoint refusal still names vllm_engine_load). */ +#define VLLM_ABI_VERSION 18 /* ── Export macro ───────────────────────────────────────────────────────────── * Marks the symbols that make up the stable ABI. Default visibility now; Task 3 @@ -707,6 +732,19 @@ typedef struct vllm_video_model_params { int32_t device; /* 0 cpu, 1 cuda */ int32_t dequant_bf16; /* 0 keep-quant, 1 dequant/stream bf16 */ int32_t fp4_resident; /* NVFP4+cuda: keep FP4 packed, Marlin W4A16 GEMM */ + /* ── v18 additions (the generalized seam) ───────────────────────────────── + * The model family to load, e.g. "minimax-h3". NULL/empty DETECTS it from + * what the checkpoint holds; an unregistered name is refused naming the + * registered ones. Never a hint — a checkpoint no family claims is refused, + * because the wrong family does not fail, it renders noise. */ + const char* family; + /* FAMILY-SPECIFIC load settings as parallel arrays of n_extras borrowed + * key/value strings (both arrays must hold n_extras non-NULL entries). + * `partition` above is the documented alias for the key "partition"; + * supplying both with DIFFERENT values is VLLM_ERR_INVALID_ARGUMENT. */ + const char* const* extra_keys; + const char* const* extra_values; + int32_t n_extras; /* 0 => none */ } vllm_video_model_params; typedef struct vllm_video_params { @@ -724,6 +762,12 @@ typedef struct vllm_video_params { float noise_aug; /* keyframe pinning strength; <= 0 => 1.0 */ /* Where frame_%06d.ppm + audio.wav land (created if absent). REQUIRED. */ const char* output_dir; + /* v18: FAMILY-SPECIFIC per-generation settings, same parallel-array shape as + * the load-time extras. MiniMax-H3 defines none, and refuses any key it does + * not know rather than ignoring it. 0 => none. */ + const char* const* extra_keys; + const char* const* extra_values; + int32_t n_extras; } vllm_video_params; /* One finished generation. OWNERSHIP: every member is library-allocated; @@ -752,6 +796,11 @@ VLLM_API vllm_status vllm_video_engine_load(const vllm_video_model_params* param vllm_video_engine** out); VLLM_API void vllm_video_engine_free(vllm_video_engine* engine); +/* v18: the family this handle RESOLVED to ("minimax-h3", ...) — the answer to + * "what did detection decide?". Points at storage the library owns for the + * lifetime of the handle; the caller must NOT free it. NULL engine => NULL. */ +VLLM_API const char* vllm_video_engine_family(const vllm_video_engine* engine); + /* Run one BLOCKING generation, filling *out. Serialized per engine handle. * VLLM_ERR_INVALID_ARGUMENT for a missing output_dir / illegal reference * combination; VLLM_ERR_RUNTIME when the pipeline refuses (undeclared or diff --git a/include/vllm/model_executor/model_loader/nvfp4_dequant.h b/include/vllm/model_executor/model_loader/nvfp4_dequant.h index 2c73b3bc2..51127b5ec 100644 --- a/include/vllm/model_executor/model_loader/nvfp4_dequant.h +++ b/include/vllm/model_executor/model_loader/nvfp4_dequant.h @@ -37,6 +37,50 @@ inline constexpr int kNvfp4GroupSize = 16; inline constexpr float kE2M1Lut[8] = {0.0F, 0.5F, 1.0F, 1.5F, 2.0F, 3.0F, 4.0F, 6.0F}; +// WHICH LOGICAL ELEMENT SITS IN WHICH NIBBLE — a PRODUCER convention, and the two +// producers this project reads DISAGREE. See .agents/specs/nvfp4-nibble-order.md. +// +// Read the wrong way round, every adjacent fp4 pair is transposed. The result is +// finite, correctly shaped and correctly scaled, so nothing downstream notices: +// a decoder emits fluent tokens, a DiT renders a plausible-but-wrong frame. +// +// This parameter IS defaulted, to kLowFirst, and that is the deliberate design: +// every caller predating .agents/specs/nvfp4-nibble-order.md consumes a ModelOpt +// or compressed-tensors checkpoint, which is low-first, so the default makes +// "this change moved nothing else" true BY CONSTRUCTION rather than by +// inspection. Four callers rely on it today — minimax_h3_nvfp4.cpp:112, +// minimax_h3_device.cpp:1311, qwen3_5.cpp:1298 and +// dense_nvfp4_gemm.h DequantNvfp4ToBLayout — and H3 reaches low-first by +// normalizing its bytes at load (MiniMaxH3Nvfp4SwapNibbles) rather than by +// passing an order. +// +// The seam where the order is NEVER defaulted is `Ltx2DequantNvfp4ToBf16` +// (ltx2_loader.h), which takes a resolved `Ltx2Nvfp4Producer` with no default, +// because that is the path where the two conventions actually meet and a default +// would let a caller that never thought about it get the silent wrong answer. +enum class Nvfp4NibbleOrder { + // Element 2j in the LOW nibble, 2j+1 in the HIGH. torchao's `pack_uint4` + // (torchao/prototype/mx_formats/kernels.py:160, + // `uint8_data[::2] | uint8_data[1::2] << 4`), which is what NVIDIA ModelOpt and + // compressed-tensors checkpoints carry, and what vLLM's own reader assumes + // (`break_fp4_bytes`, nvfp4_emulation_utils.py:321-324: `low = a_flat & 0x0F` + // then `torch.stack((low, high))`). THE DEFAULT — every caller predating + // .agents/specs/nvfp4-nibble-order.md means this one. + kLowFirst, + // Element 2j in the HIGH nibble. Lightricks' `nvfp4-prequant`, which wrote the + // first-party LTX-2.5 NVFP4 DiT (ltx-kernels/docs/NVFP4.md:27-29: "`hi_first=True` + // (default) puts element `2j` in the **high** nibble of byte `j`"; the same + // statement at ltx-core/quantization/nvfp4/linear.py:6). + // + // MiniMax-H3's community NVFP4 checkpoints are also high-first and are handled a + // DIFFERENT way — `MiniMaxH3Nvfp4SwapNibbles` normalizes the bytes at load + // (minimax_h3.h:1500-1517) because H3 also feeds a Marlin fp4-RESIDENT path, and + // one byte transform fixes both arms where a host-dequant flag fixes only one. + // Both mechanisms are deliberate; nvfp4-nibble-order.md section 3.1 records which + // to use when, and the condition under which LTX-2.5 must switch to H3's. + kHighFirst, +}; + // Decode one IEEE fp8-e4m3fn byte (1 sign, 4 exp, 3 mantissa; bias 7; no inf; // NaN = 0x7F/0xFF; 0x00 = +0) to f32. Matches // torch.Tensor.view(torch.float8_e4m3fn).to(torch.float32). @@ -45,10 +89,14 @@ float F8E4M3ToF32(uint8_t byte); // Dequantize a modelopt W4A16_NVFP4 weight matrix to bf16 (row-major bit // patterns in out_bf16). // -// packed [out_dim, in_dim/2] U8, low-nibble-first packing +// packed [out_dim, in_dim/2] U8, packed per `order` // weight_scale_fp8 [out_dim, in_dim/16] fp8-e4m3fn bytes, linear layout // weight_scale_2 per-tensor f32 global scale (multiplied, not reciprocated) // out_bf16 [out_dim, in_dim] bf16 bit patterns (caller-owned) +// order which nibble holds element 2j; DEFAULTS to the torchao / +// ModelOpt convention, so every caller written before +// .agents/specs/nvfp4-nibble-order.md is unchanged BY +// CONSTRUCTION rather than by inspection // // Requires in_dim % 16 == 0. Aborts (VT_CHECK) otherwise. // @@ -58,7 +106,8 @@ float F8E4M3ToF32(uint8_t byte); // (the safetensors reader validates every span). void DequantNvfp4ToBf16(const uint8_t* packed, const uint8_t* weight_scale_fp8, float weight_scale_2, int64_t out_dim, int64_t in_dim, - uint16_t* out_bf16); + uint16_t* out_bf16, + Nvfp4NibbleOrder order = Nvfp4NibbleOrder::kLowFirst); // Dequantize a modelopt per-tensor FP8 (W8A16) weight to bf16. The 35B gate // checkpoint stores its attention/GDN projections this way (hf_quant_config diff --git a/include/vllm/model_executor/models/dense_device_glue.h b/include/vllm/model_executor/models/dense_device_glue.h index 35afd0cfb..aa8c06af6 100644 --- a/include/vllm/model_executor/models/dense_device_glue.h +++ b/include/vllm/model_executor/models/dense_device_glue.h @@ -17,6 +17,8 @@ // DBuf — move-only pooled device allocation + tensor view. #pragma once +#include +#include #include #include #include @@ -25,6 +27,8 @@ #include "vllm/model_executor/models/qwen3_5_weights.h" // OwnedTensor #include "vllm/platforms/interface.h" #include "vt/backend.h" +#include "vt/device.h" // kNumDeviceTypes +#include "vt/dtype.h" // VT_CHECK #include "vt/ops.h" namespace vllm { @@ -62,19 +66,40 @@ inline Tensor Reshape(const Tensor& src, const std::vector& shape) { // The device-scratch residency policy (BACKEND-PLATFORM item 2), resolved from // the running device's platform. The DevicePool soft cap is platform data (0 == -// uncapped, GB10 today ⇒ pool behavior byte-for-byte unchanged). Memoized in a -// function-local static: DBuf is a per-op hot path and the process runs on ONE -// device, so the virtual dispatch is paid exactly once. Mirrors qwen3_5.cpp. +// uncapped, GB10 today ⇒ pool behavior byte-for-byte unchanged). Mirrors +// qwen3_5.cpp. +// +// Memoized PER DEVICE TYPE, not once per process. The previous single +// function-local static cached whichever device asked FIRST and then applied its +// cap to every later device — the same ambient-device assumption #516 fixed one +// layer down, and a mixed-backend process would have run a CUDA DBuf under the +// CPU platform's policy. DBuf is a per-op hot path, so the virtual dispatch is +// still paid at most once per device type. +// +// A backend whose platform was never REGISTERED now throws out of +// `platforms::GetPlatform` instead of inheriting whichever device asked first. +// That is the point: a residency cap read off another platform is a wrong +// number wearing a default's clothes. Gated by +// tests/vllm/models/test_device_pool.cpp. struct DevicePoolPolicy { size_t cap_bytes = 0; // residency_policy().device_pool_cap_bytes (0 == uncapped) }; inline DevicePoolPolicy ResolveDevicePoolPolicy(const Dev& d) { - static const DevicePoolPolicy p = [&] { - const auto rp = - vllm::platforms::GetPlatform(d.q.device.type).residency_policy(); - return DevicePoolPolicy{rp.device_pool_cap_bytes}; - }(); - return p; + // Stored as cap+1 so that 0 means "not resolved yet" and a genuine cap of 0 + // (every platform today) still caches. Racing threads resolve the same device + // type to the same value, so the benign double-resolve needs no lock. + static std::array, vt::kNumDeviceTypes> cached{}; + // Same bound, same place, as platforms::Index() (src/vllm/platforms/ + // platform.cpp) applies to this identical value before indexing ITS registry. + // An out-of-range DeviceType is only reachable by a cast, and the two lookups + // must not disagree about whether that is a throw or a stray write. + const size_t idx = static_cast(d.q.device.type); + VT_CHECK(idx < vt::kNumDeviceTypes, "invalid device type"); + const size_t seen = cached[idx].load(std::memory_order_relaxed); + if (seen != 0) return DevicePoolPolicy{seen - 1}; + const auto rp = vllm::platforms::GetPlatform(d.q.device.type).residency_policy(); + cached[idx].store(rp.device_pool_cap_bytes + 1, std::memory_order_relaxed); + return DevicePoolPolicy{rp.device_pool_cap_bytes}; } // Owned device allocation + tensor view, routed through the SHARED DevicePool so @@ -91,7 +116,11 @@ class DBuf { bytes_ = static_cast(numel) * vt::SizeOf(dt); alloc_bytes_ = bytes_ == 0 ? 1 : bytes_; cap_ = ResolveDevicePoolPolicy(d).cap_bytes; - pool_ = ActivePool(); + // THIS DEVICE's pool, unless an ActivePoolScope overrides it (the aux + // stream). Remembered so the block returns to the pool it came from even if + // this DBuf outlives the scope. See device_pool.h: there is no + // device-less pool to fall back on. + pool_ = &ActivePool(*b_); p_ = pool_->Get(*b_, alloc_bytes_); t_ = MakeTensor(p_, dt, d.q.device, shape); if (host != nullptr && bytes_ > 0) b_->Copy(d.q, p_, host, bytes_); @@ -130,16 +159,44 @@ class DBuf { b_->Synchronize(d.q); } // Relinquish the pool block WITHOUT returning it (dtor becomes a no-op); the - // caller takes over the Pool().Put obligation for alloc_bytes(). + // caller takes over the Put obligation for alloc_bytes(). Prefer + // ReleaseShared() below, which discharges that obligation correctly by + // construction. void* Release() { void* p = p_; p_ = nullptr; return p; } + // Move the block into a shared_ptr that returns it to THIS buffer's own pool + // and backend when the last owner drops it — the carrier every cross-step + // hand-off (device logits, MTP hidden states, MoE scratch) wants. + // + // It replaces ~28 copies of a hand-written deleter that closed over the byte + // count ALONE and called `Pool().Put(alloc, q)`. That idiom named neither the + // device nor the pool, so it returned every such block to the one global pool + // — a block from another device (#516), which was LIVE, and a block drawn + // from the aux-stream pool, which was not: none of the nine `Release()` sites + // sat inside or under any of the four `ActivePoolScope` regions, so the old + // deleter and the buffer's own `pool_` always agreed in practice. It was a + // hazard one new call site away from being real, and it is gone either way, + // because the carrier now captures the pool it came from rather than + // re-deriving it. + std::shared_ptr ReleaseShared() { + DevicePool* const pool = pool_; + Backend* const b = b_; + const size_t alloc = alloc_bytes_; + void* const p = Release(); + // A moved-from or already-released buffer owns nothing; a shared_ptr built + // over a null pointer with a custom deleter would still RUN that deleter and + // push null into the free list. + if (p == nullptr) return {}; + return std::shared_ptr(p, [pool, b, alloc](void* q) { pool->Put(*b, alloc, q); }); + } + private: Backend* b_; - DevicePool* pool_ = &Pool(); + DevicePool* pool_ = nullptr; void* p_ = nullptr; size_t bytes_ = 0; size_t alloc_bytes_ = 0; diff --git a/include/vllm/model_executor/models/device_pool.h b/include/vllm/model_executor/models/device_pool.h index 60dd6df72..b1d8f450b 100644 --- a/include/vllm/model_executor/models/device_pool.h +++ b/include/vllm/model_executor/models/device_pool.h @@ -1,10 +1,9 @@ // Shared process-wide caching device allocator (DevicePool) — extracted VERBATIM // from the Qwen3.6 forward (qwen3_5.cpp) so the dense Qwen3 forward (qwen3.cpp) // reuses the SAME pooled-scratch machinery instead of raw per-op Backend -// Alloc/Free. This is a pure relocation: the class body, the Pool()/AuxPool() -// singletons, and the thread-local ActivePool()/ActivePoolScope are byte-for-byte -// the qwen3_5.cpp definitions, so the 27B/35B gate-model behavior is unchanged -// (the header is included by qwen3_5.cpp in place of its old inline copies). +// Alloc/Free. The relocation was byte-for-byte the qwen3_5.cpp definitions; what +// has changed since is that a pool is now bound to ONE DEVICE (see below), and +// the accessors take the backend that names it. // // Rationale: both cudaMalloc AND cudaFree SYNCHRONIZE the whole device, so the // per-op DBuf alloc/free churn in a forward (thousands of tiny scratch buffers per @@ -15,8 +14,28 @@ // on the same queue, and CUDA stream ordering guarantees the op that last touched // the block has completed before any reused op runs — no host sync needed. Blocks // are never returned to the driver (leak at process exit, like the cublasLt -// workspace); the pool is bounded by the forward's peak concurrent scratch. The -// pool is backend-agnostic (CPU malloc/free too — a harmless bounded cache there). +// workspace); the pool is bounded by the forward's peak concurrent scratch. +// +// ONE POOL PER DEVICE (#516, .agents/specs/pool-device-key.md). Until this was +// fixed there was ONE pool for the whole process and its free list was keyed by +// byte size class alone, so a block allocated through one backend was handed to +// the next caller of that size class whatever device it was running on. One +// fault, two symptoms, chosen by direction: a cudaMalloc block reaching a +// CPU-backend forward SIGSEGVs host-side (and compute-sanitizer is CLEAN, +// because the fault is not on the device), while a host block reaching a CUDA +// forward returned a UNIFORM 0x7fff0000 quiet NaN — computed and propagated, not +// garbage read. vLLM never had the bug because it never had the design: its +// allocation handle carries the device as field 0 +// (vllm/device_allocator/__init__.py:12-14 @ pin 555967922) and torch's cache is +// per-device by construction (c10/cuda/CUDACachingAllocator.h:118-172). +// +// So a `DevicePool` is BOUND to one backend, `Pool(b)` resolves the pool for a +// device, and there is deliberately NO way to spell "the pool" without naming a +// device. The `vt::Backend*` IS the device identity: the registry hands out +// exactly one Backend* per Device{type,index} (vt/backend.h, kMaxDevicesPerType), +// and GetBackend(type) and GetBackend(Device{type,0}) return the identical +// pointer — so the device enters the key with no new virtual on vt::Backend and +// therefore no edit to any backend implementation. #pragma once #include @@ -26,9 +45,11 @@ #include #include #include +#include #include #include #include +#include #include #include "vt/backend.h" @@ -49,9 +70,21 @@ namespace vllm { // its own class bucket. VT_POOL_EXACT=1 restores exact keying (A/B measurement). class DevicePool { public: + // A pool serves exactly ONE device, named at construction. Resolve one with + // Pool(b) / AuxPool(b) rather than building your own; a directly-constructed + // pool is for tests that want an isolated free list. + explicit DevicePool(vt::Backend& b) : backend_(&b) {} + DevicePool(const DevicePool&) = delete; + DevicePool& operator=(const DevicePool&) = delete; + + // Size-class rounding is `private static`, and `tests/vt/test_cpu_isa_x86.cpp` + // exercises it directly (including the overflow throw) without a backend to + // build a pool on — this is that seam. It is deliberately `static`, so binding + // a pool to a device did not change it. static size_t SizeClassForTest(size_t bytes) { return ClassOf(bytes); } void* Get(vt::Backend& b, size_t bytes) { + RequireOwnDevice(b, "Get"); // BYPASS lane (VT_POOL_BYPASS=1) — the pool is a DETECTOR BLIND SPOT and // this is how you see through it. Two ways it hides a real defect from // compute-sanitizer: @@ -65,12 +98,7 @@ class DevicePool { // a real Free, which restores both boundaries for the detector. It is a // debugging lane only: it reinstates the per-op cudaMalloc/cudaFree sync // storm this pool exists to remove, so it is never a timing configuration. - // The backend is remembered here so the no-backend Put overload (the - // cross-step shared_ptr deleter) can free through it. - if (Bypass()) { - backend_ = &b; - return b.Alloc(bytes); - } + if (Bypass()) return b.Alloc(bytes); const size_t key = ClassOf(bytes); { std::lock_guard lk(mu_); @@ -89,13 +117,24 @@ class DevicePool { // Uncapped retention (deliberately-retained cross-step buffers: the device // logits / MTP hidden handed off via a shared_ptr deleter). Bytes are always // returned to the free list — the cross-step buffers are not cap-evicted. - void Put(size_t bytes, void* p) { - // Bypass: free for real so a later use-after-free traps. `backend_` is set by - // the Get that produced `p`, so it is non-null whenever a Put can be reached; - // the null guard keeps the lane from leaking a block if that ever stops - // holding rather than dereferencing a null backend. + // + // This used to take no backend at all, which is how ~28 copy-pasted + // `shared_ptr` deleters came to name neither the device nor the pool: they + // closed over a byte count and called `Pool().Put(alloc, q)`, so a block from + // ANY device was returned to the one global pool. The aux-stream half of that + // was LATENT rather than live: the deleter would have returned an AuxPool + // block to the main pool, but no `Release()` site sat under an + // `ActivePoolScope` — the four scope regions are leaf-ward of all nine of + // them — so the wrong-pool return was reachable only by adding a site, which + // is precisely the mistake that then costs a debugging campaign. + // `DBuf::ReleaseShared()` is now the only way to build that carrier and it + // captures the buffer's own pool and backend, so neither half can come back + // (#516). + void Put(vt::Backend& b, size_t bytes, void* p) { + RequireOwnDevice(b, "Put"); + // Bypass: free for real so a later use-after-free traps. if (Bypass()) { - if (backend_ != nullptr) backend_->Free(p); + b.Free(p); return; } const size_t key = ClassOf(bytes); @@ -110,6 +149,7 @@ class DevicePool { // When a discrete GPU sets a bound, scratch over the cap is freed to the driver // rather than pooled, so the reuse pool self-limits without a model edit. void Put(vt::Backend& b, size_t bytes, void* p, size_t cap) { + RequireOwnDevice(b, "Put"); if (Bypass()) { b.Free(p); return; @@ -137,8 +177,11 @@ class DevicePool { // // SAFETY: `free_` only ever holds blocks a DBuf already returned, so nothing // live is touched. Under VT_POOL_BYPASS the free list is always empty (Put - // frees straight through) and this is a no-op. + // frees straight through) and this is a no-op. And because a pool now holds + // ONE device's blocks, `b.Free` is guaranteed to be the allocator that made + // them — before #516 a drain could hand one device's block to another's Free. size_t Drain(vt::Backend& b) { + RequireOwnDevice(b, "Drain"); std::lock_guard lk(mu_); size_t freed = 0; for (auto& entry : free_) { @@ -156,14 +199,38 @@ class DevicePool { if (std::getenv("VT_POOL_STATS") != nullptr) { const uint64_t h = hits_.load(), m = misses_.load(); const double rate = (h + m) ? 100.0 * static_cast(h) / static_cast(h + m) : 0.0; + // The backend pointer identifies WHICH device's pool this line is about: + // a mixed-backend process now prints one line per device, and two lines + // with no way to tell them apart would be worse than one wrong line. std::fprintf(stderr, - "[DevicePool] hits=%llu misses(cudaMalloc)=%llu hit-rate=%.2f%% distinct-classes=%zu\n", + "[DevicePool backend=%p] hits=%llu misses(cudaMalloc)=%llu hit-rate=%.2f%% " + "distinct-classes=%zu\n", + static_cast(backend_), static_cast(h), static_cast(m), rate, free_.size()); } } private: + // The device check, on EVERY pool operation. A hard runtime throw and NOT an + // `assert`: the SACRED gate builds are Release/NDEBUG, where an assert is + // compiled out and the pre-fix behavior — a block silently crossing devices — + // would come straight back. It is one predictable compare against a member, + // against a `cudaMalloc` this pool exists to avoid. + // + // The only way to reach it is an `ActivePoolScope` pointing at another + // device's pool, which is precisely the mistake this row exists to make + // impossible to make quietly. + void RequireOwnDevice(vt::Backend& b, const char* op) const { + if (&b == backend_) return; + char msg[192]; + std::snprintf(msg, sizeof(msg), + "DevicePool::%s called with backend %p on a pool bound to backend %p: a scratch " + "block must never cross devices (see .agents/specs/pool-device-key.md, #516)", + op, static_cast(&b), static_cast(backend_)); + throw std::logic_error(msg); + } + // VT_POOL_BYPASS=1 turns every Get/Put into a raw driver Alloc/Free (see Get). // Read once: it must not change between an allocation and its matching free, // or a pooled block would be handed to Backend::Free (or a driver block leaked @@ -197,21 +264,66 @@ class DevicePool { } std::mutex mu_; - // Backend the last Get allocated through, so the no-backend Put overload can - // free under bypass. One device per process (see ResolveDevicePoolPolicy), so - // this is stable; unused when bypass is off. - vt::Backend* backend_ = nullptr; + // THE DEVICE, and the reason this class exists in this shape. Every block in + // `free_` was allocated by this backend and will be freed by it; nothing else + // may draw from or return to this pool. + vt::Backend* backend_; std::unordered_map> free_; size_t retained_ = 0; // bytes (class-rounded) held in free_, for the soft cap std::atomic hits_{0}; std::atomic misses_{0}; }; -inline DevicePool& Pool() { - static DevicePool p; +namespace detail { + +// Process-wide table of per-device pools. Tiny by construction: one entry per +// `vt::Backend*` the process ever allocates through, i.e. one per +// Device{type,index}. Entries are never erased, which is what lets Pool()'s +// memo below hold a raw pointer. +class PoolTable { + public: + DevicePool& For(vt::Backend& b) { + std::lock_guard lk(mu_); + for (const auto& e : pools_) + if (e.first == &b) return *e.second; + pools_.emplace_back(&b, std::unique_ptr(new DevicePool(b))); + return *pools_.back().second; + } + + private: + std::mutex mu_; + std::vector>> pools_; +}; + +inline PoolTable& MainPoolTable() { + static PoolTable t; + return t; +} + +// The (backend -> pool) resolution, memoized per thread. A DBuf resolves its +// pool on EVERY construction — thousands per forward step — and this pool's +// whole purpose is to avoid a synchronizing cudaMalloc, so paying a lock and a +// scan for it would be self-defeating. A process drives one device per host +// thread at a time, so the steady state here is a single pointer compare. +inline DevicePool& MemoizedPool(PoolTable& table, vt::Backend& b, + vt::Backend*& last_backend, DevicePool*& last_pool) { + if (last_backend == &b) return *last_pool; + DevicePool& p = table.For(b); + last_backend = &b; + last_pool = &p; return p; } +} // namespace detail + +// THE scratch pool for a device. There is deliberately no no-argument spelling: +// "the pool" without a device is the defect (#516), not an ergonomic shortcut. +inline DevicePool& Pool(vt::Backend& b) { + thread_local vt::Backend* last_backend = nullptr; + thread_local DevicePool* last_pool = nullptr; + return detail::MemoizedPool(detail::MainPoolTable(), b, last_backend, last_pool); +} + // --- Aux-stream scratch pool (ENG-MOE-SHARED-AUX) ---------------------------- // The MoE shared-expert overlap (MoeBlockFusedMarlinCuda) issues the shared MLP // on a SECOND CUDA stream concurrent with the routed experts on the main stream. @@ -228,26 +340,47 @@ inline DevicePool& Pool() { // share a live block. Blocks are handed back to the pool they came from (DBuf // stores its owning pool), so a buffer allocated in the aux region and destroyed // after the join still returns to the aux pool. +// The aux pool is per-device too: the stream distinction and the device +// distinction are independent, and a process with two devices running the MoE +// overlap needs one aux pool per device, not one shared between them. #ifdef VT_MARLIN_NVFP4 // only the Marlin MoE overlap path draws from AuxPool -inline DevicePool& AuxPool() { - static DevicePool p; - return p; +namespace detail { +inline PoolTable& AuxPoolTable() { + static PoolTable t; + return t; +} +} // namespace detail +inline DevicePool& AuxPool(vt::Backend& b) { + thread_local vt::Backend* last_backend = nullptr; + thread_local DevicePool* last_pool = nullptr; + return detail::MemoizedPool(detail::AuxPoolTable(), b, last_backend, last_pool); } #endif -// Thread-local "active" scratch pool a DBuf constructs from. Defaults to the main -// Pool(); the aux-stream overlap region swaps it to AuxPool() for the duration of -// the shared-expert issue via ActivePoolScope. Single host thread drives the +// Thread-local OVERRIDE of the scratch pool a DBuf constructs from. Null means +// "this device's own Pool(b)" — the default, and the only correct default, since +// a thread-local cannot know which device the next DBuf will be built on. The +// aux-stream overlap region swaps it to AuxPool(b) for the duration of the +// shared-expert issue via ActivePoolScope. Single host thread drives the // forward, and the aux ops are issued in one contiguous block, so the swap is a // simple RAII stack. -inline DevicePool*& ActivePool() { - thread_local DevicePool* p = &Pool(); +// +// An override pointing at ANOTHER device's pool is no longer a silent +// corruption: the pool checks its backend on every operation and throws. +inline DevicePool*& ActivePoolOverride() { + thread_local DevicePool* p = nullptr; return p; } +inline DevicePool& ActivePool(vt::Backend& b) { + DevicePool* const override_pool = ActivePoolOverride(); + return override_pool != nullptr ? *override_pool : Pool(b); +} struct ActivePoolScope { DevicePool* prev; - explicit ActivePoolScope(DevicePool* p) : prev(ActivePool()) { ActivePool() = p; } - ~ActivePoolScope() { ActivePool() = prev; } + explicit ActivePoolScope(DevicePool* p) : prev(ActivePoolOverride()) { + ActivePoolOverride() = p; + } + ~ActivePoolScope() { ActivePoolOverride() = prev; } ActivePoolScope(const ActivePoolScope&) = delete; ActivePoolScope& operator=(const ActivePoolScope&) = delete; }; diff --git a/include/vllm/model_executor/models/gemma4.h b/include/vllm/model_executor/models/gemma4.h index a5e7beee6..6aad0723d 100644 --- a/include/vllm/model_executor/models/gemma4.h +++ b/include/vllm/model_executor/models/gemma4.h @@ -147,6 +147,39 @@ Gemma4Weights LoadGemma4ForConditionalGenerationWeightsOwned( // end-to-end gate is BLOCKED on the runner allocating one uniform KV head_dim // (runner.cpp:600-646) — Gemma-4's per-layer 256/512 heads are not representable // without a shared-path change to attn_kv_ construction. See gemma4-multimodal.md. +// MODEL-DIFFUSION-LTX25 L3. What `output_hidden_states=True` returns, for the +// consumers that condition on the WHOLE stack rather than on the logits — LTX-2.5's +// text encoder is one (base_encoder.py:68-71: it runs the inner model, takes +// `outputs.hidden_states`, and never touches lm_head). +// +// `hidden_states` has `num_hidden_layers + 1` entries, each [T, H] host f32, in +// transformers' own append order: +// [0] the input embeddings, already sqrt(hidden)-scaled +// [i] the output of decoder layer i-1, for i in 1..num_hidden_layers-1 +// [L] model.norm(output of the LAST decoder layer) +// The RAW output of the last decoder layer is NOT in the tuple. A consumer that +// assumes it is gets 49 finite tensors of the right shape and the wrong content, +// which is why this order is stated here and gated in +// tests/vllm/models/test_ltx2_text_encoder.cpp rather than left to the reader. +struct Gemma4HiddenStatesResult { + // HOST f32, and that is a WIDENING of upstream's dtype — recorded here rather + // than left as bare fact, per AGENTS.md's dtype-polarity rule. Upstream runs the + // whole text tower in ONE resolved dtype, bf16 by default + // (`LTXGemmaTextEncoder.__init__`, base_encoder.py:41 + // `dtype: torch.dtype = torch.bfloat16`), and the `hidden_states` tuple it + // passes to the feature extractor is bf16. These states are downloaded from BF16 + // device buffers (gemma4.cpp:571-578) and widened on the way out because this is + // the CPU REFERENCE ARM: it is what the LTX-2.5 parity gate compares against + // upstream executed in torch float32, and every LTX entry point REFUSES a + // non-f32 compute dtype by name rather than widening silently + // (ltx2_text_encoder.h, the DTYPE note). Cost of the widening, stated so it is + // not discovered later: at the shipped 49 x 1024 x 3840 x 4B this holds ~771 MB + // host where upstream holds ~385 MB. The bf16 / FP8 / NVFP4 arms are phase L6 of + // .agents/specs/ltx-2-5.md and are OWED, not shipped. + std::vector> hidden_states; + std::vector logits; // [n_out, vocab], as Forward() returns +}; + class Gemma4Model { public: static std::vector Forward( @@ -156,6 +189,16 @@ class Gemma4Model { const HfConfig& config, vt::Queue& queue, const std::vector& logits_indices = {}); + // The same forward as Forward(), additionally returning every hidden state in + // transformers' order (see Gemma4HiddenStatesResult). Capture is OFF on every + // other entry point, so no shipped path changes shape or cost. + static Gemma4HiddenStatesResult ForwardHiddenStates( + const std::vector& token_ids, const std::vector& positions, + const v1::CommonAttentionMetadata& attn_meta, + const std::vector& attn_kv, const Gemma4Weights& weights, + const HfConfig& config, vt::Queue& queue, + const std::vector& logits_indices = {}); + static ForwardLogits ForwardDevice( const std::vector& token_ids, const std::vector& positions, const v1::CommonAttentionMetadata& attn_meta, @@ -208,6 +251,30 @@ std::vector Gemma4GenerateGreedyViaRegistry( vt::Queue& queue, int max_new_tokens, std::vector* out_margins = nullptr); +// The FULL-attention layers' "proportional" rope cos|sin table, on host in f32 — +// the exact values the forward builds (BuildProportionalRopeCache rounds them to +// bf16 to match the q/k it rotates; nothing else differs). Returns +// [max_pos + 1, head_dim]: the first head_dim/2 columns are cos and the second +// half sin, over the head_dim/2 DISTINCT angle pairs, mirroring upstream's +// `emb = cat((freqs, freqs))` with each angle stored once +// (`Gemma4UnifiedTextRotaryEmbedding.forward`, modeling_gemma4_unified.py:259-275 +// — the `cat` itself is :271; the inv_freq it consumes comes from +// modeling_rope_utils.py:187-254). +// +// This is a GATE SURFACE, and it exists because of a measurement rather than a +// preference. `partial_rotary_factor` decides how many angle pairs are rotated +// and how many are zero-padded to identity, and it is the one field on this path +// that the tower's hidden states cannot resolve: on the reduced LTX tower +// fixture, forcing it from the config's 0.25 to 1.0 displaces the worst hidden +// state by 1.09e-01 against that state's measured bf16 noise floor of 9.99e-02 +// — a ratio of 1.09, inside the tolerance the states are gated at — and a LARGER +// fixture makes it worse, not better (0.65 at head_dim 16/32, seq 32), because +// bf16 accumulation noise grows at least as fast as the rope contribution. So +// the states are the wrong instrument and this table is the right one: f32, no +// accumulation, compared element-wise against the oracle's own rotary embedding. +std::vector Gemma4ProportionalRopeCosSin(const HfConfig& config, + int64_t head_dim, int64_t max_pos); + // Wrap already-loaded Gemma-4 weights in the registered LoadedModel so a caller // that owns the weights (the mm e2e gate) can drive ModelRegistry::Forward without // re-reading the checkpoint. `Make` OWNS the moved weights; `Borrow` does NOT own diff --git a/include/vllm/model_executor/models/ltx2.h b/include/vllm/model_executor/models/ltx2.h new file mode 100644 index 000000000..f71a1a404 --- /dev/null +++ b/include/vllm/model_executor/models/ltx2.h @@ -0,0 +1,505 @@ +// LTX-2.5 — the 21.00B joint video+audio flow-matching DiT: layout + forward. +// +// Row: MODEL-DIFFUSION-ltx-2-5-ltx2-video-transformer-3d-model. Spec: +// .agents/specs/ltx-2-5.md (phase L2). Issue #435. +// +// UPSTREAM. Lightricks LTX-2, `packages/ltx-core/src/ltx_core/model/transformer/`. +// Every declaration below names the file:line it was ported from. The mapping is: +// +// Ltx2DitParams <- model_configurator.py:19-83 (LTXModelConfigurator +// .from_metadata) + model.py:48-81 (LTXModel.__init__) +// Ltx2RopeType <- rope.py:11-14 (LTXRopeType) +// Ltx2PrecomputeFreqsCis <- rope.py:198-224 (precompute_freqs_cis) +// Ltx2ApplyRotaryEmb <- rope.py:16-84 (apply_rotary_emb + both variants) +// Ltx2AdaLayerNormSingle <- adaln.py:19-45, timestep_embedding.py:6-143 +// Ltx2Attention <- attention.py:478-579 (Attention) +// Ltx2FeedForward <- feed_forward.py:6-15, gelu_approx.py:4-10 +// Ltx2TransformerBlockForward <- transformer.py:87-417 (BasicAVTransformerBlock) +// Ltx2DitForward <- model.py:492-538 (LTXModel.forward) +// EnumerateLtx2DitTensors <- the named_parameters() of the model above +// ParseLtx2DitParamsFromManifest<- the SHAPES are the config, mirroring +// ParseMiniMaxH3DitParamsFromGgufManifest +// +// WHAT THIS TU IS. The CORRECTNESS forward, gated against the upstream modules +// executed at REDUCED dimensions on CPU (scripts/gen-ltx2-goldens.py -> +// tests/vllm/models/ltx2_goldens.inc), exactly as MiniMax-H3's DiT is. Every +// projection routes through `vt::MatmulBT`; self-attention routes through +// `vt::Attention(causal=false)`; cross-attention routes through the seam this +// row added for it, `vt::AttentionCross` (vt::Attention rejects Tq != S, which no +// cross-attention can satisfy). The elementwise glue — RMSNorm, LayerNorm, +// gelu-tanh, AdaLN modulation, RoPE, the per-head attention gate — runs as +// explicit host loops so the math reads against upstream line by line. +// +// DTYPE. Everything here is f32. That is NOT a widening of a bf16 path: it is the +// PARITY dtype of the L2 gate, which compares the ALGORITHM against upstream run +// in torch float32. Upstream resolves ONE model dtype and every layer inherits it +// (model.py has no per-layer dtype at all), so the production bf16 / FP8 / NVFP4 +// arms are a single stream-dtype choice — phase L6 — and are OWED, not shipped: +// `Ltx2DitForward` REFUSES any `compute_dtype` other than `vt::DType::kF32` with a +// message naming the missing phase rather than silently computing in f32. +// +// NOT PORTED IN L2, recorded here so it cannot be discovered later: +// - Guidance perturbations (guidance/perturbations.py BatchedPerturbationConfig, +// transformer.py:330-397 `*_perturbation_mask` / `cross_attn_skip_all` and +// attention.py:545-552 `perturbation_mask`). L2 runs the no-perturbation +// configuration, whose masks are all-ones and whose flags are all false — +// upstream's own `perturbations=None` path (model.py:509-511). +// - `use_keyframes_abs_pos_embedding` (transformer_args.py:23-43). LTX-2.5's +// checkpoint does not carry the parameter; the enumeration refuses a config +// that asks for it. +// - The caption projections (text_projection.py:31-38). LTX-2.5 is a 22B-form +// checkpoint: `caption_proj_before_connector=true` puts them in the TEXT +// ENCODER, so the DiT has none (model_configurator.py:199-219). They are +// phase L3. +// - `prompt_adaln_single` (model.py:223-227). LTX-2.5 sets +// `use_prompt_adaln_single=false`; see Ltx2PromptKvCache. +#pragma once + +#include +#include +#include +#include + +#include + +#include "vt/device.h" +#include "vt/dtype.h" +#include "vt/tensor.h" + +namespace vllm { + +// --------------------------------------------------------------------------- +// Architecture +// --------------------------------------------------------------------------- + +// rope.py:11-14. SPLIT is LTX-2.5's setting; INTERLEAVED is upstream's documented +// legacy mode and is ported because a checkpoint may still select it. +enum class Ltx2RopeType { kSplit, kInterleaved }; + +// LTXModel.__init__ defaults (model.py:48-81), which are also +// LTXModelConfigurator.from_metadata's `config.get(...)` fallbacks +// (model_configurator.py:46-83). `ParseLtx2DitParams` overrides any key the +// checkpoint's `transformer` config actually carries, exactly like from_metadata. +struct Ltx2DitParams { + int64_t num_layers = 48; + // Video stream. inner_dim = num_attention_heads * attention_head_dim. + int64_t num_attention_heads = 32; + int64_t attention_head_dim = 128; + int64_t in_channels = 128; + int64_t out_channels = 128; + int64_t cross_attention_dim = 4096; // text context width for `attn2` + // Audio stream. audio_inner_dim = audio_num_attention_heads * audio_attention_head_dim. + int64_t audio_num_attention_heads = 32; + int64_t audio_attention_head_dim = 64; + int64_t audio_in_channels = 128; + int64_t audio_out_channels = 128; + // Serves TWO roles upstream: `audio_attn2`'s context width, and the `inner_dim` + // the audio<->video cross positional embedding is built at + // (transformer_args.py:364-371). Both are the audio stream width. + int64_t audio_cross_attention_dim = 2048; + + double norm_eps = 1e-6; + double positional_embedding_theta = 10000.0; + std::vector positional_embedding_max_pos = {20, 2048, 2048}; + std::vector audio_positional_embedding_max_pos = {20}; + int64_t timestep_scale_multiplier = 1000; + int64_t av_ca_timestep_scale_multiplier = 1; + bool use_middle_indices_grid = true; + Ltx2RopeType rope_type = Ltx2RopeType::kSplit; + // model_configurator.py:68 — `frequencies_precision == "float64"`. Selects the + // numpy float64 frequency ladder (rope.py:87-107) over the torch float32 one + // (:110-131); the two differ in the last f32 ulps of every RoPE angle. + bool double_precision_rope = false; + bool apply_gated_attention = false; + bool cross_attention_adaln = false; + // model_configurator.py:74-76. FALSE on LTX-2.5, which is what makes the + // cross-attention K/V timestep-independent — see Ltx2PromptKvCache. + bool use_prompt_adaln_single = true; + // model_configurator.py:77-80. LTX-2.5 (gemma4) sets ff_bias=false and leaves + // audio_ff_bias at its true default; the checkpoint's shapes agree. + bool ff_bias = true; + bool audio_ff_bias = true; + + int64_t inner_dim() const { return num_attention_heads * attention_head_dim; } + int64_t audio_inner_dim() const { return audio_num_attention_heads * audio_attention_head_dim; } + // adaln.py:14-16 — 6 base params, +3 when cross-attention AdaLN is on. + int64_t adaln_embedding_coefficient() const { return cross_attention_adaln ? 9 : 6; } + // model.py:130. + int64_t cross_pe_max_pos() const { + return positional_embedding_max_pos[0] > audio_positional_embedding_max_pos[0] + ? positional_embedding_max_pos[0] + : audio_positional_embedding_max_pos[0]; + } + // The audio<->video cross attention projects Q from the VIDEO stream but sizes + // its heads from the AUDIO stream (transformer.py:154-175), so its inner width + // is the audio one while its query/out width is the video one. This asymmetry is + // the one a square assumption silently transposes. + int64_t av_cross_inner_dim() const { return audio_inner_dim(); } +}; + +// LTXModelConfigurator.from_metadata (model_configurator.py:19-83). `metadata` is +// the checkpoint's metadata mapping; the transformer keys are read from +// `metadata["config"]["transformer"]` when that nesting is present and from the +// object itself otherwise. Every `check_config_value` upstream asserts +// (:26-44) is asserted here with the same expectation. +Ltx2DitParams ParseLtx2DitParams(const nlohmann::json& metadata); + +// One parameter of the DiT, keyed by its EXACT upstream `named_parameters()` name. +struct Ltx2TensorSpec { + std::string name; + std::vector shape; +}; + +// The weight contract: every parameter LTXModel(model_type=AudioVideo) creates, +// in a deterministic order. This IS the layout — the parity suite compares it +// against the upstream module's own named_parameters(). +std::vector EnumerateLtx2DitTensors(const Ltx2DitParams& params); + +// The SHAPES are the config: recover the geometry from a manifest alone, the way +// a ComfyUI-format checkpoint (which carries no transformer config) has to be +// read. Mirrors ParseMiniMaxH3DitParamsFromGgufManifest. Throws by name when a +// tensor the geometry is derived from is missing. +Ltx2DitParams ParseLtx2DitParamsFromManifest(const std::vector& manifest); + +// --------------------------------------------------------------------------- +// Weight views +// --------------------------------------------------------------------------- + +// `bias` is Empty() for a bias-free projection (`ff` on LTX-2.5, ff_bias=false). +struct Ltx2LinearWeight { + vt::Tensor weight; // [out_features, in_features] + vt::Tensor bias; // [out_features] or empty +}; + +// attention.py:478-518. `q_norm`/`k_norm` are torch.nn.RMSNorm(INNER_DIM) — the +// norm runs over the WHOLE projected width before the head split (:505-506, +// applied at ops.py:32-33), not per head. `to_gate_logits` is [heads, query_dim] +// (:513-514) and is empty when apply_gated_attention is false. +struct Ltx2AttentionWeights { + Ltx2LinearWeight to_q, to_k, to_v; + vt::Tensor q_norm, k_norm; // [inner_dim] + Ltx2LinearWeight to_gate_logits; + Ltx2LinearWeight to_out; // to_out.0 +}; + +// feed_forward.py:6-15. `net.0.proj` -> gelu(tanh) -> `net.2`. +struct Ltx2FeedForwardWeights { + Ltx2LinearWeight proj_in; // net.0.proj, [inner, dim] + Ltx2LinearWeight proj_out; // net.2, [dim, inner] +}; + +// adaln.py:19-45 + timestep_embedding.py:118-143. +struct Ltx2AdaLayerNormSingleWeights { + Ltx2LinearWeight linear_1; // emb.timestep_embedder.linear_1, [dim, 256] + Ltx2LinearWeight linear_2; // emb.timestep_embedder.linear_2, [dim, dim] + Ltx2LinearWeight linear; // [coefficient * dim, dim] +}; + +// transformer.py:87-189 (BasicAVTransformerBlock's parameters). +struct Ltx2BlockWeights { + Ltx2AttentionWeights attn1, attn2; + Ltx2AttentionWeights audio_attn1, audio_attn2; + Ltx2AttentionWeights audio_to_video_attn; // Q video, K/V audio + Ltx2AttentionWeights video_to_audio_attn; // Q audio, K/V video + Ltx2FeedForwardWeights ff, audio_ff; + vt::Tensor scale_shift_table; // [9, dim] (:125) + vt::Tensor audio_scale_shift_table; // [9, adim] (:150) + vt::Tensor prompt_scale_shift_table; // [2, dim] (:185) + vt::Tensor audio_prompt_scale_shift_table; // [2, adim] (:187) + vt::Tensor scale_shift_table_a2v_ca_video; // [5, dim] (:178) + vt::Tensor scale_shift_table_a2v_ca_audio; // [5, adim] (:177) +}; + +// model.py:202-287 (the model-level parameters) plus the block stack. +struct Ltx2DitWeights { + Ltx2LinearWeight patchify_proj, proj_out; + Ltx2AdaLayerNormSingleWeights adaln_single; + vt::Tensor scale_shift_table; // [2, dim] — the OUTPUT table (:230), not the block's [9, dim] + Ltx2LinearWeight audio_patchify_proj, audio_proj_out; + Ltx2AdaLayerNormSingleWeights audio_adaln_single; + vt::Tensor audio_scale_shift_table; // [2, adim] (:260) + Ltx2AdaLayerNormSingleWeights av_ca_video_scale_shift; // (:269) + Ltx2AdaLayerNormSingleWeights av_ca_audio_scale_shift; // (:274) + Ltx2AdaLayerNormSingleWeights av_ca_a2v_gate; // (:279) + Ltx2AdaLayerNormSingleWeights av_ca_v2a_gate; // (:284) + std::vector blocks; +}; + +// Bind every view in `Ltx2DitWeights` from tensors keyed by their upstream +// parameter name. A name the contract requires and the map lacks throws BY NAME +// rather than reading as zeros. Optional parameters (biases a config turns off, +// the prompt tables when cross_attention_adaln is false, the gate logits when +// gating is off) are bound only when the params say they exist. +Ltx2DitWeights BindLtx2DitWeights(const Ltx2DitParams& params, + const std::map& tensors); + +// --------------------------------------------------------------------------- +// RoPE (rope.py) +// --------------------------------------------------------------------------- + +// precompute_freqs_cis (rope.py:198-224). `positions` is the middle-indices grid +// [batch, n_pos_dims, tokens, 2] holding each patch's [start, end) bounds when +// `use_middle_indices_grid` is true, and [batch, n_pos_dims, tokens] otherwise. +// `dim` is the width the frequency ladder is built for — inner_dim for a stream's +// own RoPE, audio_cross_attention_dim for the audio<->video cross RoPE. +// +// Output layout mirrors upstream's tensors exactly: +// SPLIT -> cos/sin are [batch, heads, tokens, dim/(2*heads)] (:182-183) +// INTERLEAVED -> cos/sin are [batch, tokens, dim] (:187-195) +struct Ltx2FreqsCis { + std::vector cos, sin; + std::vector shape; +}; +// generate_freq_grid_pytorch (rope.py:110-131) / generate_freq_grid_np (:87-107): +// `theta ** linspace(0, 1, dim / (2 * n_pos_dims)) * pi/2`, built in float32 or, +// when `double_precision` is set (`frequencies_precision == "float64"`), in +// float64 and cast down. Exposed because the two ladders differ only in the last +// f32 ulps of each sample, and the audio ladder multiplies those up by four +// orders of magnitude before RoPE takes their cosine. +std::vector Ltx2FreqGrid(double theta, int64_t n_pos_dims, int64_t dim, + bool double_precision); + +// `n_pos_dims` is how many leading position axes are CONSUMED (3 for a video +// stream's own RoPE, 1 for the audio stream and for the audio<->video cross RoPE, +// which slices `positions[:, 0:1, :]` at transformer_args.py:365). +// `source_n_pos_dims` is how many the BUFFER actually carries, so a 1-axis read of +// a 3-axis video grid strides correctly. +Ltx2FreqsCis Ltx2PrecomputeFreqsCis(const double* positions, int64_t batch, int64_t tokens, + int64_t n_pos_dims, int64_t source_n_pos_dims, + bool use_middle_indices_grid, int64_t dim, + const std::vector& max_pos, double theta, + int64_t num_attention_heads, Ltx2RopeType rope_type, + bool double_precision); + +// apply_rotary_emb (rope.py:16-27) over `x` [batch, tokens, dim] IN PLACE. +// `heads` is the head count the SPLIT layout was built with. +void Ltx2ApplyRotaryEmb(float* x, int64_t batch, int64_t tokens, int64_t dim, int64_t heads, + const Ltx2FreqsCis& pe, Ltx2RopeType rope_type); + +// --------------------------------------------------------------------------- +// Bricks (each gated on its own so a failure localizes) +// --------------------------------------------------------------------------- + +// AdaLayerNormSingle.forward (adaln.py:39-45). `timesteps` is [count] ALREADY +// scaled by timestep_scale_multiplier (transformer_args.py:177). Returns the +// modulation `linear(silu(emb))` in `modulation` [count, coefficient * dim] and +// the embedding itself in `embedded` [count, dim]. +struct Ltx2AdalnOut { + std::vector modulation; + std::vector embedded; +}; +Ltx2AdalnOut Ltx2AdaLayerNormSingle(vt::Device device, const Ltx2AdaLayerNormSingleWeights& w, + const float* timesteps, int64_t count, int64_t dim); + +// FeedForward.forward (feed_forward.py:14-15): net.0.proj -> gelu(tanh) -> net.2. +std::vector Ltx2FeedForward(vt::Device device, const Ltx2FeedForwardWeights& w, + const float* x, int64_t rows, int64_t dim, int64_t inner); + +// The K/V half of Attention.forward, split out because LTX-2.5 can CACHE it: +// with `use_prompt_adaln_single=false` the prompt modulation carries no timestep +// term (transformer.py:441), so `to_k`/`to_v` over the modulated context — and +// their k_norm, and the absence of RoPE on the text path — depend only on the +// prompt. The denoise loop computes them ONCE PER REQUEST and reuses them for +// every step. Layout: k/v are [batch * context_tokens, heads * dim_head], held +// exactly as the attention op consumes them (post-norm, post-RoPE). +struct Ltx2CrossKv { + std::vector k, v; +}; + +// The identity of the PROMPT a cache was filled for. +// +// The cached K/V are timestep-independent, so reusing them across DENOISE STEPS +// is exact. They are NOT prompt-independent, and a geometry check cannot tell +// two different prompts of equal token count apart — which is precisely the case +// a pipeline hits when one cache outlives one request. Serving request 1's K/V +// to request 2 renders request 1's prompt with no shape mismatch, no non-finite +// value and no error, so the cache carries a CONTENT fingerprint over both +// streams' context tensors, their geometry, and their prompt masks, and a +// mismatch is refused by name. +// +// The digests are FNV-1a over the raw bytes: an exact-identity test, never an +// approximate one. Two prompts that differ in one f32 ulp are two prompts. +struct Ltx2PromptIdentity { + int64_t batch = 0; + int64_t video_context_tokens = 0, audio_context_tokens = 0; + int64_t video_context_dim = 0, audio_context_dim = 0; + uint64_t video_context_digest = 0, audio_context_digest = 0; + uint64_t video_mask_digest = 0, audio_mask_digest = 0; +}; + +// Per-request cache of every block's text-cross-attention K/V, both streams. +// `valid` is set by the first forward that fills it; later forwards reuse it — +// but ONLY for the prompt `prompt` records. Gated BIT-IDENTICAL against +// recomputation, and gated to REFUSE a changed prompt: a cache that silently +// diverges is exactly the failure this is here to make impossible. +struct Ltx2PromptKvCache { + bool valid = false; + Ltx2PromptIdentity prompt; + std::vector video; // one per block + std::vector audio; // one per block + + // Unbind the cache from its prompt so the next forward re-fills it for a new + // one. This is how a pipeline REUSES the allocation across requests; anything + // else is refused rather than served stale. + void Reset() { + valid = false; + prompt = Ltx2PromptIdentity{}; + video.clear(); + audio.clear(); + } +}; + +// Attention.forward (attention.py:520-579). `context` is nullptr for +// self-attention (upstream's `context = x if context is None else context`). +// `pe`/`k_pe` are nullptr when no RoPE applies (every text cross-attention). +// `bias` is the optional additive score bias, [tokens, context_tokens] or +// [1, context_tokens], nullptr for none. +struct Ltx2AttentionArgs { + int64_t batch = 1; + int64_t tokens = 0; // Tq + int64_t context_tokens = 0; // S; equals `tokens` for self-attention + int64_t query_dim = 0; + int64_t context_dim = 0; + int64_t heads = 0; + int64_t dim_head = 0; + // The q/k RMSNorm eps: `Attention.__init__`'s `norm_eps: float = 1e-6` + // (attention.py:485), handed to both RMSNorms (attention.py:505-506). + // + // This DEFAULT is read by nothing today — every construction of this struct + // assigns it — and a 10^6 mutation of it leaves every suite green. It is a + // latent trap rather than live code, so the only instrument that can hold it is + // the pin in tests/vllm/models/test_ltx2.cpp. Keep it equal to + // `Ltx2DitParams::norm_eps`, which is where every real call site sources it. + double norm_eps = 1e-6; + Ltx2RopeType rope_type = Ltx2RopeType::kSplit; + const Ltx2FreqsCis* pe = nullptr; + const Ltx2FreqsCis* k_pe = nullptr; + const float* bias = nullptr; + int64_t bias_rows = 0; + // Prompt-K/V cache hooks. `kv_in` supplies K/V to reuse instead of projecting + // the context again (the `to_v` / `to_k` / k_norm chain is skipped wholesale); + // `kv_out`, when non-null, receives the K/V this call computed. Legal only on + // a path whose K/V really are timestep-independent — the caller owns that. + const Ltx2CrossKv* kv_in = nullptr; + Ltx2CrossKv* kv_out = nullptr; +}; +std::vector Ltx2Attention(vt::Device device, const Ltx2AttentionWeights& w, const float* x, + const float* context, const Ltx2AttentionArgs& args); + +// --------------------------------------------------------------------------- +// Forward +// --------------------------------------------------------------------------- + +// One modality's inputs — the fields of `Modality` (modality.py:9-63) this phase +// consumes. Row-major host buffers. +struct Ltx2ModalityInput { + int64_t batch = 1; + int64_t tokens = 0; // T + int64_t context_tokens = 0; // S + const float* latent = nullptr; // [batch, tokens, in_channels] + const float* timesteps = nullptr; // [batch, tokens] + const float* sigma = nullptr; // [batch] + // [batch, n_pos_dims, tokens, 2] when use_middle_indices_grid, else + // [batch, n_pos_dims, tokens]. n_pos_dims is 3 for video, 1 for audio. + const double* positions = nullptr; + const float* context = nullptr; // [batch, context_tokens, context_dim] + // Optional prompt mask in {0, 1} over the context tokens, [batch, context_tokens]. + // Converted by `(mask - 1) * finfo(f32).max` exactly as + // TransformerArgsPreprocessor._prepare_attention_mask does (transformer_args.py:199-206). + const int32_t* context_mask = nullptr; + // Optional self-attention STRENGTH mask in [0, 1], [batch, tokens, tokens] or + // the key-only broadcast [batch, 1, tokens]. Converted to an additive log-space + // bias exactly as _prepare_self_attention_mask does (transformer_args.py:208-237). + const float* attention_mask = nullptr; + int64_t attention_mask_rows = 0; // `tokens` for the dense form, 1 for key-only + bool enabled = true; // Modality.enabled -> TransformerArgs.enabled +}; + +// The prompt identity a `Ltx2PromptKvCache` is bound to, over the two streams' +// context tensors, their geometry and their prompt masks — every input the +// cached K/V are a function of. Exposed so a caller can key its own per-request +// cache the same way rather than re-deriving the rule. +Ltx2PromptIdentity Ltx2PromptIdentityOf(const Ltx2DitParams& params, + const Ltx2ModalityInput& video, + const Ltx2ModalityInput& audio); + +struct Ltx2DitOutputs { + std::vector video; // [batch, video tokens, out_channels] + std::vector audio; // [batch, audio tokens, audio_out_channels] +}; + +// LTXModel.forward (model.py:492-538) for model_type=AudioVideo, plus the +// preprocessors it drives (transformer_args.py:263-411). BOTH streams are +// required: LTX-2.5 is an AudioVideo checkpoint, and the VideoOnly / AudioOnly +// model types (model.py:31-33) build a different parameter set, so they are +// refused rather than served by an ungated path. To run one stream of an AV model +// — which is what upstream's own pipeline does — clear `enabled` on the other; +// the audio<->video cross attention still reads its state, exactly as +// transformer.py:265-269 does. +// +// `compute_dtype` must be vt::DType::kF32 — see the DTYPE note at the top of this +// file. Anything else is REFUSED with a message naming phase L6. +// +// `cache`, when non-null, is the prompt-K/V cache: filled on the first call and +// reused afterwards. It is only legal when `use_prompt_adaln_single` is false — +// with the prompt AdaLN MLP enabled the K/V carry a timestep term and caching +// them would be wrong, so that combination is refused rather than approximated. +// A filled cache is bound to the PROMPT it was filled for: a call whose context +// tensors, context geometry or prompt masks differ from that prompt is REFUSED +// by name (call `Ltx2PromptKvCache::Reset()` to rebind it to a new request) +// rather than served K/V that would render the previous prompt. +Ltx2DitOutputs Ltx2DitForward(vt::Device device, const Ltx2DitParams& params, + const Ltx2DitWeights& weights, const Ltx2ModalityInput* video, + const Ltx2ModalityInput* audio, vt::DType compute_dtype, + Ltx2PromptKvCache* cache = nullptr); + +// One BasicAVTransformerBlock (transformer.py:254-417), exposed so the block is +// gateable on its own. `video_x` / `audio_x` are updated IN PLACE. +struct Ltx2BlockArgs { + int64_t batch = 1; + int64_t video_tokens = 0, audio_tokens = 0; + int64_t video_context_tokens = 0, audio_context_tokens = 0; + bool video_enabled = true, audio_enabled = true; + // Per-token AdaLN modulation, [batch, tokens, coefficient * dim]. + const float* video_timestep_modulation = nullptr; + const float* audio_timestep_modulation = nullptr; + // Audio<->video cross-attention AdaLN inputs (transformer_args.py:388-411). + const float* video_cross_scale_shift = nullptr; // [batch, video tokens, 4 * dim] + const float* video_cross_gate = nullptr; // [batch, 1, dim] + const float* audio_cross_scale_shift = nullptr; // [batch, audio tokens, 4 * adim] + const float* audio_cross_gate = nullptr; // [batch, 1, adim] + const float* video_context = nullptr; + const float* audio_context = nullptr; + const float* video_context_bias = nullptr; // additive, [batch * 1, S] broadcast + const float* audio_context_bias = nullptr; + const float* video_self_bias = nullptr; // additive, [batch * rows, T] + int64_t video_self_bias_rows = 0; + const float* audio_self_bias = nullptr; + int64_t audio_self_bias_rows = 0; + const Ltx2FreqsCis* video_pe = nullptr; + const Ltx2FreqsCis* audio_pe = nullptr; + const Ltx2FreqsCis* video_cross_pe = nullptr; + const Ltx2FreqsCis* audio_cross_pe = nullptr; + Ltx2CrossKv* video_prompt_kv = nullptr; // non-null: use if `filled`, else fill + Ltx2CrossKv* audio_prompt_kv = nullptr; + bool prompt_kv_filled = false; +}; +void Ltx2TransformerBlockForward(vt::Device device, const Ltx2DitParams& params, + const Ltx2BlockWeights& weights, const Ltx2BlockArgs& args, + float* video_x, float* audio_x); + +// --------------------------------------------------------------------------- +// Mask preparation (transformer_args.py:199-237) +// --------------------------------------------------------------------------- + +// _prepare_attention_mask (:199-206): a {0,1} prompt mask becomes +// `(mask - 1) * finfo(f32).max`, i.e. 0 for a kept key and -FLT_MAX for a masked +// one. Result is [batch, context_tokens], one broadcast row per batch element. +std::vector Ltx2PrepareContextMask(const int32_t* mask, int64_t batch, + int64_t context_tokens); + +// _prepare_self_attention_mask (:208-237): a [0,1] STRENGTH mask becomes +// `log(clamp(m, min=tiny))` for m > 0 and finfo(f32).min for m <= 0. +std::vector Ltx2PrepareSelfAttentionMask(const float* mask, int64_t count); + +} // namespace vllm diff --git a/include/vllm/model_executor/models/ltx2_audio_vae.h b/include/vllm/model_executor/models/ltx2_audio_vae.h new file mode 100644 index 000000000..e8823f252 --- /dev/null +++ b/include/vllm/model_executor/models/ltx2_audio_vae.h @@ -0,0 +1,259 @@ +// LTX-2.5 AUDIO VAE — the spectrogram decoder and its BigVGAN vocoder. +// +// LTX-2.5 (Lightricks/LTX-2) decodes one request into frames PLUS a stereo +// waveform. The waveform half is two stages: an `AudioDecoder` that turns audio +// latents back into a log-mel spectrogram, and a `Vocoder` that turns that +// spectrogram into samples. Both are pure-Python `ltx_core` modules upstream, so +// unlike MiniMax-H3's checkpoint remote code they can be executed directly as the +// oracle — which is exactly what scripts/gen-ltx2-vae-goldens.py does, at reduced +// dimensions on CPU, with both sides rebuilding weights from one deterministic +// stream so NO WEIGHT BYTE is checked in. +// +// ─── WHAT THIS IS A PORT OF (file:line on BOTH sides) ──────────────────────── +// Upstream root: Lightricks/LTX-2, packages/ltx-core/src/ltx_core/ +// OURS <- UPSTREAM +// Ltx2AudioDecoderForward <- model/audio_vae/audio_vae.py:385-400 +// (its ResnetBlock) <- model/audio_vae/resnet.py:115-176 +// (its AttnBlock) <- model/audio_vae/attention.py:16-55 +// (its Upsample) <- model/audio_vae/upsample.py:12-55 +// (its CausalConv2d) <- model/audio_vae/causal_conv_2d.py:7-64 +// (PixelNorm / GroupNorm) <- model/common/normalization.py:14-59 +// (per-channel statistics) <- model/audio_vae/ops.py:58-75 +// (the audio patchifier) <- components/patchifiers.py:287-330 +// Ltx2VocoderForward <- model/audio_vae/vocoder.py:398-438 +// (AMPBlock1 / SnakeBeta) <- model/audio_vae/vocoder.py:208-290 +// (Activation1d up/down) <- model/audio_vae/vocoder.py:104-184 +// (ResBlock1, legacy arm) <- model/audio_vae/resnet.py:12-80 +// Ltx2VocoderWithBweForward <- model/audio_vae/vocoder.py:573-630 +// (MelSTFT / _STFTFn) <- model/audio_vae/vocoder.py:441-516 +// Ltx2KaiserSincFilter1d <- model/audio_vae/vocoder.py:52-70 +// Ltx2HannSincResampleFilter1d <- model/audio_vae/vocoder.py:116-128 +// +// ─── THE THREE THINGS THAT FAIL SILENTLY ───────────────────────────────────── +// * CAUSALITY LIVES ON THE HEIGHT AXIS. The shipped `causality_axis` is +// `height` (model/audio_vae/model_configurator.py:134) and the tensor layout is +// (batch, channels, TIME, mel_bins) — so dim 2 is time and ALL of its +// convolution padding goes on the LEFT (top). Padding it symmetrically still +// produces a plausible spectrogram that simply peeks into the future. +// MEASURED CAVEAT, so nobody over-claims it: `causality_axis` governs the +// CONVOLUTIONS ONLY. The decoder's `AttnBlock`s attend over the entire +// (time, mel) map (attention.py:31-55), so with the shipped mid-block +// attention on, the decoder as a whole is NOT causal in time — upstream moves +// every output frame under a last-frame perturbation, and the test asserts +// exactly that alongside the convolution-only reach. +// * THE UPSAMPLER DROPS THE FIRST TEMPORAL ELEMENT, not the last +// (upsample.py:29-42): only the first two interpolated elements depend on a +// single input element, so undoing the encoder's pad means dropping the head. +// * THE VOCODER'S ANTI-ALIASING FILTERS ARE COMPUTED, NEVER LOADED. Both the +// kaiser-sinc window the activations use and the hann-sinc window the BWE +// resampler uses are built at construction; a checkpoint carries neither. They +// are gated on their own before the decoders are blamed for a mismatch. +#pragma once + +#include +#include +#include +#include + +namespace vllm { + +// --------------------------------------------------------------------------- +// Parameters keyed by their upstream `state_dict` name, so the module's own +// naming IS the contract. The same bag serves the audio VAE, the vocoder and the +// Conv video VAE (ltx2_video_vae.h), which is why it lives in one place. +// --------------------------------------------------------------------------- +struct Ltx2VaeWeights { + std::map> tensors; + + const std::vector& Get(const std::string& name) const; + bool Has(const std::string& name) const { return tensors.count(name) != 0; } +}; + +// causality_axis.py:4-10. The axis whose padding is one-sided; `kHeight` is the +// shipped default and, in the (b, c, time, mel) layout, it is TIME. +enum class Ltx2CausalityAxis { kNone, kWidth, kHeight, kWidthCompatibility }; + +// common/normalization.py:7-11. `kPixel` is the shipped default and is +// PARAMETER-FREE, so a pixel-norm checkpoint carries no norm tensors at all. +enum class Ltx2NormType { kGroup, kPixel }; + +// --------------------------------------------------------------------------- +// AudioDecoder (audio_vae.py:277-494) +// --------------------------------------------------------------------------- + +struct Ltx2AudioDecoderConfig { + // Names and defaults mirror AudioDecoderConfigurator.from_metadata + // (audio_vae/model_configurator.py:108-141). + int64_t ch = 128; + int64_t out_ch = 2; + std::vector ch_mult = {1, 2, 4}; + int64_t num_res_blocks = 2; + std::vector attn_resolutions = {8, 16, 32}; + int64_t resolution = 256; + int64_t z_channels = 8; + Ltx2NormType norm_type = Ltx2NormType::kPixel; + Ltx2CausalityAxis causality_axis = Ltx2CausalityAxis::kHeight; + bool mid_block_add_attention = true; + // The decoder's TARGET mel-bin count. 0 keeps whatever the latent carried, + // mirroring `mel_bins=None` (audio_vae.py:422). + int64_t mel_bins = 0; + // Only read on the GroupNorm arm; PixelNorm has no parameters at all. Neither + // is reachable from a checkpoint: `build_normalization_layer` passes `eps=1e-6` + // as a LITERAL and forwards its own `num_groups` keyword, whose default is 32 + // (normalization.py:44, 56), and no audio_vae call site passes `num_groups`. + // They are fields here so the gate can pin them. + // + // `norm_type = kGroup` is not a dead arm, but it is not what pure defaults give + // you either. `AudioDecoder.__init__` declares `norm_type = GROUP` + // (audio_vae.py:294) and, on the very next line, `causality_axis = WIDTH` + // (audio_vae.py:295) — and `ResnetBlock` refuses that combination with + // `ValueError: Causal ResnetBlock with GroupNorm is not supported` + // (resnet.py:130-131), verified by construction against the pinned upstream. A + // group-norm checkpoint is therefore one that declares `causality_axis: none` + // alongside it, which is legal and is what the group-norm golden in + // test_ltx2_vae.cpp runs. Before that arm existed this eps was never READ on + // any path, and a 100x change moved nothing. + int64_t num_groups = 32; + double norm_eps = 1e-6; + // The audio VAE reaches PixelNorm through `build_normalization_layer`, which + // passes eps=1e-6 (normalization.py:58). The VIDEO VAE constructs `PixelNorm()` + // bare and so gets its 1e-8 DEFAULT (normalization.py:22). The two are + // deliberately different fields with deliberately different defaults; see the + // honesty note on Ltx2ConvVideoDecoderConfig::pixel_norm_eps. + double pixel_norm_eps = 1e-6; + // `state_dict` prefix, so one bag can hold several modules. + std::string prefix; + + int64_t num_resolutions() const { return static_cast(ch_mult.size()); } +}; + +// A (channels, frames, mel_bins) spectrogram, channel-major and contiguous. +struct Ltx2AudioSpectrogram { + int64_t channels = 0; + int64_t frames = 0; + int64_t mel_bins = 0; + std::vector data; +}; + +// AudioDecoder.forward at batch 1: denormalize -> conv_in -> mid -> up path -> +// norm/SiLU/conv_out -> crop-or-pad to the target shape. `latent` is +// [latent_channels, latent_frames, latent_mel_bins], channel-major. +// +// Batch is fixed at 1 deliberately: the pipeline decodes one request's latents, +// and upstream's per-channel statistics broadcast over the patchified last axis +// rather than over the batch, so nothing here is batch-coupled. +Ltx2AudioSpectrogram Ltx2AudioDecoderForward(const Ltx2AudioDecoderConfig& config, + const Ltx2VaeWeights& weights, + const std::vector& latent, + int64_t latent_channels, int64_t latent_frames, + int64_t latent_mel_bins); + +// --------------------------------------------------------------------------- +// Vocoder (vocoder.py:293-438) +// --------------------------------------------------------------------------- + +struct Ltx2VocoderConfig { + // Shapes mirror Vocoder.__init__ (vocoder.py:317-341), but the two ARM + // selectors deliberately do NOT: `Vocoder.__init__` defaults to `resblock="1"` + // and `activation="snake"`, i.e. the pre-2.3 legacy arm, while `amp` and + // `snakebeta` below default to the AMP1/snakebeta arm. + // + // That is not a mismatch, it is the LTX-2.5 contract. A 2.5 checkpoint reaches + // the vocoder through VocoderConfigurator.from_metadata, whose BWE branch + // REQUIRES both — `check_config_value(vocoder_cfg, "resblock", "AMP1")` and + // `check_config_value(vocoder_cfg, "activation", "snakebeta")`, and the same + // pair again for the BWE generator (audio_vae/model_configurator.py:59-64). + // A default that mirrored `Vocoder.__init__` would therefore be a default no + // shipping checkpoint can use. The legacy arm stays reachable by setting + // `amp = false`, which is what the `resblock == "1"` branch selects + // (model_configurator.py:53), and it is gated on its own. + std::vector resblock_kernel_sizes = {3, 7, 11}; + std::vector upsample_rates = {6, 5, 2, 2, 2}; + std::vector upsample_kernel_sizes = {16, 15, 8, 4, 4}; + std::vector> resblock_dilation_sizes = {{1, 3, 5}, {1, 3, 5}, {1, 3, 5}}; + int64_t upsample_initial_channel = 1024; + // resblock == "AMP1": BigVGAN v2 AMPBlock1 with anti-aliased activations. False + // selects the legacy ResBlock1 arm (`resblock == "1"`, plain leaky ReLU), which + // is what pre-2.3 checkpoints carry (audio_vae/model_configurator.py:53). + bool amp = true; + // activation == "snakebeta" vs "snake" (vocoder.py:242). Only read when `amp`. + bool snakebeta = true; + bool use_tanh_at_final = true; + bool apply_final_activation = true; + bool use_bias_at_final = true; + int64_t output_sampling_rate = 24000; + // Snake/SnakeBeta store alpha/beta in LOG scale upstream (alpha_logscale=True, + // vocoder.py:193). + bool snake_logscale = true; + std::string prefix; +}; + +// Vocoder.forward over a STEREO spectrogram [2, frames, mel_bins], channel-major. +// Upstream hardcodes conv_pre's input width to 128 = 2 channels x 64 mel bins +// (vocoder.py:350-358), so the interleave `b s c t -> b (s c) t` is part of the +// contract: flattening channel-last instead time-smears the audio while still +// producing something that plays. Returns [2, samples]. +std::vector Ltx2VocoderForward(const Ltx2VocoderConfig& config, + const Ltx2VaeWeights& weights, const std::vector& mel, + int64_t channels, int64_t frames, int64_t mel_bins, + int64_t* out_samples); + +// --------------------------------------------------------------------------- +// VocoderWithBWE (vocoder.py:519-630) — the ltx-2.3+ arm the 2.5 checkpoint uses. +// --------------------------------------------------------------------------- + +// The floor under the BWE mel BEFORE its log: `torch.clamp(mel, min=1e-5)` +// (vocoder.py:515). Named so it can be pinned, because it sets the floor of the +// log-mel fed to the bwe_generator and REAL SILENCE reaches it in production. +// +// It is NOT a member of the invisible-constant class described in +// ltx2_video_vae.h, and the line here that said it was is corrected rather than +// carried: the ORDINARY BWE arm's mel_basis is non-negative and well-scaled and +// its raw minimum is ~4.4e-3, so that arm alone cannot move under a mutation. +// "ltx2 vae: the BWE mel log clamp is gated where it actually binds" attenuates +// mel_basis until every bin saturates the floor, and against it 1e-5 -> 1e-8 REDS +// at max|diff| = 0.144965 versus the 5e-6 band. The pin below stays anyway, for +// what no golden can see: a regeneration that moves the constant and the expected +// tensors together. +inline constexpr double kLtx2BweMelLogClamp = 1e-5; + +struct Ltx2VocoderBweConfig { + Ltx2VocoderConfig vocoder; // prefix defaults to `vocoder.` + Ltx2VocoderConfig bwe_generator; // prefix defaults to `bwe_generator.` + int64_t filter_length = 1024; + int64_t hop_length = 256; + int64_t win_length = 1024; + int64_t n_mel_channels = 64; + int64_t input_sampling_rate = 24000; + int64_t output_sampling_rate = 48000; + std::string prefix; +}; + +// The full BWE chain: vocoder -> pad to a hop multiple -> causal STFT log-mel -> +// bwe_generator residual -> + hann-sinc resampled skip -> clamp -> trim. Returns +// [2, samples] at `output_sampling_rate`. +std::vector Ltx2VocoderWithBweForward(const Ltx2VocoderBweConfig& config, + const Ltx2VaeWeights& weights, + const std::vector& mel, int64_t channels, + int64_t frames, int64_t mel_bins, + int64_t* out_samples); + +// --------------------------------------------------------------------------- +// The two COMPUTED filters. Neither is in any checkpoint; both are gated +// directly, because a wrong window makes every activation wrong at once and the +// resulting mismatch is impossible to localize. +// --------------------------------------------------------------------------- + +// kaiser_sinc_filter1d (vocoder.py:52-70). This is BigVGAN's filter, identical to +// the one the MiniMax-H3 audio VAE already ports, so it DELEGATES to that shared +// implementation rather than standing up a second copy; the golden proves the +// shared code also matches LTX's upstream. +std::vector Ltx2KaiserSincFilter1d(double cutoff, double half_width, int64_t kernel_size); + +// The BWE resampler's HANN-windowed sinc (vocoder.py:116-128) — a different +// window from the kaiser one above, equivalent to torchaudio's `resample`. Fills +// the geometry the forward pass needs; any out-pointer may be null. +std::vector Ltx2HannSincResampleFilter1d(int64_t ratio, int64_t* kernel_size, int64_t* pad, + int64_t* pad_left, int64_t* pad_right); + +} // namespace vllm diff --git a/include/vllm/model_executor/models/ltx2_audio_vae_encoder.h b/include/vllm/model_executor/models/ltx2_audio_vae_encoder.h new file mode 100644 index 000000000..7e88063e0 --- /dev/null +++ b/include/vllm/model_executor/models/ltx2_audio_vae_encoder.h @@ -0,0 +1,186 @@ +// LTX-2.5 AUDIO VAE — the ENCODER half and its mel front-end, which phase L4 +// recorded as owed. +// +// Reference AUDIO conditioning ("dub it") needs a waveform turned into latents, +// and that is two stages, not one: `AudioProcessor.waveform_to_mel` builds a +// log-mel spectrogram, then `AudioEncoder` compresses it +// (`encode_audio`, audio_vae.py:249-274). +// +// ─── WHAT THIS IS A PORT OF (file:line on BOTH sides) ──────────────────────── +// Upstream root: Lightricks/LTX-2 @ fd4ded7f, packages/ltx-core/src/ltx_core/ +// OURS <- UPSTREAM +// Ltx2AudioEncoderForward <- model/audio_vae/audio_vae.py:190-246 +// (the downsampling path) <- model/audio_vae/downsample.py:60-110 +// (Downsample) <- model/audio_vae/downsample.py:11-57 +// (mid block) <- model/audio_vae/audio_vae.py:22-57 +// (ResnetBlock / AttnBlock) <- model/audio_vae/resnet.py:115-176, +// model/audio_vae/attention.py:16-55 +// (the audio patchifier) <- components/patchifiers.py:287-305 +// (per-channel statistics) <- model/audio_vae/ops.py:58-75 +// Ltx2WaveformToLogMel <- model/audio_vae/ops.py:8-55 +// Ltx2SlaneyMelFilterbank <- torchaudio.functional.melscale_fbanks +// (reached from ops.py:20-34) +// Ltx2AudioEncoderConfig <- model/audio_vae/model_configurator.py:144-182 +// +// Every primitive is SHARED with the decoder: this file's implementation is +// compiled into ltx2_audio_vae.cpp, so `Conv2d`, `ApplyCausalPadding`, +// `PixelNorm`, `ResnetBlock` and `AttnBlock` are the exact functions the decoder +// goldens already hold. Only `Downsample` is new, because the decoder has no +// strided convolution. +// +// ─── THE FOUR THINGS THAT FAIL SILENTLY ────────────────────────────────────── +// * `Downsample`'s PRE-PAD IS ASYMMETRIC AND ITS CONV IS STRIDED +// (downsample.py:34-52). On the shipped `height` axis the pad is +// `(left, right, top, bottom) = (0, 1, 2, 0)` — TWO rows of history on the +// time axis, ONE column on the mel axis, and nothing on the other side of +// either. Padding it symmetrically shifts the whole sampling lattice by half +// a bin and produces a plausible latent, not an error. +// * THE `attn` LIST IS BUILT INSIDE THE RES-BLOCK LOOP but `curr_res` is halved +// OUTSIDE it (downsample.py:87-107), so a level either has `num_res_blocks` +// attention blocks or none, and the resolution a level is tested at is the one +// it ENTERS with, not the one it leaves with. +// * `in_ch_mult` IS `(1, *ch_mult)`, so level i reads `ch * ch_mult[i-1]` and +// writes `ch * ch_mult[i]` (downsample.py:78-85). Reading `ch_mult[i]` on both +// sides builds an encoder whose first level is already too wide. +// * THE PATCHIFIER FLATTENS `b c t f -> b t (c f)` (patchifiers.py:301-304), so +// the per-channel statistics are indexed by `c * mel_bins + f`, NOT by `c`. +// This is the same packing the decoder's `_denormalize_latents` undoes. +// +// ─── THE MEL FRONT-END IS A ONE-SIDED REFERENCE, AND THAT IS RECORDED ──────── +// diffusers has NO counterpart: there is no `torchaudio` import anywhere in +// `src/diffusers`, `AutoencoderKLLTX2Audio.encode` is never called from +// `pipelines/ltx2/*`, and audio conditioning enters diffusers only as a +// pre-computed latent. So `slaney` mel scale, `slaney` normalization, the +// centered reflect-padded STFT and `power=1.0` are attested by `ltx_core` alone. +// Everything else in this file is corroborated by both implementations. +#pragma once + +#include +#include +#include + +#include "vllm/model_executor/models/ltx2_audio_vae.h" + +namespace vllm { + +// --------------------------------------------------------------------------- +// AudioEncoder (audio_vae.py:60-246) +// --------------------------------------------------------------------------- + +struct Ltx2AudioEncoderConfig { + // Names and defaults mirror AudioEncoderConfigurator.from_metadata + // (audio_vae/model_configurator.py:162-182). + int64_t ch = 128; + int64_t in_channels = 2; + std::vector ch_mult = {1, 2, 4}; + int64_t num_res_blocks = 2; + // The SHIPPED LTX-2.5 checkpoint sets this EMPTY, so no level carries + // attention; the configurator's `{8, 16, 32}` is only the fallback when the + // checkpoint omits the key (model_configurator.py:166). + std::vector attn_resolutions = {8, 16, 32}; + int64_t resolution = 256; + int64_t z_channels = 8; + // `double_z` doubles conv_out's width; only the FIRST half is ever used + // (audio_vae.py:237), so a checkpoint trained with double_z and read without it + // still produces a latent — of the wrong channels. + bool double_z = true; + bool resamp_with_conv = true; + // The shipped checkpoint sets this FALSE (its `mid_block_add_attention`), unlike + // the decoder arm L4 gated, which ships it true. + bool mid_block_add_attention = true; + Ltx2NormType norm_type = Ltx2NormType::kPixel; + Ltx2CausalityAxis causality_axis = Ltx2CausalityAxis::kHeight; + // Only read on the GroupNorm arm; PixelNorm is parameter-free. Neither is + // reachable from a checkpoint: `build_normalization_layer` passes `eps=1e-6` as + // a LITERAL and forwards its own `num_groups` keyword, whose default is 32 + // (normalization.py:44, 56), and no audio_vae call site passes `num_groups` at + // all — so they are fields only so the gate can pin them. + // + // `norm_type = kGroup` is `AudioEncoder.__init__`'s declared default + // (audio_vae.py:82), but it is NOT what pure defaults give you: the paired + // default is `causality_axis = WIDTH` (audio_vae.py:83), and `ResnetBlock` + // refuses GroupNorm on any causal axis with + // `ValueError: Causal ResnetBlock with GroupNorm is not supported` + // (resnet.py:130-131) — verified by construction against the pinned upstream. + // So a group-norm checkpoint is one that declares `causality_axis: none` + // alongside it, which is legal and is what the group-norm golden in + // test_ltx2_vae.cpp runs. That arm is what stopped this eps from being a + // constant no arm ever read. + int64_t num_groups = 32; + double norm_eps = 1e-6; + // Reached through `build_normalization_layer`, which passes eps=1e-6 + // (normalization.py:58) — the SAME value the audio decoder gets and NOT the + // video VAE's bare 1e-8. + double pixel_norm_eps = 1e-6; + std::string prefix; + + int64_t num_resolutions() const { return static_cast(ch_mult.size()); } +}; + +// AudioEncoder.forward at batch 1 (audio_vae.py:190-203). `spectrogram` is +// [in_channels, frames, mel_bins] channel-major — upstream's +// (batch, channels, TIME, frequency), which is why dim 2 is the causal axis on +// the shipped `height` setting. +// +// Returns the NORMALIZED latent means, [z_channels, frames', mel_bins'], +// i.e. exactly what `Ltx2AudioDecoderForward` de-normalizes on the way back. +Ltx2AudioSpectrogram Ltx2AudioEncoderForward(const Ltx2AudioEncoderConfig& config, + const Ltx2VaeWeights& weights, + const std::vector& spectrogram, + int64_t channels, int64_t frames, int64_t mel_bins); + +// --------------------------------------------------------------------------- +// AudioProcessor (audio_vae/ops.py:8-55) — the mel front-end +// --------------------------------------------------------------------------- + +// `torch.log(torch.clamp(mel, min=1e-5))` (ops.py:52). A member of the +// invisible-constant class and the one that BINDS IN PRODUCTION: real silence +// drives the linear mel to 0 and this constant alone decides the value the +// encoder then sees (log(1e-5) = -11.5129...). A reduced-dimension golden built +// from a well-scaled random waveform never reaches it, so the gate holds it with +// a source-anchored assertion AND a golden arm whose input is actual silence. +inline constexpr double kLtx2AudioMelLogClamp = 1e-5; + +struct Ltx2AudioProcessorConfig { + // Every field is passed straight to `torchaudio.transforms.MelSpectrogram` + // (ops.py:20-34). `win_length` is not a separate knob upstream — it is + // `n_fft` — and `f_min` / `f_max` are hardcoded to 0 and `sample_rate / 2` + // rather than read from the checkpoint's `preprocessing.mel` block, which is a + // real (if currently harmless) upstream inconsistency: for the shipped + // 16 kHz / fmax 8000 config the two agree. + int64_t target_sample_rate = 16000; + int64_t mel_bins = 64; + int64_t mel_hop_length = 160; + int64_t n_fft = 1024; +}; + +// `melscale_fbanks(n_freqs, f_min, f_max, n_mels, sample_rate, norm="slaney", +// mel_scale="slaney")`, returned as [n_mels, n_freqs] — TRANSPOSED relative to +// torchaudio's [n_freqs, n_mels], because every consumer here contracts over the +// frequency axis. Exposed on its own because a wrong filterbank makes every mel +// bin wrong at once and the resulting mismatch is impossible to localize. +// +// `sample_rate / 2` is computed with INTEGER division, mirroring torchaudio's +// `torch.linspace(0, sample_rate // 2, n_freqs)`. +std::vector Ltx2SlaneyMelFilterbank(int64_t n_freqs, double f_min, double f_max, + int64_t n_mels, int64_t sample_rate); + +// `AudioProcessor.waveform_to_mel` (ops.py:44-55) at any channel count. +// `waveform` is [channels, samples], channel-major, at `sampling_rate`. +// +// A SAMPLE RATE THAT DOES NOT MATCH `target_sample_rate` IS REFUSED BY NAME. +// Upstream resamples with `torchaudio.functional.resample` (ops.py:40), a +// polyphase kaiser resampler for an arbitrary rational ratio; this project ports +// only the integer-ratio hann-sinc variant the BWE stage needs +// (Ltx2HannSincResampleFilter1d). Refusing is deliberate: silently treating +// 44.1 kHz samples as 16 kHz produces audio conditioning that is pitched and +// time-scaled wrong while every shape still checks out. +// +// Returns [channels, frames, mel_bins] channel-major — the layout +// `Ltx2AudioEncoderForward` takes, which is upstream's final +// `permute(0, 1, 3, 2)` (ops.py:55). +std::vector Ltx2WaveformToLogMel(const Ltx2AudioProcessorConfig& config, + const std::vector& waveform, int64_t channels, + int64_t samples, int64_t sampling_rate, int64_t* out_frames); + +} // namespace vllm diff --git a/include/vllm/model_executor/models/ltx2_conditioning.h b/include/vllm/model_executor/models/ltx2_conditioning.h new file mode 100644 index 000000000..c1308458d --- /dev/null +++ b/include/vllm/model_executor/models/ltx2_conditioning.h @@ -0,0 +1,127 @@ +// LTX-2.5 CONDITIONING ITEMS — what the VAE encoders' output is FOR. +// +// An encoded image, keyframe, reference video or reference audio is not +// conditioning yet: it becomes conditioning when its tokens are placed into the +// denoise state, with the right RoPE positions and the right denoise mask. That +// placement is upstream's `ConditioningItem.apply_to` family, and this is the +// port of it. +// +// ─── WHAT THIS IS A PORT OF (file:line on BOTH sides) ──────────────────────── +// Upstream root: Lightricks/LTX-2 @ fd4ded7f, packages/ltx-core/src/ltx_core/ +// OURS <- UPSTREAM +// Ltx2CreateVideoLatentState <- tools.py:139-186 (VideoLatentTools) +// Ltx2CreateAudioLatentState <- tools.py:246-279 (AudioLatentTools) +// Ltx2ConditionVideoByLatentIndex <- conditioning/types/latent_cond.py:22-43 +// Ltx2ConditionVideoByKeyframe <- conditioning/types/keyframe_cond.py:39-90 +// Ltx2ConditionVideoByReference <- conditioning/types/reference_video_cond.py:46-108 +// Ltx2ConditionAudioByReference <- conditioning/types/reference_audio_cond.py:33-65 +// +// The patchifiers and the coordinate maps are NOT re-implemented here: they are +// `Ltx2VideoPatchify`, `Ltx2VideoPatchBounds`, `Ltx2PixelCoords`, +// `Ltx2AudioPatchify` and `Ltx2AudioPatchTimings` from ltx2_pipeline.h, already +// gated by phase L5. +// +// ─── THE FOUR THINGS THAT FAIL SILENTLY ────────────────────────────────────── +// * `Ltx2ConditionVideoByLatentIndex` LEAVES THE NOISY TENSOR ALONE. Only +// `clean_latent` and `denoise_mask` are written (latent_cond.py:38-39); the +// noiser then composes them as +// `lerp(clean, lerp(latent, noise, ns), mask)` (components/noisers.py:31-34). +// diffusers instead writes the clean tokens into the noisy tensor too +// (pipeline_ltx2_condition.py:1002) and reaches the same answer ONLY at +// `noise_scale == 1`. Copying diffusers here would be a silent divergence at +// every other noise scale. +// * THE DENOISE MASK IS `1 - strength`, NOT `strength`. Mask 0 means "keep the +// clean value"; strength 1 means fully conditioned. Inverting it renders an +// unconditioned clip that looks like the feature not working. +// * A KEYFRAME'S CAUSAL FIX IS DISABLED UNLESS `frame_idx == 0` +// (keyframe_cond.py:45-50). The fix exists because the causal encoder's FIRST +// latent frame spans one pixel frame; a keyframe placed later has no such +// frame, so applying the fix anyway shifts it by up to `factors.time - 1`. +// * THE REFERENCE ITEM'S TWO SCALE FACTORS EACH GUARD ON `!= 1` +// (reference_video_cond.py:74, 80). Applying the temporal translation with +// `temporal_scale_factor == 1` subtracts zero — harmless — but applying the +// CLAMP unconditionally is not, and applying the spatial multiply when it is 1 +// is a no-op that hides a wrong branch. +#pragma once + +#include +#include + +#include "vllm/model_executor/models/ltx2_pipeline.h" +#include "vllm/model_executor/models/ltx2_upsampler.h" // Ltx2LatentVolume + +namespace vllm { + +// LatentState (types.py:251-287) at batch 1, carrying only the fields a +// conditioning item touches. `attention_mask` is deliberately absent: every item +// ported here passes `attention_mask=None` and there is no pre-existing mask, so +// `update_attention_mask` returns None (conditioning/mask_utils.py:110-140). An +// item that DID need one would have to grow this struct rather than silently +// dropping it, which is why the omission is stated here. +struct Ltx2LatentState { + int64_t tokens = 0; + int64_t width = 0; // channels per token + int64_t pos_dims = 0; // 3 for video (frame, height, width), 1 for audio + std::vector latent; // [tokens, width] — the NOISY tensor + std::vector clean; // [tokens, width] + std::vector mask; // [tokens] — the denoise mask, 1 = fully denoised + std::vector positions; // [pos_dims, tokens, 2] — [start, end) +}; + +// VideoLatentTools.create_initial_state (tools.py:139-186) at batch 1. +// `initial_latent` may be null, which is upstream's zero fill. `out_keyframes_mask` +// receives `_first_frame_keyframes_mask` (tools.py:186-197) — the target's FIRST +// latent frame marked, unconditionally, because the causal encoder makes it span a +// single pixel frame. May be null. +Ltx2LatentState Ltx2CreateVideoLatentState(const Ltx2VideoLatentShape& shape, int64_t patch_size, + const Ltx2ScaleFactors& factors, double fps, + bool causal_fix, const float* initial_latent = nullptr, + std::vector* out_keyframes_mask = nullptr); + +// AudioLatentTools.create_initial_state (tools.py:246-279) at batch 1. The audio +// positions are the patch grid bounds THEMSELVES, in seconds — there is no +// `get_pixel_coords` and no division by fps on this side. +Ltx2LatentState Ltx2CreateAudioLatentState(const Ltx2AudioLatentShape& shape, + const Ltx2AudioPatchifierParams& params, + const float* initial_latent = nullptr); + +// VideoConditionByLatentIndex (latent_cond.py:22-43): REPLACE the clean tokens of +// latent frame `latent_idx` with an encoded image or clip. This is the +// image-to-video / first-frame path, and `latent_idx = 0` is what +// `combined_image_conditionings` selects for `frame_idx == 0` +// (ltx-pipelines/utils/helpers.py:295-300). +// +// `conditioning` must match the target's batch, channels, height and width; +// upstream raises `ConditioningError` otherwise and so does this. +void Ltx2ConditionVideoByLatentIndex(Ltx2LatentState* state, const Ltx2VideoLatentShape& target, + int64_t patch_size, const Ltx2LatentVolume& conditioning, + double strength, int64_t latent_idx); + +// VideoConditionByKeyframeIndex (keyframe_cond.py:39-90): APPEND keyframe tokens +// whose positions are offset to pixel frame `frame_idx`. The appended tokens are +// zero in the noisy tensor and the keyframe in the clean one. +void Ltx2ConditionVideoByKeyframe(Ltx2LatentState* state, const Ltx2LatentVolume& keyframes, + int64_t patch_size, const Ltx2ScaleFactors& factors, double fps, + int64_t frame_idx, double strength, int64_t num_pixel_frames, + bool causal_fix); + +// VideoConditionByReferenceLatent (reference_video_cond.py:46-108): APPEND a +// reference video's tokens, translated into the target's frame. `downscale_factor` +// is the target/reference SPATIAL ratio and `temporal_scale_factor` the temporal +// one; both must match what the IC-LoRA was trained with, which upstream stores in +// the LoRA metadata this project does not yet read — so a caller that guesses them +// gets a plausible, wrongly-placed reference. +void Ltx2ConditionVideoByReference(Ltx2LatentState* state, const Ltx2LatentVolume& reference, + int64_t patch_size, const Ltx2ScaleFactors& factors, double fps, + int64_t downscale_factor, int64_t temporal_scale_factor, + double strength, bool causal_fix); + +// AudioConditionByReferenceLatent (reference_audio_cond.py:33-65): APPEND already +// patchified reference-audio tokens with their own timings. Takes the patchified +// form because the caller holds the encoder's `[channels, frames, mel_bins]` output +// and `Ltx2AudioPatchify` is the gated way to flatten it. +void Ltx2ConditionAudioByReference(Ltx2LatentState* state, const std::vector& tokens, + int64_t token_count, int64_t width, + const std::vector& positions, double strength); + +} // namespace vllm diff --git a/include/vllm/model_executor/models/ltx2_connector.h b/include/vllm/model_executor/models/ltx2_connector.h new file mode 100644 index 000000000..9d0eeda47 --- /dev/null +++ b/include/vllm/model_executor/models/ltx2_connector.h @@ -0,0 +1,202 @@ +// LTX-2.5 EMBEDDINGS CONNECTOR — the 1-D transformer between the Gemma-4 text +// encoder and the DiT's cross-attention. +// +// Row: MODEL-DIFFUSION-ltx-2-5-ltx2-video-transformer-3d-model. Spec: +// .agents/specs/ltx-2-5.md (phase L5). Issue #435. +// +// This brick sits BETWEEN two phases and was orphaned between them: L3 stops at +// the text encoder's output and L2 starts at the DiT's context input, so nothing +// owned the module that turns one into the other. It lands here. +// +// ─── WHAT THIS IS A PORT OF (file:line on BOTH sides) ──────────────────────── +// Upstream: Lightricks/LTX-2, +// packages/ltx-core/src/ltx_core/text_encoders/gemma/embeddings_connector.py +// OURS <- UPSTREAM +// Ltx2ConnectorForward <- :154-191 (Embeddings1DConnector.forward) +// Ltx2ConnectorBlockForward <- :41-71 (_BasicTransformerBlock1D.forward) +// Ltx2ConnectorReplaceRegisters <- :139-152 +// Ltx2ConnectorConfig <- :95-137 + :194-256 (both configurators) +// +// ─── IT IS BUILT ON THE DiT's OWN PARTS, AND SO IS THIS PORT ───────────────── +// `_BasicTransformerBlock1D` imports the DiT's `Attention`, `FeedForward` and +// RoPE verbatim (:4-11). This TU therefore routes through `vllm::Ltx2Attention`, +// `vllm::Ltx2FeedForward`, `vllm::Ltx2PrecomputeFreqsCis` and +// `vllm::Ltx2ApplyRotaryEmb` from phase L2 rather than re-deriving them — a +// second attention implementation here would be a parallel path that could drift +// from the one the DiT is gated on. +// +// ─── THE FOUR THINGS THAT FAIL SILENTLY ────────────────────────────────────── +// * `learnable_registers` IS A BFLOAT16 PARAMETER (:135-137 constructs it with +// `dtype=torch.bfloat16`). Keeping those values in f32 is WIDER than upstream: +// every register position then carries ~8 extra mantissa bits, the result is +// still finite and still shaped right, and only the padded positions move. +// AGENTS.md names this polarity exactly — a value gate cannot catch a dtype +// that is too wide — so the rounding is explicit here and gated on its own +// golden. +// * THE REGISTER TABLE IS TILED, NOT INDEXED: `repeat(seq_len // num_registers, 1)` +// (:146) walks 0..N-1, 0..N-1, …, so `seq_len` MUST be a multiple of +// `num_learnable_registers` and upstream asserts it (:144). +// * THE MASK IS REPLACED BY ZEROS once registers are substituted (:152). Every +// position is attendable afterwards — including the ones that were padding. A +// port that kept the original mask attends over fewer tokens and produces a +// different, plausible conditioning. +// * THE FINAL `rms_norm` (:189) has NO weight and torch's default eps of 1e-6 +// (utils.py:7-12). It is applied AFTER the last block, on top of the two the +// block already applies. +// +// ─── DTYPE ─────────────────────────────────────────────────────────────────── +// f32 for the activations, which is the parity dtype of this gate and matches +// what upstream computes when handed an f32 input (there is no per-layer dtype in +// the module). The ONE deliberate narrowing is `learnable_registers`, above, +// because upstream stores it narrow. The production bf16 arm is phase L6. +#pragma once + +#include +#include +#include + +#include "vllm/model_executor/models/ltx2.h" // Ltx2RopeType and the DiT's parts +#include "vllm/model_executor/models/ltx2_audio_vae.h" // Ltx2VaeWeights, the shared bag + +namespace vllm { + +// utils.py:7-12 — `torch.nn.functional.rms_norm`'s eps as `rms_norm` passes it. +// The connector uses the WEIGHTLESS form, so this is the only stabilizer in the +// residual path. NOT a member of the invisible-constant class, however near-zero +// the fixture's rows are: `rms_norm` adds the epsilon to the MEAN SQUARE, not to +// a row minimum, so it perturbs every row it normalizes. At the class's own 100x +// bar (1e-6 -> 1e-4) it REDS 5 of the arms in "ltx2 the Embeddings1DConnector +// reproduces upstream on every arm" — Split 0.0558581, Interleaved 0.104284, +// Float64 0.140343, NoRegisters 0.000542641, GatedNoBias 0.0892045. +// It is pinned as well as gated, in test_ltx2_pipeline.cpp, case "the constants +// the headers call pinned are actually pinned", which compares this against +// upstream's own `rms_norm` signature default rather than a retyped literal — +// the one check a regenerated golden cannot satisfy by moving with it. +inline constexpr double kLtx2ConnectorRmsNormEps = 1e-6; + +// Embeddings1DConnector.__init__ defaults (:95-108), which are also both +// configurators' `transformer_config.get` fallbacks (:199-218, :227-255). +struct Ltx2ConnectorConfig { + int64_t attention_head_dim = 128; + int64_t num_attention_heads = 30; + int64_t num_layers = 2; + double positional_embedding_theta = 10000.0; + // `[1]` upstream (:114-116). The RoPE grid is 1-D: one position axis over the + // token index. + std::vector positional_embedding_max_pos = {1}; + // 0 disables the substitution entirely, which is upstream's `None` (:103, :167). + int64_t num_learnable_registers = 128; + Ltx2RopeType rope_type = Ltx2RopeType::kSplit; + bool double_precision_rope = false; + bool apply_gated_attention = false; + bool ff_bias = true; + std::string prefix; + + int64_t inner_dim() const { return num_attention_heads * attention_head_dim; } +}; + +// The parameter contract, in `named_parameters()` order: `learnable_registers` +// first (a bare nn.Parameter precedes every submodule in torch's ordering), then +// each block's attention and feed-forward. +struct Ltx2ConnectorTensorSpec { + std::string name; + std::vector shape; +}; +std::vector EnumerateLtx2ConnectorTensors( + const Ltx2ConnectorConfig& config); + +// The result of a connector forward: the conditioning the DiT's cross-attention +// consumes, plus the mask it must be paired with. Upstream returns both (:191) +// and a caller that keeps its ORIGINAL mask instead of this one re-masks +// positions the connector has just filled with registers. +struct Ltx2ConnectorOutput { + std::vector hidden_states; // [batch, seq, inner_dim] + std::vector mask; // [batch, 1, 1, seq], additive +}; + +// _replace_padded_with_learnable_registers (:139-152), exposed on its own because +// it is where the bf16 narrowing lives and a defect there is otherwise absorbed +// by two transformer blocks before anyone sees it. +Ltx2ConnectorOutput Ltx2ConnectorReplaceRegisters(const Ltx2ConnectorConfig& config, + const Ltx2VaeWeights& weights, + const float* hidden_states, + const float* additive_attention_mask, + int64_t batch, int64_t seq); + +// Embeddings1DConnector.forward (:154-191). `additive_attention_mask` is +// [batch, 1, 1, seq] with 0 for a kept token and -finfo(f32).max for a padded +// one — the form `_prepare_attention_mask` produces (transformer_args.py:199-206) +// — and may be null only when the config carries no registers, which mirrors +// upstream's own unconditional dereference at :168-170. +Ltx2ConnectorOutput Ltx2ConnectorForward(const Ltx2ConnectorConfig& config, + const Ltx2VaeWeights& weights, + const float* hidden_states, + const float* additive_attention_mask, int64_t batch, + int64_t seq); + +// ─── THE PROCESSOR AROUND THE TWO CONNECTORS ───────────────────────────────── +// +// Upstream: `EmbeddingsProcessor.create_embeddings` +// (text_encoders/gemma/embeddings_processor.py:70-95). It is a SEPARATE module +// from the connector and it does three things the connector does not, each of +// which changes the conditioning silently when it is skipped: +// +// * IT RIGHT-PAD-SORTS THE FEATURES FIRST (:82-84, `_compute_right_pad_order` / +// `_apply_right_pad_order`). Upstream's own comment is "Connectors expect +// right-padded input ([valid, pad])", because the register table is indexed +// by ABSOLUTE position (`s % num_registers`) rather than by which positions +// were padded. A LEFT-padded batch handed straight to the connector puts +// real tokens where registers belong and registers where tokens belong, and +// the result is finite, correctly shaped and conditioned on nothing. +// * IT MULTIPLIES THE VIDEO ENCODING BY A BINARY MASK (:86-87) and does NOT do +// the same to the audio one (:91-93) — an asymmetry that reads like an +// oversight, is upstream's behaviour, and is mirrored rather than tidied. +// * IT RETURNS THE BINARY MASK THE DiT CONSUMES (:89), which is the connector's +// OUTPUT mask, not the caller's input one. That mask is derived with +// `encoded_mask < 0.000001` (:46-48), and BOTH values an additive mask can +// hold — 0.0 and -finfo(f32).max — satisfy it, so it is all ones for every +// input either reference can produce. See the implementation: the direction +// is surprising, `diffusers` writes the identical comparison, and the +// multiply is consequently an identity on every reachable path. +// +// THE TWO REFERENCES DISAGREE ABOUT WHERE THE SORT LIVES, and the disagreement is +// recorded rather than resolved by preference. `diffusers`' +// `LTX2ConnectorTransformer1d.forward` folds the sort INTO the connector +// (`src/diffusers/pipelines/ltx2/connectors.py`, the `torch.argsort(1 - +// binary_attn_mask, stable=True)` branch) and its comment claims that matches +// "the original LTX implementation" — which is true only because `ltx_core` +// sorts one level up, in the processor. The two compose to the same function; +// they differ in which module owns it. This port follows `ltx_core`, so +// `Ltx2ConnectorForward` stays a faithful port of `Embeddings1DConnector` and +// the sort lands here, at the processor. +struct Ltx2ConnectorEmbeddings { + std::vector video; // [batch, seq, video inner_dim] + std::vector audio; // [batch, seq, audio inner_dim] + // `binary_mask.squeeze(-1)` (:89): [batch, seq], 1.0 for a position the DiT's + // cross-attention may attend to and 0.0 for one it may not. With registers + // enabled every position is attendable, which is the whole point of them. + std::vector mask; +}; + +// `additive_attention_mask` is [batch, seq] with 0 for a kept token and +// -finfo(f32).max for a padded one, and it is required: it is what decides which +// positions become registers. +// +// NO NUMERIC ORACLE, and that is owed rather than overlooked. `Ltx2ConnectorForward` +// — the connector this wraps — is gated on five arms against EXECUTED upstream +// (`gen-ltx2-pipeline-goldens.py` section 10). This wrapper is not. Its own tests +// are PROPERTY tests: padding-side agnosticism compares two of OUR OWN calls, so +// a defect present in both arms cancels, and the binary-mask case pins a polarity +// rather than a value. A reviewer's mutations at the layer above — video scaled +// x1.5, conditioning rows reversed — passed every assertion in the suite. +// Upstream's counterpart is `EmbeddingsProcessor.create_embeddings` +// (embeddings_processor.py:70-95), reachable from the same generator that already +// executes the connector; see the closure note on `Ltx2ConditioningTrace` +// (multimodal/ltx2_video.h). +Ltx2ConnectorEmbeddings Ltx2ConnectorCreateEmbeddings( + const Ltx2ConnectorConfig& video_config, const Ltx2VaeWeights& video_weights, + const float* video_features, const Ltx2ConnectorConfig& audio_config, + const Ltx2VaeWeights& audio_weights, const float* audio_features, + const float* additive_attention_mask, int64_t batch, int64_t seq); + +} // namespace vllm diff --git a/include/vllm/model_executor/models/ltx2_device.h b/include/vllm/model_executor/models/ltx2_device.h new file mode 100644 index 000000000..484833f47 --- /dev/null +++ b/include/vllm/model_executor/models/ltx2_device.h @@ -0,0 +1,142 @@ +// LTX-2.5 DiT — the DEVICE-RESIDENT forward, and its kernel seam (phase L8). +// +// Row: MODEL-DIFFUSION-LTX25. Spec: .agents/specs/ltx-2-5.md phase L8. Issue #435. +// +// ─── WHAT THIS CLOSES ──────────────────────────────────────────────────────── +// +// Phase L7 wired LTX-2.5 through `vllm::multimodal::VideoEngine` and then had to +// REFUSE `device = 1` by name, because the two halves of the device story did not +// meet: `Ltx2DitForward` (ltx2.h:433-434) accepts only `vt::DType::kF32` and +// computes into host `std::vector`, while `Ltx2StreamDitToDevice` +// (ltx2_loader.h:266-272) stages bf16 onto the device and refuses `widen_to_f32` +// by design. This header is the half that was missing: the SAME graph with every +// activation in device memory and the stream in the checkpoint's own dtype. +// +// ─── DTYPE POLARITY, STATED FIRST ──────────────────────────────────────────── +// +// Upstream resolves ONE model dtype and every layer inherits it — `model.py` has +// no per-layer dtype at all. So the device stream is bf16, which is what +// `Ltx2StreamDitToDevice` already puts on the device and what the shipped +// checkpoints dequantize to. `kF32` is accepted too, and ONLY because it is the +// PARITY dtype of the L2 gate: the f32 arm is what lets this forward be compared +// against `tests/vllm/models/ltx2_goldens.inc` — the same goldens the CPU forward +// is held to — at f32 round-off rather than at a bf16 band. It is a gate arm, not +// a production one, and nothing widens a bf16 load to reach it. +// +// The `scale_shift_table` family stays F32 on BOTH arms. That is not a widening: +// the CHECKPOINT stores those tensors F32 (ltx2_loader.h:64-66), and narrowing a +// tensor the file itself widened would be the dtype rule applied backwards. They +// are [2, dim] / [9, dim] / [5, dim] tables — a few kilobytes against a 21 GB +// model — so the f32 read costs nothing per token. +// +// The RoPE cos/sin tables are also f32 on both arms, for the reason +// `Ltx2FreqGrid` already documents (ltx2.h:253-260): the two frequency ladders +// differ only in the last f32 ulps of every angle, and the audio ladder +// multiplies those up by four orders of magnitude before RoPE takes their cosine. +// They are [batch, heads, tokens, head_dim/2] and are built ONCE per forward on +// the host, exactly as MiniMax-H3 builds its own rope cache +// (minimax_h3_device.cpp:193-210). +// +// ─── THE KERNEL SEAM IS NEXT DOOR ──────────────────────────────────────────── +// +// `vt::OpId::kLtx2` and the seven-entry table behind it live in +// `ltx2_kernels.h`, which includes nothing but `vt/`. That split is not tidiness: +// `src/vt/cuda/cuda_ltx2.cu` includes the table, and this header pulls in +// `ltx2.h` and with it `nlohmann/json.hpp`, which nvcc has no business parsing. +// The table's contents, and why it is only seven ops, are documented there. +// +// ─── NUMERICS: CLOSE, NOT BIT-IDENTICAL ────────────────────────────────────── +// +// The f32 device arm is NOT bit-identical to `Ltx2DitForward` and does not claim +// to be. The divergence is in the SHARED ops, not in these kernels: `vt::RmsNorm` +// and `vt::LayerNorm` reduce in f32 where `RmsNormRows` / `LayerNormRows` +// accumulate in double, and MatmulBT and the attention ops use their own +// accumulation orders. f32 is what upstream torch does, so the device path is +// arguably the closer mirror. It is held to the SAME upstream goldens. +#pragma once + +#include +#include +#include +#include +#include + +#include "vllm/model_executor/models/ltx2.h" +#include "vllm/model_executor/models/ltx2_kernels.h" +#include "vt/backend.h" +#include "vt/device.h" +#include "vt/dtype.h" +#include "vt/tensor.h" + +namespace vllm { + +// --------------------------------------------------------------------------- +// Staging +// --------------------------------------------------------------------------- + +// A DiT whose every weight lives in device memory, plus the allocations that own +// it. `weights` views into `storage`; dropping this struct frees the model. +// +// This is the TEST/second path onto the device. The production path for a real +// checkpoint is `Ltx2StreamDitToDevice` (ltx2_loader.h:266-272), which never +// materializes the whole model on the host at all; this one starts from an +// already-host-resident weight set, which is exactly what the golden fixtures +// build. Both produce `Ltx2DitWeights` whose views are device-resident, and +// `Ltx2DitForwardDevice` cannot tell them apart. +struct Ltx2DitDeviceWeights { + std::vector> storage; + std::map views; + Ltx2DitWeights weights; +}; + +// Upload every tensor of `host` (keyed by its upstream parameter name, exactly +// what `Ltx2LoadDitFromSafetensors` and the golden fixtures produce) to `queue`'s +// device, converting to `stream_dtype` where the dtype policy says so. +// +// THE POLICY, and it is the whole reason this takes a dtype at all: +// * `stream_dtype == kBF16` — every projection weight and bias becomes bf16, +// which is what upstream's one-model-dtype resolution gives and what +// `Ltx2StreamDitToDevice` already puts on the device. +// * `stream_dtype == kF32` — everything stays f32. The PARITY arm; see this +// header's DTYPE note for why it exists and what it is not. +// In BOTH cases the `scale_shift_table` family stays F32, because the checkpoint +// stores it F32. `Ltx2DitTensorIsTable` is the predicate, exposed so a caller can +// assert the split rather than re-derive it. +bool Ltx2DitTensorIsTable(const std::string& name); + +Ltx2DitDeviceWeights Ltx2StageDitWeightsToDevice(vt::Queue& queue, const Ltx2DitParams& params, + const std::map& host, + vt::DType stream_dtype = vt::DType::kBF16); + +// --------------------------------------------------------------------------- +// The forward +// --------------------------------------------------------------------------- + +// `Ltx2DitForward` (ltx2.h:443-446) with every activation in device memory. +// +// `weights` must be DEVICE-RESIDENT on `queue`'s device — from +// `Ltx2StreamDitToDevice` or `Ltx2StageDitWeightsToDevice`. Handing it host +// views on a CUDA queue is the failure mode that reads as all-zeros rather than +// as an error, so the forward checks the device of every weight it binds. +// +// Inputs stay HOST pointers and outputs stay host `std::vector`, exactly +// as `MiniMaxH3DitForwardDevice` does (minimax_h3.h:1794-1798): the forward +// uploads what it needs and downloads the two output heads. That is what makes +// the device arm comparable against the SAME goldens as the CPU arm rather than +// against a second, differently-shaped harness. +// +// `compute_dtype` is the STREAM dtype and must be kBF16 (production) or kF32 +// (parity). It must agree with how the weights were staged: a bf16 stream over +// f32-staged weights compares a different model, not a different dtype policy. +// +// The prompt-K/V cache (ltx2.h:335-350) is NOT ported to this path yet and a +// non-null `cache` is REFUSED BY NAME rather than silently ignored — an ignored +// cache would recompute correctly and quietly lose the optimization, which is +// exactly the kind of divergence that is discovered a phase later. +Ltx2DitOutputs Ltx2DitForwardDevice(vt::Queue& queue, const Ltx2DitParams& params, + const Ltx2DitWeights& weights, + const Ltx2ModalityInput* video, + const Ltx2ModalityInput* audio, vt::DType compute_dtype, + Ltx2PromptKvCache* cache = nullptr); + +} // namespace vllm diff --git a/include/vllm/model_executor/models/ltx2_duration_head.h b/include/vllm/model_executor/models/ltx2_duration_head.h new file mode 100644 index 000000000..15b6a1efe --- /dev/null +++ b/include/vllm/model_executor/models/ltx2_duration_head.h @@ -0,0 +1,109 @@ +// LTX-2.5 DURATION HEAD — predicts a shot's natural length from the connector +// outputs, so a request need not state `num_frames`. +// +// Row: MODEL-DIFFUSION-ltx-2-5-ltx2-video-transformer-3d-model. Spec: +// .agents/specs/ltx-2-5.md (phase L5). Issue #435. +// +// ─── WHAT THIS IS A PORT OF (file:line on BOTH sides) ──────────────────────── +// Upstream root: Lightricks/LTX-2, packages/ltx-core/src/ltx_core/duration_head/ +// OURS <- UPSTREAM +// Ltx2DurationPredict <- duration_head.py:89-118 (DurationHead.forward) +// Ltx2DurationAttentionPool <- duration_head.py:45-49 (AttentionPooler.forward) +// Ltx2DurationHeadConfig <- duration_head.py:63-71 + model_configurator.py:22-35 +// EnumerateLtx2DurationHeadTensors <- the module's own named_parameters() +// +// ─── THE FOUR THINGS THAT FAIL SILENTLY ────────────────────────────────────── +// * THE OUTPUT IS `exp(mlp_out)`, NOT `mlp_out`. The head is trained in +// LOG-seconds and exponentiates on the way out (duration_head.py:117-118). +// Returning the raw regression gives a number that is finite, positive-ish and +// a completely different duration. +// * THE TWO STREAMS CONCATENATE ALONG THE TOKEN AXIS (`dim=1`, :113), after each +// is projected to the SHARED pooler width. Concatenating along the feature axis +// also type-checks when the widths happen to line up. +// * THE MODALITY EMBEDDING IS ADDED AFTER THE PROJECTION (:109, :111), which is +// what lets the pooler tell the streams apart. Adding it before the projection +// puts it through a different linear map per stream and loses that. The +// embedding is the ONLY thing that tags a stream — see the invariance below. +// * THE POOLER IS `torch.nn.MultiheadAttention`, whose PACKED `in_proj_weight` is +// [3 * E, E] in Q, K, V order. It is CROSS attention here — the queries are the +// learnable tokens and the keys/values are the token stream — so the three +// slices are applied to two different inputs, not one. +// +// ─── AN INVARIANCE, so nobody mistakes it for a gate hole ──────────────────── +// THE CONCAT ORDER IS NOT OBSERVABLE. A mutation that reversed the two streams' +// concatenation left every golden green, and that is correct rather than a weak +// fixture: `AttentionPooler` is cross-attention with no mask and no positional +// encoding over the token axis, so it is PERMUTATION INVARIANT and upstream +// cannot distinguish the orders either. Measured on upstream — a reversed concat +// and a random permutation each move the pooled output by 2.98e-08 (f32 +// reduction-order noise), while giving the audio stream the VIDEO modality +// embedding moves it by 4.80e-03. +// +// The consequence is worth stating plainly: what separates the two streams is the +// modality EMBEDDING, nothing else. Both facts are gated in +// tests/vllm/models/test_ltx2_pipeline.cpp, so if upstream ever gives the pooler +// positional information the invariance assertion fails and this note stops being +// true at the moment it stops being true. +// +// ─── NO MASK, BY CONSTRUCTION ──────────────────────────────────────────────── +// The pooler takes no attention mask because the connector has already replaced +// every padded position with a learnable register and zeroed its mask +// (duration_head.py:15-16, embeddings_connector.py:139-152). This port therefore +// requires the connector's output, not a raw padded batch — a caller that hands +// it padded tokens gets a duration computed over the padding, silently. +// +// ─── DTYPE ─────────────────────────────────────────────────────────────────── +// f32, the parity dtype of this gate. Upstream constructs the head in the +// pipeline's dtype (distilled.py:163-167 passes `self.dtype` = bfloat16), so the +// bf16 arm is owed by phase L6. +#pragma once + +#include +#include +#include + +#include "vllm/model_executor/models/ltx2_audio_vae.h" // Ltx2VaeWeights + +namespace vllm { + +// DurationHead.__init__ defaults (duration_head.py:63-71), which are also +// DurationHeadConfigurator.from_metadata's `config.get` fallbacks +// (model_configurator.py:28-35). The two cross-attention dims mirror the DiT's +// own `cross_attention_dim` / `audio_cross_attention_dim`. +struct Ltx2DurationHeadConfig { + int64_t video_cross_attention_dim = 4096; + int64_t audio_cross_attention_dim = 2048; + int64_t pooler_hidden_dim = 256; + int64_t num_queries = 1; + int64_t num_pooler_heads = 4; + int64_t mlp_hidden = 256; + std::string prefix; +}; + +// The parameter contract, in `named_parameters()` order. Note that the two bare +// `nn.Parameter`s (`video_modality_emb`, `audio_modality_emb`) come FIRST, before +// any submodule — that is torch's own ordering, and the parity suite asserts it. +struct Ltx2DurationHeadTensorSpec { + std::string name; + std::vector shape; +}; +std::vector EnumerateLtx2DurationHeadTensors( + const Ltx2DurationHeadConfig& config); + +// AttentionPooler.forward (duration_head.py:45-49) on its own, so a pooler defect +// localizes instead of arriving as one wrong scalar. `tokens` is +// [batch, token_count, pooler_hidden_dim]; the result is +// [batch, num_queries, pooler_hidden_dim]. +std::vector Ltx2DurationAttentionPool(const Ltx2DurationHeadConfig& config, + const Ltx2VaeWeights& weights, const float* tokens, + int64_t batch, int64_t token_count); + +// DurationHead.forward (duration_head.py:89-118). Either stream may be null, and +// both being null throws exactly as upstream does (:104-105). Returns the +// predicted duration in SECONDS, [batch]. +std::vector Ltx2DurationPredict(const Ltx2DurationHeadConfig& config, + const Ltx2VaeWeights& weights, const float* video_tokens, + int64_t video_token_count, const float* audio_tokens, + int64_t audio_token_count, int64_t batch); + +} // namespace vllm diff --git a/include/vllm/model_executor/models/ltx2_kernels.h b/include/vllm/model_executor/models/ltx2_kernels.h new file mode 100644 index 000000000..3b5af5cf9 --- /dev/null +++ b/include/vllm/model_executor/models/ltx2_kernels.h @@ -0,0 +1,155 @@ +// LTX-2.5 DiT — the DEVICE-FORWARD KERNEL SEAM (vt::OpId::kLtx2), phase L8. +// +// Row: MODEL-DIFFUSION-LTX25. Spec: .agents/specs/ltx-2-5.md phase L8. Issue #435. +// +// This header is DELIBERATELY THIN, and mirrors minimax_h3_device.h for the same +// reason: `src/vt/cuda/cuda_ltx2.cu` includes it, and a backend kernel TU must not +// have to compile the model's own headers. `ltx2_device.h` — which declares the +// forward — pulls in `ltx2.h` and with it `nlohmann/json.hpp`, so the two are +// separated rather than nvcc being asked to parse a JSON library. `ltx2_device.h` +// includes this one; nothing here includes anything but `vt/`. +// +// ─── WHY THE TABLE IS SO SHORT ─────────────────────────────────────────────── +// +// Almost the whole LTX-2.5 DiT forward is already covered by tuned SHARED vt:: +// ops, and the device port reuses them rather than growing a private kernel set: +// +// Linear -> vt::MatmulBT + vt::Add (rank-1 row-broadcast bias) +// rms_norm (no weight) -> vt::RmsNorm with an all-ones weight +// RMSNorm(inner_dim) -> vt::RmsNorm (q_norm / k_norm) +// LayerNorm(affine=False) -> vt::LayerNorm (weight = bias = nullptr) +// gelu(approximate="tanh") -> vt::GeluTanh +// self-attention -> vt::Attention(causal=false) +// cross / biased attention -> vt::AttentionCross +// dtype boundary -> vt::CastBf16 / vt::CastF32 +// +// That leaves exactly SEVEN ops the shared surface does not provide, and every +// one of them is a transcription of a named host helper in ltx2_dit.cpp / +// ltx2.cpp, in the same arithmetic order. +// +// SEAM: one OpProvider entry whose payload is a static kernels-struct of typed +// launchers, mirroring the kMiniMaxH3 precedent. Registered for BOTH kCPU +// (src/vt/cpu/cpu_ltx2.cpp) and kCUDA (src/vt/cuda/cuda_ltx2.cu) — so the device +// forward's STRUCTURE is covered by CPU CI and a GPU is needed to gate the +// KERNELS, not the port. +#pragma once + +#include + +#include "vt/device.h" +#include "vt/dtype.h" + +namespace vllm { + +namespace ltx2 { + +// The seven ops the shared vt:: surface does not express. Every one names the +// host helper it stands in for and the upstream line that helper came from. +// +// DTYPE: arithmetic is ALWAYS f32; only the load/store width varies. A bf16 +// stream therefore rounds on STORE, which is exactly where upstream's cast falls +// — there is no separate rounding pass, and no way for a fused store to drift +// from it. `dtype` below is always the STREAM dtype (kF32 or kBF16) of the +// buffers it applies to; the `float*` parameters are the f32 islands named in +// this header's DTYPE note (the scale-shift tables and the RoPE cos/sin). +struct Ltx2DeviceKernels { + // `AdaValue` (ltx2_dit.cpp:69-80, get_ada_values transformer.py:191-200): + // out[r, c] = table[table_row, c] + modulation[r, mod_index, c] + // `modulation` is the flat [rows, num_params * width] AdaLN projection, so the + // slice for row r lives at r*num_params*width + mod_index*width. Passing + // `num_params` explicitly is deliberate: reading it as anything else silently + // mixes the self-attention, feed-forward and cross-attention groups, which + // renders confidently and wrongly. + // + // `table_row` and `mod_index` differ for the AV cross GATE, where upstream + // hands in `scale_shift_table[4:]` (one row) against a one-parameter timestep + // (transformer.py:214-215): table row 4, modulation index 0. + void (*ada_value)(vt::Queue&, void* out, const float* table, const void* modulation, + int64_t rows, int64_t width, int64_t num_params, int64_t table_row, + int64_t mod_index, vt::DType dtype); + + // The affine tail of `AdaZero` (ltx2_dit.cpp:89-101, PytorchAdaZeroFunction + // ops.py:50-58) and of `ModulateContext` (ltx2_dit.cpp:120-131, + // apply_cross_attention_adaln transformer.py:420-447), IN PLACE: + // x[r, c] = x[r, c] * (1 + scale[r, c]) + shift[r, c] + // `src_row_stride` is the ROW stride of scale/shift in elements. It is `width` + // for the per-token AdaLN values and ZERO for the static [2, dim] prompt table, + // whose single row broadcasts over every token. Passing it explicitly is what + // lets one kernel serve both without a materialized broadcast. + // + // `src_dtype` is SEPARATE from `dtype` and that is the load-bearing part. The + // AdaLN values arrive at the stream dtype (they come out of `ada_value`), but + // the prompt table is F32 — the checkpoint stores it F32 and this port keeps it + // F32. One dtype for both would either narrow the table or force a + // materialized widening of `x`; reading an F32 table as bf16 would halve every + // stride and produce a plausible-looking result from the wrong memory. + void (*modulate)(vt::Queue&, void* x, const void* scale, const void* shift, int64_t rows, + int64_t width, int64_t src_row_stride, vt::DType dtype, vt::DType src_dtype); + + // The gated residual accumulate, IN PLACE: + // dst[r, c] += src[r, c] * gate[r / rows_per_gate_row, c] + // Four host sites collapse onto this one kernel: the `x + y * gate` half of + // `PostSelfAttention` (ltx2_dit.cpp:104-114, ops.py:72-82), the text + // cross-attention gate (ltx2_dit.cpp:197-202, transformer.py:250-251), the + // feed-forward gate (ltx2_dit.cpp:387-392, transformer.py:413-415), and + // `AddGatedBroadcast` (ltx2_dit.cpp:135-145, transformer.py:355-364/:386-395). + // The first three pass `rows_per_gate_row = 1`; the AV cross gate carries a + // SINGLE token row per batch element and passes the stream's token count. + void (*add_gated)(vt::Queue&, void* dst, const void* src, const void* gate, int64_t rows, + int64_t width, int64_t rows_per_gate_row, vt::DType dtype); + + // `PytorchGatedAttention` (ops.py:94-106) as ltx2.cpp:880-891 runs it, IN PLACE: + // attn[r, h * dim_head + e] *= 2 / (1 + exp(-logits[r, h])) + // Applied to the attention output BEFORE `to_out` (attention.py:576-579) and + // driven by the RAW attention input, never by the attention output — gating + // after `to_out` would be a different model. + void (*gate_heads)(vt::Queue&, void* attn, const void* logits, int64_t rows, int64_t heads, + int64_t dim_head, vt::DType dtype); + + // `Ltx2ApplyRotaryEmb` (ltx2.cpp:670-714, apply_rotary_emb rope.py:16-84) over + // `x` [batch, tokens, dim] IN PLACE. + // + // SPLIT (`interleaved = false`, LTX-2.5's setting): cos/sin are + // [batch, heads, tokens, per_head] and within one head the channels split into + // HALVES — channel r pairs with channel per_head + r, NOT with its neighbour + // (rope.py:67, d=2). `per_head` must be head_dim / 2. + // + // INTERLEAVED (`interleaved = true`, upstream's documented legacy mode): + // cos/sin are [batch, tokens, dim] and the (even, odd) pairs rotate + // (rope.py:30-40). `per_head` is ignored. + void (*rope)(vt::Queue&, void* x, const float* cos, const float* sin, int64_t batch, + int64_t tokens, int64_t dim, int64_t heads, int64_t per_head, bool interleaved, + vt::DType dtype); + + // `_process_output`'s affine (ltx2_dit.cpp:534-547, model.py:482-488), IN PLACE: + // shift_v = table[0, c] + embedded[r, c] + // scale_v = table[1, c] + embedded[r, c] + // x[r, c] = x[r, c] * (1 + scale_v) + shift_v + // Upstream forms `scale_shift_values = table + embedded` FIRST and only then + // applies the affine; folding the two additions the other way round would round + // differently, so the kernel keeps upstream's order. + void (*output_modulate)(vt::Queue&, void* x, const float* table, const void* embedded, + int64_t rows, int64_t width, vt::DType dtype); + + // Plain ELEMENTWISE SiLU in place: x[i] = x[i] / (1 + exp(-x[i])). The shared op + // set has SiluAndMul / MoeSiluMul (both GATED forms) but no ungated SiLU, which + // `Ltx2AdaLayerNormSingle` needs twice (ltx2.cpp:766, :773 — TimestepEmbedding + // .forward timestep_embedding.py:84-96 and AdaLayerNormSingle.forward + // adaln.py:44-45). + // + // Matches ltx2.cpp's `Silu` EXACTLY (x / (1 + exp(-x))); the algebraically + // equivalent x * sigmoid(x) is NOT bit-identical, and this path is gated + // against that host reference. + void (*silu)(vt::Queue&, void* x, int64_t n, vt::DType dtype); +}; + +// Resolver. Throws when nothing is registered for (kLtx2, device) — which cannot +// happen for kCPU/kCUDA in a normal build, but keeps the failure explicit rather +// than a null dereference on an unexpected backend. +const Ltx2DeviceKernels* Ltx2Device(vt::DeviceType device); +// True iff the table is registered for `device` (guards the device forward). +bool Ltx2DeviceKernelsAvailable(vt::DeviceType device); + +} // namespace ltx2 + +} // namespace vllm diff --git a/include/vllm/model_executor/models/ltx2_loader.h b/include/vllm/model_executor/models/ltx2_loader.h new file mode 100644 index 000000000..1c2da38a5 --- /dev/null +++ b/include/vllm/model_executor/models/ltx2_loader.h @@ -0,0 +1,665 @@ +// LTX-2.5 QUANTIZED LOADERS — the FP8 DiT, the torchao-NVFP4 DiT and the +// torchao-NVFP4 Gemma-4 text encoder, materialized from their SHIPPED +// checkpoints onto the contracts phases L2 and L3 already committed. +// +// Row: MODEL-DIFFUSION-LTX25, .agents/specs/ltx-2-5.md phase L6. Issue #435. +// +// ─── NO NEW QUANT SCHEME. ONE DELTA, AND IT IS A SCALE LAYOUT ──────────────── +// +// The DiT's FP8 arm is per-tensor E4M3 with an F32 `weight_scale` MULTIPLIER — +// byte for byte what `DequantFp8ToBf16` (nvfp4_dequant.h:76) already consumes. +// +// The text encoder is torchao NVFP4, NOT compressed-tensors. Measured against +// the shipped file rather than assumed: every quantized module carries a +// `torchao_nvfp4` U8[240] marker whose JSON reads +// +// {"format": "torchao_nvfp4", "block_size": 16, "scope": "full", +// "config": "NVFP4DynamicActivationNVFP4WeightConfig", +// "is_swizzled_scales": true, "use_triton_kernel": true, +// "use_dynamic_activation": true, "use_dynamic_per_tensor_scale": true} +// +// so the encoding is the SAME E2M1 nibble pair + fp8-e4m3 group scale + +// per-tensor f32 global as the modelopt W4A16 path `DequantNvfp4ToBf16` +// (nvfp4_dequant.h:59) implements, with `weight_scale_2` used as a MULTIPLIER +// and not reciprocated — confirmed numerically on the real file, not inferred: +// `model.layers.0.self_attn.q_proj` has weight_scale_2 = 1.3515e-4 and a group +// scale whose maximum is exactly 448.0, so 6 * 448 * 1.3515e-4 = 0.3633 is a +// plausible weight amax. Under the compressed-tensors DIVISOR convention +// (nvfp4_emulation.h:18-23) the same number would imply an amax of 2688/1.35e-4 +// = 1.99e7, which no trained weight has. The convention is therefore modelopt's. +// +// The ONE delta is `is_swizzled_scales`: the group-scale tensor is stored in +// the cuBLAS "block scaling factors layout" rather than the linear +// [out, in/16] the two existing paths expect. `Ltx2UnswizzleNvfp4BlockScale` +// inverts exactly the permutation vLLM's own producers apply — +// `swizzle_blockscale` (vllm/model_executor/layers/quantization/utils/ +// nvfp4_utils.py:44-49) and `to_blocked` +// (vllm/model_executor/layers/quantization/qutlass_utils.py:165-180), which are +// the same permutation written twice — and hands the result to the UNCHANGED +// `DequantNvfp4ToBf16`. Nothing else about the scheme differs. +// +// ─── AND A SECOND PRODUCER, WITH A SECOND DELTA ────────────────────────────── +// +// Recorded 2026-08-13 (phase L9a, .agents/specs/nvfp4-nibble-order.md). The above +// describes the torchao-quantized TEXT ENCODER. Lightricks' first-party NVFP4 +// **DiT** was written by their own `nvfp4-prequant`, and it differs twice: +// +// * the SAME swizzled bytes are declared in the cuBLAS-PADDED framing +// [round_up(N,128), round_up(G,4)] rather than torchao's `to_blocked` +// [32*ceil(N/128), 16*ceil(G/4)]. Same buffer, same permutation, different +// 2-D dress — `Ltx2UnswizzleNvfp4BlockScale` reads both unchanged; +// * element 2j is in the HIGH nibble, not the low one. +// +// The second is a different byte ENCODING and is why `DequantNvfp4ToBf16` grew an +// `Nvfp4NibbleOrder`. This file used to be REFUSED, with a message diagnosing its +// scale as the LINEAR layout; that diagnosis was wrong — the bytes were swizzled +// all along, and the linear shape is numerically identical to the padded framing +// for every layer here, which is exactly why a shape test could not tell. +// `Ltx2ResolveNvfp4Producer` below is what decides, and what refuses. +// +// ─── WHAT THE STORED SHAPES MEAN, AND THE TRAP IN THEM ─────────────────────── +// +// NVFP4 packs TWO values per byte along the LAST dimension, so every U8 width +// in the text encoder's header is HALF the logical one. This campaign already +// shipped that mistake once into a spec (§1.4). The loader therefore NEVER +// reads a packed width as logical: `in_features = stored_cols * 2`, always, and +// the result is cross-checked against the unpacked `model.norm.weight` [3840], +// which is BF16 and so authoritative. +// +// The SWIZZLED scale is the second half of the same trap. Its stored shape is +// [out/4, (in/16)*4] — the same element count as the linear [out, in/16], in a +// different 2-D dress. Reading it as linear type-checks, produces finite +// weights, and permutes every group scale within a 128x64 tile. The loader +// asserts the swizzled shape explicitly and refuses anything else BY NAME. +// +// ─── DTYPE ─────────────────────────────────────────────────────────────────── +// +// The default materialization is **bf16**, which is the checkpoint's own model +// dtype: `DequantFp8ToBf16` / `DequantNvfp4ToBf16` land there natively, the +// biases and norms are stored BF16, and upstream resolves ONE model dtype that +// every layer inherits. A gate cannot catch a dtype that is too WIDE, so f32 is +// never the default here. +// +// The tables (`scale_shift_table` and friends) are the annotated exception: +// they are stored F32 IN THE CHECKPOINT and are kept F32, because narrowing a +// tensor the file itself widened would be this rule applied backwards. +// +// `Ltx2WidenDitToF32` exists for one caller only — `Ltx2DitForward`, which +// phase L2 declared f32-only because f32 is its PARITY dtype against upstream +// run in torch float32 (ltx2.h:33-39). It is opt-in, it is named for what it +// does, and it is not what a production load does. +// +// ─── GB10 RESIDENCY ────────────────────────────────────────────────────────── +// +// `Ltx2StreamDitToDevice` dequantizes and uploads ONE TENSOR AT A TIME, freeing +// each host buffer before the next. Two measured findings force this and it is +// not an option: host/ATS-retagged decode weights run 20-30% slower on GB10, so +// weights are staged at LOAD; and a load-to-host-then-stage of a 21 GB DiT +// holds both copies at once, which is what wedged the box during MiniMax-H3's +// port (minimax_h3.h:1598-1606). Same shape, same reason. +// +// ─── WHAT THE SHIPPED DiT CARRIES THAT PHASE L2 DOES NOT PORT ──────────────── +// +// MEASURED 2026-08-12 from the FP8 checkpoint's own header, and reported rather +// than absorbed. The file carries four families outside the L2 contract, and +// they fall into TWO groups that this comment used to conflate — corrected +// 2026-08-13, because the conflation is what made a downstream refusal state +// something untrue about the tree for a whole phase. +// +// UNPORTED. `Ltx2LoadDitFromSafetensors` REFUSES the load by naming these, and +// only an explicit `allow_unported_modules` — which exists so the ported subset +// stays gateable — proceeds, still reporting every one of them in `unported`: +// +// prompt_adaln_single.*, audio_prompt_adaln_single.* +// Upstream builds these only when `cross_attention_adaln AND +// use_prompt_adaln_single` (model.py:222-226, :253-257). Their presence +// means the shipped LTX-2.5 sets `use_prompt_adaln_single = TRUE`, which +// contradicts .agents/specs/ltx-2-5.md §1.2 and ltx2.h:115-117 — and with +// it the prompt-K/V "free win", whose whole premise is that the prompt +// modulation carries no timestep term. +// keyframes_abs_pos_embedding [1, 4096] +// So `use_keyframes_abs_pos_embedding = TRUE`, contradicting ltx2.h:47-49. +// +// LOADED ELSEWHERE — NOT UNPORTED, and never named in that refusal: +// +// video_embeddings_connector.*, audio_embeddings_connector.* +// 8 `transformer_1d_blocks` and a `learnable_registers` [128, dim] each. +// They sit OUTSIDE the DiT contract by design, because upstream loads them +// into the TEXT ENCODER's `EmbeddingsProcessor` through +// `EMBEDDINGS_PROCESSOR_KEY_OPS` (encoder_configurator.py:331-346), and +// phase L9c materializes them here through `Ltx2LoadConnectorWeights`, +// which the video engine calls on the render path. `LoadedElsewhere` +// (ltx2_loader.cpp:417-428) is the code that says so. +// +// This paragraph previously listed the connector families among the refused +// ones, citing `ltx2_text_encoder.h`'s "records as owed" note, and phase L10's +// `encoder_path` refusal cited THIS comment as its evidence that the last hop +// could not be taken. By then it was false on both counts: the weights load, and +// the hop is taken (phase L13). Recorded rather than quietly edited, because a +// refusal whose REASON goes stale is the recurring defect of this campaign and +// this is the third instance. +#pragma once + +#include +#include +#include +#include +#include + +#include + +#include "vllm/model_executor/model_loader/nvfp4_dequant.h" // Nvfp4NibbleOrder +#include "vllm/model_executor/models/ltx2.h" +#include "vllm/model_executor/models/ltx2_audio_vae.h" +#include "vllm/model_executor/models/ltx2_connector.h" +#include "vllm/model_executor/models/ltx2_text_encoder.h" +#include "vllm/model_executor/models/ltx2_upsampler.h" +#include "vllm/model_executor/models/ltx2_video_vae.h" +#include "vt/backend.h" +#include "vt/dtype.h" +#include "vt/tensor.h" + +namespace vllm { + +class SafetensorsFile; +struct StTensor; + +// --------------------------------------------------------------------------- +// torchao NVFP4: the marker, and the one delta +// --------------------------------------------------------------------------- + +// The `.torchao_nvfp4` U8 sidecar, parsed. Every field is read rather +// than assumed, and `ParseLtx2TorchaoNvfp4Marker` REFUSES a combination this +// port does not implement — which is what makes "verify the layout before +// reusing anything" a gate and not a comment. +struct Ltx2TorchaoNvfp4Marker { + std::string format; + int64_t block_size = 0; + std::string scope; + std::string config; + bool is_swizzled_scales = false; + bool use_triton_kernel = false; + bool use_dynamic_activation = false; + bool use_dynamic_per_tensor_scale = false; +}; + +// The tensor name this port is keyed on. +inline constexpr const char* kLtx2TorchaoNvfp4MarkerSuffix = ".torchao_nvfp4"; + +// --------------------------------------------------------------------------- +// WHICH PRODUCER WROTE THIS FILE — .agents/specs/nvfp4-nibble-order.md section 3.2 +// --------------------------------------------------------------------------- +// +// The two NVFP4 checkpoints this campaign loads were written by DIFFERENT +// producers that disagree about BOTH the group scale's 2-D framing AND the nibble +// order. Getting either wrong yields finite, correctly shaped, correctly scaled, +// WRONG weights, so the choice is made from evidence and refused when the +// evidence does not fit. +// +// torchao `.torchao_nvfp4` marker present, scale stored in the +// `to_blocked` framing [32*ceil(N/128), 16*ceil(G/4)], +// LOW-nibble-first. (the Gemma-4 text encoder) +// nvfp4-prequant NO marker anywhere, scale stored in the cuBLAS-PADDED framing +// [round_up(N,128), round_up(G,4)], HIGH-nibble-first. +// (Lightricks' first-party LTX-2.5 DiT) +// +// THE MARKER IS THE DISCRIMINANT, AND ITS ABSENCE IS EVIDENCE, NOT A DEFAULT: +// torchao always emits that sidecar, so a file without one was not written by +// torchao. That is an inference from a producer's signature and it is stated here +// rather than buried, because it is the one step in this path that is not read +// directly off the file. What supports it, measured rather than argued: reading +// the first-party DiT this way correlates 0.9956 (9.46% relative rms, which IS +// NVFP4 quantization error) against the independent vonkaiser FP8 DiT of the same +// base weights, where the alternative readings give 0.0004 to 0.26; and Lightricks' +// own runtime documents exactly this layout and order +// (ltx-kernels/docs/NVFP4.md:27-29, ltx-core/quantization/nvfp4/linear.py:6-7). +// +// THE SHAPE CANNOT DISCRIMINATE ALONE, which is why the marker leads. Every layer +// of the first-party DiT has N % 128 == 0 and G % 4 == 0, so its cuBLAS-padded +// shape is NUMERICALLY IDENTICAL to the linear [N, G] one. A shape test can never +// separate those two. It is used only to CORROBORATE what the marker decided. +// +// WHAT THE REFUSAL DOES AND DOES NOT COVER — state it exactly, because the +// tempting sentence ("any other combination is refused by name") is FALSE and +// would be read as a safety guarantee. +// +// A combination the shape can SEPARATE is refused by name: a marker with the +// to_blocked shape resolves, a marker with any other shape refuses, no marker +// with the to_blocked shape refuses, and no marker with a shape that is neither +// framing refuses. What CANNOT be refused is the case the shape cannot see. A +// marker-less NVFP4 checkpoint whose `weight_scale` is stored LINEAR [N, K/16] +// — never swizzled — presents, for every geometry with N % 128 == 0 and +// G % 4 == 0, a shape numerically identical to the cuBLAS-padded one. It +// therefore resolves kNvfp4Prequant, gets the 128x4 unswizzle applied to scales +// that were never swizzled, and is read high-first. Finite, correctly shaped, +// correctly scaled, WRONG. +// +// That is not an exotic file. LINEAR [N, K/16] is precisely what NVIDIA ModelOpt, +// llm-compressor and compressed-tensors write, and it is what vLLM's own readers +// expect on disk (modelopt.py:1335-1345 and +// compressed_tensors/schemes/compressed_tensors_w4a4_nvfp4.py:73-76 both allocate +// `weight_scale` as [out, in // 16]); none of the three emits a `.torchao_nvfp4` +// sidecar. So the marker's absence excludes torchao and NOTHING ELSE. +// +// It is resolved this way anyway, deliberately, because there is no better +// evidence in the file: the shipped DiT's `__metadata__` carries exactly +// `config`, `gemma_source_checkpoint`, `model_version` and `license` — no +// `quantization_config`, no producer key, and no tensor name mentioning the +// quantizer. Unlike MiniMax-H3, whose community checkpoint DID name its +// converter, there is nothing here to key on. The correlation gate is what +// converts that inference into a measurement, and it is why the gate is the +// result rather than "the forward ran". +// +// The consequence is a tracked condition, recorded in +// .agents/specs/nvfp4-nibble-order.md section 3.1 alongside the H3 one: THIS +// LOADER MUST NOT BE POINTED AT A SECOND MARKER-LESS NVFP4 PRODUCER without +// first re-running the correlation gate against an independent oracle for that +// artifact. It resolves such a file rather than refusing it. +enum class Ltx2Nvfp4Producer { + kTorchao, // marker present: to_blocked framing, low-nibble-first + kNvfp4Prequant, // marker absent: cuBLAS-padded framing, high-nibble-first +}; + +// The two 2-D framings of the SAME swizzled byte buffer. Equal element counts, +// identical layout; `Ltx2UnswizzleNvfp4BlockScale` reads either because it keys +// off the LOGICAL rows/cols and a byte count, not off the declared shape. +std::vector Ltx2Nvfp4ToBlockedScaleShape(int64_t out_features, + int64_t in_features); +std::vector Ltx2Nvfp4PaddedScaleShape(int64_t out_features, + int64_t in_features); + +// Resolve the producer from the marker, corroborated by the stored scale shape. +// `marker` is null when the file declares none for this module. +// +// Throws std::runtime_error naming `module` when the marker and the shape +// disagree, or when the shape matches neither framing — never picking one. +Ltx2Nvfp4Producer Ltx2ResolveNvfp4Producer(const std::string& module, + const Ltx2TorchaoNvfp4Marker* marker, + const std::vector& scale_shape, + int64_t out_features, int64_t in_features); + +// The nibble order each producer packs with. +Nvfp4NibbleOrder Ltx2Nvfp4NibbleOrderFor(Ltx2Nvfp4Producer producer); + +// Parses the marker's JSON payload. Throws std::runtime_error naming `module` +// on: a non-JSON payload, a `format` other than "torchao_nvfp4", a +// `block_size` other than kNvfp4GroupSize (16), or `is_swizzled_scales` false — +// the last because an UNSWIZZLED torchao checkpoint would need the linear read, +// and silently applying the unswizzle to it permutes every scale. +Ltx2TorchaoNvfp4Marker ParseLtx2TorchaoNvfp4Marker(const std::string& module, + const StTensor& marker); + +// Invert the cuBLAS block-scaling-factors permutation. +// +// Source (vLLM `swizzle_blockscale`, nvfp4_utils.py:44-49): the [rows, cols] +// scale is padded to [round_up(rows,128), round_up(cols,4)], viewed as +// [rows/128, 4, 32, cols/4, 4] and permuted to (0, 3, 2, 1, 4). vLLM's +// `to_blocked` (qutlass_utils.py:165-180) writes the identical permutation a +// second way and is what torchao's own producer matches. +// +// So the element at logical (r, c), with r = 128*rt + 32*a + s and +// c = 4*ct + q, lives at flat offset +// +// ((((rt * (cols_padded/4)) + ct) * 32 + s) * 4 + a) * 4 + q +// +// `swizzled_bytes` must be exactly round_up(rows,128) * round_up(cols,4); +// `linear` receives rows*cols bytes. Padding rows/cols are never read. +void Ltx2UnswizzleNvfp4BlockScale(const uint8_t* swizzled, size_t swizzled_bytes, + int64_t rows, int64_t cols, uint8_t* linear); + +// One NVFP4 quantized module, dequantized to bf16 through the UNCHANGED modelopt +// path: unswizzle the group scale, then `DequantNvfp4ToBf16`. +// +// packed U8 [out, in/2] (in = stored cols * 2), packed per `producer` +// scale F8_E4M3 the SWIZZLED group scale, in `producer`'s framing +// scale_2 F32 scalar MULTIPLIER +// +// Both scale layouts are the same permutation over the same bytes; only the +// declared 2-D shape and the nibble order differ, and `producer` names which. +// Every one of those three shapes is asserted against `out`/`in` before a byte is +// read, and a mismatch throws naming `module`. `out_bf16` receives out*in bf16 bit +// patterns. +// +// `producer` is NOT defaulted. A default here would let a caller that never +// thought about the question silently get the torchao reading for a +// nvfp4-prequant file, which is precisely the failure this seam exists to refuse. +void Ltx2DequantNvfp4ToBf16(const std::string& module, const StTensor& packed, + const StTensor& scale, const StTensor& scale_2, + int64_t out_features, int64_t in_features, + Ltx2Nvfp4Producer producer, uint16_t* out_bf16); + +// --------------------------------------------------------------------------- +// The DiT +// --------------------------------------------------------------------------- + +// ComfyUI-format checkpoints — which is what both shipped LTX-2.5 DiTs are — +// prefix every parameter. Stripped before the name meets the L2 contract. +inline constexpr const char* kLtx2DitCheckpointPrefix = "model.diffusion_model."; + +// Which quantization the DiT file actually uses, DETECTED from the tensors +// rather than from a filename. +enum class Ltx2DitQuant { + kFp8, // F8_E4M3 weight + F32 scalar `_scale` (vonkaiser 22b-distilled-fp8) + kNvfp4, // U8 packed + F8_E4M3 `_scale` + F32 `_scale_2` +}; + +struct Ltx2DitLoadOptions { + // Proceed past the unported families named at the top of this header, + // reporting them in `Ltx2DitCheckpoint::unported` instead of throwing. The + // ported subset is still bound exactly; nothing is approximated. + bool allow_unported_modules = false; + // Widen the bf16 materialization to f32 for `Ltx2DitForward`, whose gate is + // f32 by declaration. Doubles the footprint; see the DTYPE note above. + bool widen_to_f32 = false; +}; + +// A host buffer owned by a loaded checkpoint. Pointer-stable: the views index +// into `bytes`, and the shared_ptr keeps that allocation alive independently of +// how the owning vector grows. +struct Ltx2HostBuffer { + std::vector bytes; + vt::DType dtype = vt::DType::kF32; +}; + +struct Ltx2DitCheckpoint { + Ltx2DitQuant quant = Ltx2DitQuant::kFp8; + // The geometry the FILE describes, including the flags whose modules this + // port does not carry. + Ltx2DitParams checkpoint_params; + // The geometry actually BOUND — `checkpoint_params` with every unported flag + // cleared. The two differing is exactly what `unported` enumerates. + Ltx2DitParams params; + // Module prefixes present in the file and outside the L2 contract, in header + // order, deduplicated. Empty means the file and the contract agree. + std::vector unported; + Ltx2DitWeights weights; + std::map views; + // Host-resident buffers (`Ltx2LoadDitFromSafetensors`). Pointer-stable. + std::vector> storage; + // Device allocations (`Ltx2StreamDitToDevice`), each with its backend's Free + // as the deleter so a staged checkpoint releases exactly like a host one. + std::vector> device_storage; +}; + +// Recover `Ltx2DitParams` from a checkpoint header alone, prefix stripped and +// packed widths doubled. Pure manifest work: no payload is touched. +Ltx2DitParams Ltx2ParseDitParamsFromCheckpoint(const SafetensorsFile& file, + Ltx2DitQuant* out_quant = nullptr); + +// Materialize the whole DiT onto the L2 contract, HOST-resident. +// +// Every name the contract requires and the file lacks throws BY NAME, so a +// missing tensor can never read as zeros. `EnumerateLtx2DitTensors` supplies +// the required set and its shapes; both are checked before anything is bound. +// +// This is the REFERENCE loader and it materializes the whole model at once +// (~21 GB bf16 for the shipped FP8 DiT). A real GB10 run uses +// `Ltx2StreamDitToDevice`. +Ltx2DitCheckpoint Ltx2LoadDitFromSafetensors(const SafetensorsFile& file, + const Ltx2DitLoadOptions& options = {}); + +// Widen a bf16 checkpoint in place to f32, rebinding every view. Only for +// `Ltx2DitForward`; see the DTYPE note. +void Ltx2WidenDitToF32(Ltx2DitCheckpoint& checkpoint); + +// The GB10 arm: dequantize and upload tensor by tensor, freeing each host +// buffer before the next, so peak residency is the device copy plus ONE tensor. +// Returns the same struct with `views` pointing at DEVICE memory and +// `device_storage` — NOT `storage`, which stays empty on this path — holding the +// device allocations; `widen_to_f32` is refused here because the point of +// staging is not to move twice the bytes. +Ltx2DitCheckpoint Ltx2StreamDitToDevice(vt::Queue& queue, const SafetensorsFile& file, + const Ltx2DitLoadOptions& options = {}); + +// --------------------------------------------------------------------------- +// The text encoder +// --------------------------------------------------------------------------- + +// One caption projection as it comes off the checkpoint: bf16 weight, bf16 +// bias. `Ltx2TextFeatureExtractorForward` refuses a config/weights disagreement +// (ltx2_text_encoder.h:260-269), and the bias is loaded here precisely because +// the case it names — reading the U8 weight and missing the BF16 bias — is what +// a two-dtype module invites. +struct Ltx2TextProjection { + std::vector weight_bf16; // [out_features, in_features] + std::vector bias_bf16; // [out_features], empty when bias=False + int64_t out_features = 0; + int64_t in_features = 0; +}; + +struct Ltx2TextEncoderLoadOptions { + // Mirror upstream's `GemmaAssets.from_single_file` refusal of a file with no + // `__metadata__`. The shipped checkpoint HAS none, so a caller holding the + // Gemma config out of band passes false — see ltx2_text_encoder.h:366-373. + bool require_config = false; + // Skip the two ~1.5 GB / ~0.8 GB projections and load only the geometry, the + // assets and the quantized-module inventory. What a manifest gate wants. + bool skip_projections = false; +}; + +struct Ltx2TextEncoderCheckpoint { + // Established from the UNPACKED `model.norm.weight` (BF16, so authoritative), + // never from a packed U8 width. Cross-checked against both projections. + int64_t gemma_hidden_size = 0; + int64_t gemma_num_hidden_layers = 0; + Ltx2TextProjection video, audio; + Ltx2GemmaAssets assets; + // Every module carrying a `torchao_nvfp4` marker, in header order. Includes + // the full multimodal tower (`vision_model.*`, `multi_modal_projector`, + // `audio_projector`) the file also ships: text conditioning is the scope, but + // a loader that CHOKES on their presence cannot read this checkpoint. + std::vector quantized_modules; +}; + +// Materialize the LTX-specific half of the text encoder — the two caption +// projections, the embedded tokenizer/asset pack and the Gemma geometry — from +// the shipped torchao-NVFP4 file. +// +// Also VALIDATES every quantized module in the file, the Gemma tower included: +// each must carry weight / weight_scale / weight_scale_2 / torchao_nvfp4 with +// shapes consistent under the packed-width and swizzled-scale rules. A module +// that does not throws BY NAME, so an unreadable tower is a load-time refusal +// and not a phase-L7 surprise. +// +// The Gemma TOWER ITSELF is not materialized here: `ltx2_text_encoder.h` +// declares no tower contract (it consumes hidden states through +// `Ltx2TextHiddenStates`, which `Gemma4Model::ForwardHiddenStates` produces). +// Wiring the tower's torchao arm onto `Gemma4Weights` is owed and named as such +// rather than half-done. +Ltx2TextEncoderCheckpoint Ltx2LoadTextEncoderFromSafetensors( + const SafetensorsFile& file, const Ltx2TextEncoderLoadOptions& options = {}); + +// Widen the bf16 projections into `Ltx2TextEncoderWeights`, whose f32 is phase +// L3's declared PARITY dtype (ltx2_text_encoder.h:73-83). Opt-in, and ~4.6 GB +// at the shipped widths — which is exactly why it is not what loading does. +Ltx2TextEncoderWeights Ltx2WidenTextProjectionsToF32( + const Ltx2TextEncoderCheckpoint& checkpoint); + +// --------------------------------------------------------------------------- +// The VAEs, the upsampler and the duration head (phase L7) +// --------------------------------------------------------------------------- +// +// Every one of these ships as a plain bf16 safetensors file whose CONFIG rides +// in the file's own `__metadata__["config"]`, exactly as upstream's +// `SafetensorsModelStateDictLoader().metadata(path)` reads it +// (video_vae/model_configurator.py:21-24, audio_vae/model_configurator.py:50, +// :109). So a sibling JSON is not merely unnecessary here — reading one would +// let a caller pair a config with weights it does not describe, which is the +// mis-load this whole seam exists to refuse. + +// The parsed `__metadata__["config"]` object. Throws BY NAME when the key is +// absent or is not a JSON object: a VAE whose geometry has to be GUESSED +// decodes to a plausible, finite, wrong picture. +nlohmann::json Ltx2ReadCheckpointConfig(const SafetensorsFile& file); + +// Adopt a `{"transformer": {...}}` DiT configuration onto the params the SHAPES +// resolved, or refuse by name. +// +// WHAT THIS DECIDES, and why it is not cosmetic. `Ltx2ParseDitParamsFromManifest` +// reads shapes, which is the only thing a ComfyUI-flavoured checkpoint offers. A +// config states what no shape encodes — `frequencies_precision`, +// `av_ca_timestep_scale_multiplier`, the positional-embedding bounds and theta, +// `norm_eps`, `use_middle_indices_grid`. Each moves every RoPE angle or every +// modulation while leaving the tensor set byte-identical, so the manifest path +// resolves a DIFFERENT MODEL from the same file and nothing downstream can tell. +// +// A config is believed only when `EnumerateLtx2DitTensors` over it reproduces the +// IDENTICAL weight contract `from_shapes` produces. That is what proves it +// describes THIS file rather than another checkpoint's config pasted beside it; +// a disagreement is refused rather than resolved in either direction, because +// taking the shapes renders with the wrong RoPE and taking the config binds the +// wrong tensors. +// +// ONE FUNCTION, because two callers must not answer this differently: the video +// engine (`Ltx2VideoEngine::Load`) and the device gate, which drives +// `Ltx2StreamDitToDevice` directly and therefore owes the same adoption. +// +// `allow_unported_modules` clears `use_keyframes_abs_pos_embedding` IN A COPY of +// the config before parsing, mirroring what the loader does for +// `use_prompt_adaln_single`: the flag is cleared for the CONTRACT, the module +// stays unported, and the checkpoint's `unported` list still names it. Without +// the opt-in `ParseLtx2DitParams` throws, which is the refusal. +// +// `source` names the config in every refusal, so a reader knows whether the +// checkpoint declared it or a caller supplied it. +Ltx2DitParams Ltx2AdoptDeclaredDitParams(const nlohmann::json& config, + const Ltx2DitParams& from_shapes, + bool allow_unported_modules, + const std::string& source); + +// --------------------------------------------------------------------------- +// The embeddings connector (phase L9c) +// --------------------------------------------------------------------------- +// +// The two `*_embeddings_connector` families live in the DiT FILE but not in the +// DiT's own weight contract: upstream loads them into the TEXT ENCODER's +// `EmbeddingsProcessor`, through `EMBEDDINGS_PROCESSOR_KEY_OPS`, which rewrites +// `model.diffusion_model.video_embeddings_connector.` to `video_connector.` +// (text_encoders/gemma/encoders/encoder_configurator.py:331-346). They are +// therefore loaded HERE, beside the DiT and not inside it, and they stay outside +// `EnumerateLtx2DitTensors`. + +enum class Ltx2ConnectorStream { kVideo, kAudio }; + +// The tensor-name prefix each stream's family carries in the checkpoint, WITHOUT +// the ComfyUI `model.diffusion_model.` prefix (`PlanDit` strips that). +const char* Ltx2ConnectorCheckpointPrefix(Ltx2ConnectorStream stream); + +// Does this DiT file carry a connector at all? Keyed on `learnable_registers`, +// the one tensor the family always has and the only one that is not per-block. +bool Ltx2CheckpointHasConnector(const SafetensorsFile& file, Ltx2ConnectorStream stream); + +// `Embeddings1DConnectorConfigurator.from_metadata` and its audio twin +// (embeddings_connector.py:194-256) applied to a `{"transformer": {...}}` object. +// +// FOUR VALUES NO SHAPE ENCODES, and one of them is not close to its default: +// LTX-2.5 declares `connector_positional_embedding_max_pos = [4096]` where the +// class default is `[1]`. `get_fractional_positions` divides the token index by +// it (rope.py:132-141), so the default turns every position into a fractional +// position 4096x too large and every RoPE angle with it. The others are +// `rope_type`, `frequencies_precision` (-> double-precision frequencies) and +// `connector_apply_gated_attention`. +// +// `positional_embedding_theta` is deliberately NOT read from the config even +// though the DiT declares one: neither configurator passes it, so upstream's +// connector always runs at the class default of 10000.0. Reading the DiT's key +// would be a re-invention, not a port — and the shipped file declares 10000.0 +// anyway, so the two agree and only the RULE differs. +Ltx2ConnectorConfig Ltx2ParseConnectorConfig(const nlohmann::json& config, + Ltx2ConnectorStream stream); + +// Materialize one connector family out of the DiT checkpoint, widened to f32 — +// which is `Ltx2ConnectorForward`'s declared parity dtype. +// +// IT IS NOT CHEAP AND THE CALLER MUST TREAT IT AS EXPENSIVE. 129 tensors is 8 +// blocks of four dim x dim projections plus a 4x-wide feed-forward, so at the +// shipped widths the video family is ~1.61G parameters and the audio family +// ~0.40G: **about 8 GB of f32 together**. That is small against the DiT's 21 GB +// on disk and 44 GB staged, and it is NOT small on a 119 GB unified-memory box +// that reboots rather than OOM-killing. The video engine therefore loads these, +// runs the connector once, and drops them inside one scope — the conditioning is +// resolved at load time, so the weights never outlive their single use. +// +// It is also the CONTRACT CHECK on the config. `Ltx2ParseConnectorConfig` reads +// values that mostly cannot be seen in a shape — but `connector_num_layers`, +// `connector_apply_gated_attention`, `connector_ff_bias` and the head geometry +// all CAN, so every enumerated tensor must exist at its enumerated shape AND no +// tensor of the family may be left over. A config that says 2 layers against a +// file carrying 8 is refused by name here rather than binding the first two and +// rendering. +Ltx2VaeWeights Ltx2LoadConnectorWeights(const SafetensorsFile& file, + const Ltx2ConnectorConfig& config); + +// `__metadata__["model_version"]` ("2.5.0"), which is what +// `detect_model_version` reads to pick a recipe (ltx-pipelines +// utils/constants.py:161) and what `should_use_ancestral_sampler` keys on +// (distilled.py:84). Empty when the file declares none — never defaulted to a +// generation, because defaulting one picks a sigma schedule that renders. +std::string Ltx2ReadCheckpointModelVersion(const SafetensorsFile& file); + +// One `SDOps` key rule: a name is KEPT when it starts with `match_prefix`, and +// that prefix is then rewritten to `replacement`. Mirrors +// `SDOps.with_matching(prefix=...)` + `.with_replacement(from, to)` +// (loader/sd_ops.py), which is how every shipped VAE file's ComfyUI-flavoured +// names are turned into the module's own `state_dict` names. A rule set is +// applied IN ORDER and the first matching rule wins, so the longer prefixes come +// first exactly as upstream's chained filters list them. +// +// A name matched by NO rule is dropped, which is upstream's behaviour and is +// what lets one file hold the encoder, the decoder and the vocoder at once. +struct Ltx2VaeKeyRule { + std::string match_prefix; + std::string replacement; +}; + +// The three shipped filters, verbatim: +// VAE_DECODER_COMFY_KEYS_FILTER video_vae/model_configurator.py:255-265 +// AUDIO_VAE_DECODER_COMFY_KEYS_FILTER audio_vae/model_configurator.py:184-190 +// VOCODER_COMFY_KEYS_FILTER audio_vae/model_configurator.py:91-103 +// The vocoder's rule strips `vocoder.` EXACTLY ONCE (`removeprefix`, :91-97), so +// `vocoder.vocoder.conv_pre` becomes `vocoder.conv_pre` and NOT `conv_pre` — +// which is the difference between the BWE chain finding its two generators and +// binding both of them to one name. +std::vector Ltx2VideoVaeDecoderKeyRules(); +std::vector Ltx2AudioVaeDecoderKeyRules(); +std::vector Ltx2VocoderKeyRules(); + +// Every tensor a rule set keeps, widened to f32 and keyed by the REWRITTEN name, +// which is `Ltx2VaeWeights`' whole contract (ltx2_audio_vae.h:60-70). BF16 and +// F32 are the only dtypes these files carry; anything else throws by name rather +// than being reinterpreted. An empty rule set keeps every name unchanged. +// +// f32 here is NOT a widening of a production path: `Ltx2VaeWeights` is declared +// f32 by phases L4/L5 because f32 is their parity dtype, and this materializes +// onto that declared contract. The VAEs together are ~1.8 GB bf16, so the +// widened copy is ~3.7 GB — small next to the DiT, and the reason the DiT does +// NOT take this path. +Ltx2VaeWeights Ltx2LoadVaeWeights(const SafetensorsFile& file, + const std::vector& rules = {}); + +// `_build_conv_video_decoder` (video_vae/model_configurator.py:81-94) applied to +// `config["vae"]`, plus `_vae_class_name_from_metadata` (:21-24) recovered into +// `*out_kind` so a `CausalDiffusionVAE` checkpoint is REFUSED by +// `Ltx2VideoDecode` rather than decoded by the conv arm. +Ltx2ConvVideoDecoderConfig Ltx2ParseConvVideoDecoderConfig(const nlohmann::json& config, + Ltx2VideoDecoderKind* out_kind); + +// `AudioDecoderConfigurator.from_metadata` (audio_vae/model_configurator.py:108-141). +Ltx2AudioDecoderConfig Ltx2ParseAudioDecoderConfig(const nlohmann::json& config); + +// `LatentUpsamplerConfigurator.from_metadata` (upsampler/model_configurator.py:14-21). +// Unlike the two VAEs this config is FLAT — the shipped +// `ltx-2.5-latent-spatial-upscaler-x2` writes its keys at the top level of +// `__metadata__["config"]`, with no `vae` wrapper — so it is read from the +// object itself. `temporal_upsample` is left as the file states it and refused +// downstream by `Ltx2LatentUpsample`, which is where the missing arm lives. +Ltx2UpsamplerConfig Ltx2ParseUpsamplerConfig(const nlohmann::json& config); + +// `VocoderConfigurator.from_metadata`, the BWE branch +// (audio_vae/model_configurator.py:49-88). The legacy flat branch (:53-56) is +// REFUSED by name: it is the pre-2.3 `resblock == "1"` vocoder, no LTX-2.5 +// checkpoint carries it, and silently building it would emit audio from the +// wrong generator. Every `check_config_value` upstream asserts is asserted here. +Ltx2VocoderBweConfig Ltx2ParseVocoderBweConfig(const nlohmann::json& config); + +} // namespace vllm diff --git a/include/vllm/model_executor/models/ltx2_pipeline.h b/include/vllm/model_executor/models/ltx2_pipeline.h new file mode 100644 index 000000000..264668416 --- /dev/null +++ b/include/vllm/model_executor/models/ltx2_pipeline.h @@ -0,0 +1,591 @@ +// LTX-2.5 PIPELINE — the flow-matching schedule, the noiser, the diffusion +// steps, guidance, the patchifiers, and the recipe table. +// +// Row: MODEL-DIFFUSION-ltx-2-5-ltx2-video-transformer-3d-model. Spec: +// .agents/specs/ltx-2-5.md (phase L5). Issue #435. +// +// ─── WHAT THIS IS A PORT OF (file:line on BOTH sides) ──────────────────────── +// Upstream A: Lightricks/LTX-2, packages/ltx-core/src/ltx_core/ +// OURS <- UPSTREAM +// Ltx2SigmaSchedule <- components/schedulers.py:21-57 (LTX2Scheduler) +// Ltx2LinearQuadraticSchedule <- components/schedulers.py:67-88 +// Ltx2GaussianNoise <- components/noisers.py:30-37 +// Ltx2EulerStep <- components/diffusion_steps.py:32-40 +// Ltx2EulerAncestralStep <- components/diffusion_steps.py:63-106 +// Ltx2Res2sStep / Ltx2SdeCoeff <- components/diffusion_steps.py:118-190 +// Ltx2EulerCfgPpStep <- components/diffusion_steps.py:208-252 +// Ltx2AncestralStep <- components/diffusion_steps.py:7-22 +// Ltx2ProjectionCoef <- components/guiders.py:363-369 +// Ltx2CfgDelta / Ltx2StgDelta <- components/guiders.py:23-27, 70-74 +// Ltx2MultiModalGuidance <- components/guiders.py:244-291 +// Ltx2GuiderParamsForSigma <- components/guiders.py:214-230, 332-335 +// Ltx2BatchedPerturbationConfig <- guidance/perturbations.py:53-143 +// Ltx2VideoPatchify / … <- components/patchifiers.py:11-134 +// Ltx2PixelCoords <- components/patchifiers.py:137-171 +// Ltx2AudioPatchify / … <- components/patchifiers.py:174-353 +// +// Upstream B: Lightricks/LTX-2, packages/ltx-pipelines/src/ltx_pipelines/ +// Ltx2ParseModelVersion <- (ltx-core) loader/helpers.py:62-81 +// Ltx2DetectPipelineParams <- utils/constants.py:130-179 +// Ltx2ShouldUseAncestralSampler <- distilled.py:62-84 +// the distilled sigma constants <- utils/constants.py:17-25 +// +// Upstream C: vLLM-Omni, vllm_omni/diffusion/models/ltx2/ltx2_recipes.py +// Ltx2PipelineRecipe / …Phase… <- ltx2_recipes.py:29-87 +// ResolveLtx2PipelineRecipe <- ltx2_recipes.py:161-175 +// +// ─── WHICH UPSTREAM OWNS WHICH VALUE ───────────────────────────────────────── +// vLLM-Omni is this project's BINDING oracle (spec section 3) and it carries NO +// 2.5 row — its table stops at 2.3. So the SHAPE of the recipe model and the +// values of every pre-2.5 row come from vLLM-Omni, and the values of the 2.4 and +// 2.5 rows come from Lightricks `ltx-pipelines`, which is the model author's own +// runtime. Each row below names its source. Where the two references DISAGREE the +// disagreement is recorded rather than resolved by preference: they ship +// different default negative prompts, and both strings are exposed +// (kLtx2LightricksNegativePrompt vs kLtx2OmniNegativePrompt in the goldens). +// +// ─── THE REFUSAL IS THE POINT ──────────────────────────────────────────────── +// `resolve_ltx_pipeline_recipe` RAISES on an unknown (kind, version) rather than +// defaulting (ltx2_recipes.py:170-175), and this port mirrors that exactly. A +// sigma schedule or a guidance scale that is plausible but wrong does not fail — +// it renders. Defaulting an unknown checkpoint generation onto 2.0's 40-step, +// STG-block-29 recipe would produce a video, and nothing downstream could tell. +// +// ─── DTYPE ─────────────────────────────────────────────────────────────────── +// Every buffer here is f32, and that is upstream's OWN width on these paths, not +// a widening: the noiser lerps in `.float()` (noisers.py:32-33), every diffusion +// step casts to `torch.float32` and back to the sample dtype +// (diffusion_steps.py:40, 83-84, 90-91, 231-240), and MultiModalGuider.calculate +// opens with `cond.float()` (guiders.py:256-260). The schedules are float32 +// tensors upstream too (schedulers.py:57, 88). Sigma SHIFT arithmetic is the one +// double: `mm`, `b` and `math.exp(sigma_shift)` are Python floats before torch +// sees them (schedulers.py:37-44), so they are computed in double here and +// rounded to f32 at exactly the point torch rounds them. +#pragma once + +#include +#include +#include + +namespace vllm { + +// --------------------------------------------------------------------------- +// Sigma schedules (components/schedulers.py) +// --------------------------------------------------------------------------- + +// schedulers.py:10-11. The token axis the shift is fitted on. These are module +// constants, not arguments: they decide how a resolution maps to a shift, so a +// port that moved either would produce a valid-looking schedule for the wrong +// resolution. +inline constexpr int64_t kLtx2BaseShiftAnchor = 1024; +inline constexpr int64_t kLtx2MaxShiftAnchor = 4096; + +// schedulers.py:41. The exponent applied to `(1/sigma - 1)`. A literal upstream, +// pinned here because `power != 1` would change every sigma while still +// producing a monotone schedule that renders. +inline constexpr double kLtx2SigmaShiftPower = 1.0; + +// LTX2Scheduler.execute's keyword defaults (schedulers.py:21-31). +struct Ltx2SchedulerParams { + double max_shift = 2.05; + double base_shift = 0.95; + bool stretch = true; + double terminal = 0.1; + // `default_number_of_tokens` — used when no latent is supplied. + int64_t default_number_of_tokens = kLtx2MaxShiftAnchor; +}; + +// LTX2Scheduler.execute (schedulers.py:21-57). Returns `steps + 1` sigmas, from +// ~1 down to exactly 0. `tokens` is `math.prod(latent.shape[2:])` — pass 0 to +// take `params.default_number_of_tokens`, which is what a caller with no latent +// yet does. Throws when `steps < 1`. +std::vector Ltx2SigmaSchedule(int64_t steps, int64_t tokens, + const Ltx2SchedulerParams& params = {}); + +// LinearQuadraticScheduler.execute (schedulers.py:67-88). `linear_steps < 0` +// takes upstream's `steps // 2` default. +std::vector Ltx2LinearQuadraticSchedule(int64_t steps, double threshold_noise = 0.025, + int64_t linear_steps = -1); + +// Which scheduler a caller is asking for. `kBeta` is upstream's third +// (schedulers.py:91-120) and is NOT ported: it inverts a Beta CDF through +// `scipy.stats.beta.ppf`, no ltx-pipelines entry point constructs it, and +// approximating an inverse incomplete beta would be a numerical port of its own. +// `Ltx2Schedule` REFUSES it by name rather than substituting LTX2Scheduler. +enum class Ltx2SchedulerKind { kLtx2, kLinearQuadratic, kBeta }; + +// The seam a caller reaches for when it holds a configured kind. Forwards to the +// two ported schedulers and throws for `kBeta`. +std::vector Ltx2Schedule(Ltx2SchedulerKind kind, int64_t steps, int64_t tokens, + const Ltx2SchedulerParams& params = {}); + +// --------------------------------------------------------------------------- +// The noiser (components/noisers.py) +// --------------------------------------------------------------------------- + +// GaussianNoiser.__call__ (noisers.py:30-37), elementwise over `count`: +// latent = lerp(latent, noise, noise_scale) +// latent = lerp(clean_latent, latent, denoise_mask) +// The SECOND lerp is the one a port gets backwards: `denoise_mask` selects +// toward the NOISED latent, so a conditioned token (mask 0) keeps its clean value +// and an unconditioned one (mask 1) is fully noised. Swapping the operands still +// produces a plausible video with the conditioning frames re-noised away. +// +// `noise` is drawn by the caller (upstream uses `torch.randn(generator=...)`, +// noisers.py:22-28); this port takes the draw as an argument so the stream stays +// the caller's, exactly as Ltx2NoiseStream does for the VAE. +std::vector Ltx2GaussianNoise(const float* latent, const float* clean_latent, + const float* denoise_mask, const float* noise, + int64_t count, float noise_scale = 1.0f); + +// --------------------------------------------------------------------------- +// Diffusion steps (components/diffusion_steps.py) +// --------------------------------------------------------------------------- + +// EulerDiffusionStep.step (diffusion_steps.py:32-40): `x + to_velocity(...) * dt`. +// `to_velocity` (utils.py:21-36) REFUSES sigma == 0 rather than dividing, so a +// terminal step_index throws here too. +std::vector Ltx2EulerStep(const float* sample, const float* denoised, + const float* sigmas, int64_t sigma_count, int64_t step_index, + int64_t count); + +// EulerAncestralDiffusionStep.step (diffusion_steps.py:63-106). The +// rectified-flow parameterization (alpha = 1 - sigma), which is LTX-2's; it is +// deliberately NOT the DDIM one `Ltx2AncestralStep` implements, and the two agree +// only at eta == 0 (the class docstring says so at :51-56). +// +// `noise` may be null only when `eta == 0`; anything else throws, mirroring +// upstream's own ValueError (:87-88). When `sigmas[step_index + 1] == 0` the +// denoised prediction is returned unchanged (:85-86). +std::vector Ltx2EulerAncestralStep(const float* sample, const float* denoised, + const float* sigmas, int64_t sigma_count, + int64_t step_index, int64_t count, double eta = 1.0, + double s_noise = 1.0, const float* noise = nullptr); + +// Res2sDiffusionStep.get_sde_coeff (diffusion_steps.py:118-155), the `sigma_up` +// arm — the only one `step` uses (:179). `sigma_up` is clamped IN to +// `sigma_next * kLtx2Res2sSigmaUpClamp` before anything else. +struct Ltx2SdeCoeff { + double alpha_ratio = 1.0; + double sigma_down = 0.0; + double sigma_up = 0.0; +}; + +// diffusion_steps.py:138. What keeps `sqrt(sigma_next^2 - sigma_up^2)` off zero, +// and it BINDS on the ordinary eta = 1 schedule rather than only on a malformed +// one. `step` forms `sigma_up = sigma_next * eta`, so eta <= 1 gives +// sigma_up <= sigma_next — but <= includes ==, and at equality `min` takes +// `sigma_next * 0.9999`, which is the whole point: without it the residual is +// exactly 0 and `sigma_down` collapses. The earlier note reasoned from the +// inequality and skipped its boundary, calling the constant invisible; it is not. +// A 1% move (0.9999 -> 0.99) REDS the Eta1 arm the suite already runs, at +// max|diff| = 0.086 (index 0) and 0.130563 (index 1), because the residual scales +// as sqrt(1 - clamp^2) and that is 10x larger at 0.99. The EtaHalf arm stays green +// (0.5 * sigma_next is below the clamp) and Eta1 index 2 stays green +// (sigma_next == 0 returns the denoised prediction unchanged, :181-182). Pinned +// as well, because a regenerated golden would move with the constant. +inline constexpr double kLtx2Res2sSigmaUpClamp = 0.9999; + +Ltx2SdeCoeff Ltx2Res2sSdeCoeff(double sigma_next, double sigma_up); + +// Res2sDiffusionStep.step (diffusion_steps.py:157-190). Returns the denoised +// prediction unchanged when `sigma_up` or `sigma_next` is 0 (:181-182). +std::vector Ltx2Res2sStep(const float* sample, const float* denoised, + const float* sigmas, int64_t sigma_count, int64_t step_index, + int64_t count, const float* noise, double eta = 0.5); + +// _get_ancestral_step (diffusion_steps.py:7-22): the DDIM / variance-exploding +// ancestral coefficients, in the rescaled `sigma / alpha` space. Used only by +// CFG++. +struct Ltx2AncestralSigmas { + double sigma_down = 0.0; + double sigma_up = 0.0; +}; +Ltx2AncestralSigmas Ltx2AncestralStep(double sigma_from, double sigma_to, double eta = 1.0); + +// diffusion_steps.py:233-235. `torch.finfo(torch.float32).eps`, the clamp that +// keeps `alpha = 1 - sigma` off zero when sigma is EXACTLY 1.0 — which the first +// step of every unstretched schedule is. Unlike most of the invisible-constant +// class this one DOES bind on a real schedule, and the goldens include the arm +// where it decides the numbers. +inline constexpr double kLtx2CfgPpAlphaEps = 1.1920928955078125e-07; + +// EulerCfgPpDiffusionStep.step (diffusion_steps.py:208-252). +std::vector Ltx2EulerCfgPpStep(const float* sample, const float* denoised, + const float* uncond_denoised, const float* sigmas, + int64_t sigma_count, int64_t step_index, int64_t count, + double eta = 1.0, double s_noise = 1.0, + const float* noise = nullptr); + +// --------------------------------------------------------------------------- +// Guidance (components/guiders.py) +// --------------------------------------------------------------------------- + +// guiders.py:368. The stabilizer under `squared_norm`. It is ~1e-8 relative +// against an O(1) denominator, so a tensor comparison cannot see it — except on +// the one arm where `project_onto` is all zeros, which the goldens carry. +inline constexpr double kLtx2ProjectionCoefEps = 1e-8; + +// projection_coef (guiders.py:363-369): per BATCH ROW, +// `dot(to_project, project_onto) / (||project_onto||^2 + eps)`. `count` is the +// number of elements per row, i.e. the product of every axis after the batch. +std::vector Ltx2ProjectionCoef(const float* to_project, const float* project_onto, + int64_t batch, int64_t count); + +// CFGGuider.delta (guiders.py:23-24) and STGGuider.delta (:70-71). Both are +// elementwise and rank-agnostic, which is why they are the two that survive +// contact with a real 5-D latent — see the refusal below. +std::vector Ltx2CfgDelta(const float* cond, const float* uncond, int64_t count, + double scale); +std::vector Ltx2StgDelta(const float* cond, const float* perturbed, int64_t count, + double scale); + +// NOT PORTED, refused by name: CFGStarRescalingGuider (guiders.py:30-52), +// LtxAPGGuider (:77-125) and LegacyStatefulAPGGuider (:128-191). +// +// THE REASON IS REACHABILITY. Nothing in the LTX-2 tree constructs any of the +// three: they appear only at their own `class` statements (:31, :78, :129), and +// every pipeline builds MultiModalGuider from MultiModalGuiderParams +// (utils/constants.py:49-68). Porting an arm upstream cannot reach would be +// inventing behaviour, so `Ltx2Guidance` refuses them by name and they are +// recorded as owed (.agents/porting-inventory.md 9.18(b)). +// +// IT IS *NOT* BECAUSE THE SHAPES CANNOT WORK — an earlier revision of this +// comment said so and was wrong, and the correction is kept here because the +// wrong version had been frozen as a golden (spec §7.0(b)). All three do +// multiply `projection_coef`'s rank-2 `(B, 1)` result straight into the latent +// (`proj_coeff * cond`, :48, :118, :184) and torch does right-align it onto the +// LAST TWO axes, but the measured predicate is +// +// raises <=> B > 1 and shape[-2] not in {1, B} +// +// so at B = 1 — the ordinary single-request video latent — it composes and is +// numerically CORRECT, `(1, 1)` being a scalar. Where it composes with B > 1 it +// is silently wrong, applying the per-batch coefficient along axis -2 rather +// than the batch axis. LtxAPG's `norm(dim=[-1, -2, -3])` (:114) is a SEPARATE +// constraint needing rank >= 3. The measured matrix is a golden +// (kLtx2GuideProbeComposes), now including B = 1 and shape[-2] == B rows, so +// upstream changing either the shapes or the reachability fails this gate +// instead of going unnoticed. +enum class Ltx2GuiderKind { + kCfg, + kStg, + kMultiModal, + kCfgStarRescaling, // refused + kLtxApg, // refused + kLegacyStatefulApg, // refused +}; + +// MultiModalGuiderParams (guiders.py:194-211) — the ONLY guider parameter object +// any ltx-pipelines entry point constructs. +struct Ltx2MultiModalGuiderParams { + double cfg_scale = 1.0; + double stg_scale = 0.0; + std::vector stg_blocks; + double rescale_scale = 0.0; + double modality_scale = 1.0; + int64_t skip_step = 0; + + // guiders.py:275-291. `math.isclose` with its DEFAULT rel_tol of 1e-9, not + // `!=`: a scale of 1.0 + 1e-12 is "no guidance" here and "guidance" under an + // exact comparison, and the difference decides whether an extra full DiT + // forward runs per step. + bool DoUnconditionalGeneration() const; + bool DoPerturbedGeneration() const; + bool DoIsolatedModalityGeneration() const; + bool ShouldSkipStep(int64_t step) const; +}; + +// MultiModalGuider.calculate (guiders.py:244-273). `uncond_*` may be null, which +// mirrors upstream's `torch.Tensor | float` union at its only reachable scalar +// value: a null stands for the float 0.0 that a disabled arm passes. +// +// The rescale (`:268-271`) divides by `pred.std()`. torch's `std` is the UNBIASED +// (N-1) estimator by default, and using the biased one instead is a small, +// everywhere, resolution-dependent gain error that no shape or finiteness check +// can see. +std::vector Ltx2MultiModalGuidance(const Ltx2MultiModalGuiderParams& params, + const float* cond, const float* uncond_text, + const float* uncond_perturbed, + const float* uncond_modality, int64_t count); + +// The seam a caller reaches for when it holds a configured kind: forwards to the +// three ported guiders and throws by name for the other three. +std::vector Ltx2Guidance(Ltx2GuiderKind kind, const Ltx2MultiModalGuiderParams& params, + const float* cond, const float* uncond_text, + const float* uncond_perturbed, const float* uncond_modality, + int64_t count); + +// One bin of the sigma-dependent factory (guiders.py:294-342). `sigma_upper_bound` +// is the bin's INCLUSIVE upper edge; bin i is `(key[i+1], key[i]]`. +struct Ltx2GuiderSigmaBin { + double sigma_upper_bound = 0.0; + Ltx2MultiModalGuiderParams params; +}; + +// _params_for_sigma_from_sorted_dict (guiders.py:214-230). `bins` need not be +// sorted — this sorts descending exactly as `from_dict` does (:329) — but it must +// be non-empty, which upstream also requires (:223-224, :327-328). +const Ltx2MultiModalGuiderParams& Ltx2GuiderParamsForSigma( + const std::vector& bins, double sigma); + +// --------------------------------------------------------------------------- +// Perturbations (guidance/perturbations.py) +// --------------------------------------------------------------------------- + +// perturbations.py:8-16. The VALUE is the row index into the mask tensor's dim 0, +// so these are not free to renumber. +enum class Ltx2PerturbationType { + kSkipVideoSelfAttn = 0, + kSkipAudioSelfAttn = 1, + kSkipA2vCrossAttn = 2, + kSkipV2aCrossAttn = 3, +}; +inline constexpr int64_t kLtx2PerturbationTypeCount = 4; + +// perturbations.py:19-33. `all_blocks` is upstream's `blocks is None`. +struct Ltx2Perturbation { + Ltx2PerturbationType type = Ltx2PerturbationType::kSkipVideoSelfAttn; + bool all_blocks = false; + std::vector blocks; +}; + +// perturbations.py:36-50. An empty list is "nothing perturbed"; it is what +// `PerturbationConfig.empty()` builds and what LTX-2.5 actually runs. +struct Ltx2PerturbationConfig { + std::vector perturbations; + bool IsPerturbed(Ltx2PerturbationType type, int64_t block) const; +}; + +// BatchedPerturbationConfig (perturbations.py:53-143): the per-block KEEP mask, +// 1 = keep and 0 = perturbed, indexed [type, block, sample]. The no-perturbation +// configuration — all ones — is the shipped LTX-2.5 path (spec section 2 routes +// STG through the guider), so it is the default here rather than a corner case. +class Ltx2BatchedPerturbationConfig { + public: + Ltx2BatchedPerturbationConfig() = default; + Ltx2BatchedPerturbationConfig(const std::vector& configs, + int64_t num_blocks); + // `BatchedPerturbationConfig.empty` (:134-143). + static Ltx2BatchedPerturbationConfig Empty(int64_t batch, int64_t num_blocks); + + int64_t batch() const { return batch_; } + int64_t num_blocks() const { return num_blocks_; } + // The whole [type, block, sample] mask, row-major. + const std::vector& block_masks() const { return masks_; } + // `mask` (:118-124): this block's per-sample keep mask for one type. + std::vector Mask(Ltx2PerturbationType type, int64_t block) const; + bool AnyInBatch(Ltx2PerturbationType type, int64_t block) const; + bool AllInBatch(Ltx2PerturbationType type, int64_t block) const; + // `batch_slice` (:111-116): a view over samples [start, end), rebuilt by + // slicing rather than by re-reading the config list. + Ltx2BatchedPerturbationConfig BatchSlice(int64_t start, int64_t end) const; + + private: + int64_t batch_ = 0; + int64_t num_blocks_ = 0; + std::vector masks_; +}; + +// --------------------------------------------------------------------------- +// Patchifiers (components/patchifiers.py) +// --------------------------------------------------------------------------- + +// VideoLatentShape (types.py:73-98) and AudioLatentShape (:134-160). +struct Ltx2VideoLatentShape { + int64_t batch = 1, channels = 0, frames = 0, height = 0, width = 0; +}; +struct Ltx2AudioLatentShape { + int64_t batch = 1, channels = 0, frames = 0, mel_bins = 0; +}; + +// SpatioTemporalScaleFactors.default() (types.py:31-33) — the Conv VAE's +// downsampling, which is what turns latent bounds into pixel timestamps. +struct Ltx2ScaleFactors { + int64_t time = 8, height = 32, width = 32; +}; + +// VideoLatentPatchifier (patchifiers.py:11-134). The temporal patch size is +// always 1 (:15) — `unpatchify` asserts it (:46) — so `patch_size` is the +// SPATIAL one only. +int64_t Ltx2VideoTokenCount(const Ltx2VideoLatentShape& shape, int64_t patch_size); +// `b c (f p1) (h p2) (w p3) -> b (f h w) (c p1 p2 p3)` (:31-38). Output is +// [batch, tokens, channels * patch_size^2]. +std::vector Ltx2VideoPatchify(const float* latent, const Ltx2VideoLatentShape& shape, + int64_t patch_size); +// `b (f h w) (c p q) -> b c f (h p) (w q)` (:52-60). +std::vector Ltx2VideoUnpatchify(const float* tokens, const Ltx2VideoLatentShape& shape, + int64_t patch_size); +// get_patch_grid_bounds (:64-134): [batch, 3, tokens, 2] of [start, end) bounds +// in (frame, height, width) order. +std::vector Ltx2VideoPatchBounds(const Ltx2VideoLatentShape& shape, int64_t patch_size); +// get_pixel_coords (:137-171). `causal_fix` rewrites the TEMPORAL axis only, and +// clamps at 0 — the first latent frame covers one pixel frame, not `time`. +std::vector Ltx2PixelCoords(const std::vector& latent_coords, int64_t batch, + int64_t tokens, const Ltx2ScaleFactors& factors, + bool causal_fix); + +// AudioPatchifier (patchifiers.py:174-353). The three rates below are its +// constructor defaults (:177-180) and they set the seconds-per-latent-frame the +// DiT's audio RoPE is indexed by, so a wrong one is a silently mistimed +// soundtrack rather than an error. +struct Ltx2AudioPatchifierParams { + int64_t sample_rate = 16000; + int64_t hop_length = 160; + int64_t audio_latent_downsample_factor = 4; + bool is_causal = true; + int64_t shift = 0; +}; +// `b c t f -> b t (c f)` (:301-306). +std::vector Ltx2AudioPatchify(const float* latent, const Ltx2AudioLatentShape& shape); +// `b t (c f) -> b c t f` (:325-331). +std::vector Ltx2AudioUnpatchify(const float* tokens, const Ltx2AudioLatentShape& shape); +// get_patch_grid_bounds (:334-353): [batch, 1, frames, 2] of [start, end) +// timestamps in SECONDS. +std::vector Ltx2AudioPatchTimings(const Ltx2AudioLatentShape& shape, + const Ltx2AudioPatchifierParams& params); + +// --------------------------------------------------------------------------- +// The recipes +// --------------------------------------------------------------------------- + +// parse_model_version (ltx-core loader/helpers.py:62-81): the dot-separated +// numeric PREFIX, stopping at the first non-numeric component, so "2.3.rc1" is +// (2, 3) and "banana" is {}. Callers that want "2.4-rc2" to compare equal to its +// own generation normalize the separator first, exactly as `detect_model_version` +// does (utils/constants.py:161) — `Ltx2DetectPipelineParams` does that here. +std::vector Ltx2ParseModelVersion(const std::string& version); + +// MultiModalGuiderParams for both streams plus the geometry defaults — +// PipelineParams (utils/constants.py:40-76). +struct Ltx2PipelineParams { + int64_t seed = 10; + int64_t stage_1_height = 512; + int64_t stage_1_width = 768; + int64_t num_frames = 121; + double frame_rate = 24.0; + int64_t num_inference_steps = 40; + int64_t default_image_crf = 33; + Ltx2MultiModalGuiderParams video_guider; + Ltx2MultiModalGuiderParams audio_guider; + + int64_t stage_2_height() const { return stage_1_height * 2; } + int64_t stage_2_width() const { return stage_1_width * 2; } +}; + +// The three generations upstream declares (utils/constants.py:80-124). +Ltx2PipelineParams Ltx2Params20(); +Ltx2PipelineParams Ltx2Params23(); +Ltx2PipelineParams Ltx2Params24(); +Ltx2PipelineParams Ltx2Params23Hq(); + +// detect_params (utils/constants.py:166-179) applied to a version STRING: the +// params of the newest generation that version is at or above. This is the rule +// that gives LTX-2.5 its parameters — (2,5) >= (2,4), so 2.5 inherits +// LTX_2_4_PARAMS — and it is why a 2.5 recipe can be written from upstream at all +// rather than being invented here. +Ltx2PipelineParams Ltx2DetectPipelineParams(const std::string& version); + +// distilled.py:62-84. Generation 2.5 and later sample STAGE 1 with the ancestral +// (SDE) Euler step; earlier ones use the deterministic one. Stage 2 is always +// deterministic — its 3-step refinement is too short to remove freshly injected +// noise (distilled.py:206-209). +inline constexpr int64_t kLtx2AncestralSinceMajor = 2; +inline constexpr int64_t kLtx2AncestralSinceMinor = 5; +inline constexpr double kLtx2AncestralEta = 1.0; +inline constexpr double kLtx2AncestralSNoise = 1.0; +// distilled.py:69-73. Offsets the loop's noise generator off the pipeline seed so +// its first draw is not bit-identical to the initial latent noise. +inline constexpr int64_t kLtx2AncestralNoiseSeedOffset = 10000; +bool Ltx2ShouldUseAncestralSampler(const std::string& version); + +// ltx2_recipes.py:38 — how a phase builds its input. +enum class Ltx2PhaseInputTransform { kInitial, kSpatialUpsample }; +// Which stepper a phase samples with (distilled.py:170-185). +enum class Ltx2StepperKind { kEuler, kEulerAncestral }; + +// LTXPhaseRecipe (ltx2_recipes.py:29-50). +struct Ltx2PhaseRecipe { + std::string name; + Ltx2MultiModalGuiderParams video_guidance; + Ltx2MultiModalGuiderParams audio_guidance; + int64_t spatial_downscale = 1; + // Empty means "derive the schedule from the scheduler at run time"; a non-empty + // one is an explicit distilled schedule and fixes `num_inference_steps`. + std::vector sigmas; + double noise_scale = 0.0; + Ltx2PhaseInputTransform input_transform = Ltx2PhaseInputTransform::kInitial; + bool allow_guidance_override = true; + bool use_official_sigma_schedule = true; + Ltx2StepperKind stepper = Ltx2StepperKind::kEuler; + double stepper_eta = 0.0; + double stepper_s_noise = 1.0; + int64_t noise_seed_offset = 0; + + // ltx2_recipes.py:48-50: `None` when the schedule is not explicit. -1 here. + int64_t num_inference_steps() const; +}; + +// LTXPipelineRecipe (ltx2_recipes.py:53-87). +struct Ltx2PipelineRecipe { + std::vector phases; + int64_t height = 512; + int64_t width = 768; + int64_t num_frames = 121; + double frame_rate = 24.0; + int64_t num_inference_steps = 40; + // Not a vLLM-Omni field: Lightricks resolves the conditioning-image CRF per + // generation (utils/constants.py:36-37, 124) and it is a property of the model + // generation, so it travels with the recipe rather than being re-derived. + int64_t default_image_crf = 33; + std::string negative_prompt; + int64_t video_output_phase = -1; + int64_t audio_output_phase = -1; + bool allow_request_sigmas = true; + bool allow_request_latents = true; + bool allow_negative_prompt = true; + bool fixed_num_inference_steps = false; + + int64_t max_spatial_downscale() const; +}; + +// resolve_ltx_pipeline_recipe (ltx2_recipes.py:170-175). Keyed on the EXACT +// (pipeline_kind, model_version) pair and throwing by name on anything else — +// never defaulting. The table: +// +// ("one_stage", "2") vLLM-Omni LTX2_ONE_STAGE_RECIPE (:109-111) +// ("one_stage", "2.3") vLLM-Omni LTX23_ONE_STAGE_RECIPE (:112-115) +// ("one_stage", "2.4") Lightricks LTX_2_4_PARAMS (constants.py:124) +// ("one_stage", "2.5") Lightricks, via _PARAMS_SINCE_VERSION (:130-133) +// ("distilled_two_stage","2") vLLM-Omni LTX2_DISTILLED_TWO_STAGE_RECIPE (:125-158) +// ("distilled_two_stage","2.5") Lightricks distilled.py + constants.py:17-23 +// ("dmd2", "2") vLLM-Omni LTX_POSITIVE_ONLY_RECIPE (:116-124) +// ("dmd2", "2.3") same +// +// The 2.4 and 2.5 rows exist here and not upstream in vLLM-Omni, which carries no +// row past 2.3 (spec section 3). Their VALUES are Lightricks', not invented: the +// one_stage rows are `detect_params`' own resolution of that version, and the +// distilled 2.5 row is the 2.0 one plus `should_use_ancestral_sampler`, which is +// the single thing distilled.py changes for generation 2.5. +Ltx2PipelineRecipe ResolveLtx2PipelineRecipe(const std::string& pipeline_kind, + const std::string& model_version); + +// --------------------------------------------------------------------------- +// Out of scope for L5, refused by name (spec section 2, "Out") +// --------------------------------------------------------------------------- + +// Each of these renders something plausible if it is silently downgraded, which +// is why none of them falls back. `Ltx2RefuseUnportedPipelineFeature` throws with +// a message naming the missing piece and the phase or row that owes it. +enum class Ltx2UnportedPipelineFeature { + kTemporalUpsampler, // model/upsampler with temporal_upsample=True + kLoraFusion, // loader/LoraPathStrengthAndSDOps + kMultishot, // ltx-pipelines multishot entry points + kInt8ConvRot, // ComfyUI-only quantization + kCfgParallelism, // ltx-pipelines/multigpu + kVideoEngineWiring, // end-to-end through vllm::multimodal::VideoEngine (L7) + kBetaScheduler, // components/schedulers.py:91-120 +}; +[[noreturn]] void Ltx2RefuseUnportedPipelineFeature(Ltx2UnportedPipelineFeature feature); + +} // namespace vllm diff --git a/include/vllm/model_executor/models/ltx2_text_encoder.h b/include/vllm/model_executor/models/ltx2_text_encoder.h new file mode 100644 index 000000000..8ecf6af73 --- /dev/null +++ b/include/vllm/model_executor/models/ltx2_text_encoder.h @@ -0,0 +1,606 @@ +// LTX-2.5 TEXT CONDITIONING — the Gemma-4 multi-layer feature aggregation, the +// two caption projections, and the embedded tokenizer/asset pack. +// +// ─── THE ONE THING THAT MAKES THIS DIFFERENT FROM EVERY OTHER TEXT ENCODER ──── +// +// LTX-2.5 does NOT condition on the encoder's last hidden state. It takes EVERY +// Gemma-4 hidden state — the embedding output plus all 48 decoder outputs, 49 in +// total — stacks them on a new LAST axis to [batch, seq, hidden, layers], +// normalizes, concatenates ACROSS THE LAYER AXIS, and projects the flattened +// result twice (feature_extractor.py:114-129). +// +// Measured on the shipped `vonkaiser/LTX-2.5-FP8-NVFP4` text encoder +// (`gemma4-12b-with-proj-nvfp4-torchao.safetensors`, 1688 tensors): +// +// text_embedding_projection.video_aggregate_embed.weight U8 [4096, 94080] +// text_embedding_projection.audio_aggregate_embed.weight U8 [2048, 94080] +// model.embed_tokens.weight U8 [262144, 1920] +// model.norm.weight BF16 [3840] +// model.layers.{0..47}.* 48 layers +// +// NVFP4 packs TWO values per byte, so those U8 widths are HALF the real feature +// counts: the projections take 188160 = 3840 x 49 inputs, not 94080, and the +// Gemma hidden size is 3840, not 1920. `model.norm.weight [3840]` is the +// independent confirmation — it is stored BF16 and therefore unpacked. +// +// ─── WHAT THIS IS A PORT OF (file:line on BOTH sides) ──────────────────────── +// Upstream root: Lightricks/LTX-2, packages/ltx-core/src/ltx_core/ +// OURS <- UPSTREAM +// Ltx2StackHiddenStates <- text_encoders/gemma/feature_extractor.py:120 +// Ltx2NormAndConcatPaddedBatch <- text_encoders/gemma/feature_extractor.py:12-45 +// Ltx2NormAndConcatPerTokenRms <- text_encoders/gemma/feature_extractor.py:48-64 +// Ltx2RescaleNorm <- text_encoders/gemma/feature_extractor.py:67-69 +// Ltx2TextFeatureExtractorForward <- text_encoders/gemma/feature_extractor.py:85-129 +// Ltx2SelectTextFeatureVariant <- text_encoders/gemma/encoders/encoder_configurator.py:163-209 +// Ltx2ConvertToAdditiveMask <- text_encoders/gemma/embeddings_processor.py:16-20 +// Ltx2ComputeRightPadOrder <- text_encoders/gemma/embeddings_processor.py:23-38 +// Ltx2ApplyRightPadOrder <- text_encoders/gemma/embeddings_processor.py:41-43 +// Ltx2ToBinaryMask <- text_encoders/gemma/embeddings_processor.py:46-48 +// Ltx2TextEncoderConditioning <- text_encoders/gemma/embeddings_processor.py:70-117 +// Ltx2LoadGemmaAssets <- text_encoders/gemma/gemma_assets.py:104-159 +// Ltx2GemmaHiddenStateContract <- text_encoders/gemma/encoders/base_encoder.py:49-71 +// +// ─── THE FOUR THINGS THAT FAIL SILENTLY ────────────────────────────────────── +// * THE CONCATENATION IS HIDDEN-MAJOR, LAYER-MINOR. `stack(..., dim=-1)` then +// `.reshape(B, T, D*L)` puts layer `l` of channel `d` at flat index +// `d * L + l`. A port that concatenates layer-major (`l * D + d`) produces a +// correctly shaped, finite, PERMUTED conditioning vector. +// * THERE ARE TWO NORMALIZATION VARIANTS AND THEY ARE NOT INTERCHANGEABLE. +// V1 is a per-batch, per-layer masked mean/range with an `8 *` scale and +// eps 1e-6; V2 is a per-token RMS over the HIDDEN axis with eps 1e-6 and no +// scale. The choice comes from config (`Ltx2SelectTextFeatureVariant`), never +// from a guess. +// * THE `+1` IS THE EMBEDDING LAYER. `num_layers = num_hidden_layers + 1` +// (encoder_configurator.py:182). Dropping it makes the projection 3840 inputs +// too narrow, which a shape check catches; taking the LAST 48 of the 49 +// instead of the first 48 plus the embedding does NOT change any shape. +// * V1'S PROJECTION HAS NO BIAS AND V2'S HAVE ONE (encoder_configurator.py:187, +// 206-208). Because the norm zeroes padded positions, a padded position's +// PROJECTED value is exactly the bias — not zero. A port that force-zeroes the +// projected pads silently diverges from upstream on every padded row. The +// DECLARED contract below (`aggregate_bias`, `*_out_features`) is therefore +// checked against the weights the loader actually supplied — see +// `Ltx2TextFeatureExtractorForward`. +// +// ─── AND THE EPSILONS, WHICH ARE A CLASS AND NOT ONE INSTANCE ──────────────── +// A constant that only changes the answer on a DEGENERATE input is invisible to a +// golden built from random values — but NEITHER of the two below is that +// constant, and the detailed note at kLtx2TextNormV1Eps already says why for the +// V1 half. Both are additive terms on an O(1) denominator against a 1e-5 band, so +// at the 100x bar they move ORDINARY random-value goldens: V1 1e-6 -> 1e-4 REDS +// "`_norm_and_concat_padded_batch`, both padding sides" at 0.000524044 and +// "FeatureExtractorV1" at 7.53999e-05 / 6.61612e-05; V2 1e-6 -> 1e-4 REDS +// "`norm_and_concat_per_token_rms`, both padding sides" at 0.00232971 and +// carries into "FeatureExtractorV2" and the hand-off at 0.000344872 / +// 0.000259042 / 0.00039053. They are held two ways all the same: their VALUE +// against upstream measured by probe, and the degenerate input on which each is +// the only thing between the port and a division by zero — that input is what +// makes a division-by-zero visible, not what makes the constant visible at all. +// When a fourth epsilon arrives, it owes the same pair — +// not a comment saying it matches upstream. +// +// ─── DTYPE ─────────────────────────────────────────────────────────────────── +// Everything here is f32, exactly as phase L2 (ltx2.h) records for the DiT: that +// is NOT a widening of a bf16 path, it is the PARITY dtype of this gate, which +// compares the ALGORITHM against upstream run in torch float32. Upstream resolves +// ONE model dtype and every layer inherits it — `LTXGemmaTextEncoder` takes a +// single `dtype` (base_encoder.py:41) and `FeatureExtractorV2.forward` casts the +// normalized tensor straight back to `encoded.dtype` (feature_extractor.py:122), +// so the production bf16 / FP8 / NVFP4 arms are a single stream-dtype choice — +// phase L6 — and are OWED, not shipped. Every entry point that takes a +// `compute_dtype` REFUSES anything but `vt::DType::kF32` with a message naming +// the missing phase rather than silently computing in f32. +#pragma once + +#include +#include +#include +#include + +#include + +#include "vllm/model_executor/models/gemma4.h" +#include "vt/dtype.h" + +namespace vllm { + +class SafetensorsFile; +namespace tok { +class Tokenizer; +} // namespace tok + +// ─────────────────────────── the hidden-state contract ─────────────────────── + +// Which hidden states LTX consumes, and in which order. +// +// `LTXGemmaTextEncoder.encode` (base_encoder.py:68-71) calls the inner Gemma +// model with `output_hidden_states=True` and passes `outputs.hidden_states` +// straight through. In transformers that tuple has `num_hidden_layers + 1` +// entries in this exact order: +// +// [0] the token embeddings, AFTER the sqrt(hidden) embed scale +// [1 .. L-1] the output of decoder layers 0 .. L-2 +// [L] the output of decoder layer L-1 AFTER `model.norm` +// +// The last entry is FINAL-NORMED and the raw output of the last decoder layer +// never appears. A port that appends L raw layer outputs plus the embeddings, or +// that appends the raw last layer and then the normed one, has 49 finite tensors +// of the right shape and the wrong content. +struct Ltx2GemmaHiddenStateContract { + static constexpr const char* kOrder = + "hidden_states[0] = embeddings * sqrt(hidden); hidden_states[i] = output of " + "decoder layer i-1; hidden_states[num_hidden_layers] = model.norm(output of " + "the LAST decoder layer). Count = num_hidden_layers + 1."; + // The count the caption projections' in_features must agree with. + static int64_t Count(int64_t num_hidden_layers) { return num_hidden_layers + 1; } +}; + +// A batch of Gemma hidden states, one pointer per layer, each [batch, seq, hidden] +// in row-major order. `layers.size()` must equal +// `Ltx2GemmaHiddenStateContract::Count(num_hidden_layers)`. +struct Ltx2TextHiddenStates { + std::vector layers; + int64_t batch = 0; + int64_t seq = 0; + int64_t hidden = 0; +}; + +// ───────────────────────────── feature aggregation ─────────────────────────── + +// feature_extractor.py:28 — ONE `eps = 1e-6` bound at the top of +// `_norm_and_concat_padded_batch` and used TWICE: in the mean's denominator +// (`denom` is defined at :34, the `+ eps` is at :35) and in the range's +// (`range_ + eps`, :41). Named here so there is a single thing to pin, and pinned +// against the value MEASURED out of upstream in +// tests/vllm/models/test_ltx2_text_encoder.cpp. +// +// Reachability, stated because it is what makes the pin necessary, and stated +// CORRECTLY because an earlier revision of this comment got it wrong in the +// direction that makes a reader relax: +// +// * `range_ + eps` (:41) is reachable only when a whole (batch, layer) slice is +// CONSTANT over its valid positions, so `range_` collapses to 0. +// * `denom + eps` (:35) is DIVISION-BY-ZERO-reachable only when a batch row has +// no valid token. But it is OUTPUT-OBSERVABLE far more widely, because its +// DTYPE is observable: `denom` is an int64 tensor and `eps` a python float, so +// upstream adds them in FLOAT32, and an f64 add differs by one f32 ulp in the +// mean. `range_ + eps` then amplifies that by 8/eps = 8e6 as `range_` -> 0. On +// a constant 0.5 stack under a 3-valid-token mask, upstream reads 0.476837158 +// and an f64 denominator reads 0.238418579 — 23842x the suite's kTol. +// +// So this constant is UNOBSERVABLE at the output only on an all-pad row or an +// all-zero mask, where :44-45 zeroes every position before anything escapes. On a +// realistic Gemma workload `range_` is O(1), so the same one-ulp perturbation +// lands around 2e-12 at the output — small, but not the "any input" the earlier +// comment claimed. Both the value and the arithmetic width are gated on the +// degenerate inputs, where they are visible. +// +// That is the ONE-ULP dtype perturbation. A VALUE move of the size the pin +// exists to catch is far louder and needs no degenerate input at all: 1e-6 -> +// 1e-4 REDS "`_norm_and_concat_padded_batch`, both padding sides" at 0.000524044 +// and "FeatureExtractorV1" at 7.53999e-05 / 6.61612e-05, on the ordinary +// random-value goldens. +inline constexpr double kLtx2TextNormV1Eps = 1e-6; + +// feature_extractor.py:61 — `torch.rsqrt(variance + 1e-6)`. DIVISION-BY-ZERO- +// reachable only when a token's whole hidden slice is zero, which is the narrow +// claim the earlier "reachable only when" was making and the wrong one to state +// alone: the epsilon is added to an O(1) variance, so it is OUTPUT-observable on +// ordinary random values too. 1e-6 -> 1e-4 REDS +// "`norm_and_concat_per_token_rms`, both padding sides" at 0.00232971, and the +// perturbation carries through the projections into "FeatureExtractorV2" and +// "the encoder -> conditioning hand-off" at 0.000344872 / 0.000259042 / +// 0.00039053 against the suite's 1e-5 kTol. Same shape as the V1 half above. +inline constexpr float kLtx2TextNormV2Eps = 1e-6f; + +// feature_extractor.py:12-64. Which of the two normalizations runs. +enum class Ltx2TextNormVariant { + // `_norm_and_concat_padded_batch` (:12-45) — per-batch, per-layer masked mean + // and range, `8 * (x - mean) / (range + 1e-6)`. The 19B / V1 checkpoints. + kPaddedBatchV1, + // `norm_and_concat_per_token_rms` (:48-64) — per-token RMS over the HIDDEN + // axis, `x * rsqrt(mean(x^2) + 1e-6)`. Upstream's docstring: "for V2 models". + kPerTokenRmsV2, +}; + +// The resolved feature-extractor shape. Produced by `Ltx2SelectTextFeatureVariant` +// from the checkpoint config; never hand-assembled on a model path. +struct Ltx2TextFeatureConfig { + Ltx2TextNormVariant variant = Ltx2TextNormVariant::kPerTokenRmsV2; + int64_t embedding_dim = 0; // gemma_text_config.hidden_size (3840) + int64_t num_layers = 0; // num_hidden_layers + 1 (49) + int64_t video_out_features = 0; // video_aggregate_embed.out_features (4096) + int64_t audio_out_features = 0; // audio_aggregate_embed.out_features (2048); 0 = absent + bool aggregate_bias = false; // V1 false (:187), V2 true (:206-208) + bool is_av = false; // V1 only (:188): the audio arm IS the video tensor + int64_t FlatDim() const { return embedding_dim * num_layers; } +}; + +// encoder_configurator.py:163-209 — the selection, mirrored including both of its +// refusals. `transformer_config` is the diffusion checkpoint's `config.transformer` +// object. +// +// none of the four V2 marker keys present -> V1 (projection lives in the DiT) +// all four present with their exact expected values -> V2 +// a partial set, or a drifted value -> throws std::runtime_error naming the keys +// +// Never infers the variant from tensor shapes: 3840 x 49 is the flat width under +// BOTH variants, so shapes cannot distinguish them. +Ltx2TextFeatureConfig Ltx2SelectTextFeatureVariant( + const nlohmann::json& transformer_config, int64_t gemma_hidden_size, + int64_t gemma_num_hidden_layers); + +// feature_extractor.py:120 — `torch.stack(hidden_states, dim=-1)`. +// Output is [batch, seq, hidden, layers], layer being the LAST (fastest) axis. +std::vector Ltx2StackHiddenStates(const Ltx2TextHiddenStates& states); + +// feature_extractor.py:12-45. `stacked` is [B, T, D, L]; `mask` is [B, T] in +// {0, 1}. Returns [B, T, D * L] with PADDED POSITIONS ZEROED. Padding-side +// agnostic — the binary mask alone decides which positions are valid. +std::vector Ltx2NormAndConcatPaddedBatch(const float* stacked, + const int32_t* mask, int64_t batch, + int64_t seq, int64_t hidden, + int64_t layers); + +// feature_extractor.py:48-64. Same shapes; per-token RMS over the hidden axis. +// Padded positions ZEROED. +std::vector Ltx2NormAndConcatPerTokenRms(const float* stacked, + const int32_t* mask, int64_t batch, + int64_t seq, int64_t hidden, + int64_t layers); + +// feature_extractor.py:67-69 — `x * sqrt(target_dim / source_dim)`, computed with +// the same `math.sqrt` of a double ratio upstream uses. V2 only. +double Ltx2RescaleNorm(int64_t target_dim, int64_t source_dim); + +// One caption projection: `torch.nn.Linear(flat_dim, out_features, bias=...)`. +// `weight` is row-major [out_features, in_features] — torch's own layout, so the +// checkpoint tensor is used as stored. +struct Ltx2TextAggregateEmbed { + std::vector weight; + std::vector bias; // empty when the Linear has bias=False + int64_t out_features = 0; + int64_t in_features = 0; +}; + +// The text encoder's projection weights. V1 populates `video` only and reports +// the same tensor for audio (`is_av`); V2 populates both. +struct Ltx2TextEncoderWeights { + Ltx2TextAggregateEmbed video; // text_embedding_projection.video_aggregate_embed + // (V1: .aggregate_embed) + Ltx2TextAggregateEmbed audio; // text_embedding_projection.audio_aggregate_embed +}; + +// The extractor output. `audio` is empty when the config has no audio projection; +// under V1's `is_av` it is a COPY of `video`, matching upstream returning the same +// tensor twice (feature_extractor.py:95-96). +struct Ltx2TextFeatures { + std::vector video; // [batch, seq, video_out_features] + std::vector audio; // [batch, seq, audio_out_features] or empty +}; + +// feature_extractor.py:85-129 — the whole extractor: stack, normalize by the +// selected variant, (V2) rescale per projection, project. `compute_dtype` must be +// vt::DType::kF32 — see the DTYPE note at the top of this header. +// +// REFUSES, by name, any disagreement between what `config` DECLARES and what +// `weights` actually carries: `aggregate_bias` vs `w.bias.empty()`, +// `*_out_features` vs `w.out_features`, and `FlatDim()` vs `w.in_features`. +// Upstream builds both Linears from the one config object +// (encoder_configurator.py:187, 206-208) and so cannot disagree with itself; a +// port that loads the config and the tensors separately can. The concrete case is +// a loader that reads `video_aggregate_embed.weight` (U8/NVFP4) and misses +// `.bias` (BF16, a different unpack path) while the config still says bias=True: +// every conditioning row is then shifted by the missing bias and every padded row +// projects to 0 rather than to the bias — finite, correctly shaped, wrong prompt. +Ltx2TextFeatures Ltx2TextFeatureExtractorForward( + const Ltx2TextHiddenStates& states, const int32_t* mask, + const Ltx2TextEncoderWeights& weights, const Ltx2TextFeatureConfig& config, + vt::DType compute_dtype = vt::DType::kF32); + +// ──────────────────── the encoder -> conditioning hand-off ─────────────────── + +// embeddings_processor.py:16-20 — `(mask - 1) * finfo(f32).max`, i.e. 0.0 for a +// kept position and -FLT_MAX for a pad. Returns [batch, 1, 1, seq] flattened. +std::vector Ltx2ConvertToAdditiveMask(const int32_t* mask, int64_t batch, + int64_t seq); + +// embeddings_processor.py:23-38 — the STABLE descending argsort of the binary +// mask that places valid positions before pads while preserving their relative +// order. Idempotent on an already right-padded input. Fills `sort_index` +// [batch, seq] and `reordered_additive_mask` [batch, 1, 1, seq]. +void Ltx2ComputeRightPadOrder(const float* additive_mask, int64_t batch, + int64_t seq, std::vector& sort_index, + std::vector& reordered_additive_mask); + +// embeddings_processor.py:41-43 — gather `features` [batch, seq, dim] along seq +// by `sort_index`. +std::vector Ltx2ApplyRightPadOrder(const float* features, + const int32_t* sort_index, int64_t batch, + int64_t seq, int64_t dim); + +// embeddings_processor.py:46-48 — `(encoded_mask < 1e-6)` as {0, 1}, [batch, seq]. +// +// MEASURED, and gated as measured: BOTH masks upstream can hand this function +// satisfy the predicate everywhere. With learnable registers on — which LTX-2.5 +// has — the connector returns `zeros_like(additive_mask)` (embeddings_connector.py:152) +// and 0.0 < 1e-6; with them off it returns the additive mask and -FLT_MAX < 1e-6 +// too. So the mask `EmbeddingsProcessor` hands the DiT is ALL ONES. That is +// upstream's behaviour, not ours to repair. +std::vector Ltx2ToBinaryMask(const float* encoded_mask, int64_t batch, + int64_t seq); + +// The conditioning `EmbeddingsProcessor.process_hidden_states` produces, up to +// the connector call. `video`/`audio` are RIGHT-PAD ORDERED features ready for +// `Embeddings1DConnector`; `additive_mask` is the matching reordered mask. +struct Ltx2TextConditioning { + std::vector video; // [batch, seq, video_out_features] + std::vector audio; // [batch, seq, audio_out_features] or empty + std::vector additive_mask; // [batch, 1, 1, seq] + std::vector sort_index; // [batch, seq] +}; + +// embeddings_processor.py:70-117, minus the two connector calls. +// +// WHERE THE CONNECTOR IS, corrected 2026-08-13. This note used to say +// `Embeddings1DConnector` (embeddings_connector.py:74-191) was NOT ported and +// that this function therefore stopped at its INPUT contract. Phase L5 ported it +// (`Ltx2ConnectorForward`, ltx2_connector.h), phase L9c put it on the render +// path with the checkpoint's own weights, and phase L13 runs this function's +// output through it per request. The stopping point is unchanged — this is still +// the processor MINUS the connector calls — but "the connector does not exist" +// was true only until L5, and a stale owed-note is how a later refusal came to +// cite a missing piece that had landed. +// +// ONE OVERLAP A CALLER MUST KNOW ABOUT. `Ltx2ConnectorCreateEmbeddings` is the +// OTHER port of embeddings_processor.py:23-43 and carries the right-pad sort +// too, so feeding this function's output into it sorts an already-sorted stream. +// That composes to the identity — a stable descending argsort of a 0/1 key is +// idempotent — and `Ltx2VideoEngine::Generate` asserts the precondition rather +// than assuming it. +Ltx2TextConditioning Ltx2TextEncoderConditioning( + const Ltx2TextHiddenStates& states, const int32_t* mask, + const Ltx2TextEncoderWeights& weights, const Ltx2TextFeatureConfig& config, + vt::DType compute_dtype = vt::DType::kF32); + +// ───────────────────── the embedded tokenizer / asset pack ─────────────────── + +// gemma_assets.py:58-159. The LTX-2.5 text encoder ships as ONE .safetensors file +// with its HuggingFace assets stored AS TENSORS, which is unusual enough that a +// loader assuming a sibling `tokenizer.json` fails on it: +// +// tokenizer_json U8 [32169626] ~32 MB, the whole tokenizer +// hf_asset__tokenizer_config.json U8 [3736] +// hf_asset__processor_config.json U8 [1382] +// hf_asset__generation_config.json U8 [255] +// hf_asset__chat_template.jinja U8 [18683] +// +// and the HF config itself in the file's `__metadata__` under `gemma_config`. +struct Ltx2GemmaAssets { + std::vector tokenizer_json; + // Sidecar name (the part after `hf_asset__`) -> raw bytes. + std::map> sidecars; + // The parsed `__metadata__["gemma_config"]` JSON. Null when `require_config` + // was false and the file carried no metadata. + nlohmann::json config; + bool has_config = false; + + // gemma_assets.py:144-151. + const std::vector& SidecarBytes(const std::string& name) const; + nlohmann::json SidecarJson(const std::string& name) const; +}; + +// gemma_assets.py:34-36 — the two names the pack format is keyed on. +inline constexpr const char* kLtx2GemmaTokenizerTensor = "tokenizer_json"; +inline constexpr const char* kLtx2GemmaAssetPrefix = "hf_asset__"; + +// gemma_assets.py:104-142 + `_require_sidecars` (:153-159). Throws +// std::runtime_error when `tokenizer_json` is missing or when either REQUIRED +// sidecar (`tokenizer_config.json`, `processor_config.json`, gemma_assets.py:38-41) +// is absent. +// +// MEASURED FINDING, reported rather than worked around: the shipped +// `vonkaiser/LTX-2.5-FP8-NVFP4` text encoder carries NO `__metadata__` block at +// all, so upstream's `GemmaAssets.from_single_file` raises on it before it reads a +// single tensor (gemma_assets.py:110-114). `require_config` mirrors that refusal +// by default; a caller that has the Gemma config from elsewhere passes false and +// gets the tensors, with `has_config` reporting which happened. +Ltx2GemmaAssets Ltx2LoadGemmaAssets(const SafetensorsFile& file, + bool require_config = true); + +// ─────────────────────── the prompt -> tokens hand-off ─────────────────────── +// +// `LTXGemmaTokenizer.tokenize_with_weights` (tokenizer.py:31-59), which is the +// ONLY tokenization on the conditioning path. Both references agree that no chat +// template is applied here — the template belongs to the separate prompt +// ENHANCEMENT path (base_encoder.py:100, diffusers pipeline_ltx2.py:624), which +// produces a plain string that then comes back through this function. + +// gemma_assets.py:162 — `TOKENIZER_MAX_LENGTH = 1024`, bound at +// base_encoder.py:231-236. Confirmed independently by diffusers, whose +// `_get_gemma_prompt_embeds` defaults `max_sequence_length: int = 1024` +// (pipeline_ltx2.py:304). +inline constexpr int64_t kLtx2GemmaTokenizerMaxLength = 1024; + +// base_encoder.py:235 — `PaddingSide.LEFT`. diffusers sets the same thing +// explicitly with the comment "Gemma expects left padding for chat-style +// prompts" (pipeline_ltx2.py:328-329). +// +// This is why `Ltx2GemmaPromptTokens` reports `first_valid` rather than assuming +// position 0: the valid tokens are the TAIL, and their absolute positions start +// at the pad count, not at zero. +enum class Ltx2GemmaPaddingSide { kLeft, kRight }; + +// One tokenized prompt, in upstream's `[max_length]` shape. +struct Ltx2GemmaPromptTokens { + // [max_length]. Pad positions carry `pad_id`. + std::vector input_ids; + // [max_length] in {0, 1} — the second element of upstream's (token, weight) + // pairs (tokenizer.py:57-59), used as the attention mask at + // base_encoder.py:65-67. + std::vector attention_mask; + // Index of the first valid token, and how many there are. On the LEFT-padded + // default `first_valid` is the pad count and the valid run is the tail. + int64_t first_valid = 0; + int64_t num_valid = 0; + // True when the prompt was longer than `max_length` and lost tokens. + bool truncated = false; +}; + +// tokenizer.py:31-59, mirrored including the parts that look like details: +// +// * `text.strip()` first (:33). diffusers strips too (pipeline_ltx2.py:333). +// * encode, then PREPEND BOS if it is not already first — CONDITIONAL, on +// upstream's own `if not input_ids or input_ids[0] != bos_id` guard +// (:44-46). A port that prepends unconditionally doubles the BOS. +// +// Two things about that, and the first one is a KNOWN DIVERGENCE rather than +// a mirrored default. Upstream calls `self.tokenizer(text, ...)` — `__call__` +// with its default `add_special_tokens=True` (tokenizer.py:37-43) — so +// upstream DOES run the post_processor and we call plain `Encode`, which +// does not. On THIS checkpoint the two are identical, because the shipped +// `post_processor` is a TemplateProcessing whose `special_tokens` map is +// EMPTY and whose template is the bare sequence, so it has nothing to add: +// measured on the shipped file, not assumed. If a future checkpoint ships a +// post_processor that DOES add something, upstream would emit it and we +// would not — so this is the line to change, not a property to rely on. +// +// What the two references actually disagree about is narrower than "one +// runs the post-processor": both let it run. `ltx_core` ALSO prepends BOS +// explicitly and says why — "Gemma 3 already emits it via post_processor; +// Gemma 4 does not, so we prepend" (tokenizer.py:12-15) — while diffusers +// relies on the post_processor alone (pipeline_ltx2.py:339), which for this +// tokenizer.json adds nothing, so following diffusers would drop token 0 of +// every prompt. `ltx_core` is the model author's own runtime and is explicit +// about the case, so it is the one followed. +// * EOS is never appended (:14). +// * truncation happens BEFORE the BOS prepend and again after (:41, :46), so a +// maximal prompt loses its LAST token to make room for BOS rather than +// losing the BOS. +// * pad to exactly `max_length` on `padding_side` (:48-54). +// +// Throws std::runtime_error when `bos_id` is negative — upstream raises for the +// same reason at tokenizer.py:34-36, because a conditioning path with no BOS is +// a different prompt, not a degraded one. +Ltx2GemmaPromptTokens Ltx2TokenizeGemmaPrompt( + const tok::Tokenizer& tokenizer, const std::string& prompt, int32_t bos_id, + int32_t pad_id, int64_t max_length = kLtx2GemmaTokenizerMaxLength, + Ltx2GemmaPaddingSide padding_side = Ltx2GemmaPaddingSide::kLeft); + +// The two ids upstream reads off the tokenizer/config rather than hardcoding. +// MEASURED on the shipped checkpoint's own `hf_asset__generation_config.json`: +// `bos_token_id` 2, `pad_token_id` 0 — and its `tokenizer_json` added_tokens +// agree (`` 0, `` 1, `` 2). +struct Ltx2GemmaSpecialIds { + int32_t bos_id = -1; + int32_t pad_id = -1; +}; + +// Resolve the two ids from the asset pack, or throw naming which one is missing. +// Reads `hf_asset__generation_config.json` first (that is where the checkpoint +// states them) and falls back to the tokenizer's own added-token table, which is +// what `LTXGemmaTokenizer` does through `tokenizer.bos_token_id` +// (tokenizer.py:35) and `tokenizer.pad_token` (:26-27). +Ltx2GemmaSpecialIds Ltx2ResolveGemmaSpecialIds(const Ltx2GemmaAssets& assets, + const tok::Tokenizer& tokenizer); + +// ───────────────────────────── the Gemma-4 TOWER ───────────────────────────── +// +// Phase L6 loaded the two caption projections and the asset pack and recorded +// "wiring the tower's torchao arm onto `Gemma4Weights` is owed and named as such +// rather than half-done" (ltx2_loader.h:325-329). This is that wiring. +// +// WHERE THE CONFIG COMES FROM, because this checkpoint cannot answer it. +// The shipped `vonkaiser` NVFP4 build carries NO `__metadata__` block at all, so +// upstream's own `GemmaAssets.from_single_file` raises on it before reading a +// tensor (gemma_assets.py:110-114) and there is nothing in the file to read a +// Gemma config out of. The config is therefore an INPUT, and a caller that has +// none gets a refusal naming the missing piece rather than a plausible default. +// +// The fields that no shape encodes, and that a default would get wrong: +// +// layer_types which layers are full vs sliding. The shipped +// tower is (sliding x 5, full) x 8, and the two +// have DIFFERENT geometry. +// global_head_dim 512 on the full layers against head_dim 256. +// num_global_key_value_heads ONE, against 8 on the sliding layers. +// attention_k_eq_v true — the full layers ship no `v_proj` at all. +// rope_parameters two rope types at two thetas, partial rotary +// 0.25 on the full arm only. +// rms_norm_eps, sliding_window, final_logit_softcapping +// +// Each moves every hidden state while leaving the tensor set byte-identical, so +// a wrong config resolves a DIFFERENT MODEL out of the same file and nothing +// downstream can tell. That is why this takes the config rather than inferring +// it: shapes can confirm a config, and cannot supply one. + +// The tower, materialized. `weights` is bf16 throughout — the NVFP4 modules are +// dequantized on the way in, which is what `Gemma4Model::ForwardHiddenStates` +// consumes and what upstream's own resolved model dtype is +// (base_encoder.py:41). At the shipped 12B that is ~24 GB of host memory, and +// the caller is told so here rather than discovering it. +struct Ltx2GemmaTower { + HfConfig config; + Gemma4Weights weights; + // Every module that arrived NVFP4-packed and was dequantized, in header order. + // A tower whose modules are all bf16 already leaves this empty. + std::vector dequantized_modules; +}; + +// Materialize the Gemma-4 tower from an LTX text-encoder safetensors file. +// +// Reads `model.embed_tokens.*`, `model.norm.weight` and `model.layers.{i}.*` and +// IGNORES everything else the file carries — `vision_model.*`, +// `multi_modal_projector`, `audio_projector` and the two caption projections are +// all present in the shipped checkpoint and none of them is on the text path. +// Ignoring is deliberate and not laziness: choking on their presence would make +// the only shipped checkpoint unreadable. +// +// Each module is taken bf16 when it is stored bf16 and dequantized through +// `Ltx2DequantTorchaoNvfp4ToBf16` when it carries a `torchao_nvfp4` marker. +// A module in neither form throws BY NAME. +// +// REFUSES, by name, rather than loading something shaped like a tower: +// * a `v_proj` present on a layer the config says is `attention_k_eq_v`, or +// absent on one it does not — the two cases are 16 kv heads apart; +// * any per-layer tensor whose width disagrees with the geometry the config +// resolves for THAT layer; +// * `hidden_size_per_layer_input` > 0, i.e. a PLE tower, since this checkpoint +// family ships no `embed_tokens_per_layer` and a silently-absent PLE is a +// different model. +Ltx2GemmaTower Ltx2LoadGemmaTowerFromSafetensors(const SafetensorsFile& file, + const nlohmann::json& gemma_config); + +// ─────────────────────── prompt -> conditioning, end to end ────────────────── +// +// `LTXGemmaTextEncoder.encode` (base_encoder.py:49-71) followed by +// `EmbeddingsProcessor.process_hidden_states` (embeddings_processor.py:70-117). +// +// WHAT THIS DOES NOT DO, and why it is not a shortcut. Upstream pads every +// prompt to 1024 and runs all 1024 rows through the tower. This runs only the +// VALID tokens, at their ORIGINAL absolute positions, and treats the pad rows as +// zero. That is equivalent, not approximate: pads are masked out of attention +// and sit causally before every valid token, and the feature extractor zeroes +// their rows anyway (feature_extractor.py:63-64). Both halves of that are +// MEASURED rather than argued — upstream's own padded-vs-short f32 spread is +// f32 round-off, and our short run reproduces the padded run's valid rows, both +// gated in tests/vllm/models/test_ltx2_text_encoder.cpp. At the shipped 1024 it +// is the difference between a 12B forward over 1024 rows and over the prompt's +// own length. +struct Ltx2PromptConditioning { + Ltx2TextConditioning conditioning; + Ltx2GemmaPromptTokens tokens; + // [max_length], the binary mask the extractor consumed — 1 on a valid token. + std::vector mask; + int64_t seq = 0; // == max_length; the DiT sees the full padded width +}; + +// `queue` runs the tower. `weights`/`config` are the caption projections and the +// resolved V1/V2 shape, exactly as `Ltx2TextFeatureExtractorForward` takes them. +Ltx2PromptConditioning Ltx2EncodePromptToConditioning( + const Ltx2GemmaTower& tower, const tok::Tokenizer& tokenizer, + const Ltx2GemmaSpecialIds& ids, const Ltx2TextEncoderWeights& weights, + const Ltx2TextFeatureConfig& feature_config, const std::string& prompt, + vt::Queue& queue, int64_t max_length = kLtx2GemmaTokenizerMaxLength); + +} // namespace vllm diff --git a/include/vllm/model_executor/models/ltx2_upsampler.h b/include/vllm/model_executor/models/ltx2_upsampler.h new file mode 100644 index 000000000..f172ba247 --- /dev/null +++ b/include/vllm/model_executor/models/ltx2_upsampler.h @@ -0,0 +1,155 @@ +// LTX-2.5 LATENT SPATIAL UPSAMPLER — stage 2 of the distilled two-stage recipe. +// +// Row: MODEL-DIFFUSION-ltx-2-5-ltx2-video-transformer-3d-model. Spec: +// .agents/specs/ltx-2-5.md (phase L5). Issue #435. +// +// ─── WHAT THIS IS A PORT OF (file:line on BOTH sides) ──────────────────────── +// Upstream root: Lightricks/LTX-2, packages/ltx-core/src/ltx_core/model/upsampler/ +// OURS <- UPSTREAM +// Ltx2LatentUpsample <- model.py:82-126 (LatentUpsampler.forward) +// Ltx2UpsamplerConfig <- model.py:25-80 + model_configurator.py:11-31 +// (ResBlock) <- res_block.py:29-37 +// (SpatialRationalResampler) <- spatial_rational_resampler.py:40-47 +// (PixelShuffleND) <- pixel_shuffle.py:31-54 +// (BlurDownsample) <- blur_downsample.py:29-53 +// Ltx2RationalForScale <- spatial_rational_resampler.py:10-14 +// Ltx2UpsampleVideoLatent <- model.py:129-143 (upsample_video) +// +// ─── WHAT SEPARATES THIS FROM THE VAE'S CONVOLUTIONS ───────────────────────── +// These are plain `torch.nn.Conv3d`/`Conv2d` with `padding=1` — ZERO padding on +// every axis INCLUDING time. The Conv video VAE next door uses `CausalConv3d`, +// which prepends replicated copies of frame 0 (convolution.py:306-307). The two +// are not interchangeable and this file does not reuse that kernel: a causal pad +// here would shift the whole clip while still producing a correctly shaped, +// finite, plausible latent. +// +// Three more things that fail silently: +// * GroupNorm's group count is HARDCODED to 32 (res_block.py:24,26; model.py:50), +// not a config key. With mid_channels 512 that is 16 channels per group; a port +// that normalized per channel or per tensor produces a valid-looking latent. +// * The residual is added BEFORE the activation, not after: `activation(x + residual)` +// (res_block.py:36). `activation(x) + residual` is the same shape and a +// different function. +// * PixelShuffleND unpacks `(c p1 p2)` with p1 taking HEIGHT and p2 taking WIDTH +// (pixel_shuffle.py:41-47). Swapping them transposes every 2x2 block. +// +// ─── NOT PORTED, refused by name ───────────────────────────────────────────── +// * `temporal_upsample` (model.py:55-72, 109-113) — spec section 2 puts the +// temporal x2 upsampler out of scope; asking for it throws. +// * `dims == 2` (model.py:85-100) — a checkpoint that sets it wants Conv2d +// everywhere, i.e. no temporal convolution at all. LTX-2.5's upsampler is +// dims=3; the 2-D arm is refused rather than approximated by the 3-D one. +// +// ─── DTYPE ─────────────────────────────────────────────────────────────────── +// f32, because this is the CPU REFERENCE arm and the gate compares the ALGORITHM +// against upstream run in torch float32. Upstream runs the upsampler in the +// pipeline's bfloat16 (distilled.py:109, 219), so the bf16 arm is owed by phase +// L6 exactly as the VAEs' is. +#pragma once + +#include +#include +#include + +#include "vllm/model_executor/models/ltx2_audio_vae.h" // Ltx2VaeWeights + +namespace vllm { + +// res_block.py:24,26 and model.py:50 — `torch.nn.GroupNorm(32, channels)`, a +// literal on all three sites. +inline constexpr int64_t kLtx2UpsamplerNormGroups = 32; +// torch's `nn.GroupNorm` default `eps` (it is not passed at any of the three +// construction sites). NOT a member of the invisible-constant class. It was +// recorded as one on a mutation that happened not to move anything, and a +// mutation that moves nothing proves nothing; at the class's OWN 100x bar +// (1e-5 -> 1e-3) it REDS all three arms of +// "ltx2 the latent spatial upsampler reproduces upstream" — +// PixelShuffle 0.0289409, Rational2 0.0347079, Rational1p5 0.0649014. The pin +// below still earns its place, because a golden regenerated with a moved eps +// moves with it and only the pin compares against torch's own default. +inline constexpr double kLtx2UpsamplerNormEps = 1e-5; + +// LatentUpsampler.__init__ defaults (model.py:25-35), which are also +// LatentUpsamplerConfigurator.from_metadata's `config.get` fallbacks +// (model_configurator.py:14-21). +struct Ltx2UpsamplerConfig { + int64_t in_channels = 128; + int64_t mid_channels = 512; + int64_t num_blocks_per_stage = 4; + int64_t dims = 3; + bool spatial_upsample = true; + bool temporal_upsample = false; + double spatial_scale = 2.0; + bool rational_resampler = false; + std::string prefix; +}; + +// _rational_for_scale (spatial_rational_resampler.py:10-14): the up/down integer +// pair for a supported scale. Upstream RAISES on anything else (:12-13) and so +// does this — an unsupported scale is a config error, never a nearest match. +struct Ltx2RationalScale { + int64_t num = 1; + int64_t den = 1; +}; +Ltx2RationalScale Ltx2RationalForScale(double scale); + +// BlurDownsample's fixed separable binomial kernel (blur_downsample.py:29-33): +// the outer product of Pascal's row `kernel_size - 1`, normalized to sum 1. It is +// COMPUTED at construction, never loaded, so both sides must build it +// independently — the same rule the audio VAE's kaiser-sinc windows follow. +// Returns [kernel_size * kernel_size], row-major. +std::vector Ltx2BlurKernel(int64_t kernel_size); +// `BlurDownsample.__init__`'s default (blur_downsample.py:14), which +// `SpatialRationalResampler` never overrides (:38) — so this default IS the +// shipped kernel width. Gated against upstream's own signature by +// test_ltx2_pipeline.cpp, case "the constants the headers call pinned are +// actually pinned", and reached NUMERICALLY as well: 5 -> 3 REDS the Rational1p5 +// arm of "ltx2 the latent spatial upsampler reproduces upstream" at 0.689782. +// Only that arm, because the blur runs on the rational `den` and 1.5 -> {3, 2} is +// the one scale the suite covers with den != 1. +inline constexpr int64_t kLtx2BlurKernelSize = 5; + +// The parameter contract: every tensor `LatentUpsampler(config)` creates, in +// `named_parameters()` order. This IS the layout — the parity suite compares it +// against the upstream module's own. +struct Ltx2UpsamplerTensorSpec { + std::string name; + std::vector shape; +}; +std::vector EnumerateLtx2UpsamplerTensors( + const Ltx2UpsamplerConfig& config); + +// A [batch, channels, frames, height, width] latent, row-major. +struct Ltx2LatentVolume { + int64_t batch = 1; + int64_t channels = 0; + int64_t frames = 0; + int64_t height = 0; + int64_t width = 0; + std::vector data; + + int64_t elems() const { return batch * channels * frames * height * width; } +}; + +// LatentUpsampler.forward (model.py:82-126), the dims == 3 spatial arm. Throws by +// name for `dims == 2`, for `temporal_upsample`, and when neither upsample flag +// is set (upstream's own ValueError at :74). +Ltx2LatentVolume Ltx2LatentUpsample(const Ltx2UpsamplerConfig& config, + const Ltx2VaeWeights& weights, + const Ltx2LatentVolume& latent); + +// upsample_video (model.py:129-143): un-normalize by the video encoder's +// per-channel statistics, upsample, re-normalize. `std_of_means` / `mean_of_means` +// are the encoder's `per_channel_statistics` (video_vae/ops.py:63-84), one value +// per LATENT channel. +// +// Exposed separately from `Ltx2LatentUpsample` because the statistics belong to +// the VAE, not to the upsampler, and a caller holding a latent that is already in +// un-normalized space must not apply them twice. +Ltx2LatentVolume Ltx2UpsampleVideoLatent(const Ltx2UpsamplerConfig& config, + const Ltx2VaeWeights& weights, + const Ltx2LatentVolume& latent, + const std::vector& std_of_means, + const std::vector& mean_of_means); + +} // namespace vllm diff --git a/include/vllm/model_executor/models/ltx2_video_vae.h b/include/vllm/model_executor/models/ltx2_video_vae.h new file mode 100644 index 000000000..bfb7022a5 --- /dev/null +++ b/include/vllm/model_executor/models/ltx2_video_vae.h @@ -0,0 +1,246 @@ +// LTX-2.5 CONV VIDEO VAE — the convolutional video decoder, and the explicit +// refusal of the diffusion one. +// +// LTX-2.5 ships TWO video decoders behind one checkpoint field +// (`config.vae._class_name`, video_vae/model_configurator.py:18-34): +// +// "CausalVideoAutoencoder" -> ConvVideoDecoder — ported here +// anything else -> NADiffusionDecoder — NOT ported, REFUSED BY NAME +// +// The diffusion decoder is a neighborhood-attention model with its own row. Per +// .agents/specs/ltx-2-5.md section 0 item 2 it is refused with a message naming +// the missing piece and NEVER silently downgraded to the conv decoder — a +// downgrade would return a lower-quality render as if it were the requested one, +// which no gate in this project can detect. +// +// ─── WHAT THIS IS A PORT OF (file:line on BOTH sides) ──────────────────────── +// Upstream root: Lightricks/LTX-2, packages/ltx-core/src/ltx_core/ +// OURS <- UPSTREAM +// Ltx2ConvVideoDecode <- model/video_vae/conv_video_decoder.py:263-357 +// (block construction) <- model/video_vae/conv_video_decoder.py:61-143 +// (ResnetBlock3D) <- model/video_vae/resnet.py:12-186 +// (UNetMidBlock3D) <- model/video_vae/resnet.py:189-277 +// (CausalConv3d) <- model/video_vae/convolution.py:266-317 +// (DepthToSpaceUpsample) <- model/video_vae/sampling.py:68-123 +// (AttnBlock3D / _RMSNorm2D) <- model/video_vae/attention.py:11-69 +// (unpatchify) <- model/video_vae/ops.py:35-60 +// (per-channel statistics) <- model/video_vae/ops.py:63-84 +// (PixArt timestep embedding) <- model/transformer/timestep_embedding.py:6-141 +// Ltx2ParseVideoDecoderKind <- model/video_vae/model_configurator.py:18-34 +// +// ─── THE FOUR THINGS THAT FAIL SILENTLY ────────────────────────────────────── +// * TEMPORAL PADDING IS A REPLICATED FIRST FRAME, NOT ZEROS. CausalConv3d +// prepends `k_t - 1` copies of frame 0 (convolution.py:306-307). MiniMax-H3's +// causal Conv3d zero-pads instead, so the two are NOT interchangeable even +// though both put every temporal pad on the LEFT. +// * TWO DIFFERENT PixelNorm EPSILONS. video_vae/resnet.py:46 and +// conv_video_decoder.py:243 construct `PixelNorm()` with its DEFAULT eps of +// 1e-8, while the audio VAE reaches PixelNorm through +// `build_normalization_layer`, which passes 1e-6 (normalization.py:58). Using +// one value for both is a silent, tiny, everywhere-bias. +// * DEPTH-TO-SPACE UNPACKS `(c p1 p2 p3)`, AND THE TEMPORAL STRIDE DROPS THE +// FIRST FRAME (sampling.py:112-120). Getting the channel order wrong shuffles +// pixels inside every 2x2 block; keeping the first frame shifts the whole clip. +// * `unpatchify` DECOMPOSES CHANNELS AS `(c p r q)` WITH `h` TAKING q AND `w` +// TAKING r (ops.py:50-58) — r and q are NOT interchangeable. +// +// ─── DTYPE ─────────────────────────────────────────────────────────────────── +// Every buffer this header names is f32, because this is the CPU REFERENCE arm. +// Upstream runs the decoder in the CHECKPOINT's dtype instead +// (`sample.to(weights_dtype)` in, `sample.to(output_dtype)` out — +// conv_video_decoder.py:283-286, 355-356), and it has none of the float32 pin the +// audio tower carries. The bf16/NVFP4 arm that inherits the checkpoint dtype is +// owed by phase L6; see ltx2_video_vae.cpp for why no gate here can catch a dtype +// that is merely too WIDE. +#pragma once + +#include +#include +#include + +#include "vllm/model_executor/models/ltx2_audio_vae.h" // Ltx2VaeWeights + +namespace vllm { + +// video_vae/enums.py:4-6. +enum class Ltx2NormLayer { kGroupNorm, kPixelNorm }; + +// video_vae/enums.py:16-20. Only the modes a decoder can actually select are +// listed; `make_conv_nd` forwards the rest to torch, which this port refuses. +enum class Ltx2PaddingMode { kZeros, kReflect, kReplicate }; + +// Which decoder a checkpoint asks for. +enum class Ltx2VideoDecoderKind { kConv, kDiffusion }; + +// `_RMSNorm2D` is `F.normalize(x, dim=1) * (sqrt(C) * gamma)` (attention.py:11-30), +// so the denominator floor is torch's `F.normalize` DEFAULT eps of 1e-12 — an L2 +// normalize, not a mean-square RMS, and not this project's usual rms_norm epsilon. +// Named so it can be pinned: mutation proves 1e-12 -> 0.0 leaves every golden +// green, because the reduced-dimension activations are O(1) and the floor never +// binds at that magnitude. It is still READ on every element, so the goldens are +// not blind to it in the other direction — 1e-12 -> 1.0 reds two encoder goldens +// at 0.000525832. And it decides whether an all-zero channel vector divides or +// produces NaN. +inline constexpr double kLtx2RmsNorm2dEps = 1e-12; + +// `config.vae._class_name` -> the decoder kind, mirroring +// `_vae_class_name_from_metadata` + `VideoDecoderConfigurator.from_metadata` +// (video_vae/model_configurator.py:18-34, 242-250): the conv decoder is selected +// by the exact string "CausalVideoAutoencoder" and by an ABSENT field (upstream's +// default); anything else is the diffusion decoder. +Ltx2VideoDecoderKind Ltx2ParseVideoDecoderKind(const std::string& vae_class_name); + +// One entry in `decoder_blocks`. `multiplier` 0 means "the upstream default for +// this block kind" — 2 for `res_x_y` (conv_video_decoder.py:98), 1 for every +// `compress_*` (conv_video_decoder.py:111, 120, 129). +struct Ltx2VideoDecoderBlock { + std::string name; + int64_t num_layers = 1; + int64_t multiplier = 0; + bool inject_noise = false; + bool residual = false; +}; + +// ─── THE INVISIBLE-CONSTANT CLASS ──────────────────────────────────────────── +// An HONEST LIMIT of these goldens, and it is a CLASS, not one instance. Any +// epsilon or floor that exists to stabilize a division is, by construction, +// hard for a reduced-dimension parity gate to reach DOWNWARD: the deterministic +// stream produces O(1) activations, so shrinking the term it guards changes +// nothing the tensor comparison can see. That is the honest form of the claim. +// "Accepts any value at all" is what this paragraph used to say, and it is FALSE +// even of its own members — the epsilon is still READ on every element, so a +// large enough value moves the output. Only a probe that FAILS TO REACH proves +// unreachable; a mutation that happens not to move anything proves nothing, and +// the direction and MAGNITUDE of the mutation are therefore part of the verdict. +// MEASURED, by mutating each in turn, with the bound each number actually holds: +// +// kMiniMaxH3SnakeEps 1e-9 -> 0.0 green +// kLtx2RmsNorm2dEps 1e-12 -> 0.0 green ...but NOT green upward: +// escalating it to 1.0 REDS "the video ENCODER (*_res family)" and "the video +// encoder CROPS a frame count that is not 1 + k*factor", both at 0.000525832 +// against the 5e-6 band. It never BINDS at the shipped value, and it is read +// regardless — the two are different statements and only the first is true +// of this constant. +// +// `kLtx2BweMelLogClamp` was listed with them and NO LONGER belongs — the third +// entry to leave this list for the same reason, which is why the verdict is now +// stated per-entry with the number that proves it. The arm that made it +// reachable, "ltx2 vae: the BWE mel log clamp is gated where it actually binds", +// landed with the pin itself; the line calling it invisible was written in the +// same change and was false the moment it shipped. 1e-5 -> 1e-8 REDS that arm at +// max|diff| = 0.144965 against the 5e-6 band (36 cases: 34 passed, 2 failed — +// the golden, and the constant assertion below it). What made it look invisible +// was the SCALE of the ordinary arm, not the constant's nature: that arm's raw +// mel minimum is ~4.4e-3 and never approaches the floor, so the reachable arm +// attenuates mel_basis by 1e-4 until every bin lands under it — and asserts the +// saturated-bin count rather than assuming it. +// +// `Ltx2ConvVideoDecoderConfig::pixel_norm_eps` was listed with them and NO LONGER +// belongs. The arm added to make `norm_eps` reachable — "ltx2 vae: the video +// decoder's norm_eps is gated where it BINDS" — runs its latent at a tenth of the +// usual scale, and that makes this epsilon a first-order term too: 1e-8 -> 1e-6 +// now REDS that arm at max|diff| = 1.69305e-04 against the 5e-6 band. The fixture +// built to close one hole closed its neighbour with it, and the line claiming +// otherwise survived the change that falsified it. It stays pinned, in "ltx2 vae: +// the two PixelNorm epsilons stay different", for the reason a pin always earns: +// a regeneration that moves the constant and the goldens together. +// +// `Ltx2ConvVideoDecoderConfig::norm_eps` was listed here and DOES NOT BELONG. It +// is read on every arm that has a `res_x_y` block, PixelNorm included, because +// `norm3` is a GroupNorm built whenever `in_channels != out_channels` +// (resnet.py:93-97) and applied at resnet.py:178. Its 1e-6 -> 1e-4 mutation +// stayed green only because the norm3 in the shipped fixture divides by a +// variance of ~0.2 five blocks deep; at 1e-6 -> 1.0 the same golden moves 1.6e-2. +// That is a sensitivity property of one fixture, not invisibility, and it is now +// gated numerically by a fixture where the epsilon is a first-order term. +// +// So every member of the class is held by a SOURCE-ANCHORED CONSTANT ASSERTION in +// tests/vllm/models/test_ltx2_vae.cpp, cited to the upstream line that sets it, +// rather than by the tensor comparison — and a constant that is added later and +// left unpinned is a new hole, not a covered one. The three names above that LEFT +// the class keep their assertions as well: their goldens now move under a +// mutation, but a golden still cannot catch a regeneration that shifts the +// constant and the expected tensors together, and only the pin can. +struct Ltx2ConvVideoDecoderConfig { + // Defaults mirror `_build_conv_video_decoder` + // (video_vae/model_configurator.py:81-94). + int64_t in_channels = 128; + int64_t out_channels = 3; + // In CHECKPOINT (encoder) order. The decoder walks it REVERSED, exactly as + // conv_video_decoder.py:222 does. + std::vector decoder_blocks; + int64_t patch_size = 4; + Ltx2NormLayer norm_layer = Ltx2NormLayer::kPixelNorm; + bool causal = false; + bool timestep_conditioning = true; + Ltx2PaddingMode spatial_padding_mode = Ltx2PaddingMode::kReflect; + int64_t base_channels = 128; + int64_t norm_num_groups = 32; + double decode_noise_scale = 0.025; + double decode_timestep = 0.05; + // The GroupNorm arm's eps, and the one `res_x_y`'s shortcut norm3 uses. + // `ResnetBlock3D.__init__` declares `eps: float = 1e-6` (video_vae/resnet.py:31) + // and hands it to every nn.GroupNorm it builds (resnet.py:44, 65, 94); + // `UNetMidBlock3D` carries the same value as `resnet_eps` (resnet.py:216). + // + // norm3 is the reason this is LIVE on a PixelNorm checkpoint too: it is built + // whenever `in_channels != out_channels` (resnet.py:93-97) and applied to the + // residual at resnet.py:178, and `norm_layer` does not gate it. Neither does a + // checkpoint key — `_make_decoder_block` passes `eps=1e-6` / `resnet_eps=1e-6` + // literally (conv_video_decoder.py:78, 103), so this field exists to be pinned + // to that literal, and is gated numerically by the norm_eps arm in + // tests/vllm/models/test_ltx2_vae.cpp. + double norm_eps = 1e-6; + // `PixelNorm()`'s DEFAULT (normalization.py:22), reached bare from + // video_vae/resnet.py:46 and conv_video_decoder.py:243 — NOT the 1e-6 the audio + // VAE gets through build_normalization_layer. + double pixel_norm_eps = 1e-8; + std::string prefix; +}; + +// The deterministic source for every `torch.randn` upstream draws, consumed in +// CALL ORDER. That is precisely the guarantee an upstream `torch.Generator` +// gives, and it is what makes the decoder's noise injection reproducible on both +// sides. A null stream means "no noise is available", which is an ERROR whenever +// the config asks for noise rather than a silent zero fill. +class Ltx2NoiseStream { + public: + virtual ~Ltx2NoiseStream() = default; + virtual std::vector Draw(int64_t count) = 0; +}; + +// A (channels, frames, height, width) clip in [-1, 1]-ish pixel space (upstream +// maps it to [0, 1] outside the decoder, conv_video_decoder.py:497-499). +struct Ltx2VideoFrames { + int64_t channels = 0; + int64_t frames = 0; + int64_t height = 0; + int64_t width = 0; + std::vector data; +}; + +// ConvVideoDecoder.forward at batch 1. `latent` is +// [latent_channels, latent_t, latent_h, latent_w], channel-major. +// +// `timestep` overrides `decode_timestep` when non-null (the decoder's own +// default is used otherwise, conv_video_decoder.py:304-305). `noise` must be +// non-null whenever `timestep_conditioning` is set or any block sets +// `inject_noise`. +Ltx2VideoFrames Ltx2ConvVideoDecode(const Ltx2ConvVideoDecoderConfig& config, + const Ltx2VaeWeights& weights, + const std::vector& latent, int64_t latent_channels, + int64_t latent_t, int64_t latent_h, int64_t latent_w, + Ltx2NoiseStream* noise, const double* timestep = nullptr); + +// The seam a caller reaches for when it holds a checkpoint rather than a decided +// kind. `kConv` forwards to Ltx2ConvVideoDecode; `kDiffusion` THROWS, naming +// NADiffusionDecoder and its missing neighborhood-attention kernel. It never +// falls back. +Ltx2VideoFrames Ltx2VideoDecode(Ltx2VideoDecoderKind kind, + const Ltx2ConvVideoDecoderConfig& config, + const Ltx2VaeWeights& weights, const std::vector& latent, + int64_t latent_channels, int64_t latent_t, int64_t latent_h, + int64_t latent_w, Ltx2NoiseStream* noise, + const double* timestep = nullptr); + +} // namespace vllm diff --git a/include/vllm/model_executor/models/ltx2_video_vae_encoder.h b/include/vllm/model_executor/models/ltx2_video_vae_encoder.h new file mode 100644 index 000000000..65faa1a29 --- /dev/null +++ b/include/vllm/model_executor/models/ltx2_video_vae_encoder.h @@ -0,0 +1,183 @@ +// LTX-2.5 CONV VIDEO VAE — the ENCODER half, which phase L4 recorded as owed. +// +// Without it there is no way to turn an input image, keyframe or reference video +// into latents, so every conditioning mode except prompt-embeds is unreachable: +// `combined_image_conditionings` calls `video_encoder(image)` and hands the +// result to `VideoConditionByLatentIndex` / `VideoConditionByKeyframeIndex` +// (ltx-pipelines/utils/helpers.py:272-308). +// +// ─── WHAT THIS IS A PORT OF (file:line on BOTH sides) ──────────────────────── +// Upstream root: Lightricks/LTX-2 @ fd4ded7f, packages/ltx-core/src/ltx_core/ +// OURS <- UPSTREAM +// Ltx2ConvVideoEncode <- model/video_vae/video_vae.py:264-336 +// (block construction) <- model/video_vae/video_vae.py:39-145 +// (SpaceToDepthDownsample) <- model/video_vae/sampling.py:12-65 +// (CausalConv3d, strided) <- model/video_vae/convolution.py:266-313 +// (ResnetBlock3D/UNetMidBlock3D)<- model/video_vae/resnet.py:12-277 +// (AttnBlock3D) <- model/video_vae/attention.py:11-69 +// (patchify) <- model/video_vae/ops.py:6-32 +// (per-channel statistics) <- model/video_vae/ops.py:63-84 +// Ltx2VideoTemporalScaleFactor <- types.py:35-53 (SpatioTemporalScaleFactors) +// Ltx2ConvVideoEncoderConfig <- model/video_vae/model_configurator.py:37-70 +// +// The primitives are SHARED with the decoder rather than copied: this encoder is +// compiled into ltx2_video_vae.cpp so it calls the very same `CausalConv3d`, +// `PixelNorm`, `ApplyNorm`, `ResnetBlock3d` and `AttnBlock3d` the decoder is +// gated on. A second copy of a causal pad is the duplicate that goes wrong +// quietly, because each copy keeps its own green gate. +// +// ─── THE FIVE THINGS THAT FAIL SILENTLY ────────────────────────────────────── +// * THE ENCODER'S DEFAULT SPATIAL PADDING IS `zeros`; THE DECODER'S IS +// `reflect` (model_configurator.py:63-67 vs :90). They read the SAME +// checkpoint key `spatial_padding_mode` on a flat CausalVideoAutoencoder +// config, so they only diverge when the key is ABSENT — and then they diverge +// silently, by a half-pixel border, in opposite directions. +// * `SpaceToDepthDownsample` DUPLICATES FRAME 0 BEFORE the space-to-depth fold +// whenever the temporal stride is 2 (sampling.py:39-40), and it does that on +// the SKIP path as well as the conv path. Skipping it shifts the whole clip by +// one latent frame while every shape still checks out. +// * ITS SKIP CONNECTION IS A GROUP MEAN, NOT A SLICE (sampling.py:50-51): the +// folded `in_channels * prod(stride)` channels are cut into `out_channels` +// contiguous groups of `group_size` and averaged. Taking the first channel of +// each group instead produces a plausible latent. +// * `patchify` DECOMPOSES SPATIAL AXES AS `(h q) (w r) -> (c p r q)` — r (WIDTH) +// is the OUTER factor and q (HEIGHT) the inner one (ops.py:20-28). This is the +// exact inverse of the decoder's `unpatchify`, and swapping r and q transposes +// every patch. +// * A STRIDED CausalConv3d STILL PREPENDS `k_t - 1` COPIES OF FRAME 0, not +// `k_t - stride` (convolution.py:305-307). The padding is decided before the +// stride is applied, so the output frame count is +// `(T + k_t - 1 - k_t) / stride + 1`. +// +// ─── DTYPE ─────────────────────────────────────────────────────────────────── +// f32 throughout, for exactly the reason ltx2_video_vae.cpp:20-44 gives for the +// decoder: this is the CPU REFERENCE arm and upstream instead runs in the +// CHECKPOINT's dtype. No gate here can catch a dtype that is merely too WIDE, +// because the generator casts every upstream parameter to f32 and the oracle +// therefore runs f32 too. The production arm is owed with the decoder's. +#pragma once + +#include +#include +#include + +#include "vllm/model_executor/models/ltx2_audio_vae.h" // Ltx2VaeWeights +#include "vllm/model_executor/models/ltx2_upsampler.h" // Ltx2LatentVolume +#include "vllm/model_executor/models/ltx2_video_vae.h" // Ltx2NormLayer, Ltx2PaddingMode + +namespace vllm { + +// video_vae/enums.py:9-13. What the last conv emits ALONGSIDE the means, and it +// changes the channel arithmetic rather than just the bookkeeping: +// kPerChannel conv_out emits 2 * out_channels; means are the first half. +// kUniform conv_out emits out_channels + 1; the single trailing logvar +// channel is REPEATED out_channels times before the split. +// kConstant conv_out emits out_channels + 1; the trailing channel is +// DISCARDED and replaced by a constant -30 block. +// kNone conv_out emits out_channels — and upstream then chunks that in +// two, leaving HALF as many mean channels as the per-channel +// statistics carry. See Ltx2ConvVideoEncode: it is refused by +// name rather than reproduced, because upstream itself raises. +enum class Ltx2LogVarianceType { kPerChannel, kUniform, kConstant, kNone }; + +// `approx_ln_0` (video_vae.py:328) — the constant log-variance the kConstant arm +// substitutes, described upstream as "the minimal clamp value in +// DiagonalGaussianDistribution objects". Named so it can be pinned: it is a +// member of the invisible-constant class, because a golden that only ever reads +// the MEANS half of the split can never see it. The gate reads the constant +// directly instead. +inline constexpr double kLtx2EncoderApproxLnZero = -30.0; + +// One entry in `encoder_blocks`, in CHECKPOINT order — the encoder walks it +// FORWARD, which is what makes it the mirror of the decoder's reversed walk. +// `multiplier` 0 means "the upstream default for this block kind": 2 for +// `res_x_y`, `compress_all_x_y`, `compress_all_res`, `compress_space_res` and +// `compress_time_res` (video_vae.py:62, 103, 114, 123, 132); the plain +// `compress_time` / `compress_space` / `compress_all` convolutions take NO +// multiplier at all and keep `in_channels` (video_vae.py:72-101). +struct Ltx2VideoEncoderBlock { + std::string name; + int64_t num_layers = 1; + int64_t multiplier = 0; +}; + +struct Ltx2ConvVideoEncoderConfig { + // Defaults mirror `_prepare_video_encoder_kwargs` + // (video_vae/model_configurator.py:55-69). + int64_t in_channels = 3; + // The LATENT width. On a flat CausalVideoAutoencoder config this is + // `latent_channels`, NOT the top-level `out_channels`, which is the decoder's + // RGB count (model_configurator.py:41-43) — reading the wrong one builds an + // encoder with 3 latent channels that still runs. + int64_t out_channels = 128; + std::vector encoder_blocks; + int64_t patch_size = 4; + Ltx2NormLayer norm_layer = Ltx2NormLayer::kPixelNorm; + Ltx2LogVarianceType latent_log_var = Ltx2LogVarianceType::kUniform; + // ZEROS is the ENCODER's default and it is NOT the decoder's `reflect`; see the + // header note above. + Ltx2PaddingMode spatial_padding_mode = Ltx2PaddingMode::kZeros; + int64_t norm_num_groups = 32; // VideoEncoder._DEFAULT_NORM_NUM_GROUPS = 32. + // HARDCODED upstream: `_make_encoder_block` passes `resnet_eps=1e-6` / + // `eps=1e-6` literally (video_vae.py:56, 66) and `conv_norm_out` takes + // `eps=1e-6` (video_vae.py:240). It is a field here only so the gate can pin + // it; there is no checkpoint key that moves it. + // + // And norm3 is the reason it is LIVE on a PixelNorm checkpoint here too, for + // the identical reason it is on the decoder's `norm_eps`: `ResnetBlock3D` + // builds `norm3 = nn.GroupNorm(num_groups=1, ..., eps=eps)` whenever + // `in_channels != out_channels` (resnet.py:93-97) and applies it to the + // residual (resnet.py:178), and `norm_layer` does not gate that. So every + // `res_x_y` encoder block reads this value even though `conv_norm_out` and + // `ApplyNorm` take their PixelNorm branches. + // + // Both halves route through ONE line in the port — ltx2_video_vae.cpp:1051,1056 + // reach :405, the same line the decoder reaches from :693,700 — but a SHARED + // LINE IS NOT AN ARGUMENT FOR LIVENESS, and this file previously offered it as + // one. :405 sits behind the `input.channels != out_channels` guard at :400, so + // even entering ResnetBlock3d is not reaching it — `res_x` passes `x.channels` + // as `out_channels` at :1051 and the guard is false. Encoder arm B does not + // reach it at all: all four blocks it holds are plain strided CausalConv3d + // (:1060-1068), so it never enters ResnetBlock3d and stays green under every + // mutation of this value. Liveness is per-arm and MEASURED — the field default + // 1e-6 -> 1e-4 reds two encoder goldens at max|diff| = 4.38839e-05 — which makes + // the numerical coverage real but PARTIAL, and is why the pin still carries the + // arms the goldens do not. + double norm_eps = 1e-6; + // `PixelNorm()`'s bare DEFAULT (normalization.py:22), same as the decoder's. + double pixel_norm_eps = 1e-8; + std::string prefix; +}; + +// `SpatioTemporalScaleFactors.from_blocks` (types.py:35-53), the temporal half. +// Every block whose name starts with `compress_time` or `compress_all` halves +// time by 2 REGARDLESS of its channel multiplier, so the factor is 2^steps. This +// is what decides the frame-count crop below, which is why it is exposed: a +// caller that picks a frame count without it silently loses trailing frames. +int64_t Ltx2VideoTemporalScaleFactor(const std::vector& blocks); + +// The spatial half of the same rule: `patch_size * 2^steps` over blocks starting +// with `compress_space` or `compress_all` (types.py:52). +int64_t Ltx2VideoSpatialScaleFactor(const std::vector& blocks, + int64_t patch_size); + +// VideoEncoder.forward at batch 1 (video_vae.py:264-336). `frames` is +// [in_channels, F, H, W] channel-major, in [-1, 1] pixel space. +// +// F MUST BE 1 + k * temporal_factor. Upstream WARNS and crops the trailing +// `(F - 1) % factor` frames rather than failing (video_vae.py:276-286); this +// mirrors the crop exactly, and `out_cropped_frames` reports how many were +// dropped so a caller can surface it instead of silently shipping a shorter +// clip. The pointer may be null. +// +// The returned latent is ALREADY NORMALIZED by `per_channel_statistics` +// (video_vae.py:336), i.e. it is in the same space the DiT and +// `Ltx2UpsampleVideoLatent` expect, and the same space `Ltx2ConvVideoDecode` +// de-normalizes on the way back. +Ltx2LatentVolume Ltx2ConvVideoEncode(const Ltx2ConvVideoEncoderConfig& config, + const Ltx2VaeWeights& weights, + const std::vector& frames, int64_t channels, + int64_t frame_count, int64_t height, int64_t width, + int64_t* out_cropped_frames = nullptr); + +} // namespace vllm diff --git a/include/vllm/model_executor/models/minimax_h3.h b/include/vllm/model_executor/models/minimax_h3.h index d981cc006..bce4e8934 100644 --- a/include/vllm/model_executor/models/minimax_h3.h +++ b/include/vllm/model_executor/models/minimax_h3.h @@ -487,6 +487,86 @@ struct MiniMaxH3AudioVaeWeights { bool Has(const std::string& name) const { return tensors.count(name) != 0; } }; +// --------------------------------------------------------------------------- +// THE SHARED 1-D BigVGAN PRIMITIVES. +// +// These are published here, next to MiniMaxH3KaiserSincFilter1d and +// MiniMaxH3GroupNorm3d, because MiniMax-H3 is not their only caller: LTX-2.5's +// audio VAE (ltx2_audio_vae.cpp) descends from the same BigVGAN lineage and needs +// exactly this arithmetic. They used to be TU-private to +// minimax_h3_audio_vae.cpp, which forced LTX to stand up a second copy — and a +// second copy of the alias-free trim geometry in particular is the kind of +// duplicate that goes wrong quietly: a fix to the pad/trim arithmetic lands in +// one file, the other keeps its own green gate, and the two audio VAEs disagree +// with nothing to say so. AGENTS.md §"Shared seams" says to extend the seam +// rather than hand-roll a parallel path, so the seam is extended here and there +// is now ONE implementation, gated by BOTH suites. +// +// The `MiniMaxH3` prefix records where they were ported first, not who may call +// them. Signals are CHANNEL-MAJOR [C, T] throughout. +// --------------------------------------------------------------------------- + +// One 1-D convolution over [C_in, T] with stride/dilation/groups. Weight is +// [C_out, C_in/groups, K]; the input must ALREADY be padded. Accumulates in +// double, and reports the produced length through `out_len`. +std::vector MiniMaxH3Conv1d(const std::vector& in, int64_t in_channels, + int64_t in_len, const std::vector& weight, + const std::vector* bias, int64_t out_channels, + int64_t kernel, int64_t stride, int64_t dilation, + int64_t groups, int64_t* out_len); + +// torch.nn.functional.conv_transpose1d over [C_in, T]. Weight is +// [C_in, C_out/groups, K]; output length is (T-1)*stride - 2*padding + K. +std::vector MiniMaxH3ConvTranspose1d(const std::vector& in, int64_t in_channels, + int64_t in_len, const std::vector& weight, + const std::vector* bias, int64_t out_channels, + int64_t kernel, int64_t stride, int64_t padding, + int64_t groups, int64_t* out_len); + +// F.pad along the time axis. `replicate` selects mode="replicate"; false is the +// zero pad an ordinary nn.Conv1d `padding=` argument performs. +std::vector MiniMaxH3Pad1d(const std::vector& in, int64_t channels, int64_t in_len, + int64_t left, int64_t right, bool replicate, int64_t* out_len); + +// The stabilizing epsilon in Snake/SnakeBeta's reciprocal, named so it can be +// pinned: upstream writes `1.0 / (beta + 1e-9)` on both sides of this port's +// lineage — LTX-2.5 at audio_vae/vocoder.py:198 (Snake) and :221 (SnakeBeta), and +// MiniMax-H3 in its BigVGAN activation. Mutation proves no reduced-dimension +// golden can tell 1e-9 from 0.0, because beta is O(1) there and never small +// enough for the term to matter; the value still decides whether a real +// checkpoint whose learned beta approaches zero divides or explodes. It is +// therefore held by a source-anchored constant assertion, not by a tensor +// comparison. +inline constexpr double kMiniMaxH3SnakeEps = 1e-9; + +// Snake / SnakeBeta: x + (b + kMiniMaxH3SnakeEps)^-1 * sin^2(a * x), in place. +// A null `beta` selects plain Snake, which reuses ALPHA as the reciprocal scale +// (LTX-2.5 vocoder.py:198); a non-null one selects SnakeBeta (vocoder.py:221), +// which is what every MiniMax-H3 checkpoint carries. `logscale` exponentiates +// both, which is how the parameters are stored. +void MiniMaxH3SnakeActivation(std::vector& x, int64_t channels, int64_t length, + const std::vector& alpha, const std::vector* beta, + bool logscale); + +// The anti-aliased activation, `Activation1d`: upsample by `ratio` -> Snake(Beta) +// -> downsample by `ratio`, both through the kaiser-sinc window with REPLICATE +// padding. MiniMax-H3 reaches it through dac_alias_free_act.py + +// dac_alias_free_resample.py; LTX-2.5 through vocoder.py:104-184. The trim +// geometry is the fragile part and the reason this is shared rather than copied. +// +// Build() computes the window once; Apply() is const and may be reused. +struct MiniMaxH3AliasFreeActivation1d { + int64_t ratio = 2; + int64_t kernel_size = 12; + std::vector filter; + + void Build(); + + std::vector Apply(const std::vector& in, int64_t channels, int64_t in_len, + const std::vector& alpha, const std::vector* beta, + bool logscale, int64_t* out_len) const; +}; + // kaiser_sinc_filter1d (dac_alias_free_filter.py:26-60) — built at load time, never // read from the checkpoint. std::vector MiniMaxH3KaiserSincFilter1d(double cutoff, double half_width, diff --git a/include/vllm/multimodal/ltx2_video.h b/include/vllm/multimodal/ltx2_video.h new file mode 100644 index 000000000..fb683ce54 --- /dev/null +++ b/include/vllm/multimodal/ltx2_video.h @@ -0,0 +1,375 @@ +// LTX-2.5 behind the GENERALIZED video seam — the second family registered with +// `vllm::multimodal::VideoEngine`, and the driving loop that turns the L2-L6 +// bricks into frames + a waveform. +// +// Row: MODEL-DIFFUSION-LTX25. Spec: .agents/specs/ltx-2-5.md phase L7. Issue #435. +// +// ─── WHAT THIS TU IS ───────────────────────────────────────────────────────── +// +// Phases L2-L6 shipped a DiT forward, a text feature extractor, two VAEs, a +// vocoder, an upsampler, a duration head, and a pipeline COMPONENT library — +// schedules, noisers, steppers, guiders, patchifiers, recipes. Nothing drove +// them: `Ltx2RefuseUnportedPipelineFeature(kVideoEngineWiring)` refused the +// composition BY NAME and named this phase as its owner. This TU is that +// composition and nothing else. It adds no numerics; every line either resolves +// a parameter, moves a buffer, or calls a brick that already has a golden. +// +// ─── WHAT IT IS A PORT OF (file:line on BOTH sides) ────────────────────────── +// Upstream: Lightricks/LTX-2 @ fd4ded7, packages/ltx-pipelines/src/ltx_pipelines/ +// OURS <- UPSTREAM +// Ltx2VideoEngine::Generate <- distilled.py:186-300 (DistilledPipeline.__call__) +// the per-phase stage call <- utils/blocks.py:500-582 (DiffusionStage.__call__) +// the latent state build <- utils/helpers.py:428-447 (create_noised_state) +// + ltx-core tools.py:139-184 / :246-280 +// the denoise loop <- utils/samplers.py:39-79 (euler_denoising_loop) +// + :26-36 (_step_state) +// the X0 conversion <- ltx-core model/transformer/model.py:590-604 +// (X0Model.forward) + utils.py:38-50 (to_denoised) +// the per-step Modality build <- utils/helpers.py:466-503 +// (modality_from_latent_state, timesteps_from_mask) +// post_process_latent <- utils/helpers.py:462-464 +// +// ─── WHAT THIS ENGINE REFUSES, AND WHY EACH WOULD RENDER ──────────────────── +// +// Two of the three below are refusals that STAND. Item 2 is a refusal that was +// LIFTED, and it is kept in this list rather than deleted because the interesting +// thing about it is its history: it is the one whose stated reason went stale +// twice, which is this campaign's recurring defect and worth leaving legible. +// +// 1. `device = 1` (CUDA) WITHOUT A CUDA BACKEND. Phase L7 refused every non-zero +// device outright: L2's forward was f32-only by declaration and L6's +// `Ltx2StreamDitToDevice` stages bf16 and refuses to widen, so no combination +// put the DiT on a GPU. **Phase L8 closed that** — `Ltx2DitForwardDevice` +// (ltx2_device.h) is the same graph with every activation in device memory and +// the stream in the checkpoint's own bf16 — so a CUDA handle now denotes a +// CUDA forward and the load succeeds. +// +// What is still refused is the SUBSTITUTION. If the CUDA backend is not +// registered in this build, the load is refused BY NAME rather than served the +// CPU forward behind a CUDA-looking handle, because that substitution is what +// would make every later timing and every "it ran on the GPU" claim false. +// +// 2. A PROMPT — NO LONGER REFUSED. Phase L13 closed the last hop, and what +// follows is the record of what each phase actually contributed, because +// this refusal went stale TWICE before it went away and a reader deserves to +// be able to re-check the reason rather than trust it. +// +// L10 made the tower RUN: the embedded tokenizer reaches a prompt string +// (`Ltx2TokenizeGemmaPrompt`, token-exact against HuggingFace on the shipped +// 262144-entry vocab), the torchao-NVFP4 Gemma-4 tower materializes onto +// `Gemma4Weights` (`Ltx2LoadGemmaTowerFromSafetensors`), it produces all 49 +// hidden states within the oracle's own bf16 noise floor, and the aggregation +// and both caption projections turn those into the 4096-wide video and +// 2048-wide audio conditioning streams (`Ltx2EncodePromptToConditioning`). +// +// L9c put the `Embeddings1DConnector` on the render path with the +// checkpoint's OWN weights (`Ltx2LoadConnectorWeights`), which is where +// upstream puts it too: `EmbeddingsProcessor` runs an 8-layer 1-D transformer +// over the caption projections before the DiT sees them +// (embeddings_processor.py:70-117), and that module ships INSIDE THE DiT FILE +// as `video_embeddings_connector.*` / `audio_embeddings_connector.*`. +// +// The two landed on separate branches and could not see each other. L10's +// refusal said the connector's weights were "still among the modules +// `Ltx2LoadDitFromSafetensors` refuses", and by the time the branches met +// that was FALSE — `ltx2_loader.cpp:417` already recorded them as loaded +// elsewhere. So `has_encoder()` is now TRUE when `encoder_path` is supplied, +// and a request's own `prompt` is tokenized, encoded, projected, run through +// the connector and handed to cross-attention, per request. +// +// WHAT THIS COSTS, stated rather than discovered: the tower is ~24 GB of host +// bf16 at the shipped 12B and it stays RESIDENT, because a prompt arrives per +// request. The connector weights do NOT stay resident — see the Impl comment +// in the .cpp; they are ~8 GB of f32 and are re-read from the still-present +// DiT file inside one scope per request, which keeps the steady state at the +// tower rather than at tower + connector. +// +// WHAT IS STILL OWED, and it is a real gap rather than a formality: the +// shipped `vonkaiser` text encoder carries NO `__metadata__` at all, so the +// Gemma config is an INPUT. It comes from the checkpoint's own +// `__metadata__["gemma_config"]` when there is one and from the +// `encoder_config_path` extra when there is not, and an encoder with neither +// is REFUSED rather than given a default — the same polarity `dit_config_path` +// already has, and for the same reason: a wrong Gemma config resolves a +// DIFFERENT MODEL out of a byte-identical tensor set. +// +// 3. Any pipeline kind / model version the recipe table does not carry. +// `ResolveLtx2PipelineRecipe` already throws rather than defaulting +// (ltx2_pipeline.h:543-562); this engine passes the checkpoint's OWN +// `model_version` through to it rather than assuming 2.5, because a checkpoint +// of another generation resolved onto 2.5's sigmas renders confidently and +// wrongly. +// +// ─── SCALE, STATED PLAINLY ─────────────────────────────────────────────────── +// +// The f32 CPU forward is the parity forward, not a production one: at the shipped +// 21.00B geometry its weights alone are ~76 GB. The bf16 DEVICE forward (L8) is +// the production residency — ~42 GB staged tensor-by-tensor — and it is what +// `device = 1` runs. Neither changes the other number this row owes: a single +// denoise step over a 512x768x121 latent is ~2.6e14 FLOPs, and there is no +// production-configuration oracle to divide by, so no speed figure is claimed +// anywhere in this family (spec §0). +#pragma once + +#include +#include + +#include "vllm/multimodal/video_engine.h" + +namespace vllm { +struct Ltx2DitParams; +} // namespace vllm + +namespace vllm::multimodal { + +// The stable registry name this family is reached under +// (VideoModelParams::family / vllm_video_model_params.family). It is the string +// `.agents/specs/ltx-2-5.md` and the L1 registry refusal test already print. +inline constexpr char kLtx2VideoFamily[] = "ltx-2.5"; + +// ── the family-specific LOAD extras (VideoModelParams::extras) ────────────── +// Every one of these is a knob upstream reads from somewhere this seam has no +// field for. An extra this family does not define is REFUSED, never ignored. + +// The audio stream's prompt-embeds file, the twin of the seam's +// `prompt_embeds_path` (which carries the VIDEO stream). LTX-2.5 conditions two +// streams at two different widths — 4096 and 2048 — and one file cannot hold +// both, so the audio half rides here. Rows of `audio_cross_attention_dim`, +// little-endian f32, and the two files must agree on their ROW COUNT because +// upstream's two encodings come from one tokenization. +inline constexpr char kLtx2AudioPromptEmbedsExtra[] = "audio_prompt_embeds_path"; + +// `resolve_ltx_pipeline_recipe`'s first key (ltx2_recipes.py:161-175). Defaults +// to "distilled_two_stage", which is what the shipped +// `ltx-2.5-22b-distilled-transformer` is: the file NAMES itself distilled and +// `DistilledPipeline` is the entry point that loads it. +inline constexpr char kLtx2PipelineKindExtra[] = "pipeline_kind"; + +// Overrides the `model_version` the DiT checkpoint declares in its own +// `__metadata__`. Present for a checkpoint that carries none; it never silently +// replaces one that does, and a mismatch between the two is reported. +inline constexpr char kLtx2ModelVersionExtra[] = "model_version"; + +// The DiT's `{"transformer": {...}}` configuration, as a JSON FILE, for a +// checkpoint whose `__metadata__` carries none. +// +// MEASURED 2026-08-12, and it is why this extra exists: of the two shipped +// LTX-2.5 DiTs only the first-party NVFP4 one carries `__metadata__` at all. +// `vonkaiser/LTX-2.5-FP8-NVFP4`'s FP8 DiT — the copy every phase before L6 gated +// against and the one L8 ran on the GPU — has NO `__metadata__` key whatsoever. +// Without a config the geometry still resolves from SHAPES, but the values no +// shape encodes fall back to the parser's defaults: `double_precision_rope = +// false` and `av_ca_timestep_scale_multiplier = 1`, against LTX-2.5's declared +// `float64` and `1000`. Both move every RoPE angle and every audio<->video +// modulation, so a silent default is a DIFFERENT MODEL rendering confidently. +// +// So a DiT that declares no config is REFUSED unless this extra names one. The +// file holds the same object the shipped checkpoints put in +// `__metadata__["config"]` — `{"transformer": {...}}` — and it is adopted through +// the IDENTICAL weight-contract check the declared path uses, so a config +// belonging to another checkpoint is refused rather than bound. +inline constexpr char kLtx2DitConfigPathExtra[] = "dit_config_path"; + +// Proceed past the module families the L2 contract does not carry — +// `prompt_adaln_single` / `audio_prompt_adaln_single` and +// `keyframes_abs_pos_embedding` (ltx2_loader.h). "1" opts in; anything else +// leaves the loader's refusal in place. The shipped DiTs all carry at least one +// of them, so this is the flag that says "gate the ported subset knowingly". +// +// The two `*_embeddings_connector` families are NOT in that set and this extra +// has nothing to do with them: they are outside the DiT contract by design and +// are materialized by `Ltx2LoadConnectorWeights`. An earlier revision of this +// comment listed them here, which is the same stale claim `ltx2_loader.h` +// carried and phase L10's refusal was built on. +inline constexpr char kLtx2AllowUnportedExtra[] = "allow_unported_modules"; + +// Run only the phases up to and including this index of the resolved recipe +// (0-based). Absent runs every phase. This exists because the two-stage recipe's +// second phase needs the latent spatial upsampler, and a run without one must +// say so rather than skipping the phase silently. +inline constexpr char kLtx2MaxPhaseExtra[] = "max_phase"; + +// How many of the supplied prompt-embeds rows are REAL tokens; the rest are +// padding. Absent means every row is real. +// +// WHY A SEAM WITH NO TOKENIZER NEEDS THIS. The embeddings connector substitutes +// its `learnable_registers` table at PADDED positions +// (embeddings_connector.py:139-152), so the padding is not inert — it is what +// decides which of the connector's inputs are learned constants rather than +// caption features. Upstream always knows this, because the tokenizer produced +// the mask. This seam takes prompt embeds from a FILE, which carries no mask, so +// without this extra the padded tail would be conditioned on as if it were text +// and every register would go unused. Recorded as a knob rather than assumed, +// and it is the field the Gemma-4 tower will supply when it lands. +inline constexpr char kLtx2PromptValidRowsExtra[] = "prompt_embeds_valid_rows"; + +// The Gemma-4 config for the tower `encoder_path` names, as a JSON FILE. +// +// MEASURED, and it is why this extra exists rather than being a convenience: +// the only shipped LTX-2.5 text encoder, `vonkaiser`'s +// `gemma4-12b-with-proj-nvfp4-torchao.safetensors`, has NO `__metadata__` block, +// so upstream's own `GemmaAssets.from_single_file` raises on it before reading a +// tensor (gemma_assets.py:110-114). The official bf16 encoder DOES carry one, +// under `__metadata__["gemma_config"]`, and that is preferred when present. +// +// What a default would get wrong is not a detail. `layer_types` decides which +// layers are full vs sliding and the two have DIFFERENT geometry; +// `global_head_dim` is 512 against `head_dim` 256; `num_global_key_value_heads` +// is ONE against 8; `attention_k_eq_v` is true, so the full layers ship no +// `v_proj` at all. Each of those moves every hidden state while leaving the +// tensor set byte-identical — a wrong config resolves a DIFFERENT MODEL out of +// the same file and nothing downstream can tell. So an encoder with neither +// source is refused, exactly as `dit_config_path`'s case is. +// +// Supplying BOTH a declaring checkpoint and this extra is refused rather than +// resolved in either direction, for the same reason. +inline constexpr char kLtx2EncoderConfigPathExtra[] = "encoder_config_path"; + +// WHAT THE LAST `Generate()` ACTUALLY HANDED THE DiT's CROSS-ATTENTION. +// +// Every field is read off the exact f32 buffers `Ltx2ModalityInput::context` +// pointed at, after the connector and immediately before the denoise loop — not +// re-derived from the inputs that produced them. +// +// WHY THIS EXISTS AS A SURFACE. Conditioning is the one thing about a render +// that the render cannot be inspected for. This project has already been burned +// on that exact point twice: L9c's reviewer found that the difference between a +// scene and a colour field was INVISIBLE to the frame analyzer (neighbour +// |dx|/sd 0.093 vs 0.033, block-mean ratios nearly identical) and took contact +// sheets to tell apart; and the campaign's standing lesson is that a golden +// reduced to `isfinite` once hid a 23842x error. A digest over the bytes that +// were actually fed is an instrument that cannot have that blind spot: it is a +// function OF those bytes, so any change to any element changes it, and it +// cannot be satisfied by a plausible-looking wrong tensor. +// +// It is not a quality claim and cannot become one. It answers "did this render +// depend on this prompt, through these weights" and nothing else. A server also +// has a use for it: "which conditioning produced this clip" is otherwise +// unanswerable after the fact. +// +// IT IS A WITNESS, NOT A GATE — and the difference is MEASURED, not argued. A +// digest detects CHANGE; it does not pin VALUES, so nothing at this level says +// the values are the ones upstream would produce. Two mutations applied to the +// composition below, each alone: +// +// * video conditioning scaled by 1.5 AFTER the connector, and +// * the conditioning rows REVERSED, putting every caption row on the wrong +// token, +// +// and BOTH passed `test_ltx2_video` at 30 cases / 499 assertions with exit 0. +// The digest moved, as it must — but no assertion says WHICH value it should +// have moved to. +// +// THAT COUNT IS THIS HEAD'S, and the distinction is the point of writing it +// down. A reviewer first measured the pair at `43aa58377`, where the suite stood +// at 485 assertions; the numbers were carried forward unchanged while the suite +// grew, so the comment named a count no run of it could produce. Re-run here +// (CPU Release, mutant recompiled and relinked each leg, tree restored +// byte-for-byte and re-verified green between legs): 499/499, exit 0, both. +// +// THE VALUE ORACLE THE COMPOSITION IS OWED. The per-brick oracles are real and +// strong: the Gemma-4 tower against a running `transformers` at a measured bf16 +// floor, `Ltx2ConnectorForward` on five arms against executed upstream, and the +// feature extractor and both caption projections against executed upstream. The +// two JOINS between them have none: `Ltx2ConnectorCreateEmbeddings` +// (ltx2_connector.h) and the `Generate` composition that chains it onto +// `Ltx2TextEncoderConditioning`. Both mutations above live in exactly that gap. +// +// The closure is specified rather than left as a wish, because the path is +// already built. `scripts/gen-ltx2-pipeline-goldens.py` imports and EXECUTES +// upstream `text_encoders/gemma/embeddings_connector.py` under a pinned SHA +// (section 10), and the composition's upstream counterpart is one function in +// the same package: `EmbeddingsProcessor.process_hidden_states` +// (embeddings_processor.py:97-117), which is feature extractor -> additive mask +// -> `create_embeddings` -> the two connectors — precisely this chain. A section +// that executes it end-to-end at the reduced dims the script already uses would +// give both joins a real numeric oracle, WITHOUT the "gate through our own +// helper" trap that makes a max|diff| of 0 prove only that two arms agree. The +// script reproduces its current output byte-for-byte (md5 53e2a6ab…9eb4, +// verified 2026-08-13), so the section can be added without disturbing anything +// already gated. Until that lands, this trace is a change detector and the +// composition's VALUES rest on the per-brick oracles either side of it. +struct Ltx2ConditioningTrace { + // True when the text tower encoded the request's own prompt; false when the + // conditioning came from `prompt_embeds_path`. + bool from_prompt = false; + std::string prompt; // the exact string that was tokenized ("" for embeds) + int64_t tokens = 0; // context rows the DiT cross-attends over + int64_t video_width = 0, audio_width = 0; + // FNV-1a over the raw little-endian f32 bytes of each stream. + uint64_t video_digest = 0, audio_digest = 0; + // max|x| per stream. A conditioning tensor that collapsed to zeros would give + // two prompts the SAME digest and RED any dependence check, but it would do so + // for the wrong reason; this says which happened. + double video_absmax = 0.0, audio_absmax = 0.0; + // True only once the `Generate` that produced this conditioning RETURNED. The + // trace is filled immediately after the connector and BEFORE the denoise loop, + // because that is the only point at which the exact buffers cross-attention + // will read still exist as such. So a `Generate` that throws in denoise, in a + // VAE decode or in the muxer leaves a fully populated trace behind for a render + // that produced no frames. Without this flag the next reader cannot tell that + // from a completed render, and "which conditioning produced this clip" would + // answer for a clip that does not exist. + bool completed = false; +}; + +// A loaded LTX-2.5 checkpoint set. Construct through +// `vllm::multimodal::LoadVideoEngine` (detection) or by declaring +// `family = kLtx2VideoFamily`; this type is exposed so a test can name it. +class Ltx2VideoEngine : public VideoEngine { + public: + static std::unique_ptr Load(const VideoModelParams& params); + + Ltx2VideoEngine(Ltx2VideoEngine&&) noexcept; + Ltx2VideoEngine& operator=(Ltx2VideoEngine&&) noexcept; + ~Ltx2VideoEngine() override; + + std::string family() const override; + vt::Device device() const override; + bool has_encoder() const override; + bool has_prompt_embeds() const override; + VideoResult Generate(const VideoGenParams& params) override; + + // The `model_version` this engine resolved its recipe with ("2.5"), and the + // pipeline kind it resolved with. Exposed because "which recipe ran" is the + // one thing a rendered clip cannot be inspected for. + const std::string& model_version() const; + const std::string& pipeline_kind() const; + + // The DiT parameters THIS ENGINE LOADED — the ones its forward actually runs + // under, after the checkpoint's own declared config has been adopted. + // + // Exposed because the values that config decides are exactly the ones a + // rendered clip cannot be inspected for and a SHAPE cannot see: + // `double_precision_rope` and `av_ca_timestep_scale_multiplier` move every + // RoPE angle and every audio<->video modulation while leaving the tensor set + // byte-identical. A test that re-derives them from the file and asserts on its + // own local copy proves nothing about what the engine bound; this accessor is + // what lets it assert on the engine. + const Ltx2DitParams& dit_params() const; + + // The conditioning the LAST `Generate()` fed cross-attention. `tokens == 0` + // before the first generation. See `Ltx2ConditioningTrace`. + // + // BY VALUE, under the same mutex `Generate` holds. Returning a reference was a + // data race on the exact use this accessor is FOR: the header offers it to a + // server ("which conditioning produced this clip"), and a server calls it from + // a thread that is not the one inside `Generate`. A reference hands the caller + // a `std::string` and two digests that a concurrent `Generate` is rewriting, + // and the lock cannot help because it is released before the caller reads. A + // copy taken while the writer is excluded is the only form that is safe to + // hand out. Costs one short string copy per call, on a path that renders video. + Ltx2ConditioningTrace last_conditioning() const; + + private: + Ltx2VideoEngine(); + struct Impl; + std::unique_ptr impl_; +}; + +// Does this checkpoint set hold an LTX-2.5 DiT? Exposed for the registry and for +// a test that wants the answer without a load. See the definition for the +// discriminator and for why it cannot collide with MiniMax-H3's. +bool DetectLtx2Video(const VideoModelParams& params); + +} // namespace vllm::multimodal diff --git a/include/vllm/multimodal/minimax_h3_video.h b/include/vllm/multimodal/minimax_h3_video.h index 3c308dd50..4156d510f 100644 --- a/include/vllm/multimodal/minimax_h3_video.h +++ b/include/vllm/multimodal/minimax_h3_video.h @@ -28,6 +28,15 @@ // Byte-identity contract: on the committed fold fixture the CPU t2va render is // byte-identical to the PRE-fold `minimax-h3-gen` binary at fc636c76 // (tests/vllm/models/test_minimax_h3_video_fold.cpp, three-arm gate). +// +// LTX-2.5 L1 (.agents/specs/ltx-2-5.md §5, issue #435): H3 is now ONE FAMILY +// behind the generalized `vllm::multimodal::VideoEngine` seam, registered as +// "minimax-h3" and reached through `LoadVideoEngine`. Nothing below changed +// behaviourally — the H3-typed params/result structs, `Load`, and the H3-typed +// `Generate` are the same entry points on the same code path, and the fold +// gate above still holds them to the same golden bytes. What is new is that the +// same handle also satisfies the abstract seam, and that the H3-specific +// `partition` field maps to/from the generic `extras["partition"]`. #pragma once #include @@ -35,6 +44,7 @@ #include #include +#include "vllm/multimodal/video_engine.h" #include "vt/device.h" namespace vllm::openai { @@ -43,6 +53,10 @@ struct VideoRequest; // entrypoints/openai/video_api.h namespace vllm::multimodal { +// The stable registry name this family is reached under (VideoModelParams::family +// / vllm_video_model_params.family). +inline constexpr char kMiniMaxH3VideoFamily[] = "minimax-h3"; + // Map the stable public video ABI device selector onto the runtime's generic // backend key. The ABI remains 0=CPU / 1=CUDA; callers below this seam dispatch // only through the returned DeviceType. @@ -118,32 +132,52 @@ struct MiniMaxH3VideoResult { // A loaded H3 video checkpoint set, weights staged once, ready to generate. // Construction throws std::runtime_error naming the problem on any mismatch. -class MiniMaxH3VideoEngine { +// +// It IS a VideoEngine (L1): the generic overrides below are thin adapters over +// the H3-typed members — same Impl, same forward, same bytes. +class MiniMaxH3VideoEngine final : public VideoEngine { public: static std::unique_ptr Load(const MiniMaxH3VideoModelParams& params); MiniMaxH3VideoEngine(MiniMaxH3VideoEngine&&) noexcept; MiniMaxH3VideoEngine& operator=(MiniMaxH3VideoEngine&&) noexcept; - ~MiniMaxH3VideoEngine(); + ~MiniMaxH3VideoEngine() override; + + // Always kMiniMaxH3VideoFamily. + std::string family() const override; // The device selected by the queue created during Load(). - vt::Device device() const; + vt::Device device() const override; // True when an encoder tower is loaded (the request PROMPT conditions the // render); false => prompt_embeds_path conditioning (or Generate refuses). - bool has_encoder() const; - bool has_prompt_embeds() const; + bool has_encoder() const override; + bool has_prompt_embeds() const override; // Run one blocking generation. Serialized internally (the staged weights are // shared state); throws std::runtime_error to fail the request. MiniMaxH3VideoResult Generate(const MiniMaxH3VideoGenParams& params); + // The generic spelling: converts, calls the H3-typed Generate above, converts + // back. There is no second code path. + VideoResult Generate(const VideoGenParams& params) override; + private: MiniMaxH3VideoEngine(); struct Impl; std::unique_ptr impl_; }; +// ── The generic <-> H3 param mapping (single-sourced so the C ABI, the HTTP +// route and the registry loader cannot drift). `partition` is the ONE +// H3-specific load field, and it rides in the generic `extras` under the key +// "partition" — absent means declared-but-unknown, exactly as an empty +// MiniMaxH3VideoModelParams::partition does (the #77 guard). ─────────────────── +MiniMaxH3VideoModelParams MiniMaxH3VideoModelParamsFromGeneric(const VideoModelParams& params); +VideoModelParams MiniMaxH3VideoModelParamsToGeneric(const MiniMaxH3VideoModelParams& params); +MiniMaxH3VideoGenParams MiniMaxH3VideoGenParamsFromGeneric(const VideoGenParams& params); +VideoResult MiniMaxH3VideoResultToGeneric(const MiniMaxH3VideoResult& result); + // The ONE mapping from a parsed /v1/videos request onto the seam's params — // library-owned so the HTTP route and the FFI cannot drift (the pre-fold // server carried this as a private lambda). `output_dir` is the job directory diff --git a/include/vllm/multimodal/video_engine.h b/include/vllm/multimodal/video_engine.h new file mode 100644 index 000000000..3119fa8ca --- /dev/null +++ b/include/vllm/multimodal/video_engine.h @@ -0,0 +1,239 @@ +// The GENERALIZED video+audio generation seam — the abstract engine every +// diffusion video family is reached through, and the checkpoint-detected +// registry that resolves WHICH family a checkpoint set belongs to. +// +// Why this exists. ARCH-ONE-SURFACE ROW 2 folded MiniMax-H3's whole assembly +// pipeline into ONE library entry point (MiniMaxH3VideoEngine), and the C ABI's +// `vllm_video_*` slice was already shaped generically — separate `dit_path` / +// `encoder_path` / `video_vae_path` / `audio_vae_path` artifacts rather than one +// model directory. Only the INTERNALS were H3-typed. LTX-2.5 is the second +// family (a 21B joint video+audio flow-matching DiT), and AGENTS.md §"Shared +// seams" is explicit that new models are ADDITIVE files reached through the +// shared surface — never a second parallel path. So the H3-typed entry point +// becomes one implementation of this interface, and a new family is a new file +// plus one REGISTER_VLLM_VIDEO_FAMILY line. +// Spec: .agents/specs/ltx-2-5.md §5. Issue #435. +// +// FAMILY-SPECIFIC FIELDS DO NOT LAND HERE. H3's `partition` (fl2va|ref2va) and +// LTX's `pipeline_kind` / `model_version` ride in `extras`, a string map, so +// adding a family adds no member to a struct every other family must then +// ignore — the same reason vLLM keeps per-architecture knobs in the config dict +// rather than in EngineArgs. +// +// NEVER GUESS A FAMILY. `LoadVideoEngine` either honours an explicitly declared +// `family`, or asks every registered family to look at the checkpoint. Exactly +// one claimant loads; zero or several is a std::runtime_error naming what was +// seen and what is registered. "There is only one family registered, so it must +// be that one" is precisely the silent mis-load this seam exists to prevent — +// an H3 GGUF handed to an LTX loader does not fail, it renders noise. +#pragma once + +#include +#include +#include +#include +#include +#include + +#include "vt/device.h" + +namespace vllm::openai { +struct VideoRequest; // entrypoints/openai/video_api.h +} + +namespace vllm::multimodal { + +// ── Load-time parameters: the checkpoint SET. Empty string == "not supplied". +// The C ABI mirror is vllm_video_model_params. ─────────────────────────────── +struct VideoModelParams { + std::string dit_path; // the denoiser: GGUF | safetensors | shard DIR + std::string encoder_path; // the text tower, when the family has one + std::string tokenizer_path; // tokenizer.json (prompt conditioning) + std::string video_vae_path, video_vae_config_path; + std::string audio_vae_path, audio_vae_config_path; + // Fallback conditioning when no encoder is configured: rows of the family's + // text width, little-endian f32. + std::string prompt_embeds_path; + + // The family to load. EMPTY means "detect from the checkpoint"; a non-empty + // value that is not registered is refused (it is never treated as a hint). + std::string family; + + int32_t device = 0; // 0 cpu, 1 cuda + int32_t dequant_bf16 = 0; // 0 keep-quant, 1 dequant/stream bf16 + int32_t fp4_resident = 0; // keep packed FP4 resident + quantized GEMM + int64_t encoder_max_layers = 0; // 0 => all layers + + // Family-specific load knobs. H3: "partition" = "fl2va" | "ref2va". + std::map extras; +}; + +// ── Per-generation parameters (the C ABI mirror is vllm_video_params). ─────── +struct VideoGenParams { + std::string prompt; // encoded when the engine has an encoder + // "" => resolved by the family from the references it was given. + std::string task; + double duration_seconds = 0.0; // <= 0 => per-task default + int64_t num_frames = 0; // <= 1 => per-task default + int64_t height = 0, width = 0; // <= 0 => aspect-derived default + int64_t steps = 0; // <= 0 => the family's default + double flow_shift = 0.0; // <= 0 => the family's default + double audio_flow_shift = 0.0; // <= 0 => the family's default + uint64_t seed = 0; + bool has_seed = false; // false => the family's fixed default noise streams + + // KEYFRAMES: binary PPM (P6), as a path or in-memory bytes (exactly one + // spelling per frame). Pin frame 0 / the last frame OF THE OUTPUT. + std::string first_frame_path, last_frame_path; + std::string first_frame_ppm; // in-memory alternative (server data: URLs) + double noise_aug = 1.0; // condition-noise augmentation (1.0 pins) + + // REFERENCES (exclusive with keyframes). + std::vector ref_image_paths; // whole reference images (PPM) + std::string ref_video_dir; // DIR of frame_%06d.ppm + std::string ref_audio_path; // 16-bit PCM WAV path... + std::string ref_audio_wav; // ...or its bytes + + // Where frame_%06d.ppm + audio.wav land (created if absent). REQUIRED. + std::string output_dir; + + // Family-specific per-request knobs (none for H3 today). + std::map extras; +}; + +// ── One finished generation (the C ABI mirror is vllm_video_result). ──────── +struct VideoResult { + std::string frame_dir; // holds frame_%06d.ppm + std::string audio_path; // 16-bit PCM WAV + int64_t frame_count = 0, width = 0, height = 0; + int64_t fps = 0, sample_rate = 0; + // The ffmpeg argv the CALLER may exec (argv[0] is "ffmpeg"). THE PROCESS + // BOUNDARY is part of this seam's contract, inherited from ROW 2: the library + // writes artifacts and COMPOSES the argv, and spawns nothing. + std::vector mux_argv; + std::string mux_output_path; // the -o target mux_argv names +}; + +// Read a family-specific extra, or `fallback` when the key is absent. +std::string VideoExtra(const std::map& extras, const std::string& key, + const std::string& fallback = std::string()); + +// Every tensor name a checkpoint artifact DECLARES, header-only — no payload is +// read, so this is safe on a checkpoint far larger than RAM. `path` may be a +// single GGUF file (including a llama.cpp split set), a single safetensors +// file, or a multi-shard directory (read through its index's weight_map, the +// same two index spellings the shard loaders accept). Returns false with *why +// holding the reason when the artifact cannot be enumerated. +// +// This is the ONE thing every family detector inspects, so that detection is a +// question about what a checkpoint HOLDS rather than what its filename says, +// and so two families cannot disagree about how a checkpoint was read. +bool ReadVideoCheckpointTensorNames(const std::string& path, std::vector* out, + std::string* why); + +// A loaded video checkpoint set, weights staged once, ready to generate. +class VideoEngine { + public: + virtual ~VideoEngine(); + + // The stable registry name of the family this engine implements. Stable + // because it is what a caller DECLARES to skip detection and what an + // unresolved load prints. + virtual std::string family() const = 0; + + // The device the queue created during load selected. + virtual vt::Device device() const = 0; + + // True when a text tower is loaded (the request PROMPT conditions the + // render); false => prompt-embeds conditioning (or Generate refuses). + virtual bool has_encoder() const = 0; + virtual bool has_prompt_embeds() const = 0; + + // Run one blocking generation. Implementations serialize internally (staged + // weights are shared state); throws std::runtime_error to fail the request. + virtual VideoResult Generate(const VideoGenParams& params) = 0; + + protected: + // Polymorphic base: constructible and movable only by derived classes. + VideoEngine() = default; + VideoEngine(const VideoEngine&) = default; + VideoEngine& operator=(const VideoEngine&) = default; + VideoEngine(VideoEngine&&) = default; + VideoEngine& operator=(VideoEngine&&) = default; +}; + +// ── The registry ──────────────────────────────────────────────────────────── +// Mirrors the ModelRegistry self-registration idiom (model_registry.h): each +// family registers itself from its OWN translation unit, so adding a family +// edits no shared array. + +// Does this checkpoint set belong to the family? Implementations INSPECT the +// artifact (tensor names, metadata) — never the file extension or the path +// spelling, both of which are chosen by whoever repackaged the checkpoint. A +// detector must not throw: an unreadable or unrecognizable artifact is `false`, +// and the caller reports it with the rest of the evidence. +using VideoFamilyDetector = std::function; + +// Load the checkpoint set as this family. Throws std::runtime_error naming the +// problem on any mismatch. +using VideoFamilyLoader = std::function(const VideoModelParams&)>; + +struct VideoFamilyRegistration { + std::string name; // the stable family name, e.g. "minimax-h3" + VideoFamilyDetector detect; + VideoFamilyLoader load; +}; + +// Add a family to the process-global registry. Throws std::runtime_error on an +// empty name, a missing detector or loader, or A NAME ALREADY REGISTERED — the +// last because two families sharing one name is the never-guess guarantee +// defeated from the inside: the listing shows one family, two claimants collapse +// into one name so the SEVERAL-claimants refusal cannot fire, and which loader +// runs falls to link order. Registrars run at static init, so a refusal there +// ends the process; that is intended, since a name collision is a build defect +// and a checkpoint handed to the wrong family renders noise rather than failing. +void RegisterVideoFamily(VideoFamilyRegistration registration); + +// Every registered family name, sorted and duplicate-free — invariants of the +// registry itself (RegisterVideoFamily inserts in order and refuses a +// collision), so they hold however static init ordered the TUs and however late +// a caller registers. This listing is what refusals print. +std::vector RegisteredVideoFamilies(); + +// Every registered family that CLAIMS this checkpoint set, sorted. Empty means +// nothing recognized it; more than one means the detectors overlap, which is a +// defect in whichever detector is too loose — resolved by refusing, never by +// picking one. +std::vector DetectVideoFamilies(const VideoModelParams& params); + +// Resolve the family and load. `params.family`, when non-empty, selects +// directly (an unregistered name is refused naming what IS registered). When +// empty, detection must produce exactly one claimant; zero or several throws +// std::runtime_error naming the checkpoint it inspected, what it found there, +// and every registered family. +std::unique_ptr LoadVideoEngine(const VideoModelParams& params); + +// The ONE mapping from a parsed /v1/videos request onto the seam's params — +// library-owned so the HTTP route and the FFI cannot drift, and family-agnostic +// so a second family serves the same endpoint without a second mapping. +// `output_dir` is the job directory the artifacts land in. +VideoGenParams VideoGenParamsFromRequest(const ::vllm::openai::VideoRequest& request, + const std::string& output_dir); + +// Static-init helper whose constructor performs the self-registration; used +// only through the REGISTER_VLLM_VIDEO_FAMILY macro. +struct VideoFamilyRegistrar { + explicit VideoFamilyRegistrar(VideoFamilyRegistration registration) { + RegisterVideoFamily(std::move(registration)); + } +}; + +// Registers one family from its own TU. Place at namespace scope inside +// `namespace vllm::multimodal { ... }`; `unique_tag` is any TU-unique token. +#define REGISTER_VLLM_VIDEO_FAMILY(unique_tag, family_name, detect_fn, load_fn) \ + namespace { \ + const ::vllm::multimodal::VideoFamilyRegistrar vllm_video_family_registrar_##unique_tag( \ + ::vllm::multimodal::VideoFamilyRegistration{(family_name), (detect_fn), (load_fn)}); \ + } // namespace + +} // namespace vllm::multimodal diff --git a/include/vllm/tokenizer/tokenizer.h b/include/vllm/tokenizer/tokenizer.h index effcd3c95..79e3d9fb7 100644 --- a/include/vllm/tokenizer/tokenizer.h +++ b/include/vllm/tokenizer/tokenizer.h @@ -47,6 +47,18 @@ class Tokenizer { // not byte-level BPE with a recognized Split pre-tokenizer regex (no silent // wrong tokenization). static Tokenizer FromHfJson(const std::string& tokenizer_json_path); + // The same parse, from bytes already in hand. `source` appears in error + // messages exactly where the path would. + // + // Exists because a tokenizer does not always arrive as a FILE. LTX-2.5's text + // encoder ships its tokenizer AS A TENSOR — `tokenizer_json` U8 [32169626] + // inside the one safetensors file, alongside `hf_asset__*` sidecars + // (gemma_assets.py:34-36) — so a loader that assumes a sibling + // `tokenizer.json` cannot read that checkpoint at all. Spilling 32 MB to a + // temporary file just to read it back would put the temp directory on a model + // path and leave a 32 MB file behind on every failure. + static Tokenizer FromHfJsonBytes(std::string_view tokenizer_json, + const std::string& source); // Loads a GGUF byte-level BPE vocab (tokenizer.ggml.* kvs). Throws // std::runtime_error unless tokenizer.ggml.model == "gpt2" and // tokenizer.ggml.pre is a recognized pre-tokenizer name. diff --git a/include/vllm/transformers_utils/hf_config.h b/include/vllm/transformers_utils/hf_config.h index 79d5a7f59..afacdd8e7 100644 --- a/include/vllm/transformers_utils/hf_config.h +++ b/include/vllm/transformers_utils/hf_config.h @@ -132,6 +132,23 @@ struct HfConfig { // path) on missing file, malformed JSON, or missing required fields. HfConfig LoadHfConfig(const std::string& path); +// The same parse, from a config object already in hand. `source` appears in +// every error message exactly where the path would, so a refusal still names +// where the config came from. +// +// Exists because a config does not always arrive as a sibling `config.json`. +// LTX-2.5's text encoder ships as ONE safetensors file whose HuggingFace config +// rides in the file's own `__metadata__["gemma_config"]` +// (gemma_assets.py:34, :110-114) — and the shipped `vonkaiser` NVFP4 build +// carries no `__metadata__` at all, so its caller has to source the config out +// of band and hand it over as an object. Writing it to a temporary file just to +// read it back would make the temp directory part of a model path. +// +// Reads no sibling `generation_config.json`: without a path there is no sibling +// to read, so `generation_config_eos_ids` stays empty rather than picking up +// whatever happens to sit in the working directory. +HfConfig ParseHfConfig(const nlohmann::json& doc, const std::string& source); + // Cheap, non-throwing peek at config.json's `architectures` array — empty on // any parse/read problem. Exists for TASK dispatch BEFORE the full text-model // HfConfig parse: a SupportsTranscription-only checkpoint (Parakeet) nests its diff --git a/include/vt/op_provider.h b/include/vt/op_provider.h index 1e6983b6a..3446b465c 100644 --- a/include/vt/op_provider.h +++ b/include/vt/op_provider.h @@ -126,9 +126,18 @@ void RegisterOpProvider(OpId op, DeviceType device, const OpProvider& provider); // keeps winning, because nothing else registers at priority 0 under that name. void RegisterOp(OpId op, DeviceType device, void* fn); +// The canonical spelling of an op — the enumerator without its `k`. The +// counterpart of DeviceTypeName (include/vt/device.h), and what every refusal +// and reference-tier warning in this seam reports, so no reader has to count +// enumerators in include/vt/ops.h to find out what was refused. Total over the +// enum: the defining switch is exhaustive and `default`-free, so appending an +// OpId without naming it fails the -Werror build. +const char* OpName(OpId op); + // The selected provider's kernel for (op, device): highest priority whose // `supports(caps)` holds, ties by name. Throws when nothing is registered or -// nothing supports the device — the pre-existing GetOp contract, unchanged. +// nothing supports the device — the pre-existing GetOp contract, naming the op +// and the device rather than their integers. void* GetOp(OpId op, DeviceType device); // DECLINE-AND-FALL-BACK. A provider kernel that cannot serve a particular call diff --git a/include/vt/ops.h b/include/vt/ops.h index 903405892..4d3f533ba 100644 --- a/include/vt/ops.h +++ b/include/vt/ops.h @@ -107,6 +107,7 @@ enum class OpId : uint8_t { kMoeRouterTopK, kMoeCombine, kAttention, + kAttentionCross, kAttentionDenseFast, kAttentionDenseFlash, kAttentionDenseFa2, @@ -384,6 +385,18 @@ enum class OpId : uint8_t { kMamba2ChunkScan, kMamba2StateUpdate, kRmsNormGatedGroup, + // LTX-2.5 DiT device-resident-forward glue table (phase L8). Only the seven + // small ops the shared vt:: surface does NOT already cover: the AdaLN table + // lookup, the AdaZero affine, the gated residual accumulate, the per-head + // attention gate, LTX's split/interleaved RoPE, the output head's + // table+embedded affine, and plain ungated SiLU. Everything else in the DiT + // forward reuses tuned shared ops (kMatmulBT, kRmsNorm, kLayerNorm, kGeluTanh, + // kAdd, kAttention, kAttentionCross), so this table stays deliberately small. + // Registered on BOTH kCPU and kCUDA (cpu_ltx2.cpp / cuda_ltx2.cu) so the + // device forward is exercised in CPU CI too; resolved via ltx2::Ltx2Device(). + // Additive: only Ltx2DitForwardDevice dispatches it. Appended before kCount + // so no existing op's id shifts. + kLtx2, kCount }; @@ -552,6 +565,26 @@ struct AttentionArgs { bool causal = true; }; +// Dense NON-CAUSAL CROSS attention args. vt::Attention requires key/value to +// carry the SAME token count as query (ops.cpp: "query/key/value token count must +// match"), which no cross-attention can satisfy: LTX-2.5's text cross-attention +// (transformer.py:113 `attn2`) and its audio<->video cross-attention +// (transformer.py:154 `audio_to_video_attn`, :166 `video_to_audio_attn`) each +// project queries from one stream and keys/values from another, so Tq != S by +// construction. This is that seam, kept SEPARATE from vt::Attention so every +// existing self-attention call stays byte-identical. +// +// query [Tq,Hq,D], key/value [S,Hkv,D], out [Tq,Hq,D]; Hq a multiple of Hkv +// (GQA broadcast, exactly as vt::Attention). Query i attends to EVERY key j in +// [0,S) — bidirectional, no causal mask; the only masking is the optional +// additive `bias` (see vt::AttentionCross). f32 softmax with max subtraction. +struct AttentionCrossArgs { + // Softmax scale applied to the qk dot product. torch SDPA's default is + // E**-0.5 with E = query.size(-1) = head_dim, which is what every LTX + // attention gets (attention.py:98). Must be set explicitly (> 0). + float scale = 0.0f; +}; + // --- Conformer / FastConformer audio-encoder op args (spike // .agents/specs/parakeet-conformer-encoder.md P1/P2/P3). ------------------------ @@ -1037,6 +1070,8 @@ using MoeCombineGateFn = void (*)(Queue&, Tensor&, const Tensor&, const Tensor&, const Tensor&); using AttentionFn = void (*)(Queue&, Tensor&, const Tensor&, const Tensor&, const Tensor&, const AttentionArgs&); +using AttentionCrossFn = void (*)(Queue&, Tensor&, const Tensor&, const Tensor&, const Tensor&, + const Tensor* /*bias*/, const AttentionCrossArgs&); // Conformer / FastConformer audio-encoder kernels (spike P1/P2/P3). using Conv2dFn = void (*)(Queue&, Tensor& /*out*/, const Tensor& /*x*/, const Tensor& /*weight*/, const Tensor* /*bias*/, const Conv2dArgs&); @@ -2375,6 +2410,45 @@ void MoeCombineGate(Queue& q, Tensor& out, const Tensor& expert_out, const Tenso void Attention(Queue& q, Tensor& out, const Tensor& query, const Tensor& key, const Tensor& value, const AttentionArgs& args); +// --- Dense non-causal CROSS attention (LTX-2.5 L2). See AttentionCrossArgs for +// why vt::Attention cannot serve it. Per q-head h (kv-head g = h/(Hq/Hkv)), +// query i in [0,Tq), keys j in [0,S): +// s[j] = scale * (query[i,h] · key[j,g]) + bias[i or 0, j] +// p = softmax_j(s) (f32, max-subtracted) +// out[i,h] = Σ_j p[j] * value[j,g] +// `bias` is an OPTIONAL rank-2 additive score bias [Tq, S] or [1, S] (the +// broadcast key-only form a padding mask produces), f32, on the same device; +// nullptr means no bias. It is additive in torch SDPA's sense — upstream builds +// it as `(mask - 1) * finfo.max` for the prompt mask (transformer_args.py:204) +// and as log-space attenuation for the self-attention strength mask (:232-237), +// so a fully masked key reaches the softmax as a large negative number, NOT as +// -inf, and an all-masked row degenerates to a uniform average exactly as torch's +// does. All softmax/accumulation math is f32. +// +// BACKENDS, recorded so it cannot be discovered later: this op ships with a CPU +// kernel (`AttentionCrossKernel`, src/vt/cpu/cpu_ops.cpp) AND — since phase L8, +// 2026-08-12 — a NATIVE CUDA kernel (src/vt/cuda/cuda_attention_cross.cu). An +// earlier revision of this paragraph recorded the CUDA one as OWED "alongside the +// LTX-2.5 device-resident forward, which is the first caller that would need it"; +// that caller arrived and so did the kernel, in the same change, and a CUDA +// device now has a real provider rather than a refusal or a unified-memory +// fallback to the host. +// +// The CUDA kernel is a structural port of `AttentionDenseFlashKernel` +// (src/vt/cuda/cuda_ops.cu), generalized on the three axes AttentionCrossArgs +// exists for. It uses the online-softmax recurrence where the CPU kernel uses the +// explicit three-pass max/exp/normalize, so the two agree to f32 summation-order +// slack and are NOT bit-identical — the same relationship `AttentionDenseFast` +// already has with `AttentionKernel`. +// +// A device with NO provider — kXPU, say — still refuses through `GetOp`, naming +// this op via `vt::OpName` rather than falling back. Callers route on what a call +// MEANS, not on whether its numbers happen to be square, so that refusal is +// deterministic per call site instead of per prompt length — see Ltx2Attention +// (src/vllm/model_executor/models/ltx2.cpp). +void AttentionCross(Queue& q, Tensor& out, const Tensor& query, const Tensor& key, + const Tensor& value, const Tensor* bias, const AttentionCrossArgs& args); + // --- Conformer / FastConformer audio-encoder kernels ------------------------- // Spike: .agents/specs/parakeet-conformer-encoder.md (rows P1/P2/P3). Upstream // mirror: transformers 5.3.0 transformers/models/parakeet/modeling_parakeet.py, diff --git a/scripts/check-agent-record.py b/scripts/check-agent-record.py index ced9050f9..8dc1b001c 100644 --- a/scripts/check-agent-record.py +++ b/scripts/check-agent-record.py @@ -57,7 +57,15 @@ # (324/373/356/310/261 is unchanged), because like the MuseGlimmer, KimiK3 # and MiniMaxH3DiT rows they carry no pinned-registry target. Bumped # because seven new rows EXIST, never to make a transition pass. - "MODEL": (AGENTS / "model-matrix.md", 369), + # 363 since 2026-08-11: +`MODEL-DIFFUSION-ltx-2-5-ltx2-video-transformer-3d-model` + # (Lightricks LTX-2.5, 21.00B joint video+audio DiT, released 2026-08). A FOURTH + # beyond-pin row, and like Muse Glimmer it is absent from `555967922` because it + # did not exist yet. Unlike the others it is also out-of-repo: the architecture + # reference is Lightricks' own `LTX-2` (`ltx-core`), and vLLM-Omni's `ltx2` module + # stops at 2.3 (`ltx2_recipes.py:162-166`), with 2.5 still OPEN upstream at + # vllm-omni#6066. Same lane as the MiniMax-H3 diffusion row. Bumped because a new + # row EXISTS, never to make a transition pass. + "MODEL": (AGENTS / "model-matrix.md", 370), # 82 since 2026-07-21: +`QUANT-NVFP4-CT-W4A16` (compressed-tensors NVFP4A16 / # W4A16 — NVFP4 weights with BF16 activations, distinct from the existing # `QUANT-NVFP4-CT-W4A4` and `QUANT-NVFP4-MO-W4A16` rows in both scheme diff --git a/scripts/gen-ltx2-gemma-tower-goldens.py b/scripts/gen-ltx2-gemma-tower-goldens.py new file mode 100644 index 000000000..507860af4 --- /dev/null +++ b/scripts/gen-ltx2-gemma-tower-goldens.py @@ -0,0 +1,656 @@ +#!/usr/bin/env python3 +"""Emit tests/vllm/models/ltx2_gemma_tower_goldens.inc — the Gemma-4 TOWER oracle. + +Phase L3 recorded that this gate could not exist: the `transformers` on the box +had no `gemma4_unified` in `CONFIG_MAPPING`, so the tower could not be built at +reduced dimensions and there was nothing to compare against. That is no longer +true, and this generator is the proof — it BUILDS AND RUNS the tower. + +What it gates, and why each piece is here rather than assumed: + + * `Gemma4Model::ForwardHiddenStates` against the REAL upstream forward. Until + now the Gemma-4 port's own header said the forward was "grounded + compiles" + with the end-to-end gate BLOCKED (gemma4.h, G1 HONEST STATUS). Compiling is + not running, and this is the first execution-grounded comparison. + + * The MIXED per-layer attention geometry the LTX tower actually ships. The + shipped 12B has 48 layers in an 8-fold `sliding, sliding, sliding, sliding, + sliding, full` pattern, and the two kinds are NOT the same shape: + + sliding : 16 q heads x head_dim 256, 8 kv heads, v_proj PRESENT + full : 16 q heads x global_head_dim 512, 1 kv head, NO v_proj + (attention_k_eq_v: V aliases K), rope_type "proportional" + with partial_rotary_factor 0.25 and theta 1e6, against the + sliding layers' plain rope at theta 1e4 + + A fixture with one uniform layer type cannot separate a port that handles + both from a port that handles one and silently applies it twice, which is + §7.0(c)'s "a fixture that cannot discriminate is the same defect as an + unpinned constant". The reduced arch below keeps BOTH kinds and keeps the + ratios (global head_dim = 2 x sliding head_dim, global kv heads = 1). + + * The LEFT-PADDING EQUIVALENCE, which is what makes a prompt affordable. + Upstream pads every prompt to 1024 (gemma_assets.py:162, + base_encoder.py:231-236, PaddingSide.LEFT) and runs all 1024 rows through a + 12B tower. Our port runs only the VALID tokens at their ORIGINAL absolute + positions. That is equivalent -- pads are masked out of attention and are + causally before every valid token, and the feature extractor zeroes their + rows anyway -- but "is equivalent" is a claim, so section 4 emits the full + left-padded oracle run and the C++ suite holds the short run's valid rows + to the padded run's valid rows. If the equivalence is ever false, that gate + is what says so, and a 100x cost claim stops resting on an argument. + + * ONE arithmetic width per state, both ways round. Section 2 is the oracle in + float32 and section 3 the SAME oracle in bfloat16; sections 4 and 4b are + that pair again for the left-padded run. Our forward carries the stream in + bf16 and widens only on the way out (gemma4.h, Gemma4HiddenStatesResult), + so bf16 is the dtype-MATCHED arm and f32 is the arm that would catch a + reduction-order defect a bf16 store absorbs. Gating only one of them has + burned this project before. + + Every leg runs on a DEEP COPY of the module. `nn.Module.to(dtype)` converts + in place and bf16 rounding is destructive, so before that fix the two + "float32" legs after the bf16 one were executing over bf16-ROUNDED weights + -- MEASURED at up to 3.90e-02 per state, of the same order as the noise + floor the tolerance itself is derived from. + + * The ROPE TABLES, in f32 (section 6), because `partial_rotary_factor` is not + resolvable from the hidden states at any fixture size this generator can + build. Section 6's note carries the measurement that establishes it. + +Both sides rebuild every weight from one deterministic FNV-1a + splitmix64 +stream keyed by the parameter's own HuggingFace NAME, exactly as +scripts/gen-ltx2-text-goldens.py does, so no weight byte is checked in and the +weight-NAMING contract is itself part of the gate. A port that merges q/k/v in +the wrong order, or that reads `v_proj` on a full-attention layer that has none, +builds a different tensor from the same names and fails. + +Upstream sources: + transformers/models/gemma4_unified/modeling_gemma4_unified.py -> the tower + Lightricks/LTX-2 packages/ltx-core/src/ltx_core/text_encoders/gemma/ + encoders/base_encoder.py:68 -> model.model(..., output_hidden_states=True) + encoders/encoder_configurator.py:68-73 -> AutoModelForImageTextToText.from_config + encoders/base_encoder.py:231-236 -> max_length 1024, PaddingSide.LEFT + diffusers src/diffusers/pipelines/ltx2/pipeline_ltx2.py:347-352 -> the same + call and the stack/flatten pack, read as an INDEPENDENT second opinion + +The real tower's `text_config` is not invented here: it is the one carried in +`__metadata__["gemma_config"]` of the OFFICIAL bf16 text encoder +(Lightricks/LTX-2.5, text_encoders/gemma4-12b-with-proj-ltx-2.5-bf16.safetensors), +read with an 84 KB HTTP range request over the safetensors header and committed +as `--real-config`. The SHIPPED vonkaiser NVFP4 file carries no `__metadata__` +at all, which is why the config has to come from somewhere and why it is not +guessed. + +Usage: + scripts/gen-ltx2-gemma-tower-goldens.py \ + --python /home/mudler/recon-cpu/venv/bin/python \ + --out tests/vllm/models/ltx2_gemma_tower_goldens.inc + +Needs a `transformers` that registers `gemma4_unified` (>= 5.8; MEASURED working +at 5.12.1, MEASURED absent at 5.3.0) plus torch and numpy. CPU only. No +checkpoint, no download, no network. +""" + +from __future__ import annotations + +import argparse +import copy +import json +import os +import subprocess +import sys +from pathlib import Path + +import numpy as np +import torch + +# --------------------------------------------------------------------------- +# ORACLE IDENTITY — asserted, never assumed (spec 7.0(1)). +# +# A decoy once produced byte-identical goldens on this project, so the module +# that answers has to be the module we meant. `transformers` 5.3.0 is present on +# the same box under a different interpreter and does NOT register +# `gemma4_unified`; picking it up by accident reads exactly like "Gemma-4 is +# unsupported" rather than like a wrong environment. +# --------------------------------------------------------------------------- + +MIN_TRANSFORMERS = (5, 8) + + +def assert_oracle_identity() -> dict: + import transformers # noqa: PLC0415 + from transformers import CONFIG_MAPPING # noqa: PLC0415 + + version = transformers.__version__ + parts = [] + for piece in version.split(".")[:2]: + digits = "".join(c for c in piece if c.isdigit()) + parts.append(int(digits) if digits else 0) + if tuple(parts) < MIN_TRANSFORMERS: + raise SystemExit( + f"transformers {version} is below the {MIN_TRANSFORMERS[0]}." + f"{MIN_TRANSFORMERS[1]} that first registers gemma4_unified. " + "Point --python at an interpreter whose transformers has it; this " + "is the exact blocker phase L3 recorded." + ) + if "gemma4_unified" not in CONFIG_MAPPING: + raise SystemExit( + f"transformers {version} at {transformers.__file__} does not register " + "'gemma4_unified' in CONFIG_MAPPING. Refusing to emit goldens from a " + "tower this interpreter cannot build." + ) + return { + "transformers_version": version, + "transformers_path": str(Path(transformers.__file__).resolve().parent), + "torch_version": torch.__version__, + } + + +def upstream_revision(root: Path) -> str: + """HEAD, plus a DIRTY marker. + + `git rev-parse HEAD` reports the committed SHA whether or not the tree has + uncommitted edits, so a revision recorded from it alone can name a commit + that is not what ran (spec 7.0(2)). + """ + if not root.is_dir(): + return "absent" + try: + head = subprocess.run( + ["git", "-C", str(root), "rev-parse", "HEAD"], + check=True, capture_output=True, text=True, + ).stdout.strip() + status = subprocess.run( + ["git", "-C", str(root), "status", "--porcelain"], + check=True, capture_output=True, text=True, + ).stdout.strip() + except Exception: # noqa: BLE001 - a tarball checkout has no git metadata + return "unknown" + return head + ("-DIRTY" if status else "") + + +# --------------------------------------------------------------------------- +# The deterministic stream — byte-identical to scripts/gen-ltx2-text-goldens.py +# and to tests/vllm/models/test_ltx2_text_encoder.cpp :: Ltx2Rand. +# --------------------------------------------------------------------------- + +_MASK64 = (1 << 64) - 1 + + +def fnv1a64(name: str) -> int: + h = 0xCBF29CE484222325 + for byte in name.encode("utf-8"): + h ^= byte + h = (h * 0x100000001B3) & _MASK64 + return h + + +def splitmix64(x: int) -> int: + x = (x + 0x9E3779B97F4A7C15) & _MASK64 + z = x + z = ((z ^ (z >> 30)) * 0xBF58476D1CE4E5B9) & _MASK64 + z = ((z ^ (z >> 27)) * 0x94D049BB133111EB) & _MASK64 + return z ^ (z >> 31) + + +def ltx2_rand(name: str, count: int) -> np.ndarray: + seed = fnv1a64(name) + out = np.empty(count, dtype=np.float64) + for i in range(count): + u = splitmix64((seed + i) & _MASK64) + out[i] = (u >> 11) * (2.0**-53) * 2.0 - 1.0 + return out + + +def gemma_param_spec(name: str) -> tuple[float, float]: + """(scale, offset) keyed ONLY by the HuggingFace parameter name. + + The offsets are not cosmetic. Gemma-4's RMSNorm is PLAIN — it multiplies by + `self.weight` directly rather than by `1 + weight` + (modeling_gemma4_unified.py:181-185), and its weights initialize to ones — so + a norm weight centred on 0 would put every normalized activation near zero + and hide a scale error. `layer_scalar` is a BUFFER initialized to ones + (:501) and applied as `hidden_states *= self.layer_scalar` (:535); leaving it + at exactly 1 would make a port that ignores it entirely pass. + """ + if name.endswith("layer_scalar"): + return 0.1, 1.0 + if name.endswith("_layernorm.weight") or name.endswith("norm.weight"): + return 0.1, 1.0 + if name.endswith("embed_tokens.weight"): + return 0.05, 0.0 + if name.endswith(".bias"): + return 0.02, 0.0 + return 0.05, 0.0 + + +def make_param(name: str, shape) -> torch.Tensor: + scale, offset = gemma_param_spec(name) + count = int(np.prod(shape)) if len(shape) else 1 + values = ltx2_rand(name, count) * scale + offset + return torch.from_numpy(values.astype(np.float32)).reshape(tuple(shape)) + + +# --------------------------------------------------------------------------- +# The reduced architecture. +# +# Every ratio the port branches on is preserved; only the magnitudes shrink. +# The layer_types pattern keeps the shipped tower's shape -- a run of sliding +# layers CLOSED by a full one, twice -- so a port that mixes the two head_dims +# up, or that resolves layer 0's geometry once and reuses it, diverges. +# --------------------------------------------------------------------------- + +HIDDEN = 32 # real 3840 +NUM_LAYERS = 12 # real 48 +HEAD_DIM = 8 # real 256 +GLOBAL_HEAD_DIM = 16 # real 512 +NUM_HEADS = 4 # real 16 +NUM_KV_HEADS = 2 # real 8 +NUM_GLOBAL_KV_HEADS = 1 # real 1 (unchanged: it IS one upstream) +INTERMEDIATE = 64 # real 15360 +VOCAB = 64 # real 262144 +SLIDING_WINDOW = 6 # real 1024 +MAX_POSITION = 128 # real 262144 + +# The shipped pattern is (sliding x 5, full) x 8. Two repeats is the smallest +# fixture in which a full layer is neither first nor last and a sliding layer +# FOLLOWS a full one -- the case a port that latches geometry once gets wrong. +LAYER_TYPES = (["sliding_attention"] * 5 + ["full_attention"]) * 2 +assert len(LAYER_TYPES) == NUM_LAYERS + +# Section 1/2: an unpadded run. Section 3: the same VALID tokens left-padded. +TOKENS = [2, 17, 41, 5, 23, 9, 60, 33] +SEQ = len(TOKENS) +PADDED_SEQ = 20 # > SLIDING_WINDOW, so the window is LIVE +PAD_ID = 0 +NUM_PAD = PADDED_SEQ - SEQ + + +def reduced_text_config(real: dict) -> dict: + """The REAL text_config with only the magnitudes reduced. + + Starting from the shipped config rather than from a hand-written dict is + deliberate: every field this generator does not name -- `attention_k_eq_v`, + `final_logit_softcapping`, `rms_norm_eps`, both `rope_parameters` entries, + `tie_word_embeddings`, `hidden_activation` -- is carried over EXACTLY, so the + fixture cannot quietly disagree with the checkpoint about a field nobody + thought to reduce. + """ + t = dict(real) + t.update( + hidden_size=HIDDEN, + num_hidden_layers=NUM_LAYERS, + num_attention_heads=NUM_HEADS, + num_key_value_heads=NUM_KV_HEADS, + head_dim=HEAD_DIM, + global_head_dim=GLOBAL_HEAD_DIM, + num_global_key_value_heads=NUM_GLOBAL_KV_HEADS, + intermediate_size=INTERMEDIATE, + vocab_size=VOCAB, + vocab_size_per_layer_input=VOCAB, + sliding_window=SLIDING_WINDOW, + max_position_embeddings=MAX_POSITION, + layer_types=list(LAYER_TYPES), + ) + return t + + +def build_tower(real_config: dict): + """Exactly `GemmaTextEncoderConfigurator.from_metadata` at reduced dims.""" + from transformers import CONFIG_MAPPING, AutoModelForImageTextToText # noqa: PLC0415 + + wrapper = dict(real_config) + wrapper["text_config"] = reduced_text_config(real_config["text_config"]) + # The vision and audio towers are not on the text-conditioning path and the + # checkpoint's copies of them are not reduced here; dropping the sub-configs + # makes transformers skip building them entirely. + wrapper.pop("vision_config", None) + wrapper.pop("audio_config", None) + + config_cls = CONFIG_MAPPING[real_config["model_type"]] + config = config_cls.from_dict(wrapper) + model = AutoModelForImageTextToText.from_config(config) + inner = model.model # base_encoder.py:68 -- the inner model, no lm_head + + filled = [] + with torch.no_grad(): + for name, param in inner.named_parameters(): + param.copy_(make_param(name, tuple(param.shape))) + filled.append((name, tuple(param.shape))) + # `layer_scalar` is a BUFFER, so named_parameters() misses it. It + # multiplies every layer's output (:535) and a port that drops it is + # invisible while it stays at its initialized 1.0. + for name, buf in inner.named_buffers(): + if name.endswith("layer_scalar"): + buf.copy_(make_param(name, tuple(buf.shape))) + filled.append((name, tuple(buf.shape))) + return model, inner, config, filled + + +def run_tower(inner, ids, mask, dtype, positions=None): + """One oracle leg, on a DEEP COPY of the module. + + `nn.Module.to(dtype)` converts parameters and buffers IN PLACE, so the + obvious `inner.to(dtype)` / `inner.to(torch.float32)` round trip does not + restore anything: bf16 rounding is destructive, and every leg after the + first bf16 one would run over bf16-ROUNDED weights while calling itself + float32. MEASURED on this fixture before the fix: re-running the identical + f32 call after the bf16 leg moved a state by up to 3.90e-02, which is + ~40x the f32 legs' own round-off and of the same order as the bf16 noise + floor the gate is calibrated on. Sections 4 and 5 were silently + order-dependent because of it. Copying is cheap here -- the reduced tower + is a few hundred KB -- and it makes every leg independent of leg order. + """ + m = copy.deepcopy(inner).to(dtype).eval() + kwargs = {} + if positions is not None: + # The absolute positions the tokens occupy in the padded batch. Left + # padding does NOT renumber them: with no explicit `position_ids` + # transformers derives them from `cache_position`, which counts the pad + # rows, so a short run has to be told where its tokens really sit. + kwargs["position_ids"] = torch.tensor([positions], dtype=torch.long) + with torch.no_grad(): + out = m( + input_ids=torch.tensor([ids], dtype=torch.long), + attention_mask=torch.tensor([mask], dtype=torch.long), + output_hidden_states=True, + **kwargs, + ) + return [h[0].to(torch.float32).contiguous().numpy() for h in out.hidden_states] + + +def rope_cos_sin(config, layer_type: str, head_dim: int, positions) -> np.ndarray: + """The oracle's OWN cos|sin table for one layer type, in float32. + + Why this is emitted at all: `partial_rotary_factor` is NOT resolvable from + the hidden states. MEASURED on this fixture, the whole difference between + the config's 0.25 and a port that ignored it and rotated fully is 1.09e-01 + at the worst state against a bf16 noise floor of 9.99e-02 -- a ratio of + 1.09, i.e. inside the tolerance the same states are gated at. Enlarging the + fixture does not rescue it: at (head_dim 16/32, seq 32) the ratio FALLS to + 0.65, because bf16 accumulation noise grows at least as fast as the rope + contribution does. So the end-to-end states are the wrong instrument, and + the right one is the table itself, in f32, with no accumulation in it. + + Built by the real `Gemma4UnifiedTextRotaryEmbedding`, which is what routes + `rope_type: "proportional"` to `_compute_proportional_rope_parameters` + (modeling_gemma4_unified.py:206-218 -- the ROPE_INIT_FUNCTIONS lookup at :207 + and the call at :218; modeling_rope_utils.py:187-254) and so is the only + thing that decides how many angle pairs are zero-padded. + + Returns [len(positions), head_dim]: the first head_dim/2 columns are cos + over the distinct angle pairs and the second half is sin, which is the + layout `BuildProportionalRopeCache` writes. + """ + from transformers.models.gemma4_unified.modeling_gemma4_unified import ( # noqa: PLC0415 + Gemma4UnifiedTextRotaryEmbedding, + ) + + rope = Gemma4UnifiedTextRotaryEmbedding(config) + pos = torch.tensor([list(positions)], dtype=torch.long) + x = torch.zeros(1, len(positions), head_dim, dtype=torch.float32) + cos, sin = rope(x, pos, layer_type=layer_type) + pairs = head_dim // 2 + # Upstream duplicates each angle (`emb = cat((freqs, freqs))`) so cos/sin are + # head_dim wide over head_dim/2 DISTINCT angles; take one copy of each. + table = torch.cat((cos[0, :, :pairs], sin[0, :, :pairs]), dim=-1) + return table.to(torch.float32).contiguous().numpy() + + +# --------------------------------------------------------------------------- +# Emission helpers (identical to scripts/gen-ltx2-text-goldens.py's) +# --------------------------------------------------------------------------- + + +def _cxx_float(value: float, digits: int) -> str: + if value != value: + return "NAN" + if value == float("inf"): + return "INFINITY" + if value == float("-inf"): + return "-INFINITY" + return repr(float(f"%.{digits}g" % value)) + + +def emit_f32(out, name: str, values) -> None: + flat = np.asarray(values, dtype=np.float32).reshape(-1).tolist() + out.write(f"inline constexpr float {name}[] = {{\n") + for i in range(0, len(flat), 6): + out.write(" " + ", ".join(_cxx_float(v, 9) + "f" for v in flat[i : i + 6]) + ",\n") + out.write("};\n\n") + + +def emit_i32(out, name: str, values) -> None: + flat = [int(v) for v in np.asarray(values).reshape(-1).tolist()] + out.write(f"inline constexpr int32_t {name}[] = {{\n") + for i in range(0, len(flat), 12): + out.write(" " + ", ".join(str(v) for v in flat[i : i + 12]) + ",\n") + out.write("};\n\n") + + +def emit_scalar(out, name: str, value) -> None: + out.write(f"inline constexpr int64_t {name} = {int(value)};\n") + + +def emit_string(out, name: str, value: str) -> None: + out.write(f'inline constexpr const char* {name} = R"JSON({value})JSON";\n\n') + + +# --------------------------------------------------------------------------- + + +def main() -> int: + ap = argparse.ArgumentParser() + ap.add_argument("--out", required=True, type=Path) + ap.add_argument( + "--real-config", + type=Path, + default=None, + help="the gemma_config JSON from the official bf16 TE's __metadata__; " + "defaults to the copy committed beside this script", + ) + ap.add_argument("--ltx2", type=Path, default=Path.home() / "_git" / "LTX-2") + ap.add_argument("--diffusers", type=Path, default=Path.home() / "_git" / "diffusers") + args = ap.parse_args() + + identity = assert_oracle_identity() + real_config_path = args.real_config or ( + Path(__file__).resolve().parent.parent + / "tests" / "vllm" / "models" / "ltx2_gemma4_text_config.json" + ) + real_config = json.loads(real_config_path.read_text()) + + model, inner, config, filled = build_tower(real_config) + + padded_ids = [PAD_ID] * NUM_PAD + TOKENS + padded_mask = [0] * NUM_PAD + [1] * SEQ + + plain_f32 = run_tower(inner, TOKENS, [1] * SEQ, torch.float32) + plain_bf16 = run_tower(inner, TOKENS, [1] * SEQ, torch.bfloat16) + padded_f32 = run_tower(inner, padded_ids, padded_mask, torch.float32) + # The padded run at the SHIPPED dtype too. Without it the left-padded rows + # have no dtype-matched oracle, and anything gated against them -- the + # prompt-to-conditioning path, which is left-padded by construction -- has + # no measured floor to be held to, only a borrowed one. + padded_bf16 = run_tower(inner, padded_ids, padded_mask, torch.bfloat16) + # The equivalence claim, isolated INSIDE the oracle and in f32 so no dtype + # noise is mixed into it: the same valid tokens, told their absolute + # positions, run WITHOUT the pads. If upstream's own two answers agree, the + # claim "dropping the pads is free" is upstream's property; our port then + # only has to inherit it, and the C++ gate can hold those two claims apart. + short_abs_f32 = run_tower( + inner, TOKENS, [1] * SEQ, torch.float32, + positions=list(range(NUM_PAD, NUM_PAD + SEQ)), + ) + equivalence = [ + float(np.abs(s - p[NUM_PAD:]).max()) + for s, p in zip(short_abs_f32, padded_f32) + ] + sys.stderr.write( + "left-pad equivalence measured INSIDE the oracle (f32, per state): " + f"max {max(equivalence):.3e}\n" + ) + + assert len(plain_f32) == NUM_LAYERS + 1, len(plain_f32) + + argv = " ".join([Path(sys.argv[0]).name] + sys.argv[1:]) + with args.out.open("w", encoding="utf-8") as out: + out.write( + "// GENERATED by scripts/gen-ltx2-gemma-tower-goldens.py — DO NOT EDIT BY HAND.\n" + "//\n" + "// The Gemma-4 TOWER oracle: the upstream HuggingFace implementation BUILT\n" + "// AND RUN at reduced dimensions, which phase L3 recorded as impossible on\n" + "// the transformers it had. Every weight is rebuilt on both sides from the\n" + "// deterministic Ltx2Rand stream keyed by the HuggingFace parameter NAME, so\n" + "// no weight byte is checked in and the naming contract is part of the gate.\n" + "//\n" + f"// Oracle: transformers {identity['transformers_version']} at\n" + f"// {identity['transformers_path']}\n" + f"// torch {identity['torch_version']}\n" + f"// Lightricks/LTX-2 revision: {upstream_revision(args.ltx2)}\n" + f"// diffusers revision: {upstream_revision(args.diffusers)}\n" + "// Regenerate with:\n" + f"// {argv}\n" + "#pragma once\n\n" + "#include \n\n" + "namespace vllm_test {\n\n" + ) + + out.write("// --- section 0: the reduced architecture ---\n") + emit_scalar(out, "kLtxTowerHidden", HIDDEN) + emit_scalar(out, "kLtxTowerNumLayers", NUM_LAYERS) + emit_scalar(out, "kLtxTowerNumStates", NUM_LAYERS + 1) + emit_scalar(out, "kLtxTowerHeadDim", HEAD_DIM) + emit_scalar(out, "kLtxTowerGlobalHeadDim", GLOBAL_HEAD_DIM) + emit_scalar(out, "kLtxTowerNumHeads", NUM_HEADS) + emit_scalar(out, "kLtxTowerNumKvHeads", NUM_KV_HEADS) + emit_scalar(out, "kLtxTowerNumGlobalKvHeads", NUM_GLOBAL_KV_HEADS) + emit_scalar(out, "kLtxTowerIntermediate", INTERMEDIATE) + emit_scalar(out, "kLtxTowerVocab", VOCAB) + emit_scalar(out, "kLtxTowerSlidingWindow", SLIDING_WINDOW) + emit_scalar(out, "kLtxTowerSeq", SEQ) + emit_scalar(out, "kLtxTowerPaddedSeq", PADDED_SEQ) + emit_scalar(out, "kLtxTowerNumPad", NUM_PAD) + emit_scalar(out, "kLtxTowerPadId", PAD_ID) + out.write("\n") + # The exact reduced text_config, so the C++ side PARSES what ran rather + # than reconstructing it field by field and drifting. + emit_string( + out, + "kLtxTowerTextConfigJson", + json.dumps(reduced_text_config(real_config["text_config"]), indent=1), + ) + out.write("// The layer types, in order. 1 = full_attention, 0 = sliding.\n") + emit_i32(out, "kLtxTowerLayerIsFull", + [1 if t == "full_attention" else 0 for t in LAYER_TYPES]) + + out.write("// --- section 1: the inputs ---\n") + emit_i32(out, "kLtxTowerTokens", TOKENS) + emit_i32(out, "kLtxTowerPaddedTokens", padded_ids) + emit_i32(out, "kLtxTowerPaddedMask", padded_mask) + + out.write( + "// --- section 2: hidden states, oracle in FLOAT32 ---\n" + "// [state][seq * hidden]. state 0 is the sqrt(hidden)-scaled embeddings;\n" + "// state i is decoder layer i-1's output; the LAST state is model.norm of\n" + "// the last layer, and the raw last-layer output never appears.\n" + ) + for i, s in enumerate(plain_f32): + emit_f32(out, f"kLtxTowerStateF32_{i}", s) + + out.write("// --- section 3: hidden states, oracle in BFLOAT16 (the SHIPPED dtype) ---\n") + for i, s in enumerate(plain_bf16): + emit_f32(out, f"kLtxTowerStateBf16_{i}", s) + + # THE NOISE FLOOR, MEASURED — the thing that makes the gate's tolerance a + # measurement instead of a number somebody picked. + # + # Upstream's own answer moves by this much when the SAME code runs at + # bf16 instead of f32, so it is the smallest difference this comparison + # can possibly resolve. Holding our bf16 forward to it says something + # precise: we are closer to upstream-in-bf16 than upstream-in-bf16 is to + # upstream-in-f32. It also cannot be relaxed to rescue a failing port — + # widening it means regenerating it, which means the oracle itself moved. + noise = [float(np.abs(a - b).max()) for a, b in zip(plain_f32, plain_bf16)] + scale = [float(np.abs(a).max()) for a in plain_f32] + out.write( + "// --- section 3b: the oracle's OWN f32-vs-bf16 spread, per state ---\n" + "// The dtype noise floor, MEASURED rather than assumed. This is the\n" + "// tolerance: a port whose bf16 answer sits inside it is indistinguishable\n" + "// from upstream at upstream's own arithmetic width, and one that sits\n" + "// outside it has a defect that bf16 rounding does not explain.\n" + ) + emit_f32(out, "kLtxTowerDtypeNoise", noise) + emit_f32(out, "kLtxTowerStateScale", scale) + + out.write( + "// --- section 4: the LEFT-PADDED run, float32 ---\n" + "// [state][padded_seq * hidden]. Rows 0..kLtxTowerNumPad-1 are pad rows and\n" + "// their contents are upstream's garbage-but-masked values; the gate reads\n" + "// only the VALID tail and holds section 2 to it.\n" + "// Every leg runs on a DEEP COPY of the module, so this one is not\n" + "// downstream of the bf16 leg's rounding and the sections are independent\n" + "// of the order they are produced in.\n" + ) + for i, s in enumerate(padded_f32): + emit_f32(out, f"kLtxTowerPaddedStateF32_{i}", s) + + out.write( + "// --- section 4b: the LEFT-PADDED run, BFLOAT16 (the SHIPPED dtype) ---\n" + "// The dtype-MATCHED arm for anything gated on the left-padded rows, and\n" + "// with section 4 it is also the padded run's own measured noise floor.\n" + ) + for i, s in enumerate(padded_bf16): + emit_f32(out, f"kLtxTowerPaddedStateBf16_{i}", s) + + out.write( + "// --- section 5: the equivalence, measured INSIDE the oracle ---\n" + "// Per state, max|short-run-at-absolute-positions - padded-run's valid rows|,\n" + "// both f32 and both on independently deep-copied modules, so this number\n" + "// carries NO dtype noise. It is upstream's own answer to 'is dropping the\n" + "// pads free?'. Our port inherits it; the C++ gate checks the two claims\n" + "// separately so a failure says which one broke.\n" + ) + emit_f32(out, "kLtxTowerPadEquivalence", equivalence) + + # SECTION 6 -- the rope table, because the hidden states cannot see it. + # + # `partial_rotary_factor` decides how many of the full-attention layers' + # angle pairs are rotated and how many are zero-padded to identity. It is + # config-carried, shape-invisible, and -- MEASURED, not assumed -- also + # invisible in the gated hidden states: setting it to 1.0 displaces the + # worst state by 1.09e-01 against a bf16 floor of 9.99e-02 (ratio 1.09), + # and a larger fixture makes that WORSE, not better (0.65 at head_dim + # 16/32, seq 32), because bf16 accumulation noise grows at least as fast. + # A gate whose tolerance is bf16 noise cannot resolve it, so the table + # itself is emitted and compared in f32. + full_head_dim = GLOBAL_HEAD_DIM + rope_positions = list(range(PADDED_SEQ)) + rope_full = rope_cos_sin(config.text_config, "full_attention", + full_head_dim, rope_positions) + out.write( + "// --- section 6: the ORACLE's FULL-attention rope cos|sin table, f32 ---\n" + "// [position][global_head_dim]: first half cos, second half sin, over the\n" + "// global_head_dim/2 DISTINCT angle pairs (upstream stores each twice).\n" + "// `rope_type: proportional` at theta 1e6 and partial_rotary_factor 0.25,\n" + "// so the trailing pairs are cos=1, sin=0. Emitted because the partial\n" + "// factor is NOT resolvable from the hidden states -- see the note above --\n" + "// and because a table that silently rotated every pair would still produce\n" + "// 13 finite, plausibly-scaled states.\n" + ) + emit_scalar(out, "kLtxTowerRopePositions", len(rope_positions)) + out.write("\n") + emit_f32(out, "kLtxTowerRopeFullCosSin", rope_full) + + out.write("} // namespace vllm_test\n") + + sys.stderr.write( + f"wrote {args.out} — {len(filled)} named tensors, " + f"{NUM_LAYERS + 1} states x {SEQ} x {HIDDEN}\n" + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/gen-ltx2-goldens.py b/scripts/gen-ltx2-goldens.py new file mode 100644 index 000000000..367cc4ddb --- /dev/null +++ b/scripts/gen-ltx2-goldens.py @@ -0,0 +1,689 @@ +#!/usr/bin/env python3 +"""Emit tests/vllm/models/ltx2_goldens.inc — the LTX-2.5 DiT parity oracle. + +LTX-2.5 (`Lightricks/LTX-2.5`) is a 21.00B joint video+audio flow-matching +diffusion transformer. Its DiT alone is ~19 GB even in NVFP4, and there is no +vLLM-Omni native path to it (.agents/specs/ltx-2-5.md section 3), so the port +cannot be gated end to end here. What CAN be gated exactly, on any CPU, is the +MATH: this generator IMPORTS THE UPSTREAM MODULES BY PATH and runs them at +REDUCED dimensions with deterministic pseudo-random weights, then emits the +resulting tensors as C++ goldens. The C++ suite regenerates the identical weights +and inputs from the identical PRNG and must reproduce these outputs, so not one +weight byte is checked in. + +This is MiniMax-H3's method (scripts/gen-minimax-h3-goldens.py), applied to an +upstream that is easier to reach: `ltx_core` needs only a sys.path entry pointing +at a LTX-2 checkout, no venv and no checkpoint. Nothing here is a restatement — +every tensor below comes out of upstream's own `LTXModel`, `Attention`, +`FeedForward`, `AdaLayerNormSingle` and `precompute_freqs_cis`. + +Upstream sources (Lightricks/LTX-2, packages/ltx-core/src/ltx_core/): + model/transformer/model.py -> the full DiT forward goldens + model/transformer/transformer.py -> exercised through that forward + model/transformer/attention.py -> the gated-attention brick + model/transformer/feed_forward.py -> the ff / audio_ff bias-asymmetry brick + model/transformer/adaln.py -> the AdaLN-single brick + model/transformer/rope.py -> the split / interleaved / float64 bricks + +Usage: + python3 scripts/gen-ltx2-goldens.py \ + --ltx2 ~/_git/LTX-2 \ + --out tests/vllm/models/ltx2_goldens.inc + +Needs torch + numpy (CPU only). +""" + +from __future__ import annotations + +import argparse +import math +import os +import subprocess +import sys +from pathlib import Path + +import numpy as np +import torch + +# --------------------------------------------------------------------------- +# Deterministic weight/input stream, mirrored bit-for-bit by the C++ suite +# (tests/vllm/models/test_ltx2.cpp :: Ltx2Rand). A per-tensor FNV-1a seed plus a +# splitmix64 counter makes every tensor independent of fill ORDER, so the two +# sides cannot silently drift by reordering their parameter construction. +# --------------------------------------------------------------------------- + +_MASK64 = (1 << 64) - 1 + + +def fnv1a64(name: str) -> int: + h = 0xCBF29CE484222325 + for byte in name.encode("utf-8"): + h ^= byte + h = (h * 0x100000001B3) & _MASK64 + return h + + +def splitmix64(x: int) -> int: + x = (x + 0x9E3779B97F4A7C15) & _MASK64 + z = x + z = ((z ^ (z >> 30)) * 0xBF58476D1CE4E5B9) & _MASK64 + z = ((z ^ (z >> 27)) * 0x94D049BB133111EB) & _MASK64 + return z ^ (z >> 31) + + +def ltx2_rand(name: str, count: int) -> np.ndarray: + """`count` values uniform in [-1, 1), reproducible from `name` alone.""" + seed = fnv1a64(name) + out = np.empty(count, dtype=np.float64) + for i in range(count): + u = splitmix64((seed + i) & _MASK64) + # Top 53 bits -> [0, 1), then map to [-1, 1). Both sides use the same + # 53-bit mantissa construction so the doubles are bit-identical. + unit = (u >> 11) * (2.0**-53) + out[i] = unit * 2.0 - 1.0 + return out + + +def make_param(name: str, shape, scale: float, offset: float = 0.0) -> torch.Tensor: + count = int(np.prod(shape)) if len(shape) else 1 + values = ltx2_rand(name, count) * scale + offset + return torch.from_numpy(values.astype(np.float32)).reshape(tuple(shape)) + + +def param_spec(name: str) -> tuple[float, float]: + """(scale, offset) for a parameter, keyed ONLY by its name. + + The C++ suite applies the identical rule, so a name either side invents that + the other does not have produces different numbers and the gate fails — which + is what makes the weight CONTRACT part of the gate rather than an assumption. + RMSNorm weights sit around 1.0 (torch initializes them to ones) so the q/k + norms do not crush the attention scores; everything else is small and centred. + """ + if name.endswith("q_norm.weight") or name.endswith("k_norm.weight"): + return 0.1, 1.0 + if name.endswith(".bias"): + return 0.02, 0.0 + return 0.05, 0.0 + + +def fill_parameters(module: torch.nn.Module) -> None: + for name, param in module.named_parameters(): + scale, offset = param_spec(name) + param.copy_(make_param(name, tuple(param.shape), scale, offset)) + + +# --------------------------------------------------------------------------- +# Reduced-dimension arch. Every ratio the port branches on is preserved: two +# streams of DIFFERENT width, a head count shared between them (which +# model_configurator.py:44 asserts) but DIFFERENT head dims, distinct video and +# audio latent widths, a text context per stream, gated attention on, cross +# attention AdaLN on, the prompt AdaLN MLP off, and the ff / audio_ff bias +# asymmetry. Only the magnitudes shrink. +# +# Two constraints the shapes must respect, both from rope.py: +# inner_dim // (2 * n_pos_dims) must be >= 1 (video: 32 // 6 = 5) +# audio_cross_attention_dim == audio inner (the cross RoPE is built at it) +# --------------------------------------------------------------------------- + +ARCH = dict( + num_attention_heads=4, + attention_head_dim=8, # video inner dim 32 + in_channels=8, + out_channels=8, + num_layers=2, + cross_attention_dim=32, + norm_eps=1e-6, + positional_embedding_theta=10000.0, + positional_embedding_max_pos=[20, 2048, 2048], + timestep_scale_multiplier=1000, + use_middle_indices_grid=True, + audio_num_attention_heads=4, + audio_attention_head_dim=4, # audio inner dim 16 + audio_in_channels=6, + audio_out_channels=6, + audio_cross_attention_dim=16, + audio_positional_embedding_max_pos=[20], + av_ca_timestep_scale_multiplier=1, + apply_gated_attention=True, + cross_attention_adaln=True, + use_prompt_adaln_single=False, + ff_bias=False, + audio_ff_bias=True, +) + +BATCH = 2 +VIDEO_GRID = (2, 2, 2) # (t, h, w) -> 8 video tokens +VIDEO_TOKENS = VIDEO_GRID[0] * VIDEO_GRID[1] * VIDEO_GRID[2] +AUDIO_TOKENS = 5 +VIDEO_CONTEXT = 3 +AUDIO_CONTEXT = 4 + + +def video_dim() -> int: + return ARCH["num_attention_heads"] * ARCH["attention_head_dim"] + + +def audio_dim() -> int: + return ARCH["audio_num_attention_heads"] * ARCH["audio_attention_head_dim"] + + +# --------------------------------------------------------------------------- +# Upstream import — by PATH, so no install and no environment leakage. +# --------------------------------------------------------------------------- + + +def load_upstream(root: Path): + src = root / "packages" / "ltx-core" / "src" + if not (src / "ltx_core").is_dir(): + raise SystemExit(f"not an LTX-2 checkout: {root} (expected {src}/ltx_core)") + sys.path.insert(0, str(src)) + import ltx_core # noqa: PLC0415 + + # ORACLE IDENTITY, asserted rather than assumed: an ltx_core installed in + # site-packages would import silently and gate against the wrong source. + resolved = Path(ltx_core.__file__).resolve() + if not str(resolved).startswith(str(src.resolve())): + raise SystemExit(f"ltx_core resolved to {resolved}, not to the checkout at {src}") + return src + + +def upstream_revision(root: Path) -> str: + try: + out = subprocess.run( + ["git", "-C", str(root), "rev-parse", "HEAD"], + check=True, + capture_output=True, + text=True, + ) + return out.stdout.strip() + except Exception: # noqa: BLE001 - a tarball checkout has no git metadata + return "unknown" + + +# --------------------------------------------------------------------------- +# Inputs +# --------------------------------------------------------------------------- + + +def video_positions() -> torch.Tensor: + """The middle-indices patch grid (patchifiers.py:95-138), int64 as upstream builds it.""" + t, h, w = VIDEO_GRID + coords = [] + for ti in range(t): + for hi in range(h): + for wi in range(w): + coords.append((ti, hi, wi)) + grid = torch.zeros(BATCH, 3, VIDEO_TOKENS, 2, dtype=torch.int64) + for idx, (ti, hi, wi) in enumerate(coords): + for axis, value in enumerate((ti, hi, wi)): + grid[:, axis, idx, 0] = value + grid[:, axis, idx, 1] = value + 1 + return grid + + +def audio_positions() -> torch.Tensor: + grid = torch.zeros(BATCH, 1, AUDIO_TOKENS, 2, dtype=torch.int64) + for idx in range(AUDIO_TOKENS): + grid[:, 0, idx, 0] = idx + grid[:, 0, idx, 1] = idx + 1 + return grid + + +def rand_input(name: str, shape, scale: float, offset: float = 0.0) -> torch.Tensor: + return make_param(name, shape, scale, offset) + + +# The masked case's inputs, defined ONCE so the emitted copies and the copies the +# forward actually runs on cannot drift apart. A {0,1} prompt mask (the padding +# form _prepare_attention_mask converts) plus a key-only [0,1] self-attention +# STRENGTH mask. +def video_context_mask() -> torch.Tensor: + return torch.tensor([[1, 1, 0], [1, 0, 0]], dtype=torch.int64) + + +def audio_context_mask() -> torch.Tensor: + return torch.tensor([[1, 1, 1, 0], [1, 1, 0, 0]], dtype=torch.int64) + + +def video_self_mask() -> torch.Tensor: + return make_param("input.video.attention_mask", (BATCH, 1, VIDEO_TOKENS), 0.5, 0.5) + + +def audio_self_mask() -> torch.Tensor: + return make_param("input.audio.attention_mask", (BATCH, 1, AUDIO_TOKENS), 0.5, 0.5) + + +# The DENSE `(B, T, T)` self-attention mask — the form transformer_args.py:212-215 +# documents as *the* dense one, where every QUERY carries its own row of key +# strengths. The key-only `(B, 1, T)` broadcast above cannot distinguish a kernel +# that reads each query's own bias row from one that reads row 0 for every query, +# because there is only one row; this shape is what separates them. +def video_self_mask_dense() -> torch.Tensor: + return make_param( + "input.video.attention_mask_dense", (BATCH, VIDEO_TOKENS, VIDEO_TOKENS), 0.5, 0.5 + ) + + +def audio_self_mask_dense() -> torch.Tensor: + return make_param( + "input.audio.attention_mask_dense", (BATCH, AUDIO_TOKENS, AUDIO_TOKENS), 0.5, 0.5 + ) + + +def build_modalities(masked: bool, audio_enabled: bool = True, dense_self_mask: bool = False): + from ltx_core.model.transformer.modality import Modality # noqa: PLC0415 + + video_ctx_mask = None + audio_ctx_mask = None + video_attn_mask = None + audio_attn_mask = None + if masked: + video_ctx_mask = video_context_mask() + audio_ctx_mask = audio_context_mask() + if dense_self_mask: + video_attn_mask = video_self_mask_dense() + audio_attn_mask = audio_self_mask_dense() + else: + video_attn_mask = video_self_mask() + audio_attn_mask = audio_self_mask() + + video = Modality( + latent=rand_input("input.video.latent", (BATCH, VIDEO_TOKENS, ARCH["in_channels"]), 0.5), + sigma=rand_input("input.video.sigma", (BATCH,), 0.25, 0.5), + timesteps=rand_input("input.video.timesteps", (BATCH, VIDEO_TOKENS), 0.25, 0.5), + positions=video_positions(), + context=rand_input( + "input.video.context", (BATCH, VIDEO_CONTEXT, ARCH["cross_attention_dim"]), 0.5 + ), + enabled=True, + context_mask=video_ctx_mask, + attention_mask=video_attn_mask, + ) + audio = Modality( + latent=rand_input( + "input.audio.latent", (BATCH, AUDIO_TOKENS, ARCH["audio_in_channels"]), 0.5 + ), + sigma=rand_input("input.audio.sigma", (BATCH,), 0.25, 0.5), + timesteps=rand_input("input.audio.timesteps", (BATCH, AUDIO_TOKENS), 0.25, 0.5), + positions=audio_positions(), + context=rand_input( + "input.audio.context", (BATCH, AUDIO_CONTEXT, ARCH["audio_cross_attention_dim"]), 0.5 + ), + enabled=audio_enabled, + context_mask=audio_ctx_mask, + attention_mask=audio_attn_mask, + ) + return video, audio + + +def build_model(rope_type_name: str, double_rope: bool): + from ltx_core.model.transformer.model import LTXModel, LTXModelType # noqa: PLC0415 + from ltx_core.model.transformer.rope import LTXRopeType # noqa: PLC0415 + + model = LTXModel( + model_type=LTXModelType.AudioVideo, + num_attention_heads=ARCH["num_attention_heads"], + attention_head_dim=ARCH["attention_head_dim"], + in_channels=ARCH["in_channels"], + out_channels=ARCH["out_channels"], + num_layers=ARCH["num_layers"], + cross_attention_dim=ARCH["cross_attention_dim"], + norm_eps=ARCH["norm_eps"], + positional_embedding_theta=ARCH["positional_embedding_theta"], + positional_embedding_max_pos=list(ARCH["positional_embedding_max_pos"]), + timestep_scale_multiplier=ARCH["timestep_scale_multiplier"], + use_middle_indices_grid=ARCH["use_middle_indices_grid"], + audio_num_attention_heads=ARCH["audio_num_attention_heads"], + audio_attention_head_dim=ARCH["audio_attention_head_dim"], + audio_in_channels=ARCH["audio_in_channels"], + audio_out_channels=ARCH["audio_out_channels"], + audio_cross_attention_dim=ARCH["audio_cross_attention_dim"], + audio_positional_embedding_max_pos=list(ARCH["audio_positional_embedding_max_pos"]), + av_ca_timestep_scale_multiplier=ARCH["av_ca_timestep_scale_multiplier"], + rope_type=LTXRopeType(rope_type_name), + double_precision_rope=double_rope, + apply_gated_attention=ARCH["apply_gated_attention"], + cross_attention_adaln=ARCH["cross_attention_adaln"], + use_prompt_adaln_single=ARCH["use_prompt_adaln_single"], + ff_bias=ARCH["ff_bias"], + audio_ff_bias=ARCH["audio_ff_bias"], + ) + fill_parameters(model) + model.eval() + return model + + +# --------------------------------------------------------------------------- +# Emission helpers +# --------------------------------------------------------------------------- + + +def emit_header(out, argv: str, revision: str) -> None: + out.write( + "// GENERATED by scripts/gen-ltx2-goldens.py — DO NOT EDIT BY HAND.\n" + "//\n" + "// LTX-2.5 DiT parity goldens produced by IMPORTING and EXECUTING the upstream\n" + "// Lightricks LTX-2 modules (packages/ltx-core/src/ltx_core/model/transformer/)\n" + "// at reduced dimensions with the deterministic Ltx2Rand stream.\n" + f"// Upstream revision: {revision}\n" + "// Regenerate with:\n" + f"// {argv}\n" + "//\n" + "// See .agents/specs/ltx-2-5.md sections 0 and 7 for why this is the gate: the\n" + "// shipped DiT is ~19 GB and vLLM-Omni carries no native 2.5 path, so the MATH is\n" + "// gated exactly here and no weight byte is checked in.\n" + "#pragma once\n\n" + "#include \n\n" + "namespace vllm_test {\n\n" + ) + + +def _cxx_float(value: float, digits: int) -> str: + """Format as a valid C++ floating literal (`0` alone is an integer literal).""" + if not math.isfinite(value): + raise ValueError(f"refusing to emit non-finite golden value: {value}") + text = f"{value:.{digits}g}" + if "." not in text and "e" not in text and "E" not in text: + text += ".0" + return text + + +def emit_f32(out, name: str, values) -> None: + flat = np.asarray(values, dtype=np.float32).reshape(-1).tolist() + out.write(f"inline constexpr float {name}[] = {{\n") + for i in range(0, len(flat), 6): + chunk = ", ".join(_cxx_float(v, 9) + "f" for v in flat[i : i + 6]) + out.write(" " + chunk + ",\n") + out.write("};\n\n") + + +def emit_f64(out, name: str, values) -> None: + flat = np.asarray(values, dtype=np.float64).reshape(-1).tolist() + out.write(f"inline constexpr double {name}[] = {{\n") + for i in range(0, len(flat), 6): + chunk = ", ".join(_cxx_float(v, 17) for v in flat[i : i + 6]) + out.write(" " + chunk + ",\n") + out.write("};\n\n") + + +def emit_i64(out, name: str, values) -> None: + flat = [int(v) for v in np.asarray(values).reshape(-1).tolist()] + out.write(f"inline constexpr int64_t {name}[] = {{\n") + for i in range(0, len(flat), 12): + out.write(" " + ", ".join(str(v) for v in flat[i : i + 12]) + ",\n") + out.write("};\n\n") + + +def emit_scalar(out, name: str, value) -> None: + out.write(f"inline constexpr int64_t {name} = {int(value)};\n") + + +def tensor(t: torch.Tensor) -> np.ndarray: + return t.detach().to(torch.float32).contiguous().numpy() + + +# --------------------------------------------------------------------------- +# Golden sections +# --------------------------------------------------------------------------- + + +def emit_arch(out) -> None: + out.write("// --- section 0: the reduced architecture, mirrored by the C++ suite ---\n") + for key in ( + "num_attention_heads", + "attention_head_dim", + "in_channels", + "out_channels", + "num_layers", + "cross_attention_dim", + "timestep_scale_multiplier", + "audio_num_attention_heads", + "audio_attention_head_dim", + "audio_in_channels", + "audio_out_channels", + "audio_cross_attention_dim", + "av_ca_timestep_scale_multiplier", + ): + emit_scalar(out, f"kLtx2Arch_{key}", ARCH[key]) + emit_scalar(out, "kLtx2Batch", BATCH) + emit_scalar(out, "kLtx2VideoTokens", VIDEO_TOKENS) + emit_scalar(out, "kLtx2AudioTokens", AUDIO_TOKENS) + emit_scalar(out, "kLtx2VideoContext", VIDEO_CONTEXT) + emit_scalar(out, "kLtx2AudioContext", AUDIO_CONTEXT) + out.write("\n") + emit_f64(out, "kLtx2VideoPositions", video_positions().to(torch.float64)) + emit_f64(out, "kLtx2AudioPositions", audio_positions().to(torch.float64)) + + +def emit_manifest(out, model) -> None: + """The upstream parameter LIST is the layout contract; gate it verbatim.""" + out.write("// --- section 1: upstream named_parameters() — the weight contract ---\n") + names = [] + ranks = [] + dims = [] + for name, param in model.named_parameters(): + names.append(name) + ranks.append(len(param.shape)) + dims.extend(int(d) for d in param.shape) + out.write(f"inline constexpr const char* kLtx2ParamNames[] = {{\n") + for name in names: + out.write(f' "{name}",\n') + out.write("};\n\n") + emit_i64(out, "kLtx2ParamRanks", ranks) + emit_i64(out, "kLtx2ParamDims", dims) + emit_scalar(out, "kLtx2ParamCount", len(names)) + out.write("\n") + + +def emit_rope(out, model) -> None: + """precompute_freqs_cis, both flavours and both frequency precisions.""" + from ltx_core.model.transformer.rope import ( # noqa: PLC0415 + LTXRopeType, + generate_freq_grid_np, + generate_freq_grid_pytorch, + precompute_freqs_cis, + ) + + out.write("// --- section 2: rope.py — the frequency ladder and both cos/sin layouts ---\n") + # The ladder itself: theta ** linspace(0, 1, n) * pi/2, in f32 and in f64. + emit_f32(out, "kLtx2FreqGridVideoF32", generate_freq_grid_pytorch(10000.0, 3, video_dim())) + emit_f32(out, "kLtx2FreqGridVideoF64", generate_freq_grid_np(10000.0, 3, video_dim())) + emit_f32(out, "kLtx2FreqGridAudioF32", generate_freq_grid_pytorch(10000.0, 1, audio_dim())) + emit_f32(out, "kLtx2FreqGridAudioF64", generate_freq_grid_np(10000.0, 1, audio_dim())) + + vpos = video_positions() + apos = audio_positions() + cross_max = max( + ARCH["positional_embedding_max_pos"][0], ARCH["audio_positional_embedding_max_pos"][0] + ) + cases = ( + ("Split", LTXRopeType.SPLIT, False), + ("Interleaved", LTXRopeType.INTERLEAVED, False), + ("Double", LTXRopeType.SPLIT, True), + ) + for tag, rope_type, double in cases: + gen = generate_freq_grid_np if double else generate_freq_grid_pytorch + vcos, vsin = precompute_freqs_cis( + vpos, + dim=video_dim(), + out_dtype=torch.float32, + theta=ARCH["positional_embedding_theta"], + max_pos=list(ARCH["positional_embedding_max_pos"]), + use_middle_indices_grid=True, + num_attention_heads=ARCH["num_attention_heads"], + rope_type=rope_type, + freq_grid_generator=gen, + ) + acos, asin = precompute_freqs_cis( + apos, + dim=audio_dim(), + out_dtype=torch.float32, + theta=ARCH["positional_embedding_theta"], + max_pos=list(ARCH["audio_positional_embedding_max_pos"]), + use_middle_indices_grid=True, + num_attention_heads=ARCH["audio_num_attention_heads"], + rope_type=rope_type, + freq_grid_generator=gen, + ) + # transformer_args.py:364-371 — the audio<->video cross RoPE: the TIME + # axis only, built at audio_cross_attention_dim. + ccos, csin = precompute_freqs_cis( + vpos[:, 0:1, :], + dim=ARCH["audio_cross_attention_dim"], + out_dtype=torch.float32, + theta=ARCH["positional_embedding_theta"], + max_pos=[cross_max], + use_middle_indices_grid=True, + num_attention_heads=ARCH["num_attention_heads"], + rope_type=rope_type, + freq_grid_generator=gen, + ) + emit_f32(out, f"kLtx2Rope{tag}VideoCos", tensor(vcos)) + emit_f32(out, f"kLtx2Rope{tag}VideoSin", tensor(vsin)) + emit_f32(out, f"kLtx2Rope{tag}AudioCos", tensor(acos)) + emit_f32(out, f"kLtx2Rope{tag}AudioSin", tensor(asin)) + emit_f32(out, f"kLtx2Rope{tag}CrossCos", tensor(ccos)) + emit_f32(out, f"kLtx2Rope{tag}CrossSin", tensor(csin)) + + +def emit_bricks(out, model) -> None: + """The leaf modules, each run standalone so a failure localizes.""" + out.write("// --- section 3: the leaf bricks (adaln, gated attention, ff) ---\n") + dim = video_dim() + adim = audio_dim() + + # AdaLayerNormSingle (adaln.py:39-45). The timesteps arrive ALREADY scaled by + # timestep_scale_multiplier, exactly as _prepare_timestep passes them. + ts = rand_input("brick.adaln.timesteps", (BATCH * VIDEO_TOKENS,), 0.25, 0.5) * float( + ARCH["timestep_scale_multiplier"] + ) + modulation, embedded = model.adaln_single(ts.flatten(), hidden_dtype=torch.float32) + emit_f32(out, "kLtx2AdalnTimesteps", tensor(ts)) + emit_f32(out, "kLtx2AdalnModulation", tensor(modulation)) + emit_f32(out, "kLtx2AdalnEmbedded", tensor(embedded)) + + # Attention with PER-HEAD GATING (attention.py:576-579). Self-attention shape, + # no RoPE and no mask, so this brick isolates the gate and its ORDER. + x = rand_input("brick.attn.x", (BATCH, VIDEO_TOKENS, dim), 0.5) + attn_out = model.transformer_blocks[0].attn1(x) + emit_f32(out, "kLtx2AttnGatedInput", tensor(x)) + emit_f32(out, "kLtx2AttnGatedOutput", tensor(attn_out)) + + # The ASYMMETRIC cross-modal pair: Q from the video stream, K/V from the audio + # stream, distinct token counts, both cross RoPE tables applied. + from ltx_core.model.transformer.rope import LTXRopeType, precompute_freqs_cis # noqa: PLC0415 + + cross_max = max( + ARCH["positional_embedding_max_pos"][0], ARCH["audio_positional_embedding_max_pos"][0] + ) + vpe = precompute_freqs_cis( + video_positions()[:, 0:1, :], + dim=ARCH["audio_cross_attention_dim"], + out_dtype=torch.float32, + theta=ARCH["positional_embedding_theta"], + max_pos=[cross_max], + use_middle_indices_grid=True, + num_attention_heads=ARCH["num_attention_heads"], + rope_type=LTXRopeType.SPLIT, + ) + ape = precompute_freqs_cis( + audio_positions()[:, 0:1, :], + dim=ARCH["audio_cross_attention_dim"], + out_dtype=torch.float32, + theta=ARCH["positional_embedding_theta"], + max_pos=[cross_max], + use_middle_indices_grid=True, + num_attention_heads=ARCH["audio_num_attention_heads"], + rope_type=LTXRopeType.SPLIT, + ) + a2v_q = rand_input("brick.a2v.q", (BATCH, VIDEO_TOKENS, dim), 0.5) + a2v_kv = rand_input("brick.a2v.kv", (BATCH, AUDIO_TOKENS, adim), 0.5) + a2v_out = model.transformer_blocks[0].audio_to_video_attn( + a2v_q, context=a2v_kv, pe=vpe, k_pe=ape + ) + emit_f32(out, "kLtx2A2vQuery", tensor(a2v_q)) + emit_f32(out, "kLtx2A2vContext", tensor(a2v_kv)) + emit_f32(out, "kLtx2A2vOutput", tensor(a2v_out)) + + # The FFN BIAS ASYMMETRY: `ff` has none, `audio_ff` has one. + ffx = rand_input("brick.ff.x", (BATCH, VIDEO_TOKENS, dim), 0.5) + emit_f32(out, "kLtx2FfInput", tensor(ffx)) + emit_f32(out, "kLtx2FfOutput", tensor(model.transformer_blocks[0].ff(ffx))) + affx = rand_input("brick.audio_ff.x", (BATCH, AUDIO_TOKENS, adim), 0.5) + emit_f32(out, "kLtx2AudioFfInput", tensor(affx)) + emit_f32(out, "kLtx2AudioFfOutput", tensor(model.transformer_blocks[0].audio_ff(affx))) + + +def emit_forward( + out, tag: str, rope_type_name: str, double_rope: bool, masked: bool, + audio_enabled: bool = True, dense_self_mask: bool = False, +) -> None: + out.write( + f"// --- forward case {tag}: rope={rope_type_name} float64_freqs={double_rope} " + f"masked={masked} audio_enabled={audio_enabled} dense_self_mask={dense_self_mask} ---\n" + ) + model = build_model(rope_type_name, double_rope) + video, audio = build_modalities(masked, audio_enabled, dense_self_mask) + with torch.no_grad(): + vx, ax = model(video=video, audio=audio, perturbations=None) + emit_f32(out, f"kLtx2Forward{tag}Video", tensor(vx)) + emit_f32(out, f"kLtx2Forward{tag}Audio", tensor(ax)) + return model + + +def emit_masks(out) -> None: + out.write("// --- section 5: the prompt and self-attention masks the masked case runs ---\n") + emit_i64(out, "kLtx2VideoContextMask", video_context_mask()) + emit_i64(out, "kLtx2AudioContextMask", audio_context_mask()) + emit_f32(out, "kLtx2VideoSelfMask", tensor(video_self_mask())) + emit_f32(out, "kLtx2AudioSelfMask", tensor(audio_self_mask())) + emit_f32(out, "kLtx2VideoSelfMaskDense", tensor(video_self_mask_dense())) + emit_f32(out, "kLtx2AudioSelfMaskDense", tensor(audio_self_mask_dense())) + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--ltx2", required=True, type=Path, help="path to a Lightricks/LTX-2 checkout") + parser.add_argument("--out", required=True, type=Path) + args = parser.parse_args() + + root = args.ltx2.expanduser() + load_upstream(root) + revision = upstream_revision(root) + + torch.set_grad_enabled(False) + argv = "python3 " + " ".join( + [os.path.relpath(sys.argv[0]), f"--ltx2 {root}", f"--out {args.out}"] + ) + + args.out.parent.mkdir(parents=True, exist_ok=True) + with args.out.open("w", encoding="utf-8") as out: + emit_header(out, argv, revision) + emit_arch(out) + reference = build_model("split", False) + emit_manifest(out, reference) + emit_rope(out, reference) + emit_bricks(out, reference) + emit_masks(out) + out.write("// --- section 4: the full DiT forward ---\n") + emit_forward(out, "Split", "split", False, False) + emit_forward(out, "Interleaved", "interleaved", False, False) + emit_forward(out, "Double", "split", True, False) + emit_forward(out, "Masked", "split", False, True) + # Modality.enabled=False on the audio stream: the audio blocks are skipped + # entirely (transformer.py:266) while the audio->video cross attention still + # reads the audio state (:268), and the audio output head still runs over the + # untouched patchified latent (model.py:527-536). + emit_forward(out, "AudioOff", "split", False, False, audio_enabled=False) + # The DENSE (B, T, T) self-attention mask (transformer_args.py:212-215). + # Every query carries its OWN row of key strengths, so this case — and + # only this case — separates a kernel that indexes the bias by query from + # one that reads bias row 0 for every query. + emit_forward(out, "DenseMask", "split", False, True, dense_self_mask=True) + out.write("} // namespace vllm_test\n") + print(f"wrote {args.out}", file=sys.stderr) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/gen-ltx2-pipeline-goldens.py b/scripts/gen-ltx2-pipeline-goldens.py new file mode 100644 index 000000000..7fa7029f8 --- /dev/null +++ b/scripts/gen-ltx2-pipeline-goldens.py @@ -0,0 +1,1444 @@ +#!/usr/bin/env python3 +"""Emit tests/vllm/models/ltx2_pipeline_goldens.inc — the LTX-2.5 PIPELINE oracle. + +Phase L5 of .agents/specs/ltx-2-5.md (issue #435): the flow-matching schedule, +the noiser, the diffusion steps, guidance and the guiders, the patchifiers, the +distilled two-stage recipe, the latent spatial x2 upsampler, the duration head +and the `Embeddings1DConnector`. + +Everything below is produced by EXECUTING the upstream modules at REDUCED +dimensions on CPU, with both sides rebuilding weights and inputs from one +deterministic FNV-1a + splitmix64 stream keyed by parameter NAME, so no weight +byte is checked in. This is the method that made L2 and L4 trustworthy +(scripts/gen-ltx2-goldens.py, scripts/gen-ltx2-vae-goldens.py). + +Upstream sources: + + A. Lightricks/LTX-2, packages/ltx-core/src/ltx_core/ (EXECUTED) + components/schedulers.py -> section 1 + components/noisers.py -> section 2 + components/diffusion_steps.py -> section 3 + components/guiders.py -> section 4 + guidance/perturbations.py -> section 5 + components/patchifiers.py -> section 6 + model/upsampler/*.py -> section 8 + duration_head/duration_head.py -> section 9 + text_encoders/gemma/embeddings_connector.py -> section 10 + + B. Lightricks/LTX-2, packages/ltx-pipelines/src/ltx_pipelines/ (READ) + utils/constants.py -> section 7, EXECUTED by path (its package __init__ + pulls in `av`, which is not a dependency of the math) + distilled.py -> section 7, read with `ast` (its imports reach the + whole pipeline stack; the constants are literals) + + C. vLLM-Omni, vllm_omni/diffusion/models/ltx2/ltx2_recipes.py (READ with `ast`) + -> section 7. This is the BINDING oracle's serving model, and it carries + NO 2.5 row — see the spec section 3. Its table is emitted so the C++ + refusal is gated against the real key set rather than against a + remembered one, and so a 2.5 row appearing upstream shows up as a + golden change instead of going unnoticed. + +Usage: + python3 scripts/gen-ltx2-pipeline-goldens.py \\ + --ltx2 ~/_git/LTX-2 \\ + --vllm-omni ~/_git/vllm-omni \\ + --out tests/vllm/models/ltx2_pipeline_goldens.inc + +Needs torch + numpy + einops (CPU only). No checkpoint, venv, or gated download. + +WHY THIS SCRIPT REFUSES THREE THINGS. Each refusal is a finding this campaign +paid for, not a defensive habit: + + 1. ORACLE IDENTITY. The resolved `ltx_core.__file__` must live under `--ltx2`. + L4 proved a decoy `ltx_core` on `sys.path` produces goldens with an IDENTICAL + md5 (spec section 7.0(b)), so `sys.path.insert` winning is not something to + assume. + 2. PROVENANCE. `git -C rev-parse HEAD` is emitted for BOTH upstreams and + asserted against a SHA the C++ suite pins. + 3. A DIRTY TREE IS REFUSED. A revision anchor read from a tree with uncommitted + edits stamps a clean SHA on goldens the SHA does not describe — the anchor + then actively misleads a bisect. `git status --porcelain` must be empty for + both upstreams. +""" + +from __future__ import annotations + +import argparse +import ast +import importlib.util +import math +import subprocess +import sys +from pathlib import Path + +import numpy as np + +_MASK64 = (1 << 64) - 1 + + +# --------------------------------------------------------------------------- +# Deterministic weight/input stream, mirrored bit-for-bit by the C++ suite +# (tests/vllm/models/test_ltx2_pipeline.cpp :: Ltx2Rand). A per-tensor FNV-1a seed +# plus a splitmix64 counter makes every tensor independent of fill ORDER, so the +# two sides cannot silently drift by reordering their parameter construction. +# Identical to the L2 and L4 generators'. +# --------------------------------------------------------------------------- + + +def fnv1a64(name: str) -> int: + h = 0xCBF29CE484222325 + for byte in name.encode("utf-8"): + h ^= byte + h = (h * 0x100000001B3) & _MASK64 + return h + + +def splitmix64(x: int) -> int: + x = (x + 0x9E3779B97F4A7C15) & _MASK64 + z = x + z = ((z ^ (z >> 30)) * 0xBF58476D1CE4E5B9) & _MASK64 + z = ((z ^ (z >> 27)) * 0x94D049BB133111EB) & _MASK64 + return z ^ (z >> 31) + + +def ltx_rand(name: str, count: int) -> np.ndarray: + """`count` values uniform in [-1, 1), reproducible from `name` alone.""" + seed = fnv1a64(name) + out = np.empty(count, dtype=np.float64) + for i in range(count): + u = splitmix64((seed + i) & _MASK64) + out[i] = ((u >> 11) * (2.0**-53)) * 2.0 - 1.0 + return out + + +def make(name: str, count: int, scale: float = 1.0, offset: float = 0.0) -> np.ndarray: + return (ltx_rand(name, count) * scale + offset).astype(np.float32) + + +# --------------------------------------------------------------------------- +# The per-parameter role rule. The C++ side implements the SAME rule keyed on the +# same names, so a divergence shows up as a golden mismatch rather than as a +# silently different tensor. +# --------------------------------------------------------------------------- + + +def param_values(name: str, shape) -> np.ndarray: + count = int(np.prod(shape)) if len(shape) else 1 + rank = len(shape) + if name.endswith(".bias") or name.endswith("in_proj_bias"): + return make(name, count, 0.02) + if rank == 1 and name.endswith(".weight"): + # A 1-D `.weight` is an affine norm gain (GroupNorm, RMSNorm), whose + # trained value sits around 1.0. Centring it on 0.0 would make every + # normalized activation ~0 and hide a scale error. + return make(name, count, 0.1, 1.0) + if ".dur." in name: + # The DURATION HEAD needs a wider fixture than everything else, and this + # is a gate property rather than a taste. Its output is + # `exp(mlp_out(...))` through a chain that ATTENUATES: at the shared 0.05 + # scale every projection shrinks its input, `log_duration` lands within + # ~0.007 of `mlp_out.bias`, and the both / video-only / audio-only arms + # collapse to within 3e-6 of each other -- BELOW this suite's round-off + # bound. A gate that cannot separate its three arms would accept an + # implementation that ignored one of the two streams entirely. MEASURED + # spreads at this fixture: 0.05 -> 2.98e-06, 0.2 -> 2.3e-03, + # 0.35 -> 4.9e-02, which is ~10^4 x the bound while keeping the predicted + # duration in a sane 0.9-1.0 second range. + return make(name, count, 0.35) + return make(name, count, 0.05) + + +def fill_module(module, prefix: str) -> list[tuple[str, int]]: + """Fill every parameter from the shared stream; return the state_dict manifest. + + Iterates `named_parameters()`, which is the ORDER the C++ side asserts. The + fill is by NAME, so the order only decides the manifest, never the values. + """ + import torch # noqa: PLC0415 + + manifest: list[tuple[str, int]] = [] + for name, param in module.named_parameters(): + values = param_values(prefix + name, tuple(param.shape)) + # `.copy_` rather than assignment: it ROUNDS into the parameter's own + # dtype. Embeddings1DConnector.learnable_registers is bfloat16 by + # construction (embeddings_connector.py:135-137), and a port that keeps + # those values in f32 is WIDER than upstream — the exact polarity + # AGENTS.md warns a value gate cannot catch, so it is baked into the + # oracle here and mirrored on the C++ side. + param.data.copy_(torch.from_numpy(values).reshape(param.shape)) + manifest.append((prefix + name, int(values.size))) + return manifest + + +# --------------------------------------------------------------------------- +# Emission +# --------------------------------------------------------------------------- + + +def _cxx_float(value: float, digits: int) -> str: + if not math.isfinite(value): + raise ValueError(f"refusing to emit non-finite golden value: {value}") + text = f"{value:.{digits}g}" + if "." not in text and "e" not in text and "E" not in text: + text += ".0" + return text + + +def emit_f32(out, name: str, values) -> None: + flat = np.asarray(values, dtype=np.float32).reshape(-1).tolist() + out.write(f"inline constexpr float {name}[] = {{\n") + for i in range(0, len(flat), 6): + chunk = ", ".join(_cxx_float(v, 9) + "f" for v in flat[i : i + 6]) + out.write(" " + chunk + ",\n") + out.write("};\n\n") + + +def emit_f64(out, name: str, values) -> None: + flat = np.asarray(values, dtype=np.float64).reshape(-1).tolist() + out.write(f"inline constexpr double {name}[] = {{\n") + for i in range(0, len(flat), 4): + chunk = ", ".join(_cxx_float(v, 17) for v in flat[i : i + 4]) + out.write(" " + chunk + ",\n") + out.write("};\n\n") + + +def emit_i64(out, name: str, values) -> None: + flat = [int(v) for v in np.asarray(values).reshape(-1).tolist()] + out.write(f"inline constexpr int64_t {name}[] = {{\n") + for i in range(0, len(flat), 12): + out.write(" " + ", ".join(str(v) for v in flat[i : i + 12]) + ",\n") + out.write("};\n\n") + + +def emit_scalar(out, name: str, value) -> None: + out.write(f"inline constexpr int64_t {name} = {int(value)};\n") + + +def emit_double(out, name: str, value: float) -> None: + out.write(f"inline constexpr double {name} = {_cxx_float(float(value), 17)};\n") + + +def emit_bool(out, name: str, value: bool) -> None: + out.write(f"inline constexpr bool {name} = {'true' if value else 'false'};\n") + + +def emit_string(out, name: str, value: str) -> None: + escaped = value.replace("\\", "\\\\").replace('"', '\\"') + out.write(f'inline constexpr const char* {name} = "{escaped}";\n') + + +def emit_manifest(out, name: str, manifest: list[tuple[str, int]]) -> None: + out.write(f"inline constexpr const char* {name}Names[] = {{\n") + for key, _ in manifest: + out.write(f' "{key}",\n') + out.write("};\n") + out.write(f"inline constexpr int64_t {name}Counts[] = {{\n") + for i in range(0, len(manifest), 10): + out.write(" " + ", ".join(str(c) for _, c in manifest[i : i + 10]) + ",\n") + out.write("};\n\n") + + +def section(out, title: str) -> None: + out.write(f"// {'-' * 74}\n// {title}\n// {'-' * 74}\n\n") + + +# --------------------------------------------------------------------------- +# Section 1 — schedulers (components/schedulers.py) +# --------------------------------------------------------------------------- + +# Each case is (tag, steps, tokens-or-None, max_shift, base_shift, stretch, terminal). +# `tokens` None exercises `default_number_of_tokens = MAX_SHIFT_ANCHOR`; a value +# exercises the `math.prod(latent.shape[2:])` path, which is what makes the shift +# resolution-dependent (schedulers.py:32). +_SCHED_CASES = ( + ("Default", 8, None, 2.05, 0.95, True, 0.1), + ("NoStretch", 8, None, 2.05, 0.95, False, 0.1), + # tokens BELOW the base anchor: sigma_shift goes negative, which is the arm a + # small render actually takes and the one where the linear fit extrapolates. + ("FewTokens", 6, 256, 2.05, 0.95, True, 0.1), + ("ManyTokens", 6, 16384, 2.05, 0.95, True, 0.1), + ("Terminal0", 5, 1024, 2.05, 0.95, True, 0.0), + ("OneStep", 1, 4096, 2.05, 0.95, True, 0.1), + # STEP COUNTS WHERE A NAIVE FORWARD linspace WALK MISSES EXACT 0. torch walks + # the second half BACKWARDS from `end` (aten RangeFactories.cpp), which is the + # only reason the terminal sigma is exactly 0. `start + step * i` instead leaves + # 5.96e-08 there, which survives the `sigmas != 0` guard at schedulers.py:42, + # takes the shift transform, and then becomes `one_minus_z[-1]` in the stretch + # branch (:52) -- so the WHOLE schedule moves and the denoise loop never reaches + # zero noise. `--steps 41` is a plain user-reachable render, not a corner. + # 24 of the first 198 counts are affected; these are the first two above 1. + ("Steps41", 41, None, 2.05, 0.95, True, 0.1), + ("Steps47", 47, None, 2.05, 0.95, True, 0.1), +) + +# (tag, steps, threshold_noise, linear_steps-or-None) +_LINQUAD_CASES = ( + ("Default", 6, 0.025, None), + ("OneStep", 1, 0.025, None), + ("Explicit", 7, 0.05, 2), + # linear_steps == steps leaves quadratic_steps == 0, the branch that skips the + # quadratic tail entirely (schedulers.py:79). + ("AllLinear", 4, 0.025, 4), +) + + +def section_schedulers(out) -> None: + import torch # noqa: PLC0415 + from ltx_core.components.schedulers import ( # noqa: PLC0415 + BASE_SHIFT_ANCHOR, + MAX_SHIFT_ANCHOR, + LinearQuadraticScheduler, + LTX2Scheduler, + ) + + section(out, "Section 1 - sigma schedules (components/schedulers.py)") + # The two anchors are module constants (schedulers.py:10-11), not arguments: + # they set the token axis the shift is fitted on, so they belong to the gate. + emit_scalar(out, "kLtx2SchedBaseShiftAnchor", BASE_SHIFT_ANCHOR) + emit_scalar(out, "kLtx2SchedMaxShiftAnchor", MAX_SHIFT_ANCHOR) + out.write("\n") + + scheduler = LTX2Scheduler() + for tag, steps, tokens, max_shift, base_shift, stretch, terminal in _SCHED_CASES: + latent = None + if tokens is not None: + # `math.prod(latent.shape[2:])` — the only thing read off the latent. + latent = torch.zeros(1, 1, tokens) + sigmas = scheduler.execute( + steps=steps, + latent=latent, + max_shift=max_shift, + base_shift=base_shift, + stretch=stretch, + terminal=terminal, + ) + assert sigmas.dtype == torch.float32 + emit_scalar(out, f"kLtx2Sched{tag}Steps", steps) + emit_scalar(out, f"kLtx2Sched{tag}Tokens", tokens if tokens is not None else 0) + emit_double(out, f"kLtx2Sched{tag}MaxShift", max_shift) + emit_double(out, f"kLtx2Sched{tag}BaseShift", base_shift) + emit_bool(out, f"kLtx2Sched{tag}Stretch", stretch) + emit_double(out, f"kLtx2Sched{tag}Terminal", terminal) + emit_f32(out, f"kLtx2Sched{tag}Golden", sigmas.numpy()) + + linear = LinearQuadraticScheduler() + for tag, steps, threshold, linear_steps in _LINQUAD_CASES: + kwargs = {"steps": steps, "threshold_noise": threshold} + if linear_steps is not None: + kwargs["linear_steps"] = linear_steps + sigmas = linear.execute(**kwargs) + emit_scalar(out, f"kLtx2LinQuad{tag}Steps", steps) + emit_double(out, f"kLtx2LinQuad{tag}Threshold", threshold) + emit_scalar(out, f"kLtx2LinQuad{tag}LinearSteps", -1 if linear_steps is None else linear_steps) + emit_f32(out, f"kLtx2LinQuad{tag}Golden", sigmas.numpy()) + + +# --------------------------------------------------------------------------- +# Section 2 — the noiser (components/noisers.py) +# --------------------------------------------------------------------------- + +_NOISE_COUNT = 24 + + +def section_noiser(out) -> None: + import torch # noqa: PLC0415 + from ltx_core.components.noisers import GaussianNoiser # noqa: PLC0415 + from ltx_core.types import LatentState # noqa: PLC0415 + + section(out, "Section 2 - GaussianNoiser (components/noisers.py:30-37)") + + latent = torch.from_numpy(make("ltx2.noiser.latent", _NOISE_COUNT, 1.0)) + clean = torch.from_numpy(make("ltx2.noiser.clean", _NOISE_COUNT, 1.0)) + noise = torch.from_numpy(make("ltx2.noiser.noise", _NOISE_COUNT, 1.0)) + # The denoise mask is upstream's per-element blend weight between the CLEAN + # latent (conditioning) and the noised one. 0 and 1 both appear in a real + # request (a conditioned frame is fully clean), so both endpoints are here. + mask_raw = make("ltx2.noiser.mask", _NOISE_COUNT, 0.5, 0.5) + mask_raw[0] = 0.0 + mask_raw[1] = 1.0 + mask = torch.from_numpy(mask_raw) + + emit_scalar(out, "kLtx2NoiserCount", _NOISE_COUNT) + emit_f32(out, "kLtx2NoiserMask", mask.numpy()) + + class _FixedNoiser(GaussianNoiser): + """The one harness adaptation, and it changes no arithmetic. + + Upstream draws `torch.randn(..., generator=self.generator)`; the C++ side + consumes an Ltx2NoiseStream. Substituting a FIXED draw keeps the lerp + chain — which is the whole of the module's math — byte-comparable without + also porting torch's Philox. + """ + + def _sample_noise(self, latent_state): # noqa: ANN001, ANN201 + return noise + + noiser = _FixedNoiser(generator=None) + for tag, scale in (("Full", 1.0), ("Half", 0.5), ("Zero", 0.0)): + state = LatentState( + latent=latent.clone(), + clean_latent=clean.clone(), + denoise_mask=mask, + # `positions` is required by the dataclass and never read by the + # noiser (noisers.py:30-37 touches latent / clean_latent / mask only). + positions=torch.zeros(_NOISE_COUNT), + ) + result = noiser(state, noise_scale=scale) + emit_double(out, f"kLtx2Noiser{tag}Scale", scale) + emit_f32(out, f"kLtx2Noiser{tag}Golden", result.latent.numpy()) + + +# --------------------------------------------------------------------------- +# Section 3 — diffusion steps (components/diffusion_steps.py) +# --------------------------------------------------------------------------- + +_STEP_COUNT = 20 +# A schedule with a mid-range sigma, a near-terminal one and the terminal 0, so +# the `sigma_next == 0` early-outs are exercised rather than described. +_STEP_SIGMAS = (1.0, 0.725, 0.421875, 0.0) + + +def section_diffusion_steps(out) -> None: + import torch # noqa: PLC0415 + from ltx_core.components.diffusion_steps import ( # noqa: PLC0415 + EulerAncestralDiffusionStep, + EulerCfgPpDiffusionStep, + EulerDiffusionStep, + Res2sDiffusionStep, + _get_ancestral_step, + ) + + section(out, "Section 3 - diffusion steps (components/diffusion_steps.py)") + + sample = torch.from_numpy(make("ltx2.step.sample", _STEP_COUNT, 1.0)) + denoised = torch.from_numpy(make("ltx2.step.denoised", _STEP_COUNT, 0.8)) + uncond = torch.from_numpy(make("ltx2.step.uncond", _STEP_COUNT, 0.8)) + noise = torch.from_numpy(make("ltx2.step.noise", _STEP_COUNT, 1.0)) + sigmas = torch.tensor(_STEP_SIGMAS, dtype=torch.float32) + + emit_scalar(out, "kLtx2StepCount", _STEP_COUNT) + emit_f32(out, "kLtx2StepSigmas", sigmas.numpy()) + emit_scalar(out, "kLtx2StepSigmaCount", len(_STEP_SIGMAS)) + out.write("\n") + + euler = EulerDiffusionStep() + for index in (0, 1): + emit_f32( + out, + f"kLtx2StepEuler{index}Golden", + euler.step(sample, denoised, sigmas, index).numpy(), + ) + + for tag, eta, s_noise in (("Eta1", 1.0, 1.0), ("Eta0", 0.0, 1.0), ("EtaHalf", 0.5, 0.75)): + stepper = EulerAncestralDiffusionStep(eta=eta, s_noise=s_noise) + emit_double(out, f"kLtx2StepAncestral{tag}Eta", eta) + emit_double(out, f"kLtx2StepAncestral{tag}SNoise", s_noise) + for index in (0, 1): + emit_f32( + out, + f"kLtx2StepAncestral{tag}Step{index}Golden", + stepper.step(sample, denoised, sigmas, index, noise=noise).numpy(), + ) + # step_index 2 has sigma_next == 0: upstream returns the denoised + # prediction outright (diffusion_steps.py:85-86). A port that instead + # divided by sigma_next would produce inf, so the branch is gated. + emit_f32( + out, + f"kLtx2StepAncestral{tag}TerminalGolden", + stepper.step(sample, denoised, sigmas, 2, noise=noise).numpy(), + ) + + res2s = Res2sDiffusionStep() + for tag, eta in (("EtaHalf", 0.5), ("Eta1", 1.0)): + emit_double(out, f"kLtx2StepRes2s{tag}Eta", eta) + for index in (0, 1): + emit_f32( + out, + f"kLtx2StepRes2s{tag}Step{index}Golden", + res2s.step(sample, denoised, sigmas, index, noise, eta=eta).numpy(), + ) + emit_f32( + out, + f"kLtx2StepRes2s{tag}TerminalGolden", + res2s.step(sample, denoised, sigmas, 2, noise, eta=eta).numpy(), + ) + + for tag, eta, s_noise in (("Eta1", 1.0, 1.0), ("Eta0", 0.0, 1.0)): + cfgpp = EulerCfgPpDiffusionStep(eta=eta, s_noise=s_noise) + emit_double(out, f"kLtx2StepCfgPp{tag}Eta", eta) + emit_double(out, f"kLtx2StepCfgPp{tag}SNoise", s_noise) + for index in (0, 1, 2): + # index 0 has sigma_s == 1.0 EXACTLY, so `alpha_s = (1 - sigma_s)` + # is 0 and the finfo(float32).eps clamp (diffusion_steps.py:233-235) + # is what keeps `d = (x - alpha_s * uncond) / sigma_s` finite. That + # makes it the ONE arm in this file where a member of the + # invisible-constant class actually decides the numbers. + emit_f32( + out, + f"kLtx2StepCfgPp{tag}Step{index}Golden", + cfgpp.step(sample, denoised, sigmas, index, uncond_denoised=uncond, + noise=noise).numpy(), + ) + + # _get_ancestral_step (diffusion_steps.py:7-22), the DDIM helper CFG++ uses. + pairs = ((1.0, 0.725), (0.725, 0.421875), (0.421875, 0.0)) + downs, ups = [], [] + for eta in (1.0, 0.5, 0.0): + for sigma_from, sigma_to in pairs: + down, up = _get_ancestral_step( + torch.tensor(sigma_from), torch.tensor(sigma_to), eta=eta + ) + downs.append(float(down)) + ups.append(float(up)) + emit_f64(out, "kLtx2AncestralHelperEtas", [1.0, 0.5, 0.0]) + emit_f64(out, "kLtx2AncestralHelperPairs", [v for pair in pairs for v in pair]) + emit_f32(out, "kLtx2AncestralHelperDown", downs) + emit_f32(out, "kLtx2AncestralHelperUp", ups) + + +# --------------------------------------------------------------------------- +# Section 4 — guiders (components/guiders.py) +# --------------------------------------------------------------------------- + +# rank-4 on purpose: LtxAPGGuider reduces over dim=[-1,-2,-3] and projection_coef +# flattens per BATCH row, so a rank-2 fixture would make the two agree by +# accident (guiders.py:114, 364-369). +_GUIDE_SHAPE = (2, 3, 4, 5) + + +def section_guiders(out) -> None: + import torch # noqa: PLC0415 + from ltx_core.components.guiders import ( # noqa: PLC0415 + CFGGuider, + CFGStarRescalingGuider, + LegacyStatefulAPGGuider, + LtxAPGGuider, + MultiModalGuider, + MultiModalGuiderFactory, + MultiModalGuiderParams, + STGGuider, + projection_coef, + ) + + section(out, "Section 4 - guiders (components/guiders.py)") + + count = int(np.prod(_GUIDE_SHAPE)) + cond = torch.from_numpy(make("ltx2.guide.cond", count, 1.0)).reshape(_GUIDE_SHAPE) + uncond = torch.from_numpy(make("ltx2.guide.uncond", count, 0.9)).reshape(_GUIDE_SHAPE) + perturbed = torch.from_numpy(make("ltx2.guide.perturbed", count, 0.7)).reshape(_GUIDE_SHAPE) + modality = torch.from_numpy(make("ltx2.guide.modality", count, 0.6)).reshape(_GUIDE_SHAPE) + + emit_i64(out, "kLtx2GuideShape", _GUIDE_SHAPE) + emit_scalar(out, "kLtx2GuideCount", count) + out.write("\n") + + emit_f32(out, "kLtx2GuideProjCoefGolden", projection_coef(cond, uncond).numpy()) + # The 1e-8 in `squared_norm` (guiders.py:368) is a member of the + # invisible-constant class: with an O(1) denominator it is ~1e-8 relative and + # no golden can see it. This arm drives `project_onto` to EXACTLY zero, where + # the constant alone decides the result (0 / 1e-8 = 0 instead of 0/0 = NaN), + # so a mutation of it moves the number rather than being absorbed. + zeros = torch.zeros(_GUIDE_SHAPE) + emit_f32(out, "kLtx2GuideProjCoefZeroGolden", projection_coef(cond, zeros).numpy()) + # ...and one where the denominator is small but NOT zero, so the epsilon is an + # additive term against a comparable magnitude. + tiny = torch.full(_GUIDE_SHAPE, 1e-5) + emit_f32(out, "kLtx2GuideProjCoefTinyGolden", projection_coef(cond, tiny).numpy()) + emit_double(out, "kLtx2GuideProjCoefTinyValue", 1e-5) + out.write("\n") + + for tag, scale in (("Scale3", 3.0), ("Scale1", 1.0)): + cfg = CFGGuider(scale=scale) + emit_double(out, f"kLtx2GuideCfg{tag}Scale", scale) + emit_bool(out, f"kLtx2GuideCfg{tag}Enabled", cfg.enabled()) + emit_f32(out, f"kLtx2GuideCfg{tag}Golden", cfg.delta(cond, uncond).numpy()) + + for tag, scale in (("Scale1", 1.0), ("Scale0", 0.0)): + stg = STGGuider(scale=scale) + emit_double(out, f"kLtx2GuideStg{tag}Scale", scale) + emit_bool(out, f"kLtx2GuideStg{tag}Enabled", stg.enabled()) + emit_f32(out, f"kLtx2GuideStg{tag}Golden", stg.delta(cond, perturbed).numpy()) + out.write("\n") + + # --- The three PROJECTION guiders, and why the port refuses them ---------- + # THE REFUSAL RESTS ON REACHABILITY, NOT ON SHAPES. An earlier revision of + # this file claimed the shape expression "raises at every rectangular rank + # >= 3, i.e. at every real (B, C, F, H, W) video latent". That premise is + # FALSE and the matrix below is what disproves it, so it is recorded here + # rather than quietly dropped (spec §7.0(b): a wrong finding frozen as a + # golden is worse than no golden). + # + # `projection_coef` returns a rank-2 `(B, 1)` tensor (guiders.py:363-369) and + # the three guiders multiply it straight into a latent (`proj_coeff * cond`, + # guiders.py:48, 118, 184). torch right-aligns, so `(B, 1)` lands on the + # latent's LAST TWO axes. The real predicate is therefore + # + # raises <=> B > 1 and shape[-2] not in {1, B} + # + # which is NOT "every video latent": at B = 1 — the ordinary single-request + # render — `(1, 1)` broadcasts as a plain scalar and the result is also + # numerically CORRECT, and `(2, 128, 8, 2, 16)` composes as well because its + # `shape[-2]` happens to equal B. Where it composes with B > 1 it is silently + # WRONG, applying the per-batch coefficient along axis -2 instead of the batch + # axis. The `norm(dim=[-1,-2,-3])` in the two threshold arms is a SEPARATE + # constraint that additionally needs rank >= 3. + # + # What actually justifies the refusal is that NOTHING UPSTREAM CONSTRUCTS + # THEM. `CFGStarRescalingGuider`, `LtxAPGGuider` and `LegacyStatefulAPGGuider` + # appear in the whole LTX-2 tree only at their own `class` statements in + # guiders.py (:31, :78, :129); every pipeline builds `MultiModalGuider` from + # `MultiModalGuiderParams` (utils/constants.py:49-68). So the arm is UNPORTED + # and refused by name and recorded as owed, per AGENTS.md. The matrix stays a + # golden so that upstream wiring one of them up shows up as a golden change. + probe_shapes = ((2, 6), (2, 3, 4), (2, 3, 4, 5), (4, 4, 4, 4), (1, 3, 4), + (1, 128, 8, 16, 16), (2, 128, 8, 2, 16)) + probe_guiders = ( + ("CFGGuider", lambda: CFGGuider(scale=3.0)), + ("STGGuider", lambda: STGGuider(scale=1.0)), + ("CFGStarRescalingGuider", lambda: CFGStarRescalingGuider(scale=3.0)), + ("LtxAPGGuiderNoThreshold", lambda: LtxAPGGuider(scale=3.0, eta=1.0, norm_threshold=0.0)), + ("LtxAPGGuiderThreshold", lambda: LtxAPGGuider(scale=3.0, eta=1.0, norm_threshold=1.0)), + ("LegacyStatefulAPGGuider", + lambda: LegacyStatefulAPGGuider(scale=2.0, eta=0.75, norm_threshold=5.0, momentum=0.5)), + ) + composes: list[int] = [] + for _, factory in probe_guiders: + for shape in probe_shapes: + left = torch.from_numpy(make("ltx2.guide.probe.a", int(np.prod(shape)))).reshape(shape) + right = torch.from_numpy(make("ltx2.guide.probe.b", int(np.prod(shape)))).reshape(shape) + try: + factory().delta(left, right) + composes.append(1) + except (RuntimeError, IndexError): + composes.append(0) + out.write("inline constexpr const char* kLtx2GuideProbeNames[] = {\n") + for name, _ in probe_guiders: + out.write(f' "{name}",\n') + out.write("};\n") + emit_scalar(out, "kLtx2GuideProbeGuiderCount", len(probe_guiders)) + emit_scalar(out, "kLtx2GuideProbeShapeCount", len(probe_shapes)) + emit_i64(out, "kLtx2GuideProbeRanks", [len(s) for s in probe_shapes]) + # The two axes the real predicate is written in: the batch, and the axis the + # `(B, 1)` coefficient actually lands on. `square` is deliberately NOT emitted + # any more — it was a mis-generalization of these two. + emit_i64(out, "kLtx2GuideProbeBatch", [s[0] for s in probe_shapes]) + emit_i64(out, "kLtx2GuideProbeSecondLast", [s[-2] for s in probe_shapes]) + emit_i64(out, "kLtx2GuideProbeComposes", composes) + out.write("\n") + + # MultiModalGuider — the ONLY guider any ltx-pipelines entry point constructs + # (utils/constants.py:49-68 builds MultiModalGuiderParams for both streams). + mm_cases = ( + ("Official", dict(cfg_scale=3.0, stg_scale=1.0, rescale_scale=0.7, + modality_scale=3.0, skip_step=0)), + ("Audio", dict(cfg_scale=7.0, stg_scale=1.0, rescale_scale=0.7, + modality_scale=3.0, skip_step=0)), + ("NoRescale", dict(cfg_scale=3.0, stg_scale=1.0, rescale_scale=0.0, + modality_scale=3.0, skip_step=0)), + ("PositiveOnly", dict(cfg_scale=1.0, stg_scale=0.0, rescale_scale=0.0, + modality_scale=1.0, skip_step=0)), + ("Skip2", dict(cfg_scale=3.0, stg_scale=1.0, rescale_scale=0.7, + modality_scale=3.0, skip_step=2)), + ) + for tag, kwargs in mm_cases: + params = MultiModalGuiderParams(stg_blocks=[29], **kwargs) + guider = MultiModalGuider(params=params) + emit_double(out, f"kLtx2GuideMm{tag}CfgScale", kwargs["cfg_scale"]) + emit_double(out, f"kLtx2GuideMm{tag}StgScale", kwargs["stg_scale"]) + emit_double(out, f"kLtx2GuideMm{tag}RescaleScale", kwargs["rescale_scale"]) + emit_double(out, f"kLtx2GuideMm{tag}ModalityScale", kwargs["modality_scale"]) + emit_scalar(out, f"kLtx2GuideMm{tag}SkipStep", kwargs["skip_step"]) + emit_bool(out, f"kLtx2GuideMm{tag}DoUncond", guider.do_unconditional_generation()) + emit_bool(out, f"kLtx2GuideMm{tag}DoPerturbed", guider.do_perturbed_generation()) + emit_bool(out, f"kLtx2GuideMm{tag}DoModality", guider.do_isolated_modality_generation()) + emit_i64( + out, + f"kLtx2GuideMm{tag}SkipStepMask", + [1 if guider.should_skip_step(step) else 0 for step in range(8)], + ) + emit_f32( + out, + f"kLtx2GuideMm{tag}Golden", + guider.calculate(cond, uncond, perturbed, modality).numpy(), + ) + + # The sigma-binned factory (guiders.py:214-230, 317-335). Keys are bin UPPER + # bounds sorted descending; the rule picks the SMALLEST key >= sigma, and a + # sigma above every key falls back to the largest. Both edges are queried. + bins = { + 1.0: MultiModalGuiderParams(cfg_scale=3.0, stg_scale=1.0, stg_blocks=[29]), + 0.5: MultiModalGuiderParams(cfg_scale=5.0, stg_scale=0.5, stg_blocks=[29]), + 0.25: MultiModalGuiderParams(cfg_scale=7.0, stg_scale=0.0, stg_blocks=[29]), + } + factory = MultiModalGuiderFactory.from_dict(bins) + queries = [2.0, 1.0, 0.75, 0.5, 0.4, 0.25, 0.1, 0.0] + emit_f64(out, "kLtx2GuideBinKeys", sorted(bins.keys(), reverse=True)) + emit_f64(out, "kLtx2GuideBinCfgScales", [bins[k].cfg_scale for k in sorted(bins, reverse=True)]) + emit_f64(out, "kLtx2GuideBinQueries", queries) + emit_f64(out, "kLtx2GuideBinResolvedCfg", [factory.params(q).cfg_scale for q in queries]) + constant = MultiModalGuiderFactory.constant( + MultiModalGuiderParams(cfg_scale=4.0, stg_blocks=[29]) + ) + emit_f64(out, "kLtx2GuideConstantResolvedCfg", + [constant.params(q).cfg_scale for q in queries]) + + +# --------------------------------------------------------------------------- +# Section 5 — perturbations (guidance/perturbations.py) +# --------------------------------------------------------------------------- + +_PERTURB_BLOCKS = 4 + + +def section_perturbations(out) -> None: + from ltx_core.guidance.perturbations import ( # noqa: PLC0415 + BatchedPerturbationConfig, + Perturbation, + PerturbationConfig, + PerturbationType, + ) + + section(out, "Section 5 - perturbations (guidance/perturbations.py)") + + emit_scalar(out, "kLtx2PerturbTypeCount", len(PerturbationType)) + emit_scalar(out, "kLtx2PerturbNumBlocks", _PERTURB_BLOCKS) + emit_scalar(out, "kLtx2PerturbSkipVideoSelfAttn", int(PerturbationType.SKIP_VIDEO_SELF_ATTN)) + emit_scalar(out, "kLtx2PerturbSkipAudioSelfAttn", int(PerturbationType.SKIP_AUDIO_SELF_ATTN)) + emit_scalar(out, "kLtx2PerturbSkipA2vCrossAttn", int(PerturbationType.SKIP_A2V_CROSS_ATTN)) + emit_scalar(out, "kLtx2PerturbSkipV2aCrossAttn", int(PerturbationType.SKIP_V2A_CROSS_ATTN)) + out.write("\n") + + # The EMPTY configuration is the one LTX-2.5 actually runs (spec section 2 puts + # STG behind the guider, and L2 ships `perturbations=None`), so its all-ones + # mask is the shipped default rather than a corner case. + empty = BatchedPerturbationConfig.empty(batch_size=3, num_blocks=_PERTURB_BLOCKS) + emit_i64(out, "kLtx2PerturbEmptyMask", empty.block_masks.to(int).numpy()) + + # A mixed batch: sample 0 skips video self-attn in blocks {1,2}, sample 1 + # skips A2V cross-attn in EVERY block (blocks=None), sample 2 is unperturbed. + configs = [ + PerturbationConfig([Perturbation(PerturbationType.SKIP_VIDEO_SELF_ATTN, [1, 2])]), + PerturbationConfig([Perturbation(PerturbationType.SKIP_A2V_CROSS_ATTN, None)]), + PerturbationConfig.empty(), + ] + mixed = BatchedPerturbationConfig(configs, num_blocks=_PERTURB_BLOCKS) + emit_i64(out, "kLtx2PerturbMixedMask", mixed.block_masks.to(int).numpy()) + emit_i64( + out, + "kLtx2PerturbMixedAny", + [ + 1 if mixed.any_in_batch(PerturbationType(direction), block) else 0 + for direction in range(len(PerturbationType)) + for block in range(_PERTURB_BLOCKS) + ], + ) + emit_i64( + out, + "kLtx2PerturbMixedAll", + [ + 1 if mixed.all_in_batch(PerturbationType(direction), block) else 0 + for direction in range(len(PerturbationType)) + for block in range(_PERTURB_BLOCKS) + ], + ) + sliced = mixed.batch_slice(1, 3) + emit_i64(out, "kLtx2PerturbSlicedMask", sliced.block_masks.to(int).numpy()) + + +# --------------------------------------------------------------------------- +# Section 6 — patchifiers (components/patchifiers.py) +# --------------------------------------------------------------------------- + +_PATCH_SIZE = 2 +_PATCH_C, _PATCH_F, _PATCH_H, _PATCH_W = 3, 2, 4, 6 + + +def section_patchifiers(out) -> None: + import torch # noqa: PLC0415 + from ltx_core.components.patchifiers import ( # noqa: PLC0415 + AudioPatchifier, + VideoLatentPatchifier, + get_pixel_coords, + ) + from ltx_core.types import ( # noqa: PLC0415 + AudioLatentShape, + SpatioTemporalScaleFactors, + VideoLatentShape, + ) + + section(out, "Section 6 - patchifiers (components/patchifiers.py)") + + video = VideoLatentPatchifier(patch_size=_PATCH_SIZE) + count = _PATCH_C * _PATCH_F * _PATCH_H * _PATCH_W + latent = torch.from_numpy(make("ltx2.patch.video", count, 1.0)).reshape( + 1, _PATCH_C, _PATCH_F, _PATCH_H, _PATCH_W + ) + tokens = video.patchify(latent) + shape = VideoLatentShape(batch=1, channels=_PATCH_C, frames=_PATCH_F, + height=_PATCH_H, width=_PATCH_W) + restored = video.unpatchify(tokens, shape) + + emit_scalar(out, "kLtx2PatchSize", _PATCH_SIZE) + emit_scalar(out, "kLtx2PatchChannels", _PATCH_C) + emit_scalar(out, "kLtx2PatchFrames", _PATCH_F) + emit_scalar(out, "kLtx2PatchHeight", _PATCH_H) + emit_scalar(out, "kLtx2PatchWidth", _PATCH_W) + emit_scalar(out, "kLtx2PatchTokenCount", video.get_token_count(shape)) + emit_scalar(out, "kLtx2PatchTokens", tokens.shape[1]) + emit_scalar(out, "kLtx2PatchTokenDim", tokens.shape[2]) + emit_f32(out, "kLtx2PatchVideoLatent", latent.numpy()) + emit_f32(out, "kLtx2PatchVideoTokens", tokens.numpy()) + emit_f32(out, "kLtx2PatchVideoRestored", restored.numpy()) + + bounds = video.get_patch_grid_bounds(shape) + emit_i64(out, "kLtx2PatchBoundsShape", list(bounds.shape)) + emit_i64(out, "kLtx2PatchBoundsGolden", bounds.numpy()) + + scale = SpatioTemporalScaleFactors.default() + emit_i64(out, "kLtx2PatchScaleFactors", [scale.time, scale.height, scale.width]) + for tag, causal in (("Plain", False), ("CausalFix", True)): + pixels = get_pixel_coords(bounds.clone(), scale, causal_fix=causal) + emit_i64(out, f"kLtx2PatchPixelCoords{tag}Golden", pixels.numpy()) + + audio_frames, audio_channels, audio_mels = 5, 2, 3 + audio_count = audio_channels * audio_frames * audio_mels + audio_latent = torch.from_numpy(make("ltx2.patch.audio", audio_count, 1.0)).reshape( + 1, audio_channels, audio_frames, audio_mels + ) + audio_shape = AudioLatentShape(batch=1, channels=audio_channels, frames=audio_frames, + mel_bins=audio_mels) + emit_scalar(out, "kLtx2PatchAudioFrames", audio_frames) + emit_scalar(out, "kLtx2PatchAudioChannels", audio_channels) + emit_scalar(out, "kLtx2PatchAudioMelBins", audio_mels) + emit_f32(out, "kLtx2PatchAudioLatent", audio_latent.numpy()) + for tag, causal, shift in (("Causal", True, 0), ("NonCausal", False, 0), ("Shift2", True, 2)): + patchifier = AudioPatchifier(patch_size=1, is_causal=causal, shift=shift) + audio_tokens = patchifier.patchify(audio_latent) + emit_scalar(out, f"kLtx2PatchAudio{tag}Shift", shift) + emit_bool(out, f"kLtx2PatchAudio{tag}IsCausal", causal) + emit_scalar(out, f"kLtx2PatchAudio{tag}TokenCount", patchifier.get_token_count(audio_shape)) + emit_f32(out, f"kLtx2PatchAudio{tag}Tokens", audio_tokens.numpy()) + emit_f32( + out, + f"kLtx2PatchAudio{tag}Restored", + patchifier.unpatchify(audio_tokens, audio_shape).numpy(), + ) + timings = patchifier.get_patch_grid_bounds(audio_shape) + emit_i64(out, f"kLtx2PatchAudio{tag}TimingShape", list(timings.shape)) + emit_f32(out, f"kLtx2PatchAudio{tag}TimingGolden", timings.numpy()) + # The two rates the timings are built from (patchifiers.py:177-180). They set + # the seconds-per-latent-frame the DiT's audio RoPE is indexed by, so a wrong + # one is a silently mistimed soundtrack rather than an error. + default_audio = AudioPatchifier(patch_size=1) + emit_scalar(out, "kLtx2PatchAudioSampleRate", default_audio.sample_rate) + emit_scalar(out, "kLtx2PatchAudioHopLength", default_audio.hop_length) + emit_scalar(out, "kLtx2PatchAudioDownsample", default_audio.audio_latent_downsample_factor) + + +# --------------------------------------------------------------------------- +# Section 7 — the pipeline recipes +# --------------------------------------------------------------------------- + + +def _module_by_path(name: str, path: Path): + spec = importlib.util.spec_from_file_location(name, path) + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +def _module_literals(path: Path, wanted: tuple[str, ...]) -> dict: + """Module-level literal assignments, read with `ast` and NOT executed. + + `ltx_pipelines/distilled.py` and vLLM-Omni's `ltx2_recipes.py` both import + their whole stacks, so neither can be executed here; their constants are + plain literals, and reading them from the source is exact. A name that + disappears upstream RAISES rather than falling back to a remembered value. + """ + tree = ast.parse(path.read_text(encoding="utf-8")) + found: dict = {} + for node in tree.body: + targets = [] + if isinstance(node, ast.Assign): + targets = [t.id for t in node.targets if isinstance(t, ast.Name)] + elif isinstance(node, ast.AnnAssign) and isinstance(node.target, ast.Name): + targets = [node.target.id] + for target in targets: + if target in wanted: + found[target] = node.value + missing = [name for name in wanted if name not in found] + if missing: + raise SystemExit(f"{path}: expected module-level constants not found: {missing}") + return found + + +def section_recipes(out, ltx2_root: Path, omni_root: Path) -> None: + section(out, "Section 7 - pipeline recipes") + + pipelines = ltx2_root / "packages" / "ltx-pipelines" / "src" / "ltx_pipelines" + constants = _module_by_path("_ltx2_pipeline_constants", pipelines / "utils" / "constants.py") + + emit_f32(out, "kLtx2DistilledSigmas", constants.DISTILLED_SIGMA_VALUES) + emit_scalar(out, "kLtx2DistilledSigmaCount", len(constants.DISTILLED_SIGMA_VALUES)) + emit_f32(out, "kLtx2Stage2DistilledSigmas", constants.STAGE_2_DISTILLED_SIGMA_VALUES) + emit_scalar(out, "kLtx2Stage2DistilledSigmaCount", + len(constants.STAGE_2_DISTILLED_SIGMA_VALUES)) + emit_f32(out, "kLtx2TdpDistilledSigmas", constants.TDP_DISTILLED_SIGMAS.numpy()) + emit_scalar(out, "kLtx2TdpDistilledSigmaCount", int(constants.TDP_DISTILLED_SIGMAS.numel())) + emit_string(out, "kLtx2LightricksNegativePrompt", constants.DEFAULT_NEGATIVE_PROMPT) + emit_scalar(out, "kLtx2DefaultImageCrf", constants.DEFAULT_IMAGE_CRF) + emit_scalar(out, "kLtx2Ltx24ImageCrf", constants.LTX_2_4_IMAGE_CRF) + emit_scalar(out, "kLtx2VideoLatentChannels", constants.VIDEO_LATENT_CHANNELS) + out.write("\n") + + def emit_params(tag: str, params) -> None: + emit_scalar(out, f"kLtx2Params{tag}Seed", params.seed) + emit_scalar(out, f"kLtx2Params{tag}Stage1Height", params.stage_1_height) + emit_scalar(out, f"kLtx2Params{tag}Stage1Width", params.stage_1_width) + emit_scalar(out, f"kLtx2Params{tag}Stage2Height", params.stage_2_height) + emit_scalar(out, f"kLtx2Params{tag}Stage2Width", params.stage_2_width) + emit_scalar(out, f"kLtx2Params{tag}NumFrames", params.num_frames) + emit_double(out, f"kLtx2Params{tag}FrameRate", params.frame_rate) + emit_scalar(out, f"kLtx2Params{tag}NumInferenceSteps", params.num_inference_steps) + emit_scalar(out, f"kLtx2Params{tag}ImageCrf", params.default_image_crf) + for stream in ("video", "audio"): + guider = getattr(params, f"{stream}_guider_params") + name = f"kLtx2Params{tag}{stream.capitalize()}" + emit_double(out, f"{name}CfgScale", guider.cfg_scale) + emit_double(out, f"{name}StgScale", guider.stg_scale) + emit_double(out, f"{name}RescaleScale", guider.rescale_scale) + emit_double(out, f"{name}ModalityScale", guider.modality_scale) + emit_scalar(out, f"{name}SkipStep", guider.skip_step) + # An EMPTY `stg_blocks` is a real configuration (LTX_2_3_HQ_PARAMS + # turns STG off, constants.py:99-114) and C++ has no zero-length + # array, so the count travels separately and the array is padded. + emit_scalar(out, f"{name}StgBlockCount", len(guider.stg_blocks)) + emit_i64(out, f"{name}StgBlocks", guider.stg_blocks or [-1]) + out.write("\n") + + emit_params("Ltx2", constants.LTX_2_PARAMS) + emit_params("Ltx23", constants.LTX_2_3_PARAMS) + emit_params("Ltx24", constants.LTX_2_4_PARAMS) + emit_params("Ltx23Hq", constants.LTX_2_3_HQ_PARAMS) + + # `_PARAMS_SINCE_VERSION` (constants.py:130-133) is the newest-generation-at- + # or-below rule that gives LTX-2.5 its parameters: (2,5) >= (2,4), so 2.5 + # inherits LTX_2_4_PARAMS. That inheritance is the whole reason a 2.5 recipe + # can be written at all, so the rule is emitted, not assumed. + since = constants._PARAMS_SINCE_VERSION # noqa: SLF001 - upstream's own name + emit_scalar(out, "kLtx2ParamsSinceCount", len(since)) + emit_i64(out, "kLtx2ParamsSinceMajor", [v[0] for v, _ in since]) + emit_i64(out, "kLtx2ParamsSinceMinor", [v[1] for v, _ in since]) + emit_i64(out, "kLtx2ParamsSinceSteps", [p.num_inference_steps for _, p in since]) + emit_i64(out, "kLtx2ParamsSinceCrf", [p.default_image_crf for _, p in since]) + + # parse_model_version + detect_params, applied to the versions this port keys + # on. Executing the real function is what makes the C++ mirror gateable. + from ltx_core.loader.helpers import parse_model_version # noqa: PLC0415 + + versions = ("", "2", "2.3", "2.4", "2.5", "2.5.1", "2.4-rc2", "2.3.rc1", "3", "banana") + parsed = [parse_model_version(v.replace("-", ".")) for v in versions] + out.write("inline constexpr const char* kLtx2VersionStrings[] = {\n") + for v in versions: + out.write(f' "{v}",\n') + out.write("};\n") + emit_scalar(out, "kLtx2VersionCount", len(versions)) + emit_i64(out, "kLtx2VersionParsedLen", [len(p) for p in parsed]) + emit_i64(out, "kLtx2VersionParsedMajor", [p[0] if len(p) > 0 else -1 for p in parsed]) + emit_i64(out, "kLtx2VersionParsedMinor", [p[1] if len(p) > 1 else -1 for p in parsed]) + emit_i64( + out, + "kLtx2VersionResolvedSteps", + [_detect_params_for(constants, p).num_inference_steps for p in parsed], + ) + emit_i64( + out, + "kLtx2VersionResolvedCrf", + [_detect_params_for(constants, p).default_image_crf for p in parsed], + ) + out.write("\n") + + # distilled.py's ancestral-sampler rule. THIS is the one thing that separates + # a 2.5 distilled run from a 2.0 one: stage 1 samples with the ancestral (SDE) + # Euler step instead of the deterministic one (distilled.py:60-84, 170-185). + literals = _module_literals( + pipelines / "distilled.py", + ("ANCESTRAL_SAMPLER_SINCE_VERSION", "ANCESTRAL_ETA", "ANCESTRAL_S_NOISE", + "ANCESTRAL_NOISE_SEED_OFFSET"), + ) + since_version = ast.literal_eval(literals["ANCESTRAL_SAMPLER_SINCE_VERSION"]) + emit_i64(out, "kLtx2AncestralSinceVersion", list(since_version)) + emit_double(out, "kLtx2AncestralEta", + ast.literal_eval(literals["ANCESTRAL_ETA"])) + emit_double(out, "kLtx2AncestralSNoise", + ast.literal_eval(literals["ANCESTRAL_S_NOISE"])) + emit_scalar(out, "kLtx2AncestralNoiseSeedOffset", + ast.literal_eval(literals["ANCESTRAL_NOISE_SEED_OFFSET"])) + emit_i64( + out, + "kLtx2VersionUsesAncestral", + [1 if tuple(p) >= since_version else 0 for p in parsed], + ) + out.write("\n") + + # --- vLLM-Omni, the BINDING oracle's serving model (spec section 3) -------- + omni = omni_root / "vllm_omni" / "diffusion" / "models" / "ltx2" / "ltx2_recipes.py" + omni_literals = _module_literals( + omni, + ("LTX_DISTILLED_SIGMAS", "LTX_STAGE_2_DISTILLED_SIGMAS", + "LTX_DEFAULT_NEGATIVE_PROMPT", "_PIPELINE_RECIPES"), + ) + omni_distilled = list(ast.literal_eval(omni_literals["LTX_DISTILLED_SIGMAS"])) + omni_stage2 = list(ast.literal_eval(omni_literals["LTX_STAGE_2_DISTILLED_SIGMAS"])) + emit_f32(out, "kLtx2OmniDistilledSigmas", omni_distilled) + emit_f32(out, "kLtx2OmniStage2DistilledSigmas", omni_stage2) + emit_string(out, "kLtx2OmniNegativePrompt", + ast.literal_eval(omni_literals["LTX_DEFAULT_NEGATIVE_PROMPT"])) + # The two references DISAGREE on the default negative prompt: Lightricks' + # carries five leading tags ("has_subtitles, has_blurbox, transition from + # black, transition to black, speech_ending_short, ") that vLLM-Omni's lacks. + # Recorded as a value, per spec section 3 ("where they disagree, the + # disagreement is the finding"), instead of one being quietly preferred. + emit_bool(out, "kLtx2NegativePromptsAgree", + constants.DEFAULT_NEGATIVE_PROMPT + == ast.literal_eval(omni_literals["LTX_DEFAULT_NEGATIVE_PROMPT"])) + + keys = [ast.literal_eval(k) for k in omni_literals["_PIPELINE_RECIPES"].keys] + emit_scalar(out, "kLtx2OmniRecipeKeyCount", len(keys)) + out.write("inline constexpr const char* kLtx2OmniRecipeKinds[] = {\n") + for kind, _ in keys: + out.write(f' "{kind}",\n') + out.write("};\n") + out.write("inline constexpr const char* kLtx2OmniRecipeVersions[] = {\n") + for _, version in keys: + out.write(f' "{version}",\n') + out.write("};\n\n") + + +def _detect_params_for(constants, parsed: tuple): + """`detect_params` (constants.py:166-179) applied to an ALREADY-parsed version.""" + for since, params in constants._PARAMS_SINCE_VERSION: # noqa: SLF001 + if tuple(parsed) >= since: + return params + return constants.LTX_2_PARAMS + + +# --------------------------------------------------------------------------- +# Section 8 — the latent spatial upsampler (model/upsampler/) +# --------------------------------------------------------------------------- + +# GroupNorm(32, mid_channels) fixes mid_channels to a multiple of 32, so 32 is the +# smallest fixture that keeps the norm's group structure intact (one channel per +# group) rather than changing it. +_UPS_IN, _UPS_MID, _UPS_BLOCKS = 6, 32, 1 +_UPS_F, _UPS_H, _UPS_W = 2, 4, 6 + + +def section_upsampler(out) -> None: + import torch # noqa: PLC0415 + from ltx_core.model.upsampler.model import LatentUpsampler # noqa: PLC0415 + from ltx_core.model.upsampler.spatial_rational_resampler import ( # noqa: PLC0415 + _rational_for_scale, + ) + + section(out, "Section 8 - latent spatial upsampler (model/upsampler/)") + + emit_scalar(out, "kLtx2UpsInChannels", _UPS_IN) + emit_scalar(out, "kLtx2UpsMidChannels", _UPS_MID) + emit_scalar(out, "kLtx2UpsBlocksPerStage", _UPS_BLOCKS) + emit_scalar(out, "kLtx2UpsFrames", _UPS_F) + emit_scalar(out, "kLtx2UpsHeight", _UPS_H) + emit_scalar(out, "kLtx2UpsWidth", _UPS_W) + # ResBlock and LatentUpsampler both hardcode GroupNorm(32, ...) + # (res_block.py:24,26; model.py:50), so the group count is a pinned constant + # and not a config key a checkpoint could move. + emit_scalar(out, "kLtx2UpsNormGroups", 32) + # BlurDownsample's kernel_size default, READ OFF upstream's own signature + # (blur_downsample.py:14). `SpatialRationalResampler` never passes one + # (spatial_rational_resampler.py:38), so this default IS the shipped kernel + # width, and it silently changes the whole binomial kernel if it moves. + import inspect # noqa: PLC0415 + + from ltx_core.model.upsampler.blur_downsample import BlurDownsample # noqa: PLC0415 + + emit_scalar( + out, + "kLtx2UpsBlurKernelSize", + inspect.signature(BlurDownsample.__init__).parameters["kernel_size"].default, + ) + + count = _UPS_IN * _UPS_F * _UPS_H * _UPS_W + latent = torch.from_numpy(make("ltx2.ups.latent", count, 1.0)).reshape( + 1, _UPS_IN, _UPS_F, _UPS_H, _UPS_W + ) + emit_f32(out, "kLtx2UpsLatent", latent.numpy()) + + # `_rational_for_scale`'s supported map (spatial_rational_resampler.py:11-14) + # decides which scales exist at all; an unsupported one RAISES upstream, which + # is the refusal the port mirrors. + supported = (0.75, 1.5, 2.0, 4.0) + emit_f64(out, "kLtx2UpsRationalScales", supported) + emit_i64(out, "kLtx2UpsRationalNum", [_rational_for_scale(s)[0] for s in supported]) + emit_i64(out, "kLtx2UpsRationalDen", [_rational_for_scale(s)[1] for s in supported]) + out.write("\n") + + arms = ( + # (tag, rational_resampler, spatial_scale) + ("PixelShuffle", False, 2.0), + ("Rational2", True, 2.0), + # den > 1, so BlurDownsample actually runs its binomial kernel rather than + # short-circuiting at stride 1 (blur_downsample.py:36-37). H and W must + # stay divisible by 2 after the 3x upshuffle for the stride-2 conv to land + # on the same grid upstream lands on. + ("Rational1p5", True, 1.5), + ) + for tag, rational, scale in arms: + module = LatentUpsampler( + in_channels=_UPS_IN, + mid_channels=_UPS_MID, + num_blocks_per_stage=_UPS_BLOCKS, + dims=3, + spatial_upsample=True, + temporal_upsample=False, + spatial_scale=scale, + rational_resampler=rational, + ) + module.eval() + manifest = fill_module(module, f"ltx2.ups.{tag}.") + result = module(latent) + emit_bool(out, f"kLtx2Ups{tag}Rational", rational) + emit_double(out, f"kLtx2Ups{tag}Scale", scale) + emit_i64(out, f"kLtx2Ups{tag}OutShape", list(result.shape)) + emit_manifest(out, f"kLtx2Ups{tag}Param", manifest) + emit_f32(out, f"kLtx2Ups{tag}Golden", result.numpy()) + + # The binomial anti-alias kernel is COMPUTED at construction, never loaded + # (blur_downsample.py:29-33), so both sides must build it independently — the + # same rule the VAE's kaiser-sinc filters follow. + from ltx_core.model.upsampler.blur_downsample import BlurDownsample # noqa: PLC0415 + + for size in (3, 5, 7): + blur = BlurDownsample(dims=2, stride=2, kernel_size=size) + emit_f32(out, f"kLtx2UpsBlurKernel{size}", blur.kernel.numpy()) + + +# --------------------------------------------------------------------------- +# Section 9 — the duration head (duration_head/duration_head.py) +# --------------------------------------------------------------------------- + +_DUR_VIDEO_DIM, _DUR_AUDIO_DIM = 16, 8 +_DUR_HIDDEN, _DUR_QUERIES, _DUR_HEADS, _DUR_MLP = 12, 2, 3, 10 +_DUR_VIDEO_TOKENS, _DUR_AUDIO_TOKENS = 5, 3 + + +def section_duration_head(out) -> None: + import torch # noqa: PLC0415 + from ltx_core.duration_head.duration_head import DurationHead # noqa: PLC0415 + + section(out, "Section 9 - duration head (duration_head/duration_head.py)") + + head = DurationHead( + video_cross_attention_dim=_DUR_VIDEO_DIM, + audio_cross_attention_dim=_DUR_AUDIO_DIM, + pooler_hidden_dim=_DUR_HIDDEN, + num_queries=_DUR_QUERIES, + num_pooler_heads=_DUR_HEADS, + mlp_hidden=_DUR_MLP, + ) + head.eval() + manifest = fill_module(head, "ltx2.dur.") + + video = torch.from_numpy( + make("ltx2.dur.video_tokens", _DUR_VIDEO_TOKENS * _DUR_VIDEO_DIM, 1.0) + ).reshape(1, _DUR_VIDEO_TOKENS, _DUR_VIDEO_DIM) + audio = torch.from_numpy( + make("ltx2.dur.audio_tokens", _DUR_AUDIO_TOKENS * _DUR_AUDIO_DIM, 1.0) + ).reshape(1, _DUR_AUDIO_TOKENS, _DUR_AUDIO_DIM) + + emit_scalar(out, "kLtx2DurVideoDim", _DUR_VIDEO_DIM) + emit_scalar(out, "kLtx2DurAudioDim", _DUR_AUDIO_DIM) + emit_scalar(out, "kLtx2DurHidden", _DUR_HIDDEN) + emit_scalar(out, "kLtx2DurQueries", _DUR_QUERIES) + emit_scalar(out, "kLtx2DurHeads", _DUR_HEADS) + emit_scalar(out, "kLtx2DurMlpHidden", _DUR_MLP) + emit_scalar(out, "kLtx2DurVideoTokens", _DUR_VIDEO_TOKENS) + emit_scalar(out, "kLtx2DurAudioTokens", _DUR_AUDIO_TOKENS) + emit_f32(out, "kLtx2DurVideoInput", video.numpy()) + emit_f32(out, "kLtx2DurAudioInput", audio.numpy()) + emit_manifest(out, "kLtx2DurParam", manifest) + + # Both streams, then each alone. The concat is along the TOKEN axis + # (duration_head.py:113), so a port that concatenated along the feature axis + # would still produce a finite duration for the both-streams arm alone. + emit_f32(out, "kLtx2DurBothGolden", head(video, audio).numpy()) + emit_f32(out, "kLtx2DurVideoOnlyGolden", head(video_tokens=video).numpy()) + emit_f32(out, "kLtx2DurAudioOnlyGolden", head(audio_tokens=audio).numpy()) + # The pooled attention output, before the MLP, so a pooler defect localizes + # instead of arriving as one wrong scalar. + projected = torch.cat( + [ + head.video_input_proj(video) + head.video_modality_emb, + head.audio_input_proj(audio) + head.audio_modality_emb, + ], + dim=1, + ) + emit_f32(out, "kLtx2DurProjectedGolden", projected.numpy()) + emit_f32(out, "kLtx2DurPooledGolden", head.attention_pooler(projected).numpy()) + + # AN INVARIANCE OF UPSTREAM'S OWN MODULE, measured rather than assumed, and + # emitted so it is gated rather than left as a silent hole. + # + # A mutation that REVERSED the token-axis concat left every golden in this + # file green. The reason is not a weak fixture: AttentionPooler is + # cross-attention with no mask and no positional encoding over the token axis + # (duration_head.py:45-49), so it is PERMUTATION INVARIANT and upstream cannot + # distinguish the two orders either. Measured on upstream: a reversed concat + # and a random permutation both move the pooled output by 2.98e-08, i.e. f32 + # reduction-order noise, while giving the audio stream the VIDEO modality + # embedding moves it by 4.80e-03. + # + # So what tags the two streams is the modality EMBEDDING, not the order, and + # that is what the gate must hold. Both goldens are emitted: the permuted pool + # (which must MATCH) and the mis-tagged one (which must NOT). + reversed_tokens = torch.cat( + [ + head.audio_input_proj(audio) + head.audio_modality_emb, + head.video_input_proj(video) + head.video_modality_emb, + ], + dim=1, + ) + emit_f32(out, "kLtx2DurPooledReversedGolden", + head.attention_pooler(reversed_tokens).numpy()) + mistagged = torch.cat( + [ + head.video_input_proj(video) + head.video_modality_emb, + head.audio_input_proj(audio) + head.video_modality_emb, + ], + dim=1, + ) + emit_f32(out, "kLtx2DurPooledMistaggedGolden", head.attention_pooler(mistagged).numpy()) + + +# --------------------------------------------------------------------------- +# Section 10 — Embeddings1DConnector (text_encoders/gemma/embeddings_connector.py) +# --------------------------------------------------------------------------- + +_CONN_HEADS, _CONN_HEAD_DIM, _CONN_LAYERS = 3, 8, 2 +_CONN_REGISTERS, _CONN_SEQ, _CONN_BATCH = 4, 8, 2 + + +def section_connector(out) -> None: + import torch # noqa: PLC0415 + from ltx_core.model.transformer.rope import LTXRopeType # noqa: PLC0415 + from ltx_core.text_encoders.gemma.embeddings_connector import ( # noqa: PLC0415 + Embeddings1DConnector, + ) + + section(out, "Section 10 - Embeddings1DConnector (text_encoders/gemma/)") + + inner = _CONN_HEADS * _CONN_HEAD_DIM + emit_scalar(out, "kLtx2ConnHeads", _CONN_HEADS) + emit_scalar(out, "kLtx2ConnHeadDim", _CONN_HEAD_DIM) + emit_scalar(out, "kLtx2ConnLayers", _CONN_LAYERS) + emit_scalar(out, "kLtx2ConnInnerDim", inner) + emit_scalar(out, "kLtx2ConnRegisters", _CONN_REGISTERS) + emit_scalar(out, "kLtx2ConnSeq", _CONN_SEQ) + emit_scalar(out, "kLtx2ConnBatch", _CONN_BATCH) + emit_double(out, "kLtx2ConnTheta", 10000.0) + + # The rms_norm eps, READ OFF upstream's own signature (utils.py:7) rather than + # retyped. Spec §7.0(a): this is the invisible-constant class — the fixture's + # rows are never near-zero, so the value comparison alone accepts a 100x + # change. The C++ header says it is "pinned here"; this is what makes that + # true, and it moves if upstream moves. + import inspect # noqa: PLC0415 + + from ltx_core.utils import rms_norm # noqa: PLC0415 + + emit_double( + out, "kLtx2ConnRmsNormEps", inspect.signature(rms_norm).parameters["eps"].default + ) + + count = _CONN_BATCH * _CONN_SEQ * inner + hidden = torch.from_numpy(make("ltx2.conn.hidden", count, 1.0)).reshape( + _CONN_BATCH, _CONN_SEQ, inner + ) + emit_f32(out, "kLtx2ConnHidden", hidden.numpy()) + + # The additive mask upstream's own preprocessor produces: 0 for a kept token, + # -finfo(f32).max for a padded one (transformer_args.py:199-206). Row 0 keeps + # 5 of 8, row 1 keeps all 8, so the register substitution is exercised on one + # row and inert on the other. + keep = np.ones((_CONN_BATCH, _CONN_SEQ), dtype=np.int32) + keep[0, 5:] = 0 + mask = torch.zeros(_CONN_BATCH, 1, 1, _CONN_SEQ, dtype=torch.float32) + mask[0, 0, 0, 5:] = -torch.finfo(torch.float32).max + emit_i64(out, "kLtx2ConnKeep", keep) + + for tag, rope_type, double_precision, registers, gated, ff_bias in ( + ("Split", LTXRopeType.SPLIT, False, _CONN_REGISTERS, False, True), + ("Interleaved", LTXRopeType.INTERLEAVED, False, _CONN_REGISTERS, False, True), + ("Float64", LTXRopeType.SPLIT, True, _CONN_REGISTERS, False, True), + # num_learnable_registers=None disables the substitution entirely and the + # mask survives into every attention (embeddings_connector.py:167-170). + ("NoRegisters", LTXRopeType.SPLIT, False, None, False, True), + # The two connector config keys a 2.5 checkpoint can flip + # (embeddings_connector.py:216-217). + ("GatedNoBias", LTXRopeType.SPLIT, False, _CONN_REGISTERS, True, False), + ): + module = Embeddings1DConnector( + attention_head_dim=_CONN_HEAD_DIM, + num_attention_heads=_CONN_HEADS, + num_layers=_CONN_LAYERS, + positional_embedding_theta=10000.0, + positional_embedding_max_pos=[1], + num_learnable_registers=registers, + rope_type=rope_type, + double_precision_rope=double_precision, + apply_gated_attention=gated, + ff_bias=ff_bias, + ) + module.eval() + manifest = fill_module(module, f"ltx2.conn.{tag}.") + hidden_states, out_mask = module(hidden.clone(), mask.clone()) + + emit_bool(out, f"kLtx2Conn{tag}Interleaved", rope_type == LTXRopeType.INTERLEAVED) + emit_bool(out, f"kLtx2Conn{tag}DoublePrecision", double_precision) + emit_scalar(out, f"kLtx2Conn{tag}Registers", registers if registers else 0) + emit_bool(out, f"kLtx2Conn{tag}Gated", gated) + emit_bool(out, f"kLtx2Conn{tag}FfBias", ff_bias) + emit_manifest(out, f"kLtx2Conn{tag}Param", manifest) + emit_f32(out, f"kLtx2Conn{tag}Golden", hidden_states.numpy()) + emit_f32(out, f"kLtx2Conn{tag}MaskGolden", out_mask.numpy()) + + if registers: + # The register substitution on its own, so a defect there localizes + # rather than arriving as a wrong final tensor. `learnable_registers` + # is a BFLOAT16 parameter (embeddings_connector.py:135-137); this + # golden is what proves the C++ side rounds to bf16 too instead of + # carrying a wider value the final comparison would absorb. + replaced, zeroed = module._replace_padded_with_learnable_registers( # noqa: SLF001 + hidden.clone(), mask.clone() + ) + emit_f32(out, f"kLtx2Conn{tag}RegistersGolden", + module.learnable_registers.float().numpy()) + emit_f32(out, f"kLtx2Conn{tag}ReplacedGolden", replaced.numpy()) + emit_f32(out, f"kLtx2Conn{tag}ZeroedMaskGolden", zeroed.numpy()) + + +# --------------------------------------------------------------------------- +# Driver +# --------------------------------------------------------------------------- + + +def load_upstream(root: Path) -> Path: + """Import `ltx_core` BY PATH from `root`, and prove that is what resolved.""" + src = root / "packages" / "ltx-core" / "src" + if not (src / "ltx_core" / "components" / "schedulers.py").is_file(): + raise SystemExit(f"no ltx_core under {src}; point --ltx2 at a Lightricks/LTX-2 checkout") + sys.path.insert(0, str(src)) + import ltx_core # noqa: PLC0415 + + resolved = Path(ltx_core.__file__).resolve() + if not resolved.is_relative_to(src.resolve()): + raise SystemExit( + f"ltx_core resolved to {resolved}, which is NOT under the checkout at {src}. " + "Refusing to generate goldens from an oracle this script did not choose." + ) + return src + + +def upstream_revision(root: Path, label: str) -> str: + """The exact upstream tree these goldens were produced from. + + A DIRTY tree is refused outright. A revision anchor is only worth having if it + describes the code that ran, and `rev-parse` on a tree with uncommitted edits + reports a clean SHA for goldens that SHA does not produce — which is worse + than no anchor, because it survives a bisect and misdirects it. + """ + try: + head = subprocess.run( + ["git", "-C", str(root), "rev-parse", "HEAD"], + check=True, capture_output=True, text=True, + ).stdout.strip() + status = subprocess.run( + ["git", "-C", str(root), "status", "--porcelain"], + check=True, capture_output=True, text=True, + ).stdout.strip() + except Exception as exc: # noqa: BLE001 + raise SystemExit( + f"cannot read the {label} revision at {root}: {exc}. These goldens are only " + "interpretable against a known upstream tree; refusing to emit an unanchored one." + ) from exc + if status: + raise SystemExit( + f"{label} checkout at {root} is DIRTY:\n{status}\n" + "Refusing to stamp a clean revision on goldens that tree does not describe." + ) + return head + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--ltx2", required=True, type=Path, + help="a checkout of Lightricks/LTX-2 (the repo root)") + parser.add_argument("--vllm-omni", required=True, type=Path, + help="a checkout of vllm-project/vllm-omni (the repo root)") + parser.add_argument("--out", required=True, type=Path) + args = parser.parse_args() + + root = args.ltx2.expanduser().resolve() + omni_root = args.vllm_omni.expanduser().resolve() + load_upstream(root) + revision = upstream_revision(root, "LTX-2") + omni_revision = upstream_revision(omni_root, "vllm-omni") + + import torch + + torch.set_grad_enabled(False) + torch.manual_seed(0) + + args.out.parent.mkdir(parents=True, exist_ok=True) + with args.out.open("w", encoding="utf-8") as out: + out.write( + "// GENERATED by scripts/gen-ltx2-pipeline-goldens.py — DO NOT EDIT BY HAND.\n" + "//\n" + "// LTX-2.5 PIPELINE goldens (phase L5), produced by executing the UPSTREAM\n" + "// ltx_core modules at reduced dimensions on CPU and by reading the recipe\n" + "// constants out of ltx-pipelines and vLLM-Omni. Weights and inputs come from\n" + "// the shared deterministic stream, so no weight byte is checked in.\n" + "// Regenerate with:\n" + "// python3 scripts/gen-ltx2-pipeline-goldens.py --ltx2 \n" + "// --vllm-omni \n" + "// --out tests/vllm/models/ltx2_pipeline_goldens.inc\n" + "//\n" + f"// Upstream revision (Lightricks/LTX-2): {revision}\n" + f"// Upstream revision (vllm-omni): {omni_revision}\n" + "//\n" + "// See .agents/specs/ltx-2-5.md section 7 for why this is the gate, and\n" + "// section 7.0 for why BOTH the identity assertion and these SHAs are here.\n" + "#pragma once\n\n#include \n\nnamespace vllm_test {\n\n" + "// The upstream trees these numbers came from. The suite asserts both equal\n" + "// the SHAs it pins, so regenerating against a DIFFERENT checkout fails the\n" + "// gate instead of silently replacing the oracle. The generator additionally\n" + "// REFUSES a dirty tree, so a SHA here always describes the code that ran.\n" + f'inline constexpr const char* kLtx2PipelineUpstreamRevision = "{revision}";\n' + f'inline constexpr const char* kLtx2OmniUpstreamRevision = "{omni_revision}";\n\n' + ) + section_schedulers(out) + section_noiser(out) + section_diffusion_steps(out) + section_guiders(out) + section_perturbations(out) + section_patchifiers(out) + section_recipes(out, root, omni_root) + section_upsampler(out) + section_duration_head(out) + section_connector(out) + out.write("} // namespace vllm_test\n") + print(f"wrote {args.out}", file=sys.stderr) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/gen-ltx2-prompt-tokens-goldens.py b/scripts/gen-ltx2-prompt-tokens-goldens.py new file mode 100644 index 000000000..661942467 --- /dev/null +++ b/scripts/gen-ltx2-prompt-tokens-goldens.py @@ -0,0 +1,197 @@ +#!/usr/bin/env python3 +"""Emit tests/vllm/models/ltx2_prompt_tokens_goldens.inc — REAL prompts, REAL vocab. + +`LTXGemmaTokenizer.tokenize_with_weights` (tokenizer.py:31-59) run against the +tokenizer the LTX-2.5 text encoder actually ships — which it ships AS A TENSOR, +`tokenizer_json` U8 [32169626] inside the safetensors file, not as a sibling +`tokenizer.json` (gemma_assets.py:34-36). + +Why this gate is separate from a synthetic one. A hand-built tokenizer fixture +proves the WRAPPER — strip, BOS prepend, left pad, truncation order — and proves +nothing about the 262144-entry SentencePiece-flavoured BPE the model was trained +with. This one runs real English prompts through the real merges and checks the +ids, so a pre-tokenizer or normalizer mismatch on the shipped vocab is caught as +a wrong id rather than as a slightly different video. + +The shipped tokenizer is the Gemma metaspace-via-normalizer form the C++ side +already recognizes: `Replace(" " -> "U+2581")` NORMALIZER plus +`Split(" ", MergedWithPrevious)` PRE-TOKENIZER, byte-fallback decoder, BPE with +262144 entries and 24 added tokens. Its `post_processor` is a TemplateProcessing +whose `special_tokens` map is EMPTY, which is exactly why `ltx_core` prepends BOS +itself and says so (tokenizer.py:12-15) — measured here rather than believed. + +Needs the checkpoint, so this is not a CI generator; the emitted .inc is, and the +C++ case that uses the 32 MB tensor is env-gated on the checkpoint being present. + +Usage: + scripts/gen-ltx2-prompt-tokens-goldens.py \ + --text-encoder $CHECKPOINT_ROOT/ltx-2.5/vonkaiser-fp8-nvfp4/text_encoders/\ +gemma4-12b-with-proj-nvfp4-torchao.safetensors \ + --out tests/vllm/models/ltx2_prompt_tokens_goldens.inc +""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import struct +import sys +from pathlib import Path + +# gemma_assets.py:162 / base_encoder.py:231-236, and diffusers +# pipeline_ltx2.py:304 independently. +MAX_LENGTH = 1024 + +# Deliberately varied. Each one is here for a reason a single prompt would miss: +# * leading/trailing whitespace, which upstream strips (tokenizer.py:33); +# * an empty prompt, which must still produce a BOS and 1023 pads; +# * punctuation and digits, which the Split/Replace pre-tokenizer treats +# differently from letters; +# * a long cinematic prompt of the kind this model is actually driven with. +PROMPTS = [ + "a red fox running through deep snow at sunrise", + " a cat sits on a windowsill, watching rain fall ", + "", + "Cinematic wide shot, 35mm: two astronauts walk across a rust-coloured dune " + "field while a ringed planet rises behind them; slow dolly-in, volumetric " + "light, 24 fps.", +] + + +def read_tokenizer_tensor(path: Path) -> bytes: + """Pull `tokenizer_json` out of the safetensors file WITHOUT loading weights.""" + with path.open("rb") as f: + header_len = struct.unpack(" MAX_LENGTH: # truncation=True + ids = ids[:MAX_LENGTH] + if not ids or ids[0] != bos_id: # :44-46 + ids = [bos_id, *ids][:MAX_LENGTH] + pad = MAX_LENGTH - len(ids) # :48-54, LEFT side + return [pad_id] * pad + ids, [0] * pad + [1] * len(ids) + + +def _emit_i32(out, name: str, values) -> None: + out.write(f"inline constexpr int32_t {name}[] = {{\n") + for i in range(0, len(values), 16): + out.write(" " + ", ".join(str(int(v)) for v in values[i : i + 16]) + ",\n") + out.write("};\n\n") + + +def _cxx_string(s: str) -> str: + out = [] + for ch in s: + if ch == '"': + out.append('\\"') + elif ch == "\\": + out.append("\\\\") + elif ch == "\n": + out.append("\\n") + elif 0x20 <= ord(ch) < 0x7F: + out.append(ch) + else: + out.extend("\\x%02x" % b for b in ch.encode("utf-8")) + out.append('" "') + return '"' + "".join(out) + '"' + + +def main() -> int: + ap = argparse.ArgumentParser() + ap.add_argument("--text-encoder", required=True, type=Path) + ap.add_argument("--out", required=True, type=Path) + args = ap.parse_args() + + # ORACLE IDENTITY, asserted: `tokenizers` is what HuggingFace's own fast + # tokenizer runs, and it is the thing our BPE is being compared against. + import tokenizers # noqa: PLC0415 + from tokenizers import Tokenizer # noqa: PLC0415 + + blob = read_tokenizer_tensor(args.text_encoder) + digest = hashlib.sha256(blob).hexdigest() + tok = Tokenizer.from_str(blob.decode("utf-8")) + + doc = json.loads(blob) + post = doc.get("post_processor") or {} + specials = post.get("special_tokens") or {} + # MEASURED, and the reason `ltx_core` prepends BOS by hand. + added = {t["content"]: t["id"] for t in doc.get("added_tokens", [])} + bos_id, pad_id = added[""], added[""] + + rows_ids, rows_mask, valid = [], [], [] + for p in PROMPTS: + ids, mask = tokenize_with_weights(tok, p, bos_id, pad_id) + assert len(ids) == MAX_LENGTH and len(mask) == MAX_LENGTH + rows_ids.append(ids) + rows_mask.append(mask) + valid.append(sum(mask)) + + with args.out.open("w", encoding="utf-8") as out: + out.write( + "// GENERATED by scripts/gen-ltx2-prompt-tokens-goldens.py — DO NOT EDIT.\n" + "//\n" + "// `LTXGemmaTokenizer.tokenize_with_weights` (tokenizer.py:31-59) over the\n" + "// tokenizer the LTX-2.5 text encoder SHIPS AS A TENSOR, run by HuggingFace\n" + f"// `tokenizers` {tokenizers.__version__}.\n" + "//\n" + f"// tokenizer_json sha256: {digest}\n" + f"// source: {args.text_encoder}\n" + "//\n" + "// MEASURED on that tokenizer, and the reason upstream prepends BOS by hand\n" + f"// (tokenizer.py:12-15): its post_processor is `{post.get('type')}` with\n" + f"// {len(specials)} special_tokens, so `add_special_tokens=True` adds NOTHING.\n" + "// diffusers relies on exactly that flag (pipeline_ltx2.py:339) and would\n" + "// therefore produce a prompt with no BOS; `ltx_core` is followed here.\n" + "#pragma once\n\n" + "#include \n\n" + "namespace vllm_test {\n\n" + ) + out.write(f"inline constexpr int64_t kLtxPromptMaxLength = {MAX_LENGTH};\n") + out.write(f"inline constexpr int32_t kLtxPromptBosId = {bos_id};\n") + out.write(f"inline constexpr int32_t kLtxPromptPadId = {pad_id};\n") + out.write(f"inline constexpr int64_t kLtxPromptCount = {len(PROMPTS)};\n") + out.write( + f'inline constexpr const char* kLtxPromptTokenizerSha256 = "{digest}";\n\n') + out.write("inline constexpr const char* kLtxPromptText[] = {\n") + for p in PROMPTS: + out.write(" " + _cxx_string(p) + ",\n") + out.write("};\n\n") + _emit_i32(out, "kLtxPromptValidCount", valid) + for i, (ids, mask) in enumerate(zip(rows_ids, rows_mask)): + _emit_i32(out, f"kLtxPromptIds_{i}", ids) + _emit_i32(out, f"kLtxPromptMask_{i}", mask) + out.write("} // namespace vllm_test\n") + + sys.stderr.write( + f"wrote {args.out} — {len(PROMPTS)} prompts, valid token counts {valid}\n") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/gen-ltx2-quant-goldens.py b/scripts/gen-ltx2-quant-goldens.py new file mode 100644 index 000000000..a54eeb766 --- /dev/null +++ b/scripts/gen-ltx2-quant-goldens.py @@ -0,0 +1,664 @@ +#!/usr/bin/env python3 +"""Emit tests/vllm/models/ltx2_quant_goldens.inc — the LTX-2.5 phase-L6 quant oracle. + +Two things need gating and they need DIFFERENT oracles, so they get different +sections and their provenance is recorded separately. + +1. THE SCALE SWIZZLE (`kLtx2Blocked*`). torchao writes NVFP4 group scales in the + cuBLAS "block scaling factors layout" + (https://docs.nvidia.com/cuda/cublas/index.html#d-block-scaling-factors-layout). + The producer is `to_blocked` in + vllm/model_executor/layers/quantization/qutlass_utils.py:165-180, whose own + header records that it was copied from + https://github.com/pytorch/ao/tree/main/torchao/prototype/mx_formats — i.e. + from the exact torchao module that quantized our checkpoint. vLLM writes the + same permutation a second time as `swizzle_blockscale` + (vllm/model_executor/layers/quantization/utils/nvfp4_utils.py:44-49). + + This generator EXECUTES that function. `import vllm.*` dies in + `vllm.distributed` on a missing `zmq`, but qutlass_utils.py itself imports + only torch, `vllm.triton_utils` and `vllm.utils.math_utils`, all of which + load, so `importlib.util.spec_from_file_location` runs the pinned FILE + directly and `to_blocked(x, backend="torch")` is called for real. The goldens + below therefore come from the producer upstream itself calls, not from a copy + of it. `--vllm` pins the checkout, and `check_transcription` additionally + diffs the body the C++ INVERSE was written against, so a change to the + permutation is caught as a source change and not only as a golden diff. + + The `backend="triton"` arm is NOT exercised: it dispatches to a Triton kernel + and this host has no active driver. That arm is a separate implementation of + the same permutation upstream, and gating it needs a GPU host. + + `swizzle_blockscale` — vLLM's SECOND writing of the same permutation — still + cannot be executed here, because it calls `.cuda()` unconditionally. It is + pinned by source fragment only, and that is stated where it is checked. + +2. THE REAL CHECKPOINT BYTES (`kLtx2Real*`). These come off the SHIPPED files on + $CHECKPOINT_ROOT — a few hundred bytes read at their own offsets, never a + payload download — and the expected values are decoded with TORCH, which is a + genuinely independent implementation of fp8-e4m3: `torch.uint8 -> + view(float8_e4m3fn) -> float()`. So the fp8 half of both dequant paths is + gated against something that is not ours. The e2m1 nibble LUT is not decoded + by torch (no fp4 dtype); it is the one already gated by the modelopt path in + tests/vllm/test_nvfp4_dequant.cpp and is reused, not re-derived. + +Usage: + python3 scripts/gen-ltx2-quant-goldens.py \\ + --vllm ~/_git/vllm \\ + --checkpoint-root /mnt/nas_share/checkpoints \\ + --out tests/vllm/models/ltx2_quant_goldens.inc + +Needs torch + numpy (CPU only). +""" + +from __future__ import annotations + +import argparse +import importlib.util +import json +import struct +import subprocess +import sys +from pathlib import Path + +import numpy as np +import torch + +# --------------------------------------------------------------------------- +# Provenance. AGENTS.md wants the upstream revision anchor; L4's decoy +# experiment (spec section 7.0(b)) showed the anchor is worthless unless the +# tree it names is the tree that ran, so a DIRTY checkout is refused outright — +# `git rev-parse HEAD` reports the committed SHA whatever the worktree holds, +# which is precisely how a clean anchor gets stamped onto drifted goldens. +# --------------------------------------------------------------------------- + + +def pinned_revision(root: Path, label: str, paths: list[str]) -> tuple[str, bool]: + """(sha, whole_tree_clean). REFUSES when any of `paths` is dirty. + + The anchor is scoped to what was actually read. A dirty file among `paths` + is fatal — that is exactly the case where `rev-parse` stamps a clean SHA onto + goldens the committed tree cannot reproduce. A dirty file ELSEWHERE is not + fatal, but it is recorded in the emitted header, because an anchor that + quietly implies more than it checked is the same defect one step removed. + """ + try: + sha = subprocess.check_output( + ["git", "-C", str(root), "rev-parse", "HEAD"], text=True + ).strip() + except Exception as exc: # noqa: BLE001 - a tarball checkout has no git metadata + raise SystemExit(f"{label}: cannot read a revision from {root}: {exc}") from exc + dirty_paths = subprocess.check_output( + ["git", "-C", str(root), "status", "--porcelain", "--"] + paths, text=True + ).strip() + if dirty_paths: + raise SystemExit( + f"{label}: the sources this generator reads are DIRTY at {sha}.\n" + " Refusing: rev-parse would stamp a CLEAN anchor onto goldens produced " + "by an edited tree, which is the exact failure .agents/specs/ltx-2-5.md " + "section 7.0(b) records. Commit or stash first.\n" + f" {dirty_paths}" + ) + whole = subprocess.check_output( + ["git", "-C", str(root), "status", "--porcelain"], text=True + ).strip() + return sha, not whole + + +# --------------------------------------------------------------------------- +# The transcription, and the pin that keeps it honest +# --------------------------------------------------------------------------- + +# The `backend="torch"` body of `to_blocked`, verbatim from +# vllm/model_executor/layers/quantization/qutlass_utils.py:167-180. Nothing here +# is EXECUTED — the goldens come from calling the real function — so this is not +# a second hand-typing that could drift from what runs. It is the text the C++ +# inverse in Ltx2UnswizzleNvfp4BlockScale was written against, pinned so that a +# change to the permutation upstream is reported as a source change rather than +# only as a silent golden diff on the next regeneration. +_TO_BLOCKED_SOURCE_ANCHOR = """ rows, cols = input_matrix.shape + n_row_blocks = cdiv(rows, 128) + n_col_blocks = cdiv(cols, 4) + + # Calculate the padded shape + padded_rows = n_row_blocks * 128 + padded_cols = n_col_blocks * 4 + + padded = input_matrix + assert (rows, cols) == (padded_rows, padded_cols) + + # Rearrange the blocks + blocks = padded.view(n_row_blocks, 128, n_col_blocks, 4).permute(0, 2, 1, 3) + rearranged = blocks.reshape(-1, 4, 32, 4).transpose(1, 2).reshape(-1, 32, 16) + + return rearranged.flatten()""" + + +# The only vLLM files this generator reads. The revision anchor is scoped to +# exactly these, and they are the ones a dirty tree is refused over. +_PINNED_VLLM_PATHS = [ + "vllm/model_executor/layers/quantization/qutlass_utils.py", + "vllm/model_executor/layers/quantization/utils/nvfp4_utils.py", + "vllm/model_executor/layers/quantization/utils/nvfp4_emulation_utils.py", +] + +# ── THE NIBBLE ORDER, PINNED ON BOTH SIDES ────────────────────────────────── +# +# .agents/specs/nvfp4-nibble-order.md. E2M1 packs two values per byte and the two +# producers we read DISAGREE about which logical element gets which nibble, so +# both conventions are anchored against a pinned, clean checkout. A silent flip +# on either side transposes every adjacent weight pair: finite, correctly shaped, +# and wrong. + +# LOW-first, vLLM's reader — and therefore torchao's and ModelOpt's, which it +# reads. Matches torchao's own producer, `pack_uint4` +# (pytorch/ao torchao/prototype/mx_formats/kernels.py:160, +# `uint8_data[::2] | uint8_data[1::2] << 4`), which is NOT pinnable here: torchao +# is not installed on this host and not vendored. vLLM's reader is the local +# witness, and it is an independent implementation of the same convention. +_VLLM_LOW_NIBBLE_ANCHORS = [ + "high = (a_flat & 0xF0) >> 4 # Upper nibbles", + "low = a_flat & 0x0F # Lower nibbles", + "combined = torch.stack((low, high), dim=1).flatten()", +] + +# HIGH-first, Lightricks' own runtime, which is what wrote the first-party +# LTX-2.5 NVFP4 DiT. +_PINNED_LTX2_PATHS = [ + "packages/ltx-kernels/docs/NVFP4.md", + "packages/ltx-kernels/csrc/nvfp4/quantize.cu", + "packages/ltx-core/src/ltx_core/quantization/nvfp4/linear.py", +] +_LTX2_HIGH_NIBBLE_ANCHORS = [ + ("packages/ltx-kernels/docs/NVFP4.md", + "`hi_first=True` (default) puts element `2j` in the **high** nibble of byte `j`;"), + ("packages/ltx-core/src/ltx_core/quantization/nvfp4/linear.py", + "element ``2j`` in the high nibble)"), + ("packages/ltx-core/src/ltx_core/quantization/nvfp4/linear.py", + "``weight_scale`` — E4M3 block scales as ``uint8``, cuBLAS 128x4 tiled layout"), + # The permutation itself, which our Ltx2UnswizzleNvfp4BlockScale inverts, and + # the `padded_cols == roundup(K/16, 4)` framing that the DiT declares. + ("packages/ltx-kernels/csrc/nvfp4/quantize.cu", + "return static_cast(tile) * 512 + (r & 31) * 16 + (r >> 5) * 4 + (col & 3);"), +] + + +def check_transcription(vllm_root: Path) -> None: + """Fail if the transcription above no longer matches the pinned checkout.""" + path = vllm_root / "vllm/model_executor/layers/quantization/qutlass_utils.py" + if not path.is_file(): + raise SystemExit(f"not a vLLM checkout: {path} is missing") + text = path.read_text(encoding="utf-8") + if _TO_BLOCKED_SOURCE_ANCHOR not in text: + raise SystemExit( + f"{path}: the transcribed `to_blocked` body no longer appears verbatim.\n" + " The swizzle this port inverts has MOVED. Re-read it and re-transcribe; " + "do not relax this check." + ) + # The second writing of the same permutation, pinned so a divergence between + # vLLM's two producers cannot pass unnoticed either. + other = vllm_root / "vllm/model_executor/layers/quantization/utils/nvfp4_utils.py" + if not other.is_file(): + raise SystemExit(f"not a vLLM checkout: {other} is missing") + otext = other.read_text(encoding="utf-8") + for fragment in ( + "padded = padded.reshape(B, M_padded // 128, 4, 32, K_padded // 4, 4)", + "swizzled = padded.permute(0, 1, 4, 3, 2, 5).contiguous().cuda()", + ): + if fragment not in otext: + raise SystemExit( + f"{other}: `swizzle_blockscale` no longer contains {fragment!r}; " + "the two vLLM producers may have diverged. Re-read both." + ) + # LOW-nibble-first, vLLM's own reader. + emu = vllm_root / "vllm/model_executor/layers/quantization/utils/nvfp4_emulation_utils.py" + if not emu.is_file(): + raise SystemExit(f"not a vLLM checkout: {emu} is missing") + etext = emu.read_text(encoding="utf-8") + for fragment in _VLLM_LOW_NIBBLE_ANCHORS: + if fragment not in etext: + raise SystemExit( + f"{emu}: `break_fp4_bytes` no longer contains {fragment!r}.\n" + " That is the LOW-nibble-first convention DequantNvfp4ToBf16 defaults " + "to. If it moved, re-read it; do not relax this check." + ) + + +def check_ltx2_nibble_order(ltx2_root: Path) -> None: + """Fail if Lightricks' HIGH-first convention is no longer stated where we read it.""" + for rel, fragment in _LTX2_HIGH_NIBBLE_ANCHORS: + path = ltx2_root / rel + if not path.is_file(): + raise SystemExit(f"not an LTX-2 checkout: {path} is missing") + if fragment not in path.read_text(encoding="utf-8"): + raise SystemExit( + f"{path}: no longer contains {fragment!r}.\n" + " The first-party NVFP4 DiT is read HIGH-nibble-first on the strength of " + "that statement (.agents/specs/nvfp4-nibble-order.md section 1). Re-read " + "it; do not relax this check." + ) + + +def load_upstream_to_blocked(vllm_root: Path): + """Return vLLM's OWN `to_blocked`, executed from the pinned checkout. + + `import vllm.model_executor...` pulls in `vllm.distributed`, which dies on a + missing `zmq` on this host. qutlass_utils.py itself needs only torch, + `vllm.triton_utils` and `vllm.utils.math_utils`, so loading the FILE through + `spec_from_file_location` runs the real function without importing the + package graph around it. That is what makes this a RUNNING oracle rather + than a transcription of one. + """ + path = vllm_root / "vllm/model_executor/layers/quantization/qutlass_utils.py" + if not path.is_file(): + raise SystemExit(f"not a vLLM checkout: {path} is missing") + # `vllm.triton_utils` / `vllm.utils.math_utils` are imported by name from + # inside the file, so the checkout has to be importable as a package root. + if str(vllm_root) not in sys.path: + sys.path.insert(0, str(vllm_root)) + spec = importlib.util.spec_from_file_location("vllm_qutlass_utils_oracle", path) + if spec is None or spec.loader is None: + raise SystemExit(f"cannot load {path} as a module") + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + to_blocked = getattr(module, "to_blocked", None) + if to_blocked is None: + raise SystemExit(f"{path}: no `to_blocked` to execute; the producer has MOVED") + return to_blocked + + +# --------------------------------------------------------------------------- +# Deterministic byte stream, mirrored bit-for-bit by the C++ suite +# --------------------------------------------------------------------------- + +_MASK64 = (1 << 64) - 1 + + +def fnv1a64(data) -> int: + h = 0xCBF29CE484222325 + if isinstance(data, str): + data = data.encode("utf-8") + for byte in data: + h ^= byte + h = (h * 0x100000001B3) & _MASK64 + return h + + +def splitmix64(x: int) -> int: + x = (x + 0x9E3779B97F4A7C15) & _MASK64 + z = x + z = ((z ^ (z >> 30)) * 0xBF58476D1CE4E5B9) & _MASK64 + z = ((z ^ (z >> 27)) * 0x94D049BB133111EB) & _MASK64 + return z ^ (z >> 31) + + +def rand_bytes(name: str, count: int) -> np.ndarray: + """`count` bytes reproducible from `name` alone. The C++ suite rebuilds these.""" + seed = fnv1a64(name) + out = np.empty(count, dtype=np.uint8) + for i in range(count): + out[i] = (splitmix64((seed + i) & _MASK64) >> 24) & 0xFF + return out + + +# --------------------------------------------------------------------------- +# Safetensors header reading — no payload beyond the named byte ranges +# --------------------------------------------------------------------------- + + +def open_header(path: Path): + fh = path.open("rb") + length = struct.unpack(" bytes: + info = header[name] + begin, end = info["data_offsets"] + take = min(nbytes, end - begin) + fh.seek(base + begin) + raw = fh.read(take) + if len(raw) != take: + raise SystemExit(f"{name}: short read ({len(raw)} of {take})") + return raw + + +def f8e4m3_to_f32(raw: bytes) -> np.ndarray: + """Decode fp8-e4m3fn with TORCH — an implementation that is not ours.""" + t = torch.frombuffer(bytearray(raw), dtype=torch.uint8).view(torch.float8_e4m3fn) + return t.float().numpy() + + +_E2M1_LUT = np.array([0.0, 0.5, 1.0, 1.5, 2.0, 3.0, 4.0, 6.0], dtype=np.float32) + + +def e2m1_nibbles_to_f32(raw: bytes) -> np.ndarray: + """Low-nibble-first E2M1 decode, the LUT nvfp4_dequant.h:37-38 already gates.""" + b = np.frombuffer(raw, dtype=np.uint8) + nib = np.empty(b.size * 2, dtype=np.uint8) + nib[0::2] = b & 0x0F + nib[1::2] = b >> 4 + mag = _E2M1_LUT[nib & 0x7] + return np.where((nib & 0x8) != 0, -mag, mag).astype(np.float32) + + +# --------------------------------------------------------------------------- +# Emission +# --------------------------------------------------------------------------- + + +def cxx_f32(v: float) -> str: + # `%.9g` of an integral value emits `64`, and `64F` is not a float literal — + # it is an integer with an unknown user-defined suffix, which is a hard + # compile error rather than a silent narrowing. Force a decimal point. + text = f"{float(v):.9g}" + if not any(c in text for c in ".eE"): + text += ".0" + return text + "F" + + +def emit_bytes(out, name: str, data) -> None: + values = ", ".join(str(int(b)) for b in data) + out.write(f"inline constexpr uint8_t {name}[] = {{{values}}};\n") + out.write(f"inline constexpr int64_t {name}Count = {len(data)};\n\n") + + +def emit_f32(out, name: str, data) -> None: + values = ", ".join(cxx_f32(v) for v in data) + out.write(f"inline constexpr float {name}[] = {{{values}}};\n") + out.write(f"inline constexpr int64_t {name}Count = {len(data)};\n\n") + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--vllm", required=True, type=Path, help="the pinned vLLM checkout") + parser.add_argument("--ltx2", required=True, type=Path, + help="the Lightricks LTX-2 checkout (the HIGH-nibble authority)") + parser.add_argument("--checkpoint-root", required=True, type=Path) + parser.add_argument("--out", required=True, type=Path) + args = parser.parse_args() + + check_transcription(args.vllm) + check_ltx2_nibble_order(args.ltx2) + to_blocked = load_upstream_to_blocked(args.vllm) + vllm_sha, vllm_clean = pinned_revision(args.vllm, "vllm", _PINNED_VLLM_PATHS) + + ltx2_sha, ltx2_clean = pinned_revision(args.ltx2, "ltx2", _PINNED_LTX2_PATHS) + + root = args.checkpoint_root / "ltx-2.5/vonkaiser-fp8-nvfp4" + dit_path = root / "transformer/ltx-2.5-22b-distilled-fp8.safetensors" + te_path = root / "text_encoders/gemma4-12b-with-proj-nvfp4-torchao.safetensors" + nvfp4_dit_path = ( + args.checkpoint_root + / "ltx-2.5/lightricks-ltx-2.5/diffusion_models" + / "ltx-2.5-22b-distilled-transformer-nvfp4.safetensors" + ) + for p in (dit_path, te_path, nvfp4_dit_path): + if not p.is_file(): + raise SystemExit(f"missing shipped checkpoint: {p}") + + args.out.parent.mkdir(parents=True, exist_ok=True) + out = args.out.open("w", encoding="utf-8") + out.write( + "// GENERATED by scripts/gen-ltx2-quant-goldens.py — DO NOT EDIT BY HAND.\n" + "//\n" + "// LTX-2.5 phases L6 + L9a (.agents/specs/ltx-2-5.md, issue #435): the\n" + "// torchao-NVFP4 scale swizzle, the NIBBLE ORDER\n" + "// (.agents/specs/nvfp4-nibble-order.md), and the three shipped checkpoints'\n" + "// own bytes.\n" + "//\n" + f"// FP8 DiT {dit_path}\n" + f"// NVFP4 TE {te_path}\n" + f"// NVFP4 DiT {nvfp4_dit_path} ({nvfp4_dit_path.stat().st_size} bytes)\n" + "//\n" + f"// LTX-2 revision (HIGH-nibble-first authority): {ltx2_sha}" + + ("" if ltx2_clean else " [worktree not fully clean; the pinned files are]") + + "\n" + f"// vLLM revision (swizzle transcription pinned against it): {vllm_sha}\n" + + ( + "// Both pinned vLLM sources are clean at that revision; the rest of that\n" + "// checkout's worktree is NOT, so the anchor covers those two files only.\n" + if not vllm_clean + else "// That vLLM checkout was entirely clean at generation time.\n" + ) + + + "// The swizzle oracle RUNS: vLLM's own `to_blocked(x, backend=\"torch\")` is\n" + "// loaded out of the pinned checkout and called. Its `backend=\"triton\"` arm and\n" + "// `swizzle_blockscale` are not executed here (no driver / unconditional\n" + "// .cuda()); both are pinned by source. See the script.\n" + "//\n" + "// Regenerate (one line; a trailing backslash in a // comment is a\n" + "// -Werror=comment line continuation):\n" + "// python3 scripts/gen-ltx2-quant-goldens.py --vllm --ltx2 " + " --checkpoint-root --out tests/vllm/models/ltx2_quant_goldens.inc\n" + "#pragma once\n\n#include \n\nnamespace vllm_test {\n\n" + ) + out.write(f'inline constexpr const char* kLtx2QuantVllmRevision = "{vllm_sha}";\n\n') + + # --- section 1: the swizzle, over shapes that exercise every tile boundary - + cases = [(128, 4), (128, 16), (256, 4), (384, 12), (128, 240)] + out.write(f"inline constexpr int64_t kLtx2BlockedCaseCount = {len(cases)};\n") + out.write("struct Ltx2BlockedCase { int64_t rows, cols; };\n") + out.write("inline constexpr Ltx2BlockedCase kLtx2BlockedCases[] = {\n") + for rows, cols in cases: + out.write(f" {{{rows}, {cols}}},\n") + out.write("};\n\n") + for idx, (rows, cols) in enumerate(cases): + src = rand_bytes(f"blocked.{rows}x{cols}", rows * cols) + t = torch.from_numpy(src.reshape(rows, cols).copy()) + blocked = to_blocked(t, backend="torch").numpy() + assert blocked.size == rows * cols, (blocked.size, rows * cols) + emit_bytes(out, f"kLtx2BlockedLinear{idx}", src) + emit_bytes(out, f"kLtx2BlockedSwizzled{idx}", blocked) + + # --- section 2: the SHIPPED text encoder's own swizzled scale --------------- + # + # The first 512 bytes of a swizzled buffer are exactly the (rt=0, ct=0) tile, + # i.e. logical rows 0..127 x cols 0..3 — self-contained, so a 512-byte read + # pins the real layout without materializing a 7.4 GB file. + te_fh, te_hdr, te_base = open_header(te_path) + te_module = "text_embedding_projection.video_aggregate_embed" + scale_name = te_module + ".weight_scale" + scale_info = te_hdr[scale_name] + real_tile = read_slice(te_fh, te_hdr, te_base, scale_name, 512) + out.write( + "// The (row-tile 0, col-tile 0) block of the SHIPPED text encoder's\n" + f"// `{scale_name}` — logical rows 0..127, cols 0..3.\n" + f"// Stored shape {scale_info['shape']} (= [out/4, (in/16)*4]).\n" + ) + emit_bytes(out, "kLtx2RealTeScaleTileSwizzled", real_tile) + # Unswizzle by inverting the oracle rather than by re-deriving it: push an + # index map (int32, because a 512-entry arange does not fit in uint8) through + # UPSTREAM's own `to_blocked` to get linear->swizzled, then scatter through + # it. Writing the permute chain out a second time here would be the same + # hand-typing the transcription used to be. + idx = torch.arange(128 * 4, dtype=torch.int32).reshape(128, 4) + fwd = to_blocked(idx, backend="torch").numpy() + linear_tile = np.empty(128 * 4, dtype=np.uint8) + linear_tile[fwd] = np.frombuffer(real_tile, dtype=np.uint8) + emit_bytes(out, "kLtx2RealTeScaleTileLinear", linear_tile) + out.write( + "// Decoded with TORCH's own fp8-e4m3fn, so the byte->value half of this\n" + "// gate is not our implementation checking itself.\n" + ) + emit_f32(out, "kLtx2RealTeScaleTileLinearF32", f8e4m3_to_f32(bytes(linear_tile))) + + scale2_raw = read_slice(te_fh, te_hdr, te_base, te_module + ".weight_scale_2", 4) + out.write( + f"inline constexpr float kLtx2RealTeScale2 = " + f"{cxx_f32(struct.unpack(' 128 packed bytes, 16 groups + out.write( + "// ---------------------------------------------------------------------\n" + "// Phase L9a: the FIRST-PARTY NVFP4 DiT vs the FP8 DiT as an ORACLE\n" + f"// {nvfp4_dit_path.name}\n" + f"// module {n4_module}\n" + f"// control {n4_control}\n" + f"// logical [{n4_out}, {n4_in}]; weight_scale stored " + f"{n4_hdr[n4_module + '.weight_scale']['shape']} = the cuBLAS-PADDED framing\n" + f"// {padded_framing}. torchao's to_blocked framing would be " + f"{to_blocked_framing};\n" + "// both dress the SAME bytes. This file carries NO torchao_nvfp4 marker.\n" + "//\n" + "// The first 2048 scale bytes are a valid standalone SWIZZLED [128, 16] grid\n" + "// (the row-tile index is 0 below row 128, so n_col_blocks drops out).\n" + f"// Rows {n4_rows} span all four 32-row quarters of that tile.\n" + ) + # The module's REAL geometry and the shape it REALLY declares, so the C++ gate + # can run the producer resolver on the artifact's own numbers rather than on a + # geometry invented in the test. + out.write(f"inline constexpr int64_t kLtx2RealDitNvfp4OutFeatures = {n4_out};\n") + out.write(f"inline constexpr int64_t kLtx2RealDitNvfp4InFeatures = {n4_in};\n") + out.write( + f"inline constexpr int64_t kLtx2RealDitNvfp4DeclaredScaleShape[] = " + f"{{{n4_hdr[n4_module + '.weight_scale']['shape'][0]}, " + f"{n4_hdr[n4_module + '.weight_scale']['shape'][1]}}};\n" + ) + out.write( + f'inline constexpr const char* kLtx2RealDitNvfp4Module =\n "{n4_module}";\n' + ) + out.write(f"inline constexpr int64_t kLtx2RealDitNvfp4ScaleTileRows = 128;\n") + out.write(f"inline constexpr int64_t kLtx2RealDitNvfp4ScaleTileCols = 16;\n") + out.write(f"inline constexpr int64_t kLtx2RealDitNvfp4RowElems = {n4_elems};\n") + out.write( + f"inline constexpr int64_t kLtx2RealDitNvfp4Rows[] = " + f"{{{', '.join(str(r) for r in n4_rows)}}};\n" + f"inline constexpr int64_t kLtx2RealDitNvfp4RowCount = {len(n4_rows)};\n" + ) + out.write(f"inline constexpr float kLtx2RealDitNvfp4Scale2 = {cxx_f32(n4_scale2)};\n\n") + emit_bytes(out, "kLtx2RealDitNvfp4ScaleTile", n4_scale_tile) + + # The packed NVFP4 weight rows, and the FP8 oracle's same rows. + def packed_rows(fh, hdr, base, module, rows, nbytes, stride): + begin = hdr[module + ".weight"]["data_offsets"][0] + acc = bytearray() + for r in rows: + fh.seek(base + begin + r * stride) + chunk = fh.read(nbytes) + if len(chunk) != nbytes: + raise SystemExit(f"{module} row {r}: short read") + acc += chunk + return bytes(acc) + + n4_packed = packed_rows(n4_fh, n4_hdr, n4_base, n4_module, n4_rows, + n4_elems // 2, n4_packed_in) + emit_bytes(out, "kLtx2RealDitNvfp4Packed", n4_packed) + n4_fh.close() + + # The ORACLE: the same rows of the same module out of the FP8 checkpoint, + # decoded by TORCH. Emitted as f32 VALUES, per section 7.0's "emit values". + dit_fh2, dit_hdr2, dit_base2 = open_header(dit_path) + for label, module in (("Oracle", n4_module), ("Control", n4_control)): + fw_shape = dit_hdr2[module + ".weight"]["shape"] + fw_scale = struct.unpack( + "