feat(ltx-2.5): LTX-2.5 joint video+audio DiT, and a video seam that is no longer MiniMax-only (#435) (#641)

Ports Lightricks LTX-2.5 -- a 21.00B joint video+audio flow-matching DiT, 48
blocks, two coupled streams (video 4096 / audio 2048) joined by explicit
audio<->video cross-attention, per-head gated attention, gelu-approximate FFN --
and generalizes the diffusion lane so a second model was ADDITIVE rather than a
fork. MiniMax-H3 moves behind the new `vllm::multimodal::VideoEngine` seam with
its fold gate holding it to the same golden bytes.

WHAT RUNS. The DiT and both VAEs including their ENCODERS and the mel front-end;
the pipeline layer (sigma schedule, guidance, latent spatial x2 upsampler,
duration head, embeddings connector); the quantized loaders, with FP8, torchao
NVFP4 and the first-party NVFP4 file all loading; the Gemma-4 12B text tower with
its embedded tokenizer, token-exact against HuggingFace on the shipped
262144-entry vocab; and a typed prompt reaching cross-attention through the
connector. Reachable as video family `ltx-2.5` through the C ABI (v18, 36
exports), the `ltx2-gen` example and the server's /v1/videos, which registers
family-agnostically.

WHAT IS OWED, stated because a reader will otherwise over-read the above. The
320x192/25f render IS a coherent scene but was REGISTER-conditioned -- synthetic
N(0, 0.2) input with 104 of 128 rows the connector's own trained
learnable_registers -- so it is the checkpoint's learned default reached through
the real connector, not a depiction of a prompt. The typed-prompt path is gated
on a reduced-dimension synthetic fixture; a real-checkpoint prompted render has
never been run, in either direction. The conditioning composition has NO value
oracle: scaling it x1.5 or reversing its rows still passes every assertion, so
`last_conditioning()` is a change detector and says so. The shipped NVFP4 tower's
output is never compared to an independent oracle. Speed is structurally PENDING:
vLLM-Omni carries no native 2.5 and its diffusers adapter is a black box, so no
production-configuration denominator exists.

ALSO FIXED, found while building the above. The shared scratch pool was
DEVICE-BLIND: one process-wide free list held blocks from every backend, so
CUDA->CPU was a SIGSEGV and CPU->CUDA was SILENT -- a uniform 0x7fff0000 all-NaN
no token gate can see (#516, and #486 is the same bug). A DevicePool is now bound
to one backend at construction and the no-argument accessors are REMOVED, which
is what caught a live instance of the same defect landing in this very campaign.

VERIFIED. Local: clean from-scratch Release build, zero warnings, 423/423 with
the run's own denominator asserted against `ctest -N`. dgx.casa (GB10, sm_121a
confirmed by cuobjdump on the binary, every fast-path arm enabled): 453 of 453
tests RAN under one `flock $HOME/gpu.lock` held 14:36:09Z-15:29:37Z; 9 failures,
all nine drawn from the pre-existing set tracked as #233/#248/#305/#614, no new
failure; test_minimax_h3 Passed 19.02s where it previously SEGFAULTed, and
test_device_pool and all seven LTX suites pass.

Issue: #435, #516, #486

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [Claude Code]

Author: localai-bot

.agents/benchmark-record.md

@@ -0,0 +1,5 @@
+# CLAIM-LTX25-L0-L7
+
+| Claim | Row IDs | Agent | Worktree / remote dir | Branch | Owned scope | State | Last update |
+|---|---|---|---|---|---|---|---|
+| `CLAIM-LTX25-L0-L7` | `MODEL-DIFFUSION-ltx-2-5-ltx2-video-transformer-3d-model` (`SPIKE`), `ROAD-V1-LTX25` | Claude Code (opus-5), operator role | isolated worktree `/home/mudler/_git/vllm.cpp-ltx25`; GPU work on dgx.casa serialized through `flock $HOME/gpu.lock` | `row/MODEL-DIFFUSION-LTX25`, issue [#435](https://github.com/mudler/vllm.cpp/issues/435) | Owns: NEW `.agents/specs/ltx-2-5.md`; the LTX-2.5 rows in `.agents/roadmap_v1.md` and `.agents/model-matrix.md`; the LTX-2.5 rows in `docs/{STATUS,BENCHMARKS,FEATURES}.md`; NEW `src/vllm/model_executor/models/ltx2_*` and `include/vllm/model_executor/models/ltx2*`; NEW `vllm::multimodal::VideoEngine` seam plus the MiniMax-H3 move behind it (BEHAVIOUR-PRESERVING only); additive ABI v13 fields in `include/vllm.h`. EXCLUDES: any change to H3's numerics or its checkpoint handling, the v12 ABI contract, DiffVAE / `NADiffusionDecoder`, the temporal x2 upsampler, LoRA fusion, multishot, `int8-convrot`, and multi-GPU / CFG parallelism, all of which are recorded as owed in the spec rather than silently dropped | `ACTIVE` | 2026-08-13 — **MERGE AUTHORITY RECORDED**: developer-directed "keep going autonomously until you get it working, fix the bugs you find along the way and merge the PRs directly once everything is proved". Scope widened beyond L0-L7 to a working END-TO-END LTX-2.5 render: L9a NVFP4 linear scale layout, L9b real render + `--video-family`, L10 the Gemma-4 tower so a prompt works, L11 the VAE encoders for image/video/audio conditioning, and the #516 pool fix that a trustworthy render depends on. Merge is gated on PROVED, which for this campaign means: fresh review PASS, the operator's own gate rerun, and no red bought by weakening a detector. |

.agents/claims/CLAIM-LTX25-L0-L7.md

@@ -780,6 +780,36 @@ add_library(vllm STATIC
   src/vllm/model_executor/models/minimax_h3_encoder_sharded.cpp
   src/vllm/model_executor/models/minimax_h3_encoder_device.cpp
   src/vllm/model_executor/models/minimax_h3_vision_gguf.cpp
+  # LTX-2.5 (row MODEL-DIFFUSION-LTX25, .agents/specs/ltx-2-5.md phase L2).
+  src/vllm/model_executor/models/ltx2.cpp
+  src/vllm/model_executor/models/ltx2_dit.cpp
+  # LTX-2.5 (row MODEL-DIFFUSION-LTX25, phase L8): the DEVICE-RESIDENT forward and
+  # the (kLtx2, device) resolver. The resolver is its own TU so the vt::GetOp cast
+  # links in CPU-only builds, exactly as minimax_h3_device_resolve.cpp is.
+  src/vllm/model_executor/models/ltx2_device_resolve.cpp
+  src/vllm/model_executor/models/ltx2_device.cpp
+  # LTX-2.5 (ROW MODEL-DIFFUSION-LTX25, phase L4): the Conv video VAE, the audio
+  # VAE and its vocoder. Additive files mirroring ltx_core's own structure.
+  src/vllm/model_executor/models/ltx2_audio_vae.cpp
+  src/vllm/model_executor/models/ltx2_video_vae.cpp
+  # LTX-2.5 (ROW MODEL-DIFFUSION-LTX25, phase L3): the Gemma-4 multi-layer feature
+  # aggregation, the two caption projections and the embedded asset pack. An
+  # additive file mirroring ltx_core's own structure.
+  src/vllm/model_executor/models/ltx2_text_encoder.cpp
+  # LTX-2.5 (ROW MODEL-DIFFUSION-LTX25, phase L5): the flow-matching pipeline,
+  # the latent spatial upsampler, the duration head and the embeddings
+  # connector. Additive files mirroring ltx_core's own structure.
+  src/vllm/model_executor/models/ltx2_pipeline.cpp
+  src/vllm/model_executor/models/ltx2_upsampler.cpp
+  src/vllm/model_executor/models/ltx2_duration_head.cpp
+  src/vllm/model_executor/models/ltx2_connector.cpp
+  # LTX-2.5 (ROW MODEL-DIFFUSION-LTX25, phase L11): the conditioning items that
+  # place the VAE ENCODERS' output into the denoise state.
+  src/vllm/model_executor/models/ltx2_conditioning.cpp
+  # LTX-2.5 (ROW MODEL-DIFFUSION-LTX25, phase L6): the quantized loaders — the
+  # FP8 and torchao-NVFP4 DiT arms, the torchao-NVFP4 text encoder, and the
+  # load-time device staging GB10's ATS penalty makes the default.
+  src/vllm/model_executor/models/ltx2_loader.cpp
   src/vllm/entrypoints/openai/video_api.cpp
   src/vllm/model_executor/models/kimi_kda.cpp
   src/vllm/model_executor/models/kimi_linear_registry.cpp
@@ -883,6 +913,15 @@ add_library(vllm STATIC
   # server's private /v1/videos twin. Builds artifacts + the ffmpeg argv and
   # SPAWNS NOTHING (the ratified process boundary).
   src/vllm/multimodal/minimax_h3_video.cpp
+  # The GENERALIZED video seam (vllm::multimodal::VideoEngine) + its
+  # checkpoint-detected family registry — LTX-2.5 L1, .agents/specs/ltx-2-5.md
+  # §5. Knows about no family; each family self-registers from its own TU.
+  src/vllm/multimodal/video_engine.cpp
+  # LTX-2.5 behind that seam — the SECOND family, and the driving loop that
+  # composes the L2-L6 bricks into frames + a waveform (phase L7). Additive:
+  # one file plus one REGISTER_VLLM_VIDEO_FAMILY line, which is the property
+  # the seam exists to have.
+  src/vllm/multimodal/ltx2_video.cpp
   src/vllm/v1/core/encoder_cache_manager.cpp
   src/vllm/tokenizer/unicode_data.cpp
   src/vllm/tokenizer/pretokenizer.cpp
@@ -1081,6 +1120,7 @@ add_library(vllm STATIC
   src/vt/cpu/cpu_threadpool.cpp
   src/vt/cpu/cpu_ops.cpp
   src/vt/cpu/cpu_minimax_h3.cpp
+  src/vt/cpu/cpu_ltx2.cpp
   src/vt/cpu/cpu_isa_arm.cpp
   src/vt/cpu/cpu_isa_x86.cpp
   src/vt/cpu/cpu_matmul_elem.cpp
@@ -1487,7 +1527,9 @@ if(VLLM_CPP_CUDA)
     src/vt/cuda/cuda_sample.cu
     src/vt/cuda/cuda_deepseek_v4.cu
     src/vt/cuda/cuda_laguna.cu
-    src/vt/cuda/cuda_minimax_h3.cu)
+    src/vt/cuda/cuda_minimax_h3.cu
+    src/vt/cuda/cuda_ltx2.cu
+    src/vt/cuda/cuda_attention_cross.cu)
   find_package(CUDAToolkit REQUIRED)
   # cublasLt is linked now so the Task 4 matmul lands without a build change.
   target_link_libraries(vllm PUBLIC CUDA::cudart CUDA::cublasLt)

.agents/model-matrix.md

@@ -186,7 +186,7 @@ configs, token-for-token the same output. Switching to it should be boring. Ever
 you get on top, most of it borrowed from whichever engine does it best:
 
 - **One 66 MiB binary instead of a 9.1 GiB install.** A flat, exception-free, llama.cpp-style C ABI
-  ([`include/vllm.h`](include/vllm.h), ABI v17, 35 functions) for C, C++, Go, or Rust. No Python
+  ([`include/vllm.h`](include/vllm.h), ABI v18, 36 functions) for C, C++, Go, or Rust. No Python
   interpreter in the process.
 - **GGUF as a first-class citizen.** Load the same quantized files llama.cpp uses, and on CPU
   **compute directly on the compressed blocks** (Q4_0/Q8_0/Q3_K/Q4_K/Q5_K/Q6_K) with no BF16
@@ -385,7 +385,7 @@ behind a model gallery, multi-model serving, the full OpenAI API surface, auth,
 ## Use it as a library (C API)
 
 Link `libvllm` and include [`include/vllm.h`](include/vllm.h): a flat, exception-free,
-llama.cpp-style C ABI (`VLLM_ABI_VERSION 17`, 35 exported functions) suitable for `dlopen` / FFI.
+llama.cpp-style C ABI (`VLLM_ABI_VERSION 18`, 36 exported functions) suitable for `dlopen` / FFI.
 
 ```c
 vllm_model_params mp = vllm_model_params_default();