From 3c6209d9be45dc05ee7a599f1d60d22f890565c6 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Mon, 17 Aug 2026 12:03:55 +0000 Subject: [PATCH] feat(LTX25-RES2S-LOOP): the res_2s loop, its second evaluation, and the guidance the naive merge would have dropped (#921) `pipeline_kind=res2s_two_stage` serves upstream's `TI2VidTwoStagesHQPipeline` -- the high-quality arm. New TU `ltx2_samplers.{h,cpp}`, mirroring upstream's own `utils/` partition: a stepper advances a substep, a sampler decides how many there are. Ported at `fd4ded7f`: `Ltx2Phi` (`utils/res2s.py:4-22`), `Ltx2GetRes2sCoefficients` + `Ltx2PhiCache` (`:25-62`), `Ltx2Res2sNormalizeNoise` (`utils/samplers.py:160-170`), `Ltx2Res2sDenoisingLoop` (`:208-447`), and `Res2sTwoStageRecipe` (`ti2vid_two_stages_hq.py:59-340` + `utils/constants.py:95-115`). THE SAMPLER IS THE HQ VARIANT. Serving the HQ preset on the Euler loop renders a plausible clip that is quietly not HQ, at roughly half the model evaluations the preset was tuned for, and no shape or token gate can see it. The gate is therefore an exact DiT-evaluation count -- `2n+1` when the schedule ends at 0, else `2n` -- asserted with the eval-sigma sequence beside it, so two forwards at the same sigma also fails. End to end: 7 and 11 evaluations at 3 and 5 steps on `res2s_two_stage`, against 3 and 5 on `one_stage`. `phi` is a CANCELLATION CLIFF, not a series expansion. Upstream guards only `abs(z) < 1e-10` and otherwise evaluates the quotient directly, so its own `phi(2,-1e-10)` is 0.0 and `phi(2,-1e-8)` is 1.1102230246251563. A Taylor expansion near zero -- the numerically BETTER port -- returns 0.5 and diverges from what the model actually ran. Pinned with `==`. GUIDANCE, which the merge nearly dropped. `daeff67f2` (#1092) landed the guided video denoiser into the same phase loop. A textual resolution keeping `Evaluate` as a bare `Ltx2DitForward` would have made the HQ preset the only unguided video arm in the tree -- upstream's stage 1 runs a `GuidedDenoiser` at cfg 3.0/7.0 and rescale 0.45 (`ti2vid_two_stages_hq.py:271-281`) -- and the evaluation count cannot see it, because a denoiser call is one evaluation guided or not. So `Evaluate` builds the `Ltx2X0Model` lambda and calls `Ltx2GuidedDenoise`, and a SECOND counter was added: `dit_forwards` counts actual `Ltx2DitForward` calls and is `3 * (2n+1)` on HQ stage 1 (cond + uncond + modality), asserted exactly at two step counts with `forwards != evaluations` as its own assertion. Stripping guidance from the res_2s arm alone is RED. `step_index` mirrors upstream literally: `step_idx` at the first evaluation (`samplers.py:301`), a literal 0 at the substep beside its one-element schedule (`:385`), `n_full_steps` at the terminal one (`:437`). Since `should_skip_step` is `step % (skip_step + 1) != 0`, the literal 0 makes the substep unskippable at any `skip_step` -- inert on the HQ preset's own `skip_step = 0`, live for a request override. A mutation survivor the review did not list: the substep's x0 conversion must use the latent THAT EVALUATION was handed, because the substep runs over `x_mid`. Reading the stream latent moves the whole substep prediction and nothing could see it, since the loop's arithmetic is gated with a fixture denoiser that performs no conversion. Now gated and RED. The engine's `VT_CHECK` was a tautology -- both operands came from the same `stats` object and `2n+1 > n` holds for every n -- and is now the trace delta against `stats.evaluations`. The argument is executable: the same under-counting defect beside the restored old check is GREEN. Both generator scripts are committed rather than described. `scripts/gen-ltx2-res2s-goldens.py` imports upstream's own `phi`, `get_res2s_coefficients`, `Res2sDiffusionStep`, `post_process_latent`, `_channelwise_normalize` and `res2s_audio_video_denoising_loop` at the pin and reproduces `ltx2_res2s_goldens.inc` byte for byte -- which is the evidence the goldens are upstream's and not this port's. Supersedes #1101, whose branch carried a merge commit with a bare subject and no trailer block. `check-commit-trailers.py` walks merges, that commit was a first-parent ancestor of every candidate head, and repairing it would have needed a force-push. Same tree, one commit, block intact. Owed, not claimed: no render on real weights, and no oracle-run comparison -- everything is gated against upstream SOURCE at `fd4ded7f`. The HQ preset is host-only here, because its `modality_scale = 3.0` asks for the isolated-modality pass and `Ltx2DitForwardDevice` takes no perturbations; that is #1092's owed device work, not newly incurred. Closes #921. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/specs/ltx25-res2s-loop.md | 843 ++++++++++++++++++ .agents/specs/ltx25-resolution-envelope.md | 6 + CMakeLists.txt | 7 + docs/FEATURES.md | 1 + docs/USAGE.md | 59 +- .../model_executor/models/ltx2_pipeline.h | 68 +- .../model_executor/models/ltx2_samplers.h | 319 +++++++ include/vllm/multimodal/ltx2_video.h | 94 ++ scripts/gen-ltx2-res2s-goldens.py | 410 +++++++++ scripts/mutation-harness.py | 245 +++++ .../model_executor/models/ltx2_pipeline.cpp | 249 +++++- .../model_executor/models/ltx2_samplers.cpp | 407 +++++++++ src/vllm/multimodal/ltx2_video.cpp | 371 ++++++-- tests/vllm/models/ltx2_res2s_goldens.inc | 203 +++++ tests/vllm/models/test_ltx2_pipeline.cpp | 689 +++++++++++++- tests/vllm/multimodal/test_ltx2_video.cpp | 307 +++++++ 16 files changed, 4171 insertions(+), 107 deletions(-) create mode 100644 .agents/specs/ltx25-res2s-loop.md create mode 100644 include/vllm/model_executor/models/ltx2_samplers.h create mode 100644 scripts/gen-ltx2-res2s-goldens.py create mode 100644 scripts/mutation-harness.py create mode 100644 src/vllm/model_executor/models/ltx2_samplers.cpp create mode 100644 tests/vllm/models/ltx2_res2s_goldens.inc diff --git a/.agents/specs/ltx25-res2s-loop.md b/.agents/specs/ltx25-res2s-loop.md new file mode 100644 index 000000000..307973d05 --- /dev/null +++ b/.agents/specs/ltx25-res2s-loop.md @@ -0,0 +1,843 @@ +# LTX-2.5 — the `res_2s` denoising loop, and the second evaluation no shape check can see + +Row: `LTX25-RES2S-LOOP`. Campaign: [`ltx-2-5.md`](ltx-2-5.md) (operator-owned; +**not edited by this row**). Issue: +[#921](https://github.com/mudler/vllm.cpp/issues/921). Parent campaign issue: +[#644](https://github.com/mudler/vllm.cpp/issues/644). Previous owner: +[`ltx25-resolution-envelope.md`](ltx25-resolution-envelope.md) `## Owed`, which +mirrored the geometry half and explicitly did not take the sampler. + +Upstream pin: + +| Reference | Revision | +|---|---| +| Lightricks/LTX-2 (`packages/ltx-core`, `packages/ltx-pipelines`) | `fd4ded7f2d88d3da713abcdd4ad41ecc4a9314ca` | + +Verified at the local checkout `/home/mudler/_git/LTX-2` before any anchor below +was taken: `git rev-parse HEAD` = `fd4ded7f2d88d3da713abcdd4ad41ecc4a9314ca`, +`git status --short` empty. Base: `origin/main` at `b5756ea8c`. + +vLLM implements nothing in this class. `vllm-project/vllm-omni` stops at LTX-2.3 +and carries no `res_2s` sampler at all, so Lightricks is the reference for this +row under AGENTS.md `## When vLLM has no implementation`, recorded in +[`.agents/oracles/`](../oracles/). + +--- + +## 0. Honesty statement + +**What lands.** The `res_2s` second-order sampler — `phi`, the RK coefficients, +the two transformer evaluations per step, the bong anchor refinement, both SDE +injections, and the terminal step — plus a `res2s_two_stage` recipe that reaches +it from `pipeline_kind`, which is a LOAD knob and therefore reaches `ltx2-gen`, +the C ABI and the server on their default configurations. + +**What is measured and what is not.** Every numeric claim below is gated against +**upstream's own executing code**, not against a restatement of it: `res2s.py`, +`samplers.py`, `diffusion_steps.py` and `helpers.post_process_latent` are +imported from the checkout at the pin and run, and their outputs are the goldens. +Only three things are substituted, and each is one this port reproduces exactly: +the denoiser (a fixed quadratic), the noise **draw** (`torch.randn`, whose stream +this port does not have — see §4.3), and two media-IO modules (`av`, +`OpenImageIO`) that the import chain pulls in and nothing numeric touches. + +**No render on real weights is claimed.** `dgx.casa` is contended and OOM-reboots +under a second job; a sampler is exactly the thing CPU goldens gate well. The +real-checkpoint HQ render is owed and named under `## Owed`. + +**Upstream ships no tests at this pin.** `find /home/mudler/_git/LTX-2 -name +'test_*.py'` returns 0 lines. "Port the upstream tests in the same change" has +nothing to port, so §5 does the stronger thing available: run upstream's own +functions and pin their output. + +--- + +## 1. What upstream does, with anchors on both sides + +### 1.1 The pipeline, and what makes it HQ + +**This section said "exactly three things" until 2026-08-17 and the count was +wrong.** It was load bearing, because it was the argument for what this row had +to port, and it is corrected here by *diffing the two files* at the pin rather +than by adding to a list. `diff ti2vid_two_stages.py ti2vid_two_stages_hq.py` +at `fd4ded7f` shows at least seven differences. Three are the ones this row +took: + +1. `stepper = Res2sDiffusionStep()` (`ti2vid_two_stages_hq.py:258`), passed to + both stages (`:285`, `:319`). +2. `loop = res2s_audio_video_denoising_loop`, passed to **both** stages + (`:292`, `:335`). +3. `LTX_2_3_HQ_PARAMS` (`utils/constants.py:95-115`): 15 steps, stage 1 at + `1088 // 2` x `1920 // 2`, STG off on both modalities, video rescale 0.45, + audio rescale 1.0, cfg 3.0 / 7.0, modality 3.0. + +And these are the others, each with its disposition: + +4. **Stage 1 loads the distilled LoRA at `distilled_lora_strength_stage_1`** + (`:92-101`, `:151-154`) where the plain pipeline loads none on that stage + (`ti2vid_two_stages.py:140`). The distilled LoRA is out of scope for every + LTX row here and is already named under §2 Out; the substance is unchanged, + the count was not. +5. **Stage 1's schedule is `execute(latent=empty_latent, steps=...)`** + (`:260-267`) against the plain pipeline's `execute(steps=...)`. + `schedulers.py:32` is `tokens = math.prod(latent.shape[2:]) if latent is not + None else default_number_of_tokens`, and `default_number_of_tokens` is 4096, + so this is a **resolution-dependent shift** rather than a fixed one. This + port's engine always derives from `target_tokens`, so stage 1 **coincides** + with upstream here. The divergence, if any, is on the PLAIN two-stage arm and + is not this row's to move; recorded so the next reader does not read the + coincidence as a design. +6. **`GuidedDenoiser` (`:271-281`) replaces `FactoryGuidedDenoiser`.** Both + reduce to `_guided_denoise` (`utils/denoisers.py:61-211`) — the difference is + whether the guider params are constant or built per sigma — so the port takes + the same seam either way. This is the item §3.5 below is about. +7. `hq_2_stage_arg_parser` replaces `default_2_stage_arg_parser` + + `resolve_cli_params`, and the guider parameters are typed + `MultiModalGuiderParams` rather than `... | MultiModalGuiderFactory`. A CLI + surface, not a behaviour. + +Stage 1 runs at `width // 2, height // 2` (`:238-243`) under a `GuidedDenoiser` +with a negative context (`:271-281`); stage 2 runs at full resolution under a +`SimpleDenoiser` (`:316`) with `stage_2_sigmas` defaulting to +`STAGE_2_DISTILLED_SIGMAS` (`:193`) and re-noising to `stage_2_sigmas[0]` +(`:327`, `:332`), after a spatial upsample of the stage-1 latent (`:297`). + +### 1.2 The loop runs on its own defaults, and that is a finding + +`DiffusionStage.__call__` calls the loop with **six** keyword arguments and no +others — `sigmas`, `video_state`, `audio_state`, `stepper`, `transformer`, +`denoiser` (`utils/blocks.py:566-573`). Nothing in the HQ pipeline overrides any +other parameter. So every remaining knob of +`res2s_audio_video_denoising_loop` (`samplers.py:208-223`) takes its declared +default on the shipped arm: + +| Parameter | Value on the HQ arm | Anchor | +|---|---|---| +| `noise_seed` | `-1` | `samplers.py:215` | +| `noise_seed_substep` | `None` -> `noise_seed + 10000` = `9999` | `samplers.py:216`, `:265-266` | +| `eta` | `0.5` (step level); substeps are **always** `0.5` | `samplers.py:217`, `:274` | +| `bongmath` | `True` | `samplers.py:218` | +| `bongmath_max_iter` | `100` | `samplers.py:219` | +| `new_noise_fn` | `_get_new_noise` (normalized), **not** `_get_plain_noise` | `samplers.py:220`, `:164-170` | +| `model_dtype` | `torch.bfloat16` | `samplers.py:221` | + +**`model_dtype` is `torch.bfloat16` upstream and f32 here**, so this loop stores +its latent at twice upstream's width. That is a port-wide pre-existing choice +rather than this row's — every LTX-2.5 host path in this tree is f32 — and it is +named because `AGENTS.md` "Inherit vLLM defaults" says a dtype that is too WIDE +is invisible to every correctness gate this project owns. Stated at the code in +`ltx2_samplers.h` as well, not only here. +| `legacy_mode` | `True` | `samplers.py:222` | + +Two of these are load bearing and would be easy to get wrong by analogy with the +already-ported ancestral arm: + +**The SDE noise does not depend on the request seed.** `noise_seed = -1` is a +constant, not the pipeline's `seed`. The initial latent still depends on the +seed through the noiser; the loop's own injections do not. The ancestral arm +does the opposite (`distilled.py:69-73` derives its seed from the pipeline's), +so mirroring by analogy would have been wrong. Mirrored here by giving the loop +upstream's own default parameters and having the engine call it the way +`DiffusionStage.__call__` does. + +**The noise is normalized.** `euler_ancestral_denoising_loop` defaults +`new_noise_fn=_get_plain_noise` (`samplers.py:574`), a bare `torch.randn`. The +`res_2s` loop defaults to `_get_new_noise` (`samplers.py:220`), which draws in +`highest_precision_float` and then applies `(n - n.mean()) / n.std()` followed by +`_channelwise_normalize` (`samplers.py:164-170`, `:160-161`). Two loops, two noise +functions, at two adjacent lines in one file. + +**`legacy_mode=True` means the timestep conversion does NOT happen.** +`_inject_sde_noise` (`samplers.py:173-205`) converts sigmas through +`timesteps_from_mask` only when `legacy_mode` is false (`:188-192`); on the HQ +arm it hands the stepper the raw schedule and applies `post_process_latent` +afterwards (`:202-203`). + +### 1.3 `phi` is a cancellation cliff, not a series expansion + +`phi(j, neg_h)` (`res2s.py:4-22`) returns `1 / j!` when `abs(neg_h) < 1e-10`, and +otherwise evaluates `(exp(z) - sum_{k 0.03` + (`:357-364`): `bongmath_max_iter` unconditional iterations of + `x_anchor = x_mid - h * a21 * eps_1; eps_1 = denoised_1 - x_anchor`. + There is no early exit. Both `x_anchor` and `eps_1` are carried forward. +10. **Evaluation 2** at `sigmas = [sub_sigma]`, `step_index = 0`, over the + mid-state cast to `model_dtype` (`:369-386`), then `post_process_latent` + (`:389-392`). +11. `eps_2 = denoised_2 - x_anchor`; + `x_next = x_anchor + h * (b1 * eps_1 + b2 * eps_2)` (`:397-407`). +12. **Step SDE injection** at `eta`, with the loop's own **float32** schedule and + `step_idx = i` (`:412-427`). +13. `state.latent = x_next.to(model_dtype)` (`:430-433`). + +Then, when `sigmas[-1] == 0`, one final evaluation at index `n_full_steps` — +which is the injected `0.0011` — whose `post_process_latent`'d prediction becomes +the state outright (`:436-445`). + +**The DiT evaluation count is therefore exactly `2 * n_full_steps + 1` when the +caller's schedule ends at 0, and `2 * n_full_steps` when it does not.** Measured +against upstream: a 5-sigma schedule ending at 0 gives 9; a 4-sigma schedule not +ending at 0 gives 6; a 2-sigma schedule ending at 0 gives 3. The already-shipped +Euler arm gives `n_full_steps` and `n_full_steps` — **half**. This count is the +discriminator this whole row rests on, because no shape check, no frame count and +no rendered clip can tell the two samplers apart. + +### 1.5 The precision split is upstream's, at two levels + +The loop works in `hp` = float64 on CPU (`samplers.py:262`, "float64 on CUDA/CPU +for ODE numerical stability"), and writes back to `model_dtype`. The +already-ported ancestral loop does the opposite and steps in **float32** +(`samplers.py:550` floats the SAMPLE; the denoised operand was already floated +at `:484`, so a reader looking for two `.float()` calls at `:550-551` finds +one). Two loops, two precisions, +stated by upstream at both sites. + +Inside the loop the SDE coefficients themselves split again, and this one is +implicit rather than stated: + +* the **substep** injection is handed `torch.stack([sigma, sub_sigma])`, both + `hp` (`samplers.py:342`), so `get_sde_coeff` runs in **float64**; +* the **step** injection is handed the loop's own `sigmas` (`samplers.py:415`, + `:425`), which `DiffusionStage` created as **float32** + (`ti2vid_two_stages_hq.py:268`), so `get_sde_coeff` runs in **float32**. + +Mirrored rather than unified: one templated implementation, instantiated at the +two scalar types, so there is no second copy of the formula. §3.2. + +--- + +## 2. Scope + +### In + +* `phi`, `get_res2s_coefficients` and the phi cache (`res2s.py:1-62`). +* `res2s_audio_video_denoising_loop` (`samplers.py:208-447`), including + `_get_new_noise`'s normalization (`samplers.py:160-170`) and + `_inject_sde_noise`'s legacy arm (`samplers.py:173-205`). +* `Res2sDiffusionStep.step` / `.get_sde_coeff` at **float64**, by templating the + already-gated float32 implementation rather than copying it. +* `Ltx2StepperKind::kRes2s`. +* A `res2s_two_stage` recipe row, from `LTX_2_3_HQ_PARAMS` and + `ti2vid_two_stages_hq.py`. +* The engine dispatch, so `pipeline_kind=res2s_two_stage` reaches the loop from + `include/vllm.h`, `ltx2-gen` and the server. +* A `dit_evaluations` counter on `Ltx2ConditioningTrace`, because the count is + the only observable that separates the two samplers. + +### Out, and refused or recorded rather than dropped + +* **The prompt enhancer, the distilled LoRA per stage, and DiffVAE.** Already out + of scope for every LTX row here; unchanged by this one. +* **Bit-exact SDE noise against upstream.** The draw is `torch.randn` on a seeded + `torch.Generator`; this port has `SplitMixGaussian`. Already true of the + ancestral arm, which ships. Recorded in §4.3, not hidden. +* **`legacy_mode=False`.** Nothing upstream reaches it on this pipeline + (`DiffusionStage` passes no `legacy_mode`), so mirroring means not building a + selection surface for it. Recorded under `## Owed`. +* **`gradient_estimating_euler_denoising_loop`** (`samplers.py:84-152`) and + `EulerCfgPpDiffusionStep`. Different samplers, no pipeline in scope selects + them. + +--- + +## 3. Design + +### 3.1 A new translation unit, mirroring upstream's own file + +`include/vllm/model_executor/models/ltx2_samplers.h` + +`src/vllm/model_executor/models/ltx2_samplers.cpp`, mirroring +`ltx-pipelines/utils/samplers.py` and `utils/res2s.py`. The steppers stay in +`ltx2_pipeline.{h,cpp}`, which mirrors `ltx-core/components/diffusion_steps.py`. +That is upstream's own partition: a *stepper* advances one substep, a *sampler* +decides how many substeps there are and what is evaluated between them, and they +live in different packages upstream. + +### 3.2 The loop takes hooks, because upstream's takes a `denoiser` + +`res2s_audio_video_denoising_loop` is a free function whose model access is a +`Denoiser` callable (`samplers.py:214`). Mirroring that shape is also what +makes the evaluation count gateable: a test supplies a counting denoiser and +asserts an exact number. + +``` +struct Ltx2Res2sHooks { + // `denoiser(transformer, video_state, audio_state, sigmas, step_index)` + std::function&, const std::vector&, + double sigma, int64_t step_index, + std::vector&, std::vector&)> denoise; + // `post_process_latent` (utils/helpers.py:461-463) + std::function(std::vector, bool is_video)> post_process; + // `new_noise_fn` (samplers.py:220) + std::function(int64_t count, bool is_video, bool substep)> new_noise; +}; +``` + +`denoise` takes a **scalar** sigma rather than a schedule plus an index, because +all three upstream call sites reduce to `sigmas[step_index]` at +`SimpleDenoiser.__call__` / `GuidedDenoiser.__call__` (`utils/denoisers.py:237`) +and the substep call site already passes a one-element schedule with index 0 +(`samplers.py:384-385`). Passing the pair would invite a caller to index it +differently from upstream. + +**`step_index` is still passed, and that is §3.6.** It is a *second* argument +upstream's `Denoiser` takes, and the denoiser reads it for something other than +the sigma. + +The step arithmetic is one templated core in `ltx2_pipeline.cpp`: + +``` +template std::vector Res2sStepImpl(...); +``` + +instantiated three times — `` for the existing, already-gated +`Ltx2Res2sStep`; `` for the step-level injection; `` +for the substep injection. One formula, three dtypes, matching §1.5. The +selection is an enum named after the two upstream call sites, not a bare bool. + +### 3.3 The engine hoists its per-evaluation body + +`ltx2_video.cpp`'s phase loop currently builds `Ltx2ModalityInput`, runs the +denoiser, and post-processes, all inline in the step loop. This row hoists that +into one `Evaluate(video_latent, audio_latent, sigma, step_index)` lambda that +**both** arms call: the Euler/ancestral loop calls it once per step, the res_2s +loop calls it through `hooks.denoise`. No second forward path is written by +hand, and the keyframe-mask guards, the frozen-sigma handling and the trace +updates are reached identically from both. + +`im.trace.dit_evaluations` increments inside `Evaluate`, so it counts every arm, +across every phase. + +### 3.5 The res_2s evaluations go through `Ltx2GuidedDenoise` + +**Added 2026-08-17 at the merge onto `main`.** `daeff67f2` (#1092/#1102) landed +the guided video denoiser into the same phase-loop region this row edits, so +this row and that one both own the body of `Evaluate`. Resolving the conflict +*textually* — keeping this row's bare `Ltx2DitForward` — would have made the HQ +preset **the only unguided video arm in the tree**, at cfg 1.0 where upstream +tunes it at 3.0, and **no gate this row had could see it**: the evaluation count +is the sampler's factor, not the denoiser's. + +Upstream's HQ stage 1 builds a `GuidedDenoiser` and hands it to +`res2s_audio_video_denoising_loop` (`ti2vid_two_stages_hq.py:271-281`, `:292`), +exactly as `ti2vid_one_stage.py:221-226` hands one to the Euler loop. The +sampler decides how many denoiser calls happen; the denoiser decides how many +forwards each call is. So `Evaluate` builds the `Ltx2X0Model` lambda and calls +`Ltx2GuidedDenoise`, and both samplers reach it. + +Stage 2 is a `SimpleDenoiser` upstream (`:316`). Here that is the recipe's +default-constructed `Ltx2MultiModalGuiderParams` — `_POSITIVE_ONLY_GUIDER` +(`denoisers.py:25-28`), cfg 1.0 / stg 0.0 / modality 1.0 — which assembles ONE +pass and a `calculate` whose every term is zero. That equivalence is +`ltx25-guided-video.md` §10's, not a new claim here. + +**The gate is a SECOND counter, because the first one cannot move.** +`Ltx2ConditioningTrace::dit_forwards` counts actual `Ltx2DitForward` calls +inside the x0 lambda; `dit_evaluations` counts denoiser calls. On the HQ stage 1 +they are `3 * (2n + 1)` and `2n + 1` — cond, uncond and mod, because cfg is 3.0 +and modality is 3.0 and stg is 0.0 — and an unguided arm makes them EQUAL. §5.2 +asserts both exactly on two step counts, and §8's mutation strips the guidance +and shows RED. + +### 3.6 The second evaluation's `step_index`: upstream's literal 0 + +Upstream's `Denoiser` signature is +`denoiser(transformer, video_state, audio_state, sigmas, step_index)` and the +res_2s loop passes **three different values** for it: + +| Evaluation | `sigmas` | `step_index` | Anchor | +|---|---|---|---| +| first | the loop's schedule | `step_idx` | `samplers.py:301` | +| substep | `torch.stack([sub_sigma])` | **`0`**, a literal | `samplers.py:384-385` | +| terminal | the loop's schedule | `n_full_steps` | `samplers.py:437` | + +**The decision is to mirror this exactly, including the literal 0.** It is not +cosmetic: the denoiser reads `step_index` through `should_skip_step`, which is +`step % (skip_step + 1) != 0` (`guiders.py:287-291`), so `0 % anything == 0` +makes the substep evaluation **unskippable at any `skip_step`**. + +On the shipped HQ preset this is **inert**: `LTX_2_3_HQ_PARAMS` sets +`skip_step = 0` on both modalities (`constants.py:104`, `:112`) and +`should_skip_step` returns False for every step. It is **not** inert for a +request that sets `video_skip_step` or `audio_skip_step`, which +`ltx25-guided-video.md` §4.5 exposes. There, passing the loop counter at the +substep would skip the same steps the first evaluation skipped and render the +first-order trajectory under the second-order sampler's schedule — at the right +evaluation count, the right shape and the right frame count. + +Gated by `Ltx2Res2sLoopStats::eval_step_indices` plus the fixture's own record +of what arrived, against goldens taken from upstream's own loop. Two independent +records, so a build that recorded one value and passed another fails rather than +agreeing with itself. + +### 3.4 The `res2s_two_stage` recipe + +Two phases, from §1.1: + +| | phase 0 `generate_lowres_hq` | phase 1 `refine_hq` | +|---|---|---| +| `spatial_downscale` | 2 (`:238-243`) | 1 | +| `sigmas` | empty — derived, 15 steps (`:260-267`) | `STAGE_2_DISTILLED_SIGMAS` (`:193`) | +| `noise_scale` | 1.0 | `stage_2_sigmas[0]` (`:327`) | +| `input_transform` | `kInitial` | `kSpatialUpsample` (`:297`) | +| `stepper` | `kRes2s` | `kRes2s` (`:285`, `:319`) | +| guidance | HQ params, override allowed | override not allowed (`SimpleDenoiser`, `:316`) | + +Recipe level: `height`/`width` from `Ltx2Params23Hq().stage_2_*()`, +`num_inference_steps = 15`, `allow_request_sigmas = true` and +`fixed_num_inference_steps = false` (stage 1's schedule really is derived from +`num_inference_steps`), `allow_negative_prompt = true` (stage 1 builds a +`GuidedDenoiser` with a negative context, unlike the distilled arm). + +`("res2s_two_stage", "2.5")` only. `LTX_2_3_HQ_PARAMS` is a plain constant that +overrides every generation-varying knob (`constants.py:91-94` says so), so there +is no `detect_params` lineage to spread it across versions, and this port's +checkpoint is 2.5. + +**The name.** `res2s_two_stage` is already the string +`test_ltx2_pipeline.cpp:1186` uses as a NEGATIVE control for the refusal table. +That case must be repointed in this change, and §5 makes the repointing visible +rather than silent. + +--- + +## 4. Risks + +### 4.1 The one that renders + +Serving the HQ preset's 15 steps and 0.45 rescale on the Euler loop renders a +plausible clip at **half** the model evaluations the preset was tuned for. There +is no pixel, count or shape that says so. Mitigation: `dit_evaluations`, asserted +to an exact number both at the loop and end to end through `Generate`. + +### 4.2 A "better" `phi` + +§1.3. Mitigation: goldens taken from upstream's own `phi` at the cliff, including +`phi2(-1e-10) == 0.0` exactly. + +### 4.3 The noise stream diverges from upstream + +`torch.randn` on a seeded `torch.Generator` is not reproducible here. Consequence: +`res2s` renders are not bit-comparable with upstream, exactly as the shipped +ancestral arm is not. What IS gated: the normalization `_get_new_noise` applies +after the draw, and the loop arithmetic under an injected deterministic noise +function. Stated, not hidden. + +### 4.4 The bong loop is a fixed point, and 100 iterations is not arbitrary + +`eps_1 <- (denoised_1 - x_mid) + h * a21 * eps_1` contracts with ratio +`h * a21`, which the `h < 0.5` guard bounds under `0.25`. It converges to machine +precision long before iteration 100, so an early exit would be numerically +invisible — which is exactly why the loop is mirrored as written and +`bongmath_max_iter` is a parameter rather than a constant folded away. + +### 4.5 Stage 2 needs the spatial upsampler + +`input_transform = kSpatialUpsample` reaches `Ltx2UpsampleVideoLatent`, which +refuses a spatiotemporal upsampler by name. That is the pre-existing behaviour of +`distilled_two_stage` phase 1 and is unchanged here; the HQ arm inherits both the +capability and its refusal. + +--- + +## 5. Tests and evidence + +All goldens are generated by running **upstream's own code** at the pin. The +generator is [`scripts/gen-ltx2-res2s-goldens.py`](../../scripts/gen-ltx2-res2s-goldens.py), +committed beside the nine other `scripts/gen-ltx2-*.py`, and the header of +`tests/vllm/models/ltx2_res2s_goldens.inc` names it. + +**This section previously said the generator was "recorded in this section" and +it was not** — not in this file, not in `scripts/`, not anywhere in the tree, so +a later reader could not regenerate a single number. Repaired on 2026-08-17 by +writing the generator and checking it against the committed file: at `fd4ded7f` +it reproduces `ltx2_res2s_goldens.inc` **byte for byte** apart from the header +line naming it and the `EvalStepIndices` arrays §3.6 adds. That reproduction is +the evidence the committed goldens are what upstream produced; a generator that +merely ran would not have been. + +Regenerate and diff with: + +``` +python3 scripts/gen-ltx2-res2s-goldens.py --ltx2 /path/to/LTX-2 \ + --out tests/vllm/models/ltx2_res2s_goldens.inc +``` + +It refuses a dirty upstream checkout and refuses a revision that is not the pin, +because a SHA in the header that does not describe the code that ran reads as a +pin while the oracle is whatever was in the working tree. + +The substitutions are §0's three plus `model_dtype`: upstream's loop declares +`torch.bfloat16` (`samplers.py:221`) and the generator passes `torch.float32`, +this port's model dtype. §1.2 records that divergence. + +### 5.1 `test_ltx2_pipeline` + +1. **"ltx2 res2s phi mirrors upstream at the small-z cliff"** — `phi(1, z)` and + `phi(2, z)` at 14 values of `z` spanning `0`, both sides of the `1e-10` + guard, and the mid range. Asserted **exactly** (`==`) at `z = 0`, + `z = -1e-11`, `z = -1e-10` and `z = -1e-9`, where upstream's values are + `1.0/0.5`, `1.0/0.5`, `1.000000082740371/0.0` and `0.9999999717180684/0.0`. + A series-expansion port fails on the third and fourth rows. +2. **"ltx2 res2s coefficients mirror upstream"** — `a21`, `b1`, `b2` at 11 values + of `h` including `1e-12`, `1e-10` and `1e-8`, i.e. the cliff carried into the + coefficients. +3. **"ltx2 res2s the noise normalization is applied"** — `_get_new_noise`'s two + normalization steps against upstream's own `_channelwise_normalize` on a fixed + input. Positive control: a zero-filled buffer must NOT reproduce the golden. +4. **"ltx2 res2s the loop evaluates TWICE per step"** — the discriminator. Four + fixtures, a counting denoiser, and an exact expected count: + + | Fixture | `sigmas` | `n_full` | expected evaluations | forces | + |---|---|---|---|---| + | `BongOn` | `0.9, 0.8, 0.7, 0.62` | 3 | **6** | `h < 0.5` and `sigma > 0.03` on every step | + | `BongOffByH` | `0.9, 0.5, 0.25, 0.12` | 3 | **6** | every `h >= 0.5`; every `sigma > 0.03` | + | `BongOffBySigma` | `0.03, 0.028, 0.026, 0.025` | 3 | **6** | every `h < 0.5`; every `sigma <= 0.03` | + | `TerminalZero` | `1.0, 0.75, 0.5, 0.25, 0.0` | 4 | **9** = 2*4+1 | the injected `0.0011` tail | + + The sequence of sigmas the denoiser was called at is asserted too, so a build + that ran two evaluations at the *same* sigma fails: `BongOn` must see + `0.9, 0.848528, 0.8, 0.748331, 0.7, 0.658787`, i.e. `sqrt(sigma*sigma_next)` + interleaved. `TerminalZero`'s last two are `0.0165831` and `0.0011`. + + **Making the expected value impossible to hit by accident:** the counts are 6 + and 9, never 0 and never the step count, so neither a stub that evaluates + nothing nor one that evaluates once per step can pass. `9 != 4` and `6 != 3` + are the assertions that separate this sampler from the shipped one. +5. **"ltx2 res2s the bong refinement is reached, and only in its own branch"** — + the same four fixtures run with `bongmath` true and false. `BongOn` and + `TerminalZero` must **differ**; `BongOffByH` and `BongOffBySigma` must be + **byte-identical**. Both goldens are carried, so "differ" is not asserted + against a value this port computed. This is how each branch is forced and how + the forcing is shown to have worked: the `h` fixture keeps every sigma above + 0.03 and the sigma fixture keeps every `h` below 0.5, so neither can be + passing for the other's reason. `sigma > 0.03` is strict and the fixture + starts at exactly `0.03`. +6. **"ltx2 res2s the loop reproduces upstream"** — final video and audio latents + for all four fixtures, against upstream's own loop output. The denoise mask is + `1,1,0,1,0,1` with a distinct clean latent, so `post_process_latent` is not the + identity and a build that dropped the blend fails at three positions. +7. **"ltx2 the res2s_two_stage recipe is upstream's HQ preset"** — the §3.4 + table, plus that `("res2s_two_stage", "2.3")` still refuses by name. + +### 5.2 `test_ltx2_video` — the production path + +8. **"ltx2 video: the HQ pipeline evaluates the DiT twice per step"** — a + `pipeline_kind=res2s_two_stage` load on the reduced-dimension fixture, + `engine->Generate(...)`, and `trace.dit_evaluations` asserted against the + number the two phases' schedules imply. Compared **against the same render on + `one_stage`**, which must report strictly fewer, so the assertion cannot pass + by both arms being the same. +9. **"ltx2 video: the HQ pipeline stage 1 is GUIDED, three forwards per + evaluation"** — the §3.5 gate. `dit_evaluations` and `dit_forwards` asserted + EXACTLY on two step counts (7/21 and 11/33), the relation + `forwards == 3 * evaluations` derived rather than only read, the four HQ + guider scales, and `pass_ran` for `cond`/`uncond`/`mod` with `ptb` absent. + `forwards != evaluations` is stated as its own assertion, because that is the + sentence an unguided arm's RED has to print. +10. **"ltx2 video: the res_2s SUBSTEP converts x0 against the midpoint, not the + state"** — §9.3. The per-arm invariant over the four `res2s_substep_*` + vectors, with the midpoint displacement as the non-vacuity bound. + +### 5.3 Reachability + +The production entry point is `vllm_video_generate` -> `VideoEngine::Generate` -> +`Ltx2VideoEngine::Generate` -> the phase loop's `kRes2s` dispatch. The mutation +is §8: delete the dispatch, rerun, show RED. + +--- + +## 6. Gates + +``` +cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DVLLM_CPP_CUDA=OFF +cmake --build build -j6 +ctest --test-dir build -j4 --output-on-failure +``` + +Reported with `CONFIGURE_EXIT`, `BUILD_EXIT`, `: error:` count, `ctest -N`, +`CTEST_EXIT`, the pass/fail line, load and free disk, and positive controls for +`No space left` and `BFD assertion`. + +Known-red and not this row's: `windows-msvc-*` (#584). Load-dependent: +`test_async_llm` (#294), `test_engine_core_proc` (#1052), `test_serve_low_tools` +(#428), `test_cpu_x86_llamacpp_floor` exiting 4 as `NO_QUIET_WINDOW` (#618). + +--- + +## 7. Stop conditions + +* Return `NEEDS_DECISION` rather than narrowing if the `res2s_two_stage` recipe + cannot be made reachable from `pipeline_kind` without changing the C ABI. +* Do not take the GPU. A sampler is gated by CPU goldens; `dgx.casa` is contended + and OOM-reboots under a second job. +* Do not "fix" `phi`. §1.3. + +--- + +## 8. What the mutation pass found + +Twenty mutations over five rounds. Each row carries `git diff --stat`, whether it +**BUILT** with its compile-error count, and the exit code captured directly into +a variable — because a mutation that fails to build and one that never applied +both read exactly like a passing test. This row adds a fourth column, the doctest +CASE and ASSERTION counts, and it earned its place in round 2 (see M6b). + +Restores are `git checkout --` against a clean tree, verified by sha256 and +re-stamped with `os.utime`, because a restored file older than its object makes +ninja skip the rebuild and carry the previous mutation's binary forward. The +harness is [`scripts/mutation-harness.py`](../../scripts/mutation-harness.py). + +**This section previously said the harness was `mutate.py`, "recorded beside the +golden generator", and neither file existed.** Repaired on 2026-08-17 by writing +the harness. It refuses a dirty working tree, refuses a mutation whose anchor is +absent or ambiguous rather than running a clean tree and reporting a pass, runs +the WHOLE binary rather than a `--test-case` filter, and refuses to score +anything as a survivor when the case or assertion count is zero — the four +false-green shapes this campaign has paid for, in one place. + +| # | Mutation | BUILT | cc-err | EXIT | cases | asserts | Verdict | +|---|---|---|---|---|---|---|---| +| M1 | delete the `kRes2s` dispatch in the phase loop | YES | 0 | **1** | 1/1F | 22/6F | **DETECTED** — the reachability proof | +| M2 | widen `phi`'s guard to 1e-4, i.e. the series-expansion port | YES | 0 | **1** | 1/1F | 34/9F | DETECTED | +| M3 | drop the SECOND evaluation and reuse the first | YES | 0 | **1** | 1/1F | 7/2F | DETECTED | +| M4 | delete the bong refinement | YES | 0 | **1** | 1/1F | 28/4F | DETECTED | +| M5 | relax the bong `sigma > 0.03` to `>=` | YES | 0 | 0 | 1/0F | 28/0F | **SURVIVED — unobservable, see below** | +| M6a-d | remove one of the two `normalize(noise)` calls | YES | 0 | 0 | 1/0F | 9/0F | **NO-OP mutation, see below** | +| M6e | neuter `normalize`'s shared body | YES | 0 | **1** | 1/1F | 9/4F | DETECTED | +| M7a | collapse the step-level SDE width onto f64, at `kRoundOff` | YES | 0 | 0 | 1/0F | 34/0F | SURVIVED — **fixed**, see below | +| M7b | the same, under a one-ulp bound | YES | 0 | **1** | 1/1F | 42/3F | DETECTED | +| M8a | read the loop's eta at the substep, no eta≠0.5 fixture | YES | 0 | 0 | 1/0F | 34/0F | SURVIVED — **fixed**, see below | +| M8b | the same, with the `Eta1` fixture | YES | 0 | **1** | 1/1F | 42/2F | DETECTED | +| M9 | give the HQ recipe's stage 1 the Euler stepper | YES | 0 | **1** | 1/1F | 22/6F | DETECTED — the defect #921 names | +| M10a | engine hands the loop its RAW draw | YES | 0 | 0 | 1/0F | 22/0F | SURVIVED — **fixed**, see below | +| M10b | the same, with `res2s_noise_moment_error` | YES | 0 | **1** | 1/1F | 25/2F | DETECTED | +| M11 | drop the final evaluation at the injected 0.0011 | YES | 0 | **1** | 1/1F | 72/3F | DETECTED | +| M12 | skip the 0.0011 substitution | YES | 0 | **1** | 1/1F | 95/3F | DETECTED | +| M13 | decode stage 2's audio, which the pipeline discards | YES | 0 | **1** | 1/1F | 43/1F | DETECTED | +| M14 | drop `legacy_mode`'s post-injection blend | YES | 0 | **1** | 1/1F | 42/24F | DETECTED | +| M15a | swap the video/audio draw order, stateless fixture noise | YES | 0 | 0 | 1/0F | 42/0F | SURVIVED — **fixed**, see below | +| M15b | the same, with a stateful fixture generator | YES | 0 | **1** | 1/1F | 42/10F | DETECTED | +| M16 | reassociate `h * a21 * eps` | YES | 0 | 0 | 1/0F | 42/0F | **SURVIVED — below one ulp, see below** | +| M17 | refine the anchor but not `eps_1` | YES | 0 | **1** | 1/1F | 42/3F | DETECTED | +| M18a | `bongmath_max_iter` 100 -> 1, wrong FILE | — | — | — | — | — | **ANCHOR NOT FOUND** — printed, not silent | +| M18b | the same, in the header | YES | 0 | **1** | 1/1F | 42/6F | DETECTED | +| M19 | stop incrementing `dit_evaluations` | YES | 0 | **1** | 1/1F | 25/7F | DETECTED | +| M20 | give the HQ recipe's stage 2 the Euler stepper | YES | 0 | **1** | 1/1F | 43/1F | DETECTED | + +### The four survivors that were fixed + +**M7a, M8a, M10a and M15a each found a real hole, and each is now closed.** + +* **M7a** — the loop golden ran at this file's `kRoundOff` of 5e-6, and the + float32/float64 SDE-coefficient split (§1.5) moves the result by about 1e-7. + The tolerance was a claim this port could not defend. Measured: 3 of 5 fixtures + are BIT-EXACT against upstream and 2 differ by 2.98e-08, one ulp at 0.5. The + bound is now **1e-7**, and a direct case pins the two coefficient arms apart. +* **M8a** — the substep injection is pinned at eta 0.5 whatever the loop's eta is + (samplers.py:273-274), and every fixture ran at the loop's own default of 0.5, + where the two are the same number. An **`Eta1` fixture** (eta = 1.0, generated + from upstream) is the only thing that separates them. +* **M10a** — `Ltx2Res2sNormalizeNoise` was gated as a FUNCTION while whether the + ENGINE calls it was gated by nothing: the end-to-end case checks counts, and + normalization changes no count. `Ltx2ConditioningTrace::res2s_noise_moment_error` + now observes it, asserted below 1e-9, which a raw Gaussian draw cannot reach. +* **M15a** — the fixture's noise hook was STATELESS and returned the same values + for video and audio, so swapping the two injections changed nothing. Upstream's + generator ADVANCES, so within one step the two modalities get different + tensors and the order decides which. The fixture's generator is now stateful on + both sides of the comparison. + +### The two survivors that stand, and why they are not holes + +* **M5 — `sigma > 0.03` against `sigma >= 0.03` is unobservable.** The schedule + is float32, so `0.03f` widens to 0.029999999329447746, which is below the + double `0.03` the guard compares against. No float32 schedule can hold the + boundary value, so no fixture can reach it. Upstream compares the same widened + float32 against the same Python float (samplers.py:357), so the strictness is + unobservable THERE too. The test comment previously claimed this case pinned + it; that claim is now removed and replaced by this derivation. +* **M16 — reassociating `h * a21 * eps`.** Upstream forms the scalar product + first (`h * a21 * eps_1_video`, samplers.py:322) and this port mirrors that. + The reassociated form differs by less than one ulp at this fixture's scale, so + the association is mirrored but is **not separately observable**. Recorded + rather than asserted, because a case claiming to gate it would be a tautology. + +### The two mutations that could not fail, and what that cost + +**M6a through M6d each removed ONE of the two `normalize(noise)` calls, and each +read as a survivor for four rounds.** They were not survivors. The two calls are +idempotent on a rank-2 latent — the header says so in as many words — so removing +either leaves the other doing the whole job. Only M6e, which neuters the shared +body, is a mutation at all. + +**M6b's `-tc` filter matched NOTHING**: the case name was truncated to `ltx2 res2s +the loop NORMALIZES its noise` and the case is `...its noise, unlike the ancestral +loop`. doctest printed `SUCCESS!` with exit 0 over **zero cases**. `git diff +--stat` was correct, the build was clean, and the exit code was 0 — all three +standard facts said "passing test". The CASE COUNT column is the only thing that +caught it, which is the argument for printing it. + +**M18a's anchor was in the header and the mutation targeted the `.cpp`.** The +harness printed `ANCHOR NOT FOUND` rather than running a clean tree and reporting +a pass, which is the fourth shape this campaign has paid for. + +### Reachability + +The chain is `vllm_video_generate` -> `VideoEngine::Generate` -> +`Ltx2VideoEngine::Generate` -> the phase loop's `kRes2s` dispatch. M1 deletes the +last hop and the end-to-end case goes RED (exit 1, 6 failed assertions). The +`pipeline_kind` load extra reaches all three surfaces: `ltx2-gen`'s +`--pipeline-kind` passes the string straight through with no allowlist +(`examples/ltx2_gen/main.cpp:239`), the C ABI takes it as a video load extra, and +the server takes `--video-extra pipeline_kind=res2s_two_stage`. + +--- + +## Owed + +* [#921](https://github.com/mudler/vllm.cpp/issues/921) is closed by this row. +* A real-checkpoint HQ render on `dgx.casa`, and a rendered-clip comparison + against the Euler arm at the same preset. Not attempted here (§0). +* `legacy_mode=False` (`samplers.py:188-192`) — the `timesteps_from_mask` + conversion inside `_inject_sde_noise`. Unreachable upstream from any pipeline in + scope; no selection surface built. +* Bit-exact SDE noise against upstream's `torch.randn` stream (§4.3), which the + already-shipped ancestral arm owes on the same grounds. + +## Owed, added by the implementation + +* **The `sigma > 0.03` strictness is ungated and cannot be gated** through a + float32 schedule (§8, M5). Not filed as an issue: there is no defect and no + fixture that would close it. +* **The `h * a21` association is mirrored but unobservable** at this fixture's + scale (§8, M16). +* **`Ltx2Res2sNormalizeNoise`'s idempotent second call** is unreachable as a + distinct behaviour on a rank-2 latent; it exists because a batched latent would + make it real, and nothing here can tell. + +## 9. The merge onto `main`, and the review repair (2026-08-17) + +The fresh review confirmed the sampler itself: upstream's own +`res2s_audio_video_denoising_loop` imported at `fd4ded7f` and run, 6/6/6/9/6 +evaluations, eval-sigma sequences matching at max diff `0.000e+00`, final +latents within 4.9e-10, `bong_moved` matching on all five fixtures, 14/14 phi +rows and 11/11 coefficient rows bit-exact, 11/11 mutations detected. None of +that is revisited. Three findings were repaired. + +### 9.1 The merge is not textual (§3.5) + +`daeff67f2` (#1092/#1102) landed the guided video denoiser into this row's +phase-loop region. `git merge-tree` reported three conflict hunks in +`src/vllm/multimodal/ltx2_video.cpp` plus `tests/vllm/multimodal/test_ltx2_video.cpp`, +`docs/USAGE.md` and `docs/FEATURES.md`. Resolved deliberately per §3.5 and §3.6. + +### 9.2 The mutation table for the repair + +Run with [`scripts/mutation-harness.py`](../../scripts/mutation-harness.py), +which prints all four facts and refuses a mutation whose anchor is absent. +Baselines: `test_ltx2_video` 75 cases / 2249 assertions / exit 0, +`test_ltx2_pipeline` 50 cases / 2961 assertions / exit 0. + +| # | Mutation | Binary | BUILT | cc-err | EXIT | cases/asserts | Verdict | +|---|---|---|---|---|---|---|---| +| M1 | the HQ arm alone is UNGUIDED, i.e. the naive textual merge | video | YES | 0 | **1** | 75/1F | 2249/8F | DETECTED | +| M2 | delete the `kRes2s` dispatch (reachability) | video | YES | 0 | **1** | 75/2F | 2249/12F | DETECTED | +| M3 | stop counting `dit_forwards` | video | YES | 0 | **1** | 75/1F | 2249/6F | DETECTED | +| M4 | the substep x0 converts against the STREAM latent | video | YES | 0 | **1** | 75/1F | 2249/2F | DETECTED — **was a SURVIVOR, see 9.3** | +| M5 | the loop under-counts the substep evaluation | video | YES | 0 | **1** | 75/3F | 2188/0F | DETECTED | +| M6 | never advance the per-phase evaluation index | video | YES | 0 | **1** | 75/2F | 2233/2F | DETECTED | +| M7 | give the HQ recipe's stage 1 the Euler stepper | video | YES | 0 | **1** | 75/3F | 2237/13F | DETECTED | +| M8 | **M5 beside the OLD tautological check** | video | YES | 0 | **0** | 75/0F | 2249/0F | **SURVIVED — see 9.4** | +| P1 | the substep passes the loop counter, records 0 | pipeline | YES | 0 | **1** | 50/1F | 2961/9F | DETECTED | +| P2 | the substep passes AND records the loop counter | pipeline | YES | 0 | **1** | 50/1F | 2961/27F | DETECTED | +| P3 | the terminal evaluation passes `step_index` 0 | pipeline | YES | 0 | **1** | 50/1F | 2961/1F | DETECTED | + +### 9.3 M4 was a survivor, and what it found + +**On the first pass M4 was GREEN**: exit 0, 74 cases, 2234 assertions, nothing +failed. The substep evaluation runs over `x_mid` (`samplers.py:369-378`) and its +x0 conversion must use the latent that evaluation was handed. Reading +`video.latent` instead moves the whole substep prediction by +`x_mid - x_anchor` and **no instrument in this tree could see it**: the loop's +own arithmetic is gated with a FIXTURE denoiser that never performs a +conversion, and the engine's counters, eval sigmas, bong count and rendered clip +are all invariant under it. + +Closed by `Ltx2ConditioningTrace::res2s_substep_*` and a case that asserts +`cond == latent - timesteps * velocity` over the four recorded vectors, with the +midpoint displacement as the non-vacuity bound. M4 is now RED. + +### 9.4 M8 is why the engine's `VT_CHECK` was rewritten + +The check beside `Ltx2Res2sDenoisingLoop` read +`stats.evaluations > stats.full_steps`. Both operands are fields of the same +struct and `2n + 1 > n` holds for every `n >= 1`, so it could not fail for any +build. Its own comment claimed it "checks the two counters agree", and +`im.trace.dit_evaluations` was never compared against anything. + +**Measured rather than argued.** M8 applies M5's defect — the loop stops +counting its substep evaluation — beside the restored old check, and the suite +is **GREEN at exit 0, 75 cases, 2249 assertions**. The same defect against the +trace-delta form is exit 1. The check now compares +`im.trace.dit_evaluations - evaluations_before` against `stats.evaluations`, +which is the engine's count against the loop's. + +## Owed, added by the review repair + +* **The HQ preset is host-only.** Its `modality_scale = 3.0` asks for the + isolated-modality pass, and `Ltx2DitForwardDevice` takes no `perturbations` + argument, so the guidance resolution refuses that arm before the loop. That is + `ltx25-guided-video.md`'s owed device work (#1092's follow-up), inherited here + rather than newly incurred; `docs/USAGE.md` states it. +* **The merge commit `da54d350e161` carries no trailer block** and + `scripts/check-commit-trailers.py` walks merge commits. It cannot be repaired + in place without a force-push, which this project forbids. Reported to the + operator; the sanctioned route is a fresh branch and a superseding pull + request, which is not this row's decision to take. + +## Now + +`ACTIVE` — implemented on `row/LTX25-RES2S-LOOP`, review findings repaired, +merged onto `origin/main` at `2e025247e`, awaiting re-review. diff --git a/.agents/specs/ltx25-resolution-envelope.md b/.agents/specs/ltx25-resolution-envelope.md index 5f357acc4..efd17570d 100644 --- a/.agents/specs/ltx25-resolution-envelope.md +++ b/.agents/specs/ltx25-resolution-envelope.md @@ -513,6 +513,12 @@ takes a different path entirely. on the Euler loop would render a plausible clip that is quietly not HQ at roughly half the model evaluations the preset was tuned for. No HQ recipe row is added by this row, so nothing can select it and nothing lands dead. + **TAKEN by row `LTX25-RES2S-LOOP`, spec + [`ltx25-res2s-loop.md`](ltx25-res2s-loop.md).** The entry stays here rather + than being deleted, because this file is where the issue's owner was recorded + and the pointer is the provenance; that spec's own `## Owed` carries what + remains of it, which is a real-checkpoint render and the `legacy_mode=False` + arm. - `TI2VidTwoStagesPipeline` as a recipe row — stage 1 on the scheduler-derived schedule under full CFG, stage 2 on `STAGE_2_DISTILLED_SIGMAS` with guidance off (`ti2vid_two_stages.py:243-308`). Distinct from the distilled two-stage diff --git a/CMakeLists.txt b/CMakeLists.txt index 898640405..04a59738f 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -847,6 +847,13 @@ add_library(vllm STATIC # the latent spatial upsampler, the duration head and the embeddings # connector. Additive files mirroring ltx_core's own structure. src/vllm/model_executor/models/ltx2_pipeline.cpp + # LTX-2.5 (ROW LTX25-RES2S-LOOP, issue #921): the res_2s second-order sampler. + # Its own TU because upstream partitions it that way — a STEPPER advances one + # substep and lives in ltx-core/components/diffusion_steps.py, which + # ltx2_pipeline.cpp mirrors, while a SAMPLER decides how many substeps there + # are and what is evaluated between them and lives in + # ltx-pipelines/utils/samplers.py. Two upstream packages, two files here. + src/vllm/model_executor/models/ltx2_samplers.cpp src/vllm/model_executor/models/ltx2_upsampler.cpp # LTX-2.5 (ROW LTX25-DFR-PIPELINE, issue #986): the DFR canvas layout — the # keyframe segment grid, the temporal tile ranges and the latent stitch. Its diff --git a/docs/FEATURES.md b/docs/FEATURES.md index ebec16432..1f9974f04 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -174,6 +174,7 @@ in `ltx2_text_encoder.cpp` is the call that would have to change. | LTX-2.5 Conv VAE decode threading | LTX-2.5 video VAE | `test_ltx2_vae` "the decode DISPATCHES its convolutions to the CPU threadpool" and "...BIT-IDENTICAL across thread counts", through `Ltx2VideoDecodeStreaming`; 34 golden margins UNCHANGED; TSan clean | **Parallel** over CONV output lines via `vt::cpu::ParallelForRows` ([#1009](https://github.com/mudler/vllm.cpp/issues/1009)). ~9x at 16-20 workers, contended box, 21-23% spread. Bit-identical at any count | | LTX-2.5 retake (`RetakePipeline`, regenerate a time window) | LTX-2.5 DiT + video VAE encoder | `test_ltx2_retake` 4/4 (69 assertions) and 4 `test_ltx2_video` cases entering through `Generate`; mask, conform and the four-way plan pinned to upstream `fd4ded7f` | `--pipeline-kind retake` on `ltx2-gen`. Source is a `frame_%06d.ppm` DIRECTORY; a container is REFUSED (no demuxer). Geometry comes from the clip. A folder has no audio, so the soundtrack is generated | | LTX-2.5 text-to-audio (`T2AOneStagePipeline`) | LTX-2.5 DiT + audio VAE, no video VAE | `test_ltx2_video`'s `ltx2 t2a:` cases, entering through `Generate`; 18 mutations, 17 DETECTED (four by review of a conditional-only #1039 gate) and the 18th proven an identity, not a blind spot | `--pipeline-kind t2a_one_stage`. NO picture: 0 frames, no mux argv. The only AUDIO-ONLY guided arm (CFG + STG, 3 forwards/step), so it needs a text tower. CPU only; the device forward is refused by name | +| LTX-2.5 HQ preset (`TI2VidTwoStagesHQPipeline`, `res_2s` sampler) | LTX-2.5 DiT | 6 `test_ltx2_pipeline` cases + 2 `test_ltx2_video` cases through `Generate`, vs UPSTREAM'S OWN loop run at `fd4ded7f`: video latents BIT-EXACT on 3 of 5 fixtures, 1 ulp on 2. 20 mutations, 18 DETECTED | `--pipeline-kind res2s_two_stage`, 2.5 only. TWO denoiser calls per step plus a terminal one, and stage 1 is GUIDED at cfg 3.0 / modality 3.0, so 15 + 3 steps is 38 calls and 100 forwards. The preset IS the sampler | | LTX-2.5 T2A guidance space | LTX-2.5 DiT (T2A arm) | `test_ltx2_video` "the guider is handed x0 predictions" through `Generate`, on all 3 arms plus the guider output and the Euler input; a seam case puts the two spaces 1.5e-07 apart at rescale 0 and 0.352 at 0.7 | Combines **denoised (x0)**, mirroring `X0Model` (`model.py:590-604`). Was velocity space, which agrees only at rescale 0 ([#1039](https://github.com/mudler/vllm.cpp/issues/1039)) | | LTX-2.5 VIDEO guidance | LTX-2.5 DiT, joint video+audio | `test_ltx2_video`'s `ltx2 one_stage:` cases through `Generate`; all FOUR arms carry the x0 invariant and the guider output replays EXACTLY | `--pipeline-kind one_stage` runs `_guided_denoise`: 4 forwards/step, combined per modality in **x0**. Was ONE unguided forward, every `video_guidance` field dead ([#1092](https://github.com/mudler/vllm.cpp/issues/1092)) | | LTX-2.5 cross-attention perturbations | LTX-2.5 DiT | `test_ltx2_video` gates each direction ALONE, on a forward where one stream is PRESENT but DISABLED so only that one runs: the flag moves the stream it writes, the other leaves it bit-identical. Swapping the two is RED | `SKIP_A2V_CROSS_ATTN` / `SKIP_V2A_CROSS_ATTN` ported, which `modality_scale = 3.0` selects on every video row. The DEVICE forward takes no perturbations, so that pass is refused there by name | diff --git a/docs/USAGE.md b/docs/USAGE.md index ab4f6e450..8245c7aa0 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -2634,8 +2634,8 @@ or without the ComfyUI `model.diffusion_model.` prefix. Each family reads its ow knobs from `extras`. H3 takes `partition`. LTX-2.5 takes `audio_prompt_embeds_path` (the audio stream's conditioning, the twin of the seam's `prompt_embeds_path`, which carries the video stream), `pipeline_kind` -(default `distilled_two_stage`; also `one_stage`, `dmd2`, `dfr`, `retake` and -`t2a_one_stage`), `model_version` (only for a checkpoint that +(default `distilled_two_stage`; also `one_stage`, `res2s_two_stage`, `dmd2`, +`dfr`, `retake` and `t2a_one_stage`), `model_version` (only for a checkpoint that declares none), `dit_config_path`, `encoder_config_path`, `negative_prompt_embeds_path` and `negative_audio_prompt_embeds_path` (the negative half of the same fallback, for the unconditional forward), @@ -3126,13 +3126,14 @@ CHECKPOINT_ROOT=... VLLM_CPP_LTX2_TOWER_E2E=1 \ Recipes resolve on an EXACT `(pipeline_kind, model_version)` pair and refuse anything else by name rather than defaulting, because a plausible but wrong sigma -schedule or guidance scale renders a video instead of failing. **Fifteen** pairs -resolve, derived from `ResolveLtx2PipelineRecipe` (`ltx2_pipeline.cpp:1288-1333`): +schedule or guidance scale renders a video instead of failing. **Sixteen** pairs +resolve, derived from `ResolveLtx2PipelineRecipe`: | `pipeline_kind` | resolving `model_version` | |---|---| | `one_stage` | 2, 2.3, 2.4, 2.5 | | `distilled_two_stage` | 2, 2.5 | +| `res2s_two_stage` | **2.5 only** | | `dfr` | **2.5 only** | | `dmd2` | 2, 2.3 | | `retake` | 2, 2.5 | @@ -3143,8 +3144,54 @@ This list ran to ten until 2026-08-17, omitting `dfr` entirely and all four DFR's base stage rests on generated keyframe slots, which need a checkpoint declaring `use_keyframes_abs_pos_embedding`, and the 2.0 distilled row predates that parameter — so resolving DFR onto it would build a recipe the engine must -then refuse at load. Refusing at the recipe table names the version instead -(`ltx2_pipeline.cpp:1306-1313`). +then refuse at load. Refusing at the recipe table names the version instead. + +### `res2s_two_stage`: the high-quality preset, and why it is a sampler + +`res2s_two_stage` is `TI2VidTwoStagesHQPipeline`. Against the plain two-stage +pipeline it changes the SAMPLER on both stages — the `res_2s` second-order +method instead of Euler — and takes `LTX_2_3_HQ_PARAMS`: 15 steps, STG off, +video rescale 0.45, cfg 3.0 video / 7.0 audio, modality 3.0. Those are not the +only differences (stage 1 also loads the distilled LoRA, derives its schedule +from the stage-1 latent shape, and runs a `GuidedDenoiser` where the plain +pipeline runs a `FactoryGuidedDenoiser`), so do not read the sampler swap as an +exhaustive list. It resolves at 2.5 only, because that preset is a plain +constant upstream with no per-generation lineage to spread it over. + +Fifteen steps is not fewer forwards, and it is not even 15 model calls. The +`res_2s` loop evaluates the denoiser TWICE per step — once at the step's sigma +and once at the geometric mean of that sigma and the next — and once more at a +terminal sigma the schedule injects. Stage 1's 15 steps is therefore 31 denoiser +calls, and stage 2's frozen 3-step schedule adds 7, for **38 calls per render**. +Stage 1 is also GUIDED, so each of its calls is three transformer forwards +(conditional, unconditional, isolated-modality) against stage 2's one: **100 +transformer forwards** for a full render, where `one_stage` at its own 30-step +default runs 30 calls. Expect the HQ preset to cost several times the 30-step +arm and to look better, not to be faster. + +That is also why the preset cannot be reached by passing its numbers to another +kind. `--steps 15` on `one_stage` renders a finished, correctly sized, plausible +clip at a fraction of the model evaluations the preset was tuned for, and no +property of the output says so. Ask for the pipeline, not for its step count. + +```sh +ltx2-gen --pipeline-kind res2s_two_stage \ + --prompt "a cinematic shot of ..." \ + --height 1088 --width 1920 --frames 121 +``` + +`pipeline_kind` is a LOAD knob, so this reaches the C API and the server too: a +server started with `--video-extra pipeline_kind=res2s_two_stage` renders every +request on the HQ preset. + +Three limits, stated rather than left to be found. The stage-2 spatial upsample +is the same one `distilled_two_stage` uses and carries the same refusal when the +checkpoint has no latent upsampler. The loop's SDE noise is drawn from this +port's own generator rather than upstream's seeded `torch.randn`, so a render is +not bit-comparable with Lightricks' — the same limit the ancestral arm already +ships with. And stage 1's guidance asks for an isolated-modality pass, which the +device-resident forward cannot perturb, so this preset is host-only until that +is closed; both are recorded in `.agents/specs/ltx25-res2s-loop.md`. ### Retake: regenerating a time window of an existing clip diff --git a/include/vllm/model_executor/models/ltx2_pipeline.h b/include/vllm/model_executor/models/ltx2_pipeline.h index 93c75cc22..78bc0ad42 100644 --- a/include/vllm/model_executor/models/ltx2_pipeline.h +++ b/include/vllm/model_executor/models/ltx2_pipeline.h @@ -210,6 +210,46 @@ std::vector Ltx2Res2sStep(const float* sample, const float* denoised, const float* sigmas, int64_t sigma_count, int64_t step_index, int64_t count, const float* noise, double eta = 0.5); +// ─── THE SAME STEP, AT THE PRECISION EACH CALL SITE ACTUALLY HANDS IT ──────── +// +// `Res2sDiffusionStep.step` has no dtype of its own: it takes whatever its +// tensors carry, and the res_2s loop hands it two DIFFERENT combinations. Both +// are mirrored rather than unified onto one, because the difference is real +// arithmetic and putting the conversion where upstream puts it is the rule. +// +// SUBSTEP (samplers.py:337-352). `sigmas = torch.stack([sigma, sub_sigma])`, +// and both are `hp` (:291-292, :315). So `get_sde_coeff` runs in FLOAT64. +// +// STEP (samplers.py:412-427). `sigmas` is the loop's own schedule, which +// `DiffusionStage` created as FLOAT32 (ti2vid_two_stages_hq.py:268). So +// `get_sde_coeff` runs in FLOAT32 — the residual `sqrt(sigma_next^2 - +// sigma_up^2)`, `alpha_ratio` and `sigma_down` are all f32 quantities — while +// the SAMPLE and the noise are still f64 and the result is f64. +// +// The values in both cases are f64, because `sample` is `x_anchor` (`hp`) and +// `output_dtype = denoised_sample.dtype` is `hp` too (diffusion_steps.py:180). +// +// One implementation, instantiated at the two scalar types; there is no second +// copy of the formula. The selection is an enum naming the two upstream call +// sites rather than a bare bool, so a reader can check the claim. +enum class Ltx2Res2sScheduleWidth { + // samplers.py:415, :425 — the loop's float32 schedule. + kF32Schedule, + // samplers.py:342, :350 — the [sigma, sub_sigma] pair, both float64. + kF64Schedule, +}; + +// `Res2sDiffusionStep.get_sde_coeff` computed in float64 rather than float32. +// The f32 arm stays `Ltx2Res2sSdeCoeff` above and keeps its goldens. +Ltx2SdeCoeff Ltx2Res2sSdeCoeffHp(double sigma_next, double sigma_up); + +// `Res2sDiffusionStep.step` over float64 samples. `width` decides only the +// precision the SIGMAS and therefore the coefficients are computed at. +std::vector Ltx2Res2sStepHp(const double* sample, const double* denoised, + const double* sigmas, int64_t sigma_count, + int64_t step_index, int64_t count, const double* noise, + double eta, Ltx2Res2sScheduleWidth width); + // _get_ancestral_step (diffusion_steps.py:7-22): the DDIM / variance-exploding // ancestral coefficients, in the rescaled `sigma / alpha` space. Used only by // CFG++. @@ -518,7 +558,20 @@ bool Ltx2ShouldUseAncestralSampler(const std::string& version); // ltx2_recipes.py:38 — how a phase builds its input. enum class Ltx2PhaseInputTransform { kInitial, kSpatialUpsample }; // Which stepper a phase samples with (distilled.py:170-185). -enum class Ltx2StepperKind { kEuler, kEulerAncestral }; +// +// `kRes2s` is not only a stepper: it selects a whole SAMPLER. Upstream keeps the +// two choices separate — `DiffusionStage.__call__` takes `stepper` and `loop` +// independently (utils/blocks.py:512-513) — but they are not independently +// selectable in practice, because `res2s_audio_video_denoising_loop` REFUSES any +// stepper that is not a `Res2sDiffusionStep` (samplers.py:276-277) and no other +// loop constructs one. `TI2VidTwoStagesHQPipeline` passes both together, to both +// stages (ti2vid_two_stages_hq.py:285/:292 and :319/:335). One enumerator +// therefore carries both, and the alternative — a separate loop field whose only +// legal combination is this one — would publish a selection surface upstream +// does not have and three combinations that must then be refused. +// +// Row LTX25-RES2S-LOOP, issue #921. Spec .agents/specs/ltx25-res2s-loop.md. +enum class Ltx2StepperKind { kEuler, kEulerAncestral, kRes2s }; // LTXPhaseRecipe (ltx2_recipes.py:29-50). struct Ltx2PhaseRecipe { @@ -624,6 +677,19 @@ void Ltx2AssertResolution(int64_t height, int64_t width, int64_t divisor); // ("one_stage", "2.5") Lightricks, via _PARAMS_SINCE_VERSION (:130-133) // ("distilled_two_stage","2") vLLM-Omni LTX2_DISTILLED_TWO_STAGE_RECIPE (:125-158) // ("distilled_two_stage","2.5") Lightricks distilled.py + constants.py:17-23 +// ("res2s_two_stage", "2.5") Lightricks ti2vid_two_stages_hq.py:59-340 plus +// LTX_2_3_HQ_PARAMS (constants.py:95-115). Row +// LTX25-RES2S-LOOP, #921. The res_2s sampler on +// BOTH stages, 15 steps, STG off. 2.5 only, and +// not by analogy with the one_stage rows: +// `LTX_2_3_HQ_PARAMS` is a plain constant that +// overrides every generation-varying knob +// (constants.py:91-94 says so), so there is no +// `detect_params` lineage to spread it across +// versions. THE SAMPLER IS THE PRESET: this +// recipe on `kEuler` would render a finished, +// correctly sized, plausible clip at half the +// model evaluations 15 steps was tuned for // ("dmd2", "2") vLLM-Omni LTX_POSITIVE_ONLY_RECIPE (:116-124) // ("dmd2", "2.3") same // ("dfr", "2.5") Lightricks dfr_pipeline.py:155-561 (row diff --git a/include/vllm/model_executor/models/ltx2_samplers.h b/include/vllm/model_executor/models/ltx2_samplers.h new file mode 100644 index 000000000..c1a783293 --- /dev/null +++ b/include/vllm/model_executor/models/ltx2_samplers.h @@ -0,0 +1,319 @@ +// LTX-2.5 SAMPLERS — the res_2s second-order denoising loop. +// +// Row: LTX25-RES2S-LOOP. Spec: .agents/specs/ltx25-res2s-loop.md. Issue #921. +// +// ─── WHAT THIS IS A PORT OF (file:line on BOTH sides) ──────────────────────── +// Upstream: Lightricks/LTX-2 @ fd4ded7f, +// packages/ltx-pipelines/src/ltx_pipelines/ +// OURS <- UPSTREAM +// Ltx2Phi <- utils/res2s.py:4-22 +// Ltx2Res2sCoefficients <- utils/res2s.py:25-62 +// Ltx2Res2sNormalizeNoise <- utils/samplers.py:160-170 +// Ltx2Res2sDenoisingLoop <- utils/samplers.py:208-447 +// +// ─── WHY THIS IS A SEPARATE TRANSLATION UNIT ───────────────────────────────── +// Upstream's own partition. A *stepper* advances one substep and lives in +// `ltx-core/components/diffusion_steps.py`, which this port mirrors in +// `ltx2_pipeline.{h,cpp}`. A *sampler* decides how many substeps there are, what +// is evaluated between them, and in what order, and lives in +// `ltx-pipelines/utils/samplers.py`. They are different packages upstream and +// they are different files here. +// +// ─── THE SAMPLER *IS* THE HQ VARIANT ───────────────────────────────────────── +// `TI2VidTwoStagesHQPipeline` differs from `TI2VidTwoStagesPipeline` in SEVERAL +// things, and this loop is two of them: `stepper=Res2sDiffusionStep()` +// (ti2vid_two_stages_hq.py:258) and `loop=res2s_audio_video_denoising_loop` +// passed to both stages (:292, :335). The others, measured by diffing the two +// files at `fd4ded7f` rather than asserted: `LTX_2_3_HQ_PARAMS` +// (utils/constants.py:95-115); stage 1 loads the distilled LoRA at +// `distilled_lora_strength_stage_1` where the plain pipeline loads none on that +// stage (:92-101 against ti2vid_two_stages.py:140); the stage-1 schedule is +// derived as `execute(latent=empty_latent, steps=...)` against the plain +// pipeline's `execute(steps=...)`, which `schedulers.py:32` makes a +// RESOLUTION-DEPENDENT shift rather than the 4096-token default; and +// `GuidedDenoiser` (:271-281) replaces `FactoryGuidedDenoiser`. This comment +// said "exactly three things" until 2026-08-17, and the count was wrong in a +// load-bearing way, because it was the argument for what this row had to port. +// +// So a build that served the HQ preset's 15 steps and 0.45 rescale on the Euler +// loop would render a plausible clip at HALF the denoiser calls the preset was +// tuned for, and there is no shape, frame count, sample rate or pixel that says +// so. The one observable that separates the two samplers is the number of +// denoiser evaluations, which is why `Ltx2Res2sLoopStats::evaluations` exists +// and why the suite asserts an exact number rather than a bound. +// +// AND THE COUNT OF EVALUATIONS CANNOT SEE THE OTHER HALF. Each evaluation on +// the HQ stage 1 is THREE transformer forwards, because `GuidedDenoiser` runs +// the conditional, unconditional and isolated-modality passes +// (denoisers.py:100-137) at cfg 3.0 and modality 3.0. An arm that ran this +// sampler around a bare unguided forward reports the same evaluation count this +// file gates. `Ltx2ConditioningTrace::dit_forwards` is the second counter, and +// it is what the engine's gate reads. +// +// ─── DTYPE, AND WHY IT IS NOT f32 HERE ─────────────────────────────────────── +// This is the one LTX-2.5 path whose interior is DOUBLE, and that is upstream's +// own choice stated in upstream's own words: `hp = highest_precision_float(...)` +// with the comment "float64 on CUDA/CPU for ODE numerical stability" +// (samplers.py:261-262). Every anchor, epsilon, midpoint and combination below +// is `double`; the LATENT that enters and leaves is f32, which is this port's +// `model_dtype`, at the positions upstream writes `.to(model_dtype)` — :370, +// :375, :431, :433, :442 and :445. +// +// AND THE TWO `model_dtype`s ARE NOT THE SAME WIDTH. Upstream's loop declares +// `model_dtype: torch.dtype = torch.bfloat16` (samplers.py:221) and the HQ +// pipeline overrides nothing (`DiffusionStage.__call__` passes six keyword +// arguments, utils/blocks.py:566-573), so upstream stores this latent at bf16 +// where this port stores it at f32 — twice the bytes on the largest buffer in +// the loop. That is a PORT-WIDE pre-existing choice, not this row's: every +// LTX-2.5 host path here is f32 (`ltx2.h`), and narrowing one loop's storage +// would put a bf16 tensor into an f32 pipeline. It is stated here because +// `AGENTS.md` "Inherit vLLM defaults" says a wider dtype is invisible to every +// correctness gate this project owns, so it has to be written down where the +// divergence lives rather than discovered later. +// +// The already-ported ANCESTRAL loop does the opposite and steps in float32 +// (samplers.py:550 calls `.float()` on the SAMPLE; the denoised operand was +// already floated at :484, so only one `.float()` sits at the step call and a +// reader looking for two at :550-551 finds one). Two loops, two +// precisions, in one file. Neither is a widening choice made here. +#pragma once + +#include +#include +#include +#include +#include + +namespace vllm { + +// --------------------------------------------------------------------------- +// The exponential integrator (utils/res2s.py) +// --------------------------------------------------------------------------- + +// `phi(j, neg_h)` (res2s.py:4-22). +// +// phi_j(z) = (e^z - sum_{k, double>; + +// `get_res2s_coefficients` (res2s.py:25-62). `c2` is the substep position and is +// 0.5 on every reachable path (samplers.py:288). +struct Ltx2Res2sCoefficients { + double a21 = 0.0; // c2 * phi_1(-h * c2) (res2s.py:48-50) + double b1 = 0.0; // phi_1(-h) - b2 (res2s.py:59-60) + double b2 = 0.0; // phi_2(-h) / c2 (res2s.py:54-56) +}; +Ltx2Res2sCoefficients Ltx2GetRes2sCoefficients(double h, Ltx2PhiCache& phi_cache, + double c2 = 0.5); + +// --------------------------------------------------------------------------- +// The noise (utils/samplers.py:155-170) +// --------------------------------------------------------------------------- + +// The normalization half of `_get_new_noise` (samplers.py:164-170): a global +// `(n - mean) / std`, then `_channelwise_normalize` (:160-161), which on this +// port's rank-2 [tokens, width] latent covers the same elements and is therefore +// the identity up to rounding. BOTH ARE APPLIED ANYWAY, in upstream's order, +// because "idempotent" is a property of the data this port happens to hand it +// and not of the function; a batched latent would make the second one real. +// +// THE DRAW ITSELF IS NOT HERE, and that is the honest boundary. Upstream draws +// `torch.randn` on a seeded `torch.Generator`; this port has `SplitMixGaussian`. +// The streams differ, so a res_2s render is not bit-comparable with upstream — +// exactly as the already-shipped ancestral arm is not. What IS mirrored is which +// noise function each loop uses, and that is not the same for the two: +// `euler_ancestral_denoising_loop` defaults to `_get_plain_noise`, a bare +// `randn` (samplers.py:574), and the res_2s loop defaults to `_get_new_noise`, +// which normalizes (samplers.py:220). Two loops, two noise functions, ten lines +// apart. Reading one off the other would drop this step silently. +// +// `std` is UNBIASED (torch's default, n-1 denominator), matching `Tensor.std()`. +std::vector Ltx2Res2sNormalizeNoise(std::vector noise); + +// The two seeds upstream's loop declares (samplers.py:215-216, :265-266). +// +// `-1` IS A CONSTANT, NOT THE REQUEST'S SEED, and that is the fact most likely +// to be got wrong by analogy. `DiffusionStage.__call__` passes the loop SIX +// keyword arguments — sigmas, video_state, audio_state, stepper, transformer, +// denoiser — and no others (utils/blocks.py:566-573), so `noise_seed` keeps its +// declared default on every reachable path. The already-ported ancestral arm +// does the opposite and derives its seed from the pipeline's +// (distilled.py:69-73), which is why this is stated rather than assumed. +inline constexpr int64_t kLtx2Res2sNoiseSeed = -1; +inline constexpr int64_t kLtx2Res2sNoiseSeedSubstepOffset = 10000; + +// --------------------------------------------------------------------------- +// The loop (utils/samplers.py:208-447) +// --------------------------------------------------------------------------- + +// `_inject_sde_noise`'s substep call fixes eta at 0.5 "for compatibility with +// the original implementation" (samplers.py:273-274) regardless of the step-level +// eta. Step level takes the loop's `eta`, which is 0.5 by default (:217). +inline constexpr double kLtx2Res2sSubstepEta = 0.5; +inline constexpr double kLtx2Res2sEta = 0.5; +// samplers.py:218-219. +inline constexpr bool kLtx2Res2sBongMath = true; +inline constexpr int64_t kLtx2Res2sBongMathMaxIter = 100; +// samplers.py:288 — "Midpoint for res_2s". +inline constexpr double kLtx2Res2sC2 = 0.5; +// samplers.py:357 — the bong guard, `h < 0.5 and sigma > 0.03`. STRICT on both +// sides: a schedule sitting at exactly 0.03 does NOT refine. +inline constexpr double kLtx2Res2sBongMaxH = 0.5; +inline constexpr double kLtx2Res2sBongMinSigma = 0.03; +// samplers.py:281-282 — the minimal sigma injected in place of a terminal zero, +// "to avoid division by zero". It becomes a real schedule entry, so the loop's +// last full step lands on it and the final evaluation happens AT it. +inline constexpr float kLtx2Res2sTerminalSigma = 0.0011f; + +// What the loop needs from its caller. Upstream's loop takes a `transformer` and +// a `Denoiser` callable (samplers.py:213-214) rather than reaching for a model, +// and mirroring that shape is also what makes the evaluation count gateable: a +// test supplies a counting denoiser and asserts an exact number. +struct Ltx2Res2sHooks { + // `denoiser(transformer, video_state, audio_state, sigmas, step_index)` + // (samplers.py:301, :380-386). Writes each modality's DENOISED prediction — + // upstream's `X0Model` returns x0, not velocity (ltx-core + // model/transformer/model.py:590-604 is the forward that converts; + // utils/blocks.py:480-482 only shows that the loop is handed that TYPE, and + // the `utils/` prefix matters because a second `blocks.py` exists under + // ltx-core model/video_vae/transformer/) — at the + // model dtype, which here is f32. + // + // A SCALAR SIGMA, not a schedule and an index into it, because all three + // upstream call sites reduce to `sigmas[step_index]` inside the denoiser + // (utils/denoisers.py:237) and the substep one already passes a ONE-element + // schedule with index 0 (samplers.py:384-385). Handing a pair to this hook + // would invite a caller to index it differently from upstream. + // + // `step_index` IS STILL PASSED, because it is a SECOND argument upstream's + // `Denoiser` takes and the denoiser reads it for something other than the + // sigma: `should_skip_step` is `step % (skip_step + 1) != 0` + // (guiders.py:287-291). The three call sites pass three different things — + // `step_idx` (samplers.py:301), a literal `0` (samplers.py:385) and + // `n_full_steps` (samplers.py:437) — so the substep evaluation is never + // skipped whatever the request's `skip_step` is. Deriving it here from the + // loop counter instead would silently skip half of a step's evaluations on a + // request that sets `skip_step`, and no rendered frame would show it. + // + // `double`, AND THE NARROWING BELONGS TO THE CALLER. The two evaluations are + // handed different widths upstream: the first gets an entry of the float32 + // schedule (samplers.py:301) and the second gets `sub_sigma`, which is + // float64 (`torch.stack([sub_sigma])`, samplers.py:384). This port's DiT + // interface takes `const float*` for `Modality.sigma`, so a narrowing has to + // happen somewhere; it happens at that interface, in the engine, and not here, + // so the loop stays the shape upstream's is. + std::function& video_latent, + const std::vector& audio_latent, double sigma, int64_t step_index, + std::vector& denoised_video, + std::vector& denoised_audio)> + denoise; + + // `post_process_latent(x, denoise_mask, clean)` (utils/helpers.py:461-463), + // per modality. Kept as a hook rather than taking the mask and the clean + // latent as arguments because the engine already owns both inside its own + // stream struct, and a second copy of the blend is the shape this project has + // recorded going wrong. + // + // ONE `double` HOOK FOR BOTH OF UPSTREAM'S WIDTHS, and the reason is a + // property of the data rather than of the function. Upstream calls + // `post_process_latent` at the model dtype on a denoiser result + // (samplers.py:305, :390, :441) and at `hp` on a sample inside + // `_inject_sde_noise` (samplers.py:203). The blend is + // `denoised * mask + clean * (1 - mask)`, and every LTX-2.5 denoise mask is + // 0 or 1 — `create_initial_state` writes ones and a conditioning zeroes whole + // token rows — so the result is exactly one operand or the other and no + // rounding is reachable at either width. The loop still narrows the + // model-dtype call sites back to f32 afterwards, mirroring + // `.to(denoised.dtype)`, so a mask that ever stopped being 0/1 would show as a + // difference rather than silently taking the wider path. + std::function(std::vector x, bool is_video)> post_process; + + // `new_noise_fn(state.latent, generator)` (samplers.py:220, :187). `substep` + // selects between upstream's TWO generators (samplers.py:267-268), which are + // seeded `noise_seed` and `noise_seed + 10000` so the substep draw is not + // bit-identical to the step draw. + std::function(int64_t count, bool is_video, bool substep)> new_noise; +}; + +// Both modalities' state, in and out. Upstream carries a `LatentState` per +// modality and allows either to be absent (samplers.py:231); `present` is that +// `None`. +struct Ltx2Res2sModality { + std::vector latent; // model_dtype (f32 here) + bool present = false; +}; + +// Reported so the caller can assert what happened, because nothing in the +// returned latents can. `evaluations` is the discriminator this row rests on. +struct Ltx2Res2sLoopStats { + // Total denoiser calls. `2 * full_steps + 1` when the caller's schedule ends + // at 0, `2 * full_steps` when it does not. + int64_t evaluations = 0; + // `n_full_steps` (samplers.py:279), taken BEFORE the terminal sigma injection. + int64_t full_steps = 0; + // Steps on which `bongmath and h < 0.5 and sigma > 0.03` held (samplers.py:357). + int64_t bong_steps = 0; + // The sigma each evaluation ran at, in call order. Every odd entry is + // `sqrt(sigma * sigma_next)` (samplers.py:315), so a build that evaluated + // twice at the SAME sigma is visible here and nowhere else. + std::vector eval_sigmas; + // The `step_index` each evaluation was handed, in the same call order. It is + // NOT the position in this vector and it is not the loop counter: upstream + // passes `step_idx`, then a literal `0` for the substep, then `n_full_steps` + // for the terminal evaluation (samplers.py:301, :385, :437). The denoiser + // reads it through `should_skip_step` (guiders.py:287-291), so on a request + // with `skip_step != 0` the sequence decides which evaluations run a forward + // at all — and nothing in the returned latents, the evaluation count or a + // rendered frame records which value was passed. + std::vector eval_step_indices; +}; + +// Loop parameters, in upstream's own declaration order and with upstream's own +// defaults (samplers.py:208-223). They are defaults HERE for the same reason +// they are defaults THERE: `DiffusionStage.__call__` overrides none of them. +struct Ltx2Res2sLoopParams { + double eta = kLtx2Res2sEta; + bool bongmath = kLtx2Res2sBongMath; + int64_t bongmath_max_iter = kLtx2Res2sBongMathMaxIter; + double c2 = kLtx2Res2sC2; +}; + +// `res2s_audio_video_denoising_loop` (samplers.py:208-447). `legacy_mode` is +// TRUE on every reachable path (samplers.py:222 default, never overridden), so +// `_inject_sde_noise` hands the stepper the raw schedule and applies +// `post_process_latent` afterwards (:202-203) instead of converting sigmas +// through `timesteps_from_mask` (:188-192). The false arm is not built; nothing +// upstream selects it and a selection surface for it would be invented here. +Ltx2Res2sLoopStats Ltx2Res2sDenoisingLoop(const std::vector& sigmas, + Ltx2Res2sModality& video, + Ltx2Res2sModality& audio, + const Ltx2Res2sHooks& hooks, + const Ltx2Res2sLoopParams& params = {}); + +} // namespace vllm diff --git a/include/vllm/multimodal/ltx2_video.h b/include/vllm/multimodal/ltx2_video.h index 2ae214989..fdb175542 100644 --- a/include/vllm/multimodal/ltx2_video.h +++ b/include/vllm/multimodal/ltx2_video.h @@ -746,6 +746,100 @@ struct Ltx2ConditioningTrace { uint64_t retake_latent_digest = 0; double retake_latent_absmax = 0.0; + // ── THE SAMPLER (row LTX25-RES2S-LOOP, #921) ────────────────────────────── + // + // TWO COUNTERS, BECAUSE THERE ARE TWO QUESTIONS AND ONE NUMBER CANNOT ANSWER + // BOTH. A render's DiT work is `evaluations x forwards-per-evaluation`. The + // sampler decides the first factor and the denoiser decides the second, and a + // build can get either wrong while producing a clip of the same shape, frame + // count, sample rate and file size. + // + // `dit_evaluations` is every DENOISER CALL this render made, across every + // phase and every step, and it is the only thing that separates the res_2s + // sampler from the first-order one. The two samplers differ in that one calls + // the denoiser TWICE per step (samplers.py:301 and :380-386) plus once at the + // terminal sigma (:437). Serving the HQ preset's 15 steps on the Euler loop + // would make 15 calls where upstream makes 31, and no output check in this + // tree could tell. + // + // The count is `2 * steps + 1` per res_2s phase when that phase's schedule + // ends at 0 and `2 * steps` when it does not, against `steps` for the Euler + // and ancestral arms — so a build that selected the wrong sampler reports a + // number that is close to half, not a number that is wrong by one. + // + // Incremented at ONE site, inside the shared `Evaluate` lambda that every + // sampler goes through, so no arm can make a call this misses. A second + // increment beside the res_2s loop's own returned `evaluations` would let the + // two drift; the engine asserts they agree instead. + int64_t dit_evaluations = 0; + // `dit_forwards` is every ACTUAL `Ltx2DitForward` this render ran, counted + // inside the `Ltx2X0Model` lambda the guided denoiser drives. One evaluation + // is one to four forwards — `cond`, `uncond`, `ptb`, `mod` + // (denoisers.py:100-137) — so this is the factor `dit_evaluations` cannot see. + // + // IT EXISTS BECAUSE THE EVALUATION COUNT IS BLIND TO GUIDANCE. Upstream's HQ + // stage 1 runs a `GuidedDenoiser` at cfg 3.0 / 7.0 with modality 3.0 + // (ti2vid_two_stages_hq.py:271-281, constants.py:99-114), which is three + // forwards per evaluation. An arm that ran the res_2s sampler around a bare + // unguided forward keeps `dit_evaluations` at exactly `2 * steps + 1`, renders + // a plausible clip at cfg 1.0 where the preset was tuned at 3.0, and moves no + // other number in this struct. This one drops from `3 * (2 * steps + 1)` to + // `2 * steps + 1`, which is why it is asserted rather than described. + int64_t dit_forwards = 0; + // Steps on which the bong anchor refinement ran, i.e. on which + // `bongmath and h < 0.5 and sigma > 0.03` held (samplers.py:357). Zero on + // every non-res_2s pipeline. It is reported separately from the evaluation + // count because the refinement changes the latent WITHOUT changing how many + // forwards ran, so the counter above is blind to it. + int64_t res2s_bong_steps = 0; + // The largest deviation from a standardized draw across every noise tensor the + // res_2s loop was handed: `max(|mean|, |sd - 1|)` over each draw, maximum over + // all of them. Zero on every non-res_2s pipeline. + // + // IT EXISTS BECAUSE THE WIRING IS INVISIBLE OTHERWISE. `_get_new_noise` + // normalizes (samplers.py:164-170) and `_get_plain_noise` does not + // (:155-157); the res_2s loop takes the first and the ancestral loop takes + // the second, ten lines apart in one file. `Ltx2Res2sNormalizeNoise` is gated + // as a FUNCTION by `test_ltx2_pipeline`, but whether the engine's hook calls + // it is a different claim, and nothing about a rendered clip, a token count or + // an evaluation count can answer it. MEASURED: with the engine handing the + // loop its raw draw, the end-to-end suite stayed GREEN — mutation M10 in + // .agents/specs/ltx25-res2s-loop.md section 8 — which is why this field was + // added rather than the wiring being left as a claim. + // + // A NORMALIZED draw drives this to ~1e-15 by construction. A raw Gaussian + // draw cannot: its sample mean is O(1/sqrt(n)) and its sample deviation is + // O(1/sqrt(n)) away from 1, so on any latent this engine builds the two are + // orders of magnitude apart rather than close. + double res2s_noise_moment_error = 0.0; + + // WHAT THE SECOND EVALUATION WAS HANDED, and what it returned for the + // conditional pass. Empty on every arm but `res2s_two_stage`, and written at + // phase 0's SECOND evaluation, which on that arm is the substep. + // + // IT EXISTS BECAUSE THE SUBSTEP'S x0 CONVERSION HAS NO OTHER OBSERVABLE. The + // res_2s substep runs over `x_mid` (samplers.py:369-378), a state that never + // becomes the stream's own latent, so `to_denoised` there must use the latent + // THAT EVALUATION was handed and not `video.latent`. Those are the same tensor + // everywhere else in this file, which is what makes the wrong one an easy + // write and an invisible one. + // + // MEASURED: with the conversion reading `video.latent`, the whole + // `test_ltx2_video` suite stayed GREEN at 74 cases and 2234 assertions. The + // loop's own arithmetic is gated against upstream with a FIXTURE denoiser, so + // that gate never sees the engine's conversion; the clip, the evaluation + // count, the eval sigmas and the bong count are all blind to it. + // + // The gate is the per-arm invariant `cond == latent - timesteps * velocity` + // over THESE tensors — an equation between four recorded vectors, not a + // magnitude — plus the non-vacuity that `res2s_substep_latent` differs from + // `video_first_latent`, which is what says the midpoint moved at all. + std::vector res2s_substep_latent; + std::vector res2s_substep_timesteps; + std::vector res2s_substep_cond; + std::vector res2s_substep_cond_velocity; + double res2s_substep_sigma = 0.0; + // ── TEXT-TO-AUDIO: what the audio-only render actually ran (#1005) ──────── // // Zero and false everywhere on a pipeline that is not `t2a_one_stage`. diff --git a/scripts/gen-ltx2-res2s-goldens.py b/scripts/gen-ltx2-res2s-goldens.py new file mode 100644 index 000000000..949b81e05 --- /dev/null +++ b/scripts/gen-ltx2-res2s-goldens.py @@ -0,0 +1,410 @@ +#!/usr/bin/env python3 +"""Emit tests/vllm/models/ltx2_res2s_goldens.inc — the LTX-2.5 res_2s oracle. + +Row LTX25-RES2S-LOOP, issue #921, spec .agents/specs/ltx25-res2s-loop.md. + +Every number this writes is what UPSTREAM'S OWN CODE RETURNED. `phi`, +`get_res2s_coefficients`, `Res2sDiffusionStep`, `post_process_latent`, +`_get_new_noise`'s two normalization steps and +`res2s_audio_video_denoising_loop` are imported from a Lightricks/LTX-2 checkout +and run. Nothing here is transcribed from reading the source, and nothing is +recomputed by a local reimplementation — which for this module is the whole +point, because the values it is most important to pin are the ones a CORRECT +implementation gets wrong. `phi(2, -1e-10)` is 0.0 upstream, not 0.5, because +the formula cancels just outside its own 1e-10 guard; a Taylor series near zero +is numerically better and diverges from the model's runtime. + +THE FOUR SUBSTITUTIONS, each one a thing this port reproduces exactly: + +1. THE DENOISER. Upstream's loop takes a `Denoiser` callable and never reaches + for a model (samplers.py:214), so a fixed quadratic stands in for the 21B + transformer. It is quadratic and not affine on purpose: a build that + evaluated once per step and reused the result cannot land on the same + trajectory by luck. +2. THE NOISE DRAW. Upstream draws `torch.randn` on a seeded `torch.Generator`; + this port has `SplitMixGaussian` and does not have that stream. The draw is + replaced by the same deterministic pattern the C++ fixture uses, so the loop + ARITHMETIC around the injection is gated even though the stream is not. The + NORMALIZATION upstream applies after its draw is gated separately and + against upstream's own code (`kLtx2Res2sNoise*` below). +3. `model_dtype`. Upstream's loop declares `torch.bfloat16` (samplers.py:221); + this passes `torch.float32`, which is this port's model dtype on every + LTX-2.5 host path. Recorded as a divergence in ltx2_samplers.h rather than + hidden here. +4. TWO MEDIA-IO MODULES. `import ltx_pipelines.utils.samplers` pulls + `ltx_core.color.hlg`, which imports PyAV, and the image path imports + OpenImageIO. Neither is vendored here and nothing numeric touches either, so + both are stubbed as empty modules BEFORE the import. If upstream ever routes + a number through them this stub is what breaks, loudly, rather than a value + silently changing. + +Regenerate with: + + python3 scripts/gen-ltx2-res2s-goldens.py --ltx2 /path/to/LTX-2 \\ + --out tests/vllm/models/ltx2_res2s_goldens.inc + +and diff. The committed file is what this script emits at `fd4ded7f`; a +difference is either an upstream change or a defect in one of the two. +""" + +from __future__ import annotations + +import argparse +import math +import pathlib +import struct +import subprocess +import sys +import types + +PIN = "fd4ded7f" + +# ─── the fixture, which is INPUT and therefore stated here rather than read ─── +# +# Six elements, because the loop's arithmetic is elementwise and six is enough +# to carry a mask that is not all ones. The mask and the clean latent matter: +# with an all-ones mask `post_process_latent` is the identity and a build that +# dropped the blend passes. Three of the six positions are pinned to the clean +# latent, so a dropped blend fails at three positions rather than nowhere. +LATENT_COUNT = 6 +VIDEO_0 = [i / 6.0 for i in range(LATENT_COUNT)] +AUDIO_0 = [0.5 - i / 12.0 for i in range(LATENT_COUNT)] +MASK = [1.0, 1.0, 0.0, 1.0, 0.0, 1.0] +CLEAN = [-0.3, 0.2, 0.7, -0.1, 0.4, 0.05] + +# The raw vector `_get_new_noise`'s normalization is measured on. An INPUT, not +# an upstream output: what upstream produces from it is the golden below it. The +# expression is written out rather than the six doubles being pasted in, because +# a normalization golden is only meaningful beside the exact bits it was taken +# over, and `-1/3` and `((0*13+5)%17)/3 - 2` are the same number to fifteen +# digits and not to seventeen. +NOISE_RAW = [((i * 13 + 5) % 17) / 3.0 - 2.0 for i in range(6)] + +PHI_Z = [ + 0.0, -1e-12, -1e-11, -1e-10, -1e-09, -1e-08, -1e-06, + -0.001, -0.125, -0.25, -0.5, -1.0, -2.0, -5.0, +] +COEFF_H = [1e-12, 1e-10, 1e-08, 1e-06, 0.01, 0.125, 0.25, 0.5, 1.0, 3.0, 7.0] + +# name -> (sigmas, eta). Each forces one branch of the bong guard +# `bongmath and h < 0.5 and sigma > 0.03` (samplers.py:357) and says which: +# BongOn every h < 0.5 AND every sigma > 0.03 -> refines +# BongOffByH every h >= 0.5, every sigma > 0.03 -> off by h alone +# BongOffBySigma every h < 0.5, every sigma <= 0.03 -> off by sigma alone +# TerminalZero a schedule ending at 0 -> the injected 0.0011 +# Eta1 BongOn's schedule at eta 1.0 -> separates the loop's +# eta from the substep's +# pinned 0.5 (:273-274) +FIXTURES = [ + ("BongOn", [0.9, 0.8, 0.7, 0.62], 0.5), + ("BongOffByH", [0.9, 0.5, 0.25, 0.12], 0.5), + ("BongOffBySigma", [0.03, 0.028, 0.026, 0.025], 0.5), + ("TerminalZero", [1.0, 0.75, 0.5, 0.25, 0.0], 0.5), + ("Eta1", [0.9, 0.8, 0.7, 0.62], 1.0), +] + + +def git_revision(root: pathlib.Path) -> str: + """The checkout's SHA, and REFUSE a dirty tree. + + A SHA that does not describe the code that ran is worse than no SHA: it + reads as a pin while the oracle is whatever was in the working tree. + """ + head = subprocess.run( + ["git", "-C", str(root), "rev-parse", "HEAD"], + check=True, capture_output=True, text=True, + ).stdout.strip() + dirty = subprocess.run( + ["git", "-C", str(root), "status", "--short"], + check=True, capture_output=True, text=True, + ).stdout.strip() + if dirty: + raise SystemExit( + f"{root} has uncommitted changes; the goldens would carry a SHA that " + f"does not describe the code that produced them:\n{dirty}" + ) + return head + + +def load_upstream(root: pathlib.Path): + """Import upstream's own modules, with the two media-IO stubs.""" + for pkg in ("ltx-pipelines", "ltx-core"): + src = root / "packages" / pkg / "src" + if not src.is_dir(): + raise SystemExit(f"{src} is not a directory; is --ltx2 an LTX-2 checkout?") + sys.path.insert(0, str(src)) + for name in ("av", "av.video", "av.audio", "OpenImageIO"): + sys.modules.setdefault(name, types.ModuleType(name)) + + import torch # noqa: PLC0415 + from ltx_core.components.diffusion_steps import Res2sDiffusionStep # noqa: PLC0415 + from ltx_core.types import LatentState # noqa: PLC0415 + from ltx_pipelines.utils import samplers # noqa: PLC0415 + from ltx_pipelines.utils.res2s import get_res2s_coefficients, phi # noqa: PLC0415 + from ltx_pipelines.utils.types import DenoisedLatentResult # noqa: PLC0415 + + return types.SimpleNamespace( + torch=torch, + Res2sDiffusionStep=Res2sDiffusionStep, + LatentState=LatentState, + samplers=samplers, + phi=phi, + get_res2s_coefficients=get_res2s_coefficients, + DenoisedLatentResult=DenoisedLatentResult, + ) + + +def make_state(up, values, *, reversed_conditioning: bool): + """One modality's `LatentState`. + + THE AUDIO SIDE REVERSES THE MASK AND THE CLEAN LATENT. Not decoration: with + both modalities carrying the same conditioning, a build that fed one + stream's mask or clean latent to the other produces the identical result and + nothing says so. Reversed, the two disagree at four of six positions. + """ + torch = up.torch + mask = list(reversed(MASK)) if reversed_conditioning else MASK + clean = list(reversed(CLEAN)) if reversed_conditioning else CLEAN + return up.LatentState( + latent=torch.tensor([values], dtype=torch.float32), + denoise_mask=torch.tensor([mask], dtype=torch.float32), + positions=torch.zeros(1, LATENT_COUNT, dtype=torch.float32), + clean_latent=torch.tensor([clean], dtype=torch.float32), + ) + + +class QuadraticDenoiser: + """The stand-in for the 21B transformer, in upstream's `Denoiser` shape. + + `0.5x + 0.25 - 0.125x^2` on video and `-0.25x + 0.1 - 0.0625x^2` on audio, + in the model dtype and in that operation order, mirrored exactly by the C++ + fixture in tests/vllm/models/test_ltx2_pipeline.cpp. The two modalities get + DIFFERENT functions so a build that fed one stream's state to the other is + visible rather than symmetric. + """ + + def __init__(self, up): + self.up = up + self.eval_sigmas: list[float] = [] + self.eval_step_indices: list[int] = [] + + def __call__(self, transformer, video_state=None, audio_state=None, sigmas=None, step_index=0): + # `sigmas[step_index]` is what every upstream denoiser reduces the pair + # to (denoisers.py:237), so the record below is the sigma the model saw. + self.eval_sigmas.append(float(sigmas[step_index].item())) + self.eval_step_indices.append(int(step_index)) + result_v = None + result_a = None + if video_state is not None: + v = video_state.latent + result_v = self.up.DenoisedLatentResult(denoised=0.5 * v + 0.25 - 0.125 * (v * v)) + if audio_state is not None: + a = audio_state.latent + result_a = self.up.DenoisedLatentResult(denoised=-0.25 * a + 0.1 - 0.0625 * (a * a)) + return result_v, result_a + + +class PatternNoise: + """The stand-in for `torch.randn`, mirroring the C++ fixture's hook. + + STATEFUL PER GENERATOR, because upstream's generators advance. Within one + step the video injection and the audio injection are two draws from the SAME + generator, so they receive DIFFERENT tensors and the ORDER of the two calls + decides which modality gets which. A stateless stand-in makes swapping the + two injections invisible, which is exactly the mutation that survived until + this was made stateful. + + The two generators are told apart by their seed rather than by identity, + because the loop constructs them itself (samplers.py:267-268): the step + generator is seeded `noise_seed` and the substep generator + `noise_seed + 10000`. + """ + + def __init__(self, up, noise_seed: int): + self.up = up + self.step_seed = up.torch.Generator().manual_seed(noise_seed).initial_seed() + self.draws = {False: 0, True: 0} + + def __call__(self, x, generator): + substep = generator.initial_seed() != self.step_seed + draw = self.draws[substep] + count = x.numel() + values = [ + ((i * 7 + 3 + (1 if substep else 0) + 13 * draw) % 11) / 5.0 - 1.0 + for i in range(count) + ] + self.draws[substep] = draw + 1 + return self.up.torch.tensor(values, dtype=self.up.torch.float64).reshape(x.shape) + + +def run_loop(up, sigmas, eta, bongmath): + """One `res2s_audio_video_denoising_loop` call, at this port's model dtype.""" + torch = up.torch + denoiser = QuadraticDenoiser(up) + video_out, audio_out = up.samplers.res2s_audio_video_denoising_loop( + sigmas=torch.tensor(sigmas, dtype=torch.float32), + video_state=make_state(up, VIDEO_0, reversed_conditioning=False), + audio_state=make_state(up, AUDIO_0, reversed_conditioning=True), + stepper=up.Res2sDiffusionStep(), + transformer=None, + denoiser=denoiser, + eta=eta, + bongmath=bongmath, + new_noise_fn=PatternNoise(up, noise_seed=-1), + model_dtype=torch.float32, + ) + return { + "video": video_out.latent.reshape(-1).tolist(), + "audio": audio_out.latent.reshape(-1).tolist(), + "eval_sigmas": denoiser.eval_sigmas, + "eval_step_indices": denoiser.eval_step_indices, + "evaluations": len(denoiser.eval_sigmas), + } + + +# ─── emission ───────────────────────────────────────────────────────────────── + + +def f32(value: float) -> str: + """A float32 literal in the shortest form that round-trips at 9 digits. + + ROUNDED THROUGH float32 FIRST. A Python float is a double, so emitting + `0.9` where the C++ array holds `0.899999976f` would put a value in the + header that the compiler then rounds to something else — a golden that + describes the double the generator held rather than the float the test + compares. + """ + value = struct.unpack("f", struct.pack("f", value))[0] + text = f"{value:.9g}" + if "." not in text and "e" not in text and "E" not in text: + text += ".0" + return text + "f" + + +def f64(value: float) -> str: + return repr(float(value)) + + +def array(kind: str, name: str, values, fmt) -> str: + body = [fmt(v) for v in values] + lines = [] + for i in range(0, len(body), 3): + lines.append(" " + ", ".join(body[i:i + 3])) + return f"inline constexpr {kind} {name}[] = {{\n" + ",\n".join(lines) + "};\n" + + +def emit(up, revision: str) -> str: + torch = up.torch + out = [] + out.append( + f"// GENERATED from Lightricks/LTX-2 @ {revision[:8]} by\n" + "// scripts/gen-ltx2-res2s-goldens.py. Do not hand-edit.\n" + "#pragma once\n\n" + "#include \n\n" + "namespace vllm_test {\n\n" + ) + + out.append( + "// res2s.py:4-22. `phi(j, z)` at j = 1 and j = 2, INCLUDING the small-z\n" + "// cliff: the guard is `abs(z) < 1e-10` and outside it the formula\n" + "// cancels catastrophically, so upstream's own phi2(-1e-10) is 0.0 and\n" + "// phi2(-1e-8) is 1.1102230246251563. These are upstream's values, not a\n" + "// series expansion's, and a 'better' port fails here.\n" + ) + out.append(array("double", "kLtx2PhiZ", PHI_Z, f64)) + out.append(array("double", "kLtx2Phi1", [up.phi(1, z) for z in PHI_Z], f64)) + out.append(array("double", "kLtx2Phi2", [up.phi(2, z) for z in PHI_Z], f64)) + out.append(f"inline constexpr int64_t kLtx2PhiCount = {len(PHI_Z)};\n\n") + + coeffs = [up.get_res2s_coefficients(h, {}, 0.5) for h in COEFF_H] + out.append("// res2s.py:25-62, c2 = 0.5 (samplers.py:288).\n") + out.append(array("double", "kLtx2Res2sCoeffH", COEFF_H, f64)) + out.append(array("double", "kLtx2Res2sCoeffA21", [c[0] for c in coeffs], f64)) + out.append(array("double", "kLtx2Res2sCoeffB1", [c[1] for c in coeffs], f64)) + out.append(array("double", "kLtx2Res2sCoeffB2", [c[2] for c in coeffs], f64)) + out.append(f"inline constexpr int64_t kLtx2Res2sCoeffCount = {len(COEFF_H)};\n\n") + + for name, sigmas, eta in FIXTURES: + bong = run_loop(up, sigmas, eta, bongmath=True) + nobong = run_loop(up, sigmas, eta, bongmath=False) + hs = [ + -math.log(sigmas[i + 1] / sigmas[i]) + for i in range(len(sigmas) - 1) + if sigmas[i + 1] != 0.0 + ] + moved = bong["video"] != nobong["video"] + out.append( + f"// {name}: sigmas {sigmas}, eta {eta}, " + f"h [{', '.join(f'{h:.6f}' for h in hs)}], " + f"bong changed the result: {moved}\n" + ) + out.append(f"inline constexpr double kLtx2Res2s{name}Eta = {f64(eta)};\n") + out.append(array("float", f"kLtx2Res2s{name}Sigmas", sigmas, f32)) + out.append( + f"inline constexpr int64_t kLtx2Res2s{name}SigmaCount = {len(sigmas)};\n" + f"inline constexpr int64_t kLtx2Res2s{name}Evaluations = {bong['evaluations']};\n" + ) + out.append(array("double", f"kLtx2Res2s{name}EvalSigmas", bong["eval_sigmas"], f64)) + # The `step_index` each call was handed, in call order. NOT the position + # in this vector: upstream passes `step_idx`, then a literal 0 for the + # substep, then `n_full_steps` for the terminal evaluation + # (samplers.py:301, :385, :437). The denoiser reads it through + # `should_skip_step` (guiders.py:287-291), so it decides which + # evaluations run a forward at all on a request with `skip_step != 0`. + out.append( + array("int64_t", f"kLtx2Res2s{name}EvalStepIndices", bong["eval_step_indices"], str) + ) + out.append(array("float", f"kLtx2Res2s{name}Video", bong["video"], f32)) + out.append(array("float", f"kLtx2Res2s{name}Audio", bong["audio"], f32)) + out.append( + f"inline constexpr bool kLtx2Res2s{name}BongMoved = " + f"{'true' if moved else 'false'};\n" + ) + out.append(array("float", f"kLtx2Res2s{name}NoBongVideo", nobong["video"], f32)) + out.append("\n") + + out.append(f"inline constexpr int64_t kLtx2Res2sLatentCount = {LATENT_COUNT};\n") + out.append(array("float", "kLtx2Res2sVideo0", VIDEO_0, f32)) + out.append(array("float", "kLtx2Res2sAudio0", AUDIO_0, f32)) + out.append(array("float", "kLtx2Res2sMask", MASK, f32)) + out.append(array("float", "kLtx2Res2sClean", CLEAN, f32)) + out.append("\n") + + raw = torch.tensor([NOISE_RAW], dtype=torch.float64) + normalized = up.samplers._channelwise_normalize( # noqa: SLF001 + (raw - raw.mean()) / raw.std() + ) + out.append( + "// samplers.py:160-170. `_get_new_noise` normalizes globally and then\n" + "// channelwise; the DRAW itself is torch.randn, whose stream this port\n" + "// does not have, so only the normalization is gated.\n" + ) + out.append(array("double", "kLtx2Res2sNoiseRaw", NOISE_RAW, f64)) + out.append( + array("double", "kLtx2Res2sNoiseNormalized", normalized.reshape(-1).tolist(), f64) + ) + out.append("\n} // namespace vllm_test\n") + return "".join(out) + + +def main() -> None: + ap = argparse.ArgumentParser() + ap.add_argument("--ltx2", required=True, type=pathlib.Path, + help="a Lightricks/LTX-2 checkout at the pinned revision") + ap.add_argument("--out", required=True, type=pathlib.Path) + args = ap.parse_args() + + revision = git_revision(args.ltx2) + if not revision.startswith(PIN): + raise SystemExit( + f"{args.ltx2} is at {revision}, not the pinned {PIN}. Advancing the pin " + "reconciles the row's spec and every gate that reads these goldens; it is " + "not something this generator may do silently." + ) + up = load_upstream(args.ltx2) + args.out.write_text(emit(up, revision)) + print(f"wrote {args.out}") + + +if __name__ == "__main__": + main() diff --git a/scripts/mutation-harness.py b/scripts/mutation-harness.py new file mode 100644 index 000000000..009ac1600 --- /dev/null +++ b/scripts/mutation-harness.py @@ -0,0 +1,245 @@ +#!/usr/bin/env python3 +"""Run a mutation pass and print the FOUR facts that stop a false green. + +Row LTX25-RES2S-LOOP, issue #921. Written for that row's section 8 and kept +general, because every one of the four failure modes below was paid for here in +a different file. + +A mutation pass answers one question: does the gate detect this defect? The +answer is the test binary's exit code, and there are four distinct ways to get +exit 0 from a mutation that proved nothing: + +1. THE MUTATION NEVER APPLIED. The anchor text moved, or was in the header + rather than the .cpp. `git diff --stat` is empty, the build is clean, the + exit code is 0, and it reads exactly like a passing test. This harness + REFUSES the run when the anchor is absent, and prints the diffstat when it is + not. +2. THE MUTATION DID NOT BUILD. A failed compile leaves the previous binary on + disk and running it reports a pass over unmutated code. This harness prints + whether the build succeeded and how many `: error:` lines it emitted, and + marks a non-building mutation `BUILD_FAILED` rather than scoring it. +3. THE FILTER MATCHED NOTHING. doctest's `--test-case` splits its argument on + COMMAS, so a case name containing one is truncated and matches zero cases — + and doctest then prints `SUCCESS!` and exits 0. This harness runs the WHOLE + BINARY by default and asserts a NON-ZERO case count and a non-zero assertion + count before it will call anything a survivor. +4. THE BINARY WAS NOT REBUILT. `git checkout --` restores a file with an old + mtime, so ninja can decide the object is current and carry the previous + mutation's binary forward. Every restore here re-stamps the file with + `os.utime(None)`. + +Usage: + + python3 scripts/mutation-harness.py --build build \\ + --test test_ltx2_pipeline \\ + --mutation "res2s-second-eval:src/vllm/model_executor/models/ltx2_samplers.cpp:\\ +hooks.denoise(mid_v, mid_a, sub_sigma:hooks.denoise(video.latent, audio.latent, sigma" + +Each `--mutation` is `NAME:PATH:FIND:REPLACE` (the first two colons split; +FIND and REPLACE are separated by the last colon-free split, so pass them with +`--find`/`--replace` when either contains a colon). A mutation file may also be +supplied with `--plan FILE`, one JSON object per line: + + {"name": "...", "path": "...", "find": "...", "replace": "..."} + +The tree is restored byte-for-byte after every mutation, verified by sha256, and +the harness refuses to start on a dirty working tree so a restore failure cannot +be mistaken for the developer's own edit. +""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import os +import pathlib +import re +import subprocess +import sys + +CASE_RE = re.compile(r"test cases:\s*(\d+)\s*\|\s*(\d+) passed\s*\|\s*(\d+) failed") +ASSERT_RE = re.compile(r"assertions:\s*(\d+)\s*\|\s*(\d+) passed\s*\|\s*(\d+) failed") + + +def sha256(path: pathlib.Path) -> str: + return hashlib.sha256(path.read_bytes()).hexdigest() + + +def require_clean(root: pathlib.Path) -> None: + dirty = subprocess.run( + ["git", "-C", str(root), "status", "--porcelain"], + check=True, capture_output=True, text=True, + ).stdout.strip() + if dirty: + raise SystemExit( + "the working tree is dirty. A mutation harness that starts from " + "uncommitted edits cannot tell its own restore failure from your work:\n" + + dirty + ) + + +def diffstat(root: pathlib.Path) -> str: + return subprocess.run( + ["git", "-C", str(root), "diff", "--stat"], + check=True, capture_output=True, text=True, + ).stdout.strip().replace("\n", " ; ") + + +def build(build_dir: pathlib.Path, target: str) -> tuple[bool, int, str]: + """Returns (built, compile_error_count, tail).""" + proc = subprocess.run( + ["cmake", "--build", str(build_dir), "--target", target, "-j", "6"], + capture_output=True, text=True, + ) + text = proc.stdout + proc.stderr + return proc.returncode == 0, text.count(": error:"), text[-1200:] + + +def run_binary(build_dir: pathlib.Path, target: str) -> dict: + """Run the WHOLE binary. Exit code captured directly, never after a pipe.""" + binary = build_dir / "tests" / target + if not binary.is_file(): + return {"exit": None, "cases": 0, "failed_cases": 0, "asserts": 0, + "failed_asserts": 0, "note": f"{binary} does not exist"} + proc = subprocess.run([str(binary)], capture_output=True, text=True) + code = proc.returncode + text = proc.stdout + proc.stderr + cases = CASE_RE.search(text) + asserts = ASSERT_RE.search(text) + return { + "exit": code, + "cases": int(cases.group(1)) if cases else 0, + "failed_cases": int(cases.group(3)) if cases else 0, + "asserts": int(asserts.group(1)) if asserts else 0, + "failed_asserts": int(asserts.group(3)) if asserts else 0, + # A thrown doctest case prints `0 failed` beside `Status: FAILURE!`, so + # the summary line is not the authority and the exit code is. + "status_failure": "Status: FAILURE!" in text, + "note": "", + } + + +def restore(root: pathlib.Path, path: str, want_sha: str) -> None: + subprocess.run(["git", "-C", str(root), "checkout", "--", path], check=True) + full = root / path + # RE-STAMP. A restored file older than its object makes ninja skip the + # rebuild and carry the previous mutation's binary into the next run. + os.utime(full, None) + got = sha256(full) + if got != want_sha: + raise SystemExit(f"restore of {path} did not reproduce the original: {got} != {want_sha}") + + +def apply_mutation(root: pathlib.Path, path: str, find: str, replace: str) -> bool: + full = root / path + if not full.is_file(): + print(f" ANCHOR NOT FOUND: {path} does not exist") + return False + text = full.read_text() + hits = text.count(find) + if hits != 1: + print(f" ANCHOR NOT FOUND: {hits} occurrences of the find text in {path} " + f"(exactly one is required, so a moved or duplicated anchor is a " + f"refusal rather than a silent no-op)") + return False + full.write_text(text.replace(find, replace, 1)) + os.utime(full, None) + return True + + +def parse_mutations(args) -> list[dict]: + out: list[dict] = [] + if args.plan: + for line in pathlib.Path(args.plan).read_text().splitlines(): + line = line.strip() + if line and not line.startswith("#"): + out.append(json.loads(line)) + if args.name: + out.append({"name": args.name, "path": args.path, + "find": args.find, "replace": args.replace}) + if not out: + raise SystemExit("no mutations: pass --plan or --name/--path/--find/--replace") + return out + + +def main() -> None: + ap = argparse.ArgumentParser() + ap.add_argument("--root", type=pathlib.Path, default=pathlib.Path.cwd()) + ap.add_argument("--build", type=pathlib.Path, required=True) + ap.add_argument("--test", required=True, help="the ctest/doctest binary target name") + ap.add_argument("--plan", help="a file of one JSON mutation per line") + ap.add_argument("--name") + ap.add_argument("--path") + ap.add_argument("--find") + ap.add_argument("--replace", default="") + args = ap.parse_args() + + root = args.root.resolve() + require_clean(root) + mutations = parse_mutations(args) + + # THE BASELINE IS PART OF THE EVIDENCE. A suite that is already red, or that + # runs zero cases, makes every mutation below unreadable. + ok, errors, tail = build(args.build, args.test) + if not ok: + raise SystemExit(f"the UNMUTATED tree does not build ({errors} errors):\n{tail}") + base = run_binary(args.build, args.test) + print(f"BASELINE {args.test}: exit={base['exit']} cases={base['cases']} " + f"({base['failed_cases']} failed) assertions={base['asserts']} " + f"({base['failed_asserts']} failed)") + if base["exit"] != 0 or base["cases"] == 0 or base["asserts"] == 0: + raise SystemExit( + "the baseline is not a clean, non-empty green. Zero cases or zero " + "assertions is a SKIP wearing a pass, and every mutation below would " + "read as a survivor." + ) + + rows = [] + for mutation in mutations: + name, path = mutation["name"], mutation["path"] + print(f"\n--- {name} ({path})") + want_sha = sha256(root / path) + if not apply_mutation(root, path, mutation["find"], mutation["replace"]): + rows.append((name, "-", "-", "-", "-", "-", "ANCHOR NOT FOUND")) + continue + stat = diffstat(root) + try: + built, errors, tail = build(args.build, args.test) + if not built: + print(f" diff: {stat}\n BUILT: NO compile_err: {errors}\n{tail}") + rows.append((name, stat, "NO", str(errors), "-", "-", "BUILD_FAILED")) + continue + result = run_binary(args.build, args.test) + code = result["exit"] + print(f" diff: {stat}") + print(f" BUILT: YES compile_err: {errors}") + print(f" EXIT: {code} cases: {result['cases']}/{result['failed_cases']}F " + f"assertions: {result['asserts']}/{result['failed_asserts']}F " + f"status_failure: {result['status_failure']}") + if result["cases"] == 0 or result["asserts"] == 0: + verdict = "NO CASES RAN" + elif code != 0: + verdict = "DETECTED" + else: + verdict = "SURVIVED" + print(f" VERDICT: {verdict}") + rows.append((name, stat, "YES", str(errors), str(code), + f"{result['cases']}/{result['failed_cases']}F " + f"{result['asserts']}/{result['failed_asserts']}F", verdict)) + finally: + restore(root, path, want_sha) + + print("\n| # | Mutation | diff --stat | BUILT | cc-err | EXIT | cases/asserts | Verdict |") + print("|---|---|---|---|---|---|---|---|") + for i, row in enumerate(rows, 1): + print(f"| M{i} | " + " | ".join(row) + " |") + + # Rebuild once at the end so the tree the developer is left with matches the + # sources, rather than the last mutation's objects. + build(args.build, args.test) + sys.exit(0 if all(r[-1] == "DETECTED" for r in rows) else 1) + + +if __name__ == "__main__": + main() diff --git a/src/vllm/model_executor/models/ltx2_pipeline.cpp b/src/vllm/model_executor/models/ltx2_pipeline.cpp index 7291ecbb5..ba13d4b06 100644 --- a/src/vllm/model_executor/models/ltx2_pipeline.cpp +++ b/src/vllm/model_executor/models/ltx2_pipeline.cpp @@ -304,23 +304,32 @@ std::vector Ltx2EulerAncestralStep(const float* sample, const float* deno return out; } -Ltx2SdeCoeff Ltx2Res2sSdeCoeff(double sigma_next, double sigma_up) { - // diffusion_steps.py:136-155, the `sigma_up is not None` arm — the only one - // `Res2sDiffusionStep.step` reaches (:179). - const float next = static_cast(sigma_next); - float up = static_cast(sigma_up); - up = std::min(up, next * static_cast(kLtx2Res2sSigmaUpClamp)); +namespace { - const float sigma_signal = 1.0f - next; // `sigmax` defaults to ones_like - const float residual = std::sqrt(std::max(next * next - up * up, 0.0f)); - float alpha_ratio = sigma_signal + residual; - float down = residual / alpha_ratio; +// `Res2sDiffusionStep.get_sde_coeff` (diffusion_steps.py:136-155), the +// `sigma_up is not None` arm — the only one `step` reaches (:179). +// +// TEMPLATED ON THE SIGMA TYPE, because upstream's has no dtype of its own and +// the res_2s loop reaches it at TWO precisions: float64 from the substep's +// `[sigma, sub_sigma]` pair (samplers.py:342) and float32 from the loop's own +// schedule at step level (samplers.py:415). Instantiating one formula twice is +// what keeps that from becoming a second copy. +template +Ltx2SdeCoeff Res2sSdeCoeffImpl(double sigma_next, double sigma_up) { + const Sigma next = static_cast(sigma_next); + Sigma up = static_cast(sigma_up); + up = std::min(up, next * static_cast(kLtx2Res2sSigmaUpClamp)); + + const Sigma sigma_signal = static_cast(1) - next; // `sigmax` defaults to ones_like + const Sigma residual = std::sqrt(std::max(next * next - up * up, static_cast(0))); + Sigma alpha_ratio = sigma_signal + residual; + Sigma down = residual / alpha_ratio; // :149-153 — the NaN scrubbing, which is what keeps a degenerate schedule from // poisoning the whole latent. - if (std::isnan(up)) up = 0.0f; + if (std::isnan(up)) up = static_cast(0); if (std::isnan(down)) down = next; - if (std::isnan(alpha_ratio)) alpha_ratio = 1.0f; + if (std::isnan(alpha_ratio)) alpha_ratio = static_cast(1); Ltx2SdeCoeff coeff; coeff.alpha_ratio = alpha_ratio; @@ -329,36 +338,85 @@ Ltx2SdeCoeff Ltx2Res2sSdeCoeff(double sigma_next, double sigma_up) { return coeff; } -std::vector Ltx2Res2sStep(const float* sample, const float* denoised, - const float* sigmas, int64_t sigma_count, int64_t step_index, - int64_t count, const float* noise, double eta) { +// `Res2sDiffusionStep.step` (diffusion_steps.py:157-190). `Sigma` is the width +// the SCHEDULE and therefore the coefficients are computed at; `Value` is the +// width of the sample, the noise and the result. Upstream reaches three +// combinations across this port's call sites and they are the three +// instantiations below. +template +std::vector Res2sStepImpl(const Value* sample, const Value* denoised, + const Sigma* sigmas, int64_t sigma_count, int64_t step_index, + int64_t count, const Value* noise, double eta) { RequireStepIndex(sigma_count, step_index); - const float sigma = sigmas[step_index]; - const float sigma_next = sigmas[step_index + 1]; - const Ltx2SdeCoeff coeff = - Ltx2Res2sSdeCoeff(sigma_next, static_cast(sigma_next * static_cast(eta))); + const Sigma sigma = sigmas[step_index]; + const Sigma sigma_next = sigmas[step_index + 1]; + const Ltx2SdeCoeff coeff = Res2sSdeCoeffImpl( + static_cast(sigma_next), + static_cast(sigma_next * static_cast(eta))); // :181-182 — returned UNCHANGED, not cast, when either is zero. - if (coeff.sigma_up == 0.0 || sigma_next == 0.0f) { - return std::vector(denoised, denoised + count); + if (coeff.sigma_up == 0.0 || sigma_next == static_cast(0)) { + return std::vector(denoised, denoised + count); } Require(noise != nullptr, "ltx2 Res2s step: requires a noise tensor"); - const float alpha_ratio = static_cast(coeff.alpha_ratio); - const float sigma_down = static_cast(coeff.sigma_down); - const float sigma_up = static_cast(coeff.sigma_up); - const float denom = sigma - sigma_next; + const Sigma alpha_ratio = static_cast(coeff.alpha_ratio); + const Sigma sigma_down = static_cast(coeff.sigma_down); + const Sigma sigma_up = static_cast(coeff.sigma_up); + // The SUBTRACTION happens at the schedule's own width, which is what upstream + // does: `sigma - sigma_next` is a tensor op between two schedule entries + // before the f64 numerator ever divides by it (diffusion_steps.py:185). + const Sigma denom = sigma - sigma_next; - std::vector out(static_cast(count)); + std::vector out(static_cast(count)); for (int64_t i = 0; i < count; ++i) { const size_t k = static_cast(i); - const float eps_next = (sample[k] - denoised[k]) / denom; - const float denoised_next = sample[k] - sigma * eps_next; - out[k] = alpha_ratio * (denoised_next + sigma_down * eps_next) + sigma_up * noise[k]; + const Value eps_next = (sample[k] - denoised[k]) / static_cast(denom); + const Value denoised_next = sample[k] - static_cast(sigma) * eps_next; + out[k] = static_cast(alpha_ratio) * + (denoised_next + static_cast(sigma_down) * eps_next) + + static_cast(sigma_up) * noise[k]; } return out; } +} // namespace + +Ltx2SdeCoeff Ltx2Res2sSdeCoeff(double sigma_next, double sigma_up) { + return Res2sSdeCoeffImpl(sigma_next, sigma_up); +} + +Ltx2SdeCoeff Ltx2Res2sSdeCoeffHp(double sigma_next, double sigma_up) { + return Res2sSdeCoeffImpl(sigma_next, sigma_up); +} + +std::vector Ltx2Res2sStep(const float* sample, const float* denoised, + const float* sigmas, int64_t sigma_count, int64_t step_index, + int64_t count, const float* noise, double eta) { + return Res2sStepImpl(sample, denoised, sigmas, sigma_count, step_index, count, + noise, eta); +} + +std::vector Ltx2Res2sStepHp(const double* sample, const double* denoised, + const double* sigmas, int64_t sigma_count, + int64_t step_index, int64_t count, const double* noise, + double eta, Ltx2Res2sScheduleWidth width) { + if (width == Ltx2Res2sScheduleWidth::kF64Schedule) { + return Res2sStepImpl(sample, denoised, sigmas, sigma_count, step_index, + count, noise, eta); + } + // The step-level arm. The schedule really is float32 upstream, so it is + // narrowed HERE rather than at the call site: narrowing at the call site would + // put the conversion one frame away from the arithmetic it changes, and the + // next reader would have to reconstruct which of the two widths ran. + std::vector narrowed(static_cast(sigma_count)); + for (int64_t i = 0; i < sigma_count; ++i) { + narrowed[static_cast(i)] = static_cast(sigmas[i]); + } + return Res2sStepImpl(sample, denoised, narrowed.data(), sigma_count, + step_index, count, noise, eta); +} + Ltx2AncestralSigmas Ltx2AncestralStep(double sigma_from, double sigma_to, double eta) { Ltx2AncestralSigmas result; // :17-18 — `if not eta`, i.e. exactly 0.0, short-circuits before any division. @@ -1283,6 +1341,134 @@ Ltx2PipelineRecipe RetakeRecipe(const std::string& version) { return recipe; } +// `TI2VidTwoStagesHQPipeline` (ti2vid_two_stages_hq.py:59, `__call__` at :174). +// Row LTX25-RES2S-LOOP, issue #921. +// +// ─── WHAT MAKES IT HQ, VERIFIED RATHER THAN INHERITED ──────────────────────── +// Two of the differences from `TI2VidTwoStagesPipeline` are the SAMPLER: +// `stepper=Res2sDiffusionStep()` (:258) and +// `loop=res2s_audio_video_denoising_loop` passed to BOTH stages (:292, :335). +// A third is `LTX_2_3_HQ_PARAMS` (utils/constants.py:95-115). THEY ARE NOT THE +// ONLY THREE, and this comment said they were until 2026-08-17. Diffing the two +// files at `fd4ded7f` also shows: stage 1 loads the distilled LoRA at +// `distilled_lora_strength_stage_1` (:92-101, :151-154) where the plain +// pipeline loads none on that stage; stage 1's schedule is derived as +// `execute(latent=empty_latent, steps=...)` (:260-267) against the plain +// pipeline's `execute(steps=...)`, which `schedulers.py:32` turns into a +// resolution-dependent shift instead of the 4096-token default; and +// `GuidedDenoiser` (:271-281) replaces `FactoryGuidedDenoiser`. +// +// Two of those four are ALREADY what this recipe does and one is out of scope. +// The schedule: this engine always derives from `target_tokens` +// (`ltx2_video.cpp`'s `Ltx2SigmaSchedule` call), which is the latent-aware form, +// so stage 1 coincides with upstream here — the divergence, if any, is on the +// PLAIN two-stage arm and is not this recipe's to move. The denoiser: stage 1's +// `video_guidance` below reaches `Ltx2GuidedDenoise`, which is +// `_guided_denoise` — the one function `GuidedDenoiser` and +// `FactoryGuidedDenoiser` share (utils/denoisers.py:61-211) — so the difference +// between the two upstream classes is WHERE the params come from and not what +// runs. The distilled LoRA per stage is out of scope for every LTX row here and +// is named in the row's spec section 2 rather than silently absent. +// +// So this recipe is NOT the distilled two-stage one with different numbers. The +// res_2s loop evaluates the transformer TWICE per step, which is the whole +// reason the preset can afford 15 steps against the 2.4 lineage's 30. A recipe +// that carried these guidance scales and this step count on `kEuler` would +// render a finished, plausible, correctly-sized clip at half the model +// evaluations it was tuned for, and no output check could tell. +// +// ─── 2.5 ONLY, AND NOT BY ANALOGY WITH THE ONE-STAGE ROWS ──────────────────── +// `LTX_2_3_HQ_PARAMS` is a plain constant, not a `replace` of a neighbour, and +// upstream says why in its own comment (constants.py:91-94): "it overrides every +// knob that varies between generations, so there is nothing for it to inherit +// from a detected checkpoint". There is therefore no `detect_params` lineage to +// spread this across versions the way `one_stage` and `t2a_one_stage` are +// spread, and the one generation-dependent value it does NOT carry — +// `default_image_crf` — is resolved from the checkpoint by the pipeline's own +// `ImageConditioner` (the same comment), which is `Ltx2DetectPipelineParams` +// here. +Ltx2PipelineRecipe Res2sTwoStageRecipe(const std::string& version) { + Ltx2PipelineRecipe recipe; + const Ltx2PipelineParams params = Ltx2Params23Hq(); + + Ltx2PhaseRecipe stage1; + stage1.name = "generate_lowres_hq"; + // :238-243 — `width // 2, height // 2`, the same halving the distilled + // two-stage arm applies. + stage1.spatial_downscale = 2; + // :260-267 — `stage_1_sigmas` defaults to None and is then built by + // `LTX2Scheduler().execute(latent, steps=num_inference_steps)`. So this phase + // has NO frozen schedule: it is derived, and 15 steps is what derives it. This + // is the one place this recipe differs in KIND from the distilled two-stage + // one, whose stage 1 carries `DISTILLED_SIGMAS` and cannot honour a step + // override. + stage1.noise_scale = 1.0; + // :271-281 — a `GuidedDenoiser` with a negative context and the HQ guider + // params, so guidance is live and a request may override it. + stage1.video_guidance = params.video_guider; + stage1.audio_guidance = params.audio_guider; + stage1.stepper = Ltx2StepperKind::kRes2s; + + Ltx2PhaseRecipe stage2; + stage2.name = "refine_hq"; + // :193 — `stage_2_sigmas: torch.Tensor = STAGE_2_DISTILLED_SIGMAS`, a DEFAULT + // ARGUMENT, so the schedule is frozen for this phase even though stage 1's is + // not. + stage2.sigmas = Stage2DistilledSigmas(); + // :327, :332 — both modality specs re-noise to `stage_2_sigmas[0].item()`, + // which is what makes the upsampled latent valid at that noise level. + stage2.noise_scale = Stage2DistilledSigmas().front(); + // :297 — `self.upsampler(video_state.latent[:1])`. + stage2.input_transform = Ltx2PhaseInputTransform::kSpatialUpsample; + // :316 — `SimpleDenoiser`, "single transformer call, no guidance" + // (utils/denoisers.py:215). Nothing a request sends can turn guidance back on. + stage2.allow_guidance_override = false; + stage2.use_official_sigma_schedule = false; + // :319/:335 — the SAME stepper and the SAME loop as stage 1. This is where the + // HQ pipeline parts company with the distilled two-stage one, whose stage 2 is + // always deterministic Euler because a 3-step refinement cannot remove freshly + // injected noise (distilled.py:206-209). That argument does not transfer: the + // HQ pipeline passes `stepper` and `loop` to `self.stage_2` explicitly, and + // "the schedule is short" is not a reason this port may substitute a different + // sampler than the one upstream hands it. + stage2.stepper = Ltx2StepperKind::kRes2s; + + recipe.phases = {stage1, stage2}; + // `assert_resolution(is_two_stage=True)` (:199), and the arguments describe + // the FINAL output — stage 1 runs at half of it. + recipe.height = params.stage_2_height(); + recipe.width = params.stage_2_width(); + recipe.num_frames = params.num_frames; + recipe.frame_rate = params.frame_rate; + // constants.py:96 — 15, against the 2.4 lineage's 30. Half the steps, and + // twice the evaluations per step. + recipe.num_inference_steps = params.num_inference_steps; + // NOT from the HQ params: constants.py:91-94 says `default_image_crf` is the + // one generation-dependent value this preset does not fix, and that the + // pipeline resolves it from the checkpoint instead. + recipe.default_image_crf = Ltx2DetectPipelineParams(version).default_image_crf; + // :210 — `self.prompt_encoder([prompt, negative_prompt], ...)`, and stage 1's + // guider consumes the negative encoding. Unlike the distilled arm, this + // pipeline HAS a negative prompt. + recipe.negative_prompt = LightricksNegativePrompt(); + recipe.video_output_phase = 1; + // :313-314 — "Stage 2 refines video only; discard its audio", so the audio + // that leaves is STAGE 1's. `video_state, _ = self.stage_2(...)` at :315 is + // the discard, and `self.audio_decoder(audio_state.latent)` at :339 reads the + // name stage 1 bound. Writing 1 here would decode the audio the pipeline + // throws away — a soundtrack that is finite, the right length and the wrong + // take. + recipe.audio_output_phase = 0; + // Stage 1's schedule really is derived from `num_inference_steps`, so a + // request may set it. Stage 2's is frozen by its own default argument and is + // unaffected either way, exactly as upstream's two parameters are. + recipe.allow_request_sigmas = true; + recipe.allow_request_latents = true; + recipe.allow_negative_prompt = true; + recipe.fixed_num_inference_steps = false; + return recipe; +} + } // namespace Ltx2PipelineRecipe ResolveLtx2PipelineRecipe(const std::string& pipeline_kind, @@ -1311,6 +1497,11 @@ Ltx2PipelineRecipe ResolveLtx2PipelineRecipe(const std::string& pipeline_kind, // resolving DFR onto it would build a recipe whose first stage the engine // must then refuse at load. Refusing at the recipe table names the version. if (model_version == "2.5") return DfrRecipe(model_version); + } else if (pipeline_kind == "res2s_two_stage") { + // 2.5 only — see `Res2sTwoStageRecipe`: `LTX_2_3_HQ_PARAMS` is a plain + // constant with no `detect_params` lineage, so there is no second version to + // resolve it onto. + if (model_version == "2.5") return Res2sTwoStageRecipe(model_version); } else if (pipeline_kind == "dmd2") { if (model_version == "2" || model_version == "2.3") return PositiveOnlyRecipe(); } else if (pipeline_kind == "retake") { diff --git a/src/vllm/model_executor/models/ltx2_samplers.cpp b/src/vllm/model_executor/models/ltx2_samplers.cpp new file mode 100644 index 000000000..5026933f4 --- /dev/null +++ b/src/vllm/model_executor/models/ltx2_samplers.cpp @@ -0,0 +1,407 @@ +// LTX-2.5 SAMPLERS — the res_2s second-order denoising loop. +// +// Row: LTX25-RES2S-LOOP. Spec: .agents/specs/ltx25-res2s-loop.md. Issue #921. +// Ported from Lightricks/LTX-2 @ fd4ded7f, +// packages/ltx-pipelines/src/ltx_pipelines/utils/{res2s,samplers}.py. +// +// The header carries the port map, the dtype argument and the warning about +// `phi`. This file carries the arithmetic, anchored line by line. +#include "vllm/model_executor/models/ltx2_samplers.h" + +#include +#include +#include +#include +#include + +#include "vllm/model_executor/models/ltx2_pipeline.h" + +namespace vllm { +namespace { + +[[noreturn]] void Refuse(const std::string& message) { throw std::runtime_error(message); } + +void Require(bool condition, const std::string& message) { + if (!condition) Refuse(message); +} + +// `math.factorial` over the only two values `get_res2s_coefficients` reaches. +double Factorial(int64_t j) { + double out = 1.0; + for (int64_t k = 2; k <= j; ++k) out *= static_cast(k); + return out; +} + +std::vector ToHp(const std::vector& x) { + return std::vector(x.begin(), x.end()); +} + +// `.to(model_dtype)` (samplers.py:370, :375, :431, :433, :442, :445). +std::vector ToModelDtype(const std::vector& x) { + std::vector out(x.size()); + for (size_t i = 0; i < x.size(); ++i) out[i] = static_cast(x[i]); + return out; +} + +} // namespace + +// --------------------------------------------------------------------------- +// The exponential integrator (utils/res2s.py:4-62) +// --------------------------------------------------------------------------- + +double Ltx2Phi(int64_t j, double neg_h) { + Require(j >= 1, "ltx2 res2s phi: j must be >= 1, got " + std::to_string(j)); + // res2s.py:13-16. The threshold is EXACTLY 1e-10 and the comparison is + // STRICT, so -1e-10 itself takes the formula branch below and returns + // upstream's cancelled value rather than 1/j!. See the header: this is a + // guard against dividing by zero, NOT a series expansion, and replacing it + // with one would make this port disagree with the model's own runtime. + if (std::fabs(neg_h) < 1e-10) return 1.0 / Factorial(j); + + // res2s.py:19 — the remainder sum_{k key{j, neg_h}; + const auto hit = phi_cache.find(key); + if (hit != phi_cache.end()) return hit->second; + const double result = Ltx2Phi(j, neg_h); + phi_cache.emplace(key, result); + return result; + }; + + Ltx2Res2sCoefficients coeff; + // res2s.py:48-50 — a21 = c2 * phi_1(-h * c2). + const double neg_h_c2 = -h * c2; + coeff.a21 = c2 * get_phi(1, neg_h_c2); + // res2s.py:54-56 — b2 = phi_2(-h) / c2. + const double neg_h_full = -h; + coeff.b2 = get_phi(2, neg_h_full) / c2; + // res2s.py:59-60 — b1 = phi_1(-h) - b2. IN THIS ORDER: b2 is computed first + // upstream and b1 is defined against it, so an implementation that derived b2 + // from b1 would invert the dependency and reorder the cache insertions. + coeff.b1 = get_phi(1, neg_h_full) - coeff.b2; + return coeff; +} + +// --------------------------------------------------------------------------- +// The noise (utils/samplers.py:155-170) +// --------------------------------------------------------------------------- + +std::vector Ltx2Res2sNormalizeNoise(std::vector noise) { + Require(noise.size() >= 2, + "ltx2 res2s noise: the normalization divides by an UNBIASED standard " + "deviation (torch's default), which is undefined for fewer than 2 elements"); + const auto normalize = [](std::vector& x) { + const double n = static_cast(x.size()); + const double mean = std::accumulate(x.begin(), x.end(), 0.0) / n; + double sq = 0.0; + for (const double v : x) sq += (v - mean) * (v - mean); + // torch's `Tensor.std()` is UNBIASED by default: the denominator is n - 1. + const double sd = std::sqrt(sq / (n - 1.0)); + for (double& v : x) v = (v - mean) / sd; + }; + // samplers.py:169 — the global normalize... + normalize(noise); + // ...and :170 -> :160-161, `_channelwise_normalize` over the last two dims. + // On this port's rank-2 [tokens, width] latent those two dims ARE every + // element, so this repeats the operation above and is the identity up to + // rounding. Applied anyway, in upstream's order: idempotence here is a + // property of THIS port's rank, not of the function, and dropping the call + // would be a divergence that a batched latent would make visible. + normalize(noise); + return noise; +} + +// --------------------------------------------------------------------------- +// The loop (utils/samplers.py:208-447) +// --------------------------------------------------------------------------- + +Ltx2Res2sLoopStats Ltx2Res2sDenoisingLoop(const std::vector& sigmas_in, + Ltx2Res2sModality& video, + Ltx2Res2sModality& audio, + const Ltx2Res2sHooks& hooks, + const Ltx2Res2sLoopParams& params) { + // samplers.py:257-259. + Require(video.present || audio.present, + "ltx2 res2s loop: at least one of video_state or audio_state must be provided " + "(samplers.py:258-259)"); + Require(static_cast(hooks.denoise) && static_cast(hooks.post_process) && + static_cast(hooks.new_noise), + "ltx2 res2s loop: the denoiser, post_process_latent and new_noise hooks are all " + "required. Two of the three are upstream PARAMETERS — `denoiser` (samplers.py:214) and " + "`new_noise_fn` (:220); `post_process_latent` is a module-level import upstream calls " + "directly (:305, :390, :441), and it is a hook here only because the engine already " + "owns the mask and the clean latent"); + Require(sigmas_in.size() >= 2, + "ltx2 res2s loop: a schedule needs at least two sigmas, got " + + std::to_string(sigmas_in.size())); + + Ltx2Res2sLoopStats stats; + // samplers.py:279. TAKEN BEFORE THE INJECTION BELOW, which is why it still + // counts the CALLER's steps after the schedule grows by one. + stats.full_steps = static_cast(sigmas_in.size()) - 1; + + // samplers.py:280-282 — "inject minimal sigma value to avoid division by + // zero". The zero is REPLACED by 0.0011 and a new zero appended, so the last + // full step lands on 0.0011 and the final evaluation happens AT it rather + // than at a sigma the model cannot be conditioned on. + std::vector sigmas = sigmas_in; + const bool terminal_zero = sigmas.back() == 0.0f; + if (terminal_zero) { + sigmas.back() = kLtx2Res2sTerminalSigma; + sigmas.push_back(0.0f); + } + + // samplers.py:284 — step sizes in log space, on the MODIFIED schedule, with + // the widening to hp BEFORE the division (`sigmas[1:].to(hp) / + // sigmas[:-1].to(hp)`). The final entry is +inf when the schedule ends at 0; + // it is computed anyway, exactly as upstream computes the whole vector, and + // the loop never reads it. + std::vector hs(sigmas.size() - 1); + for (size_t i = 0; i + 1 < sigmas.size(); ++i) { + hs[i] = -std::log(static_cast(sigmas[i + 1]) / static_cast(sigmas[i])); + } + + // samplers.py:287-288. + Ltx2PhiCache phi_cache; + const double c2 = params.c2; + + // The `sigmas` the step-level injection is handed (samplers.py:415, :425) is + // this f32 schedule. `Ltx2Res2sStepHp` takes doubles and narrows internally + // under `kF32Schedule`, so widen once here rather than per step. + const std::vector sigmas_hp = ToHp(sigmas); + + std::vector denoised_v, denoised_a; + std::vector x_anchor_v, x_anchor_a, eps_1_v, eps_1_a, x_mid_v, x_mid_a; + + for (int64_t step_idx = 0; step_idx < stats.full_steps; ++step_idx) { + const size_t s = static_cast(step_idx); + // samplers.py:291-292. + const double sigma = static_cast(sigmas[s]); + const double sigma_next = static_cast(sigmas[s + 1]); + + // samplers.py:294-296 — the anchor is the state as it stands, in hp. + if (video.present) x_anchor_v = ToHp(video.latent); + if (audio.present) x_anchor_a = ToHp(audio.latent); + + // ── STAGE 1: evaluate at the current point (samplers.py:298-307) ──────── + denoised_v.clear(); + denoised_a.clear(); + // :301 — the loop's OWN counter is this call's `step_index`. + hooks.denoise(video.latent, audio.latent, sigma, step_idx, denoised_v, denoised_a); + stats.evaluations += 1; + stats.eval_sigmas.push_back(sigma); + stats.eval_step_indices.push_back(step_idx); + // :304-307 — post_process at the MODEL DTYPE, hence the narrowing back. + if (video.present && !denoised_v.empty()) { + denoised_v = ToModelDtype(hooks.post_process(ToHp(denoised_v), true)); + } + if (audio.present && !denoised_a.empty()) { + denoised_a = ToModelDtype(hooks.post_process(ToHp(denoised_a), false)); + } + + const double h = hs[s]; // :309 + // :311-312. + const Ltx2Res2sCoefficients coeff = Ltx2GetRes2sCoefficients(h, phi_cache, c2); + // :314-315 — "sqrt is a hardcode for c2 = 0.5". + const double sub_sigma = std::sqrt(sigma * sigma_next); + // `h * a21` is a scalar-scalar product upstream before it ever meets a + // tensor (:322), so it is formed once here for the same reason. + const double h_a21 = h * coeff.a21; + + // ── the substep point (samplers.py:317-332) ───────────────────────────── + const auto build_mid = [&](bool present, const std::vector& denoised, + const std::vector& anchor, std::vector& eps, + std::vector& mid) { + if (!present || denoised.empty()) { + eps.clear(); + mid.clear(); + return; + } + eps.resize(anchor.size()); + mid.resize(anchor.size()); + for (size_t k = 0; k < anchor.size(); ++k) { + eps[k] = static_cast(denoised[k]) - anchor[k]; + mid[k] = anchor[k] + h_a21 * eps[k]; + } + }; + build_mid(video.present, denoised_v, x_anchor_v, eps_1_v, x_mid_v); + build_mid(audio.present, denoised_a, x_anchor_a, eps_1_a, x_mid_a); + + // ── SDE noise injection at the substep (samplers.py:334-352) ──────────── + // + // VIDEO FIRST, THEN AUDIO, and the order is load bearing: both draws come + // from ONE generator, so swapping them hands each modality the other's + // noise. Upstream fixes the order at :337 and :345. + // + // eta is 0.5 here whatever the loop's own eta is — ":273-274, substep eta is + // always default 0.5 for compatibility with the original implementation". + // The schedule is the f64 pair [sigma, sub_sigma] at index 0 (:342, :350). + const double substep_sigmas[2] = {sigma, sub_sigma}; + const auto inject = [&](bool present, std::vector& x, bool is_video, + const std::vector& sample, const double* sched, + int64_t sched_count, int64_t idx, double eta, + Ltx2Res2sScheduleWidth width, bool substep) { + if (!present || x.empty()) return; + const int64_t count = static_cast(x.size()); + // :187 — the noise is drawn over `state.latent`, i.e. the modality's own + // element count, before the stepper is entered. + const std::vector noise = hooks.new_noise(count, is_video, substep); + Require(static_cast(noise.size()) == count, + "ltx2 res2s loop: the noise hook returned " + std::to_string(noise.size()) + + " values for a " + std::to_string(count) + "-element latent"); + x = Ltx2Res2sStepHp(sample.data(), x.data(), sched, sched_count, idx, count, + noise.data(), eta, width); + // :202-203 — `legacy_mode` is TRUE on every reachable path, so the blend + // happens AFTER the step rather than the sigmas being converted before it. + x = hooks.post_process(std::move(x), is_video); + }; + inject(video.present, x_mid_v, true, x_anchor_v, substep_sigmas, 2, 0, + kLtx2Res2sSubstepEta, Ltx2Res2sScheduleWidth::kF64Schedule, true); + inject(audio.present, x_mid_a, false, x_anchor_a, substep_sigmas, 2, 0, + kLtx2Res2sSubstepEta, Ltx2Res2sScheduleWidth::kF64Schedule, true); + + // ── the bong iteration (samplers.py:354-364) ──────────────────────────── + // + // A FIXED-POINT REFINEMENT OF THE ANCHOR, and both the anchor and eps_1 are + // carried into the final combination below. The guard is upstream's, with + // both comparisons STRICT: a schedule sitting at exactly sigma = 0.03 does + // not refine. + // + // There is no early exit and `bongmath_max_iter` iterations always run. + // Left as written: the map contracts with ratio `h * a21`, which the h < 0.5 + // guard bounds under 0.25, so it converges to machine precision long before + // iteration 100 and an early exit would be numerically invisible — which is + // exactly why removing the parameter would be untestable and is not done. + if (params.bongmath && h < kLtx2Res2sBongMaxH && sigma > kLtx2Res2sBongMinSigma) { + stats.bong_steps += 1; + for (int64_t iter = 0; iter < params.bongmath_max_iter; ++iter) { + if (!x_mid_v.empty() && !eps_1_v.empty()) { + for (size_t k = 0; k < x_mid_v.size(); ++k) { + x_anchor_v[k] = x_mid_v[k] - h_a21 * eps_1_v[k]; + eps_1_v[k] = static_cast(denoised_v[k]) - x_anchor_v[k]; + } + } + if (!x_mid_a.empty() && !eps_1_a.empty()) { + for (size_t k = 0; k < x_mid_a.size(); ++k) { + x_anchor_a[k] = x_mid_a[k] - h_a21 * eps_1_a[k]; + eps_1_a[k] = static_cast(denoised_a[k]) - x_anchor_a[k]; + } + } + } + } + + // ── STAGE 2: evaluate at the substep point, WITH noise (samplers.py:366-392) + // + // THE SECOND EVALUATION. This is the half of the sampler that a token count, + // a frame count, a shape check and a rendered pixel are all blind to, and + // dropping it leaves a working-looking renderer running the first-order + // method at the HQ preset's step count. + const std::vector mid_v = + (video.present && !x_mid_v.empty()) ? ToModelDtype(x_mid_v) : video.latent; + const std::vector mid_a = + (audio.present && !x_mid_a.empty()) ? ToModelDtype(x_mid_a) : audio.latent; + std::vector denoised_v2, denoised_a2; + // A LITERAL ZERO, not `step_idx` (samplers.py:385). Upstream builds a + // one-element schedule `torch.stack([sub_sigma])` for this call and indexes + // it at 0, so the pair `(sigmas, step_index)` the denoiser receives is + // `([sub_sigma], 0)` on EVERY step. The scalar sigma above carries the first + // half of that; this carries the second, and it is not cosmetic: the + // denoiser reads `step_index` through `should_skip_step` + // (guiders.py:287-291), so `0 % (skip_step + 1) == 0` makes the substep + // evaluation unskippable at any `skip_step`. Passing the loop counter here + // would skip it on the same steps the first evaluation is skipped on, which + // is a first-order trajectory wearing the second-order sampler's schedule. + // Inert on the HQ preset itself, whose `skip_step` is 0 (constants.py:104, + // :112), and live for a request that overrides it. + hooks.denoise(mid_v, mid_a, sub_sigma, /*step_index=*/0, denoised_v2, denoised_a2); + stats.evaluations += 1; + stats.eval_sigmas.push_back(sub_sigma); + stats.eval_step_indices.push_back(0); + if (video.present && !denoised_v2.empty()) { + denoised_v2 = ToModelDtype(hooks.post_process(ToHp(denoised_v2), true)); + } + if (audio.present && !denoised_a2.empty()) { + denoised_a2 = ToModelDtype(hooks.post_process(ToHp(denoised_a2), false)); + } + + // ── the final combination (samplers.py:394-407) ───────────────────────── + // + // `x_anchor + h * (b1 * eps_1 + b2 * eps_2)`, in that association: the two + // weighted epsilons are summed and the sum is scaled by h, which is not the + // same rounding as scaling each term. + std::vector x_next_v, x_next_a; + const auto combine = [&](bool present, const std::vector& anchor, + const std::vector& eps1, const std::vector& d2, + std::vector& out) { + if (!present || anchor.empty() || eps1.empty() || d2.empty()) { + out.clear(); + return; + } + out.resize(anchor.size()); + for (size_t k = 0; k < anchor.size(); ++k) { + const double eps_2 = static_cast(d2[k]) - anchor[k]; + out[k] = anchor[k] + h * (coeff.b1 * eps1[k] + coeff.b2 * eps_2); + } + }; + combine(video.present, x_anchor_v, eps_1_v, denoised_v2, x_next_v); + combine(audio.present, x_anchor_a, eps_1_a, denoised_a2, x_next_a); + + // ── SDE noise injection at the step level (samplers.py:409-427) ───────── + // + // The loop's OWN eta, and the loop's OWN float32 schedule at `step_idx`. + // Both differ from the substep call above, and both differences are + // upstream's. + inject(video.present, x_next_v, true, x_anchor_v, sigmas_hp.data(), + static_cast(sigmas_hp.size()), step_idx, params.eta, + Ltx2Res2sScheduleWidth::kF32Schedule, false); + inject(audio.present, x_next_a, false, x_anchor_a, sigmas_hp.data(), + static_cast(sigmas_hp.size()), step_idx, params.eta, + Ltx2Res2sScheduleWidth::kF32Schedule, false); + + // samplers.py:429-433. + if (video.present && !x_next_v.empty()) video.latent = ToModelDtype(x_next_v); + if (audio.present && !x_next_a.empty()) audio.latent = ToModelDtype(x_next_a); + } + + // ── the final step (samplers.py:435-445) ────────────────────────────────── + // + // "Final step if we need to fully remove the noise." It runs at index + // `n_full_steps`, which after the injection above is the 0.0011 entry, and its + // prediction becomes the state OUTRIGHT — there is no stepper call, so nothing + // re-noises the finished latent. This is the `+ 1` in `2 * n_full_steps + 1`. + if (terminal_zero) { + denoised_v.clear(); + denoised_a.clear(); + // :437 — `n_full_steps`, which is one past the last full step's index and is + // the position the injected 0.0011 now occupies. + hooks.denoise(video.latent, audio.latent, + static_cast(sigmas[static_cast(stats.full_steps)]), + stats.full_steps, denoised_v, denoised_a); + stats.evaluations += 1; + stats.eval_sigmas.push_back(static_cast(sigmas[static_cast(stats.full_steps)])); + stats.eval_step_indices.push_back(stats.full_steps); + if (video.present && !denoised_v.empty()) { + video.latent = ToModelDtype(hooks.post_process(ToHp(denoised_v), true)); + } + if (audio.present && !denoised_a.empty()) { + audio.latent = ToModelDtype(hooks.post_process(ToHp(denoised_a), false)); + } + } + + return stats; +} + +} // namespace vllm diff --git a/src/vllm/multimodal/ltx2_video.cpp b/src/vllm/multimodal/ltx2_video.cpp index 2cd0be1af..7b4ff74bc 100644 --- a/src/vllm/multimodal/ltx2_video.cpp +++ b/src/vllm/multimodal/ltx2_video.cpp @@ -36,6 +36,7 @@ #include "vllm/model_executor/models/ltx2_image_preprocess.h" #include "vllm/model_executor/models/ltx2_loader.h" #include "vllm/model_executor/models/ltx2_pipeline.h" +#include "vllm/model_executor/models/ltx2_samplers.h" #include "vllm/model_executor/models/ltx2_retake.h" #include "vllm/model_executor/models/ltx2_t2a.h" #include "vllm/model_executor/models/ltx2_text_encoder.h" @@ -232,13 +233,22 @@ void FromLatentState(const Ltx2LatentState& in, StreamState* s) { // denoised * mask + clean * (1 - mask) // The mask is PER TOKEN and the latent is per token x channel, so the mask // broadcasts along the channel axis exactly as torch's trailing-axis rule does. -std::vector PostProcessLatent(const std::vector& denoised, const StreamState& state) { - std::vector out(denoised.size()); +// TEMPLATED ON THE VALUE TYPE because upstream calls this at two widths and the +// res_2s loop reaches both: at the model dtype on a denoiser result +// (samplers.py:305, :390, :441) and at `hp` on a sample inside +// `_inject_sde_noise` (samplers.py:203). One implementation, two +// instantiations; a second copy of the blend is the shape this campaign has +// recorded going wrong. The mask is 0 or 1 on every LTX-2.5 path, so the result +// is exactly one operand or the other and the two widths agree. +template +std::vector PostProcessLatent(const std::vector& denoised, + const StreamState& state) { + std::vector out(denoised.size()); for (int64_t t = 0; t < state.tokens; ++t) { - const float m = state.mask[static_cast(t)]; + const Value m = static_cast(state.mask[static_cast(t)]); for (int64_t c = 0; c < state.width; ++c) { const size_t i = static_cast(t * state.width + c); - out[i] = denoised[i] * m + state.clean[i] * (1.0F - m); + out[i] = denoised[i] * m + static_cast(state.clean[i]) * (static_cast(1) - m); } } return out; @@ -374,7 +384,8 @@ constexpr char kLtx2DurationHeadPathExtra[] = "duration_head_path"; // they are no longer trusted: the list below is derived from this file on every // run and compared, and the failure prints the replacement to paste in. // READER ANCHORS (derived and gated by test_ltx2_video): -// 798 808 809 871 967 983 985 1076 1101 1206 1247 1289 1291 +// 809 819 820 882 978 994 996 1087 1112 1217 1258 1300 1302 + const char* const kKnownLoadExtras[] = { kLtx2AudioPromptEmbedsExtra, kLtx2PipelineKindExtra, kLtx2ModelVersionExtra, kLtx2AllowUnportedExtra, kLtx2MaxPhaseExtra, kLtx2DitConfigPathExtra, @@ -3297,17 +3308,11 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { phase.name + "'), so a `steps` override is refused rather than applied"); } - // ── the denoise loop (samplers.py:39-79 / :488-558) ───────────────────── - // The ancestral arm's loop generator is seeded from the pipeline seed plus - // the recipe's own offset (distilled.py:69-73, :177-183) — a separate stream - // from the state noise, so its first draw is not the initial latent's. - SplitMixGaussian loop_noise(seed + static_cast(phase.noise_seed_offset)); - const int64_t sigma_count = static_cast(sigmas.size()); - // This phase's two guiders, resolved once. `GuidedDenoiser` is constructed // per stage upstream and holds its guiders for the whole loop - // (ti2vid_one_stage.py:221-226), so resolving them per step would let a - // request override change meaning halfway down a schedule. + // (ti2vid_one_stage.py:221-226, ti2vid_two_stages_hq.py:271-281), so + // resolving them per step would let a request override change meaning + // halfway down a schedule. const Ltx2MultiModalGuiderParams& video_guidance = phase_guidance[static_cast(phase_index)].video; const Ltx2MultiModalGuiderParams& audio_guidance = @@ -3324,8 +3329,77 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { // skipped step reuses them instead of running a forward. std::vector last_denoised_video; std::vector last_denoised_audio; - for (int64_t step = 0; step + 1 < sigma_count; ++step) { - const float sigma = sigmas[static_cast(step)]; + // Phase 0's FIRST evaluation is what `RecordFirstGuidedStep` describes. On + // the first-order arm that is step 0, which is what this was before the + // res_2s loop existed; on the res_2s arm it is the first of that step's TWO + // evaluations (samplers.py:301), because the second one runs over a midpoint + // state and at a substep sigma and would describe a different call. + // AND THE SECOND EVALUATION IS RECORDED SEPARATELY, which is why this is a + // counter rather than a bool. The res_2s substep runs over `x_mid`, a state + // that never becomes the stream's own latent (samplers.py:369-378), so the + // x0 conversion there is the one place in this file where "the latent" and + // "the latent this evaluation was handed" are different tensors. MEASURED: + // with the conversion reading `video.latent` instead, the whole + // `test_ltx2_video` suite stayed GREEN at 74 cases and 2234 assertions — + // the clip, the counts, the eval sigmas and the bong count are all blind to + // it, because the loop's own arithmetic is gated with a FIXTURE denoiser and + // the engine's conversion is not in that loop. + int64_t phase_evaluation_index = 0; + + // ── ONE EVALUATION, SHARED BY EVERY SAMPLER ───────────────────────────── + // + // Upstream's samplers all take a `Denoiser` callable and never reach for a + // model (samplers.py:213-214, :45), which is why the loops differ only in + // how many times, at which sigmas and at which step indices they call it. + // This lambda is that callable, and BOTH arms below go through it: the + // first-order loop calls it once per step, `Ltx2Res2sDenoisingLoop` calls it + // twice per step plus once at the end. + // + // AND IT IS THE GUIDED DENOISER, on both arms. Upstream's HQ stage 1 hands + // `res2s_audio_video_denoising_loop` a `GuidedDenoiser` + // (ti2vid_two_stages_hq.py:271-281, :292) exactly as the one-stage pipeline + // hands its Euler loop one (ti2vid_one_stage.py:221-226), so the sampler + // decides HOW MANY denoiser calls happen and the denoiser decides how many + // forwards each call is. Routing res_2s around `Ltx2GuidedDenoise` would + // make the HQ preset the only unguided video arm in the tree — a plausible + // clip at cfg 1.0 where the preset was tuned at 3.0 — and the evaluation + // count, which is what this row's gate reads, would not move by one. + // + // Hoisted rather than duplicated because a second forward path written by + // hand would be a second place to forget the keyframe marker, the frozen + // scalar sigma or the device/host split — and every one of those omissions + // renders a finished clip. It also makes `dit_evaluations` a single + // increment that no arm can bypass. + // + // It takes the latent as an ARGUMENT rather than reading `video.latent`, + // because the res_2s loop's second evaluation runs over a MIDPOINT state + // that never becomes the stream's own latent (samplers.py:369-378). + // + // `sigma` is a `double` on the way in and narrows here. That narrowing is + // upstream's own boundary rather than a shortcut: `Modality.sigma` reaches + // the DiT as a tensor of the model's dtype, and this port's + // `Ltx2ModalityInput::sigma` is a `const float*`. The res_2s substep sigma + // is float64 up to this line (samplers.py:315, :384) and float32 after it. + // + // `step_index` IS THE DENOISER'S OWN ARGUMENT, not the sampler's loop + // counter. Upstream's `Denoiser` signature is + // `denoiser(transformer, video_state, audio_state, sigmas, step_index)`, and + // the res_2s loop passes THREE different values for it: `step_idx` at the + // first evaluation (samplers.py:301), a literal `0` at the substep + // evaluation beside a one-element schedule (samplers.py:384-385), and + // `n_full_steps` at the terminal one (samplers.py:437). It is read by + // `should_skip_step` (`step % (skip_step + 1) != 0`, guiders.py:287-291), so + // the substep evaluation is NEVER skipped whatever `skip_step` is. That is + // inert on the HQ preset, whose `skip_step` is 0 (constants.py:104, :112), + // and it is NOT inert for a request that sets `video_skip_step`. Passing the + // loop counter here instead would skip half of a res_2s step's evaluations + // on such a request and render at the first-order method's cost with the + // second-order sampler's schedule. + const auto Evaluate = [&](const std::vector& v_latent, + const std::vector& a_latent, double sigma_hp, + int64_t step_index, std::vector& v_denoised, + std::vector& a_denoised) { + const float sigma = static_cast(sigma_hp); const std::vector v_timesteps = TimestepsFromMask(video, sigma); const std::vector a_timesteps = TimestepsFromMask(audio, sigma); // The SECOND half of upstream's `frozen` on the VIDEO side @@ -3343,7 +3417,7 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { vin.batch = 1; vin.tokens = video.tokens; vin.context_tokens = context_tokens; - vin.latent = video.latent.data(); + vin.latent = v_latent.data(); vin.timesteps = v_timesteps.data(); vin.sigma = &sigma_row; vin.positions = video.positions.data(); @@ -3401,7 +3475,7 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { ain.batch = 1; ain.tokens = audio.tokens; ain.context_tokens = context_tokens; - ain.latent = audio.latent.data(); + ain.latent = a_latent.data(); ain.timesteps = a_timesteps.data(); // The SECOND half of upstream's `frozen` (utils/types.py:104-106): the // per-modality scalar sigma is forced to 0, "not only per-token @@ -3423,7 +3497,8 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { // ── the X0 MODEL (model.py:590-604), and the guided denoiser ────────── // // `DiffusionStage` never hands the loop the raw velocity model: it hands - // `X0Model(builder.build(...))` (utils/blocks.py:480-482). So `to_denoised` + // `X0Model(builder.build(...))` (utils/blocks.py:480-482, the forward it + // wraps at ltx-core model/transformer/model.py:590-604). So `to_denoised` // belongs HERE, inside the wrapper, applied to EVERY pass on its way out of // the forward — and the guider downstream combines already-denoised // tensors. Converting once after the guider instead is a DIFFERENT function @@ -3457,15 +3532,28 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { im.compute_dtype) : Ltx2DitForward(im.device, im.dit.params, im.dit.weights, v, a, im.compute_dtype, /*cache=*/nullptr, p); + // EVERY ACTUAL DiT FORWARD IS COUNTED HERE, and that is a different + // number from `dit_evaluations` one level up. One denoiser evaluation is + // one to four forwards (cond, uncond, ptb, mod — denoisers.py:100-137), + // so the two counters answer two questions that no output can: WHICH + // SAMPLER ran, and WHETHER THE ARM WAS GUIDED. An unguided HQ render + // keeps `dit_evaluations` at 2n+1 and drops this one from 3(2n+1) to + // 2n+1, and nothing else about the clip changes. + im.trace.dit_forwards += 1; Ltx2X0Outputs out; out.video_velocity = velocity.video; out.audio_velocity = velocity.audio; // The PER-TOKEN timesteps, not the schedule scalar: a conditioned token // sits at timestep 0 and using the scalar there re-noises it. - out.video = - ToDenoised(video.latent, velocity.video, v_timesteps, video.tokens, video.width); - out.audio = - ToDenoised(audio.latent, velocity.audio, a_timesteps, audio.tokens, audio.width); + // + // AND THE LATENT IS THE ONE THIS EVALUATION WAS HANDED, not the stream's + // own. They are the same tensor on the first-order arm and on the res_2s + // first evaluation, and they are NOT the same on the res_2s substep + // evaluation, which runs over `x_mid` (samplers.py:369-378). Reading + // `video.latent` here would convert the substep's velocity against the + // wrong sample and still return a finite, correctly shaped prediction. + out.video = ToDenoised(v_latent, velocity.video, v_timesteps, video.tokens, video.width); + out.audio = ToDenoised(a_latent, velocity.audio, a_timesteps, audio.tokens, audio.width); return out; }; @@ -3477,68 +3565,211 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { denoise_in.video_guider = video_guidance; denoise_in.audio_guider = audio_guidance; denoise_in.num_blocks = im.dit.params.num_layers; - denoise_in.step_index = step; + denoise_in.step_index = step_index; denoise_in.last_denoised_video = &last_denoised_video; denoise_in.last_denoised_audio = &last_denoised_audio; const Ltx2GuidedDenoiseResult guided = Ltx2GuidedDenoise(x0_model, denoise_in); - // `post_process_latent` is applied by the LOOP to the guider's OUTPUT - // (samplers.py:35, :484), never per arm inside the denoiser. Pinning the - // conditioned tokens per arm would make every arm agree on exactly those - // tokens, which zeroes the guidance delta precisely where a keyframe or a - // reference clip is conditioning — a render that is correct everywhere the - // conditioning is absent. - // + // THE ONE PLACE A DENOISER EVALUATION IS COUNTED. Every sampler reaches + // it, so a build that ran the wrong number of them cannot report the right + // count. This is the only observable that separates the res_2s sampler + // from the first-order one — the clip, its shape, its frame count, its + // sample rate and its file size are identical between them — which is why + // it is a counter rather than a comment. `dit_forwards` inside the x0 + // model above is the other half: this one counts CALLS, that one counts + // FORWARDS, and only the second moves when guidance is dropped. + im.trace.dit_evaluations += 1; + // `last_denoised_*` keeps what the GUIDER returned, before the // post-process, because that is what `_last_denoised_video` holds // (denoisers.py:299-300) and what a skipped step reuses. last_denoised_video = guided.video_denoised; last_denoised_audio = guided.audio_denoised; - const std::vector v_denoised = PostProcessLatent(guided.video_denoised, video); - const std::vector a_denoised = PostProcessLatent(guided.audio_denoised, audio); - if (phase_index == 0 && step == 0) { - RecordFirstGuidedStep(&im.trace, guided, video.latent, v_timesteps, - static_cast(sigma), v_denoised); + if (phase_index == 0 && phase_evaluation_index == 0) { + // `stepper_input` is the POST-PROCESSED prediction, which is what both + // samplers hand their stepper: the first-order loop through + // `_step_state` (samplers.py:35) and the res_2s loop at :305. Computed + // here rather than taken from the caller so the res_2s arm, whose + // post-process runs inside the sampler at f64, records the same quantity + // the Euler arm does. + RecordFirstGuidedStep(&im.trace, guided, v_latent, v_timesteps, + static_cast(sigma), + PostProcessLatent(guided.video_denoised, video)); } + // THE SUBSTEP EVALUATION, whose x0 conversion has no other observable. + // Recorded on the res_2s arm alone, because on a first-order arm the + // second evaluation is just step 1 and `video_first_*` already describes + // the shape. See `res2s_substep_*` in ltx2_video.h. + if (phase_index == 0 && phase_evaluation_index == 1 && + phase.stepper == Ltx2StepperKind::kRes2s) { + const size_t cond = static_cast(Ltx2DenoisePass::kCond); + im.trace.res2s_substep_latent = v_latent; + im.trace.res2s_substep_timesteps = v_timesteps; + im.trace.res2s_substep_cond = guided.video_pass[cond]; + im.trace.res2s_substep_cond_velocity = guided.video_pass_velocity[cond]; + im.trace.res2s_substep_sigma = static_cast(sigma); + } + phase_evaluation_index += 1; + + // RAW, not post-processed. `post_process_latent` belongs to the SAMPLER + // upstream, not to the denoiser: the first-order loop applies it inside + // `_step_state` (samplers.py:35) and the res_2s loop applies it at four + // separate points (:305, :390, :203, :441), one of which is after an SDE + // injection rather than after an evaluation. Folding it in here would put + // it in three of those four places and silently drop the fourth. + v_denoised = guided.video_denoised; + a_denoised = guided.audio_denoised; + }; - const bool terminal = sigmas[static_cast(step + 1)] == 0.0F; - if (phase.stepper == Ltx2StepperKind::kEulerAncestral) { - if (terminal) { - // samplers.py:545-547 — the terminal step IS the denoised prediction; - // taking an ancestral step there would re-noise the finished latent. - video.latent = v_denoised; - audio.latent = a_denoised; - if (phase_index == 0 && step == 0) im.trace.video_first_next_latent = video.latent; - continue; - } - const std::vector v_noise = - loop_noise.Draw(static_cast(video.latent.size())); - const std::vector a_noise = - loop_noise.Draw(static_cast(audio.latent.size())); - video.latent = PostProcessLatent( - Ltx2EulerAncestralStep(video.latent.data(), v_denoised.data(), sigmas.data(), - sigma_count, step, static_cast(video.latent.size()), - phase.stepper_eta, phase.stepper_s_noise, v_noise.data()), - video); - audio.latent = PostProcessLatent( - Ltx2EulerAncestralStep(audio.latent.data(), a_denoised.data(), sigmas.data(), - sigma_count, step, static_cast(audio.latent.size()), - phase.stepper_eta, phase.stepper_s_noise, a_noise.data()), - audio); - } else { - video.latent = Ltx2EulerStep(video.latent.data(), v_denoised.data(), sigmas.data(), + // ── the denoise loop ──────────────────────────────────────────────────── + // The ancestral arm's loop generator is seeded from the pipeline seed plus + // the recipe's own offset (distilled.py:69-73, :178-184) — a separate stream + // from the state noise, so its first draw is not the initial latent's. + SplitMixGaussian loop_noise(seed + static_cast(phase.noise_seed_offset)); + const int64_t sigma_count = static_cast(sigmas.size()); + + if (phase.stepper == Ltx2StepperKind::kRes2s) { + // ── the res_2s second-order sampler (samplers.py:208-447) ───────────── + // + // Row LTX25-RES2S-LOOP, issue #921. `TI2VidTwoStagesHQPipeline` passes + // `loop=res2s_audio_video_denoising_loop` to BOTH of its stages + // (ti2vid_two_stages_hq.py:292, :335), and this is that loop. + // + // THE PARAMETERS ARE THE LOOP'S OWN DEFAULTS, DELIBERATELY. + // `DiffusionStage.__call__` hands the loop six keyword arguments and no + // others (utils/blocks.py:566-573), so nothing on the HQ path overrides + // eta, bongmath, the iteration cap, the noise function or the seeds. + // Passing anything else here would be this port inventing a knob. + // + // THE SEEDS ARE CONSTANTS AND NOT `seed`. `noise_seed` defaults to -1 + // (samplers.py:215) and the substep stream to -1 + 10000 + // (samplers.py:265-266), so the res_2s SDE injections do not depend on the + // request's seed at all — the initial latent still does, through the + // noiser. The ancestral arm one branch up does the opposite. Mirrored + // rather than made consistent, because consistency here would be a + // divergence. + SplitMixGaussian res2s_step_noise(static_cast(kLtx2Res2sNoiseSeed)); + SplitMixGaussian res2s_substep_noise( + static_cast(kLtx2Res2sNoiseSeed + kLtx2Res2sNoiseSeedSubstepOffset)); + + Ltx2Res2sHooks hooks; + hooks.denoise = Evaluate; + hooks.post_process = [&](std::vector x, bool is_video) { + return PostProcessLatent(x, is_video ? video : audio); + }; + // `_get_new_noise` (samplers.py:164-170): draw, then normalize. The DRAW + // is this port's `SplitMixGaussian` rather than upstream's seeded + // `torch.randn`, so the stream differs — as it already does on the + // shipped ancestral arm — and only the normalization is mirrored. Which + // NOISE FUNCTION each loop uses is mirrored too, and the two loops do not + // agree: the ancestral one defaults to the un-normalized + // `_get_plain_noise` (samplers.py:574). + hooks.new_noise = [&](int64_t count, bool /*is_video*/, bool substep) { + SplitMixGaussian& stream = substep ? res2s_substep_noise : res2s_step_noise; + const std::vector raw = stream.Draw(count); + std::vector noise = + Ltx2Res2sNormalizeNoise(std::vector(raw.begin(), raw.end())); + // OBSERVED, not asserted in prose. Whether this hook normalizes is + // invisible in the rendered clip, the token count and the evaluation + // count alike, and a build that returned `raw` here left the whole + // end-to-end suite green. See `res2s_noise_moment_error`. + double mean = 0.0; + for (const double v : noise) mean += v; + mean /= static_cast(noise.size()); + double sq = 0.0; + for (const double v : noise) sq += (v - mean) * (v - mean); + const double sd = std::sqrt(sq / static_cast(noise.size() - 1)); + im.trace.res2s_noise_moment_error = std::max( + im.trace.res2s_noise_moment_error, std::max(std::fabs(mean), std::fabs(sd - 1.0))); + return noise; + }; + + Ltx2Res2sModality res2s_video{video.latent, true}; + Ltx2Res2sModality res2s_audio{audio.latent, true}; + // TWO INDEPENDENT COUNTERS, and the check below is only worth running + // because they are independent. `stats.evaluations` is the LOOP's own + // count; `im.trace.dit_evaluations` is incremented inside `Evaluate`, i.e. + // by the ENGINE, once per call the loop actually made. This delta is what + // makes the comparison an observation rather than an identity: the + // previous form of this check compared `stats.evaluations` against + // `stats.full_steps`, both fields of the same struct, and `2n + 1 > n` + // holds for every n >= 1, so it could not fail for any build. + const int64_t evaluations_before = im.trace.dit_evaluations; + const Ltx2Res2sLoopStats stats = + Ltx2Res2sDenoisingLoop(sigmas, res2s_video, res2s_audio, hooks); + video.latent = std::move(res2s_video.latent); + audio.latent = std::move(res2s_audio.latent); + const int64_t engine_evaluations = im.trace.dit_evaluations - evaluations_before; + VT_CHECK(engine_evaluations == stats.evaluations, + "ltx2 video: the res_2s loop reports " + std::to_string(stats.evaluations) + + " denoiser evaluations and the engine counted " + + std::to_string(engine_evaluations) + ". The loop counts its own calls and the " + "engine counts the ones that reached `Evaluate`, so a disagreement means a " + "call was made without reaching the shared evaluation — the one place the " + "keyframe marker, the frozen scalar sigma, the guided denoiser and the " + "host/device split are all applied."); + VT_CHECK(engine_evaluations > stats.full_steps, + "ltx2 video: the res_2s sampler evaluates the denoiser TWICE per step plus once at " + "a terminal zero sigma (samplers.py:301, :380-386, :437), so the engine cannot " + "count as many evaluations as the loop has steps. A count at or below the step " + "count means the second evaluation was skipped, which renders a finished, " + "correctly sized, plausible clip at half the model evaluations the HQ preset was " + "tuned for."); + im.trace.res2s_bong_steps += stats.bong_steps; + } else { + for (int64_t step = 0; step + 1 < sigma_count; ++step) { + const float sigma = sigmas[static_cast(step)]; + std::vector v_raw, a_raw; + // `step` IS the denoiser's `step_index` on this arm — upstream's + // first-order loop passes its own loop counter straight through + // (samplers.py:45, :503) — which is what `should_skip_step` reads. + Evaluate(video.latent, audio.latent, static_cast(sigma), step, v_raw, a_raw); + // `_step_state` (samplers.py:35) blends before it steps. + const std::vector v_denoised = PostProcessLatent(v_raw, video); + const std::vector a_denoised = PostProcessLatent(a_raw, audio); + + const bool terminal = sigmas[static_cast(step + 1)] == 0.0F; + if (phase.stepper == Ltx2StepperKind::kEulerAncestral) { + if (terminal) { + // samplers.py:545-547 — the terminal step IS the denoised + // prediction; taking an ancestral step there would re-noise the + // finished latent. + video.latent = v_denoised; + audio.latent = a_denoised; + if (phase_index == 0 && step == 0) im.trace.video_first_next_latent = video.latent; + continue; + } + const std::vector v_noise = + loop_noise.Draw(static_cast(video.latent.size())); + const std::vector a_noise = + loop_noise.Draw(static_cast(audio.latent.size())); + video.latent = PostProcessLatent( + Ltx2EulerAncestralStep(video.latent.data(), v_denoised.data(), sigmas.data(), sigma_count, step, - static_cast(video.latent.size())); - audio.latent = Ltx2EulerStep(audio.latent.data(), a_denoised.data(), sigmas.data(), + static_cast(video.latent.size()), + phase.stepper_eta, phase.stepper_s_noise, v_noise.data()), + video); + audio.latent = PostProcessLatent( + Ltx2EulerAncestralStep(audio.latent.data(), a_denoised.data(), sigmas.data(), sigma_count, step, - static_cast(audio.latent.size())); + static_cast(audio.latent.size()), + phase.stepper_eta, phase.stepper_s_noise, a_noise.data()), + audio); + } else { + video.latent = Ltx2EulerStep(video.latent.data(), v_denoised.data(), sigmas.data(), + sigma_count, step, + static_cast(video.latent.size())); + audio.latent = Ltx2EulerStep(audio.latent.data(), a_denoised.data(), sigmas.data(), + sigma_count, step, + static_cast(audio.latent.size())); + } + // What the sampler WROTE, recorded after the step rather than derived + // from what was recorded before it. It is the only observable that says + // which tensor the stepper was actually handed: a second `ToDenoised` on + // the way in leaves every other recorded field untouched. + if (phase_index == 0 && step == 0) im.trace.video_first_next_latent = video.latent; } - // What the sampler WROTE, recorded after the step rather than derived from - // what was recorded before it. It is the only observable that says which - // tensor the stepper was actually handed: a second `ToDenoised` on the way - // in leaves every other recorded field untouched. - if (phase_index == 0 && step == 0) im.trace.video_first_next_latent = video.latent; } // `clear_conditioning` + `unpatchify` (blocks.py:575-580, in that order). diff --git a/tests/vllm/models/ltx2_res2s_goldens.inc b/tests/vllm/models/ltx2_res2s_goldens.inc new file mode 100644 index 000000000..ae1e3f0b0 --- /dev/null +++ b/tests/vllm/models/ltx2_res2s_goldens.inc @@ -0,0 +1,203 @@ +// GENERATED from Lightricks/LTX-2 @ fd4ded7f by +// scripts/gen-ltx2-res2s-goldens.py. Do not hand-edit. +#pragma once + +#include + +namespace vllm_test { + +// res2s.py:4-22. `phi(j, z)` at j = 1 and j = 2, INCLUDING the small-z +// cliff: the guard is `abs(z) < 1e-10` and outside it the formula +// cancels catastrophically, so upstream's own phi2(-1e-10) is 0.0 and +// phi2(-1e-8) is 1.1102230246251563. These are upstream's values, not a +// series expansion's, and a 'better' port fails here. +inline constexpr double kLtx2PhiZ[] = { + 0.0, -1e-12, -1e-11, + -1e-10, -1e-09, -1e-08, + -1e-06, -0.001, -0.125, + -0.25, -0.5, -1.0, + -2.0, -5.0}; +inline constexpr double kLtx2Phi1[] = { + 1.0, 1.0, 1.0, + 1.000000082740371, 0.9999999717180684, 0.999999993922529, + 0.9999994999843054, 0.9995001666249781, 0.9400247793232364, + 0.8847968677143805, 0.7869386805747332, 0.6321205588285577, + 0.43233235838169365, 0.1986524106001829}; +inline constexpr double kLtx2Phi2[] = { + 0.5, 0.5, 0.5, + 0.0, 0.0, 1.1102230246251563, + 0.5000444502911705, 0.4998333750227957, 0.4798017654141091, + 0.46081252914247806, 0.4261226388505337, 0.36787944117144233, + 0.2838338208091532, 0.16026951787996344}; +inline constexpr int64_t kLtx2PhiCount = 14; + +// res2s.py:25-62, c2 = 0.5 (samplers.py:288). +inline constexpr double kLtx2Res2sCoeffH[] = { + 1e-12, 1e-10, 1e-08, + 1e-06, 0.01, 0.125, + 0.25, 0.5, 1.0, + 3.0, 7.0}; +inline constexpr double kLtx2Res2sCoeffA21[] = { + 0.5, 0.5, 0.4999999969612645, + 0.4999998749477541, 0.498752080731768, 0.48469549749219354, + 0.4700123896616182, 0.44239843385719024, 0.3934693402873666, + 0.2589566132838567, 0.13854323093966878}; +inline constexpr double kLtx2Res2sCoeffB1[] = { + 0.0, 1.000000082740371, -1.2204460553277836, + -8.940059803563827e-05, -0.0016583582791218632, -0.019578751504981895, + -0.036828190570575625, -0.06530659712633424, -0.103638323514327, + -0.1387705935377022, -0.10220830485081611}; +inline constexpr double kLtx2Res2sCoeffB2[] = { + 1.0, 0.0, 2.2204460492503126, + 1.000088900582341, 0.9966749833623112, 0.9596035308282183, + 0.9216250582849561, 0.8522452777010674, 0.7357588823428847, + 0.4555082374150809, 0.2449351788557369}; +inline constexpr int64_t kLtx2Res2sCoeffCount = 11; + +// BongOn: sigmas [0.9, 0.8, 0.7, 0.62], eta 0.5, h [0.117783, 0.133531, 0.121361], bong changed the result: True +inline constexpr double kLtx2Res2sBongOnEta = 0.5; +inline constexpr float kLtx2Res2sBongOnSigmas[] = { + 0.899999976f, 0.800000012f, 0.699999988f, + 0.620000005f}; +inline constexpr int64_t kLtx2Res2sBongOnSigmaCount = 4; +inline constexpr int64_t kLtx2Res2sBongOnEvaluations = 6; +inline constexpr double kLtx2Res2sBongOnEvalSigmas[] = { + 0.8999999761581421, 0.8485281325067245, 0.800000011920929, + 0.7483314765582876, 0.699999988079071, 0.658786760603827}; +inline constexpr int64_t kLtx2Res2sBongOnEvalStepIndices[] = { + 0, 0, 1, + 0, 2, 0}; +inline constexpr float kLtx2Res2sBongOnVideo[] = { + -0.322195023f, 0.323642969f, 0.699999988f, + 0.249027193f, 0.400000006f, 0.576920807f}; +inline constexpr float kLtx2Res2sBongOnAudio[] = { + 0.244931653f, 0.400000006f, 0.0785670504f, + 0.699999988f, 0.159960002f, -0.233123973f}; +inline constexpr bool kLtx2Res2sBongOnBongMoved = true; +inline constexpr float kLtx2Res2sBongOnNoBongVideo[] = { + -0.21995157f, 0.451260477f, 0.699999988f, + 0.552530169f, 0.400000006f, 0.592931151f}; + +// BongOffByH: sigmas [0.9, 0.5, 0.25, 0.12], eta 0.5, h [0.587787, 0.693147, 0.733969], bong changed the result: False +inline constexpr double kLtx2Res2sBongOffByHEta = 0.5; +inline constexpr float kLtx2Res2sBongOffByHSigmas[] = { + 0.899999976f, 0.5f, 0.25f, + 0.119999997f}; +inline constexpr int64_t kLtx2Res2sBongOffByHSigmaCount = 4; +inline constexpr int64_t kLtx2Res2sBongOffByHEvaluations = 6; +inline constexpr double kLtx2Res2sBongOffByHEvalSigmas[] = { + 0.8999999761581421, 0.670820384364601, 0.5, + 0.3535533905932738, 0.25, 0.1732050788211701}; +inline constexpr int64_t kLtx2Res2sBongOffByHEvalStepIndices[] = { + 0, 0, 1, + 0, 2, 0}; +inline constexpr float kLtx2Res2sBongOffByHVideo[] = { + 0.241032124f, 0.415953547f, 0.699999988f, + 0.426735073f, 0.400000006f, 0.526328683f}; +inline constexpr float kLtx2Res2sBongOffByHAudio[] = { + 0.0932270736f, 0.400000006f, 0.0732859746f, + 0.699999988f, 0.0699004084f, 0.0386947282f}; +inline constexpr bool kLtx2Res2sBongOffByHBongMoved = false; +inline constexpr float kLtx2Res2sBongOffByHNoBongVideo[] = { + 0.241032124f, 0.415953547f, 0.699999988f, + 0.426735073f, 0.400000006f, 0.526328683f}; + +// BongOffBySigma: sigmas [0.03, 0.028, 0.026, 0.025], eta 0.5, h [0.068993, 0.074108, 0.039221], bong changed the result: False +inline constexpr double kLtx2Res2sBongOffBySigmaEta = 0.5; +inline constexpr float kLtx2Res2sBongOffBySigmaSigmas[] = { + 0.0299999993f, 0.0280000009f, 0.0260000005f, + 0.0250000004f}; +inline constexpr int64_t kLtx2Res2sBongOffBySigmaSigmaCount = 4; +inline constexpr int64_t kLtx2Res2sBongOffBySigmaEvaluations = 6; +inline constexpr double kLtx2Res2sBongOffBySigmaEvalSigmas[] = { + 0.029999999329447746, 0.028982753615772204, 0.02800000086426735, + 0.026981475821224496, 0.026000000536441803, 0.025495098020929436}; +inline constexpr int64_t kLtx2Res2sBongOffBySigmaEvalStepIndices[] = { + 0, 0, 1, + 0, 2, 0}; +inline constexpr float kLtx2Res2sBongOffBySigmaVideo[] = { + 0.119263649f, 0.260755271f, 0.699999988f, + 0.486651659f, 0.400000006f, 0.701571345f}; +inline constexpr float kLtx2Res2sBongOffBySigmaAudio[] = { + 0.275088519f, 0.400000006f, 0.194240034f, + 0.699999988f, 0.11664068f, 0.0714727566f}; +inline constexpr bool kLtx2Res2sBongOffBySigmaBongMoved = false; +inline constexpr float kLtx2Res2sBongOffBySigmaNoBongVideo[] = { + 0.119263649f, 0.260755271f, 0.699999988f, + 0.486651659f, 0.400000006f, 0.701571345f}; + +// TerminalZero: sigmas [1.0, 0.75, 0.5, 0.25, 0.0], eta 0.5, h [0.287682, 0.405465, 0.693147], bong changed the result: True +inline constexpr double kLtx2Res2sTerminalZeroEta = 0.5; +inline constexpr float kLtx2Res2sTerminalZeroSigmas[] = { + 1.0f, 0.75f, 0.5f, + 0.25f, 0.0f}; +inline constexpr int64_t kLtx2Res2sTerminalZeroSigmaCount = 5; +inline constexpr int64_t kLtx2Res2sTerminalZeroEvaluations = 9; +inline constexpr double kLtx2Res2sTerminalZeroEvalSigmas[] = { + 1.0, 0.8660254037844386, 0.75, + 0.6123724356957945, 0.5, 0.3535533905932738, + 0.25, 0.016583123906848306, 0.0010999999940395355}; +inline constexpr int64_t kLtx2Res2sTerminalZeroEvalStepIndices[] = { + 0, 0, 1, + 0, 2, 0, + 3, 0, 4}; +inline constexpr float kLtx2Res2sTerminalZeroVideo[] = { + 0.445208162f, 0.445210904f, 0.699999988f, + 0.44826746f, 0.400000006f, 0.451412231f}; +inline constexpr float kLtx2Res2sTerminalZeroAudio[] = { + 0.0724363402f, 0.400000006f, 0.082616441f, + 0.699999988f, 0.0833412334f, 0.0882256776f}; +inline constexpr bool kLtx2Res2sTerminalZeroBongMoved = true; +inline constexpr float kLtx2Res2sTerminalZeroNoBongVideo[] = { + 0.443982691f, 0.447306544f, 0.699999988f, + 0.448465914f, 0.400000006f, 0.450835288f}; + +// Eta1: sigmas [0.9, 0.8, 0.7, 0.62], eta 1.0, h [0.117783, 0.133531, 0.121361], bong changed the result: True +inline constexpr double kLtx2Res2sEta1Eta = 1.0; +inline constexpr float kLtx2Res2sEta1Sigmas[] = { + 0.899999976f, 0.800000012f, 0.699999988f, + 0.620000005f}; +inline constexpr int64_t kLtx2Res2sEta1SigmaCount = 4; +inline constexpr int64_t kLtx2Res2sEta1Evaluations = 6; +inline constexpr double kLtx2Res2sEta1EvalSigmas[] = { + 0.8999999761581421, 0.8485281325067245, 0.800000011920929, + 0.7483314765582876, 0.699999988079071, 0.658786760603827}; +inline constexpr int64_t kLtx2Res2sEta1EvalStepIndices[] = { + 0, 0, 1, + 0, 2, 0}; +inline constexpr float kLtx2Res2sEta1Video[] = { + -0.513900995f, 0.366209567f, 0.699999988f, + 0.682274103f, 0.400000006f, -0.190410003f}; +inline constexpr float kLtx2Res2sEta1Audio[] = { + -0.362084478f, 0.400000006f, 0.0627472028f, + 0.699999988f, 0.390976906f, -0.051062014f}; +inline constexpr bool kLtx2Res2sEta1BongMoved = true; +inline constexpr float kLtx2Res2sEta1NoBongVideo[] = { + -0.510560513f, 0.363734573f, 0.699999988f, + 0.686390102f, 0.400000006f, -0.188381493f}; + +inline constexpr int64_t kLtx2Res2sLatentCount = 6; +inline constexpr float kLtx2Res2sVideo0[] = { + 0.0f, 0.166666672f, 0.333333343f, + 0.5f, 0.666666687f, 0.833333313f}; +inline constexpr float kLtx2Res2sAudio0[] = { + 0.5f, 0.416666657f, 0.333333343f, + 0.25f, 0.166666672f, 0.0833333358f}; +inline constexpr float kLtx2Res2sMask[] = { + 1.0f, 1.0f, 0.0f, + 1.0f, 0.0f, 1.0f}; +inline constexpr float kLtx2Res2sClean[] = { + -0.300000012f, 0.200000003f, 0.699999988f, + -0.100000001f, 0.400000006f, 0.0500000007f}; + +// samplers.py:160-170. `_get_new_noise` normalizes globally and then +// channelwise; the DRAW itself is torch.randn, whose stream this port +// does not have, so only the normalization is gated. +inline constexpr double kLtx2Res2sNoiseRaw[] = { + -0.33333333333333326, -1.6666666666666667, 2.666666666666667, + 1.3333333333333335, 0.0, -1.3333333333333335}; +inline constexpr double kLtx2Res2sNoiseNormalized[] = { + -0.27066598098038347, -1.0826639239215337, 1.5563293906372047, + 0.7443314476960544, -0.06766649524509592, -0.8796644381862462}; + +} // namespace vllm_test diff --git a/tests/vllm/models/test_ltx2_pipeline.cpp b/tests/vllm/models/test_ltx2_pipeline.cpp index ce4c0c548..571d3f09f 100644 --- a/tests/vllm/models/test_ltx2_pipeline.cpp +++ b/tests/vllm/models/test_ltx2_pipeline.cpp @@ -41,8 +41,14 @@ #include #include "ltx2_pipeline_goldens.inc" +// Row LTX25-RES2S-LOOP (#921). Its own file rather than rows appended to +// `ltx2_pipeline_goldens.inc`: that file is written by +// scripts/gen-ltx2-pipeline-goldens.py and edited by several concurrent rows of +// this campaign, and a per-row file is the shape `AGENTS.md ## Records` asks for. +#include "ltx2_res2s_goldens.inc" #include "vllm/model_executor/models/ltx2.h" +#include "vllm/model_executor/models/ltx2_samplers.h" #include "vllm/model_executor/models/ltx2_connector.h" #include "vllm/model_executor/models/ltx2_duration_head.h" #include "vllm/model_executor/models/ltx2_upsampler.h" @@ -1183,7 +1189,17 @@ TEST_CASE("ltx2 the recipe table mirrors vLLM-Omni's, and refuses everything els {"distilled_two_stage", "2.3"}, {"dmd2", "2.5"}, {"retake", "2.3"}, - {"res2s_two_stage", "2.5"}, + // WAS `{"res2s_two_stage", "2.5"}`, and it is repointed here rather + // than deleted. That pair was this list's stand-in for "a kind the + // table has never heard of", and row LTX25-RES2S-LOOP (#921) made it + // a SERVED row — so leaving it would have asserted a refusal for a + // capability that ships, which is the failure mode #923 retired an + // enumerator over. `res2s_two_stage` at 2.3 keeps the pair's job: + // 2.5 is the only version `LTX_2_3_HQ_PARAMS` resolves onto, because + // it is a plain constant with no `detect_params` lineage + // (constants.py:91-94). + {"res2s_two_stage", "2.3"}, + {"hq_two_stage", "2.5"}, {"", ""}, }) { const std::string message = RefusalMessage( @@ -2642,3 +2658,674 @@ TEST_CASE("ltx2 the processor's binary mask mirrors a comparison that looks back INFO("padded rows = ", pad, " max|diff| vs an all-valid mask = ", masked_vs_unmasked); CHECK(masked_vs_unmasked > 0.0); } + +// =========================================================================== +// The res_2s sampler — row LTX25-RES2S-LOOP, issue #921 +// +// Every golden below came out of UPSTREAM'S OWN CODE at Lightricks/LTX-2 +// fd4ded7f: `phi`, `get_res2s_coefficients`, `Res2sDiffusionStep`, +// `post_process_latent` and `res2s_audio_video_denoising_loop` were imported +// from the checkout and run. Three things were substituted and each is one this +// port reproduces exactly — the denoiser (a fixed quadratic), the noise DRAW +// (`torch.randn`, whose stream this port does not have) and two media-IO modules +// the import chain pulls in and nothing numeric touches. The generator is +// recorded in .agents/specs/ltx25-res2s-loop.md section 5. +// =========================================================================== + +namespace { + +double MaxAbsDiffD(const std::vector& got, const double* want, size_t count) { + REQUIRE(got.size() == count); + double worst = 0.0; + for (size_t i = 0; i < count; ++i) worst = std::max(worst, std::fabs(got[i] - want[i])); + return worst; +} + +// The reduced fixture every loop case below runs on: 6 elements, a denoise mask +// that is NOT all ones, and a clean latent that differs from the state, so +// `post_process_latent` is not the identity and a build that dropped the blend +// fails at the masked positions rather than passing. +struct Res2sFixture { + std::vector video_mask, video_clean, audio_mask, audio_clean; + int64_t evaluations = 0; + std::vector eval_sigmas; + // The `step_index` each call was handed, recorded by the DENOISER rather than + // read back off `Ltx2Res2sLoopStats`. Two independent records of the same + // fact: the stats vector says what the loop believes it passed and this one + // says what arrived, so a build that recorded one value and passed another is + // visible. It matters because `should_skip_step` reads it + // (guiders.py:287-291) and nothing in the returned latents does. + std::vector eval_step_indices; + // One counter per upstream generator (samplers.py:267-268). + int64_t step_draws = 0, substep_draws = 0; + + Res2sFixture() { + const size_t n = static_cast(vllm_test::kLtx2Res2sLatentCount); + video_mask.assign(vllm_test::kLtx2Res2sMask, vllm_test::kLtx2Res2sMask + n); + video_clean.assign(vllm_test::kLtx2Res2sClean, vllm_test::kLtx2Res2sClean + n); + // The generator reverses both on the audio side, so a build that fed one + // modality's mask to the other is visible rather than symmetric. + audio_mask.assign(video_mask.rbegin(), video_mask.rend()); + audio_clean.assign(video_clean.rbegin(), video_clean.rend()); + } + + vllm::Ltx2Res2sHooks Hooks() { + vllm::Ltx2Res2sHooks hooks; + // The substituted denoiser: `0.5x + 0.25 - 0.125x^2` on video and + // `-0.25x + 0.1 + 0.0625x^2` on audio, in the model dtype and in the same + // operation order the generator used. QUADRATIC and not affine on purpose, + // so a build that evaluated once and reused the result cannot land on the + // same trajectory by luck. + hooks.denoise = [this](const std::vector& v, const std::vector& a, double sigma, + int64_t step_index, std::vector& dv, std::vector& da) { + evaluations += 1; + eval_sigmas.push_back(sigma); + eval_step_indices.push_back(step_index); + dv.resize(v.size()); + for (size_t i = 0; i < v.size(); ++i) { + dv[i] = 0.5f * v[i] + 0.25f - 0.125f * (v[i] * v[i]); + } + da.resize(a.size()); + for (size_t i = 0; i < a.size(); ++i) { + da[i] = -0.25f * a[i] + 0.1f - 0.0625f * (a[i] * a[i]); + } + }; + hooks.post_process = [this](std::vector x, bool is_video) { + const std::vector& mask = is_video ? video_mask : audio_mask; + const std::vector& clean = is_video ? video_clean : audio_clean; + for (size_t i = 0; i < x.size(); ++i) { + const double m = static_cast(mask[i]); + x[i] = x[i] * m + static_cast(clean[i]) * (1.0 - m); + } + return x; + }; + // The generator's stand-in for `torch.randn`: a fixed pattern, offset by + // which of upstream's two generators would have drawn it + // (samplers.py:267-268) and by HOW MANY draws that generator has already + // made. + // + // STATEFUL ON PURPOSE, because upstream's generator is. `_get_new_noise` + // draws from a seeded `torch.Generator` that advances, so within one step + // the video draw and the audio draw are different tensors and the ORDER of + // the two calls decides which modality receives which. MEASURED: with this + // hook stateless — the same values for every call — swapping the video and + // audio injections left this whole suite green, because both modalities were + // being handed identical noise. The engine's own hook draws from one + // `SplitMixGaussian` per stream, where that swap is a real defect. + hooks.new_noise = [this](int64_t count, bool /*is_video*/, bool substep) { + int64_t& draw = substep ? substep_draws : step_draws; + std::vector out(static_cast(count)); + for (int64_t i = 0; i < count; ++i) { + out[static_cast(i)] = + static_cast((i * 7 + 3 + (substep ? 1 : 0) + 13 * draw) % 11) / 5.0 - 1.0; + } + draw += 1; + return out; + }; + return hooks; + } +}; + +} // namespace + +TEST_CASE("ltx2 res2s phi mirrors upstream AT THE SMALL-Z CLIFF") { + // THE POINT OF THIS CASE IS THAT A BETTER IMPLEMENTATION FAILS IT. + // + // `phi` (res2s.py:4-22) guards only `abs(z) < 1e-10` and otherwise evaluates + // `(exp(z) - remainder) / z^j` directly, which cancels catastrophically just + // outside the guard. Upstream's own phi2(-1e-10) is 0.0 and its phi2(-1e-8) is + // 1.1102230246251563. A port that used a Taylor series near zero — the + // numerically correct thing to do — returns 0.5 at both and DIVERGES FROM THE + // MODEL'S OWN RUNTIME. Asserted EXACTLY, not within a tolerance, because a + // tolerance wide enough to cover the cancellation would accept the series. + REQUIRE(vllm_test::kLtx2PhiCount == 14); + for (int64_t i = 0; i < vllm_test::kLtx2PhiCount; ++i) { + const double z = vllm_test::kLtx2PhiZ[i]; + const double got1 = vllm::Ltx2Phi(1, z); + const double got2 = vllm::Ltx2Phi(2, z); + INFO("z = ", z, " phi1 got = ", got1, " want = ", vllm_test::kLtx2Phi1[i], + " phi2 got = ", got2, " want = ", vllm_test::kLtx2Phi2[i]); + CHECK(got1 == vllm_test::kLtx2Phi1[i]); + CHECK(got2 == vllm_test::kLtx2Phi2[i]); + } + + // The threshold is EXACTLY 1e-10 and STRICT, so these two neighbouring inputs + // land on opposite branches. Stated apart from the sweep because it is the one + // constant the whole function turns on, and a sweep that happened to omit one + // side would not say so. + CHECK(vllm::Ltx2Phi(2, -1e-11) == 0.5); + CHECK(vllm::Ltx2Phi(2, -1e-10) == 0.0); + // phi_j(0) = 1/j! past the two values the coefficients use, so the factorial + // is gated rather than being correct only where it is inlined. + CHECK(vllm::Ltx2Phi(3, 0.0) == 1.0 / 6.0); + CHECK(vllm::Ltx2Phi(4, 0.0) == 1.0 / 24.0); + // j < 1 is not a value upstream's callers produce (res2s.py:49, :55, :59 pass + // 1 and 2), so it is refused rather than returning a plausible number. + CHECK_FALSE(RefusalMessage([] { (void)vllm::Ltx2Phi(0, -0.5); }).empty()); +} + +TEST_CASE("ltx2 res2s coefficients mirror upstream, cliff included") { + REQUIRE(vllm_test::kLtx2Res2sCoeffCount == 11); + for (int64_t i = 0; i < vllm_test::kLtx2Res2sCoeffCount; ++i) { + const double h = vllm_test::kLtx2Res2sCoeffH[i]; + vllm::Ltx2PhiCache cache; + const vllm::Ltx2Res2sCoefficients got = vllm::Ltx2GetRes2sCoefficients(h, cache, 0.5); + INFO("h = ", h, " a21 = ", got.a21, " b1 = ", got.b1, " b2 = ", got.b2); + CHECK(got.a21 == vllm_test::kLtx2Res2sCoeffA21[i]); + CHECK(got.b1 == vllm_test::kLtx2Res2sCoeffB1[i]); + CHECK(got.b2 == vllm_test::kLtx2Res2sCoeffB2[i]); + // res2s.py:39 — three entries per distinct h: (1, -h*c2), (2, -h), (1, -h). + // Asserted because the cache is a mirrored STRUCTURE that changes no value, + // so nothing else here would notice it disappearing. + CHECK(cache.size() == 3u); + CHECK(cache.count(std::pair{1, -h * 0.5}) == 1u); + CHECK(cache.count(std::pair{1, -h}) == 1u); + CHECK(cache.count(std::pair{2, -h}) == 1u); + } + + // The cliff carried INTO the coefficients, which is where it bites: at + // h = 1e-10 upstream's b2 collapses to 0 and b1 becomes phi1's own + // cancellation residue. A "fixed" phi gives b2 = 1.0 and b1 = 0.0 here. + vllm::Ltx2PhiCache cache; + const vllm::Ltx2Res2sCoefficients cliff = vllm::Ltx2GetRes2sCoefficients(1e-10, cache, 0.5); + CHECK(cliff.b2 == 0.0); + CHECK(cliff.b1 == 1.000000082740371); + + // A cache SHARED across calls returns what a fresh one does. Upstream relies + // on this by threading one cache through the whole loop (samplers.py:287), and + // a keying defect — dropping `j`, say — shows here and nowhere else. + vllm::Ltx2PhiCache shared; + for (int64_t i = 0; i < vllm_test::kLtx2Res2sCoeffCount; ++i) { + const vllm::Ltx2Res2sCoefficients got = + vllm::Ltx2GetRes2sCoefficients(vllm_test::kLtx2Res2sCoeffH[i], shared, 0.5); + CHECK(got.a21 == vllm_test::kLtx2Res2sCoeffA21[i]); + CHECK(got.b1 == vllm_test::kLtx2Res2sCoeffB1[i]); + CHECK(got.b2 == vllm_test::kLtx2Res2sCoeffB2[i]); + } + // THIRTY, not thirty-three, and the shortfall is the cache doing its job. + // Three of these h values are twice another, and `a21` asks for phi at + // `-h * 0.5` while `b1` asks for it at `-h`: h = 0.25 reuses the entry + // h = 0.125 made, h = 0.5 reuses h = 0.25's, and h = 1.0 reuses h = 0.5's. + // So 11 * 3 - 3 = 30. Asserted as the exact number rather than as an + // inequality, because "fewer than 33" is also what a cache keyed on `neg_h` + // ALONE would report — and that cache would return phi_1 where phi_2 was + // asked for. The `count()` assertions in the sweep above pin the key's shape; + // this pins how many survived sharing. + CHECK(shared.size() == 30u); +} + +TEST_CASE("ltx2 res2s the loop NORMALIZES its noise, unlike the ancestral loop") { + // `_get_new_noise` (samplers.py:164-170) against `_get_plain_noise` + // (:155-157). The res_2s loop defaults to the first and the ancestral loop to + // the second, ten lines apart in one file, so reading one off the other drops + // this step and nothing about the rendered clip says so. + const size_t n = static_cast(vllm_test::kLtx2Res2sLatentCount); + const std::vector raw(vllm_test::kLtx2Res2sNoiseRaw, + vllm_test::kLtx2Res2sNoiseRaw + n); + const std::vector got = vllm::Ltx2Res2sNormalizeNoise(raw); + const double worst = MaxAbsDiffD(got, vllm_test::kLtx2Res2sNoiseNormalized, n); + INFO("max|diff| against upstream's own _channelwise_normalize = ", worst); + CHECK(worst < 1e-12); + + // THE EXPECTED VALUE IS NOT REACHABLE BY ACCIDENT, three ways. A pass-through + // returns `raw`, which is far from the golden; the golden's mean is 0 and its + // unbiased standard deviation is 1, and neither is true of the input. + CHECK(MaxAbsDiffD(raw, vllm_test::kLtx2Res2sNoiseNormalized, n) > 0.5); + double sum = 0.0, sq = 0.0; + for (const double v : got) sum += v; + for (const double v : got) sq += (v - sum / static_cast(n)) * (v - sum / static_cast(n)); + CHECK(std::fabs(sum) < 1e-12); + CHECK(std::fabs(std::sqrt(sq / static_cast(n - 1)) - 1.0) < 1e-12); + // A zero-filled buffer has no standard deviation to divide by, so it must NOT + // reproduce the golden — the shape a sibling row's width test passed on. + // + // ASSERTED ON `isnan` AND NOT THROUGH `MaxAbsDiffD`, because the distance + // instrument cannot see this. `std::max(worst, NaN)` returns `worst`, so a + // buffer of NaNs measures as max|diff| = 0 and the control reads as "the zeros + // reproduced the golden exactly" — which is how this assertion first passed + // for the opposite of its stated reason. This file's own header records the + // same NaN drop in `MaxAbsDiff`; the lesson had to be re-learned here. + const std::vector zeros(n, 0.0); + const std::vector from_zeros = vllm::Ltx2Res2sNormalizeNoise(zeros); + CHECK(std::isnan(from_zeros[0])); + CHECK_FALSE(std::isnan(got[0])); + // Fewer than two elements has no unbiased standard deviation, so it is refused + // rather than dividing by zero and handing the sampler a NaN latent. + CHECK_FALSE( + RefusalMessage([] { (void)vllm::Ltx2Res2sNormalizeNoise(std::vector{1.0}); }) + .empty()); +} + +TEST_CASE("ltx2 res2s the SDE coefficients run at TWO widths, as upstream hands them") { + // `Res2sDiffusionStep.get_sde_coeff` has no dtype of its own, and the res_2s + // loop reaches it at two: the SUBSTEP injection is handed + // `torch.stack([sigma, sub_sigma])`, both `hp` (samplers.py:342), and the STEP + // injection is handed the loop's own schedule, which `DiffusionStage` created + // as float32 (ti2vid_two_stages_hq.py:268, samplers.py:415). + // + // THE TWO ARMS MUST DISAGREE, or the split is a comment rather than a + // behaviour. They disagree by about one part in 1e7, which is exactly why the + // loop golden above needed a one-ulp bound to see it and why this case exists + // beside it: a golden that cannot separate two implementations is not gating + // the choice between them. + const double sigma_next = 0.62; + const double sigma_up = sigma_next * 0.5; + const vllm::Ltx2SdeCoeff f32 = vllm::Ltx2Res2sSdeCoeff(sigma_next, sigma_up); + const vllm::Ltx2SdeCoeff f64 = vllm::Ltx2Res2sSdeCoeffHp(sigma_next, sigma_up); + INFO("f32 alpha_ratio = ", f32.alpha_ratio, " f64 alpha_ratio = ", f64.alpha_ratio, + " f32 sigma_down = ", f32.sigma_down, " f64 sigma_down = ", f64.sigma_down); + CHECK(f32.alpha_ratio != f64.alpha_ratio); + CHECK(f32.sigma_down != f64.sigma_down); + // ...and they agree to float32 precision, so "they differ" is not a defect in + // one of them. + CHECK(std::fabs(f32.alpha_ratio - f64.alpha_ratio) < 1e-6); + CHECK(std::fabs(f32.sigma_down - f64.sigma_down) < 1e-6); + // `sigma_up` is clamped IN before anything else (diffusion_steps.py:138), on + // both arms. + const vllm::Ltx2SdeCoeff clamped = vllm::Ltx2Res2sSdeCoeffHp(0.5, 2.0); + CHECK(clamped.sigma_up <= 0.5 * vllm::kLtx2Res2sSigmaUpClamp); + // The float64 arm computes the residual in float64, so at the clamp boundary + // it is NOT the float32 arm's value — the residual scales as + // sqrt(1 - clamp^2), which is where the two widths part most visibly. + CHECK(vllm::Ltx2Res2sSdeCoeff(0.5, 2.0).sigma_down != clamped.sigma_down); +} + +TEST_CASE("ltx2 res2s the loop evaluates the transformer TWICE per step") { + // THE DISCRIMINATOR THIS WHOLE ROW RESTS ON. + // + // The res_2s sampler calls the denoiser at `sigmas[i]` (samplers.py:301) and + // again at `sqrt(sigma * sigma_next)` (:315, :380-386), plus once more at the + // injected terminal sigma (:437). The already-shipped Euler arm calls it ONCE + // per step. The two return a clip of the same shape, the same frame count and + // the same sample rate, so this count is the only thing that separates them. + // + // The expected numbers are 6 and 9, chosen so that no stub reaches them: a + // build that evaluates nothing reports 0, a build that evaluates once per step + // reports 3 and 4, and `2 * steps` alone reports 8 on the terminal fixture. + struct Case { + const char* tag; + const float* sigmas; + int64_t sigma_count; + int64_t evaluations; + const double* eval_sigmas; + const int64_t* eval_step_indices; + int64_t full_steps; + }; + const Case cases[] = { + {"BongOn", vllm_test::kLtx2Res2sBongOnSigmas, vllm_test::kLtx2Res2sBongOnSigmaCount, + vllm_test::kLtx2Res2sBongOnEvaluations, vllm_test::kLtx2Res2sBongOnEvalSigmas, + vllm_test::kLtx2Res2sBongOnEvalStepIndices, 3}, + {"BongOffByH", vllm_test::kLtx2Res2sBongOffByHSigmas, + vllm_test::kLtx2Res2sBongOffByHSigmaCount, vllm_test::kLtx2Res2sBongOffByHEvaluations, + vllm_test::kLtx2Res2sBongOffByHEvalSigmas, + vllm_test::kLtx2Res2sBongOffByHEvalStepIndices, 3}, + {"BongOffBySigma", vllm_test::kLtx2Res2sBongOffBySigmaSigmas, + vllm_test::kLtx2Res2sBongOffBySigmaSigmaCount, + vllm_test::kLtx2Res2sBongOffBySigmaEvaluations, + vllm_test::kLtx2Res2sBongOffBySigmaEvalSigmas, + vllm_test::kLtx2Res2sBongOffBySigmaEvalStepIndices, 3}, + {"TerminalZero", vllm_test::kLtx2Res2sTerminalZeroSigmas, + vllm_test::kLtx2Res2sTerminalZeroSigmaCount, + vllm_test::kLtx2Res2sTerminalZeroEvaluations, + vllm_test::kLtx2Res2sTerminalZeroEvalSigmas, + vllm_test::kLtx2Res2sTerminalZeroEvalStepIndices, 4}, + }; + + for (const Case& c : cases) { + Res2sFixture fixture; + const std::vector sigmas(c.sigmas, c.sigmas + c.sigma_count); + vllm::Ltx2Res2sModality video{ + std::vector(vllm_test::kLtx2Res2sVideo0, + vllm_test::kLtx2Res2sVideo0 + vllm_test::kLtx2Res2sLatentCount), + true}; + vllm::Ltx2Res2sModality audio{ + std::vector(vllm_test::kLtx2Res2sAudio0, + vllm_test::kLtx2Res2sAudio0 + vllm_test::kLtx2Res2sLatentCount), + true}; + const vllm::Ltx2Res2sLoopStats stats = + vllm::Ltx2Res2sDenoisingLoop(sigmas, video, audio, fixture.Hooks()); + + INFO("fixture = ", c.tag, " evaluations = ", stats.evaluations, " want ", c.evaluations); + // Upstream's count, measured by running upstream's loop with a counting + // denoiser. The loop's own tally and the hook's own tally must AGREE, so a + // build that reported the number without running the forwards fails. + CHECK(stats.evaluations == c.evaluations); + CHECK(fixture.evaluations == c.evaluations); + CHECK(stats.full_steps == c.full_steps); + // ...and it is not the first-order count. Asserted as an inequality against + // the step count so the case cannot pass by both numbers happening to match. + CHECK(stats.evaluations > 2 * stats.full_steps - 1); + CHECK(stats.evaluations != stats.full_steps); + + // THE SIGMAS THEMSELVES, so a build that ran two forwards at the SAME sigma + // — which would keep the count right and the sampler wrong — fails here. + REQUIRE(stats.eval_sigmas.size() == static_cast(c.evaluations)); + REQUIRE(fixture.eval_sigmas.size() == static_cast(c.evaluations)); + for (int64_t i = 0; i < c.evaluations; ++i) { + INFO("fixture = ", c.tag, " evaluation ", i, " at sigma ", stats.eval_sigmas[i], + " want ", c.eval_sigmas[i]); + CHECK(std::fabs(stats.eval_sigmas[i] - c.eval_sigmas[i]) < 1e-12); + CHECK(stats.eval_sigmas[i] == fixture.eval_sigmas[i]); + } + // Every ODD entry is the geometric mean of its neighbours — `sqrt(sigma * + // sigma_next)`, upstream's "hardcode for c2 = 0.5" (samplers.py:314-315). + // Derived here rather than only read from the golden, so the golden and the + // rule check each other. + for (int64_t i = 0; i + 1 < c.full_steps * 2; i += 2) { + const double sigma = static_cast(sigmas[static_cast(i / 2)]); + const double next = (i / 2 + 1 < c.sigma_count - 1 || sigmas[c.sigma_count - 1] != 0.0f) + ? static_cast(sigmas[static_cast(i / 2 + 1)]) + : static_cast(vllm::kLtx2Res2sTerminalSigma); + CHECK(std::fabs(stats.eval_sigmas[i + 1] - std::sqrt(sigma * next)) < 1e-9); + } + + // THE `step_index` EACH EVALUATION WAS HANDED, which is a SECOND argument + // upstream's `Denoiser` takes and which nothing about the returned latents, + // the evaluation count or a rendered frame records. Upstream passes three + // different things for it — `step_idx` at the first evaluation + // (samplers.py:301), a LITERAL 0 at the substep (samplers.py:385, beside a + // one-element schedule) and `n_full_steps` at the terminal one + // (samplers.py:437) — and the goldens carry the sequence upstream's own loop + // produced. + // + // IT IS NOT COSMETIC. The denoiser reads it through `should_skip_step`, + // which is `step % (skip_step + 1) != 0` (guiders.py:287-291). At the HQ + // preset's `skip_step = 0` every value behaves alike, so this whole + // distinction is INERT on the shipped arm — and it is live the moment a + // request sets `video_skip_step`, where passing the loop counter at the + // substep would skip half of a res_2s step's evaluations and render the + // first-order trajectory under the second-order sampler's schedule. + // + // Asserted against BOTH records: `stats` says what the loop believes it + // passed and `fixture` says what arrived, so a build that recorded one value + // and passed another fails rather than agreeing with itself. + REQUIRE(stats.eval_step_indices.size() == static_cast(c.evaluations)); + REQUIRE(fixture.eval_step_indices.size() == static_cast(c.evaluations)); + for (int64_t i = 0; i < c.evaluations; ++i) { + INFO("fixture = ", c.tag, " evaluation ", i, " step_index ", stats.eval_step_indices[i], + " want ", c.eval_step_indices[i]); + CHECK(stats.eval_step_indices[i] == c.eval_step_indices[i]); + CHECK(fixture.eval_step_indices[i] == c.eval_step_indices[i]); + } + // And the RULE the goldens encode, derived here rather than only read, so + // the two check each other: every substep evaluation is at index 0, every + // full-step evaluation is at its own step, and the terminal one is at + // `n_full_steps`. + for (int64_t step = 0; step < c.full_steps; ++step) { + CHECK(stats.eval_step_indices[2 * step] == step); + CHECK(stats.eval_step_indices[2 * step + 1] == 0); + } + if (c.evaluations == 2 * c.full_steps + 1) { + CHECK(stats.eval_step_indices[c.evaluations - 1] == c.full_steps); + } + } +} + +TEST_CASE("ltx2 res2s the bong refinement runs in its own branch and nowhere else") { + // `bongmath and h < 0.5 and sigma > 0.03` (samplers.py:357), with both + // comparisons STRICT. Each branch is forced by a fixture that CANNOT be + // satisfying the other condition: + // + // BongOn h = 0.118, 0.134, 0.121 sigma = 0.9 .. 0.62 -> runs + // BongOffByH h = 0.588, 0.693, 0.734 sigma = 0.9 .. 0.12 -> h blocks it + // BongOffBySigma h = 0.069, 0.074, 0.039 sigma = 0.03 .. 0.025 -> sigma blocks it + // + // The `h` fixture keeps every sigma above 0.03 and the sigma fixture keeps + // every h below 0.5, so neither is off for the other's reason. + // + // WHAT THIS CASE DOES *NOT* GATE, stated because it was claimed here and was + // false. The sigma fixture starts at the literal 0.03, and that was written as + // "pinning the inequality as strict". It does not. The schedule is float32, so + // `0.03f` widens to 0.029999999329447746, which is BELOW the double 0.03 the + // guard compares against — the boundary is a value no float32 schedule can + // hold, so `>` and `>=` are indistinguishable through this loop's interface. + // MEASURED: relaxing the guard to `>=` leaves this case green (mutation M5 in + // .agents/specs/ltx25-res2s-loop.md section 8). Upstream compares the same + // widened float32 against the same Python float (samplers.py:357), so the + // strictness is unobservable THERE too and no fixture can be built for it. + // Recorded as ungated rather than left reading as covered. + struct Case { + const char* tag; + const float* sigmas; + int64_t sigma_count; + int64_t bong_steps; + bool bong_moved; + const float* no_bong_video; + }; + const Case cases[] = { + {"BongOn", vllm_test::kLtx2Res2sBongOnSigmas, vllm_test::kLtx2Res2sBongOnSigmaCount, 3, + vllm_test::kLtx2Res2sBongOnBongMoved, vllm_test::kLtx2Res2sBongOnNoBongVideo}, + {"BongOffByH", vllm_test::kLtx2Res2sBongOffByHSigmas, + vllm_test::kLtx2Res2sBongOffByHSigmaCount, 0, + vllm_test::kLtx2Res2sBongOffByHBongMoved, vllm_test::kLtx2Res2sBongOffByHNoBongVideo}, + {"BongOffBySigma", vllm_test::kLtx2Res2sBongOffBySigmaSigmas, + vllm_test::kLtx2Res2sBongOffBySigmaSigmaCount, 0, + vllm_test::kLtx2Res2sBongOffBySigmaBongMoved, + vllm_test::kLtx2Res2sBongOffBySigmaNoBongVideo}, + {"TerminalZero", vllm_test::kLtx2Res2sTerminalZeroSigmas, + vllm_test::kLtx2Res2sTerminalZeroSigmaCount, 2, + vllm_test::kLtx2Res2sTerminalZeroBongMoved, + vllm_test::kLtx2Res2sTerminalZeroNoBongVideo}, + }; + + for (const Case& c : cases) { + const std::vector sigmas(c.sigmas, c.sigmas + c.sigma_count); + const auto run = [&](bool bongmath) { + Res2sFixture fixture; + vllm::Ltx2Res2sModality video{ + std::vector(vllm_test::kLtx2Res2sVideo0, + vllm_test::kLtx2Res2sVideo0 + vllm_test::kLtx2Res2sLatentCount), + true}; + vllm::Ltx2Res2sModality audio{ + std::vector(vllm_test::kLtx2Res2sAudio0, + vllm_test::kLtx2Res2sAudio0 + vllm_test::kLtx2Res2sLatentCount), + true}; + vllm::Ltx2Res2sLoopParams params; + params.bongmath = bongmath; + const vllm::Ltx2Res2sLoopStats stats = + vllm::Ltx2Res2sDenoisingLoop(sigmas, video, audio, fixture.Hooks(), params); + return std::pair, vllm::Ltx2Res2sLoopStats>{video.latent, stats}; + }; + + const auto on = run(true); + const auto off = run(false); + INFO("fixture = ", c.tag, " bong steps = ", on.second.bong_steps, " want ", c.bong_steps); + // WHICH STEPS refined, counted. A build whose guard used `<=` on either + // comparison, or `or` for `and`, reports a different number here even where + // the latents happen to agree. + CHECK(on.second.bong_steps == c.bong_steps); + CHECK(off.second.bong_steps == 0); + // Turning the refinement off never changes how many forwards ran, which is + // why the evaluation count above cannot see this branch at all. + CHECK(on.second.evaluations == off.second.evaluations); + + // The refinement's EFFECT, against upstream's own bongmath=False run rather + // than against a value this port computed. `bong_moved` came out of the + // generator, so "it changed the result" is upstream's observation. + const double moved = MaxAbsDiff(on.first, off.first.data(), off.first.size()); + INFO("fixture = ", c.tag, " max|on - off| = ", moved, " upstream says moved = ", + c.bong_moved); + CHECK((moved > 0.0) == c.bong_moved); + // ...and the bongmath=False arm matches upstream's bongmath=False output, so + // "identical" is not being satisfied by both arms being broken the same way. + const double against_upstream = + MaxAbsDiff(off.first, c.no_bong_video, off.first.size()); + INFO("fixture = ", c.tag, " max|diff| vs upstream (bongmath off) = ", against_upstream); + CHECK(against_upstream < kRoundOff); + } +} + +TEST_CASE("ltx2 res2s the loop reproduces upstream") { + // THE BOUND IS ONE f32 ulp, NOT THIS FILE'S `kRoundOff`. + // + // Measured against upstream's own loop output, three of the five fixtures come + // back BIT-EXACT and the other two move by 2.98e-08, which is one ulp at 0.5. + // The bound is set there on purpose. At `kRoundOff` (5e-6) this case cannot + // see the float32/float64 split the step-level SDE coefficients run at + // (samplers.py:415 against :342), which shifts the result by about 1e-7 + // relative — MEASURED: with the split collapsed onto float64 this case stayed + // GREEN at 5e-6 and REDS at 1e-7. A tolerance is a claim about how much + // disagreement is round-off, and 5e-6 was a claim this port could not defend. + constexpr double kOneUlp = 1e-7; + struct Case { + const char* tag; + const float* sigmas; + int64_t sigma_count; + double eta; + const float* video; + const float* audio; + }; + const Case cases[] = { + {"BongOn", vllm_test::kLtx2Res2sBongOnSigmas, vllm_test::kLtx2Res2sBongOnSigmaCount, + vllm_test::kLtx2Res2sBongOnEta, vllm_test::kLtx2Res2sBongOnVideo, + vllm_test::kLtx2Res2sBongOnAudio}, + {"BongOffByH", vllm_test::kLtx2Res2sBongOffByHSigmas, + vllm_test::kLtx2Res2sBongOffByHSigmaCount, vllm_test::kLtx2Res2sBongOffByHEta, + vllm_test::kLtx2Res2sBongOffByHVideo, vllm_test::kLtx2Res2sBongOffByHAudio}, + {"BongOffBySigma", vllm_test::kLtx2Res2sBongOffBySigmaSigmas, + vllm_test::kLtx2Res2sBongOffBySigmaSigmaCount, vllm_test::kLtx2Res2sBongOffBySigmaEta, + vllm_test::kLtx2Res2sBongOffBySigmaVideo, vllm_test::kLtx2Res2sBongOffBySigmaAudio}, + {"TerminalZero", vllm_test::kLtx2Res2sTerminalZeroSigmas, + vllm_test::kLtx2Res2sTerminalZeroSigmaCount, vllm_test::kLtx2Res2sTerminalZeroEta, + vllm_test::kLtx2Res2sTerminalZeroVideo, vllm_test::kLtx2Res2sTerminalZeroAudio}, + // THE ONLY FIXTURE THAT SEPARATES THE TWO ETAS. The substep injection is + // pinned at 0.5 whatever the loop's own eta is (samplers.py:273-274), and + // with the loop at its own default of 0.5 the two are the same number, so + // a build that read `eta` at the substep is INVISIBLE on every fixture + // above. MEASURED: that build stayed green on all four before this row + // was added. + {"Eta1", vllm_test::kLtx2Res2sEta1Sigmas, vllm_test::kLtx2Res2sEta1SigmaCount, + vllm_test::kLtx2Res2sEta1Eta, vllm_test::kLtx2Res2sEta1Video, + vllm_test::kLtx2Res2sEta1Audio}, + }; + + const size_t n = static_cast(vllm_test::kLtx2Res2sLatentCount); + for (const Case& c : cases) { + Res2sFixture fixture; + const std::vector sigmas(c.sigmas, c.sigmas + c.sigma_count); + vllm::Ltx2Res2sModality video{ + std::vector(vllm_test::kLtx2Res2sVideo0, vllm_test::kLtx2Res2sVideo0 + n), true}; + vllm::Ltx2Res2sModality audio{ + std::vector(vllm_test::kLtx2Res2sAudio0, vllm_test::kLtx2Res2sAudio0 + n), true}; + vllm::Ltx2Res2sLoopParams params; + params.eta = c.eta; + (void)vllm::Ltx2Res2sDenoisingLoop(sigmas, video, audio, fixture.Hooks(), params); + + const double vworst = MaxAbsDiff(video.latent, c.video, n); + const double aworst = MaxAbsDiff(audio.latent, c.audio, n); + INFO("fixture = ", c.tag, " eta = ", c.eta, " video max|diff| = ", vworst, + " audio max|diff| = ", aworst); + CHECK(vworst < kOneUlp); + CHECK(aworst < kOneUlp); + + // `post_process_latent` IS APPLIED, and the fixture's mask is what makes + // that checkable: at the two positions the video mask zeroes, the result + // must be the CLEAN latent exactly, whatever the sampler did. A build that + // dropped the blend returns a denoised value there and fails. + for (size_t i = 0; i < n; ++i) { + if (vllm_test::kLtx2Res2sMask[i] != 0.0f) continue; + INFO("fixture = ", c.tag, " masked video position ", i); + CHECK(video.latent[i] == vllm_test::kLtx2Res2sClean[i]); + } + // The two modalities did not receive each other's mask: the audio mask is + // the video one reversed, so the positions that must hold `clean` differ. + for (size_t i = 0; i < n; ++i) { + if (vllm_test::kLtx2Res2sMask[n - 1 - i] != 0.0f) continue; + INFO("fixture = ", c.tag, " masked audio position ", i); + CHECK(audio.latent[i] == vllm_test::kLtx2Res2sClean[n - 1 - i]); + } + } + + // A loop with no modality at all is refused (samplers.py:258-259), rather than + // returning two empty latents that a caller would decode into a blank clip. + Res2sFixture fixture; + vllm::Ltx2Res2sModality absent_v{{}, false}, absent_a{{}, false}; + const std::vector sigmas{1.0f, 0.5f, 0.0f}; + const std::string refusal = RefusalMessage( + [&] { (void)vllm::Ltx2Res2sDenoisingLoop(sigmas, absent_v, absent_a, fixture.Hooks()); }); + INFO("refusal = ", refusal); + CHECK_FALSE(refusal.empty()); + CHECK(Mentions(refusal, "samplers.py:258-259")); +} + +TEST_CASE("ltx2 the res2s_two_stage recipe is upstream's HQ preset") { + const vllm::Ltx2PipelineRecipe hq = vllm::ResolveLtx2PipelineRecipe("res2s_two_stage", "2.5"); + REQUIRE(hq.phases.size() == 2u); + + // THE THING THAT MAKES IT HQ. `stepper=Res2sDiffusionStep()` and + // `loop=res2s_audio_video_denoising_loop` on BOTH stages + // (ti2vid_two_stages_hq.py:258, :285/:292, :319/:335). Asserted against the + // distilled two-stage recipe in the same case, which selects a DIFFERENT + // sampler on each of its phases, so this cannot pass by every recipe having + // the same value. + CHECK(hq.phases[0].stepper == vllm::Ltx2StepperKind::kRes2s); + CHECK(hq.phases[1].stepper == vllm::Ltx2StepperKind::kRes2s); + const vllm::Ltx2PipelineRecipe distilled = + vllm::ResolveLtx2PipelineRecipe("distilled_two_stage", "2.5"); + CHECK(distilled.phases[0].stepper == vllm::Ltx2StepperKind::kEulerAncestral); + CHECK(distilled.phases[1].stepper == vllm::Ltx2StepperKind::kEuler); + + // LTX_2_3_HQ_PARAMS (constants.py:95-115): 15 steps, STG OFF on both + // modalities, video rescale 0.45 and audio rescale 1.0. Fifteen against the + // 2.4 lineage's thirty is the whole economics of the preset — half the steps, + // twice the evaluations each. + CHECK(hq.num_inference_steps == 15); + CHECK(vllm::ResolveLtx2PipelineRecipe("one_stage", "2.5").num_inference_steps == 30); + CHECK(hq.phases[0].video_guidance.stg_scale == 0.0); + CHECK(hq.phases[0].audio_guidance.stg_scale == 0.0); + CHECK(hq.phases[0].video_guidance.stg_blocks.empty()); + CHECK(hq.phases[0].audio_guidance.stg_blocks.empty()); + CHECK(hq.phases[0].video_guidance.cfg_scale == 3.0); + CHECK(hq.phases[0].audio_guidance.cfg_scale == 7.0); + CHECK(hq.phases[0].video_guidance.rescale_scale == 0.45); + CHECK(hq.phases[0].audio_guidance.rescale_scale == 1.0); + + // Stage 1 halves (:238-243) and DERIVES its schedule from + // `num_inference_steps` (:260-267) — the one place this recipe differs in KIND + // from the distilled two-stage one, whose stage 1 carries frozen sigmas. + CHECK(hq.phases[0].spatial_downscale == 2); + CHECK(hq.phases[0].sigmas.empty()); + CHECK_FALSE(distilled.phases[0].sigmas.empty()); + CHECK(hq.allow_request_sigmas); + CHECK_FALSE(hq.fixed_num_inference_steps); + + // Stage 2 upsamples (:297), takes STAGE_2_DISTILLED_SIGMAS by DEFAULT ARGUMENT + // (:193), re-noises to its own first sigma (:327, :332) and runs a + // `SimpleDenoiser` (:316) that no request may re-arm. + CHECK(hq.phases[1].spatial_downscale == 1); + CHECK(hq.phases[1].input_transform == vllm::Ltx2PhaseInputTransform::kSpatialUpsample); + CHECK(hq.phases[1].sigmas == distilled.phases[1].sigmas); + CHECK(hq.phases[1].noise_scale == hq.phases[1].sigmas.front()); + CHECK_FALSE(hq.phases[1].allow_guidance_override); + + // :313-315, :339 — "Stage 2 refines video only; discard its audio". The audio + // that leaves is STAGE 1's, and taking stage 2's would decode a soundtrack the + // pipeline throws away: finite, the right length, the wrong take. + CHECK(hq.video_output_phase == 1); + CHECK(hq.audio_output_phase == 0); + + // :210 — the prompt encoder is handed `[prompt, negative_prompt]` and stage 1 + // builds a `GuidedDenoiser` with the negative encoding, so unlike the + // distilled arm this pipeline HAS a negative prompt. + CHECK(hq.allow_negative_prompt); + CHECK_FALSE(hq.negative_prompt.empty()); + CHECK(distilled.negative_prompt.empty()); + + // The geometry is the FINAL output's; stage 1 runs at half of it. + // `assert_resolution(is_two_stage=True)` (:199) is what the engine then + // enforces against a request. + CHECK(hq.max_spatial_downscale() == 2); + + // 2.5 ONLY. `LTX_2_3_HQ_PARAMS` is a plain constant with no `detect_params` + // lineage (constants.py:91-94), so there is no second version to resolve it + // onto and every other pair still refuses BY NAME. + for (const std::string& version : {std::string("2"), std::string("2.3"), std::string("2.4"), + std::string("2.6")}) { + const std::string message = RefusalMessage( + [&] { (void)vllm::ResolveLtx2PipelineRecipe("res2s_two_stage", version); }); + INFO("version = ", version, " refusal = ", message); + CHECK_FALSE(message.empty()); + CHECK(Mentions(message, "Unsupported LTX pipeline kind/version")); + CHECK(Mentions(message, version)); + } +} diff --git a/tests/vllm/multimodal/test_ltx2_video.cpp b/tests/vllm/multimodal/test_ltx2_video.cpp index 09fb59f9c..8758420eb 100644 --- a/tests/vllm/multimodal/test_ltx2_video.cpp +++ b/tests/vllm/multimodal/test_ltx2_video.cpp @@ -5887,6 +5887,313 @@ TEST_CASE("ltx2 t2a: rescale_scale 0 is the control because both spaces agree th CHECK(at_default > 100.0 * at_zero); } +// ─── the HQ arm reaches the res_2s sampler (row LTX25-RES2S-LOOP, #921) ───── +// +// THIS IS THE REACHABILITY CASE, and it is deliberately not a unit test of the +// loop — `test_ltx2_pipeline` already gates the arithmetic against upstream's +// own output. This one enters through the production path a user arrives on: +// `LoadVideoEngine` with the `pipeline_kind` LOAD extra, then +// `VideoEngine::Generate`, which is what `vllm_video_generate`, `ltx2-gen` and +// the server all call. Deleting the `kRes2s` dispatch in `ltx2_video.cpp`'s +// phase loop must red this case; a unit test of `Ltx2Res2sDenoisingLoop` would +// stay green, because it proves the class works and never that anything +// reaches it. +// +// WHAT IT ASSERTS IS A COUNT, because a count is the only thing that separates +// the two samplers. The rendered clip, its shape, its frame count and its +// sample rate are identical whichever one ran. +TEST_CASE("ltx2 video: the HQ pipeline evaluates the DiT twice per step") { + Workspace ws; + + // `steps` -> forwards, for each arm. The res_2s loop runs two evaluations per + // step plus one at the terminal sigma the schedule injects (samplers.py:281, + // :437), and the first-order loop runs one per step. TWO step counts, so an + // off-by-one cannot satisfy both, and the ratio is close to two rather than a + // difference of one. + const auto forwards = [&ws](const std::string& kind, int64_t steps, const std::string& tag) { + vllm::multimodal::VideoModelParams mp = FixtureParams(ws.paths); + mp.extras[vllm::multimodal::kLtx2PipelineKindExtra] = kind; + // Stage 1 only. Both recipes' second phase needs the latent spatial + // upsampler, which the fixture does not carry and which is refused BY NAME + // in its own case above — that refusal is not what this case is about. + mp.extras[vllm::multimodal::kLtx2MaxPhaseExtra] = "0"; + const std::unique_ptr engine = + vllm::multimodal::LoadVideoEngine(mp); + REQUIRE(engine != nullptr); + auto* ltx2 = dynamic_cast(engine.get()); + REQUIRE(ltx2 != nullptr); + vllm::multimodal::VideoGenParams gen = FixtureGen(ws.root + "/" + tag); + gen.steps = steps; + // `one_stage` resolves `stg_blocks = [28]` (constants.py:86-87) and this + // fixture's DiT has two blocks, so its PERTURBED pass is refused by name + // unless the request names a block that exists. The HQ preset ships + // `stg_blocks = []` beside `stg_scale = 0.0` (constants.py:105, :113) and + // asks for no perturbed pass at all, so it needs no override — and giving it + // one would put a request override on the arm this case is measuring. + if (kind == "one_stage") OneStageFixtureGuidance(&gen); + (void)engine->Generate(gen); + return ltx2->last_conditioning(); + }; + + const vllm::multimodal::Ltx2ConditioningTrace hq3 = forwards("res2s_two_stage", 3, "hq3"); + const vllm::multimodal::Ltx2ConditioningTrace hq5 = forwards("res2s_two_stage", 5, "hq5"); + const vllm::multimodal::Ltx2ConditioningTrace euler3 = forwards("one_stage", 3, "e3"); + const vllm::multimodal::Ltx2ConditioningTrace euler5 = forwards("one_stage", 5, "e5"); + + INFO("res2s: 3 steps -> " << hq3.dit_evaluations << " forwards, 5 steps -> " + << hq5.dit_evaluations << "; euler: 3 -> " + << euler3.dit_evaluations << ", 5 -> " << euler5.dit_evaluations); + // 2 * steps + 1. The schedule `Ltx2SigmaSchedule` builds terminates at exactly + // 0 (gated in test_ltx2_pipeline), so the terminal evaluation always happens. + CHECK(hq3.dit_evaluations == 7); + CHECK(hq5.dit_evaluations == 11); + // ...against the first-order arm on the SAME request. Both numbers are read + // off a real render rather than one being computed from the other, so the + // comparison cannot be satisfied by both arms sharing a defect. + CHECK(euler3.dit_evaluations == 3); + CHECK(euler5.dit_evaluations == 5); + CHECK(hq3.dit_evaluations > 2 * euler3.dit_evaluations); + CHECK(hq5.dit_evaluations > 2 * euler5.dit_evaluations); + // A ZERO WOULD ALSO BE "not equal to the Euler count", and zero is what a + // build that never ran the loop reports. Ruled out explicitly. + CHECK(euler3.dit_evaluations > 0); + + // THE BONG REFINEMENT IS REACHED ON THE PRODUCTION SCHEDULE, not only on the + // hand-built fixtures in test_ltx2_pipeline. It changes the latent without + // changing how many forwards ran, so the counter above is blind to it and this + // is the only place a real render says it happened. + CHECK(hq3.res2s_bong_steps > 0); + CHECK(hq5.res2s_bong_steps > 0); + // ...and never on a first-order arm, which has no anchor to refine. + CHECK(euler3.res2s_bong_steps == 0); + CHECK(euler5.res2s_bong_steps == 0); + + // THE NOISE THE ENGINE HANDED THE LOOP WAS NORMALIZED. `_get_new_noise` + // (samplers.py:164-170) is what the res_2s loop takes, against the ancestral + // loop's un-normalized `_get_plain_noise` (:155-157) ten lines away. That the + // FUNCTION normalizes is gated in test_ltx2_pipeline; that this engine calls + // it is a different claim, and MEASURED: with the hook handing over its raw + // draw instead, every assertion above stayed green. + // + // 1e-9 is unreachable for a raw Gaussian draw, whose sample moments miss by + // O(1/sqrt(n)) on any latent this fixture builds, and trivial for a + // normalized one, which is exact to rounding. + INFO("res2s noise moment error = " << hq3.res2s_noise_moment_error); + CHECK(hq3.res2s_noise_moment_error < 1e-9); + CHECK(hq5.res2s_noise_moment_error < 1e-9); + // Zero — not "small" — on an arm that runs no res_2s draw at all, so the + // field cannot read as satisfied by never having been written. + CHECK(euler3.res2s_noise_moment_error == 0.0); + + // BOTH ARMS BUILT THEIR SCHEDULE THE SAME WAY, which is what lets the two + // counts be compared at all: each recipe leaves stage 1's sigmas empty and + // therefore derives them from `steps` through `Ltx2SigmaSchedule`, so the + // difference between 7 and 3 is the SAMPLER and not a different schedule. + // Their token counts differ — the HQ stage 1 halves the request + // (ti2vid_two_stages_hq.py:238-243) and `one_stage` does not — which is why + // the counts above are asserted absolutely rather than only as a ratio. + CHECK(hq3.schedule_tokens > 0); + CHECK(euler3.schedule_tokens > 0); + CHECK(hq3.video_tokens < euler3.video_tokens); +} + +// ─── the HQ arm is GUIDED, and the evaluation count cannot see that ───────── +// +// THIS IS A SEPARATE CASE FROM THE ONE ABOVE BECAUSE IT IS A SEPARATE DEFECT, +// and the one above is blind to it. A render's DiT work is +// `evaluations x forwards-per-evaluation`. The sampler decides the first factor +// and the denoiser decides the second, and `dit_evaluations` — the whole +// instrument of the case above — is exactly the first factor. Route the res_2s +// loop around a bare `Ltx2DitForward` instead of `Ltx2GuidedDenoise` and +// `dit_evaluations` stays at 2n+1, `res2s_bong_steps` stays right, the eval +// sigmas stay right, the clip keeps its shape, frame count, sample rate and file +// size, and the preset renders at cfg 1.0 where upstream tuned it at 3.0. +// +// Upstream's HQ stage 1 runs a `GuidedDenoiser` (ti2vid_two_stages_hq.py:271-281) +// built from `LTX_2_3_HQ_PARAMS` — cfg 3.0 video / 7.0 audio, rescale 0.45, +// modality 3.0, stg 0.0, stg_blocks [] (utils/constants.py:99-114). So each of +// stage 1's evaluations is THREE transformer forwards: `cond` always +// (denoisers.py:100), `uncond` because cfg != 1.0 (:102-109, guiders.py:275-277) +// and `mod` because modality_scale != 1.0 (:121-137, guiders.py:283-285). No +// `ptb`, because stg_scale is 0.0. +TEST_CASE("ltx2 video: the HQ pipeline stage 1 is GUIDED, three forwards per evaluation") { + Workspace ws; + + const auto render = [&ws](const std::string& kind, int64_t steps, const std::string& tag) { + vllm::multimodal::VideoModelParams mp = FixtureParams(ws.paths); + mp.extras[vllm::multimodal::kLtx2PipelineKindExtra] = kind; + // Stage 1 only, for the reason the case above gives: the second phase needs + // the latent spatial upsampler the fixture does not carry. + mp.extras[vllm::multimodal::kLtx2MaxPhaseExtra] = "0"; + const std::unique_ptr engine = + vllm::multimodal::LoadVideoEngine(mp); + REQUIRE(engine != nullptr); + auto* ltx2 = dynamic_cast(engine.get()); + REQUIRE(ltx2 != nullptr); + vllm::multimodal::VideoGenParams gen = FixtureGen(ws.root + "/" + tag); + gen.steps = steps; + // The HQ preset ships `stg_blocks = []` on both modalities beside + // `stg_scale = 0.0`, so unlike `one_stage` it needs no block override to run + // on a reduced-block fixture — the perturbed pass is not requested at all. + (void)engine->Generate(gen); + return ltx2->last_conditioning(); + }; + + const vllm::multimodal::Ltx2ConditioningTrace hq3 = render("res2s_two_stage", 3, "ghq3"); + const vllm::multimodal::Ltx2ConditioningTrace hq5 = render("res2s_two_stage", 5, "ghq5"); + + // THE GUIDER THE PHASE RESOLVED, so a recipe that quietly lost `LTX_2_3_HQ_PARAMS` + // fails here rather than rendering at the defaults. + CHECK(hq3.video_guidance_cfg_scale == 3.0); + CHECK(hq3.video_guidance_stg_scale == 0.0); + CHECK(hq3.video_guidance_rescale_scale == 0.45); + CHECK(hq3.video_guidance_modality_scale == 3.0); + // ...and the seam RAN, recorded at the call rather than copied from the params + // above. `RecordFirstGuidedStep` reads `pass_ran`, which the denoiser sets when + // it issues the forward. + REQUIRE(hq3.video_guided); + CHECK(hq3.video_cond_forwards == 1); + CHECK(hq3.video_uncond_forwards == 1); + CHECK(hq3.video_perturbed_forwards == 0); + CHECK(hq3.video_modality_forwards == 1); + + // THE COUNT THAT MOVES WHEN GUIDANCE IS DROPPED, and the one that does not. + // + // `dit_evaluations` is 2n+1 whether or not the arm is guided; `dit_forwards` + // is three times that when it is and equal to it when it is not. Both are + // asserted EXACTLY and on TWO step counts, so neither an off-by-one nor a + // constant factor can satisfy both. + INFO("hq3: evaluations = " << hq3.dit_evaluations << " forwards = " << hq3.dit_forwards); + INFO("hq5: evaluations = " << hq5.dit_evaluations << " forwards = " << hq5.dit_forwards); + CHECK(hq3.dit_evaluations == 7); + CHECK(hq5.dit_evaluations == 11); + CHECK(hq3.dit_forwards == 21); + CHECK(hq5.dit_forwards == 33); + // The relation, derived rather than only read off the two numbers, so a change + // to one of the four constants above cannot be absorbed by changing another. + CHECK(hq3.dit_forwards == 3 * hq3.dit_evaluations); + CHECK(hq5.dit_forwards == 3 * hq5.dit_evaluations); + // AN UNGUIDED ARM IS EXACTLY `forwards == evaluations`, which is the mutation + // this case exists for. Stated as its own assertion rather than left implicit + // in the multiplier, because that is the sentence the RED has to print. + CHECK(hq3.dit_forwards != hq3.dit_evaluations); + + // ...against the arm whose guidance this tree already gated. `one_stage` + // resolves cfg 3.0, stg 1.0 AND modality 3.0, so it runs all FOUR passes and + // the two arms differ in the pass SET as well as in the sampler. Read off a + // real render rather than computed from the HQ numbers. + vllm::multimodal::VideoModelParams mp = FixtureParams(ws.paths); + mp.extras[vllm::multimodal::kLtx2PipelineKindExtra] = "one_stage"; + mp.extras[vllm::multimodal::kLtx2MaxPhaseExtra] = "0"; + const std::unique_ptr engine = + vllm::multimodal::LoadVideoEngine(mp); + REQUIRE(engine != nullptr); + auto* ltx2 = dynamic_cast(engine.get()); + REQUIRE(ltx2 != nullptr); + vllm::multimodal::VideoGenParams gen = FixtureGen(ws.root + "/g1s"); + gen.steps = 3; + OneStageFixtureGuidance(&gen); + (void)engine->Generate(gen); + const vllm::multimodal::Ltx2ConditioningTrace euler3 = ltx2->last_conditioning(); + CHECK(euler3.dit_evaluations == 3); + CHECK(euler3.dit_forwards == 12); + CHECK(euler3.video_perturbed_forwards == 1); + // The HQ arm runs FEWER forwards per evaluation and MORE evaluations, so + // neither counter on its own separates the two arms and both are needed. + CHECK(hq3.dit_evaluations > euler3.dit_evaluations); + CHECK(hq3.dit_forwards > euler3.dit_forwards); +} + +// ─── the SUBSTEP evaluation converts against the MIDPOINT it was handed ───── +// +// The res_2s second evaluation runs over `x_mid` (samplers.py:369-378), a state +// that never becomes the stream's own latent. Everywhere else in `ltx2_video.cpp` +// "the latent" and "the latent this evaluation was handed" are the same tensor, +// which is what makes `ToDenoised(video.latent, ...)` an easy write here and an +// invisible one: MEASURED, with that substitution in place this whole file +// stayed GREEN at 74 cases and 2234 assertions. The clip, the evaluation count, +// the forward count, the eval sigmas and the bong count are all blind to it, and +// the loop's own arithmetic is gated with a FIXTURE denoiser that never performs +// this conversion at all. +TEST_CASE("ltx2 video: the res_2s SUBSTEP converts x0 against the midpoint, not the state") { + Workspace ws; + vllm::multimodal::VideoModelParams mp = FixtureParams(ws.paths); + mp.extras[vllm::multimodal::kLtx2PipelineKindExtra] = "res2s_two_stage"; + mp.extras[vllm::multimodal::kLtx2MaxPhaseExtra] = "0"; + const std::unique_ptr engine = + vllm::multimodal::LoadVideoEngine(mp); + REQUIRE(engine != nullptr); + auto* ltx2 = dynamic_cast(engine.get()); + REQUIRE(ltx2 != nullptr); + vllm::multimodal::VideoGenParams gen = FixtureGen(ws.root + "/sub"); + gen.steps = 3; + (void)engine->Generate(gen); + const vllm::multimodal::Ltx2ConditioningTrace t = ltx2->last_conditioning(); + + // The substep ran at all, and it ran on the res_2s arm. + REQUIRE(t.res2s_substep_latent.size() == t.video_first_latent.size()); + REQUIRE(!t.res2s_substep_latent.empty()); + REQUIRE(t.res2s_substep_cond.size() == t.res2s_substep_latent.size()); + REQUIRE(t.res2s_substep_cond_velocity.size() == t.res2s_substep_latent.size()); + // ONE TIMESTEP PER TOKEN, not per element: `timesteps_from_mask` is per token + // and `to_denoised` broadcasts it across the token's whole row. A conditioned + // token sits at timestep 0, which is why the scalar sigma cannot stand in. + const size_t tokens = t.res2s_substep_timesteps.size(); + REQUIRE(tokens > 0); + REQUIRE(t.res2s_substep_latent.size() % tokens == 0); + const size_t width = t.res2s_substep_latent.size() / tokens; + + // NON-VACUITY, twice, because both zeros make the assertion below trivially + // true. The midpoint MOVED — `x_mid = x_anchor + h * a21 * eps_1` + // (samplers.py:322) is not the anchor — so a build that evaluated the substep + // over the unmoved state would satisfy the invariant against either tensor and + // this case would prove nothing. + const auto abs_max = [](const std::vector& v) { + double m = 0.0; + for (const float x : v) m = std::max(m, std::abs(static_cast(x))); + return m; + }; + const auto abs_diff = [](const std::vector& a, const std::vector& b) { + REQUIRE(a.size() == b.size()); + double m = 0.0; + for (size_t i = 0; i < a.size(); ++i) { + m = std::max(m, std::abs(static_cast(a[i]) - static_cast(b[i]))); + } + return m; + }; + const double moved = abs_diff(t.res2s_substep_latent, t.video_first_latent); + INFO("midpoint moved by " << moved); + REQUIRE(moved > 1e-6); + REQUIRE(abs_max(t.res2s_substep_cond_velocity) > 1e-6); + // ...and the substep sigma is the geometric mean, not the step's own + // (samplers.py:314-315), so this really is the second evaluation. + CHECK(t.res2s_substep_sigma < t.video_first_sigma); + CHECK(t.res2s_substep_sigma > 0.0); + + // THE INVARIANT: `x0 == latent - timesteps * velocity` (model.py:590-604), + // over the latent THIS evaluation was handed. An equation between four + // recorded vectors, not a magnitude, so no fixture scale satisfies it by + // accident. With the conversion reading `video.latent` the residual is + // exactly `video_first_latent - res2s_substep_latent`, whose max is the + // `moved` printed above. + double worst = 0.0; + for (size_t token = 0; token < tokens; ++token) { + const double sigma = static_cast(t.res2s_substep_timesteps[token]); + for (size_t w = 0; w < width; ++w) { + const size_t i = token * width + w; + const double want = static_cast(t.res2s_substep_latent[i]) - + sigma * static_cast(t.res2s_substep_cond_velocity[i]); + worst = std::max(worst, std::abs(static_cast(t.res2s_substep_cond[i]) - want)); + } + } + INFO("substep |x0 - (latent - t*v)| = " << worst << " against a midpoint that moved " << moved); + CHECK(worst < 1e-5); + // And the residual is orders of magnitude below the displacement it would be + // if the wrong latent had been used, so the tolerance above cannot be + // absorbing the defect. + CHECK(worst < 0.01 * moved); +} + // ─── row LTX25-GUIDED-VIDEO (#1092): the guided VIDEO denoiser ────────────── // // The video denoise loop ran ONE unguided forward per step and applied