feat(LTX25-RES2S-LOOP): the res_2s loop, its second evaluation, and the guidance the naive merge would have dropped (#921)

.agents/specs/ltx25-resolution-envelope.md

@@ -513,6 +513,12 @@ takes a different path entirely.
   on the Euler loop would render a plausible clip that is quietly not HQ at
   roughly half the model evaluations the preset was tuned for. No HQ recipe row
   is added by this row, so nothing can select it and nothing lands dead.
+  **TAKEN by row `LTX25-RES2S-LOOP`, spec
+  [`ltx25-res2s-loop.md`](ltx25-res2s-loop.md).** The entry stays here rather
+  than being deleted, because this file is where the issue's owner was recorded
+  and the pointer is the provenance; that spec's own `## Owed` carries what
+  remains of it, which is a real-checkpoint render and the `legacy_mode=False`
+  arm.
 - `TI2VidTwoStagesPipeline` as a recipe row — stage 1 on the scheduler-derived
   schedule under full CFG, stage 2 on `STAGE_2_DISTILLED_SIGMAS` with guidance
   off (`ti2vid_two_stages.py:243-308`). Distinct from the distilled two-stage

CMakeLists.txt

@@ -847,6 +847,13 @@ add_library(vllm STATIC
   # the latent spatial upsampler, the duration head and the embeddings
   # connector. Additive files mirroring ltx_core's own structure.
   src/vllm/model_executor/models/ltx2_pipeline.cpp
+  # LTX-2.5 (ROW LTX25-RES2S-LOOP, issue #921): the res_2s second-order sampler.
+  # Its own TU because upstream partitions it that way — a STEPPER advances one
+  # substep and lives in ltx-core/components/diffusion_steps.py, which
+  # ltx2_pipeline.cpp mirrors, while a SAMPLER decides how many substeps there
+  # are and what is evaluated between them and lives in
+  # ltx-pipelines/utils/samplers.py. Two upstream packages, two files here.
+  src/vllm/model_executor/models/ltx2_samplers.cpp
   src/vllm/model_executor/models/ltx2_upsampler.cpp
   # LTX-2.5 (ROW LTX25-DFR-PIPELINE, issue #986): the DFR canvas layout — the
   # keyframe segment grid, the temporal tile ranges and the latent stitch. Its

docs/FEATURES.md

@@ -174,6 +174,7 @@ in `ltx2_text_encoder.cpp` is the call that would have to change.
 | LTX-2.5 Conv VAE decode threading | LTX-2.5 video VAE | `test_ltx2_vae` "the decode DISPATCHES its convolutions to the CPU threadpool" and "...BIT-IDENTICAL across thread counts", through `Ltx2VideoDecodeStreaming`; 34 golden margins UNCHANGED; TSan clean | **Parallel** over CONV output lines via `vt::cpu::ParallelForRows` ([#1009](https://github.com/mudler/vllm.cpp/issues/1009)). ~9x at 16-20 workers, contended box, 21-23% spread. Bit-identical at any count |
 | LTX-2.5 retake (`RetakePipeline`, regenerate a time window) | LTX-2.5 DiT + video VAE encoder | `test_ltx2_retake` 4/4 (69 assertions) and 4 `test_ltx2_video` cases entering through `Generate`; mask, conform and the four-way plan pinned to upstream `fd4ded7f` | `--pipeline-kind retake` on `ltx2-gen`. Source is a `frame_%06d.ppm` DIRECTORY; a container is REFUSED (no demuxer). Geometry comes from the clip. A folder has no audio, so the soundtrack is generated |
 | LTX-2.5 text-to-audio (`T2AOneStagePipeline`) | LTX-2.5 DiT + audio VAE, no video VAE | `test_ltx2_video`'s `ltx2 t2a:` cases, entering through `Generate`; 18 mutations, 17 DETECTED (four by review of a conditional-only #1039 gate) and the 18th proven an identity, not a blind spot | `--pipeline-kind t2a_one_stage`. NO picture: 0 frames, no mux argv. The only AUDIO-ONLY guided arm (CFG + STG, 3 forwards/step), so it needs a text tower. CPU only; the device forward is refused by name |
+| LTX-2.5 HQ preset (`TI2VidTwoStagesHQPipeline`, `res_2s` sampler) | LTX-2.5 DiT | 6 `test_ltx2_pipeline` cases + 2 `test_ltx2_video` cases through `Generate`, vs UPSTREAM'S OWN loop run at `fd4ded7f`: video latents BIT-EXACT on 3 of 5 fixtures, 1 ulp on 2. 20 mutations, 18 DETECTED | `--pipeline-kind res2s_two_stage`, 2.5 only. TWO denoiser calls per step plus a terminal one, and stage 1 is GUIDED at cfg 3.0 / modality 3.0, so 15 + 3 steps is 38 calls and 100 forwards. The preset IS the sampler |
 | LTX-2.5 T2A guidance space | LTX-2.5 DiT (T2A arm) | `test_ltx2_video` "the guider is handed x0 predictions" through `Generate`, on all 3 arms plus the guider output and the Euler input; a seam case puts the two spaces 1.5e-07 apart at rescale 0 and 0.352 at 0.7 | Combines **denoised (x0)**, mirroring `X0Model` (`model.py:590-604`). Was velocity space, which agrees only at rescale 0 ([#1039](https://github.com/mudler/vllm.cpp/issues/1039)) |
 | LTX-2.5 VIDEO guidance | LTX-2.5 DiT, joint video+audio | `test_ltx2_video`'s `ltx2 one_stage:` cases through `Generate`; all FOUR arms carry the x0 invariant and the guider output replays EXACTLY | `--pipeline-kind one_stage` runs `_guided_denoise`: 4 forwards/step, combined per modality in **x0**. Was ONE unguided forward, every `video_guidance` field dead ([#1092](https://github.com/mudler/vllm.cpp/issues/1092)) |
 | LTX-2.5 cross-attention perturbations | LTX-2.5 DiT | `test_ltx2_video` gates each direction ALONE, on a forward where one stream is PRESENT but DISABLED so only that one runs: the flag moves the stream it writes, the other leaves it bit-identical. Swapping the two is RED | `SKIP_A2V_CROSS_ATTN` / `SKIP_V2A_CROSS_ATTN` ported, which `modality_scale = 3.0` selects on every video row. The DEVICE forward takes no perturbations, so that pass is refused there by name |

docs/USAGE.md

@@ -2676,8 +2676,8 @@ or without the ComfyUI `model.diffusion_model.` prefix. Each family reads its ow
 knobs from `extras`. H3 takes `partition`. LTX-2.5 takes
 `audio_prompt_embeds_path` (the audio stream's conditioning, the twin of the
 seam's `prompt_embeds_path`, which carries the video stream), `pipeline_kind`
-(default `distilled_two_stage`; also `one_stage`, `dmd2`, `dfr`, `retake` and
-`t2a_one_stage`), `model_version` (only for a checkpoint that
+(default `distilled_two_stage`; also `one_stage`, `res2s_two_stage`, `dmd2`,
+`dfr`, `retake` and `t2a_one_stage`), `model_version` (only for a checkpoint that
 declares none), `dit_config_path`, `encoder_config_path`,
 `negative_prompt_embeds_path` and `negative_audio_prompt_embeds_path` (the
 negative half of the same fallback, for the unconditional forward),
@@ -3168,13 +3168,14 @@ CHECKPOINT_ROOT=... VLLM_CPP_LTX2_TOWER_E2E=1 \
 
 Recipes resolve on an EXACT `(pipeline_kind, model_version)` pair and refuse
 anything else by name rather than defaulting, because a plausible but wrong sigma
-schedule or guidance scale renders a video instead of failing. **Fifteen** pairs
-resolve, derived from `ResolveLtx2PipelineRecipe` (`ltx2_pipeline.cpp:1288-1333`):
+schedule or guidance scale renders a video instead of failing. **Sixteen** pairs
+resolve, derived from `ResolveLtx2PipelineRecipe`:
 
 | `pipeline_kind` | resolving `model_version` |
 |---|---|
 | `one_stage` | 2, 2.3, 2.4, 2.5 |
 | `distilled_two_stage` | 2, 2.5 |
+| `res2s_two_stage` | **2.5 only** |
 | `dfr` | **2.5 only** |
 | `dmd2` | 2, 2.3 |
 | `retake` | 2, 2.5 |
@@ -3185,8 +3186,54 @@ This list ran to ten until 2026-08-17, omitting `dfr` entirely and all four
 DFR's base stage rests on generated keyframe slots, which need a checkpoint
 declaring `use_keyframes_abs_pos_embedding`, and the 2.0 distilled row predates
 that parameter — so resolving DFR onto it would build a recipe the engine must
-then refuse at load. Refusing at the recipe table names the version instead
-(`ltx2_pipeline.cpp:1306-1313`).
+then refuse at load. Refusing at the recipe table names the version instead.
+
+### `res2s_two_stage`: the high-quality preset, and why it is a sampler
+
+`res2s_two_stage` is `TI2VidTwoStagesHQPipeline`. Against the plain two-stage
+pipeline it changes the SAMPLER on both stages — the `res_2s` second-order
+method instead of Euler — and takes `LTX_2_3_HQ_PARAMS`: 15 steps, STG off,
+video rescale 0.45, cfg 3.0 video / 7.0 audio, modality 3.0. Those are not the
+only differences (stage 1 also loads the distilled LoRA, derives its schedule
+from the stage-1 latent shape, and runs a `GuidedDenoiser` where the plain
+pipeline runs a `FactoryGuidedDenoiser`), so do not read the sampler swap as an
+exhaustive list. It resolves at 2.5 only, because that preset is a plain
+constant upstream with no per-generation lineage to spread it over.
+
+Fifteen steps is not fewer forwards, and it is not even 15 model calls. The
+`res_2s` loop evaluates the denoiser TWICE per step — once at the step's sigma
+and once at the geometric mean of that sigma and the next — and once more at a
+terminal sigma the schedule injects. Stage 1's 15 steps is therefore 31 denoiser
+calls, and stage 2's frozen 3-step schedule adds 7, for **38 calls per render**.
+Stage 1 is also GUIDED, so each of its calls is three transformer forwards
+(conditional, unconditional, isolated-modality) against stage 2's one: **100
+transformer forwards** for a full render, where `one_stage` at its own 30-step
+default runs 30 calls. Expect the HQ preset to cost several times the 30-step
+arm and to look better, not to be faster.
+
+That is also why the preset cannot be reached by passing its numbers to another
+kind. `--steps 15` on `one_stage` renders a finished, correctly sized, plausible
+clip at a fraction of the model evaluations the preset was tuned for, and no
+property of the output says so. Ask for the pipeline, not for its step count.
+
+```sh
+ltx2-gen --pipeline-kind res2s_two_stage \
+         --prompt "a cinematic shot of ..." \
+         --height 1088 --width 1920 --frames 121
+```
+
+`pipeline_kind` is a LOAD knob, so this reaches the C API and the server too: a
+server started with `--video-extra pipeline_kind=res2s_two_stage` renders every
+request on the HQ preset.
+
+Three limits, stated rather than left to be found. The stage-2 spatial upsample
+is the same one `distilled_two_stage` uses and carries the same refusal when the
+checkpoint has no latent upsampler. The loop's SDE noise is drawn from this
+port's own generator rather than upstream's seeded `torch.randn`, so a render is
+not bit-comparable with Lightricks' — the same limit the ancestral arm already
+ships with. And stage 1's guidance asks for an isolated-modality pass, which the
+device-resident forward cannot perturb, so this preset is host-only until that
+is closed; both are recorded in `.agents/specs/ltx25-res2s-loop.md`.
 
 ### Retake: regenerating a time window of an existing clip
 

include/vllm/model_executor/models/ltx2_pipeline.h

@@ -210,6 +210,46 @@ std::vector<float> Ltx2Res2sStep(const float* sample, const float* denoised,
                                  const float* sigmas, int64_t sigma_count, int64_t step_index,
                                  int64_t count, const float* noise, double eta = 0.5);
 
+// ─── THE SAME STEP, AT THE PRECISION EACH CALL SITE ACTUALLY HANDS IT ────────
+//
+// `Res2sDiffusionStep.step` has no dtype of its own: it takes whatever its
+// tensors carry, and the res_2s loop hands it two DIFFERENT combinations. Both
+// are mirrored rather than unified onto one, because the difference is real
+// arithmetic and putting the conversion where upstream puts it is the rule.
+//
+//   SUBSTEP (samplers.py:337-352). `sigmas = torch.stack([sigma, sub_sigma])`,
+//   and both are `hp` (:291-292, :315). So `get_sde_coeff` runs in FLOAT64.
+//
+//   STEP (samplers.py:412-427). `sigmas` is the loop's own schedule, which
+//   `DiffusionStage` created as FLOAT32 (ti2vid_two_stages_hq.py:268). So
+//   `get_sde_coeff` runs in FLOAT32 — the residual `sqrt(sigma_next^2 -
+//   sigma_up^2)`, `alpha_ratio` and `sigma_down` are all f32 quantities — while
+//   the SAMPLE and the noise are still f64 and the result is f64.
+//
+// The values in both cases are f64, because `sample` is `x_anchor` (`hp`) and
+// `output_dtype = denoised_sample.dtype` is `hp` too (diffusion_steps.py:180).
+//
+// One implementation, instantiated at the two scalar types; there is no second
+// copy of the formula. The selection is an enum naming the two upstream call
+// sites rather than a bare bool, so a reader can check the claim.
+enum class Ltx2Res2sScheduleWidth {
+  // samplers.py:415, :425 — the loop's float32 schedule.
+  kF32Schedule,
+  // samplers.py:342, :350 — the [sigma, sub_sigma] pair, both float64.
+  kF64Schedule,
+};
+
+// `Res2sDiffusionStep.get_sde_coeff` computed in float64 rather than float32.
+// The f32 arm stays `Ltx2Res2sSdeCoeff` above and keeps its goldens.
+Ltx2SdeCoeff Ltx2Res2sSdeCoeffHp(double sigma_next, double sigma_up);
+
+// `Res2sDiffusionStep.step` over float64 samples. `width` decides only the
+// precision the SIGMAS and therefore the coefficients are computed at.
+std::vector<double> Ltx2Res2sStepHp(const double* sample, const double* denoised,
+                                    const double* sigmas, int64_t sigma_count,
+                                    int64_t step_index, int64_t count, const double* noise,
+                                    double eta, Ltx2Res2sScheduleWidth width);
+
 // _get_ancestral_step (diffusion_steps.py:7-22): the DDIM / variance-exploding
 // ancestral coefficients, in the rescaled `sigma / alpha` space. Used only by
 // CFG++.
@@ -518,7 +558,20 @@ bool Ltx2ShouldUseAncestralSampler(const std::string& version);
 // ltx2_recipes.py:38 — how a phase builds its input.
 enum class Ltx2PhaseInputTransform { kInitial, kSpatialUpsample };
 // Which stepper a phase samples with (distilled.py:170-185).
-enum class Ltx2StepperKind { kEuler, kEulerAncestral };
+//
+// `kRes2s` is not only a stepper: it selects a whole SAMPLER. Upstream keeps the
+// two choices separate — `DiffusionStage.__call__` takes `stepper` and `loop`
+// independently (utils/blocks.py:512-513) — but they are not independently
+// selectable in practice, because `res2s_audio_video_denoising_loop` REFUSES any
+// stepper that is not a `Res2sDiffusionStep` (samplers.py:276-277) and no other
+// loop constructs one. `TI2VidTwoStagesHQPipeline` passes both together, to both
+// stages (ti2vid_two_stages_hq.py:285/:292 and :319/:335). One enumerator
+// therefore carries both, and the alternative — a separate loop field whose only
+// legal combination is this one — would publish a selection surface upstream
+// does not have and three combinations that must then be refused.
+//
+// Row LTX25-RES2S-LOOP, issue #921. Spec .agents/specs/ltx25-res2s-loop.md.
+enum class Ltx2StepperKind { kEuler, kEulerAncestral, kRes2s };
 
 // LTXPhaseRecipe (ltx2_recipes.py:29-50).
 struct Ltx2PhaseRecipe {
@@ -624,6 +677,19 @@ void Ltx2AssertResolution(int64_t height, int64_t width, int64_t divisor);
 //   ("one_stage",          "2.5")  Lightricks, via _PARAMS_SINCE_VERSION (:130-133)
 //   ("distilled_two_stage","2")    vLLM-Omni LTX2_DISTILLED_TWO_STAGE_RECIPE (:125-158)
 //   ("distilled_two_stage","2.5")  Lightricks distilled.py + constants.py:17-23
+//   ("res2s_two_stage",    "2.5")  Lightricks ti2vid_two_stages_hq.py:59-340 plus
+//                                  LTX_2_3_HQ_PARAMS (constants.py:95-115). Row
+//                                  LTX25-RES2S-LOOP, #921. The res_2s sampler on
+//                                  BOTH stages, 15 steps, STG off. 2.5 only, and
+//                                  not by analogy with the one_stage rows:
+//                                  `LTX_2_3_HQ_PARAMS` is a plain constant that
+//                                  overrides every generation-varying knob
+//                                  (constants.py:91-94 says so), so there is no
+//                                  `detect_params` lineage to spread it across
+//                                  versions. THE SAMPLER IS THE PRESET: this
+//                                  recipe on `kEuler` would render a finished,
+//                                  correctly sized, plausible clip at half the
+//                                  model evaluations 15 steps was tuned for
 //   ("dmd2",               "2")    vLLM-Omni LTX_POSITIVE_ONLY_RECIPE (:116-124)
 //   ("dmd2",               "2.3")  same
 //   ("dfr",                "2.5")  Lightricks dfr_pipeline.py:155-561 (row