diff --git a/.agents/issue-index.md b/.agents/issue-index.md index 69cf1b2c4..fbc56c04e 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -220,7 +220,14 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#442](https://github.com/mudler/vllm.cpp/issues/442) | `SPEC-DSPARK` | W8: capture the T=1+k speculative VERIFY shape (the measured 0.870x-0.981x gap) | perf | | [#513](https://github.com/mudler/vllm.cpp/issues/513) | `SPEC-DSPARK` | DFlash/DSpark draft loading hardcodes the VL weight prefix, so a text-only Qwen3.5 target cannot share its embedding | bug | | [#536](https://github.com/mudler/vllm.cpp/issues/536) | `ROAD-V1-C3` | C3's named tail records DSpark as unspiked three days after it landed, and files TLI under the wrong lane | bug | +| [#611](https://github.com/mudler/vllm.cpp/issues/611) | `ROAD-V1-LTX25` | `duration_head_path` is accepted by `kKnownLoadExtras` and read by NOTHING, so a caller who points at a duration head silently gets the recipe default. Row `LTX25-RETIRE-DEAD-ARMS` (spec [`ltx25-retire-dead-arms.md`](specs/ltx25-retire-dead-arms.md)) refuses it by name and records the full accepted/read inventory of all ten extras. Stays open: constructing a duration head and the AUTO-duration path are still owed | bug | +| [#691](https://github.com/mudler/vllm.cpp/issues/691) | `ROAD-V1-LTX25` | The LTX-2.5 out-of-scope ledger's reachable/marker split gates the MESSAGE TEXT, not reachability: give `kLoraFusion` a product call site and the ledger test still passes while the shipped refusal tells the caller `DECLARED, NOT REQUESTABLE` right after they requested it. Copy the anti-tautological shape of the extras inventory gate | bug | +| [#692](https://github.com/mudler/vllm.cpp/issues/692) | `ROAD-V1-LTX25` | Campaign spec `ltx-2-5.md:291` still records the FABRICATED `multishot` arm as owed after row `LTX25-RETIRE-DEAD-ARMS` retired it; same line needs `CFG parallelism` renamed to multi-GPU. Operator-owned file, so filed rather than edited by that row | bug | | [#886](https://github.com/mudler/vllm.cpp/issues/886) | `POLICY-NOTHING-LANDS-DEAD` | The seam checkers police where a capability routes and nothing polices whether anything reaches it, so a change can land wired downward and driven only by its own test, as tensor parallelism did (guide [`reachability.md`](reachability.md)) | feature | +| [#889](https://github.com/mudler/vllm.cpp/issues/889) | `LTX25-RETIRE-DEAD-ARMS` | `kBetaScheduler` shipped as a REACHABLE refusal in the header, `docs/FEATURES.md` and `docs/USAGE.md` with zero product callers: its site sits inside `Ltx2Schedule`, which nothing calls, and the engine calls `Ltx2SigmaSchedule` directly. Upstream constructs `BetaScheduler` nowhere either — all seven ltx-pipelines entry points hard-code `LTX2Scheduler()` — so it is reclassified as a marker rather than wired | bug | +| [#890](https://github.com/mudler/vllm.cpp/issues/890) | `LTX25-RETIRE-DEAD-ARMS` | `ltx2_video.cpp`'s temporal-upsampler guard tested `temporal_upsample` alone, which every BOTH-flags config also satisfies, so a genuine spatiotemporal checkpoint was told it is the temporal x2 arm and the ledger refusal naming the spatiotemporal one was unreachable from a request. Narrowed to `temporal_upsample && !spatial_upsample`, with a both-flags fixture driven through `LoadVideoEngine` | bug | +| [#892](https://github.com/mudler/vllm.cpp/issues/892) | `LTX25-RETIRE-DEAD-ARMS` | The `kMultiGpuParallelism` marker shipped "zero `cfg` hits in either multigpu tree" in the header and "three forms" in the user-visible refusal. At LTX-2 `fd4ded7f` it is 5 hits against 33 files as the control, and there is a fourth form, `BatchParallelGemmaBuilder` (`multigpu/bp_gemma_builder.py:42`). The spec's grep was path-filtered past `docs/multigpu/`, which is #604 again | bug | +| [#893](https://github.com/mudler/vllm.cpp/issues/893) | `LTX25-RETIRE-DEAD-ARMS` | AGENTS.md `## Nothing lands dead` disclosure for the #889 repair: `Ltx2Schedule` is an unreached dispatcher whose only driver is its own unit test. No wiring wave is coming, because upstream selects no scheduler, so the open decision is retire-or-keep-as-recorded-mirror. Listed under `## Owed` in [`ltx25-retire-dead-arms.md`](specs/ltx25-retire-dead-arms.md) | bug | | [#896](https://github.com/mudler/vllm.cpp/issues/896) | — | No gate refuses a new unchecked `static_cast<...LoadedModel&>` in a registry entry point. The #847 sweep decided the checker is warranted — the class is exactly grep-able, unlike the unaligned-read class in #627, and it regrows the moment a new model port copies its neighbour — and deliberately did not bundle a repository-wide refusal gate into a 30-file mechanical sweep. Listed under `## Owed` in [`registry-downcast-sweep.md`](specs/registry-downcast-sweep.md) §6 | feature | | [#897](https://github.com/mudler/vllm.cpp/issues/897) | — | `ModelAs`'s `const LoadedModel&` overload has no caller anywhere in the tree: every registered `prepare`/`forward` takes a NON-const `LoadedModel&`, so all 35 call sites select the non-const overload and its `Model&` result merely binds to the `const auto&`. Proven by deleting the overload and compiling all 30 swept TUs `-fsyntax-only`. It is dead in the way that reads as covered — #847 cited its existence as the reason the 14 `const` sites needed no thought. Listed under `## Owed` in [`registry-downcast-sweep.md`](specs/registry-downcast-sweep.md) §3.3 | bug | | [#904](https://github.com/mudler/vllm.cpp/issues/904) | `ROAD-V1-LTX25` | `main` red on `sanitize-cpu (address,undefined)` for a THIRD reason after the two [#730](https://github.com/mudler/vllm.cpp/issues/730) enumerates, both of which are now closed: `~Ltx2DitDeviceWeights` frees the staged DiT buffers on the main thread (`ltx2_device.cpp:1088`) while a `vt::cpu` threadpool worker is still reading one inside `AddKernel` (`cpu_layernorm.cpp:33`), so the staged weights' lifetime is not joined to the in-flight parallel op that reads them. Deterministic, 5 runs / 5 aborts on unmodified `e8048ef63`; the rest of the suite is 477/478 and `test_ltx2_video` now PASSES | bug | diff --git a/.agents/specs/ltx25-retire-dead-arms.md b/.agents/specs/ltx25-retire-dead-arms.md new file mode 100644 index 000000000..7a70897ce --- /dev/null +++ b/.agents/specs/ltx25-retire-dead-arms.md @@ -0,0 +1,919 @@ +# LTX-2.5 — retire the arms that do not exist, and refuse the extra that is ignored + +Row: `LTX25-RETIRE-DEAD-ARMS`. Campaign: [`ltx-2-5.md`](ltx-2-5.md) (operator-owned; +not edited by this row). Issues: +[#644](https://github.com/mudler/vllm.cpp/issues/644) items D–I and N, +[#611](https://github.com/mudler/vllm.cpp/issues/611). +Pattern this row is an instance of: [#604](https://github.com/mudler/vllm.cpp/issues/604). + +Upstream pins: + +| Reference | Revision | Local checkout verified at | +|---|---|---| +| Lightricks/LTX-2 | `fd4ded7f2d88d3da713abcdd4ad41ecc4a9314ca` | `git rev-parse HEAD`, 2026-08-13; re-verified 2026-08-14 | +| huggingface/diffusers | `3a2f35d4efa4c059c8bfb3bc0d6c906264895c81` | `git rev-parse HEAD`, 2026-08-13; re-verified 2026-08-14 | + +Anchors here were derived from those two checkouts by this row. **Four of them +were wrong and were found by review, not by the row** — see §7, which records +what was not re-derived rather than claiming everything was. Where an ABSENCE is +asserted, §1.1 carries the command and the control hits that prove the command +ran; treat any absence in this file without that pairing as unverified. + +## 0. What is wrong today + +`Ltx2UnportedPipelineFeature` (`include/vllm/model_executor/models/ltx2_pipeline.h:580-587`) +is a ledger of seven arms this port refuses by name. Read as a whole it claims +more than is true, in four separate ways: + +1. **One arm is fabricated.** `kMultishot` refuses "multishot generation" and its + enum comment cites *"ltx-pipelines multishot entry points"*. No such entry + point exists. Neither does the symbol, the string, or the concept. +2. **One arm's anchor names something upstream does not do.** `kCfgParallelism` + is anchored to `ltx-pipelines/multigpu`, which contains no CFG batching at + all — the four real parallelisms are orthogonal to guidance, and the recipe + this port runs uses no guidance in the first place. +3. **One arm is stale.** `kVideoEngineWiring` says the end-to-end wiring "is + phase L7, not L5". L7 landed in `cefacd2d0`. +4. **Five of the seven have no product call site.** Recording them as "refused by + name" overstates what exists: nothing a caller can send reaches them. They are + declared-out-of-scope markers, and the code did not say so. + +Separately, and user-visible: `duration_head_path` is accepted by +`kKnownLoadExtras` (`src/vllm/multimodal/ltx2_video.cpp:257`) and **read by +nothing**. A caller who points at a duration head gets the recipe default and is +told nothing. + +## 1. What the references actually contain + +### 1.1 `multishot` — FABRICATED + +Searched as a **subject**, not as our own phrasing (the self-confirming-grep trap, +#604) — and **every absence below was re-derived on 2026-08-14 with a POSITIVE +CONTROL in the same command**, after two of the claims in this section shipped +false and a third was offered as exhaustive when it was not. A +grep that returns nothing proves nothing on its own — the same command has to +return the hits you know are there. The two commands, verbatim: + +```sh +# LTX-2 @ fd4ded7f — `multishot` ABSENCE, controls `scene cuts` / +# `single continuous take` / `DurationPredictor` in the SAME command. +git grep -n -i -E 'multi[_ -]?shot|scene cuts|single continuous take|DurationPredictor' +# -> 0 lines matching multi[_ -]?shot, 24 control lines. +# RE-MEASURED 2026-08-15: this said 22. The verbatim command returns 24 (0 either +# way), and the count is the positive control, so a wrong control is a defect in +# the evidence even when the conclusion holds. Split: `git grep -c` on the +# control alternation alone is also 24, so all 24 are controls. +# LTX-2 @ fd4ded7f — every `shot` FILE, control `DurationPredictor`. +git grep -l -E '\bshots?\b' # -> 7 files (see below) +# LTX-2 @ fd4ded7f — `storyboard` ABSENCE, control `shot list` same command. +git grep -n -i -E 'storyboard|shot list' # -> 1 line, README.md:136, the control. +# LTX-2 @ fd4ded7f — what the `multi` prefix actually attaches to. +git grep -h -o -i -E 'multi[_ -]?(gpu|scale|stage|shot|clip|prompt|segment)' \ + | sort | uniq -c | sort -rn +# -> 94 multigpu, 44 multi-GPU, 25 Multi-GPU, 4 MULTI_GPU, 2 multi-prompt. +# diffusers @ 3a2f35d4 — same absence in the ltx2 pipelines package. +git grep -n -i -E 'multi[_ -]?shot|scene cuts|hard cuts|single continuous take' \ + -- src/diffusers/pipelines/ltx2/ +# -> 0 lines matching multi[_ -]?shot, 4 control lines. +``` + +| Query | LTX-2 `fd4ded7f` | diffusers `3a2f35d4` | +|---|---|---| +| `multi[_ -]?shot` (case-insensitive) | 0 hits, alongside 24 control hits | 0 hits, alongside 4 control hits | +| `\bshots?\b`, whole repository | **7 files**, none a generation mode | — | +| `multi[_ -]?(gpu\|scale\|stage\|shot\|clip\|prompt\|segment)` | `multi[-_ ]?gpu` **167**, `multi-prompt` **2** — and nothing else | — | +| `scene`, whole repository | **three senses, none a generation mode** — see §1.1a | only the prompt-guidance sense inside `pipelines/ltx2/` (`scene cuts` ×3, `scene opens` ×4) | +| `storyboard` | 0 hits, alongside its control | 0 hits | + +An earlier revision of this table said the `multi` prefix attaches to +`multi-GPU`/`multi_gpu` **only**. It does not: `multi-prompt` appears twice +(`ltx-pipelines/utils/args.py:618`, "Helps multi-prompt enhance"; and +`utils/denoisers.py:174`, "multi-prompt benchmark panels"). Both are *batching +several captions through one run* — the enhancer's KV cache and a benchmark +panel — not composing several takes into one output, so the conclusion is +unaffected and the "only" was still wrong. Recorded rather than quietly +corrected, for the same reason as everything else in this section. + +The **seven** `shot` files, with what the word means in each. An earlier revision +of this list said five and presented that as exhaustive (review finding M1). The +cause is recorded because it is reusable: that query was +`\bshots?\b` **restricted to `*.py *.md *.json *.yaml`**, and upstream's prompt +files are `.txt`. The filter, not the pattern, hid the answer — and it hid +exactly the two files that matter most here, because they are the ones that ship +at inference. Re-run without a path filter it is seven: + +- `ltx-core/duration_head/duration_head.py:1,5` — "predicts **shot** duration", + i.e. the natural length of ONE camera take. +- `ltx-core/duration_head/__init__.py:1` — same. +- `ltx-core/text_encoders/gemma/encoders/prompts/gemma4_i2v_system_prompt.txt:13,14,17` + — camera **shot type** in the prompt-enhancer's instructions. +- `ltx-core/text_encoders/gemma/encoders/prompts/gemma4_t2v_system_prompt.txt:11,12,15` + — same. +- `ltx-pipelines/utils/blocks.py:804` — `DurationPredictor`, "Predicts **shot** + duration (in frames)". +- `ltx-trainer/src/ltx_trainer/captioning.py:41,42,54,57` — captioning prose, for + describing an EXISTING video as training data. +- `LTX-2/README.md:59,136` — an example prompt ("a medium close-up **shot**") and + the prompting guide ("think like a cinematographer describing a **shot** list"). + +Upstream's "shot" is a single continuous take. There is no multi-shot generation +mode, and nothing that composes several takes into one output. + +The nearest thing in either tree runs the other direction: +`ltx-trainer/captioning.py:54` tells the CAPTIONER "the video contains multiple +shots, describe each one in turn" — an instruction for describing an input video +during dataset preparation, not for generating one. Named here so the next porter +who greps `shots` does not have to re-derive that it is the trainer. + +**The `shot` claim, corrected on the diffusers side too.** This section used to +call `utils.py:217` — "Shot type (exactly one: extreme wide shot / wide shot / +medium shot / medium close-up / close-up / extreme close-up)" — the "stronger +check" because it came from a second, independent reference. It is not +independent: that sentence is **verbatim** LTX-2's own +`gemma4_t2v_system_prompt.txt:12`, and `utils.py:251` is verbatim +`gemma4_i2v_system_prompt.txt:14`. diffusers vendored Lightricks' gemma4 system +prompts wholesale. It corroborates that upstream's `shot` is a camera shot type, +and it does so at a second pin, but it is one source quoted twice, not two +sources agreeing. diffusers' own contribution is the duration head's docstring, +`duration_head.py:83`, "Predicts the natural duration of the **shot** implied by +a caption", and its module list: `pipeline_ltx2`, `_condition`, +`_diffusion_decode`, `_hdr_lora`, `_ic_lora`, `_image2video`, `_latent_upsample`, +plus the components. None is multi-shot. + +`ltx-pipelines`' actual entry points, from `docs/pipelines.md` ("Full reference +for all 11 pipelines") and the module list: +`ti2vid_one_stage`, `ti2vid_two_stages`, `ti2vid_two_stages_hq`, +`ti2vid_two_stages_mgpu`, `ti2vid_two_stages_hq_mgpu`, `distilled`, +`distilled_mgpu`, `ic_lora`, `hdr_ic_lora`, `a2vid_two_stage`, `t2a_one_stage`, +`dubit`, `retake`, `dfr_pipeline`, `keyframe_interpolation`. None is multi-shot. + +### 1.1a `scene` — three senses, and the one that was missed twice + +**This is the third #604 instance this row shipped, and the one that took three +review rounds to find** (review finding B1). The header, this table and the PR +body all said "the only `scene` hit is PySceneDetect in the TRAINER". That is +false at `fd4ded7f`, and it was false in a SHIPPED header — the failure scenario +being a porter who greps `scene`, finds "scene cuts" prohibited in an inference +prompt, and concludes we missed a multi-shot path our own header denied. + +`scene` has three senses upstream, established by reading **every** hit of +`git grep -n -i scene` — no path filter, since a path filter is what hid the +`shot` answer above — rather than by grepping our own vocabulary. The anchors +below are representative of each sense, not a transcript of every line; the +command above is the transcript: + +| Sense | Where | Is it a generation mode? | +|---|---|---| +| `scene-linear` HDR colour (scene-referred light) | `ltx-core/color/hlg.py:1,5,44,74,93,256`, `ltx-core/hdr.py:29,132,146,162,171`, `ltx-pipelines/docs/hdr.md:15,16,34`, `utils/media_io/{color_config,encode,exr}.py`, `utils/args.py:813,861`, `ic_lora.py:422`, `retake.py:178`, `hdr_ic_lora.py:18`, `CHANGELOG.md:24` | no — colour science | +| PySceneDetect splitting | `ltx-trainer/scripts/split_scenes.py`, `ltx-trainer/docs/*`, `ltx-trainer/pyproject.toml:28` (`scenedetect>=0.6.5.2`), `ltx-trainer/AGENTS.md:85`, `.claude/skills/train-model/**` | no — a **training-data** preprocessor, and the only CODE sense | +| narrative prose, i.e. prompt-writing guidance | **`ltx-core`, which ships at INFERENCE**: `text_encoders/gemma/encoders/prompts/gemma3_i2v_system_prompt.txt:5,6,18,20`, `gemma3_t2v_system_prompt.txt:1,5,24,25`, `gemma4_i2v_system_prompt.txt:7,15,23,27`, `gemma4_t2v_system_prompt.txt:5,13,21,25`; plus `README.md:136` and `ltx-trainer/captioning.py:39` | no — instructions to the prompt enhancer | + +The third sense is the one the old sentence denied, and `README.md:136` is +**the same line** the header cites two clauses earlier for "a cinematographer +describing a shot list" — it reads "chronological descriptions of actions and +scenes". + +**The disposition does not move; it is STRENGTHENED.** Read as a subject rather +than as a keyword, the shipped enhancer prompts instruct the model *against* +multi-shot output: + +- `gemma3_i2v_system_prompt.txt:6` — "Inaccurate descriptions may cause scene cuts." +- `gemma3_i2v_system_prompt.txt:18`, `gemma3_t2v_system_prompt.txt:24` — "No + timestamps or cuts: DO NOT use timestamps or describe scene cuts unless + explicitly requested." +- `gemma4_i2v_system_prompt.txt:3` — "**Single continuous take — no hard cuts.**" + +diffusers carries the same four lines (`utils.py:121,161,182,240`), because it +vendored the same prompts. An upstream whose own prompt enhancer is told to +produce one continuous take is affirmative evidence that no multi-shot generation +mode exists — which is what `kMultishot` claimed to refuse. + +**Disposition: RETIRE.** This is a defect in our record, not a gap in our port. +There is nothing to owe, so recording it as owed is the error. The enumerator is +removed and the retirement recorded in the header, in this spec, and in the +commit message — which is where an exception's reason lives (AGENTS.md, "there is +no waiver registry"). + +### 1.2 `int8-convrot` — real absence, deliberately out of scope + +The inference quantization kinds LTX-2 defines, exhaustively +(`ltx-pipelines/utils/quantization_factory.py:22-26`, a `str`-valued enum with an +`assert_never` at `:50`, the file's last line — re-derived 2026-08-14, review +finding L1. The block quoted below starts at the `class` statement on `:22` and +its four members are `:23-26`, which is the range the header comment and the +shipped refusal message cite; `:23-27` was wrong at both ends and this spec was +the last place still carrying it): + +```python +class QuantizationKind(str, Enum): + FP8_CAST = "fp8-cast" + FP8_SCALED_MM = "fp8-scaled-mm" + NVFP4_CAST = "nvfp4-cast" + NVFP4_PREQUANT = "nvfp4-prequant" +``` + +No int8 arm. `convrot` / `conv_rot` / `quarot` / `spinquant` are 0 hits across the +repository, re-run at these pins. + +**Correction, 2026-08-13 (review finding F2).** An earlier revision of this +section put `hadamard` in that 0-hit list and concluded "no rotation of any kind". +That was itself the #604 pattern this row exists to close — an absence asserted +from a grep of our own vocabulary — and it is false. LTX-2 vendors Tri Dao's +fast-Hadamard-transform family in `ltx-kernels` +(`csrc/ops/include/fast_hadamard_transform.h`, `_common.h`, `_special.h`), and it +is explicitly **quantization-coupled**: `DequantHadamardParamsBase:54`, +`QuantHadamardParamsBase:71`, and `QuantMax = 127.0` +(`fast_hadamard_transform_common.h:14`). A fast Hadamard transform *is* the "rot" +of rotation-based quantization, so the word we should have searched for was there +all along. + +What the conclusion rests on instead, re-derived here: + +- **Nothing calls it.** `hadamard_mult_thread` / `hadamard_mult_warp` are defined + at `fast_hadamard_transform_common.h:102,123` and `_special.h:12,31,58,93` and + invoked at **no site in `csrc/`**. The `fast_hadamard_transform_cuda` + instantiations are commented out (`rms_norm_rope_cuda.cu:225-228`), and + `DequantHadamardParamsBase` / `QuantHadamardParamsBase` / `QuantMax` are + declared and referenced nowhere. +- **The live kernel that carries the Hadamard NAME does not rotate.** The only + params struct in use is `NormRopeHadamardParamsBase` + (`rms_norm_rope.cpp:12,19,95-98`, `rms_norm_rope_cuda.cu:188`), and + `norm_rope_cvt_kernel` (`rms_norm_rope_cuda.cu:51-170`) is RMS-norm → RoPE → + store, with no Hadamard step. The name is vestigial, inherited from the fork's + ancestry. +- **And where it is instantiated it is fp8 or bf16, never int8**: + `` and `` + (`rms_norm_rope_cuda.cu:219-223`), matching `rms_norm_split_rope_cuda.cu:196,200`. + +Corrected statement, same conclusion: **upstream vendors a +quantization-coupled Hadamard family, wires none of it, and never exposes a +rotation as a `QuantizationKind`.** So `int8-convrot` is still unreachable +upstream — but by dead code, not by absence. + +`int8` upstream is **unreachable, not absent**. Everything *wired* is trainer-only: + +- `ltx-trainer/src/ltx_trainer/gemma_8bit.py:33-36` — bitsandbytes `LLM.int8()` + for the Gemma backbone during LoRA training. +- `ltx-trainer/src/ltx_trainer/quantization.py:11-15` — optimum-quanto precisions + (`int8-quanto`, `int4-quanto`, …) for the trainer. + +**But one int8 kernel lives in an INFERENCE package** (review finding F1, and the +same false-absence pattern shipped inside the user-visible refusal message): +`ltx-kernels/src/ltx_kernels/blockwise/triton_ops.py` defines `_kernel` at +`:25-37` and `run_quantize_kernel` at `:40-50` — a per-row int8 quantize with fp32 +scales, output dtype `torch.int8` at `:43`, not a `uint8` buffer — aliased +`rowwise_int_quantize_triton` at `:436`. It is **dead**: that alias is its only +reference anywhere in the repository, and `blockwise/functional.py:12-18` +re-exports five other names and not this one. Context for why it is there at all: +the package is a fork of Lightricks' int8 kernel library retargeted to +fp8/fp6/nvfp4, and its custom-op namespace is still literally `q8_kernels_ops` +(`functional.py:25`). + +Every other `int8` match in the repository is `uint8`: pixel buffers, packed NVFP4 +nibbles, block-streaming staging. + +**Disposition: KEEP, re-anchored.** Unchanged by the two corrections above — +nothing wired upstream reaches int8, and no rotation is exposed as a quantization +kind. What changes is the sentence: the refusal now says int8 is UNREACHABLE +(trainer-only for anything wired, plus one dead kernel in `ltx-kernels`) instead +of claiming it appears "only in the trainer". Its *kind* also changes: it is a +declared-out-of-scope marker, not a reachable refusal. + +### 1.3 CFG parallelism — the name describes something upstream does not do + +**THE ORIGINAL VERSION OF THIS SECTION SAID `cfg` IS 0 HITS IN BOTH MULTI-GPU +TREES, AND IT WAS WRONG (#892, review finding F3).** The claim was scoped to +`ltx-pipelines/src/ltx_pipelines/multigpu/` and `ltx-core/src/ltx_core/multigpu/`, +which are the two SOURCE trees — and that path filter excludes +`ltx-pipelines/docs/multigpu/`, where the answer is written out in prose. Fourth +instance of #604 in this row, and the one that proves the row's own lesson twice +over: a path filter is an absence claim too, and this section is where that +sentence was written. + +Re-derived without the filter, with the file list as the positive control in the +same pass: + +```sh +# LTX-2 @ fd4ded7f — every `multigpu` path, no source/docs split. +git ls-files -- '*multigpu*' # -> 33 files, the CONTROL +git grep -n -i cfg -- '*multigpu*' # -> 5 lines, NOT 0 +``` + +Three of the five are the incidental `model_cfg` local in a documentation code +block (`docs/multigpu/sequence-parallel.md:81,84,85`). **Two are substantive**, +and they are the ones that matter: + +> `docs/multigpu/gemma.md:103-104` — "Batch-parallel is beneficial only when there +> is **more than one prompt to encode** — the typical **CFG** case, positive + +> negative (B=2 on 2 ranks = one prompt per rank, both forwards concurrent). The +> **distilled** pipeline runs **without CFG**: its `__call__` accepts a single +> `prompt` and no `negative_prompt`, so there is only one prompt to encode and no +> work to partition." + +That paragraph is better evidence than the absence ever was. It says in upstream's +own words that the recipe this port runs has no CFG pair, which is the reason CFG +batching is inapplicable here — a reason, rather than a claim that a string does +not appear. + +**And there are FOUR forms, not three.** The fourth sits in the very directory the +refusal cites: + +| Form | Anchor | What it splits | +|---|---|---| +| Sequence parallel | `multigpu/sp_builder.py:25` (`SequenceParallelBuilder`), `ltx-core multigpu/transformer/sequence_parallel.py`, all-to-all attention | the token axis of one denoise step | +| Tiled data parallel | `multigpu/tdp_builder.py:25` (`TiledDataParallelBuilder`) | spatial tiles, **upscale stage only** | +| Distributed VAE decode | `ltx-core multigpu/vae/distributed_decoder.py:204-256` (`DistributedVideoDecoder.decode_video`) | latent tiles across ranks, driver blends | +| **Batch-parallel Gemma encoding** | `multigpu/bp_gemma_builder.py:42` (`BatchParallelGemmaBuilder`), wrapping `ltx-core multigpu/gemma/batch_parallel_wrapper.py` | a **prompt list** across ranks, one replica per rank | + +The four `BuilderProtocol`/`DelegatingBuilder` implementors under +`ltx-pipelines/src/ltx_pipelines/multigpu/` are exactly these, verified by +`git grep -n 'BuilderProtocol' -- packages/ltx-pipelines/src/ltx_pipelines/multigpu/`. +The fourth is the closest thing upstream has to CFG batching, which is precisely +why omitting it made the "none of them is CFG batching" sentence weaker rather +than safer: the honest statement is that the form which WOULD carry a CFG pair is +the one upstream tells you not to use for the distilled pipeline. + +Upstream states the purpose in its own words +(`ltx-pipelines/docs/multigpu/README.md:7`, inside the ⚠️ block at `:5-16`): + +> **Multi-GPU (MGPU) is a latency tool, not a memory tool.** + +and adds that the mutable transformer is a **full replica on every GPU**, so it +cannot make a checkpoint fit. + +And CFG is not in the path this port runs at all. The distilled recipe denoises +with `SimpleDenoiser` at both stages (`ltx-pipelines/distilled.py:266`, `:295`), +documented as "**single transformer call, no guidance**" +(`utils/denoisers.py:3`); the guider it degenerates to is +`MultiModalGuiderParams(cfg_scale=1.0, stg_scale=0.0, modality_scale=1.0)` +(`utils/denoisers.py:25-26`), "only runs the conditioned pass and returns cond +unchanged". A `cfg_scale` of 1.0 is one pass, so there is no second pass to place +on a second GPU. + +Stronger still, and true even for the recipes that *do* guide: upstream's guided +path is not distributed either. `utils/denoisers.py:7-8` records that +`GuidedDenoiser` and `FactoryGuidedDenoiser` share `_guided_denoise`, "which +batches all guidance passes into a single transformer call". So upstream's CFG is +a batch dimension, never a second device — which is why no amount of `multigpu` +code was ever going to be CFG parallelism, and why the old enumerator name could +not have been right for any recipe. + +**Disposition: RENAME + re-anchor.** `kCfgParallelism` → `kMultiGpuParallelism`, +anchored to the four real forms, with the reason it is out of scope stated as +what it is: a single-node multi-GPU **latency** feature, on a port whose target is +one GB10. The disposition never moved across the F3 repair; only the evidence +under it was wrong, and the corrected evidence supports it more strongly. + +Gated by `test_ltx2_pipeline`'s "the multi-GPU marker note states the CFG evidence +correctly" (the header) and by the multi-GPU block in "ltx2 every out-of-scope +feature is refused BY NAME" (the shipped message), on the same shape as the int8 +marker: the retired sentence must be absent and the replacement evidence present. +Neither can verify the upstream fact — there is no upstream checkout in this tree +— so the commands above, with their control, remain the derivation. + +### 1.4 `kLoraFusion` — real upstream, correctly refused + +Verified present: `ltx-core/loader/primitives.py:160` +(`class LoraPathStrengthAndSDOps(NamedTuple)`), exported at +`loader/__init__.py:14,46`, consumed by `loader/single_gpu_model_builder.py:21` +and `block_streaming/builder.py:34,90`, fused by `loader/fuse_loras.py`. The +anchor stands; only its *kind* is corrected (marker, not reachable refusal). + +### 1.5 `kVideoEngineWiring` — LANDED + +`cefacd2d0` ("feat(ltx-2.5): LTX-2.5 joint video+audio DiT, and a video seam that +is no longer MiniMax-only (#435) (#641)", 2026-08-13) shipped exactly this: the +composition through `vllm::multimodal::VideoEngine`, reachable through the C ABI +as video family `ltx-2.5`. `include/vllm/multimodal/ltx2_video.h:12` already +speaks of the refusal in the past tense. + +**Disposition: RETIRE.** A refusal whose subject shipped is a false statement, not +a record of debt. + +### 1.6 The five with no product call site + +`grep` over `src/`, `include/`, `examples/`, `tests/` for every enumerator: + +| Enumerator | Product call site | +|---|---| +| `kTemporalUpsampler` | `src/vllm/model_executor/models/ltx2_upsampler.cpp:395` | +| `kBetaScheduler` | `src/vllm/model_executor/models/ltx2_pipeline.cpp:199` | +| `kLoraFusion` | **none** | +| `kMultishot` | **none** | +| `kInt8ConvRot` | **none** | +| `kCfgParallelism` | **none** | +| `kVideoEngineWiring` | **none** | + +**Re-derived at integration, 2026-08-15.** The table above was measured at this +row's base. `2e9d95e74` then landed the temporal-only x2 upsampler on the same +issue (#644) and renamed the enumerator `kTemporalUpsampler` → +`kSpatiotemporalUpsampler`, because only the BOTH-flags arm is still unported. +Re-derived against `origin/main` `9a8615672`, the reachable call site is +`src/vllm/model_executor/models/ltx2_upsampler.cpp:465` and the enumerator is the +spatiotemporal one; `kBetaScheduler` is unmoved at `ltx2_pipeline.cpp:199`. The +ported temporal arm is reachable from no shipped pipeline, which is why +`docs/FEATURES.md` still records it as gated and UNDRIVEN. + +**THE SPLIT WAS ONE REACHABLE AND FOUR MARKERS, NOT TWO AND THREE (#889, review +finding F1).** The table above answers "does this enumerator appear in `src/`?", +and that is the wrong question. A refusal is reachable only if something CALLS +the function that holds it, and `kBetaScheduler`'s call site +`ltx2_pipeline.cpp:199` sits inside `Ltx2Schedule`, which has **zero** call sites +in `src/`, `include/` or `examples/`. The engine bypasses the dispatcher and calls +`Ltx2SigmaSchedule` directly, in `ltx2_video.cpp`'s phase driver; no ABI field, load extra or +CLI flag carries a scheduler kind. The only thing that ever trips the arm is +`test_ltx2_pipeline`'s "ltx2 the Beta scheduler is refused by name, never substituted", constructing the enumerator by +hand — the **test-only driver** shape that AGENTS.md `## Nothing lands dead` and +[`.agents/reachability.md`](../reachability.md) name. + +The corrected table, with the question stated the way it has to be asked: + +| Enumerator | Site of the refusal | Function holding it | Product callers of that function | Kind | +|---|---|---|---|---| +| `kSpatiotemporalUpsampler` | `ltx2_upsampler.cpp:465` | `Ltx2LatentUpsample` ← `Ltx2UpsampleVideoLatent` | `ltx2_video.cpp`, the refine phase | **reachable** | +| `kBetaScheduler` | `ltx2_pipeline.cpp:199` | `Ltx2Schedule` | **none** | marker | +| `kLoraFusion` | — | — | — | marker | +| `kInt8ConvRot` | — | — | — | marker | +| `kMultiGpuParallelism` | — | — | — | marker | + +**UPSTREAM SETTLES IT, WHICH IS WHY THE REPAIR IS A RECLASSIFICATION AND NOT A +WIRING.** The alternative repair — route the engine through `Ltx2Schedule` so the +claim becomes true — would have to invent a scheduler-selection surface that +upstream does not have. At Lightricks/LTX-2 `fd4ded7f`, `BetaScheduler` is defined +at `ltx-core components/schedulers.py:91` and **constructed nowhere in the +repository**; all seven `ltx-pipelines` entry points hard-code `LTX2Scheduler()` +(`ti2vid_one_stage.py:81`, `ti2vid_two_stages.py:87`, `ti2vid_two_stages_hq.py:90`, +`a2vid_two_stage.py:78`, `t2a_one_stage.py:67`, `keyframe_interpolation.py:82`, +`retake.py:96`), verified by +`git grep -n 'LTX2Scheduler\|LinearQuadraticScheduler\|BetaScheduler'` with the +definition file excluded — a command whose own output is its control, since it +returns the seven `LTX2Scheduler()` constructions it must. vLLM-Omni `a4ea67a21` +has **zero** hits for the name, against 81 files containing `ltx` as the control. +Mirroring upstream therefore means this port has no scheduler-kind field either, +and `kBetaScheduler` records an unported upstream component rather than a refusal +a request can trip. + +`Ltx2Schedule` itself is then an unreached dispatcher, disclosed under `## Owed` +and tracked by #893 as required by `## Nothing lands dead`. No wiring wave is +coming for it, because there is nothing upstream to wire it to; the open decision +is retire-or-keep. + +**The classification is now derived rather than written.** +`test_ltx2_pipeline`'s "the reachable/marker split matches the source" walks +`src/`, `include/` and `examples/`, counts callers of each arm's entry function, +and requires the message to agree with what it finds — in both directions, so a +caller appearing later fails the marker wording just as its absence failed the +reachable wording. Two positive controls ride in the same walk, because this is an +absence claim about our own tree and this row has shipped four of those already. +It is the anti-tautological shape #691 asks for, applied to the beta arm; #691 +stays open for the other three markers, which have no call site at all to derive. + +The five are enumerated only by the `reachable`/`markers` vectors in +`test_ltx2_pipeline`'s "ltx2 every out-of-scope feature is refused BY NAME". The +line numbers that used to be written here are deliberately gone: they went stale +twice inside this pull request, and a case name is stable where a line range is +not. There is no request field, load extra, CLI flag or scheduler kind that asks +for a LoRA, an int8-convrot checkpoint, a second GPU or a Beta schedule, so no +caller can trip any of the four. + +That is not a defect on its own — a declared boundary is worth having. The defect +is calling it a refusal. The header and the messages now distinguish: + +- **reachable refusal** — a product path constructs the condition and throws, and + something a user arrives through calls the function that holds it; +- **declared-out-of-scope marker** — a record of what upstream has and this port + does not, reached only by the ledger test. + +## 2. `duration_head_path` (#611) + +### 2.1 The full `kKnownLoadExtras` audit + +Every key the family accepts, and whether any code reads it. Reader anchors are in +`src/vllm/multimodal/ltx2_video.cpp` unless noted, **as of `3c6706cd0`** — this +table is a dated record, not a live one; see the note below it. + +| Key | Constant | Reader | Status | +|---|---|---|---| +| `audio_prompt_embeds_path` | `kLtx2AudioPromptEmbedsExtra` | `:969`, `:982` | READ | +| `pipeline_kind` | `kLtx2PipelineKindExtra` | `:805` | READ | +| `model_version` | `kLtx2ModelVersionExtra` | `:789` | READ | +| `allow_unported_modules` | `kLtx2AllowUnportedExtra` | `:638` | READ | +| `max_phase` | `kLtx2MaxPhaseExtra` | `:807` | READ | +| `dit_config_path` | `kLtx2DitConfigPathExtra` | `:693` | READ | +| `prompt_embeds_valid_rows` | `kLtx2PromptValidRowsExtra` | `:1010` | READ | +| `encoder_config_path` | `kLtx2EncoderConfigPathExtra` | `:864` | READ | +| `upsampler_path` | (literal) | `:839` | READ | +| `duration_head_path` | `kLtx2DurationHeadPathExtra` | **none** | **was ACCEPTED AND IGNORED; now refused by name** | + +Nine of ten are wired. `duration_head_path` is the only defect, so the sweep this +row owes is complete and closes the "the sweep that found this one did not cover +them all" clause of #611. + +**These numbers rotted twice inside this one PR (review finding F3).** The first +set shipped off by 37 lines and named nine lines that read nothing — cited, worst +of all, *in the file they were wrong about*. Correcting them, a merge of +`origin/main` moved the real ones by another 25. A hand-written `file:line` is +stale by the next commit, so the anchors are no longer maintained by hand: the +`READER ANCHORS` comment in `ltx2_video.cpp` carries the one live copy, and +`test_ltx2_video`'s "the recorded reader anchors are the ones in the source" case +derives them from the source on every run and fails with the replacement list +printed. The table above stays because a spec is a dated record and the SHA is +named; the code comment is the live one. + +One documentation gap found by the same sweep and fixed here: the paragraph in +`docs/USAGE.md` beginning "Two families are registered." lists the LTX-2.5 extras +and omitted `encoder_config_path` entirely, though it is defined and read. + +That paragraph was cited as `docs/USAGE.md:1650-1654`, and it is not there — at +this row's head it starts at `:1949` and the extras list runs to `:1955`. A third +stale anchor in the same spec (review finding, nit). Named by its OPENING WORDS +here rather than re-pinned to a number, for the reason §2.1 already gives about +the reader anchors: this file moved twice inside one pull request, and a phrase +survives a merge that a line number does not. + +### 2.2 Why it is inert + +`ltx2_duration_head.h` / `ltx2_duration_head.cpp` port `DurationHead` and +`AttentionPooler` and gate them as bricks. Nothing in `ltx2_video.cpp` constructs +one. The AUTO-duration path (`resolve_num_frames`, `ltx-pipelines/utils/blocks.py`) +therefore cannot run, and `Generate` computes `frames` from +`duration_seconds * fps` directly (`ltx2_video.cpp:1152`). + +The stated reason for this moved twice already (#604's pattern, and the finding +that produced #611). The reason recorded here is the current one: **no head is +constructed**, so a supplied path is a file the engine never opens. + +### 2.3 Fix + +Refuse `duration_head_path` **by name** when it is supplied and non-empty, with a +message that says the head is unported, that the recipe default would otherwise be +substituted silently, and what to use instead (`num_frames`, or `duration`, which +is exact arithmetic against the recipe frame rate). + +Rejected alternative: dropping the key from `kKnownLoadExtras`. That produces +"unknown load extra", which is *wrong* — the key is defined by this family and its +meaning is understood; what is missing is the implementation. AGENTS.md requires +"a message naming the missing piece", and "unknown key" does not name it. + +Rejected alternative: reading it and constructing the head. That is the real fix +and it stays owed — it needs the connector-output plumbing the head consumes +(`duration_head.py:89-118` takes audio and/or video connector token states), which +is a different row. Refusing is the cheap correct answer until then. + +## 3. Scope + +**In.** + +1. Retire `kMultishot` and `kVideoEngineWiring`; record both retirements in the + header, this spec, and the commit message. +2. Rename `kCfgParallelism` → `kMultiGpuParallelism` and re-anchor its message to + the three real parallelisms plus the reason CFG is not in our path. +3. Re-anchor `kInt8ConvRot`'s comment and message to record the verified absence + at these pins. +4. Split the ledger into reachable refusals and declared-out-of-scope markers, in + the header comment, in the messages, and in the test. +5. Refuse `duration_head_path` by name (#611). +6. `docs/USAGE.md`: the extras paragraph, corrected on both counts. +7. Added by the review repair: correct the int8 sentence in the shipped refusal + (§1.2), correct this spec's own false `hadamard` absence (§1.2), derive the + reader anchors instead of writing them by hand (§2.1), and split the + `docs/FEATURES.md` row so the public surface stops calling the two REACHABLE + arms unrequestable (§1.6). No disposition changes. +8. Added by the THIRD review round: correct the false `scene` absence in the + shipped header, in §1.1's table and in the PR body (§1.1a); correct the `shot` + enumeration from five files to seven and record the path filter that caused it + (§1.1); correct `quantization_factory.py:23-27` to `:22-26` (§1.2); replace + §7's "every claim in §1 was re-derived" with what actually happened; and gate + the header's retirement note with a RED-first case so the sentence cannot come + back. No disposition changes — the `scene` evidence strengthens the + `kMultishot` retirement rather than weakening it. + +**Out.** + +- `.agents/specs/ltx-2-5.md` — operator-owned. This row does not edit it. Its §2 + "Out" list still names `multishot`; correcting that is the operator's edit, and + this spec is the record it would cite. Filed as + [#692](https://github.com/mudler/vllm.cpp/issues/692) by the review repair so + `main` stops carrying the claim unowned. +- Making the reachable/marker split gate REACHABILITY rather than message text. + The ledger test asserts what each message says; nothing asserts that a marker + still has no product call site, so giving `kLoraFusion` one would leave the test + green while the shipped refusal told a caller it is not requestable immediately + after they requested it. Filed as + [#691](https://github.com/mudler/vllm.cpp/issues/691), with the extras inventory + gate named as the anti-tautological shape to copy. +- Constructing a duration head, and the AUTO-duration path. Stays owed (#611 + remains open after this row, retitled by the fix rather than closed by it — + see §7). +- `kTemporalUpsampler` (row `LTX25-TEMPORAL-UPSAMPLER`), image conditioning (row + `LTX25-IMAGE-COND-FIX`), tiled decode (row `LTX25-TILED-DECODE`), AdaLN claims + (row `LTX25-ADALN-CLAIMS`). No file in this diff is theirs. + +## 4. Tests + +RED first for the behavioural change; the records changes are gated by the ledger +test's own assertions. + +1. **`ltx2 duration_head_path is REFUSED by name` (new, + `tests/vllm/multimodal/test_ltx2_video.cpp`).** Builds a valid load, adds + `extras["duration_head_path"]`, and requires a throw whose message names the + key, names the duration head as the missing piece, and names the alternative. + **RED before the fix**: the load succeeds, because nothing reads the key. That + is the defect stated as a test. +2. **`ltx2 an accepted load extra is READ by something` (new).** Asserts the + inventory of §2.1 does not silently grow: every key in `kKnownLoadExtras` + either round-trips through a reader or is refused by name. Implemented as the + two known-inert keys being refused and the rest being accepted, so adding a + tenth decorative key fails. +3. **`ltx2 every out-of-scope feature is refused BY NAME`** (existing, updated). + The list drops from 7 to 5 entries — a CHANGED COUNT, reported as such — and + splits into `reachable` and `markers`, with the marker messages required to say + they are not requestable. Adds a guard that no refusal message mentions + `multishot` again. + +Three more added by the review repair, each RED before its fix: + +4. **The int8 marker's own evidence** (an addition to case 3). The message shipped + the sentence "int8 appears upstream only in the trainer", which §1.2 shows is + false. The case now requires the message NOT to contain it and to name + `ltx-kernels`. **RED before**: both assertions failed against the shipped text. +5. **`ltx2 video: the recorded reader anchors are the ones in the source`** (new, + `tests/vllm/multimodal/test_ltx2_video.cpp`). Derives each served key's reader + line from `ltx2_video.cpp` itself and compares it with the `READER ANCHORS` + comment, and separately requires every `kLtx2DurationHeadPathExtra` reference + after the array to sit inside `CheckUnservedExtras`. It hard-codes no line + number of its own, so it cannot go stale; the obligation it creates falls on + whoever moves a reader, in the file they are already editing. **RED before**: + `Recorded: []. Actual: [638 693 789 805 807 839 864 969 1010]`. +6. **`ltx2 docs/FEATURES.md never calls a REACHABLE refusal unrequestable`** (new, + `tests/vllm/models/test_ltx2_pipeline.cpp`). The public surface had re-merged + the reachable/marker split into one "Declared, not requestable" row naming the + temporal upsampler and `BetaScheduler`. Anti-vacuous: it also requires exactly + one such LTX-2.5 row to exist, so renaming the row away fails it. **RED + before**: both reachable arms found on `FEATURES.md:328`. + +One more added by the third review round: + +7. **`ltx2 the kMultishot retirement note states the scene evidence correctly`** + (new, `tests/vllm/models/test_ltx2_pipeline.cpp`). Reads + `include/vllm/model_executor/models/ltx2_pipeline.h` through a new + `LTX2_PIPELINE_HEADER_PATH` compile definition, flattens the comment block so a + reflow cannot make it vacuous, and requires the retired false sentence to be + absent and the replacement evidence to be present. Anti-vacuous: it first + requires **exactly one** ``kMultishot` — FABRICATED` note in the file, so + deleting or renaming the paragraph fails rather than passes. + + The three positive assertions are SCOPED TO THE NOTE, not run over the whole + flattened header, and the description above used to say otherwise. A file-wide + `find` could not fail for `ltx-core`, which occurs six times in this header, so + that assertion survived deleting the clause it exists to hold — proven by + mutation, which left the case green at 8/8. The shipped case slices the + paragraph from its marker to the next enumerator heading and asserts + `scene cuts`, the CLAIM `ships at INFERENCE inside \`ltx-core\`` rather than + the bare package name, and `system_prompt` inside that slice, with a + `note.size() > 400` guard so a bad slice fails loudly instead of quietly + passing. **RED before**: 4 of its 8 assertions + failed on `b718f580f` — `flat.find("the only \`scene\` hit is PySceneDetect")` + returned 29407 where npos was required, and `scene cuts` / `system_prompt` + both returned npos. **Anti-vacuity proven by mutation**: renaming the note's + marker in a staged copy gives `REQUIRE( 0 == 1 )`; the header was restored and + verified by sha256. + + What it can and cannot prove is stated in the case's own comment: it holds the + header's TEXT, so the false sentence cannot return, but it cannot verify the + upstream claim — no upstream checkout exists in this tree. That verification is + §1.1's commands with their positive controls, run against the two pins, and it + is not automatable here. Saying so is better than a test that appears to gate + it and does not. + +Three more added by the fourth review round (F1 #889, F2 #890, F3 #892), each RED +before its fix: + +8. **`ltx2 the reachable/marker split matches the source`** (new, + `tests/vllm/models/test_ltx2_pipeline.cpp`). The gate F1 was missing. It walks + `src/`, `include/` and `examples/` under a new `VLLM_CPP_SOURCE_ROOT` compile + definition, counts product callers of each arm's ENTRY FUNCTION — not + occurrences of the enumerator, which is the question that gave the wrong answer + — and requires the refusal message to agree, **in both directions**: no caller + demands `DECLARED, NOT REQUESTABLE`, and a caller appearing later forbids it. + Two positive controls ride in the same walk, plus a `files.size() > 100` floor, + because a walk that opened nothing reports every symbol as unreachable. + **RED before**: the walk printed `Ltx2UpsampleVideoLatent callers = + ltx2_video.cpp:1515`, `Ltx2SigmaSchedule callers = ltx2_video.cpp:1689`, + `Ltx2Schedule callers = ` — both controls live, the claim zero — and the + `DECLARED, NOT REQUESTABLE` assertion failed against the shipped message. +9. **`ltx2 video: a SPATIOTEMPORAL upsampler checkpoint is refused as + SPATIOTEMPORAL, not as temporal`** (new subcase in + `tests/vllm/multimodal/test_ltx2_video.cpp`). Closes the hole F2's reviewer + named in their own evidence: no fixture drove a BOTH-FLAGS checkpoint through + the product path, so a mutation could not separate "unreachable" from + "untested". It writes a both-flags upsampler with `WriteReducedUpsampler` and + drives it through `LoadVideoEngine` + `Generate` — the production entry point, + per `## Nothing lands dead`, not a direct call to `Ltx2LatentUpsample`. + Asserts the message names `SPATIOTEMPORAL` and is neither the temporal-only + diagnosis nor a shape complaint. **RED before**: `it is the TEMPORAL x2 + upsampler` and `Supply the spatial upsampler` both matched, and `SPATIOTEMPORAL` + did not. +10. **`ltx2 the multi-GPU marker note states the CFG evidence correctly`** (new, + `tests/vllm/models/test_ltx2_pipeline.cpp`) plus a multi-GPU block in the + ledger case. Same shape as the int8 marker's evidence gate, for the same class + of defect: the retired sentence must be absent from the header and the shipped + message must carry `four forms`, `BatchParallelGemmaBuilder` and `gemma.md`. + Anti-vacuous twice — exactly one `kMultiGpuParallelism` enumerator must exist, + and the positive checks are SCOPED to the sliced note, because `kInt8ConvRot`'s + own comment already contains the word `four`. **RED before**: 4 assertions in + the ledger case (`three forms` present, `four forms` / `BatchParallelGemmaBuilder` + / `gemma.md` absent) and `flat.find("zero \`cfg\` hits")` returned 32030 where + npos was required. + + One consequence worth recording, because it cost a build: the header may not + QUOTE the retired sentence while explaining it. A text gate matches text and + cannot tell a quotation from a claim, so the corrected note describes the old + sentence instead of reproducing it, and says why in the comment. + +## 5. Risks + +| Risk | Mitigation | +|---|---| +| Removing an enumerator breaks an out-of-tree caller | `Ltx2UnportedPipelineFeature` is internal (`include/vllm/model_executor/`), not part of `include/vllm.h`. Grep shows 3 call sites, all in-tree. | +| Refusing `duration_head_path` breaks a working caller | It cannot: no code reads it, so no caller was getting anything from it. A caller who passes it today is being silently ignored, which is the bug. | +| The ledger test's case count changes and reads as a regression | Stated up front in the gate report: 7 → 5 entries in one case, plus 2 new cases. The exit code is the authority. | +| A concurrent LTX row edits the same header | Four live rows named; none owns `ltx2_pipeline.h:575-590`, `ltx2_video.cpp:250-300`, or the ledger test case. `LTX25-TEMPORAL-UPSAMPLER` owns `kTemporalUpsampler`, which this row does not touch. | + +## 6. Stop conditions + +- Return `NEEDS_DECISION` if `multishot` turns out to exist under vocabulary not + searched here — the retirement is then wrong and the arm must be re-anchored + instead. +- Return `NEEDS_DECISION` on a collision with a live LTX row. +- Do not close #611 if the refusal cannot be made to fail RED first. + +## 7. Outcome + +**Four of the claims in §1 were NOT re-derived, and four rounds of review found +them one at a time.** The sentence that stood here said "every claim in §1 was +re-derived"; it was written before the later rounds, and it was not +true when it was written either. What actually happened: + +- Round 1 found the `hadamard` absence in §1.2 false, and the "int8 appears only + in the trainer" sentence false **in the shipped refusal message**. +- Round 2 found nine reader anchors in §2.1 stale, cited in the file they were + wrong about. +- Round 3 found `scene` asserted trainer-only **in the shipped header**, the + `shot` file list presented as exhaustive at five when it is seven, and a + `quantization_factory.py` range wrong at both ends. Re-deriving the rest of + §1.1's table in the same pass turned up a fourth: the `multi` prefix was + recorded as `multi-GPU`/`multi_gpu` **only**, and `multi-prompt` is there twice. + Nobody had asked about that row; it was wrong because the row had never been + re-run either. +- Round 4 found the `cfg` absence in §1.3 false and a fourth multigpu form + omitted, from a grep PATH-FILTERED to the two source trees — the same mechanism + as the `shot` list in round 3, in a section that had already written the lesson + down. It also found the reachable/marker split itself wrong about + `kBetaScheduler`, and the guard at `ltx2_video.cpp` shadowing the one refusal + the split was still right about. + +Every one of the four is the same defect: an absence asserted from our own +vocabulary, with no positive control, and in three cases behind a path filter that +excluded the files carrying the answer. **This row's stated purpose is retiring +instances of [#604](https://github.com/mudler/vllm.cpp/issues/604), and it +shipped four of them** — three of those into files a user or a porter reads. That +self-record is worth more than the claim it replaces, because the claim was the +mechanism: writing "every claim was re-derived" is what made the next reader stop +checking. §1.1 now carries the commands with their positive controls inline, so +the claim is reproducible instead of asserted, and +`test_ltx2_pipeline`'s "the kMultishot retirement note states the scene evidence +correctly" holds the header's text so the false sentence cannot come back. + +Results, as they now stand: + +- `multishot`: **fabricated, confirmed** — and the confirmation is now stronger + than the original. 0 hits for the term in either reference against 24 and 4 + control hits in the same commands; the only upstream sense of "shot" is one + camera take across all seven files; and the shipped prompt-enhancer prompts + instruct the model to keep a "Single continuous take — no hard cuts" + (§1.1, §1.1a). Retired. +- `int8-convrot`: **unreachable upstream, and the first wording of that was wrong.** + Four inference quantization kinds exist and none is int8, and no rotation is + exposed as one — but int8 is not trainer-only (a dead per-row int8 kernel lives + in the `ltx-kernels` inference package) and `hadamard` is not 0 hits (a whole + quantization-coupled Hadamard family is vendored, and wired to nothing). Both + corrections are in §1.2; neither moves the disposition. Kept, re-anchored. +- CFG parallelism: **the name was wrong, the exclusion is right, and the evidence + under it was wrong twice.** Not 0 `cfg` hits but 5, against 33 files as the + control, because the grep was path-filtered past `docs/multigpu/`; and four + forms, not three, the fourth being `BatchParallelGemmaBuilder`. Upstream's own + README calls MGPU a latency tool, the distilled recipe runs `SimpleDenoiser` + with no guidance, and `gemma.md:104` states outright that the distilled pipeline + "runs without CFG" — which is a reason where the old sentence was an absence + (§1.3, #892). Renamed and re-anchored. +- `kVideoEngineWiring`: **stale, confirmed.** L7 landed in `cefacd2d0`. Retired. +- The reachable/marker split: **one reachable and four markers, and the row + shipped it as two and three.** The ledger now says which kind each is, because + "refused by name" overstated a marker — and the fourth review round found that + the corrected ledger still overstated one, because §1.6 asked whether an + enumerator appears in `src/` when the question is whether anything CALLS the + function holding it. `kBetaScheduler` sits in `Ltx2Schedule`, which nothing + calls (§1.6, #889). Upstream constructs `BetaScheduler` nowhere either, so the + repair is the reclassification and not a wiring; `Ltx2Schedule` is disclosed as + unreached under `## Owed` (#893). The split is now derived from the tree by a + test rather than maintained by hand in two vectors. +- `duration_head_path` was the **only** unread key of ten (§2.1). Now refused by + name. The full inventory is the durable half of this row: it means the next + person asking "which extras are decorative?" reads a table instead of grepping. + +What this row deliberately did **not** do: construct a duration head. #611 stays +open for that, with its user-visible half — silent substitution of the recipe +default — closed. + +**The review's own findings, recorded because they are the point of the row.** +Across four rounds all but one finding was evidence accuracy, not logic: a false +absence inside the shipped refusal (§1.2), a false absence in this spec's §1.2 — +the one search term that mattered — nine stale reader anchors cited in the file +they were wrong about (§2.1), a public-doc row that re-merged the split the ledger +had just made, a false absence inside the shipped HEADER (§1.1a), a five-item list +offered as exhaustive when it is seven (§1.1), a line range wrong at both ends +(§1.2), and a `cfg` absence plus a missing fourth form in §1.3 and the shipped +header (#892). A row whose subject is #604 committed four instances of #604, and +each one survived at least one review that had already been told to look for +exactly this. That is not irony to note and move past; it is why the anchors are +now derived rather than written, why the int8 absence is stated as UNREACHABLE +with its dead kernel named, why the `hadamard` search this spec got wrong is +spelled out in full rather than quietly corrected, and why every absence in §1.1 +now ships with the command that proves it AND the control hits that prove the +command ran. + +**The exception is round 4's other two findings, and they are a different +defect.** F1 (#889) and F2 (#890) were not inaccurate evidence — they were +CLASSIFICATION and CONTROL FLOW. `kBetaScheduler` was published as reachable +because §1.6 asked whether an enumerator appears in `src/`, which is a question +whose true answer is the wrong answer. And the one arm that genuinely was +reachable had its refusal shadowed by a guard on `temporal_upsample` that every +both-flags config also satisfies, so between them the ledger's reachable column +was empty in practice while both surfaces published two entries. A record can be +false without a single false citation in it, and this row had been correcting +citations for three rounds while the thing the citations described was wrong. + +The transferable lessons, now two: + +**A path filter is an absence claim too.** The `shot` enumeration is the case +where this is provable rather than inferred — the pattern `\bshots?\b` was right +and the filter `*.py *.md *.json *.yaml` was wrong, over a tree whose prompt files +are `.txt`, and it returns exactly the five files that were recorded. §1.3's `cfg` +claim is the second provable case: the pattern was right and the filter to the two +SOURCE trees excluded the `docs/` tree that answers it. (The `scene` claim's +original command was never written down, so why *it* missed `ltx-core` is not +recoverable; that it was never reproduced is the point.) A grep that returns +nothing has to be re-run without its narrowing, and with a control, before it is +written down as an absence. + +**Reachability is a claim about CALLERS, and `git grep` for the symbol does not +answer it.** `Ltx2Schedule` holds the beta refusal, appears in the header and the +source, is covered by a passing unit test, and is called by nothing. AGENTS.md +`## Nothing lands dead` and [`.agents/reachability.md`](../reachability.md) landed +in `8f49ac3be` naming this exact shape — the test-only driver — while this repair +was in flight. The gate this row now ships derives the split by counting callers +of each arm's entry function, with positive controls in the same walk, and fails +in both directions. + +## Owed + +Disclosed under AGENTS.md `## Nothing lands dead`, which requires the unreached +thing, its owning row and its issue to be named rather than left for the next +reader: + +- **`Ltx2Schedule` / `Ltx2SchedulerKind` are unreached** — + [#893](https://github.com/mudler/vllm.cpp/issues/893), owned by this row. The + dispatcher at `src/vllm/model_executor/models/ltx2_pipeline.cpp:191` has zero + call sites in `src/`, `include/` or `examples/`; the engine calls + `Ltx2SigmaSchedule` directly and its only driver is its own unit test. This is + not a staged slice awaiting a wiring wave: upstream selects no scheduler + anywhere (§1.6), so there is nothing to wire it to, and the open decision is + whether to retire it the way `kMultishot` was or keep it as a deliberately + unreached mirror of upstream's `SchedulerProtocol`. Deciding that is a behavior + change with its own red-first evidence, which is why the F1 repair labelled the + arm honestly and did not also delete the dispatcher. +- **`kLoraFusion`, `kInt8ConvRot` and `kMultiGpuParallelism` have no call site at + all** — [#691](https://github.com/mudler/vllm.cpp/issues/691), owned by this + row. The derived split gates the beta arm because there is a function to count + callers of; the other three markers have nothing to derive from, so their kind + is still asserted rather than proven. #691 stays open for exactly that gap. +- **A duration head is still not constructed** — + [#611](https://github.com/mudler/vllm.cpp/issues/611), owned by this row. Its + user-visible half, silent substitution of the recipe default, is closed. +- **`.agents/specs/ltx-2-5.md` §2 "Out" still lists `multishot`** — + [#692](https://github.com/mudler/vllm.cpp/issues/692). Operator-owned file. + +## Now + +Row `LTX25-RETIRE-DEAD-ARMS` is `DONE`. The ledger carries five entries split by +kind — **one reachable, four markers** — and the split is derived from the tree by +`test_ltx2_pipeline`'s "the reachable/marker split matches the source" rather than +maintained by hand. The one reachable arm is no longer shadowed: a both-flags +upsampler checkpoint driven through `LoadVideoEngine` is refused as +SPATIOTEMPORAL. `duration_head_path` is refused by name with a RED-first test; the +`kKnownLoadExtras` inventory is recorded in §2.1; the header's `kMultishot` and +`kMultiGpuParallelism` notes are both gated by `test_ltx2_pipeline` against the +false absences they shipped (§1.1a, §1.3). Owed items are listed above. +`.agents/specs/ltx-2-5.md` §2 "Out" still lists `multishot` and is the operator's +to correct, citing §1.1 here. diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 90123af1e..da1ece71a 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -332,6 +332,9 @@ CPU elementwise GEMM (f32/f16/bf16) runs AVX2 and AVX-512 tiers on x86 where the |---|---|---| | Kimi-Linear-48B-A3B (KDA + NoPE-MLA + MoE hybrid) | **Runner fold LANDS (ROW 7 §21, #122): the ENGINE/SERVER surface serves Kimi at the 122/128 golden profile (engine==CLI 128/128); STRICT stays closed (intrinsic p7 near-tie)** | server 19.0 tok/s wall / CLI 18.9 vs vLLM ~21 (~0.90×), speed residual named (§21) | | Muse Glimmer 30B (Meta) | Text gated at **reduced depth 4/52** only; vision wired but never reference-checked | [spec](../.agents/specs/muse-glimmer.md) / [#268](https://github.com/mudler/vllm.cpp/issues/268). Full depth, multi-step decode, image/video, server path and parser scoping open. vLLM speed OPEN GAP; llama.cpp bar #333 | +| LTX-2.5 AUTO duration (the duration head) | Brick ported, never constructed | `duration_head_path` is REFUSED by name rather than accepted-and-ignored ([#611](https://github.com/mudler/vllm.cpp/issues/611)); supplying a head cannot load one. Give `num_frames` or `duration` | +| LTX-2.5 arms a request CAN reach | Refused by name at the call site | The spatiotemporal latent upsampler (both flags set). Supplying that checkpoint names that arm, not the temporal one. The temporal-only x2 arm is ported, not refused | +| LTX-2.5 arms nothing can request | Declared, not requestable | LoRA fusion, `int8-convrot` (ComfyUI-only), single-node multi-GPU, `BetaScheduler` (upstream selects no scheduler either). No flag or extra asks for these. `multishot` was RETIRED: no such entry point exists upstream | | Multi-GPU execution | Hardware-blocked | TP proven equal to tp=1 on CPU; no 2-GPU box to run it | | LoRA end to end | CPU brick landed | Unwired standalone; not usable through the server | | Multimodal over HTTP | Image request path wired; forward + codec pending | `ROAD-V1-MM` W1-W3 landed. Open: no mm-forward on `Request.mm_features`; no image codec. Video/audio/multi-image now **refuse** with HTTP 400 rather than drop ([#686](https://github.com/mudler/vllm.cpp/issues/686)) | diff --git a/docs/USAGE.md b/docs/USAGE.md index 02dae8fee..7843db298 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -1955,14 +1955,47 @@ knobs from `extras`. H3 takes `partition`. LTX-2.5 takes `audio_prompt_embeds_path` (the audio stream's conditioning, the twin of the seam's `prompt_embeds_path`, which carries the video stream), `pipeline_kind` (default `distilled_two_stage`), `model_version` (only for a checkpoint that -declares none), `dit_config_path`, `allow_unported_modules`, `max_phase`, -`prompt_embeds_valid_rows`, `upsampler_path` and `duration_head_path`. An extra a -family does not define is refused, never ignored. One caveat inside that set: -`duration_head_path` is accepted but INERT — the duration head is ported and gated -as a brick, nothing in the video engine constructs one, and no code reads that -key, so supplying it neither loads a head nor enables an AUTO duration. Give +declares none), `dit_config_path`, `encoder_config_path`, +`allow_unported_modules`, `max_phase`, `prompt_embeds_valid_rows`, +`upsampler_path` and `duration_head_path`. An extra a family does not define is +refused, never ignored. One caveat inside that set: `duration_head_path` is +defined but UNSERVED — the duration head is ported and gated as a brick, and +nothing in the video engine constructs one — so supplying it is **refused by +name** at load rather than accepted. It used to be accepted and read by nothing, +which silently substituted the recipe default for the file you named. Give `num_frames` (or `duration`, which is exact arithmetic against the recipe's frame -rate) instead. +rate) instead. Every other key in that list reaches a reader. + +One LTX-2.5 arm is refused where a render would otherwise silently downgrade: +the spatiotemporal latent upsampler. It is reachable — supplying that checkpoint +as `upsampler_path` gets a refusal naming the arm you actually supplied. The +spatiotemporal upsampler is the arm with `spatial_upsample` AND +`temporal_upsample` set, which upstream builds as a different operator +(`Conv3d(mid, 8*mid)` + `PixelShuffleND(3)`). The temporal-only x2 upsampler is +**ported** and is not refused; nothing shipped drives it yet, so it is gated +rather than served. Four more are +recorded as out of scope but are **not requestable**, so no flag or extra can +reach them: LoRA fusion, `int8-convrot`, single-node multi-GPU, and +`BetaScheduler`. Their messages +say `DECLARED, NOT REQUESTABLE` so the two kinds are not confused. +`BetaScheduler` is in that group rather than the reachable one because upstream +selects it nowhere: every `ltx-pipelines` entry point hard-codes +`LTX2Scheduler()`, so there is no scheduler-kind field to mirror and nothing here +carries one either. `int8-convrot` +in particular is a ComfyUI-ecosystem format: upstream LTX-2's own inference +quantization kinds are `fp8-cast`, `fp8-scaled-mm`, `nvfp4-cast` and +`nvfp4-prequant`, and nothing wired upstream reaches int8 at all. + +What is **not** on that list, and why: **multi-shot or multi-scene generation.** +A request that composes several camera takes into one output has no flag here +because upstream LTX-2 has no such mode to mirror — its `shot` is one continuous +take, and its own prompt-enhancement prompts instruct the model to keep a "single +continuous take" and not to describe scene cuts. `scene` does appear across the +upstream tree, in three unrelated senses (`scene-linear` HDR colour, PySceneDetect +in the trainer's dataset preprocessor, and that prompt-writing guidance); none of +them is a generation mode. This port carried a `multishot` refusal until +2026-08-13, which was a defect in our own record rather than a gap, and it was +retired. Generate one take per request. `prompt_embeds_valid_rows` is how many of the supplied conditioning rows are real tokens; absent, every row is. It matters because the embeddings connector diff --git a/examples/ltx2_gen/main.cpp b/examples/ltx2_gen/main.cpp index 28ed2cc0a..8aa7b0a31 100644 --- a/examples/ltx2_gen/main.cpp +++ b/examples/ltx2_gen/main.cpp @@ -179,6 +179,9 @@ int main(int argc, char** argv) { else if (f == "--model-version") SetExtra("model_version", Need(argc, argv, ++i, f.c_str())); else if (f == "--pipeline-kind") SetExtra("pipeline_kind", Need(argc, argv, ++i, f.c_str())); else if (f == "--upsampler") SetExtra("upsampler_path", Need(argc, argv, ++i, f.c_str())); + // Kept although the library REFUSES this extra by name (#611): the duration + // head is unported, and forwarding the flag gets the caller that named + // refusal instead of "unknown option", which says nothing about why. else if (f == "--duration-head") SetExtra("duration_head_path", Need(argc, argv, ++i, f.c_str())); else if (f == "--max-phase") SetExtra("max_phase", Need(argc, argv, ++i, f.c_str())); diff --git a/include/vllm/model_executor/models/ltx2_pipeline.h b/include/vllm/model_executor/models/ltx2_pipeline.h index 906891732..3e87cfeff 100644 --- a/include/vllm/model_executor/models/ltx2_pipeline.h +++ b/include/vllm/model_executor/models/ltx2_pipeline.h @@ -114,8 +114,24 @@ std::vector Ltx2LinearQuadraticSchedule(int64_t steps, double threshold_n // `Ltx2Schedule` REFUSES it by name rather than substituting LTX2Scheduler. enum class Ltx2SchedulerKind { kLtx2, kLinearQuadratic, kBeta }; -// The seam a caller reaches for when it holds a configured kind. Forwards to the -// two ported schedulers and throws for `kBeta`. +// The seam a caller would reach for if it held a configured kind. NOTHING IN +// `src/`, `include/` OR `examples/` CALLS THIS, and that is upstream's shape, not +// an omission: no ltx-pipelines entry point selects a scheduler either, so there +// is no request field to carry a kind and the engine calls `Ltx2SigmaSchedule` +// directly, in `ltx2_video.cpp`'s phase driver. Say "no caller" rather than "the +// seam a caller reaches for": that wording is what published `kBetaScheduler` as +// a reachable refusal (#889). No line number: that file moves on every merge, and +// this row has already shipped three anchors that went stale inside one branch. +// +// UNREACHED, AND DELIBERATELY SO UNTIL #893 DECIDES OTHERWISE. Under AGENTS.md +// `## Nothing lands dead` this is the "test-only driver" shape, and the rule asks +// for the unreached thing, its owning row and its issue to be named rather than +// left for the next reader to discover. Owning row LTX25-RETIRE-DEAD-ARMS, which +// lists it under `## Owed`. There is no wiring wave coming — upstream has no +// scheduler selection to mirror — so the open question is retire-or-keep, not +// when to wire it. +// +// Forwards to the two ported schedulers and throws for `kBeta`. std::vector Ltx2Schedule(Ltx2SchedulerKind kind, int64_t steps, int64_t tokens, const Ltx2SchedulerParams& params = {}); @@ -571,23 +587,110 @@ Ltx2PipelineRecipe ResolveLtx2PipelineRecipe(const std::string& pipeline_kind, const std::string& model_version); // --------------------------------------------------------------------------- -// Out of scope for L5, refused by name (spec section 2, "Out") +// Out of scope, refused by name (spec section 2 "Out"; the 2026-08-13 grounding +// pass is .agents/specs/ltx25-retire-dead-arms.md, row LTX25-RETIRE-DEAD-ARMS) // --------------------------------------------------------------------------- // Each of these renders something plausible if it is silently downgraded, which // is why none of them falls back. `Ltx2RefuseUnportedPipelineFeature` throws with -// a message naming the missing piece and the phase or row that owes it. +// a message naming the missing piece and the row that owes it. +// +// TWO KINDS live here, and conflating them overstated what this port refuses: +// +// REACHABLE REFUSAL — a product path constructs the condition and throws, so a +// caller CAN trip it. `kSpatiotemporalUpsampler` (ltx2_upsampler.cpp:465) is +// the ONE. `ltx2_video.cpp` reaches it through `Ltx2UpsampleVideoLatent` when a +// phase asks for the spatial-upsample transform. The TEMPORAL-ONLY x2 +// upsampler is NOT among them: it is ported (`2e9d95e74`, spec +// .agents/specs/ltx25-temporal-upsampler.md), which is why the enumerator that +// used to be spelled `kTemporalUpsampler` now names the spatiotemporal arm +// only. Nothing shipped drives the ported arm yet, so it is gated, not served. +// +// The definition above is a claim about CALLERS, and it takes a caller to +// satisfy it. A `case` label is not one. `kBetaScheduler` was published here as +// the second reachable arm for the whole of row LTX25-RETIRE-DEAD-ARMS, and it +// is not: its call site `ltx2_pipeline.cpp:199` sits inside `Ltx2Schedule`, +// which nothing calls (#889). Recorded rather than quietly moved, because the +// row's subject is exactly this — a classification asserted instead of derived, +// and AGENTS.md `## Nothing lands dead` now names the shape it took. The split +// is gated by `test_ltx2_pipeline`'s "the reachable/marker split matches the +// source", which walks src/, include/ and examples/ and carries two positive +// controls in the same walk. That gate is the anti-tautological shape #691 +// asked for, for the beta arm; #691 stays open for the other three markers. +// +// DECLARED-OUT-OF-SCOPE MARKER — no request field, load extra or CLI flag asks +// for it, so nothing outside the ledger test reaches it. It is a record of what +// upstream HAS and this port does NOT, which is worth keeping; calling it a +// refusal is what was wrong. `kBetaScheduler`, `kLoraFusion`, `kInt8ConvRot` +// and `kMultiGpuParallelism` are markers, and their messages say so. +// +// TWO ENUMERATORS WERE RETIRED on 2026-08-13, recorded here because the +// retirement IS the record — a reader who finds them in git history needs to know +// they did not simply move: +// +// `kMultishot` — FABRICATED. It refused "multishot generation" and cited +// "ltx-pipelines multishot entry points". No such entry point, symbol or string +// exists in Lightricks/LTX-2 @ fd4ded7f or huggingface/diffusers @ 3a2f35d4. +// Searched as a SUBJECT rather than by our own phrasing: upstream's only sense +// of "shot" is ONE camera take (duration_head.py:1,5 "predicts shot duration"; +// README.md:136 "a cinematographer describing a shot list"). `scene` has THREE +// senses upstream and none is a generation mode: `scene-linear` HDR colour +// (ltx-core color/hlg.py, hdr.py), PySceneDetect in the TRAINER — the only CODE +// sense — and prompt-writing guidance, which ships at INFERENCE inside +// `ltx-core`, in text_encoders/gemma/encoders/prompts/ as +// gemma{3,4}_{i2v,t2v}_system_prompt.txt. That third sense is why the +// retirement HOLDS rather than being undermined: those prompts tell the +// enhancer NOT to describe scene cuts and to keep a "Single continuous take" +// (gemma3_i2v:6,18, gemma3_t2v:24, gemma4_i2v:3). A defect in our record is not +// a gap in our port, so there was nothing to owe. +// +// Recorded because it is the row's own subject: this paragraph used to claim +// that `scene` appeared upstream ONLY as PySceneDetect in the trainer. It was +// an absence asserted from our own vocabulary with no positive control — #604 — +// shipped in the header of the row that exists to retire #604 instances, and it +// took a third review round to find. The derivation, with its positive control +// in the same command, is .agents/specs/ltx25-retire-dead-arms.md §1.1. +// +// `kVideoEngineWiring` — LANDED. It said the end-to-end composition through +// `vllm::multimodal::VideoEngine` "is phase L7, not L5"; L7 shipped in +// `cefacd2d0`. A refusal whose subject shipped is a false statement. enum class Ltx2UnportedPipelineFeature { + // Reachable refusal. Singular. // model/upsampler with BOTH flags set. The temporal-ONLY arm is ported // (.agents/specs/ltx25-temporal-upsampler.md); this one is a different // operator — `Conv3d(mid, 8*mid)` + `PixelShuffleND(3)`, model.py:55-59. kSpatiotemporalUpsampler, - kLoraFusion, // loader/LoraPathStrengthAndSDOps - kMultishot, // ltx-pipelines multishot entry points - kInt8ConvRot, // ComfyUI-only quantization - kCfgParallelism, // ltx-pipelines/multigpu - kVideoEngineWiring, // end-to-end through vllm::multimodal::VideoEngine (L7) - kBetaScheduler, // components/schedulers.py:91-120 + // Declared-out-of-scope markers. + kBetaScheduler, // ltx-core components/schedulers.py:91-120. A MARKER because + // upstream constructs it nowhere: all seven ltx-pipelines entry + // points hard-code `LTX2Scheduler()`, and vLLM-Omni @ a4ea67a21 + // has zero hits for the name. Mirroring that means no + // scheduler-kind field here either, so nothing reaches the + // refusal — `Ltx2Schedule`, which holds it, has no caller. + kLoraFusion, // ltx-core loader/primitives.py:160 (LoraPathStrengthAndSDOps), + // fused by loader/fuse_loras.py + kInt8ConvRot, // ComfyUI-ecosystem quantization, and NOT an LTX-2 arm: the four + // inference kinds upstream defines are fp8-cast / fp8-scaled-mm / + // nvfp4-cast / nvfp4-prequant (quantization_factory.py:23-26). + // `convrot` is nowhere at all; int8 is UNREACHABLE rather than + // absent — trainer-only for anything wired, plus one DEAD kernel + // in ltx-kernels (triton_ops.py:35,43). §1.2 of the row spec + kMultiGpuParallelism, // ltx-pipelines/multigpu — four forms: sequence-parallel, + // tiled data parallel, distributed VAE decode, and + // batch-parallel Gemma encoding (bp_gemma_builder.py:42, + // `BatchParallelGemmaBuilder`), which partitions a prompt + // list across ranks. None is CFG batching, and the reason + // is upstream's own: docs/multigpu/gemma.md:103-104 calls a + // positive+negative pair "the typical CFG case" and records + // that the DISTILLED pipeline this port runs takes no + // negative_prompt, so it "runs without CFG". This used to + // assert instead that the string was absent from both + // multigpu trees, which came from a PATH-FILTERED grep that + // excluded the docs/ tree carrying the answer: 5 hits, not + // 0, against 33 files as the control (#892, §1.3 of the row + // spec). The false sentence is not repeated here, because + // the gate on it matches TEXT and cannot tell a quotation + // from a claim }; [[noreturn]] void Ltx2RefuseUnportedPipelineFeature(Ltx2UnportedPipelineFeature feature); diff --git a/include/vllm/multimodal/ltx2_video.h b/include/vllm/multimodal/ltx2_video.h index b74f38d0f..f0d4f20b3 100644 --- a/include/vllm/multimodal/ltx2_video.h +++ b/include/vllm/multimodal/ltx2_video.h @@ -9,10 +9,13 @@ // Phases L2-L6 shipped a DiT forward, a text feature extractor, two VAEs, a // vocoder, an upsampler, a duration head, and a pipeline COMPONENT library — // schedules, noisers, steppers, guiders, patchifiers, recipes. Nothing drove -// them: `Ltx2RefuseUnportedPipelineFeature(kVideoEngineWiring)` refused the +// them: an `Ltx2UnportedPipelineFeature::kVideoEngineWiring` refusal named the // composition BY NAME and named this phase as its owner. This TU is that // composition and nothing else. It adds no numerics; every line either resolves -// a parameter, moves a buffer, or calls a brick that already has a golden. +// a parameter, moves a buffer, or calls a brick that already has a golden. (That +// enumerator was RETIRED in row LTX25-RETIRE-DEAD-ARMS once L7 landed in +// `cefacd2d0` — a refusal whose subject shipped is a false statement, not a +// record of debt. It is named here in the past tense on purpose.) // // ─── WHAT IT IS A PORT OF (file:line on BOTH sides) ────────────────────────── // Upstream: Lightricks/LTX-2 @ fd4ded7, packages/ltx-pipelines/src/ltx_pipelines/ diff --git a/src/vllm/model_executor/models/ltx2_pipeline.cpp b/src/vllm/model_executor/models/ltx2_pipeline.cpp index cbb9d704a..a54f87763 100644 --- a/src/vllm/model_executor/models/ltx2_pipeline.cpp +++ b/src/vllm/model_executor/models/ltx2_pipeline.cpp @@ -1138,7 +1138,17 @@ Ltx2PipelineRecipe ResolveLtx2PipelineRecipe(const std::string& pipeline_kind, void Ltx2RefuseUnportedPipelineFeature(Ltx2UnportedPipelineFeature feature) { const std::string owed = - " Not ported by phase L5; recorded as owed in .agents/specs/ltx-2-5.md."; + " Recorded as owed in .agents/specs/ltx-2-5.md; grounded against Lightricks/LTX-2 " + "fd4ded7f in .agents/specs/ltx25-retire-dead-arms.md."; + // A marker is not a refusal a caller can trip, and saying so is the point: this + // enum used to read as six live refusals when only ONE has a product call site. + // "Two" is what this said until review found that the second, `kBetaScheduler`, + // sits inside `Ltx2Schedule`, which nothing calls — see the case below its + // enumerator in the header, and `test_ltx2_pipeline`'s "the reachable/marker + // split matches the source", which derives that from the tree. + const std::string marker = + " DECLARED, NOT REQUESTABLE: no request field or load extra asks for this, so nothing " + "but the out-of-scope ledger reaches this message."; switch (feature) { case Ltx2UnportedPipelineFeature::kSpatiotemporalUpsampler: Refuse("ltx2: the SPATIOTEMPORAL latent upsampler (spatial_upsample AND " @@ -1148,24 +1158,82 @@ void Ltx2RefuseUnportedPipelineFeature(Ltx2UnportedPipelineFeature feature) { "and PixelShuffleND(1). Owed and recorded in " ".agents/specs/ltx25-temporal-upsampler.md section 2, under the campaign " ".agents/specs/ltx-2-5.md."); + case Ltx2UnportedPipelineFeature::kBetaScheduler: + // A MARKER, not a reachable refusal, and the correction is upstream's. This + // case label is inside `Ltx2Schedule`, which no product code calls: the + // engine calls `Ltx2SigmaSchedule` directly, in `ltx2_video.cpp`'s phase + // driver, and no ABI field, load extra or CLI flag carries a kind. That mirrors LTX-2 + // @ fd4ded7f, where `BetaScheduler` is DEFINED at ltx-core + // components/schedulers.py:91 and CONSTRUCTED nowhere: all seven pipelines + // hard-code `LTX2Scheduler()` (ti2vid_one_stage.py:81, ti2vid_two_stages.py:87, + // ti2vid_two_stages_hq.py:90, a2vid_two_stage.py:78, t2a_one_stage.py:67, + // keyframe_interpolation.py:82, retake.py:96), and vLLM-Omni @ a4ea67a21 has + // zero hits for the name. Publishing this as reachable would have promised a + // selection surface that upstream does not have. + Refuse("ltx2: BetaScheduler (ltx-core components/schedulers.py:91-120) is not ported. It " + "inverts a Beta CDF through scipy.stats.beta.ppf. Upstream constructs it nowhere — " + "every ltx-pipelines entry point hard-codes LTX2Scheduler() — so mirroring upstream " + "means this port has no scheduler-kind field either." + + marker + owed); case Ltx2UnportedPipelineFeature::kLoraFusion: - Refuse("ltx2: LoRA fusion (loader/LoraPathStrengthAndSDOps) is out of scope." + owed); - case Ltx2UnportedPipelineFeature::kMultishot: - Refuse("ltx2: multishot generation is out of scope." + owed); + Refuse("ltx2: LoRA fusion (ltx-core loader/primitives.py:160 LoraPathStrengthAndSDOps, " + "fused by loader/fuse_loras.py) is out of scope." + + marker + owed); case Ltx2UnportedPipelineFeature::kInt8ConvRot: - Refuse("ltx2: the int8-convrot quantization (ComfyUI-only) is out of scope." + owed); - case Ltx2UnportedPipelineFeature::kCfgParallelism: - Refuse("ltx2: CFG / multi-GPU parallelism (ltx-pipelines/multigpu) is out of scope." + - owed); - case Ltx2UnportedPipelineFeature::kVideoEngineWiring: - Refuse("ltx2: end-to-end wiring through vllm::multimodal::VideoEngine is phase L7, not " - "L5." + - owed); - case Ltx2UnportedPipelineFeature::kBetaScheduler: - Refuse("ltx2: BetaScheduler (components/schedulers.py:91-120) is not ported. It inverts " - "a Beta CDF through scipy.stats.beta.ppf, and no ltx-pipelines entry point " - "constructs it." + - owed); + // VERIFIED UNREACHABLE so nobody re-audits it, and stated as UNREACHABLE rather + // than ABSENT because absent is what the first version of this message claimed + // and it was false. At LTX-2 @ fd4ded7f: `convrot` / `conv_rot` / `quarot` / + // `spinquant` really are 0 hits, and the four inference quantization kinds are + // exhaustive (quantization_factory.py:23-26, `assert_never` at :50). But int8 + // is NOT trainer-only. `ltx-kernels` — an inference package — carries a per-row + // int8 quantize kernel with fp32 scales (blockwise/triton_ops.py:25-50, out + // dtype `torch.int8` at :43), aliased `rowwise_int_quantize_triton` at :436. + // That alias is its ONLY reference: blockwise/functional.py:12-18 re-exports + // five names and not this one, so nothing constructs it. The package is a fork + // of Lightricks' int8 kernel library retargeted to fp8/fp6/nvfp4 — its custom-op + // namespace is still literally `q8_kernels_ops` (functional.py:25) — and the + // int8 half is what was left behind. Nothing wired reaches int8, which is why + // the disposition is unchanged; only the sentence was wrong. + Refuse("ltx2: the int8-convrot quantization is out of scope. It is a ComfyUI-ecosystem " + "format, not an LTX-2 arm: upstream's own inference kinds are fp8-cast, " + "fp8-scaled-mm, nvfp4-cast and nvfp4-prequant (ltx-pipelines/utils/" + "quantization_factory.py:23-26), and int8 is UNREACHABLE upstream — trainer-only " + "for anything wired (ltx-trainer gemma_8bit.py:33-36, quantization.py:11-15), plus " + "one DEAD per-row int8 quantize kernel in the ltx-kernels inference package " + "(blockwise/triton_ops.py:35,43, reached only by its own alias at :436)." + + marker + owed); + case Ltx2UnportedPipelineFeature::kMultiGpuParallelism: + // The old spelling was `kCfgParallelism`, which named something upstream does + // not do. There is no CFG pass to split here in the first place: the distilled + // recipe denoises with SimpleDenoiser at both stages (distilled.py:266,295), + // "single transformer call, no guidance" (utils/denoisers.py:3). + // + // THIS MESSAGE SAID "three forms and none of them is CFG batching", AND BOTH + // HALVES WERE WRONG. The count missed `BatchParallelGemmaBuilder` + // (multigpu/bp_gemma_builder.py:42), a fourth `BuilderProtocol` in the very + // directory cited above. And the CFG half rested on a grep PATH-FILTERED to + // the two source trees, which excluded `ltx-pipelines/docs/multigpu/`: re-run + // over `-- '*multigpu*'` (33 files, the control) it is 5 hits, not 0, two of + // them prose about CFG at docs/multigpu/gemma.md:103-104. + // + // The disposition did not move; it got stronger. gemma.md:104 says the + // distilled pipeline runs "without CFG", so the one form that WOULD batch a + // CFG pair is the one upstream tells you not to use for the recipe this port + // runs. Stating the reason beats asserting an absence — §1.3 of the row spec. + Refuse("ltx2: single-node multi-GPU parallelism (ltx-pipelines/multigpu) is out of " + "scope. Upstream has four forms: sequence-parallel " + "(multigpu/sp_builder.py:25), tiled data parallel " + "(multigpu/tdp_builder.py:25, upscale stage only), distributed VAE decode " + "(ltx-core multigpu/vae/distributed_decoder.py:204-256) and batch-parallel Gemma " + "encoding (multigpu/bp_gemma_builder.py:42 BatchParallelGemmaBuilder), which " + "partitions a PROMPT LIST across ranks. None is CFG batching, and the fourth is " + "the closest thing to it: upstream's own docs/multigpu/gemma.md:103-104 calls a " + "positive+negative pair 'the typical CFG case' and then records that the DISTILLED " + "pipeline — the one this port runs — takes no negative_prompt and so 'runs without " + "CFG', leaving nothing to partition. It is a LATENCY tool, " + "not a memory tool (docs/multigpu/README.md:5-16), and this port targets one " + "GB10." + + marker + owed); } Refuse("ltx2: unknown unported pipeline feature." + owed); } diff --git a/src/vllm/multimodal/ltx2_video.cpp b/src/vllm/multimodal/ltx2_video.cpp index fb7143ea1..08d9392d4 100644 --- a/src/vllm/multimodal/ltx2_video.cpp +++ b/src/vllm/multimodal/ltx2_video.cpp @@ -257,15 +257,36 @@ int64_t ExtraInt(const std::map& extras, const std::st } } +// The one key this family DEFINES and does not SERVE. `Ltx2DurationPredict` is +// ported and gated as a brick (`ltx2_duration_head.h`), but nothing here +// constructs one, so a supplied path names a file the engine never opens. +constexpr char kLtx2DurationHeadPathExtra[] = "duration_head_path"; + // Every extra key this family DEFINES. An extra outside this set is refused // rather than ignored, for the same reason H3 refuses one // (minimax_h3_video.cpp): a mistyped knob that is silently dropped renders the // DEFAULT and looks like the feature not working. +// +// DEFINED IS NOT THE SAME AS SERVED, and conflating the two was #611: nine of +// these ten reach a reader, and `duration_head_path` reached none, so supplying a +// duration head substituted the recipe default in silence — the failure mode this +// very list exists to prevent, one level in. It stays in the list because the +// family DOES define the key and DOES know what it means; `CheckUnservedExtras` +// refuses it by name instead, which is a different and truer message than +// "unknown load extra". The full audit is in +// .agents/specs/ltx25-retire-dead-arms.md §2.1. +// +// The first hand-written set of these anchors named nine lines that were readers +// of NOTHING, in this very file, and a later merge moved the real ones again. So +// they are no longer trusted: the list below is derived from this file on every +// run and compared, and the failure prints the replacement to paste in. +// READER ANCHORS (derived and gated by test_ltx2_video): +// 660 715 811 827 829 899 924 1029 1070 const char* const kKnownLoadExtras[] = { kLtx2AudioPromptEmbedsExtra, kLtx2PipelineKindExtra, kLtx2ModelVersionExtra, kLtx2AllowUnportedExtra, kLtx2MaxPhaseExtra, kLtx2DitConfigPathExtra, kLtx2PromptValidRowsExtra, kLtx2EncoderConfigPathExtra, - "upsampler_path", "duration_head_path", + "upsampler_path", kLtx2DurationHeadPathExtra, }; // FNV-1a over the raw bytes of a float buffer — the `Ltx2ConditioningTrace` @@ -304,6 +325,27 @@ void CheckKnownExtras(const std::map& extras) { } } +// A key this family DEFINES but does not SERVE, refused BY NAME when supplied +// (#611). The alternative — accepting it — is the worst of the three options: +// worse than refusing, and worse than not defining the key, because the caller +// pointed at a specific file and got the recipe default with no diagnostic. +// +// Deliberately NOT the "unknown load extra" path above. That message says the +// family does not define the key, which is false here and would send the reader +// looking for a typo instead of for the unported head. +void CheckUnservedExtras(const std::map& extras) { + const std::string duration_head = VideoExtra(extras, kLtx2DurationHeadPathExtra); + if (!duration_head.empty()) { + Fail("the '" + std::string(kLtx2DurationHeadPathExtra) + "' extra names '" + duration_head + + "', but the duration head is NOT WIRED into this engine: `Ltx2DurationPredict` is ported " + "and gated as a brick (ltx2_duration_head.h, upstream duration_head.py:89-118) and " + "nothing here constructs one, so that file would never be opened and an AUTO duration " + "would fall back to the recipe default. Give 'num_frames', or 'duration' (exact " + "arithmetic against the recipe frame rate), instead. Refused rather than ignored; " + "recorded as owed in .agents/specs/ltx25-retire-dead-arms.md (#611)."); + } +} + // `detect_model_version` normalizes the separator before parsing // (utils/constants.py:161), and the recipe table is keyed on the two-component // spelling ("2.5"), not on the checkpoint's three-component "2.5.0". Reduce it @@ -533,6 +575,7 @@ Ltx2ConditioningTrace Ltx2VideoEngine::last_conditioning() const { std::unique_ptr Ltx2VideoEngine::Load(const VideoModelParams& params) { if (params.dit_path.empty()) Fail("dit_path is required"); CheckKnownExtras(params.extras); + CheckUnservedExtras(params.extras); auto engine = std::unique_ptr(new Ltx2VideoEngine()); engine->impl_ = std::make_unique(); @@ -1338,12 +1381,12 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { // needs an encoded prompt this engine cannot produce", and since `has_encoder` // above the engine produces exactly that. What is missing now is the head // itself — `ltx2_duration_head.h` is ported and gated as a brick, but nothing - // here constructs one, and `duration_head_path` is accepted in - // `kKnownLoadExtras` while NO code reads it (grep: it appears at that one - // site). So the extra is inert rather than wired, and that is recorded as owed - // rather than left to be discovered by someone who supplies it and gets the - // recipe default. An explicit duration is exact arithmetic, so it is served; - // the AUTO path is what is missing, and `num_frames` is how to avoid it. + // here constructs one. `duration_head_path` used to be ACCEPTED while no code + // read it, so a caller who supplied a head silently landed on this line + // instead; `CheckUnservedExtras` now refuses that key by name at load (#611, + // .agents/specs/ltx25-retire-dead-arms.md §2). What remains owed is the head + // itself. An explicit duration is exact arithmetic, so it is served; the AUTO + // path is what is missing, and `num_frames` is how to avoid it. frames = static_cast(std::llround(gen.duration_seconds * fps)); } if (frames < 1) Fail("num_frames resolved to " + std::to_string(frames)); @@ -1438,7 +1481,18 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { // the difference between "you gave me the wrong checkpoint" and "something // is 3 frames short". Ported and gated, not driven — see // .agents/specs/ltx25-temporal-upsampler.md section 7. - if (im.upsampler_cfg.temporal_upsample) { + // + // `&& !spatial_upsample` IS LOAD-BEARING. This guard used to test + // `temporal_upsample` alone, which every BOTH-flags config also satisfies, + // so it fired by implication over the same variable and told the caller who + // supplied a genuine SPATIOTEMPORAL checkpoint that they had handed over the + // temporal one — wrong on both counts, and pointing them at the arm they + // already had. It also shadowed the ledger refusal at + // `ltx2_upsampler.cpp:465`, which names the spatiotemporal arm and was + // therefore unreachable from any request. Narrowed here so a both-flags + // config falls THROUGH to that refusal. Gated by test_ltx2_video's + // "a SPATIOTEMPORAL upsampler checkpoint is refused as SPATIOTEMPORAL". + if (im.upsampler_cfg.temporal_upsample && !im.upsampler_cfg.spatial_upsample) { Fail("phase '" + phase.name + "' needs the latent SPATIAL x2 upsampler, but the checkpoint at 'upsampler_path' " "declares temporal_upsample=true, i.e. it is the TEMPORAL x2 upsampler. That arm " diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index b498dc1c4..9c7fbea01 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -258,6 +258,13 @@ target_compile_definitions(test_video_engine PRIVATE # LTX-2.5 L7: the family behind the VideoEngine seam and the driving loop, run # over a reduced-dimension checkpoint set written in the SHIPPED file format. vllm_cpp_add_test(test_ltx2_video vllm/multimodal/test_ltx2_video.cpp) +# The reader-anchor claim this suite gates is a claim ABOUT a source file, so it +# reads that file. Recorded `file:line` anchors go stale within the same PR here +# (a merge of origin/main moved all nine of these by 25 lines mid-review), which +# is why the anchors are derived from the source and compared with the recorded +# ones rather than trusted. +target_compile_definitions(test_ltx2_video PRIVATE + LTX2_VIDEO_SOURCE_PATH="${CMAKE_SOURCE_DIR}/src/vllm/multimodal/ltx2_video.cpp") # LTX-2.5 image conditioning (row LTX25-IMAGE-COND, issue #644): the CHAIN from # PPM bytes to conditioned, noised tokens, gated against upstream ltx_core AND # ltx_pipelines executed at reduced dimensions. ltx2_image_cond_goldens.inc @@ -294,6 +301,23 @@ target_include_directories(test_ltx2_text_encoder PRIVATE ${CMAKE_CURRENT_SOURCE # next to the test source. vllm_cpp_add_test(test_ltx2_pipeline vllm/models/test_ltx2_pipeline.cpp) target_include_directories(test_ltx2_pipeline PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}/vllm/models) +# The out-of-scope ledger separates a REACHABLE refusal from a declared marker, and +# docs/FEATURES.md is where that distinction reaches a user. It had re-merged the two +# into one "Declared, not requestable" row, so the public surface contradicted the +# ledger it projects; the case below reads the doc and holds it to the split. +# The ledger's RETIREMENT note carries an absence claim about upstream, and it +# shipped false twice in the same header. The case that guards it reads the header +# itself rather than a copy of the sentence kept in the test. +# The ledger's reachable/marker split is a claim about the PRODUCT TREE — that +# something calls the function holding a reachable refusal, and that nothing calls +# the one holding a marker. It was wrong about `kBetaScheduler` for the whole of +# this row because it was maintained by hand, so the case now walks src/, include/ +# and examples/ under this root and derives it, with two positive controls in the +# same walk (an absence claim about our own tree is still an absence claim). +target_compile_definitions(test_ltx2_pipeline PRIVATE + VLLM_CPP_FEATURES_DOC_PATH="${CMAKE_SOURCE_DIR}/docs/FEATURES.md" + LTX2_PIPELINE_HEADER_PATH="${CMAKE_SOURCE_DIR}/include/vllm/model_executor/models/ltx2_pipeline.h" + VLLM_CPP_SOURCE_ROOT="${CMAKE_SOURCE_DIR}") # LTX-2.5 L6: the quantized loaders, gated three ways — the SHIPPED checkpoints' # own manifests, a few hundred of their own bytes against torch, and synthetic # files for whole-model materialization. All three .inc files live beside the diff --git a/tests/vllm/models/test_ltx2_pipeline.cpp b/tests/vllm/models/test_ltx2_pipeline.cpp index b526653ad..b01b8fc3a 100644 --- a/tests/vllm/models/test_ltx2_pipeline.cpp +++ b/tests/vllm/models/test_ltx2_pipeline.cpp @@ -27,11 +27,15 @@ #include #include +#include #include #include #include +#include +#include #include #include +#include #include #include #include @@ -1231,26 +1235,593 @@ TEST_CASE("ltx2 the two references disagree on the default negative prompt, and // Out-of-scope refusals // =========================================================================== -TEST_CASE("ltx2 every L5 out-of-scope feature is refused BY NAME") { - // Spec section 2 "Out", plus the L7 boundary. A silent downgrade of any of - // these produces a video, which is exactly why none of them may fall back. - const std::vector> owed = { - {vllm::Ltx2UnportedPipelineFeature::kSpatiotemporalUpsampler, "temporal"}, +// THE LIST SHRANK FROM SEVEN TO FIVE on 2026-08-13, and the shrink is the point. +// Two enumerators were retired by row LTX25-RETIRE-DEAD-ARMS (#644): +// `kMultishot`, which refused a feature that exists in NEITHER reference, and +// `kVideoEngineWiring`, whose subject shipped in `cefacd2d0`. See +// .agents/specs/ltx25-retire-dead-arms.md §1.1 and §1.5. A CHANGED CASE COUNT +// here is that retirement, not a lost assertion. +TEST_CASE("ltx2 every out-of-scope feature is refused BY NAME") { + // Spec section 2 "Out". A silent downgrade of any of these produces a video, + // which is exactly why none of them may fall back. + // + // REACHABLE REFUSALS: a product path constructs the condition, so a caller can + // trip this. ONE of the five is, not two. The engine reaches + // `ltx2_upsampler.cpp:465` through `Ltx2UpsampleVideoLatent`, which + // `ltx2_video.cpp` calls when a phase asks for the spatial-upsample transform. + // + // `kBetaScheduler` USED TO BE LISTED HERE and is not reachable. Its call site + // `ltx2_pipeline.cpp:199` sits inside `Ltx2Schedule`, and `Ltx2Schedule` has no + // product caller at all — the engine calls `Ltx2SigmaSchedule` directly + // in `ltx2_video.cpp`'s phase driver — and no ABI field, load extra or CLI flag + // carries a scheduler kind. A refusal is only reachable if something CALLS the function + // holding it; an enumerator with a `case` label is not a caller. The case + // "ltx2 the reachable/marker split matches the source" below derives that + // reachability from the tree rather than restating it here. + const std::vector> reachable = { + {vllm::Ltx2UnportedPipelineFeature::kSpatiotemporalUpsampler, "SPATIOTEMPORAL"}, + }; + // DECLARED-OUT-OF-SCOPE MARKERS: nothing a caller can send reaches these, so + // the message must not claim otherwise. Recording them as refusals overstated + // what this port has, which is the defect this row closes. + const std::vector> markers = { + {vllm::Ltx2UnportedPipelineFeature::kBetaScheduler, "BetaScheduler"}, {vllm::Ltx2UnportedPipelineFeature::kLoraFusion, "LoRA"}, - {vllm::Ltx2UnportedPipelineFeature::kMultishot, "multishot"}, {vllm::Ltx2UnportedPipelineFeature::kInt8ConvRot, "int8-convrot"}, - {vllm::Ltx2UnportedPipelineFeature::kCfgParallelism, "parallelism"}, - {vllm::Ltx2UnportedPipelineFeature::kVideoEngineWiring, "VideoEngine"}, - {vllm::Ltx2UnportedPipelineFeature::kBetaScheduler, "BetaScheduler"}, + {vllm::Ltx2UnportedPipelineFeature::kMultiGpuParallelism, "multi-GPU"}, }; - for (const auto& item : owed) { + + std::vector> all = reachable; + all.insert(all.end(), markers.begin(), markers.end()); + for (const auto& item : all) { const std::string message = RefusalMessage([&] { vllm::Ltx2RefuseUnportedPipelineFeature(item.first); }); INFO("feature = ", item.second, " refusal = ", message); CHECK(Mentions(message, item.second)); // Naming WHERE the work is owed is what keeps it from being rediscovered. CHECK(Mentions(message, "ltx-2-5.md")); + // The RETIRED arm must not come back: a refusal that cites a feature neither + // reference has sends the next reader looking upstream for it. + CHECK_FALSE(Mentions(message, "multishot")); + } + for (const auto& item : reachable) { + const std::string message = + RefusalMessage([&] { vllm::Ltx2RefuseUnportedPipelineFeature(item.first); }); + INFO("reachable = ", item.second, " refusal = ", message); + CHECK_FALSE(Mentions(message, "DECLARED, NOT REQUESTABLE")); + } + for (const auto& item : markers) { + const std::string message = + RefusalMessage([&] { vllm::Ltx2RefuseUnportedPipelineFeature(item.first); }); + INFO("marker = ", item.second, " refusal = ", message); + CHECK(Mentions(message, "DECLARED, NOT REQUESTABLE")); + } + + // THE ABSENCE THIS MESSAGE STATES IS ITSELF EVIDENCE, so it is gated. The first + // version of this marker ended "int8 appears upstream only in the trainer" — a + // false-absence claim of exactly the kind row LTX25-RETIRE-DEAD-ARMS exists to + // retire (#604), shipped inside a user-visible refusal. LTX-2 @ fd4ded7f carries + // a per-row int8 quantize kernel with fp32 scales in `ltx-kernels`, which is an + // INFERENCE package, not the trainer: `blockwise/triton_ops.py:35,43`, aliased + // `rowwise_int_quantize_triton` at `:436`. It is dead — that alias is its only + // reference and `blockwise/functional.py:12-18` does not re-export it — so the + // disposition is unchanged and only the sentence was wrong. See + // .agents/specs/ltx25-retire-dead-arms.md §1.2. + { + const std::string message = RefusalMessage([] { + vllm::Ltx2RefuseUnportedPipelineFeature(vllm::Ltx2UnportedPipelineFeature::kInt8ConvRot); + }); + INFO("int8 marker = ", message); + CHECK_FALSE(Mentions(message, "only in the trainer")); + // The true statement names where the one inference-side int8 lives, so a + // reader who greps upstream and finds it is not left thinking we missed it. + CHECK(Mentions(message, "ltx-kernels")); + } + + // THE MULTI-GPU MARKER'S OWN EVIDENCE, for the same reason and after the same + // defect. This message shipped "Upstream has three forms and none of them is + // CFG batching", and the header above the enum shipped "zero `cfg` hits in + // either multigpu tree". Both are false at LTX-2 @ fd4ded7f, and both are #604 + // again: the spec's grep was correct but PATH-FILTERED to the two SOURCE trees, + // excluding `ltx-pipelines/docs/multigpu/` where the answer lives. Re-derived + // without the filter, with the file list as its own positive control: + // + // git ls-files -- '*multigpu*' -> 33 files (the control) + // git grep -n -i cfg -- '*multigpu*' -> 5 lines, NOT zero + // + // Two of the five are substantive, at `docs/multigpu/gemma.md:103-104`. And + // there is a FOURTH `BuilderProtocol` in the very directory the message cites: + // `multigpu/bp_gemma_builder.py:42` `BatchParallelGemmaBuilder`, wrapping + // `ltx-core multigpu/gemma/batch_parallel_wrapper.py`, which partitions a PROMPT + // LIST across ranks. + // + // The disposition is unchanged and in fact stronger: `gemma.md:104` says the + // distilled pipeline runs "without CFG", so the one form that would batch a + // CFG pair is the one upstream tells you not to use for the recipe this port + // runs. Only the sentences were wrong. See + // .agents/specs/ltx25-retire-dead-arms.md §1.3. + { + const std::string message = RefusalMessage([] { + vllm::Ltx2RefuseUnportedPipelineFeature( + vllm::Ltx2UnportedPipelineFeature::kMultiGpuParallelism); + }); + INFO("multi-GPU marker = ", message); + // The retired count. "three forms" was an undercount produced by a path + // filter, so the count itself is the assertion. + CHECK_FALSE(Mentions(message, "three forms")); + CHECK(Mentions(message, "four forms")); + // The form the undercount missed, by name, so a reader who greps the cited + // directory and finds a fourth builder is not left thinking we missed it. + CHECK(Mentions(message, "BatchParallelGemmaBuilder")); + // And the REASON CFG batching is inapplicable, cited to the upstream line + // that says it, rather than asserted as an absence of the string `cfg`. + CHECK(Mentions(message, "gemma.md")); + } +} + +// THE REACHABLE/MARKER SPLIT, DERIVED FROM THE TREE INSTEAD OF ASSERTED. +// +// The ledger above and `docs/USAGE.md` both make a claim about the PRODUCT CODE: +// that a render asking for a reachable arm gets a refusal, and that nothing a +// caller can send reaches a marker. Until this case existed, that claim was two +// hand-maintained vectors and a paragraph, and it was wrong about +// `kBetaScheduler` for the whole of this row: the header defined reachable as "a +// caller CAN trip it", `docs/USAGE.md` published "Both are reachable", and the +// refusal was inside `Ltx2Schedule`, which nothing calls. +// +// THE RULE THIS ENCODES. A refusal is reachable only if something CALLS the +// function that holds it. A `case` label in a switch is not a caller, and an +// enumerator that appears in `src/` proves only that the compiler can see it. +// So the check is on the ENTRY FUNCTION of each arm's chain: +// +// kSpatiotemporalUpsampler `Ltx2LatentUpsample` (ltx2_upsampler.cpp:465) +// <- `Ltx2UpsampleVideoLatent` (:566) +// <- `ltx2_video.cpp`, the phase that upsamples +// kBetaScheduler `Ltx2Schedule` (ltx2_pipeline.cpp:199) +// <- NOTHING +// +// TWO POSITIVE CONTROLS, because this case is an ABSENCE claim about our own tree +// and #604 is the row's whole subject. A scan that reports zero because it opened +// no files, or because it cannot match a symbol of this shape, reports the same +// zero as a genuine absence. So the same walk, with the same predicate and the +// same exclusions, must find `Ltx2UpsampleVideoLatent` called from product code +// (the reachable arm, proving the walk sees callers at all) and `Ltx2SigmaSchedule` +// called from product code (proving a SCHEDULER entry point in this same header +// is findable, so the zero for `Ltx2Schedule` is not an artifact of the name). +namespace { + +// Every product translation unit: `src/`, `include/` and `examples/`. Tests are +// deliberately excluded — a unit test constructing an enumerator by hand is +// exactly what a marker is, so counting it as a caller would erase the split. +std::vector ProductSources() { + std::vector out; + for (const char* dir : {"src", "include", "examples"}) { + const std::filesystem::path root = std::filesystem::path(VLLM_CPP_SOURCE_ROOT) / dir; + if (!std::filesystem::is_directory(root)) continue; + for (const std::filesystem::directory_entry& e : + std::filesystem::recursive_directory_iterator(root)) { + if (!e.is_regular_file()) continue; + const std::string ext = e.path().extension().string(); + if (ext == ".cpp" || ext == ".cc" || ext == ".h" || ext == ".hpp" || ext == ".cu") { + out.push_back(e.path()); + } + } + } + return out; +} + +// Product files mentioning `symbol(`, excluding the files that DECLARE and DEFINE +// it. Reported as `path:line` strings so a failure names the caller rather than a +// count the reader then has to go and find. +std::vector ProductCallSites(const std::vector& files, + const std::string& symbol, + const std::vector& owning_files) { + std::vector hits; + for (const std::filesystem::path& path : files) { + const std::string name = path.filename().string(); + if (std::find(owning_files.begin(), owning_files.end(), name) != owning_files.end()) continue; + std::ifstream in(path); + if (!in.good()) continue; + std::string line; + size_t line_no = 0; + while (std::getline(in, line)) { + ++line_no; + if (line.find(symbol + "(") != std::string::npos) { + hits.push_back(path.filename().string() + ":" + std::to_string(line_no)); + } + } + } + return hits; +} + +std::string Join(const std::vector& v) { + std::string s; + for (const std::string& x : v) s += (s.empty() ? "" : ", ") + x; + return s.empty() ? "" : s; +} + +} // namespace + +TEST_CASE("ltx2 the reachable/marker split matches the source") { + const std::vector files = ProductSources(); + // ANTI-VACUOUS, control zero: a walk that opened nothing reports every symbol + // as unreachable and passes the half of this case that matters least. + REQUIRE_MESSAGE(files.size() > 100, + "the product-source walk found only " << files.size() + << " files; VLLM_CPP_SOURCE_ROOT is wrong"); + + // ── CONTROL ONE: the reachable arm's entry function IS called ────────────── + const std::vector upsample_callers = + ProductCallSites(files, "Ltx2UpsampleVideoLatent", + {"ltx2_upsampler.cpp", "ltx2_upsampler.h", "ltx2_video_vae_encoder.h"}); + INFO("Ltx2UpsampleVideoLatent callers = " << Join(upsample_callers)); + CHECK_MESSAGE(!upsample_callers.empty(), + "the SPATIOTEMPORAL refusal is published as REACHABLE, but nothing in src/, " + "include/ or examples/ calls Ltx2UpsampleVideoLatent. Either the engine stopped " + "upsampling, or this walk is broken — resolve which before trusting the zero " + "below"); + + // ── CONTROL TWO: a scheduler entry point in the SAME header is findable ──── + const std::vector sigma_callers = + ProductCallSites(files, "Ltx2SigmaSchedule", {"ltx2_pipeline.cpp", "ltx2_pipeline.h"}); + INFO("Ltx2SigmaSchedule callers = " << Join(sigma_callers)); + CHECK_MESSAGE(!sigma_callers.empty(), + "no product caller of Ltx2SigmaSchedule either, so the walk cannot see scheduler " + "call sites and the Ltx2Schedule zero below proves nothing"); + + // ── THE CLAIM: the Beta refusal is reached by no product path ────────────── + const std::vector schedule_callers = + ProductCallSites(files, "Ltx2Schedule", {"ltx2_pipeline.cpp", "ltx2_pipeline.h"}); + INFO("Ltx2Schedule callers = " << Join(schedule_callers)); + + const std::string beta = RefusalMessage([] { + vllm::Ltx2RefuseUnportedPipelineFeature(vllm::Ltx2UnportedPipelineFeature::kBetaScheduler); + }); + INFO("BetaScheduler refusal = " << beta); + + // The two must AGREE, in both directions, which is what makes this a gate on + // the classification rather than a restatement of it. No caller means marker; + // a caller appearing later means the marker wording has to go. + if (schedule_callers.empty()) { + CHECK_MESSAGE(Mentions(beta, "DECLARED, NOT REQUESTABLE"), + "nothing in src/, include/ or examples/ calls Ltx2Schedule, so no request can " + "reach the BetaScheduler refusal, yet its message does not say DECLARED, NOT " + "REQUESTABLE. Either route the engine through Ltx2Schedule or classify it as a " + "marker, per .agents/specs/ltx25-retire-dead-arms.md §1.6"); + } else { + CHECK_MESSAGE(!Mentions(beta, "DECLARED, NOT REQUESTABLE"), + "Ltx2Schedule now HAS a product caller (" + << Join(schedule_callers) + << "), so the BetaScheduler refusal is reachable and must stop calling " + "itself unrequestable. Move it back to the reachable group in the ledger " + "case, docs/FEATURES.md and docs/USAGE.md"); + } +} + +// THE PUBLIC SURFACE MUST NOT RE-MERGE WHAT THE LEDGER SPLIT. `docs/FEATURES.md` +// is where the reachable/marker distinction reaches a user, and it carried ONE +// "Declared, not requestable" row listing all five arms — including the two a +// caller CAN trip. That is the same conflation the ledger above stopped making, +// reintroduced one surface out. Gated here rather than left to review because a +// doc row and an enum drift silently. +// +// Deliberately narrow: it asserts only that no row calling something +// unrequestable names a REACHABLE arm. It does not police the wording of the doc. +TEST_CASE("ltx2 docs/FEATURES.md never calls a REACHABLE refusal unrequestable") { + std::ifstream in(VLLM_CPP_FEATURES_DOC_PATH); + REQUIRE_MESSAGE(in.good(), "cannot open " << VLLM_CPP_FEATURES_DOC_PATH); + std::stringstream buf; + buf << in.rdbuf(); + const std::string doc = buf.str(); + REQUIRE(doc.size() > 1000); + + // The ONE arm with a product call site: `ltx2_upsampler.cpp:465` constructs the + // SPATIOTEMPORAL upsampler condition, and `ltx2_video.cpp` reaches it through + // `Ltx2UpsampleVideoLatent`. Named by the word the doc uses for it, since that + // is what a reader sees. + // + // `betascheduler` WAS IN THIS LIST and had to come out. It is a marker, not a + // reachable refusal — `Ltx2Schedule`, the only function holding its call site, + // has no product caller — so the doc's markers row now names it, and a check + // that no "not requestable" row may name it would fire on the correct sentence. + // Kept as a comment rather than deleted because the wrong classification is what + // this repair fixes; see the ledger case above and §1.6 of the row spec. + // + // `spatiotemporal upsampler`, NOT a bare `upsampler`, and that is the whole + // repair. `2e9d95e74` ported the TEMPORAL-ONLY x2 upsampler on this same issue + // and renamed the enumerator, so there are now two upsampler arms in this + // model: one refused, one shipped-but-undriven. A bare `upsampler` matches both + // and therefore cannot tell a refusal from a shipped feature — it fires on a + // correct sentence about the ported arm and stays silent on the distinction it + // exists to police. Matched case-insensitively because the doc capitalizes the + // word for emphasis in one row and not the other. + const std::vector reachable_words = {"spatiotemporal upsampler"}; + + auto lowered = [](const std::string& text) { + std::string out = text; + for (char& c : out) c = static_cast(std::tolower(static_cast(c))); + return out; + }; + + size_t rows_examined = 0; + size_t reachable_rows = 0; + size_t line_no = 0; + size_t at = 0; + while (at <= doc.size()) { + const size_t end = doc.find('\n', at); + const std::string line = doc.substr(at, end == std::string::npos ? std::string::npos : end - at); + ++line_no; + const std::string lower = lowered(line); + // A table row that makes the not-requestable claim, in either casing the doc + // uses for it. + const bool is_row = !line.empty() && line[0] == '|'; + const bool claims_unrequestable = lower.find("not requestable") != std::string::npos; + const bool is_ltx = line.find("LTX-2.5") != std::string::npos; + if (is_row && claims_unrequestable && is_ltx) { + ++rows_examined; + for (const std::string& word : reachable_words) { + INFO("FEATURES.md:" << line_no << " = " << line); + CHECK_MESSAGE(lower.find(word) == std::string::npos, + "a row claiming 'not requestable' names the REACHABLE arm '" + << word + << "'; split the reachable refusals out, per " + ".agents/specs/ltx25-retire-dead-arms.md §1.6"); + } + } + // AND THE REACHABLE ROW ITSELF, because the defect this repair closes was + // produced by a MERGE rather than by an author: `2e9d95e74` landed the + // temporal-only arm while this row's doc edit still called that arm refused, + // and both files auto-merged clean, so nothing said a word. Publishing a + // refusal for a shipped feature is #604 in the most user-facing surface there + // is, which is why the doc must name the arm that is ACTUALLY refused. + if (is_row && is_ltx && lower.find("refused by name at the call site") != std::string::npos) { + ++reachable_rows; + INFO("FEATURES.md:" << line_no << " = " << line); + CHECK_MESSAGE(lower.find("spatiotemporal") != std::string::npos, + "the LTX-2.5 reachable-refusal row must name the SPATIOTEMPORAL upsampler. " + "The temporal-only x2 arm is PORTED (2e9d95e74); a row that calls it " + "refused publishes a refusal for a shipped feature"); + } + if (end == std::string::npos) break; + at = end + 1; + } + // ANTI-VACUOUS. Without these the case passes when a row is renamed away and + // proves nothing — the failure mode this whole row is about. + CHECK_MESSAGE(rows_examined == 1, + "expected exactly ONE LTX-2.5 'not requestable' row in docs/FEATURES.md, found " + << rows_examined); + CHECK_MESSAGE(reachable_rows == 1, + "expected exactly ONE LTX-2.5 'refused by name at the call site' row in " + "docs/FEATURES.md, found " + << reachable_rows); +} + +// THE RETIREMENT NOTE'S OWN EVIDENCE, held to what upstream actually says. +// +// Retiring `kMultishot` rests on an ABSENCE claim about upstream, and the header +// above the enum is where a porter reads it. That sentence shipped wrong: it said +// the only `scene` hit upstream was PySceneDetect in the TRAINER. At +// Lightricks/LTX-2 @ fd4ded7f `scene` has THREE senses, and the third — +// prompt-writing guidance — lives in `ltx-core`, which ships at INFERENCE. So a +// porter greps `scene`, finds "scene cuts" in a shipped prompt-enhancer prompt, +// and concludes we missed a multi-shot path that our own header told them did not +// exist. Third instance of #604 inside the row whose subject is retiring #604. +// +// The disposition did not move — it got STRONGER. Those prompts instruct the +// enhancer NOT to describe scene cuts and to keep a "Single continuous take" +// (gemma3_i2v:18, gemma3_t2v:24, gemma4_i2v:3), which is affirmative evidence +// that no multi-shot generation mode exists. +// +// WHAT THIS CASE CAN AND CANNOT PROVE. It reads the shipped header and holds its +// text, so the false sentence cannot come back and the true evidence cannot be +// dropped. It CANNOT verify the upstream claim — no upstream checkout exists in +// this tree — so the derivation, with the positive control in the same command, +// lives in .agents/specs/ltx25-retire-dead-arms.md §1.1. +TEST_CASE("ltx2 the kMultishot retirement note states the scene evidence correctly") { + std::ifstream in(LTX2_PIPELINE_HEADER_PATH); + REQUIRE_MESSAGE(in.good(), "cannot open " << LTX2_PIPELINE_HEADER_PATH); + std::stringstream buf; + buf << in.rdbuf(); + const std::string header = buf.str(); + REQUIRE(header.size() > 1000); + + // The note is prose wrapped across comment lines, so every claim below is + // matched against a flattened copy: comment markers dropped, runs of whitespace + // collapsed to one space. Without this a reflow of the paragraph would silently + // turn every assertion vacuous. + std::string flat; + flat.reserve(header.size()); + bool pending_space = false; + for (size_t i = 0; i < header.size(); ++i) { + const char c = header[i]; + if (c == '/' && i + 1 < header.size() && header[i + 1] == '/') { + i += 1; + pending_space = true; + continue; + } + if (std::isspace(static_cast(c)) != 0) { + pending_space = true; + continue; + } + if (pending_space && !flat.empty()) flat += ' '; + pending_space = false; + flat += c; + } + + // ANTI-VACUOUS. Every assertion below is about ONE paragraph; if that paragraph + // is renamed or removed they all pass while proving nothing. + size_t notes = 0; + for (size_t at = flat.find("`kMultishot` — FABRICATED"); at != std::string::npos; + at = flat.find("`kMultishot` — FABRICATED", at + 1)) { + ++notes; + } + REQUIRE_MESSAGE(notes == 1, + "expected exactly ONE `kMultishot` — FABRICATED retirement note in " + << LTX2_PIPELINE_HEADER_PATH << ", found " << notes); + + // The sentence that was false. It asserted an upstream absence from our own + // vocabulary, with no positive control, in a SHIPPED header. + CHECK_MESSAGE(flat.find("the only `scene` hit is PySceneDetect") == std::string::npos, + "the retired false claim is back: `scene` is not trainer-only upstream — " + "ltx-core's shipped gemma prompt files carry it, see " + ".agents/specs/ltx25-retire-dead-arms.md §1.1"); + CHECK(flat.find("only `scene` hit") == std::string::npos); + + // SCOPE THE POSITIVE CHECKS TO THE NOTE. Searching the whole flattened header + // for a positive phrase is what made the `ltx-core` assertion below structurally + // unable to fail: `ltx-core` occurs SIX times in this header — the upstream map + // at :8 and :28, `parse_model_version` at :452, `kLoraFusion` at :629, and twice + // inside the note — so a file-wide `find` survived deleting the clause it exists + // to hold. Proved by mutation: rewriting the note to say the guidance ships + // "inside the TRAINER ONLY" left the case GREEN at 8/8. It also never went red + // in this row's own red-first run (4 of 8 failed; this was not one of them). + // + // The paragraph runs from the retirement marker to the NEXT enumerator heading, + // derived rather than pinned to a line number, because a recorded anchor in this + // header went stale inside this very pull request. + const size_t note_at = flat.find("`kMultishot` — FABRICATED"); + REQUIRE(note_at != std::string::npos); + const std::string heading_tail = "` — "; + size_t note_end = std::string::npos; + for (size_t at = flat.find("`k", note_at + 1); at != std::string::npos; + at = flat.find("`k", at + 1)) { + const size_t close = flat.find('`', at + 1); + if (close == std::string::npos) break; + if (flat.compare(close, heading_tail.size(), heading_tail) == 0) { + note_end = at; + break; + } + } + if (note_end == std::string::npos) note_end = flat.size(); + const std::string note = flat.substr(note_at, note_end - note_at); + // ANTI-VACUOUS, again: an empty or truncated slice would pass a negative check + // and fail a positive one for the wrong reason. The note is a long paragraph. + REQUIRE_MESSAGE(note.size() > 400, + "the `kMultishot` retirement note sliced to " << note.size() + << " chars; the slice is wrong"); + + // And the evidence that replaced it, by the three things a porter needs: that + // the narrative sense is PROMPT GUIDANCE, that it ships in `ltx-core` rather + // than only the trainer, and that the guidance FORBIDS scene cuts. + // + // The middle one is asserted as the CLAIM, not as the package name. The axis it + // holds — the guidance ships at inference, it is not trainer-only — is exactly + // the axis that shipped false for two review rounds, and a bare `ltx-core` does + // not hold it: neither sibling below covers it either, since both survive the + // trainer-only mutation. + CHECK_MESSAGE(note.find("scene cuts") != std::string::npos, + "the retirement note must name the prompt guidance it now rests on"); + CHECK_MESSAGE(note.find("ships at INFERENCE inside `ltx-core`") != std::string::npos, + "the note must state WHERE the third `scene` sense ships. Trainer-only is what " + "it wrongly said for two rounds, and `ltx-core` shipping at inference is the " + "whole reason the retirement holds — see " + ".agents/specs/ltx25-retire-dead-arms.md §1.1"); + CHECK(note.find("system_prompt") != std::string::npos); +} + +// THE MULTI-GPU MARKER'S NOTE, held to what upstream actually contains. +// +// Same shape as the case above and the same defect, found in the same review. The +// header shipped "NOT CFG batching: zero `cfg` hits in either multigpu tree". At +// LTX-2 @ fd4ded7f that is FIVE hits, not zero, and two of them are substantive +// prose about CFG. The spec's grep was right and its PATH FILTER was wrong: it +// covered `ltx-pipelines/src/.../multigpu/` and `ltx-core/src/.../multigpu/` and +// therefore excluded `ltx-pipelines/docs/multigpu/`, where the answer is written +// out. That is this row's own transferable lesson — a path filter is an absence +// claim too — committed by the row a fourth time. +// +// The header is where a porter reads the claim, so the header is what this holds. +// The derivation, with `git ls-files -- '*multigpu*'` as its positive control, +// is .agents/specs/ltx25-retire-dead-arms.md §1.3. +// +// WHAT THIS CAN AND CANNOT PROVE, stated for the same reason as above: it gates +// the TEXT, so the false sentence cannot return and the correction cannot be +// dropped. It cannot verify the upstream fact — there is no upstream checkout in +// this tree — and pretending otherwise is worse than saying so. +namespace { + +// Comment markers dropped, whitespace runs collapsed. A reflow of the paragraph +// must not turn an assertion vacuous, which is why nothing below matches raw +// header text. +std::string FlattenHeaderComments(const std::string& header) { + std::string flat; + flat.reserve(header.size()); + bool pending_space = false; + for (size_t i = 0; i < header.size(); ++i) { + const char c = header[i]; + if (c == '/' && i + 1 < header.size() && header[i + 1] == '/') { + i += 1; + pending_space = true; + continue; + } + if (std::isspace(static_cast(c)) != 0) { + pending_space = true; + continue; + } + if (pending_space && !flat.empty()) flat += ' '; + pending_space = false; + flat += c; + } + return flat; +} + +} // namespace + +TEST_CASE("ltx2 the multi-GPU marker note states the CFG evidence correctly") { + std::ifstream in(LTX2_PIPELINE_HEADER_PATH); + REQUIRE_MESSAGE(in.good(), "cannot open " << LTX2_PIPELINE_HEADER_PATH); + std::stringstream buf; + buf << in.rdbuf(); + const std::string header = buf.str(); + REQUIRE(header.size() > 1000); + const std::string flat = FlattenHeaderComments(header); + + // ANTI-VACUOUS: every assertion below is about the ONE enumerator comment. If + // it is renamed or deleted they all pass while proving nothing. + size_t notes = 0; + for (size_t at = flat.find("kMultiGpuParallelism,"); at != std::string::npos; + at = flat.find("kMultiGpuParallelism,", at + 1)) { + ++notes; } + REQUIRE_MESSAGE(notes == 1, "expected exactly ONE `kMultiGpuParallelism` enumerator in " + << LTX2_PIPELINE_HEADER_PATH << ", found " << notes); + + // The sentence that was false, in the two spellings it could come back as. An + // absence asserted from our own vocabulary, with no positive control, in a + // SHIPPED header — #604, in the row that exists to retire #604. + CHECK_MESSAGE(flat.find("zero `cfg` hits") == std::string::npos, + "the retired false claim is back: `cfg` is 5 hits across the multigpu trees at " + "LTX-2 @ fd4ded7f, two of them substantive prose in docs/multigpu/gemma.md. See " + ".agents/specs/ltx25-retire-dead-arms.md §1.3"); + CHECK(flat.find("zero cfg hits") == std::string::npos); + + // SCOPE THE POSITIVE CHECKS TO THE NOTE. A file-wide `find` for a word like + // `four` is structurally unable to fail here: `kInt8ConvRot`'s own comment + // already says "the four inference kinds upstream defines". The note runs from + // its enumerator to the end of the enum, derived rather than pinned to a line + // number, because a recorded anchor in this header went stale mid-review. + const size_t note_at = flat.find("kMultiGpuParallelism,"); + REQUIRE(note_at != std::string::npos); + size_t note_end = flat.find("};", note_at); + if (note_end == std::string::npos) note_end = flat.size(); + const std::string note = flat.substr(note_at, note_end - note_at); + // ANTI-VACUOUS, again: a truncated slice passes every negative check and fails + // every positive one for the wrong reason. + REQUIRE_MESSAGE(note.size() > 200, "the kMultiGpuParallelism note sliced to " + << note.size() << " chars; the slice is wrong"); + + // And the evidence that replaced it: the corrected count, the form the path + // filter hid, and the upstream line that gives the REASON rather than an + // absence of a string. + CHECK_MESSAGE(note.find("four") != std::string::npos, + "the marker must state the corrected COUNT; three was an undercount produced by " + "a path filter, and the count is the part that was wrong"); + CHECK_MESSAGE(note.find("BatchParallelGemmaBuilder") != std::string::npos, + "the marker must name the FOURTH multigpu form, which sits in the very directory " + "the message cites (multigpu/bp_gemma_builder.py:42)"); + CHECK_MESSAGE(note.find("gemma.md") != std::string::npos, + "the marker must cite the upstream line stating the distilled pipeline runs " + "without CFG, which is why CFG batching is inapplicable here"); } // =========================================================================== diff --git a/tests/vllm/multimodal/test_ltx2_video.cpp b/tests/vllm/multimodal/test_ltx2_video.cpp index f5dd645c5..60de0d69d 100644 --- a/tests/vllm/multimodal/test_ltx2_video.cpp +++ b/tests/vllm/multimodal/test_ltx2_video.cpp @@ -25,8 +25,10 @@ #include #include #include +#include #include #include +#include #include #include #include @@ -449,6 +451,50 @@ TEST_CASE("ltx2 video: the second phase upsamples, and refuses when it cannot") CHECK(msg.find("the upsampled latent is") == std::string::npos); } } + // THE ARM THE GUARD ABOVE WAS SHADOWING, and the reason this subcase exists at + // all. `if (im.upsampler_cfg.temporal_upsample)` is satisfied by a BOTH-flags + // checkpoint as well as a temporal-only one, so a genuine SPATIOTEMPORAL + // checkpoint was told it is the temporal x2 upsampler and pointed at the spatial + // one. Wrong on both counts: it is neither, it is the third arm, and the ledger + // refusal that names it (`ltx2_upsampler.cpp:465`) sat behind a guard that could + // not be reached from a request. + // + // The defect is an IMPLICATION between two guards over one variable, which no + // fixture could see because nothing drove a both-flags config through + // `LoadVideoEngine` — a review could prove the ledger refusal unmutated but not + // separate "unreachable" from "untested". This subcase closes that: it is the + // both-flags checkpoint, driven through the product path, asserting the caller + // is told which arm they actually supplied. + SUBCASE("a SPATIOTEMPORAL upsampler checkpoint is refused as SPATIOTEMPORAL, not as temporal") { + vllm::Ltx2UpsamplerConfig spatiotemporal = + ltx2_fixture::ReducedUpsamplerConfig(ltx2_fixture::ReducedDitParams().in_channels); + spatiotemporal.spatial_upsample = true; + spatiotemporal.temporal_upsample = true; + const std::string path = ws.root + "/spatiotemporal_upsampler.safetensors"; + ltx2_fixture::WriteReducedUpsampler(spatiotemporal, path); + + vllm::multimodal::VideoModelParams mp = FixtureParams(ws.paths); + mp.extras["upsampler_path"] = path; + const std::unique_ptr engine = + vllm::multimodal::LoadVideoEngine(mp); + try { + (void)engine->Generate(FixtureGen(ws.root + "/spatiotemporal_ups")); + FAIL("a spatiotemporal upsampler checkpoint must be refused by name"); + } catch (const std::exception& e) { + const std::string msg = e.what(); + INFO(msg); + // The arm they ACTUALLY supplied, which is the whole repair. + CHECK(msg.find("SPATIOTEMPORAL") != std::string::npos); + // ...and NOT the temporal-only diagnosis, which is what the shadowing guard + // produced. Asserted on the sentence that only that guard emits, because + // both messages legitimately contain the word `temporal`. + CHECK(msg.find("it is the TEMPORAL x2 upsampler") == std::string::npos); + CHECK(msg.find("Supply the spatial upsampler") == std::string::npos); + // Not a shape complaint either: the refusal has to land before any weight + // is touched, which is what the ledger arm promises. + CHECK(msg.find("the upsampled latent is") == std::string::npos); + } + } SUBCASE("with one, the render lands at the FULL requested size") { vllm::multimodal::VideoModelParams mp = FixtureParams(ws.paths); mp.extras["upsampler_path"] = ws.paths.upsampler; @@ -598,6 +644,266 @@ TEST_CASE("ltx2 video: an unknown extra is refused, not ignored") { } } +// A key this family DEFINES but does not serve is the worse half of the same +// defect, and the one an "unknown extra" check cannot see. `duration_head_path` +// was in `kKnownLoadExtras` and read by NOTHING (#611): supplying a duration head +// loaded no head, opened no file, and handed back the recipe default with no +// diagnostic. AGENTS.md requires an unimplemented arm to be refused with a +// message naming the missing piece, so it is refused rather than accepted. +// +// Dropping the key from `kKnownLoadExtras` instead would produce "unknown load +// extra", which is a DIFFERENT and wrong claim — the family defines the key and +// understands what it means; what is missing is the head. Hence the assertion on +// the missing piece and the alternative, not only on the key. +TEST_CASE("ltx2 video: duration_head_path is REFUSED by name, not silently ignored") { + Workspace ws; + vllm::multimodal::VideoModelParams mp = FixtureParams(ws.paths); + // Any path at all: the point is that NOTHING opens it. Naming a file that does + // exist keeps a not-found error from standing in for the refusal. + mp.extras["duration_head_path"] = ws.paths.dit; + try { + (void)vllm::multimodal::LoadVideoEngine(mp); + FAIL("duration_head_path is served by no code; accepting it substitutes the recipe default"); + } catch (const std::exception& e) { + const std::string msg = e.what(); + INFO(msg); + CHECK(msg.find("duration_head_path") != std::string::npos); + // The MISSING PIECE, which is what separates this from "unknown key". + CHECK(msg.find("duration head") != std::string::npos); + // And what to use instead, so the refusal is actionable. + CHECK(msg.find("num_frames") != std::string::npos); + // Not the unknown-key message: that one would say the family does not define + // it, and this family does. + CHECK(msg.find("unknown load extra") == std::string::npos); + } +} + +// The INVENTORY, so the defect above cannot come back as a different key. Every +// extra this family accepts is either read by something or refused by name; a +// tenth decorative key fails this case rather than waiting to be discovered by +// the caller who supplies it. +// +// The audit behind it is in .agents/specs/ltx25-retire-dead-arms.md §2.1: nine of +// the ten keys have a reader and `duration_head_path` was the only one with none. +// The reader LINES are deliberately not repeated here — they moved twice while +// this row was in review. They live in one place, the READER ANCHORS comment in +// `ltx2_video.cpp`, and the case below derives them and holds that comment to it. +TEST_CASE("ltx2 video: every accepted load extra is READ by something") { + Workspace ws; + // The keys with a reader. + const std::vector served = { + vllm::multimodal::kLtx2AudioPromptEmbedsExtra, vllm::multimodal::kLtx2PipelineKindExtra, + vllm::multimodal::kLtx2ModelVersionExtra, vllm::multimodal::kLtx2AllowUnportedExtra, + vllm::multimodal::kLtx2MaxPhaseExtra, vllm::multimodal::kLtx2DitConfigPathExtra, + vllm::multimodal::kLtx2PromptValidRowsExtra, vllm::multimodal::kLtx2EncoderConfigPathExtra, + "upsampler_path", + }; + // The keys the family defines and does NOT serve. Growing this list is a + // deliberate act; growing it silently is the defect #611 records. + const std::vector refused = {"duration_head_path"}; + + // THE HANDLE ON THE REAL ARRAY. The unknown-extra refusal builds its listing + // from `kKnownLoadExtras` itself, so parsing that listing gates the ACTUAL + // accepted set rather than a copy of it maintained here. Without this the two + // vectors above would be true by construction and would gate nothing. + std::string listing; + { + vllm::multimodal::VideoModelParams mp = FixtureParams(ws.paths); + mp.extras["definitely_not_a_key"] = "1"; + try { + (void)vllm::multimodal::LoadVideoEngine(mp); + FAIL("an unknown extra must be refused"); + } catch (const std::exception& e) { + const std::string msg = e.what(); + const size_t at = msg.find("This family defines: "); + REQUIRE(at != std::string::npos); + listing = msg.substr(at + std::string("This family defines: ").size()); + } + } + INFO("listing = " << listing); + // Every name this row inventoried is still accepted... + for (const std::string& key : served) CHECK(listing.find(key) != std::string::npos); + for (const std::string& key : refused) CHECK(listing.find(key) != std::string::npos); + // ...and there is no ELEVENTH name that this inventory has never seen. The + // separator is ", ", so the count is one more than the separators. + size_t names = 1; + for (size_t at = listing.find(", "); at != std::string::npos; at = listing.find(", ", at + 2)) { + ++names; + } + CHECK_MESSAGE(names == served.size() + refused.size(), + "kKnownLoadExtras grew; add the key to `served` (with its reader) or to " + "`refused` (with a by-name refusal), per .agents/specs/ltx25-retire-dead-arms.md"); + + // And the unserved half is refused rather than accepted. + for (const std::string& key : refused) { + vllm::multimodal::VideoModelParams mp = FixtureParams(ws.paths); + mp.extras[key] = ws.paths.dit; + INFO("key = " << key); + try { + (void)vllm::multimodal::LoadVideoEngine(mp); + FAIL("an accepted-but-unread extra must be refused by name"); + } catch (const std::exception& e) { + const std::string msg = e.what(); + INFO(msg); + CHECK(msg.find(key) != std::string::npos); + CHECK(msg.find("unknown load extra") == std::string::npos); + } + } +} + +// THE ANCHORS FOR THAT INVENTORY, DERIVED RATHER THAN TRUSTED. +// +// The case above proves each key is accepted or refused; it cannot prove WHERE a +// key is read, and "nine of ten reach a reader at these lines" is the claim this +// row rests on. That claim shipped wrong: the recorded anchors were nine lines +// that named no reader at all, in the very file they were recorded in. Then, in +// review, a merge of `origin/main` moved all nine again. A `file:line` written by +// hand is stale by the next commit, so this derives them from the source and +// holds the recorded list to what it finds. When it fails it prints the answer. +// +// It does NOT assert absolute line numbers of its own — nothing here to rot. The +// only obligation it creates is on whoever moves a reader: update the one comment +// block in the same file they are already editing. +namespace { + +std::string ReadSourceFile(const char* path) { + std::ifstream in(path); + REQUIRE_MESSAGE(in.good(), "cannot open " << path); + std::stringstream buf; + buf << in.rdbuf(); + return buf.str(); +} + +std::vector SplitLines(const std::string& text) { + std::vector lines; + size_t at = 0; + while (at <= text.size()) { + const size_t end = text.find('\n', at); + lines.push_back(text.substr(at, end == std::string::npos ? std::string::npos : end - at)); + if (end == std::string::npos) break; + at = end + 1; + } + return lines; +} + +// 1-based index of the ONLY line containing `needle`, or 0. Uniqueness is the +// point: an anchor that matches twice anchors nothing, and existence alone is +// what let the stale numbers survive. +size_t UniqueLineWith(const std::vector& lines, const std::string& needle) { + size_t found = 0; + size_t count = 0; + for (size_t i = 0; i < lines.size(); ++i) { + if (lines[i].find(needle) != std::string::npos) { + ++count; + found = i + 1; + } + } + return count == 1 ? found : 0; +} + +std::string JoinNumbers(const std::vector& v) { + std::string s; + for (size_t n : v) s += (s.empty() ? "" : " ") + std::to_string(n); + return s; +} + +} // namespace + +TEST_CASE("ltx2 video: the recorded reader anchors are the ones in the source") { + const std::string source = ReadSourceFile(LTX2_VIDEO_SOURCE_PATH); + const std::vector lines = SplitLines(source); + REQUIRE(lines.size() > 500); + + // Everything is measured relative to the accepted-keys array, so a stray + // mention in the file header cannot be mistaken for a reader. + const size_t array_line = UniqueLineWith(lines, "const char* const kKnownLoadExtras[] = {"); + REQUIRE_MESSAGE(array_line != 0, "kKnownLoadExtras[] declaration is not unique in the source"); + size_t array_end = 0; + for (size_t i = array_line; i < lines.size(); ++i) { + if (lines[i] == "};") { + array_end = i + 1; + break; + } + } + REQUIRE(array_end > array_line); + + // The nine SERVED keys, by the token each is spelled with in the source. Order + // is irrelevant — the comparison is on the sorted set — so this list is not a + // second place the anchors live. + const std::vector served_tokens = { + "kLtx2AudioPromptEmbedsExtra", "kLtx2PipelineKindExtra", "kLtx2ModelVersionExtra", + "kLtx2AllowUnportedExtra", "kLtx2MaxPhaseExtra", "kLtx2DitConfigPathExtra", + "kLtx2PromptValidRowsExtra", "kLtx2EncoderConfigPathExtra", "\"upsampler_path\"", + }; + std::vector derived; + for (const std::string& token : served_tokens) { + size_t first = 0; + for (size_t i = array_end; i < lines.size(); ++i) { + if (lines[i].find(token) != std::string::npos) { + first = i + 1; + break; + } + } + INFO("token = " << token); + CHECK_MESSAGE(first != 0, "no reader found after kKnownLoadExtras for " << token); + if (first != 0) derived.push_back(first); + } + REQUIRE(derived.size() == served_tokens.size()); + std::sort(derived.begin(), derived.end()); + + // The RECORDED list, parsed out of the one comment line that carries it. + const size_t marker = UniqueLineWith(lines, "READER ANCHORS (derived and gated by"); + REQUIRE_MESSAGE(marker != 0, + "the READER ANCHORS comment marker is missing or not unique in the source"); + const std::string recorded_line = lines[marker]; // the line AFTER the marker (1-based) + std::vector recorded; + for (size_t i = 0; i < recorded_line.size();) { + if (std::isdigit(static_cast(recorded_line[i]))) { + size_t j = i; + while (j < recorded_line.size() && std::isdigit(static_cast(recorded_line[j]))) { + ++j; + } + recorded.push_back(static_cast(std::stoul(recorded_line.substr(i, j - i)))); + i = j; + } else { + ++i; + } + } + std::sort(recorded.begin(), recorded.end()); + CHECK_MESSAGE(recorded == derived, "the reader anchors recorded in ltx2_video.cpp are STALE. " + "Recorded: [" + << JoinNumbers(recorded) << "]. Actual: [" + << JoinNumbers(derived) + << "]. Paste the actual list into the READER ANCHORS " + "comment; the spec's §2.1 table is dated and stays."); + + // And the UNSERVED key is touched only by the refusal, never by a reader. This + // is the half a "has a reader" sweep cannot express. + const size_t refuse_line = UniqueLineWith(lines, "void CheckUnservedExtras("); + REQUIRE(refuse_line != 0); + size_t refuse_end = 0; + for (size_t i = refuse_line; i < lines.size(); ++i) { + if (lines[i] == "}") { + refuse_end = i + 1; + break; + } + } + REQUIRE(refuse_end > refuse_line); + size_t duration_hits = 0; + for (size_t i = array_end; i < lines.size(); ++i) { + if (lines[i].find("kLtx2DurationHeadPathExtra") == std::string::npos) continue; + ++duration_hits; + const size_t at = i + 1; + const bool inside_refusal = (at >= refuse_line) && (at <= refuse_end); + CHECK_MESSAGE(inside_refusal, + "ltx2_video.cpp:" << at + << " touches the duration-head extra OUTSIDE " + "CheckUnservedExtras; if it now has a real reader, move " + "it to the served list and drop the refusal (#611)"); + } + CHECK(duration_hits > 0); +} + // ─── the config the SHAPES cannot see ─────────────────────────────────────── // // WHY THESE ASSERT ON THE ENGINE AND NOT ON A LOCAL. The L7 repair is one line in