merge: L10 into the campaign -- the last row, and 8 wrong anchors it swept out

Reviewed PASS at 131328e after three FAIL rounds. The last one is the reason
to read this: told to fix ONE wrong file:line, the implementer swept every
citation the branch adds -- 19 of them -- and found EIGHT wrong at six sites.
Fixing the instance would have left seven.

Three of the eight are one anchor repeated. modeling_rope_utils.py:187-245 was
cited three times as the authority for that function's ZERO PADDING, which is at
:246 -- one line past the cited end, so the range stops inside the torch.cat(
argument list immediately before the argument that IS the whole claim. Our own
gemma4.cpp:300-312 has the identical shape: it closes on `} else {`, leaving the
vt::CastF32 pair at :313-314 that the sentence charges outside the range. A range
that stops just short of its own subject is worse than a missing citation,
because it looks checked.

The reviewer then re-extracted the citations independently and confirmed the
sweep missed none, resolved all eight corrections itself at the pins, verified
the transformers files are md5-identical between the venv and the clone so the
numbering is not install-specific, and proved code identity by comment-stripping
and hashing rather than by eye.

Three conflicts, none of them a union.

docs/USAGE.md's first: the base says "A typed prompt works" because L13 landed
it; L10 still opened "There is no prompt", stale on arrival. Took the base's
framing but KEPT L10's tokenization-divergence paragraph, which is unique to it
and still true -- upstream tokenizes through __call__ with add_special_tokens
defaulting True and so runs the post_processor, while this port calls plain
encode and prepends BOS; identical only because the shipped post_processor's
special-token map is EMPTY, measured on the file rather than assumed.

Its second and docs/FEATURES.md both took the base wholesale: the base's parity
paragraph is a strict superset carrying anchors, the ltx2.h refusal list and a
provenance note, and L10's row still said first-party NVFP4 does not load (L9A
landed it) and described an encoder_path refusal L13 lifted.

Issue: #435, #604

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [Claude Code]

Author: mudler

docs/FEATURES.md

@@ -147,7 +147,13 @@ speed-pending, which [BENCHMARKS.md](BENCHMARKS.md) tracks.
 ### Standalone and non-registered lanes
 
 These run through dedicated forwards, not the `REGISTER_VLLM_MODEL` registry, so
-they sit outside the gated list above.
+they sit outside the gated list above. One caveat the LTX-2.5 row is too narrow
+to carry: its text tower's prompt tokenization mirrors upstream only while the
+checkpoint's tokenizer `post_processor` adds nothing. The shipped one is MEASURED
+empty, so this port's plain encode plus an explicit BOS prepend matches
+upstream's `add_special_tokens=True` today; a checkpoint with a non-empty
+`post_processor` would tokenize differently here, and `Ltx2TokenizeGemmaPrompt`
+in `ltx2_text_encoder.cpp` is the call that would have to change.
 
 | Lane | Tested checkpoint(s) | Correctness gate | Speed vs reference |
 |---|---|---|---|

docs/USAGE.md

@@ -408,6 +408,15 @@ them to 4096 and 2048, and passes both streams through the embeddings connector
 before cross-attention. The tower is ~24 GB of host bf16 and stays resident,
 because a prompt arrives per request.
 
+One tokenization detail is a KNOWN DIVERGENCE rather than a mirror, and it is
+checkpoint-conditional: upstream tokenizes through the HuggingFace `__call__`
+with its default `add_special_tokens=True`, so it runs the tokenizer's
+post_processor, while this port calls the plain encode and prepends BOS by hand.
+On the shipped checkpoint the two are identical — its post_processor declares an
+EMPTY special-token map, measured on the shipped file rather than assumed — so
+nothing is lost today. A checkpoint whose post_processor DID add tokens would
+tokenize differently here.
+
 `--encoder-config` supplies the Gemma config, and it is required for the only
 shipped encoder: `vonkaiser`'s
 `gemma4-12b-with-proj-nvfp4-torchao.safetensors` carries no `__metadata__` at

include/vllm/model_executor/models/gemma4.h

@@ -251,6 +251,30 @@ std::vector<int32_t> Gemma4GenerateGreedyViaRegistry(
     vt::Queue& queue, int max_new_tokens,
     std::vector<float>* out_margins = nullptr);
 
+// The FULL-attention layers' "proportional" rope cos|sin table, on host in f32 —
+// the exact values the forward builds (BuildProportionalRopeCache rounds them to
+// bf16 to match the q/k it rotates; nothing else differs). Returns
+// [max_pos + 1, head_dim]: the first head_dim/2 columns are cos and the second
+// half sin, over the head_dim/2 DISTINCT angle pairs, mirroring upstream's
+// `emb = cat((freqs, freqs))` with each angle stored once
+// (`Gemma4UnifiedTextRotaryEmbedding.forward`, modeling_gemma4_unified.py:259-275
+// — the `cat` itself is :271; the inv_freq it consumes comes from
+// modeling_rope_utils.py:187-254).
+//
+// This is a GATE SURFACE, and it exists because of a measurement rather than a
+// preference. `partial_rotary_factor` decides how many angle pairs are rotated
+// and how many are zero-padded to identity, and it is the one field on this path
+// that the tower's hidden states cannot resolve: on the reduced LTX tower
+// fixture, forcing it from the config's 0.25 to 1.0 displaces the worst hidden
+// state by 1.09e-01 against that state's measured bf16 noise floor of 9.99e-02
+// — a ratio of 1.09, inside the tolerance the states are gated at — and a LARGER
+// fixture makes it worse, not better (0.65 at head_dim 16/32, seq 32), because
+// bf16 accumulation noise grows at least as fast as the rope contribution. So
+// the states are the wrong instrument and this table is the right one: f32, no
+// accumulation, compared element-wise against the oracle's own rotary embedding.
+std::vector<float> Gemma4ProportionalRopeCosSin(const HfConfig& config,
+                                                int64_t head_dim, int64_t max_pos);
+
 // Wrap already-loaded Gemma-4 weights in the registered LoadedModel so a caller
 // that owns the weights (the mm e2e gate) can drive ModelRegistry::Forward without
 // re-reading the checkpoint. `Make` OWNS the moved weights; `Borrow` does NOT own

include/vllm/model_executor/models/ltx2_text_encoder.h

@@ -450,19 +450,29 @@ struct Ltx2GemmaPromptTokens {
 // tokenizer.py:31-59, mirrored including the parts that look like details:
 //
 //   * `text.strip()` first (:33). diffusers strips too (pipeline_ltx2.py:333).
-//   * encode with NO special tokens added by the post_processor. The shipped
-//     tokenizer's `post_processor` is a TemplateProcessing whose `special_tokens`
-//     map is EMPTY, so it would add nothing anyway — measured on the shipped
-//     file, not assumed.
-//   * then PREPEND BOS unconditionally if it is not already first (:44-46).
-//     THIS IS WHERE THE TWO REFERENCES DISAGREE and upstream is followed:
-//     `ltx_core` prepends explicitly and says why — "Gemma 3 already emits it
-//     via post_processor; Gemma 4 does not, so we prepend" (tokenizer.py:12-15)
-//     — while diffusers passes `add_special_tokens=True` and relies on the
-//     post_processor (pipeline_ltx2.py:339), which for THIS tokenizer.json adds
-//     nothing. Following diffusers would drop token 0 of every prompt. Recorded
-//     rather than silently resolved: `ltx_core` is the model author's own
-//     runtime and is explicit about the case.
+//   * encode, then PREPEND BOS if it is not already first — CONDITIONAL, on
+//     upstream's own `if not input_ids or input_ids[0] != bos_id` guard
+//     (:44-46). A port that prepends unconditionally doubles the BOS.
+//
+//     Two things about that, and the first one is a KNOWN DIVERGENCE rather than
+//     a mirrored default. Upstream calls `self.tokenizer(text, ...)` — `__call__`
+//     with its default `add_special_tokens=True` (tokenizer.py:37-43) — so
+//     upstream DOES run the post_processor and we call plain `Encode`, which
+//     does not. On THIS checkpoint the two are identical, because the shipped
+//     `post_processor` is a TemplateProcessing whose `special_tokens` map is
+//     EMPTY and whose template is the bare sequence, so it has nothing to add:
+//     measured on the shipped file, not assumed. If a future checkpoint ships a
+//     post_processor that DOES add something, upstream would emit it and we
+//     would not — so this is the line to change, not a property to rely on.
+//
+//     What the two references actually disagree about is narrower than "one
+//     runs the post-processor": both let it run. `ltx_core` ALSO prepends BOS
+//     explicitly and says why — "Gemma 3 already emits it via post_processor;
+//     Gemma 4 does not, so we prepend" (tokenizer.py:12-15) — while diffusers
+//     relies on the post_processor alone (pipeline_ltx2.py:339), which for this
+//     tokenizer.json adds nothing, so following diffusers would drop token 0 of
+//     every prompt. `ltx_core` is the model author's own runtime and is explicit
+//     about the case, so it is the one followed.
 //   * EOS is never appended (:14).
 //   * truncation happens BEFORE the BOS prepend and again after (:41, :46), so a
 //     maximal prompt loses its LAST token to make room for BOS rather than

scripts/gen-ltx2-gemma-tower-goldens.py

@@ -35,17 +35,28 @@
     12B tower. Our port runs only the VALID tokens at their ORIGINAL absolute
     positions. That is equivalent -- pads are masked out of attention and are
     causally before every valid token, and the feature extractor zeroes their
-    rows anyway -- but "is equivalent" is a claim, so section 3 emits the full
+    rows anyway -- but "is equivalent" is a claim, so section 4 emits the full
     left-padded oracle run and the C++ suite holds the short run's valid rows
     to the padded run's valid rows. If the equivalence is ever false, that gate
     is what says so, and a 100x cost claim stops resting on an argument.
 
   * ONE arithmetic width per state, both ways round. Section 2 is the oracle in
-    float32 and section 4 the SAME oracle in bfloat16. Our forward carries the
-    stream in bf16 and widens only on the way out (gemma4.h,
-    Gemma4HiddenStatesResult), so bf16 is the dtype-MATCHED arm and f32 is the
-    arm that would catch a reduction-order defect a bf16 store absorbs. Gating
-    only one of them has burned this project before.
+    float32 and section 3 the SAME oracle in bfloat16; sections 4 and 4b are
+    that pair again for the left-padded run. Our forward carries the stream in
+    bf16 and widens only on the way out (gemma4.h, Gemma4HiddenStatesResult),
+    so bf16 is the dtype-MATCHED arm and f32 is the arm that would catch a
+    reduction-order defect a bf16 store absorbs. Gating only one of them has
+    burned this project before.
+
+    Every leg runs on a DEEP COPY of the module. `nn.Module.to(dtype)` converts
+    in place and bf16 rounding is destructive, so before that fix the two
+    "float32" legs after the bf16 one were executing over bf16-ROUNDED weights
+    -- MEASURED at up to 3.90e-02 per state, of the same order as the noise
+    floor the tolerance itself is derived from.
+
+  * The ROPE TABLES, in f32 (section 6), because `partial_rotary_factor` is not
+    resolvable from the hidden states at any fixture size this generator can
+    build. Section 6's note carries the measurement that establishes it.
 
 Both sides rebuild every weight from one deterministic FNV-1a + splitmix64
 stream keyed by the parameter's own HuggingFace NAME, exactly as
@@ -84,6 +95,7 @@
 from __future__ import annotations
 
 import argparse
+import copy
 import json
 import os
 import subprocess
@@ -317,7 +329,20 @@ def build_tower(real_config: dict):
 
 
 def run_tower(inner, ids, mask, dtype, positions=None):
-    m = inner.to(dtype).eval()
+    """One oracle leg, on a DEEP COPY of the module.
+
+    `nn.Module.to(dtype)` converts parameters and buffers IN PLACE, so the
+    obvious `inner.to(dtype)` / `inner.to(torch.float32)` round trip does not
+    restore anything: bf16 rounding is destructive, and every leg after the
+    first bf16 one would run over bf16-ROUNDED weights while calling itself
+    float32. MEASURED on this fixture before the fix: re-running the identical
+    f32 call after the bf16 leg moved a state by up to 3.90e-02, which is
+    ~40x the f32 legs' own round-off and of the same order as the bf16 noise
+    floor the gate is calibrated on. Sections 4 and 5 were silently
+    order-dependent because of it. Copying is cheap here -- the reduced tower
+    is a few hundred KB -- and it makes every leg independent of leg order.
+    """
+    m = copy.deepcopy(inner).to(dtype).eval()
     kwargs = {}
     if positions is not None:
         # The absolute positions the tokens occupy in the padded batch. Left
@@ -332,9 +357,45 @@ def run_tower(inner, ids, mask, dtype, positions=None):
             output_hidden_states=True,
             **kwargs,
         )
-    states = [h[0].to(torch.float32).contiguous().numpy() for h in out.hidden_states]
-    inner.to(torch.float32)
-    return states
+    return [h[0].to(torch.float32).contiguous().numpy() for h in out.hidden_states]
+
+
+def rope_cos_sin(config, layer_type: str, head_dim: int, positions) -> np.ndarray:
+    """The oracle's OWN cos|sin table for one layer type, in float32.
+
+    Why this is emitted at all: `partial_rotary_factor` is NOT resolvable from
+    the hidden states. MEASURED on this fixture, the whole difference between
+    the config's 0.25 and a port that ignored it and rotated fully is 1.09e-01
+    at the worst state against a bf16 noise floor of 9.99e-02 -- a ratio of
+    1.09, i.e. inside the tolerance the same states are gated at. Enlarging the
+    fixture does not rescue it: at (head_dim 16/32, seq 32) the ratio FALLS to
+    0.65, because bf16 accumulation noise grows at least as fast as the rope
+    contribution does. So the end-to-end states are the wrong instrument, and
+    the right one is the table itself, in f32, with no accumulation in it.
+
+    Built by the real `Gemma4UnifiedTextRotaryEmbedding`, which is what routes
+    `rope_type: "proportional"` to `_compute_proportional_rope_parameters`
+    (modeling_gemma4_unified.py:206-218 -- the ROPE_INIT_FUNCTIONS lookup at :207
+    and the call at :218; modeling_rope_utils.py:187-254) and so is the only
+    thing that decides how many angle pairs are zero-padded.
+
+    Returns [len(positions), head_dim]: the first head_dim/2 columns are cos
+    over the distinct angle pairs and the second half is sin, which is the
+    layout `BuildProportionalRopeCache` writes.
+    """
+    from transformers.models.gemma4_unified.modeling_gemma4_unified import (  # noqa: PLC0415
+        Gemma4UnifiedTextRotaryEmbedding,
+    )
+
+    rope = Gemma4UnifiedTextRotaryEmbedding(config)
+    pos = torch.tensor([list(positions)], dtype=torch.long)
+    x = torch.zeros(1, len(positions), head_dim, dtype=torch.float32)
+    cos, sin = rope(x, pos, layer_type=layer_type)
+    pairs = head_dim // 2
+    # Upstream duplicates each angle (`emb = cat((freqs, freqs))`) so cos/sin are
+    # head_dim wide over head_dim/2 DISTINCT angles; take one copy of each.
+    table = torch.cat((cos[0, :, :pairs], sin[0, :, :pairs]), dim=-1)
+    return table.to(torch.float32).contiguous().numpy()
 
 
 # ---------------------------------------------------------------------------
@@ -408,6 +469,11 @@ def main() -> int:
     plain_f32 = run_tower(inner, TOKENS, [1] * SEQ, torch.float32)
     plain_bf16 = run_tower(inner, TOKENS, [1] * SEQ, torch.bfloat16)
     padded_f32 = run_tower(inner, padded_ids, padded_mask, torch.float32)
+    # The padded run at the SHIPPED dtype too. Without it the left-padded rows
+    # have no dtype-matched oracle, and anything gated against them -- the
+    # prompt-to-conditioning path, which is left-padded by construction -- has
+    # no measured floor to be held to, only a borrowed one.
+    padded_bf16 = run_tower(inner, padded_ids, padded_mask, torch.bfloat16)
     # The equivalence claim, isolated INSIDE the oracle and in f32 so no dtype
     # noise is mixed into it: the same valid tokens, told their absolute
     # positions, run WITHOUT the pads. If upstream's own two answers agree, the
@@ -523,19 +589,60 @@ def main() -> int:
             "// [state][padded_seq * hidden]. Rows 0..kLtxTowerNumPad-1 are pad rows and\n"
             "// their contents are upstream's garbage-but-masked values; the gate reads\n"
             "// only the VALID tail and holds section 2 to it.\n"
+            "// Every leg runs on a DEEP COPY of the module, so this one is not\n"
+            "// downstream of the bf16 leg's rounding and the sections are independent\n"
+            "// of the order they are produced in.\n"
         )
         for i, s in enumerate(padded_f32):
             emit_f32(out, f"kLtxTowerPaddedStateF32_{i}", s)
 
+        out.write(
+            "// --- section 4b: the LEFT-PADDED run, BFLOAT16 (the SHIPPED dtype) ---\n"
+            "// The dtype-MATCHED arm for anything gated on the left-padded rows, and\n"
+            "// with section 4 it is also the padded run's own measured noise floor.\n"
+        )
+        for i, s in enumerate(padded_bf16):
+            emit_f32(out, f"kLtxTowerPaddedStateBf16_{i}", s)
+
         out.write(
             "// --- section 5: the equivalence, measured INSIDE the oracle ---\n"
             "// Per state, max|short-run-at-absolute-positions - padded-run's valid rows|,\n"
-            "// both f32, so this number carries NO dtype noise. It is upstream's own\n"
-            "// answer to 'is dropping the pads free?'. Our port inherits it; the C++\n"
-            "// gate checks the two claims separately so a failure says which one broke.\n"
+            "// both f32 and both on independently deep-copied modules, so this number\n"
+            "// carries NO dtype noise. It is upstream's own answer to 'is dropping the\n"
+            "// pads free?'. Our port inherits it; the C++ gate checks the two claims\n"
+            "// separately so a failure says which one broke.\n"
         )
         emit_f32(out, "kLtxTowerPadEquivalence", equivalence)
 
+        # SECTION 6 -- the rope table, because the hidden states cannot see it.
+        #
+        # `partial_rotary_factor` decides how many of the full-attention layers'
+        # angle pairs are rotated and how many are zero-padded to identity. It is
+        # config-carried, shape-invisible, and -- MEASURED, not assumed -- also
+        # invisible in the gated hidden states: setting it to 1.0 displaces the
+        # worst state by 1.09e-01 against a bf16 floor of 9.99e-02 (ratio 1.09),
+        # and a larger fixture makes that WORSE, not better (0.65 at head_dim
+        # 16/32, seq 32), because bf16 accumulation noise grows at least as fast.
+        # A gate whose tolerance is bf16 noise cannot resolve it, so the table
+        # itself is emitted and compared in f32.
+        full_head_dim = GLOBAL_HEAD_DIM
+        rope_positions = list(range(PADDED_SEQ))
+        rope_full = rope_cos_sin(config.text_config, "full_attention",
+                                 full_head_dim, rope_positions)
+        out.write(
+            "// --- section 6: the ORACLE's FULL-attention rope cos|sin table, f32 ---\n"
+            "// [position][global_head_dim]: first half cos, second half sin, over the\n"
+            "// global_head_dim/2 DISTINCT angle pairs (upstream stores each twice).\n"
+            "// `rope_type: proportional` at theta 1e6 and partial_rotary_factor 0.25,\n"
+            "// so the trailing pairs are cos=1, sin=0. Emitted because the partial\n"
+            "// factor is NOT resolvable from the hidden states -- see the note above --\n"
+            "// and because a table that silently rotated every pair would still produce\n"
+            "// 13 finite, plausibly-scaled states.\n"
+        )
+        emit_scalar(out, "kLtxTowerRopePositions", len(rope_positions))
+        out.write("\n")
+        emit_f32(out, "kLtxTowerRopeFullCosSin", rope_full)
+
         out.write("}  // namespace vllm_test\n")
 
     sys.stderr.write(

scripts/gen-ltx2-prompt-tokens-goldens.py

@@ -83,7 +83,13 @@ def tokenize_with_weights(tok, text: str, bos_id: int, pad_id: int):
     only thing borrowed is the algorithm, and every step cites its line.
     """
     text = text.strip()                                       # :33
-    ids = list(tok.encode(text, add_special_tokens=False).ids)  # :38-43
+    # :37-43 is `self.tokenizer(text, ...)` — `__call__` with its DEFAULT
+    # `add_special_tokens=True`, so upstream runs the post_processor. This passes
+    # False deliberately, to transcribe what the C++ `Encode` does; the two agree
+    # only because the measured post_processor here has an EMPTY `special_tokens`
+    # map (asserted into the emitted header below). On a checkpoint where it added
+    # something, THIS line and the C++ call would both have to change.
+    ids = list(tok.encode(text, add_special_tokens=False).ids)  # :37-43
     if len(ids) > MAX_LENGTH:                                 # truncation=True
         ids = ids[:MAX_LENGTH]
     if not ids or ids[0] != bos_id:                           # :44-46