feat(MODEL-MM-indextts2): the DiT transformer stack, so the S2Mel estimator is COMPLETE front to tail (#634) - #748
Merged
Conversation
…imator is COMPLETE front to tail (#634) Ports `gpt_fast/model.py:161-191` (Transformer.forward): N blocks, the U-Net skip routing across them, and the final AdaptiveLayerNorm. With #738, #739, #741 and #745 the S2Mel DiT now exists end to end -- front end, stack, tail -- each gated against upstream's own modules. This is composition only. The block is `dit::Block`, the routing is `dit_skip::Plan`, the merge is `dit_skip::ApplySkip`, and nothing here reimplements any of them. The rotary table is passed IN, emitted by the generator from upstream's own `freqs_cis`, so the gate measures composition rather than a second copy of `precompute_freqs_cis`. One upstream quirk is recorded rather than tidied: a `skip_in_linear` is built on EVERY layer when `uvit_skip_connection` is set, including layers that never receive a skip, so the checkpoint carries unused ones. They load and are left alone. A case asserts that layer 0's skip_in_linear cannot affect the output at all, while a receiving layer's must. 3 cases / 52 assertions against upstream Transformer.forward. Five mutations, five caught, all compile_err=0, file byte-identical after: the skip merged against itself instead of the popped value, emitting a layer's INPUT instead of its output, FIFO instead of LIFO, the final norm dropped, and the receive moved after the block. A SIXTH mutation was withdrawn as badly designed rather than reported as a gate hole. Pushing before the block WITHOUT removing the push after it leaves the stack holding both, and because a layer's input IS the previous layer's output, every pop still returned the same tensor -- a genuine no-op, not a defect the gate missed. Rewritten to emit the input INSTEAD of the output, it fails. Still no render: the talker, the reference encoders and BigVGAN are unported, so nothing yet produces the conditioning this estimator consumes, and correctness against vLLM-Omni needs the oracle pin (#633). FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
localai-bot
pushed a commit
that referenced
this pull request
Aug 14, 2026
Merges the PINNED SHA `32d82c64d`, not the ref. `origin/main` is shared with other worktrees in this checkout, and it advanced from `5e646d98c` to `32d82c64d` mid-merge when another session fetched — which made main's own newer files (the `dit_front` / `dit_stack` IndexTTS2 cluster, #745/#748) read as "silently dropped by the merge" in every `git diff origin/main` afterwards. They were never dropped; they are simply not in `5e646d98c`. That first attempt was aborted rather than patched, because the repair it invited — restoring files from the newer main into a tree merged from the older one — produced a CMakeLists listing TUs the tree did not contain. Merging an immutable SHA is what makes the verification below mean anything. Three conflicts, all resolved against that SHA: * `tests/vllm/multimodal/test_ltx2_video.cpp` — both sides added a different include in the same sorted run: this branch's `ltx2_video_vae_encoder.h` and main's `vllm/platforms/interface.h`. Both are kept, in sorted order. Proved by COMPILING, not by reading: `BUILD_EXIT=0` and the suite runs. * `.agents/roadmap_v1.md` and `docs/FEATURES.md` — keyed records, resolved by taking main's file WHOLESALE and reapplying this row's key onto it, never by accepting the automatic three-way result. Each file is now byte-identical to `32d82c64d` apart from exactly one line: the `#644` row and the LTX-2.5 DiT row. An earlier merge in this campaign auto-merged two `#644` rows into a duplicate that `check-agent-record` caught; taking one side wholesale is what makes that shape impossible rather than merely unlikely. The combined `#644` row names both campaign rows and both specs. The FEATURES row carries main's prompt-AdaLN facts plus this row's served image-conditioning arm, inside the 220-char cell and 600-char entry budgets. `docs/USAGE.md`, `CMakeLists.txt` and `tests/CMakeLists.txt` auto-merged, and were each verified against the pinned SHA rather than trusted: both CMake files are PURELY ADDITIVE (zero removed lines; this branch adds the two `ltx2_video_vae_encoder_load` / `ltx2_image_preprocess` TUs and the one `test_ltx2_image_cond` registration), and USAGE's only removal is the paragraph this row supersedes, with main's own additions still present. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
feat(MODEL-MM-indextts2): the DiT transformer stack, so the S2Mel estimator is COMPLETE front to tail (#634)
Ports
gpt_fast/model.py:161-191(Transformer.forward): N blocks, the U-Netskip routing across them, and the final AdaptiveLayerNorm. With #738, #739,
#741 and #745 the S2Mel DiT now exists end to end -- front end, stack, tail --
each gated against upstream's own modules.
This is composition only. The block is
dit::Block, the routing isdit_skip::Plan, the merge isdit_skip::ApplySkip, and nothing herereimplements any of them. The rotary table is passed IN, emitted by the
generator from upstream's own
freqs_cis, so the gate measures compositionrather than a second copy of
precompute_freqs_cis.One upstream quirk is recorded rather than tidied: a
skip_in_linearis builton EVERY layer when
uvit_skip_connectionis set, including layers that neverreceive a skip, so the checkpoint carries unused ones. They load and are left
alone. A case asserts that layer 0's skip_in_linear cannot affect the output at
all, while a receiving layer's must.
3 cases / 52 assertions against upstream Transformer.forward.
Five mutations, five caught, all compile_err=0, file byte-identical after: the
skip merged against itself instead of the popped value, emitting a layer's INPUT
instead of its output, FIFO instead of LIFO, the final norm dropped, and the
receive moved after the block.
A SIXTH mutation was withdrawn as badly designed rather than reported as a gate
hole. Pushing before the block WITHOUT removing the push after it leaves the
stack holding both, and because a layer's input IS the previous layer's output,
every pop still returned the same tensor -- a genuine no-op, not a defect the
gate missed. Rewritten to emit the input INSTEAD of the output, it fails.
Still no render: the talker, the reference encoders and BigVGAN are unported, so
nothing yet produces the conditioning this estimator consumes, and correctness
against vLLM-Omni needs the oracle pin (#633).
Issue: #634.
🤖 Generated with Claude Code