feat(MODEL-MM-indextts2): talker embedding scaffolding — every stage now has gated numerics (#634) - #724
Merged
Merged
Conversation
…now has gated numerics (#634) FOLLOWING_AGENTS_PROTOCOL The talker (`UnifiedVoice`, model_v2.py) runs the GPT-2 backbone already on main over a sequence assembled from text tokens, mel codes and a speaker latent. This ports that assembly. 5 cases / 46 assertions. ITS POSITION EMBEDDINGS ARE NOT GPT-2's `wpe`. `LearnedPositionEmbeddings` (model_v2.py:244-256) is a SEPARATE learned table added on top of the token embedding BEFORE the backbone runs, so the backbone's own wpe applies as well. Treating them as one table silently halves the positional signal. `get_fixed_embedding(ind)` is the INCREMENTAL-DECODE path: at step n the position is n. Returning row 0 every step makes every generated frame believe it is the first, which still decodes to audio and destroys the prosody. No shape check can see it, so a case walks all eight steps and asserts the row identity. NO GOLDEN FILE HERE, deliberately. This layer is pure INDEXING; a captured tensor would only restate the table it was captured from. The fixture instead builds a table whose row r holds r*10 + d, so a row's identity is readable from any single element and the cases assert WHICH row was read. Capacity is also bounded -- `text_pos_embedding.emb.num_embeddings` is the talker's real request limit (infer_v2_5.py:427), so running past the end throws instead of reading adjacent memory. MUTATION EVIDENCE with compile status, reported by CASE COUNT: always returning row 0, dropping the capacity bound, and dropping the position add each RED at compile_err=0. The third mutation's first attempt reported compile_err=1 (unused parameter under -Werror) and was re-run as a 0.0F multiply. MILESTONE: every one of the six pipeline stages now has its numerics ported and gated against upstream — w2v-bert, EnhancedCodec, CAMPPlus, the length regulator, the GPT-2 talker backbone and S2Mel/BigVGAN. What remains is different in KIND: composition (wiring the stages into one pipeline), the checkpoint loader, and only then a render. NO e2e claim, and the lane still refuses by name. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The talker (
UnifiedVoice) runs the GPT-2 backbone already on main over a sequence assembled from text tokens, mel codes and a speaker latent. This ports that assembly. 5 cases / 46 assertions.Its position embeddings are not GPT-2's
wpe.LearnedPositionEmbeddingsis a separate learned table added on top of the token embedding before the backbone runs — so the backbone's ownwpeapplies as well. Treating them as one silently halves the positional signal.get_fixed_embedding(ind)is the incremental-decode path: at step n the position is n. Returning row 0 every step makes every generated frame believe it's the first — audio still comes out, prosody is destroyed, and no shape check sees it. A case walks all eight steps asserting row identity.No golden file here, deliberately
This layer is pure indexing; a captured tensor would only restate the table it came from. The fixture builds a table where row
rholdsr*10+d, so a row's identity is readable from any element, and the cases assert which row was read. Capacity is bounded too —text_pos_embedding.emb.num_embeddingsis the talker's real request limit, so running past the end throws rather than reading adjacent memory.Mutation evidence
Always returning row 0, dropping the capacity bound, dropping the position add — each RED at
compile_err=0, reported by case count.Milestone
Every one of the six pipeline stages now has its numerics ported and gated against upstream — w2v-bert, EnhancedCodec, CAMPPlus, length regulator, GPT-2 talker backbone, S2Mel/BigVGAN.
What remains is different in kind: composition (wiring the stages into one pipeline), the checkpoint loader, and only then a render. No e2e claim — the lane still refuses by name.