native Moshi: port LM, Depformer, and Mimi encoder onto the image/session/leaf discipline

[sydneyrenee]

Problem

Moshi is a supported model that is currently unwired from the product path: the LFM2 native cutover deliberately left it offline/oracle-only, and no issue owns bringing it back. Two independent gaps block it:

1. The native Mimi archive is decoder-only. native/src/mimi/ implements the decode graph (quantizer → upsample → transformer → SEANet). The encoder side — SEANet encoder stack, encoder transformer with streaming KV, downsample/project to latent rate, and split residual VQ producing input codebooks — is entirely absent. Native Mimi therefore cannot turn user PCM into codes, which Moshi full-duplex requires every frame.
2. The Moshi LM/Depformer has no native port. No native multistream LM step, text head/sampler, or Depformer codebook recurrence exists. The deleted Rust/Candle RealtimeMoshi is not a fallback and must not return.

This is a dedicated later tranche, not part of the LFM2 FIFO program (#171–#175) and explicitly out of scope for V2 blocks (#159). Tracked here so the sequencing decision (LFM2 first, Moshi unwired meanwhile) stays visible and the tranche has a home when it starts.

Contract

• spec 08 (specs/11-kcoro-native-migration/08-native-moshi-runtime.md) is the normative design; the supported subset is exactly what the deleted Rust admitted: model_type == "moshi", unconditioned Moshiko Candle-layout checkpoints, no LoRA, no CFG, current v0.1 multistream config, one full-duplex stream per conversation.
• Same discipline as the LFM2 path: one resident image (Mimi binds only the distinct LFM_WEIGHT_COMPONENT_MIMI; the LFM2.5 loader must keep refusing it), opaque handles, no Rust math, assembly leaves for all math, no production scalar fallback, no post-readiness allocation.
• First-frame behavior is load-bearing and must be preserved exactly: current code encodes frame one, then resets Mimi state, decrements skip_frames, and still feeds those codes to the LM. Resetting before encode or discarding the codes changes the stream. Pin with a fixture.
• Soft interrupt discards output by epoch without resetting Mimi/LM/Depformer stream state; mic pause stops frame submission without synthesizing silence; explicit new-stream reset is a separate command. Three distinct operations, not one ambiguous interrupt().
• Frame passes are full-pass units: stop/interrupt observed between frames, never inside convolution/attention/sampling/codebook loops.
• No turn VAD in Moshi mode. Backpressure drops stale not-yet-submitted capture frames; it never resets stream state or blocks the hardware callback.
• Native SentencePiece piece decoding (id_to_piece, word-boundary marker → space, suppress empty), including split UTF-8 and control tokens.

Work

1. Freeze the supported Moshi/Mimi schema inventory (weight names, shapes, dtypes, cache dims, control tokens, sampler params) into native schema tests before numerical porting.
2. Extend the loader for LM + Mimi + SentencePiece components of one bundle.
3. Port the Mimi encoder as explicit stages with parity fixtures through emitted input codebooks, including first-frame reset ordering.
4. Port one streaming LM step; compare every layer/KV boundary.
5. Port text sampling and Depformer recurrence with deterministic RNG fixtures (shared draw stream semantics).
6. Reuse the shared MimiDecodePlan/MimiDecodeState split; decode into playback reservations at equal rate or via the route's prepared native resampler.
7. Mount PassKind::MoshiFrame on the fixed team; native frame slots, epochs, and exact wake behavior per spec 04's frame-mode rules.
8. Native output resampling with stream continuity (StreamingPcmResampler carry in conversation/session state).
9. A/B the complete frame pass against frozen authoritative traces (per native inference: restore authoritative full-value faithfulness gates before FIFO fusion #170's fixture contract).
10. App gate: continuous full duplex, simultaneous speech, soft interrupt without reset, device switch, long-run soak, clean teardown; only then make Moshi selectable in product settings.

Acceptance

• Model open accepts/rejects exactly the supported checkpoint subset with typed errors and no Candle code.
• Encoder stage outputs and input codebooks match golden fixtures across first frame, steady state, silence, chunk boundaries, and cache wrap.
• First-frame encode/reset/feed ordering is token-exact.
• LM hidden rows, KV, text logits/tokens, Depformer logits/codes, sampler states match traces at every boundary.
• One hour continuous full duplex: no post-warmup allocation, no growing queue, no reset under pressure, no audio discontinuity.
• Soft interrupt drops old output without resetting stream state.
• Stop during encode/LM/Depformer/decode/playback-full/idle joins once without polling or deadlock.
• Production dependency graph contains no Rust moshi crate or Candle through the Moshi feature path.
• The phrase "native Moshi" is not used for anything short of the above.

Related

• Normative design: spec 08
• Faithfulness fixture contract: native inference: restore authoritative full-value faithfulness gates before FIFO fusion #170
• Release matrix: native voice: complete cross-architecture, sanitizer, soak, and fairness release gates #160
• Real-checkpoint gate: voice: automate the native real-checkpoint truth gate and preserve evidence #147
• V2 blocks (non-goal there): flashkern V2: implement independent four-lane BlockDomains and measured dual-program execution #159

— K3

[sydneyrenee]

Implementation handoff — Moshi is continuous frame recurrence, not turn-scoped LFM2

This tranche must remain isolated until LFM2 #170–#176 work is stable. Do not reuse LFM2 turn machinery merely because both models produce audio.

Key architectural differences

• Moshi continuously consumes Mimi codes at frame cadence; Sesame may annotate UI but must not gate inference frames.
• Mic pause means no submitted frame, not synthetic silence and not a model reset.
• Soft interrupt suppresses stale publication by epoch while LM/Mimi/Depformer stream state continues.
• New-stream reset is explicit and distinct.
• Backpressure drops stale not-yet-submitted capture frames; it never blocks the callback or silently resets recurrence.
• There is no Rust/Candle fallback.

Missing model work

The native Mimi implementation is decoder-only. A complete encoder is required: SEANet encoder, encoder transformer/cache, downsample/project, and residual VQ input codebooks. The LFM2.5 custom detokenizer from #175 is unrelated and cannot encode user audio.

The Moshi LM and Depformer also require native schema, weights, recurrence, sampler, and SentencePiece handling. Do not claim “native Moshi” after only wiring Mimi decode.

Load-bearing first-frame behavior

Pin the current supported semantics before refactoring: encode frame one, then reset Mimi state, decrement skip state, and still feed those produced codes to the LM. This looks counterintuitive, so a conventional cleanup will likely “fix” it incorrectly.

Execution discipline

Each frame is one correlated pass ticket advanced by callbacks. Assembly owns tiles; no yield inside a kernel. No per-frame allocation, polling, sleeps, channels, host loops, or timer-driven progress. Full-pass stop/interrupt boundaries occur between frames.

Start with schema and immutable traces, then Mimi encoder, then one LM step, then sampling/Depformer, then decode/playback. Run long continuous-state tests; isolated frames cannot validate Moshi.