Expose experimental Voxtral Realtime STT backend selection

[rgbkrk]

Summary

This is a focused Voxtral Realtime STT routing slice. The workspace already had native Rust/Candle Voxtral Realtime model code in voice-voxtral; this PR makes it reachable through the shared STT model abstraction and the existing user-facing STT surfaces without changing the default Whisper path.

Changes:

• Add backend/model resolution helpers to voice-stt, including voxtral-realtime parsing and Voxtral model-name inference for mistralai/Voxtral-Mini-4B-Realtime-2602.
• Add --stt-backend, --stt-model, and --stt-max-new-tokens to voice listen and voice transcribe.
• Move CLI local listen/transcribe, realtime foreground STT, JSON-RPC, MCP, and daemon STT storage from direct WhisperModel usage to backend-neutral voice_stt::SttModel.
• Report STT load device and load duration in the CLI load banner.
• Preserve existing default behavior: Whisper remains default, and non-explicit single-shot voice listen still prefers the daemon.

What runs today

The following route now selects the native Voxtral Realtime STT backend, loads the full model on Metal, and transcribes a real audio file:

target/debug/voice transcribe --stt-backend voxtral --stt-max-new-tokens 64 /tmp/voxtral-realtime-stt-smoke.wav

Observed output:

Loading speech-to-text model (backend=voxtral, model=mistralai/Voxtral-Mini-4B-Realtime-2602)...
Model loaded on metal:0 in 2.09s. Ready to listen.

Transcribing: /tmp/voxtral-realtime-stt-smoke.wav
Audio: 2.5s (39946 samples at 16000Hz)

(43 tokens)
The quick brown fox jumps over the lazy dog.

Total command timing from /usr/bin/time -p: real 7.80, user 4.57, sys 1.31.

The native Realtime implementation is not a stub: voice-voxtral has tests covering realtime config parsing, tensor contract metadata, audio padding, mel generation, causal audio transformer modules, projector modules, sliding causal masks, tiny text decode, and tiny end-to-end transcriber paths over audio embeddings and 16 kHz samples.

Real audio smoke status

Generated real speech input:

say -v Samantha -o /tmp/voxtral-realtime-stt-smoke.aiff "The quick brown fox jumps over the lazy dog."
afconvert -f WAVE -d LEI16@16000 -c 1 /tmp/voxtral-realtime-stt-smoke.aiff /tmp/voxtral-realtime-stt-smoke.wav
afinfo /tmp/voxtral-realtime-stt-smoke.wav

Input:

• /tmp/voxtral-realtime-stt-smoke.wav
• WAV, 16 kHz, mono, Int16
• 2.496625 seconds
• Phrase: "The quick brown fox jumps over the lazy dog."

Attempt:

/usr/bin/time -p cargo run -p voice -- transcribe --stt-backend voxtral --stt-max-new-tokens 64 /tmp/voxtral-realtime-stt-smoke.wav

Result:

• The full checkpoint now loads successfully from the local HF cache.
• Device: metal:0
• CLI-reported model load time: 2.09s
• Transcript: The quick brown fox jumps over the lazy dog.
• Generated tokens: 43
• Total command wall time: 7.80s

Cache state:

• Cache root: /Users/kylekelley/.cache/huggingface/hub/models--mistralai--Voxtral-Mini-4B-Realtime-2602
• Snapshot: /Users/kylekelley/.cache/huggingface/hub/models--mistralai--Voxtral-Mini-4B-Realtime-2602/snapshots/2769294da9567371363522aac9bbcfdd19447add
• Present: config.json, params.json, processor_config.json, tekken.json
• Full weight file: consolidated.safetensors, 8,859,462,744 bytes
• Weight blob: /Users/kylekelley/.cache/huggingface/hub/models--mistralai--Voxtral-Mini-4B-Realtime-2602/blobs/263f178fe752c90a2ae58f037a95ed092db8b14768b0978b8c48f66979c8345d
• Snapshot link: /Users/kylekelley/.cache/huggingface/hub/models--mistralai--Voxtral-Mini-4B-Realtime-2602/snapshots/2769294da9567371363522aac9bbcfdd19447add/consolidated.safetensors
• Cache disk use after completion: 8.3 GiB

Download note: the completed path used resumable single-stream curl -L --fail --continue-at - against the Hugging Face resolve URL. Earlier multi-range aria2c produced sparse partial state and Xet 403s; that state was removed before completion.

Verification

Passed:

cargo check -p voice-stt -p voice-daemon -p voice
cargo test -p voice-stt backend
cargo test -p voice parses_stt_backend_options_for_listen_and_transcribe
cargo test -p voice-voxtral realtime
scripts/verify_pre_pullfrog_local_review.sh --base origin/main
/usr/bin/time -p target/debug/voice transcribe --stt-backend voxtral --stt-max-new-tokens 64 /tmp/voxtral-realtime-stt-smoke.wav

scripts/verify_pre_pullfrog_local_review.sh --base origin/main covered:

• cargo fmt --check
• git diff --check
• git diff --cached --check
• git diff --check 817af92020fc3eeea10b3833afd4c70dbbad42d5...HEAD
• cargo test -p voice
• cargo test -p voice-daemon
• cargo test -p voice-stt
• cargo clippy -p voice -p voice-daemon -p voice-stt --all-targets -- -D warnings

Known blockers and risks

• The smoke is a short generated English WAV, not a broad accuracy or long-form anti-repetition evaluation.
• No memory/RSS/Metal allocation number is recorded yet.
• The current native decode path is an offline greedy path over precomputed audio embeddings. It does not yet implement production realtime KV cache, incremental audio cache, sliding-window cache trimming, commit semantics, or partial transcript semantics.
• The next highest-value checks are longer speech fixtures, repetition-prone audio, multilingual input, and parity comparison against vLLM or Transformers for the same delay and max_new_tokens.

Primary sources checked

• Hugging Face model card and files for mistralai/Voxtral-Mini-4B-Realtime-2602
• HF commit 2769294da9567371363522aac9bbcfdd19447add
• Transformers configuration_voxtral_realtime.py
• Transformers modeling_voxtral_realtime.py
• vLLM Voxtral Realtime recipe

[pullfrog]

This run was cancelled 🛑

The workflow was cancelled before completion. Please check the link below for details.

  ^{｜ View workflow run ｜ via Pullfrog ｜ 𝕏}