Draft: Add Parakeet MLX STT feasibility harness

[rgbkrk]

Summary

This is an external Parakeet feasibility harness and documentation slice, not a native Rust Parakeet backend.

• Adds eval/evaluate.py --engine parakeet-mlx so the existing WER/CER scorer can run MLX Parakeet on the same recording fixtures as the current voice CLI path.
• Adds opt-in PARAKEET_MLX=1 support to eval/compare.sh and eval/synth_eval.sh; existing Whisper model matrices stay unchanged by default.
• Documents current Parakeet artifact status, local MLX smoke evidence, streaming constraints, Candle blockers, and a native Rust integration path in docs/parakeet-stt-feasibility.md.

Feasibility status

• MLX-ready Parakeet artifacts exist today:
  • mlx-community/parakeet-tdt-0.6b-v3
  • mlx-community/parakeet-tdt-0.6b-v2
  • mlx-community/parakeet-ctc-1.1b
• parakeet-mlx runs locally on Apple Silicon without CUDA or NeMo runtime assumptions.
• Current best use in voice is an explicit eval/final-pass harness. This PR does not route voice listen, voice transcribe, or the daemon through Python/MLX.
• Native Candle support still needs a real Parakeet/FastConformer backend, tokenizer/preprocessor parity, and decoder work. CTC is the smallest native first target; TDT/RNNT is larger.

Local smoke evidence

Parakeet TDT v3 first run:

/usr/bin/time -p uvx --from parakeet-mlx parakeet-mlx \
  eval/recordings/001.wav \
  --model mlx-community/parakeet-tdt-0.6b-v3 \
  --output-dir /tmp/voice-parakeet-smoke-v3 \
  --output-format json \
  --verbose

• First run including package/model download: real 810.15
• Transcript: Fox jumps over the lazy dog.
• Expected: The quick brown fox jumps over the lazy dog.
• Cache: ~/.cache/huggingface/hub/models--mlx-community--parakeet-tdt-0.6b-v3
• Cached model blob size: 2,508,288,736 bytes

Cached Parakeet TDT v3 eval:

/usr/bin/time -p uv run --with parakeet-mlx python eval/evaluate.py \
  --recordings eval/recordings \
  --engine parakeet-mlx \
  --json-out /tmp/voice-parakeet-eval-v3.json

• Wall time: real 5.30
• Audio duration: 17.1s
• RTF: 0.2816
• Exact: 0/2
• Mean WER: 41.7%
• Outputs: 001 => Fox jumps over the lazy dog., 002 => Shells by the seashore.

Current Rust Whisper default comparison:

git lfs pull
cargo build -p voice
/usr/bin/time -p python3 eval/evaluate.py \
  --recordings eval/recordings \
  --voice target/debug/voice \
  --model distil-whisper/distil-large-v3.5 \
  --json-out /tmp/voice-whisper-eval-large-v3-5.json

• cargo build -p voice passed after LFS data was fetched into this Codex worktree.
• Wall time: real 7.67
• Audio duration: 17.1s
• RTF: 0.4445
• Exact: 0/2
• Mean WER: 36.1%
• Outputs: 001 => The fox jumps over the lazy dog., 002 => Shells by the seashore.

Parakeet TDT v2:

• Artifact confirmed through HF API.
• Local smoke was attempted with mlx-community/parakeet-tdt-0.6b-v2 and interrupted after real 182.06 because no visible model-cache progress occurred beyond a 40 KB repo stub.

Verification

git diff --check
python3 -m unittest eval.test_evaluate
bash -n eval/compare.sh
bash -n eval/synth_eval.sh
git lfs pull
cargo build -p voice
uvx --from parakeet-mlx parakeet-mlx --help
uv run --with parakeet-mlx python eval/evaluate.py --recordings eval/recordings --engine parakeet-mlx --json-out /tmp/voice-parakeet-eval-v3.json
python3 eval/evaluate.py --recordings eval/recordings --voice target/debug/voice --model distil-whisper/distil-large-v3.5 --json-out /tmp/voice-whisper-eval-large-v3-5.json

Review fixes

Follow-up commit 122a831 addresses the first adversarial review pass:

• Parakeet transcription infrastructure errors now set error_count and make eval/evaluate.py exit nonzero by default.
• --allow-errors is available only for exploratory runs that intentionally want empty/error transcripts scored without failing the process.
• eval/compare.sh and eval/synth_eval.sh now run the optional Parakeet leg through uv run --with parakeet-mlx python eval/evaluate.py, so a clean checkout with uv does not need a preinstalled parakeet-mlx script.

Review-fix verification:

git diff --check
python3 -m unittest eval.test_evaluate
bash -n eval/compare.sh
bash -n eval/synth_eval.sh
cargo build -p voice
uv run --with parakeet-mlx python -c "import shutil; print(shutil.which('parakeet-mlx'))"
python3 eval/evaluate.py --recordings eval/recordings --engine parakeet-mlx --parakeet-bin /tmp/missing-parakeet-mlx --json-out /tmp/voice-parakeet-missing-bin-after-fix.json
python3 eval/evaluate.py --recordings eval/recordings --engine parakeet-mlx --parakeet-bin /tmp/missing-parakeet-mlx --allow-errors --json-out /tmp/voice-parakeet-missing-bin-allow-errors.json
uv run --with parakeet-mlx python eval/evaluate.py --recordings eval/recordings --engine parakeet-mlx --json-out /tmp/voice-parakeet-eval-v3-after-fix.json

Missing-bin behavior after the fix:

• Default strict command exits 1.
• It prints error: 2 transcription item(s) failed; use --allow-errors for exploratory scoring.
• The JSON contains "error_count": 2.
• The same missing-bin command exits 0 only when --allow-errors is set.

Review status

Draft intentionally. This should get adversarial Codex review before it is marked ready, especially around the harness API shape and the feasibility conclusion.

[pullfrog]

✅ No new issues found.

Reviewed changes — Adds an external Parakeet MLX eval harness to eval/evaluate.py alongside a feasibility doc; the existing Whisper eval path is unchanged and the new path is opt-in everywhere.

• eval/evaluate.py — Parakeet MLX engine — renames transcribe → transcribe_voice, adds transcribe_parakeet_mlx + dispatcher, threads engine/parakeet_bin/parakeet_cache_dir through evaluate(), extends JSON output with engine, error_count, and Parakeet binary metadata, and adds strict-by-default exit behavior with --allow-errors escape hatch.
• eval/compare.sh + eval/synth_eval.sh — appends an opt-in PARAKEET_MLX=1 block using uv run --with parakeet-mlx; existing Whisper model matrix is untouched.
• eval/test_evaluate.py — adds four new unit tests covering parakeet_txt_output_path, build_parakeet_mlx_command, and the exit_code_for_summary strict/lenient paths.
• docs/parakeet-stt-feasibility.md — records smoke evidence, streaming assessment, native Candle blockers, and a recommended next slice. All 11 tests pass.

ℹ️ Nitpicks

• eval/compare.sh:44 and eval/synth_eval.sh:57 invoke uv run --with parakeet-mlx python eval/evaluate.py while the adjacent Whisper legs use python3. Within uv run this works fine (uv always exposes python in the managed venv), but it's inconsistent — python3 in both paths would be more uniform.
• evaluate() and its call sites always receive a parakeet_bin argument and emit "parakeet_binary" in the JSON result even when engine == "voice". The field is harmless but adds noise to Whisper eval JSON files; passing None or omitting the field for the voice engine would keep the output schema tighter.

  ^{｜ View workflow run ｜ Using us.anthropic.claude-sonnet-4-6 ｜ 𝕏}