Optional forced-alignment pipeline stage for word-level timestamps

[christopherthompson81]

Motivation

Vernacula's editor wants word-level timestamps for click-to-position playback, highlight-during-playback, and segment-boundary editing. Today the picture is uneven across backends:

Backend	Native word-level timings?	Today's behaviour
Parakeet	Yes (RNN-T emits per-frame predictions)	Real timings
Whisper Turbo	**Yes (native `<	0.00
Cohere Transcribe	No	`BuildSyntheticTokenTimestamps` linear interpolation across the segment duration
Qwen3-ASR	No	Same synthetic linear interpolation
Granite Speech 4.1	No	Same synthetic linear interpolation (PR #35)
VibeVoice-ASR	Per-utterance (built-in diarization gives utterance timings, not word)	Utterance timings
IndicConformer	Yes (CTC head, per-frame)	Real timings

The synthetic-linear approach is acceptable for short segments but accumulates error on long ones — a long word at the end of a 30 s segment ends up timed too early, and click-to-position is off by seconds.

Rather than chase per-backend extraction (DTW on Whisper-style cross-attention, or research-grade self-attention DTW on Granite's decoder-only LM), an optional forced-alignment pass would unlock real word-level timings for every backend that doesn't emit them natively, using one shared infrastructure.

Architecture: model-agnostic, with pluggable acoustic backbones

The design should be acoustic-model-agnostic. The work splits into:

1. A common `ForcedAligner` interface in `Vernacula.Base`: `(float[] audio, string text, string language) → IReadOnlyList`.
2. Pluggable implementations behind that interface — each one wraps a CTC acoustic model + the Viterbi alignment algorithm.
3. Settings toggle picks the active aligner; default-pick is hardware/language/license-driven.

This mirrors how Vernacula already handles ASR backends (multiple implementations behind a common pattern). It avoids prematurely committing to a single model when we don't yet have benchmark data, and makes adding future aligners cheap.

Option space

There are roughly five families. Most production systems use family A; family C is the "you already have Whisper, why bother" path.

Family	Examples	Trade-off
A. CTC character-aligners	MMS-300M aligner, wav2vec2-large-960h, NeMo NFA + Conformer-CTC	Single model, character-level. Most modern, simplest pipeline.
B. CTC phoneme-aligners + G2P	WhisperX classic, wav2vec2-phoneme finetunes per lang	More accurate at phoneme boundaries; per-language G2P dictionaries required. WhisperX default.
C. Whisper-internal cross-attention DTW	whisper-timestamped, stable-ts, HF `word_timestamps=True`	Only works for Whisper; useless for Granite/Cohere.
D. Classical GMM-HMM	Montreal Forced Aligner, Gentle	CPU-only, deterministic, lower accuracy, dependency hell (Kaldi).
E. Library-only orchestration	`ctc-segmentation`, `torchaudio.functional.forced_align`	Not a model; the alignment algorithm. Pair with any A-family model.

Vernacula needs family A or B; family C doesn't generalise; D doesn't ONNX-export.

Realistic shortlist after Vernacula-specific filters

Filters: must be ONNX-exportable, must integrate with HF manifest download flow, must be permissively licensed, single-model preferred over per-language matrix.

Candidate	Languages	Size	Notes
MMS-300M Forced Aligner (`MahmoudAshraf/mms-300m-1130-forced-aligner`)	1130+	300M	Single multilingual model. Permissive license on the derivative. ONNX-exportable via Optimum. The "obvious default if quality is acceptable everywhere."
NeMo NFA + Conformer-CTC	~25-30 in flagship NVIDIA multilingual checkpoints, more via per-language matrix	100-600M depending on model	Reuses our existing NeMo export tooling (`scripts/nemo_export/`, same path as Parakeet/Sortformer). Well-tuned on English. Operational tax: per-language matrix or commit to one multilingual checkpoint with narrower coverage. License: NVIDIA OML, permissive but verify per-checkpoint.
Wav2vec2-large-960h (`facebook/wav2vec2-large-960h`)	English only	300M	WhisperX default for English. Reference-quality on English; would need per-language wav2vec2-XLSR finetunes for other languages.

Comparison axes (the framework)

These are the dimensions to score every candidate on. Vernacula-specific weights in the rightmost column.

Axis	Weight	Notes
Language coverage	High	Must cover the union of Cohere (14) + Granite (6) + Qwen3 (29) at minimum.
Word boundary accuracy (MAE in ms)	High	Aim for ≤ 50 ms MAE on the languages we ship transcripts for.
Robustness on noisy / accented speech	Medium	Vernacula handles podcasts and conference recordings, not just clean studio.
Model size	Medium	300M fine; 1B borderline; 3B+ no.
Per-utterance latency	Medium	Fires once per VAD segment.
License	Hard gate	No NC clauses for Vernacula's distribution.
ONNX exportability	Hard gate	Vernacula ships ONNX. Must export cleanly.
Community / maintenance	Medium	Bus factor matters for 2-year horizon.
Tokenization compat	Medium	Multilingual aligners need to handle CJK, Arabic, Indic scripts.
G2P dependency	Soft	Adding a per-language G2P matrix is operational pain. Family A avoids this.

Evaluation methodology

Before committing to a default, run a benchmark:

1. Build a small ground-truth test set. ~10 clips per language for the top 6-8 languages we care about (en, fr, de, es, ja, zh, ar, hi). LibriSpeech for English; Common Voice / MLS for others; manual word-boundary annotation where the corpus doesn't ship it.
2. Score each candidate on (MAE, P90, P99) word-boundary error. `MAE ≤ 50 ms` for the languages we ship is effectively indistinguishable from "perfect" for editor UX. P99 catches the painful click-to-wrong-place tail.
3. Eyeball-test 5 clips per language. Subjective but cheap, catches systematic drift the metrics miss.
4. Measure latency per utterance. Per-segment inference time × typical segment count = added wall-clock per file.
5. Verify ONNX export end-to-end. Run the export, load with ORT, parity smoke against PyTorch reference. Catches weird ops before committing a week of integration.

The benchmark drives the default-pick. Worst-case outcome: MMS is acceptable everywhere → ship MMS. Best-case: NFA dominates English with a small enough operational tax → ship NFA for English, MMS for rest.

Integration sketch

• New `scripts/forced_align_export/` (and possibly `scripts/nemo_forced_align_export/` for NFA) producing ONNX bundles, following per-backend export pattern.
• `Vernacula.Base.IForcedAligner` interface: `(float[] audio, string text, string language) → IReadOnlyList`.
• Implementations: `MmsForcedAligner`, `NemoNfaAligner`, etc. Viterbi DP shared as a static helper since both rely on the same algorithm over CTC log-probs.
• New HF repos under `christopherthompson81/` for whichever bundles get picked.
• `Settings → Word-level alignment → [Off / Auto when backend lacks native timings / Always]`. Default `Auto` once a default model is benchmarked.
• Settings → "Forced-aligner model" picker if we ship more than one option.
• `TranscriptionService`: when the active backend doesn't emit native timings AND the setting is on, run the aligner per-segment and replace the synthetic timestamps with real ones before `db.UpdateResult`.
• `AsrLanguageSupport.HasNativeWordTimings(AsrBackend)` helper to centralise per-backend gating.
• Editor side: no changes needed if the timestamps schema stays the same.

Out of scope (separate follow-ups)

• Phoneme-level timings — alignment models can produce them but the editor wouldn't use them today.
• Reusing the aligner for non-ASR jobs (subtitle re-alignment, audio-book chaptering, …). Worth scoping later but not part of this.
• Switching backends with native timings (Whisper, Parakeet, IndicConformer) over to the aligner for consistency. Their native timings are at least as good; touching them is risk for no benefit.

Definition of done

• Acoustic-model-agnostic `IForcedAligner` interface in `Vernacula.Base` with a shared Viterbi DP helper.
• At least one concrete implementation shipping (MMS-300M as the most likely default).
• ONNX export pipeline under `scripts/_export/` with parity smoke against the PyTorch reference.
• C# `tests/Smoke/` parity harness.
• HF bundle uploaded with a model card under `scripts/hf_readmes/`.
• Wired into `TranscriptionService` behind the Settings toggle; works end-to-end on at least Granite Speech and Cohere with verified word-level click-to-position in the editor.
• Cross-backend smoke on a multilingual clip — at least English + one non-English language (e.g. French or Japanese for Granite, Mandarin for Cohere).
• Benchmark results documented in `docs/dev/forced_alignment_investigation.md` covering at least the languages Vernacula ships transcripts for (en, fr, de, es, ja, zh, ar, hi).
• Setting defaults to `Auto` so users on backends without native timings get the upgrade without configuration.

References

• WhisperX paper: https://arxiv.org/abs/2303.00747
• MMS forced aligner card: https://huggingface.co/MahmoudAshraf/mms-300m-1130-forced-aligner
• NeMo NFA tool: https://github.com/NVIDIA/NeMo/tree/main/tools/nemo_forced_aligner
• `torchaudio.functional.forced_align` reference: https://docs.pytorch.org/audio/main/generated/torchaudio.functional.forced_align.html
• Surfaced from PR Granite Speech 4.1 GUI integration (#33) #35 review on Granite Speech GUI integration — Granite is the immediate beneficiary but every text-only ASR backend benefits.