Context
VideoLingo uses WhisperX for local ASR. An optional FunASR backend is useful for local Chinese, English, and Japanese subtitle workflows, especially when users want SenseVoice's native word timestamps or CPU inference. WhisperX remains the default.
Implementation
PR #584 adds a direct Python integration:
core/asr_backend/funasr_local.py with iic/SenseVoiceSmall as the default model- the same segment/word contract consumed by VideoLingo's existing transcription pipeline
- exact ffmpeg clip extraction and clip-offset timestamp mapping
- native FunASR word timestamps with a bounded token-level fallback
- cached CPU/CUDA models and per-model inference locks
- localized Streamlit runtime, model, device, and supported-language controls
- an opt-in installer path:
python installer.py --with-funasr
- launcher preflight, configuration defaults, tests, and English/Chinese docs
Validation
The implementation was tested with Python 3.10.20, FunASR 1.3.14, PyTorch/Torchaudio 2.8.0+cu128, Transformers 4.57.6, WhisperX 3.8.6, and an NVIDIA H100.
- 13 focused unit/wiring tests pass.
- Real SenseVoice audio passes on both CUDA and CPU, including non-zero clip offsets.
- The downstream
process_transcription path preserves every timestamped token.
- A 45-second sample produced 149 monotonic, in-bounds word timestamps.
- The localized Streamlit controls pass AppTest.
- Installer health, Ruff, compilation, JSON/YAML validation, and diff checks pass.
Scope and licensing
The first UI version exposes Chinese, English, and Japanese because those languages match VideoLingo's current downstream handling. The model remains configurable for compatible FunASR checkpoints.
The FunASR and SenseVoice repository source code is MIT. Model weights follow the license on each model card; the default SenseVoiceSmall model card links its own model license. No unverified speed or accuracy claims are assumed here.
Context
VideoLingo uses WhisperX for local ASR. An optional FunASR backend is useful for local Chinese, English, and Japanese subtitle workflows, especially when users want SenseVoice's native word timestamps or CPU inference. WhisperX remains the default.
Implementation
PR #584 adds a direct Python integration:
core/asr_backend/funasr_local.pywithiic/SenseVoiceSmallas the default modelpython installer.py --with-funasrValidation
The implementation was tested with Python 3.10.20, FunASR 1.3.14, PyTorch/Torchaudio 2.8.0+cu128, Transformers 4.57.6, WhisperX 3.8.6, and an NVIDIA H100.
process_transcriptionpath preserves every timestamped token.Scope and licensing
The first UI version exposes Chinese, English, and Japanese because those languages match VideoLingo's current downstream handling. The model remains configurable for compatible FunASR checkpoints.
The FunASR and SenseVoice repository source code is MIT. Model weights follow the license on each model card; the default SenseVoiceSmall model card links its own model license. No unverified speed or accuracy claims are assumed here.