onnx-asr comparison

[grikdotnet]

Hi! Thank you for this package.

I briefly measured performance of libcrispasr from the windows build with a python ctypes adapter against ONNX. I used a Parakeet model (parakeet-tdt-0.6b-v3-q8_0.gguf), and a 16-bit quant for onnx.
4070 GPU, I saw it was loaded.

Both CUDA and Vulkan builds show 5 times slower inference comparably to ONNX.

Are windows builds not optimized?

[CrispStrobe]

hm which of the 4 prebuilt libcrispasr releases did you use?

[grikdotnet]

Hi! I tested "libcrispasr-windows-x86_64-vulkan.tar.gz" and "libcrispasr-windows-x86_64-cuda.tar.gz" from v0.6.0.

Speech recognition works correctly, the GPU is consumed, and CPU is not.

For onnx I use the onnxruntime-directml EP with onnx-asr library.

[CrispStrobe]

can you retest with latest version please?

[grikdotnet]

Sure, should I build it from the source?
I could probably make a minimal project here on github so one could reproduce.

[CrispStrobe]

thanks! yes you would have to build from source. next release needs a few other fixes still working...

[grikdotnet]

Compiled with CUDA 13.1

(venv) stt> python misc\benchmark_asr_engines.py --engine both --audio tests\fixtures\en.wav
audio:       tests\fixtures\en.wav (60.000s)
window:      4.000s
warmups:     1
runs:        3

onnx-asr
  load:        2.495s
  runs:        [0.628, 0.628, 0.628]
  mean run:    0.628s for 60.000s audio
  x realtime:  95.54x
  calls/run:   15
  call p50:    42.5ms
  call p95:    44.7ms
  text sample: 'Um when I think about normative questions, it'
ggml_cuda_init: found 1 CUDA devices (Total VRAM: 8187 MiB):
  Device 0: NVIDIA GeForce RTX 4070 Laptop GPU, compute capability 8.9, VMM: yes, VRAM: 8187 MiB
parakeet: vocab=8192  d_model=1024  n_layers=24  n_heads=8  ff=4096  pred=640  joint=640
parakeet: BN folded into conv_dw weights for 24 layers

crisp-asr
  load:        0.424s
  runs:        [2.451, 2.825, 3.387]
  mean run:    2.888s for 60.000s audio
  x realtime:  20.78x
  calls/run:   15
  call p50:    194.5ms
  call p95:    262.8ms
  text sample: 'Um when I think about normative questions is'

stt> python misc\benchmark_asr_engines.py --engine  crisp-asr --audio tests\fixtures\en.wav
audio:       tests\fixtures\en.wav (60.000s)
window:      4.000s
warmups:     1
runs:        3
ggml_cuda_init: found 1 CUDA devices (Total VRAM: 8187 MiB):
  Device 0: NVIDIA GeForce RTX 4070 Laptop GPU, compute capability 8.9, VMM: yes, VRAM: 8187 MiB
parakeet: vocab=8192  d_model=1024  n_layers=24  n_heads=8  ff=4096  pred=640  joint=640
parakeet: BN folded into conv_dw weights for 24 layers

crisp-asr
  load:        0.427s
  runs:        [2.224, 2.291, 2.631]
  mean run:    2.382s for 60.000s audio
  RTF:         0.040
  x realtime:  25.19x
  calls/run:   15
  call p50:    155.9ms
  call p95:    198.5ms
  text sample: 'Um when I think about normative questions is'

I notice higher CPU consumption by a python process as well, from around 5% with ONNX to 17% with CrispASR. But the difference is too big, CPU does not seem to be a bottleneck here.

There was an attempts to implement ASR in GGUF in the past, failing exactly due to the speed issue:
https://github.com/jason-ni/parakeet.cpp
ggml-org/whisper.cpp#3118

[CrispStrobe]

could you paste benchmark_asr_engines.py? which version did you build and with what build params?

btw, quick note in jason-ni's attempt

[grikdotnet]

could you paste benchmark_asr_engines.py?

benchmark_asr_engines.py

CrispAsrAdapter.py

which version did you build and with what build params?
I built the master branch. Cleaned up the build folder with build scripts, sorry.