Convert to Web AI to run model locally in browser so no need for Python backend

[jasonmayes]

Web AI allows you to run many models in the web browser via WebGPU allowing users to run LLMs like Gemma 3N multimodal and beyond in real-time on video feeds. I was wondering if you could also offer your demo in pure Web AI form to run the model client side either via Google's LiteRT.js or Microsoft ONNX Runtime Web?

[vorojar]

Thanks for the suggestion! I've looked into this and there are some significant challenges:

Model conversion complexity: Qwen3-TTS consists of multiple components (LLM + audio codec decoder), and converting them to ONNX format is non-trivial — many custom operators may not be supported by ONNX Runtime Web yet.

Model size: The two models (Qwen3-TTS-0.6B + CustomVoice) total over 2GB. Users would need to download all of that just to open the page, which is a poor experience.

Performance: Even on my RTX 4090 with CUDA + PyTorch, generating a few seconds of speech already takes noticeable time. In the browser via WebGPU, the same task would likely take 30-60 seconds or more due to the overhead of WebGPU vs native CUDA and the autoregressive nature of TTS generation. On integrated GPUs it could take several minutes, making it impractical for real use.

I think this could become viable in the future as WebGPU matures and smaller distilled TTS models become available, but for now the performance gap is too large. A hosted API is a more practical path for wider accessibility at this point.

Appreciate the idea though — will keep an eye on how the Web AI ecosystem evolves!

[jasonmayes]

You may want to reach out to Xenova (creator of Transformers.js) who has converted a lot of LLMs to ONNX format that Transformers.js uses to execute via WebGPU on the web via Microsoft ONNX Runtime Web. He may be able to guide you there as my background is more on the Google stack as I work over there, but Xenova has more experience with ONNX.

With regards AI UX, yes there is a one time model download, but that is once (can be cached - I made this lib to help with that) and the cost savings / privacy benefits outweigh a 3 minute download on Wifi connection one time for many. One can also implement a hybrid approach where you start with cloud solution and download web ai model in background and then flip when ready locally. So many ways to tackle the UX - much simpler to solve that than the actual conversion :-)

Right now in all my projects I have been using the Kokoro TTS port running via WebGPU as it sounds great and is fast - I just split by sentences to ensure there is not much latency before generation as a paragraph would be too long of course. My issue with this model is I cant make custom voices that are "on brand" for a customer or such as it doesn't support custom voices beyond voice mixing of existing out of the box ones.

Seems that given your benchmarks with CUDA that this new model is very heavy - my apologies I thought it was generating in near real-time so had potential to also run elsewhere at reasonable speeds too! Certainly one to watch for the future though - for context the first stable diffusion model ported to Web AI 2 years ago took 10 seconds, and just 2 years later it now runs in about 460ms! So my fingers and toes are crossed that something similar can happen here for TTS models.

Happy to keep in touch for the future should you see some promise. Appreciate you trying and looking into it.