feat(audio): play streamed TTS audio chunks via Web Audio

[yuuhikaze]

Summary

Frontend counterpart of Open-LLM-VTuber/Open-LLM-VTuber#419. The backend can now deliver each TTS sentence as a stream of small PCM chunks (audio-stream-start / audio-stream-chunk / audio-stream-end) instead of one monolithic base64 WAV. This PR implements the playback side: a sentence starts playing as soon as its first chunks arrive, cutting time-to-first-audio roughly in half on my setup (~2s → <1s) and eliminating the dead air between sentences.

Fully backward compatible: the legacy audio message path is untouched, and against a backend without #419 nothing changes.

Design

utils/stream-audio-player.ts (new) — a singleton around Web Audio:

• Decodes base64 int16 PCM chunks and schedules them gaplessly as AudioBufferSourceNodes against a running time cursor. The cursor only re-anchors to the context clock when genuinely behind, so in-time chunks butt-join sample-exactly with no boundary cracks.
• Deterministic jitter cushion: a stream starts playing (and resumes after an underrun) only once 0.6s of audio is buffered, or the stream already ended, or a 1.5s wall cap expires — a delivery stall shorter than the cushion can never split a word. If an underrun still occurs, the cushion ratchets up by 0.3s (capped at 1.5s) for the rest of the session and logs [stream-audio] underrun #N — cushion → X.Xs, so every machine converges on its own worst-case jitter without manual tuning, and a clean console verifies gap-free playback.
• play() is promise-based and rides the existing audioTaskQueue, so streamed and legacy sentences stay strictly ordered with zero changes to the queue.

Lip-sync — streamed audio bypasses the WAV-file handler, so the player's graph routes through an AnalyserNode and lappmodel.ts takes max(wavHandlerRms, streamingRms) per frame. File-based playback drives the mouth exactly as before; streamed playback now does too.

Integration points — websocket-handler.tsx handles the three new message types (and drops streams that arrive while interrupted/listening); use-audio-task.ts adds addStreamAudioTask mirroring the display-text/expression/talk-motion handling of the file path; audio-manager.ts stops the stream player from the existing interrupt path.

Verification

Typecheck is clean relative to main's baseline. Live-tested for several days against the streaming backend + vLLM-Omni/Qwen3-TTS: first audio lands in under a second, sentences join without gaps, mid-word artifacts are eliminated by the cushion (validated by an instrumented console — zero underrun logs in normal operation), interrupts stop playback cleanly, and legacy file-based payloads (silent display payloads, non-streaming engines) continue to work interleaved with streams.

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features
  • Added streamed audio playback with chunked delivery for more continuous narration.
  • Enabled real-time subtitle/chat updates during streamed playback, including automatic Live2D “Talk”/expression triggering.
• Bug Fixes
  • Stopping or interrupting audio now fully halts both regular and streamed playback.
  • Lip-sync now better follows streamed audio loudness by using streaming RMS for more accurate animation timing.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 9fe9c508-62e0-43b3-ae98-55dd33502255

📥 Commits

Reviewing files that changed from the base of the PR and between a6f118a and 880fc33.

📒 Files selected for processing (1)

• src/renderer/src/utils/stream-audio-player.ts

🚧 Files skipped from review as they are similar to previous changes (1)

• src/renderer/src/utils/stream-audio-player.ts

---

📝 Walkthrough

Walkthrough

Adds streamed PCM audio playback, routes stream events through WebSocket and audio-task handling, stops streamed playback with other audio state, and incorporates streaming RMS into Live2D lip-sync updates.

Changes

Streaming audio pipeline

Layer / File(s)	Summary
Stream player lifecycle and playback src/renderer/src/utils/stream-audio-player.ts	Adds PCM decoding, buffering, gapless scheduling, jitter handling, interruption, and live RMS measurement.
WebSocket and audio-task wiring src/renderer/src/services/websocket-service.tsx, src/renderer/src/services/websocket-handler.tsx, src/renderer/src/hooks/utils/use-audio-task.ts	Extends stream message types, handles stream lifecycle events, and queues streamed playback alongside existing audio tasks.
Interruption and lip-sync integration src/renderer/src/utils/audio-manager.ts, src/renderer/WebSDK/src/lappmodel.ts	Stops active streams during audio resets and combines streaming RMS with existing lip-sync input.

Estimated code review effort: 4 (Complex) | ~60 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Server
  participant WebSocketHandler
  participant UseAudioTask
  participant StreamAudioPlayer
  participant Live2D
  Server-->>WebSocketHandler: audio-stream-start
  WebSocketHandler->>StreamAudioPlayer: openStream(stream_id)
  WebSocketHandler->>UseAudioTask: addStreamAudioTask(options)
  Server-->>WebSocketHandler: audio-stream-chunk
  WebSocketHandler->>StreamAudioPlayer: pushChunk(stream_id, chunk)
  Server-->>WebSocketHandler: audio-stream-end
  WebSocketHandler->>StreamAudioPlayer: endStream(stream_id)
  UseAudioTask->>StreamAudioPlayer: play(stream_id)
  Live2D->>StreamAudioPlayer: getStreamingLipSyncRms()
  StreamAudioPlayer-->>Live2D: RMS value

Loading

Poem

A rabbit hears the bits arrive,
In little chunks they hop alive—
Streamed tunes flow warm, then fade away,
And whiskers sync to every play. 🐰🎧

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: streamed TTS audio chunk playback via Web Audio.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

Warning

There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.

🔧 ESLint

If the error stems from missing dependencies, add them to the package.json file. For unrecoverable errors (e.g., due to private dependencies), disable the tool in the CodeRabbit configuration.

src/renderer/src/utils/stream-audio-player.ts

ESLint skipped: missing config or dependency (missing-dependency). The ESLint configuration references a package that is not available in the sandbox.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

src/renderer/src/services/websocket-handler.tsx (1)

319-319: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Add addStreamAudioTask to handleWebSocketMessage deps. The callback uses addStreamAudioTask, but it isn’t included in the dependency array, so the hook can capture a stale reference and keep failing react-hooks/exhaustive-deps.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/renderer/src/services/websocket-handler.tsx` at line 319, The
handleWebSocketMessage callback is using addStreamAudioTask but the hook
dependency list does not include it, which can leave a stale reference captured
by useCallback/useEffect dependencies. Update the dependency array for
handleWebSocketMessage to include addStreamAudioTask alongside the existing
dependencies so react-hooks/exhaustive-deps is satisfied and the latest function
reference is always used.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@src/renderer/src/services/websocket-handler.tsx`:
- Line 319: The handleWebSocketMessage callback is using addStreamAudioTask but
the hook dependency list does not include it, which can leave a stale reference
captured by useCallback/useEffect dependencies. Update the dependency array for
handleWebSocketMessage to include addStreamAudioTask alongside the existing
dependencies so react-hooks/exhaustive-deps is satisfied and the latest function
reference is always used.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 22270fc8-4436-41d5-b231-861c609599b1

📥 Commits

Reviewing files that changed from the base of the PR and between d176e7d and baa9562.

📒 Files selected for processing (6)

• src/renderer/WebSDK/src/lappmodel.ts
• src/renderer/src/hooks/utils/use-audio-task.ts
• src/renderer/src/services/websocket-handler.tsx
• src/renderer/src/services/websocket-service.tsx
• src/renderer/src/utils/audio-manager.ts
• src/renderer/src/utils/stream-audio-player.ts

[yuuhikaze]

Review feedback addressed in a6f118a: addStreamAudioTask added to handleWebSocketMessage's dependency array, consistent with the file's existing convention (addAudioTask was already listed). Typecheck unchanged vs baseline.

[yuuhikaze]

Pushed three follow-up fixes from extended live testing:

• 880fc33 / c28d941 — real-world stutter fix. The scheduling loop is greedy (every delivered chunk is scheduled into WebAudio immediately), so an empty delivery queue is the normal state when the TTS server delivers at ~1× realtime. The player nevertheless counted it as an underrun, paused scheduling for a cushion refill while the already-scheduled audio drained — manufacturing the exact gap it was meant to prevent — and ratcheted the session cushion toward its cap, delaying every later sentence. It now checks the scheduled runway (nextStartTime - currentTime) first: while runway remains it simply waits for the next chunk (sample-exact butt-join on arrival); the ratchet only fires on genuine starvation. Verified on a machine where TTS delivery hovers at ~1× realtime: phantom underruns went from 6/session to 0, with a single genuine underrun correctly ratcheting the cushion once.
• 880fc33 also persists the learned cushion in localStorage, so each machine converges on its own worst-case jitter once, instead of re-paying one first-turn underrun every session.
• dbdc463 — release the audio device when playback stops. The shared AudioContext kept rendering silence after a turn, holding the OS output stream open until the browser's silence timeout. Suspending on stopAll() (resumed transparently by the next play()) lets stream-activity-keyed OS audio policies — e.g. WirePlumber role-based ducking — react the moment speech ends.

Typecheck remains at the pre-existing baseline; no wire-protocol changes.