Skip to content

feat(voice): add OpenAI speech-to-text - #232

Draft
johnmatthewtennant wants to merge 17 commits into
openai-tts-basefrom
add-openai-stt-tts
Draft

feat(voice): add OpenAI speech-to-text#232
johnmatthewtennant wants to merge 17 commits into
openai-tts-basefrom
add-openai-stt-tts

Conversation

@johnmatthewtennant

@johnmatthewtennant johnmatthewtennant commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds OpenAI live speech-to-text on top of the OpenAI speech-output backend.

  • Opens a dedicated Realtime transcription session with intent=transcription and configures gpt-live-transcribe for input transcription.
  • Keeps Berd VAD as the turn-boundary authority, bounds idle pre-roll, commits long continuous speech, and orders completed transcripts by committed item ID.
  • Applies timeouts and cancellation to connection, writes, final commits, and shutdown draining so stopping cannot wait indefinitely on the network.
  • Reconciles every audio batch and server event against mute epochs so pre-mute audio and stale transcripts cannot cross an effective mute edge.
  • Preserves final transcript delivery within the shared shutdown deadline and drains earlier committed results even if the final commit fails.
  • Surfaces input-only readiness independently from speech output and shows the hang-up glyph when a failed call remains in the error state.

Reviewer-reproducible examples

  1. Configure OpenAI in Berd and select OpenAI speech-to-text under Settings > Voice.
  2. Start Voice Conversation and speak through several pauses. Transcript events should arrive while speaking, and each VAD-finalized turn should be delivered once in order.
  3. Mute during an utterance, then unmute and speak again. Audio and transcript events from the prior mute epoch must not be delivered.
  4. Speak continuously for more than 30 seconds. Transcription should commit bounded turns instead of buffering without limit.
  5. End the call during an active utterance or unavailable network. Stop should complete within its bounded shutdown path without the 11-second recognizer-detach error.
  6. Force a setup or API failure. The call control should show the red hang-up glyph and surface the actionable error.

@johnmatthewtennant johnmatthewtennant changed the title feat(voice): add OpenAI speech backends feat(voice): add OpenAI speech-to-text Aug 26, 2026
@johnmatthewtennant
johnmatthewtennant changed the base branch from main to openai-tts-base August 26, 2026 19:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant