Skip to content

Add ElevenLabs Scribe transcription #257

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
HadiAssadDiab wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Add ElevenLabs Scribe transcription #257

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,15 +38,15 @@ FreeFlow is a free Mac dictation app inspired by [Wispr Flow](https://wisprflow.
- **Custom shortcuts:** Customize both hold-to-talk and toggle dictation shortcuts. If your toggle shortcut extends your hold shortcut, you can start in hold mode and press the extra modifier keys to latch into tap mode without stopping the recording.
- **Context-aware cleanup:** FreeFlow can read nearby app context so names, terms, and phrases are spelled correctly when you dictate into email, terminals, docs, and other apps.
- **Custom vocabulary:** Add names, jargon, and project-specific words that FreeFlow should preserve during cleanup.
- **OpenAI-compatible providers:** Use Groq by default, or configure a custom model and API URL in settings.
- **Provider choices:** Use Groq by default, select ElevenLabs Scribe v2 for transcription, or configure OpenAI-compatible model IDs and API URLs in settings.

## Edit Mode

Edit Mode lets you highlight existing text and transform it with a spoken instruction, like "make this shorter" or "turn this into bullets." Enable it in settings, then use your normal dictation shortcut on selected text, or choose Manual mode to require an extra modifier key.

## Privacy

There is no FreeFlow server, so FreeFlow does not store or retain your data. The only information that leaves your computer are API calls to your configured transcription and LLM provider.
There is no FreeFlow server, so FreeFlow does not store or retain your data. The only information that leaves your computer are API calls to your selected transcription provider and your configured cleanup/context LLM provider.

## Custom Cleanup

Expand Down Expand Up @@ -82,6 +82,10 @@ FreeFlow can use OpenAI-compatible local or self-hosted providers instead of Gro

Local models are often slower than hosted providers, especially on cold start, long recordings, or busy hardware.

## Using ElevenLabs Scribe

FreeFlow can use ElevenLabs Scribe v2 for speech-to-text while continuing to use your OpenAI-compatible provider for cleanup, Edit Mode, and context. Open Settings, expand Providers, choose ElevenLabs Scribe as the transcription provider, and enter an ElevenLabs API key. Realtime streaming uses Scribe v2 Realtime when the realtime toggle is enabled.

<details>
<summary>Configure longer timeouts for local models</summary>

Expand Down
97 changes: 86 additions & 11 deletions Sources/AppState.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -201,8 +201,10 @@ final class AppState: ObservableObject, @unchecked Sendable {
private let apiKeyStorageKey = "groq_api_key"
private let apiBaseURLStorageKey = "api_base_url"
private let transcriptionModelStorageKey = "transcription_model"
private let transcriptionProviderStorageKey = "transcription_provider"
private let transcriptionAPIURLStorageKey = "transcription_api_url"
private let transcriptionAPIKeyStorageKey = "transcription_api_key"
private let elevenLabsAPIKeyStorageKey = "elevenlabs_api_key"
private let postProcessingModelStorageKey = "post_processing_model"
private let postProcessingFallbackModelStorageKey = "post_processing_fallback_model"
private let contextModelStorageKey = "context_model"
Expand Down Expand Up @@ -301,6 +303,12 @@ final class AppState: ObservableObject, @unchecked Sendable {
}
}

@Published var transcriptionProvider: TranscriptionProvider {
didSet {
UserDefaults.standard.set(transcriptionProvider.rawValue, forKey: transcriptionProviderStorageKey)
}
}

@Published var transcriptionAPIURL: String {
didSet {
persistOptionalAPIValue(transcriptionAPIURL, account: transcriptionAPIURLStorageKey)
Expand All @@ -313,6 +321,12 @@ final class AppState: ObservableObject, @unchecked Sendable {
}
}

@Published var elevenLabsAPIKey: String {
didSet {
persistOptionalAPIValue(elevenLabsAPIKey, account: elevenLabsAPIKeyStorageKey)
}
}

@Published var transcriptionModel: String {
didSet {
UserDefaults.standard.set(transcriptionModel, forKey: transcriptionModelStorageKey)
Expand Down Expand Up @@ -605,7 +619,7 @@ final class AppState: ObservableObject, @unchecked Sendable {
private var pendingManualCommandInvocation = false
private var pendingShortcutStartTask: Task<Void, Never>?
private var pendingShortcutStartMode: RecordingTriggerMode?
private var realtimeService: RealtimeTranscriptionService?
private var realtimeService: RealtimeTranscriptionClient?
private var automaticTerminationDisabled = false
private var activeAudioInterruption: ActiveAudioInterruption?
private var pendingOverlayDismissToken: UUID?
Expand All @@ -622,9 +636,13 @@ final class AppState: ObservableObject, @unchecked Sendable {
let hasCompletedSetup = UserDefaults.standard.bool(forKey: "hasCompletedSetup")
let apiKey = Self.loadStoredAPIKey(account: apiKeyStorageKey)
let apiBaseURL = Self.loadStoredAPIBaseURL(account: "api_base_url")
let transcriptionProvider = TranscriptionProvider(
rawValue: UserDefaults.standard.string(forKey: transcriptionProviderStorageKey) ?? ""
) ?? .openAICompatible
let transcriptionModel = UserDefaults.standard.string(forKey: transcriptionModelStorageKey) ?? Self.defaultTranscriptionModel
let transcriptionAPIURL = Self.loadOptionalStoredAPIValue(account: transcriptionAPIURLStorageKey)
let transcriptionAPIKey = Self.loadStoredAPIKey(account: transcriptionAPIKeyStorageKey)
let elevenLabsAPIKey = Self.loadStoredAPIKey(account: elevenLabsAPIKeyStorageKey)
let postProcessingModel = UserDefaults.standard.string(forKey: postProcessingModelStorageKey) ?? Self.defaultPostProcessingModel
let postProcessingFallbackModel = UserDefaults.standard.string(forKey: postProcessingFallbackModelStorageKey) ?? Self.defaultPostProcessingFallbackModel
let contextModel = UserDefaults.standard.string(forKey: contextModelStorageKey) ?? Self.defaultContextModel
Expand Down Expand Up @@ -724,8 +742,10 @@ final class AppState: ObservableObject, @unchecked Sendable {
self.hasCompletedSetup = hasCompletedSetup
self.apiKey = apiKey
self.apiBaseURL = apiBaseURL
self.transcriptionProvider = transcriptionProvider
self.transcriptionAPIURL = transcriptionAPIURL
self.transcriptionAPIKey = transcriptionAPIKey
self.elevenLabsAPIKey = elevenLabsAPIKey
self.transcriptionModel = transcriptionModel
self.postProcessingModel = postProcessingModel
self.postProcessingFallbackModel = postProcessingFallbackModel
Expand Down Expand Up @@ -831,7 +851,7 @@ final class AppState: ObservableObject, @unchecked Sendable {
}
}

static let defaultAPIBaseURL = "https://api.groq.com/openai/v1"
static let defaultAPIBaseURL = TranscriptionService.defaultOpenAICompatibleBaseURL

private struct StoredShortcutConfiguration {
let hold: ShortcutBinding
Expand Down Expand Up @@ -981,24 +1001,43 @@ final class AppState: ObservableObject, @unchecked Sendable {
}

private var resolvedTranscriptionBaseURL: String {
if transcriptionProvider == .elevenLabs {
return TranscriptionService.defaultElevenLabsBaseURL
}
let trimmed = transcriptionAPIURL.trimmingCharacters(in: .whitespacesAndNewlines)
return trimmed.isEmpty ? apiBaseURL : trimmed
}

private var resolvedTranscriptionAPIKey: String {
let trimmed = transcriptionAPIKey.trimmingCharacters(in: .whitespacesAndNewlines)
if transcriptionProvider == .elevenLabs {
return elevenLabsAPIKey.trimmingCharacters(in: .whitespacesAndNewlines)
}
return trimmed.isEmpty ? apiKey : trimmed
}

func makeTranscriptionService() throws -> TranscriptionService {
try TranscriptionService(
provider: transcriptionProvider,
apiKey: resolvedTranscriptionAPIKey,
baseURL: resolvedTranscriptionBaseURL,
transcriptionModel: transcriptionModel,
language: resolvedTranscriptionLanguage
Comment on lines 1019 to 1025

@coderabbitai coderabbitai Bot Jul 6, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Use the ElevenLabs batch model when the ElevenLabs provider is selected.

Line 1024 passes the shared transcriptionModel, which defaults to whisper-large-v3; that sends the OpenAI-compatible default into the ElevenLabs client instead of Scribe v2.

🐛 Proposed fix 
+    private var resolvedTranscriptionModel: String {
+        switch transcriptionProvider {
+        case .openAICompatible:
+            return transcriptionModel
+        case .elevenLabs:
+            return TranscriptionService.defaultElevenLabsModel
+        }
+    }
+
     func makeTranscriptionService() throws -> TranscriptionService {
         try TranscriptionService(
             provider: transcriptionProvider,
             apiKey: resolvedTranscriptionAPIKey,
             baseURL: resolvedTranscriptionBaseURL,
-            transcriptionModel: transcriptionModel,
+            transcriptionModel: resolvedTranscriptionModel,
             language: resolvedTranscriptionLanguage
         )
     }
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
func makeTranscriptionService() throws -> TranscriptionService {
try TranscriptionService(
provider: transcriptionProvider,
apiKey: resolvedTranscriptionAPIKey,
baseURL: resolvedTranscriptionBaseURL,
transcriptionModel: transcriptionModel,
language: resolvedTranscriptionLanguage
private var resolvedTranscriptionModel: String {
switch transcriptionProvider {
case .openAICompatible:
return transcriptionModel
case .elevenLabs:
return TranscriptionService.defaultElevenLabsModel
}
}
func makeTranscriptionService() throws -> TranscriptionService {
try TranscriptionService(
provider: transcriptionProvider,
apiKey: resolvedTranscriptionAPIKey,
baseURL: resolvedTranscriptionBaseURL,
transcriptionModel: resolvedTranscriptionModel,
language: resolvedTranscriptionLanguage
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@Sources/AppState.swift` around lines 1019 - 1025, In
makeTranscriptionService, the ElevenLabs path is still using the shared
transcriptionModel default instead of the ElevenLabs batch model. Update the
TranscriptionService initialization so that when transcriptionProvider is
ElevenLabs it passes the ElevenLabs-specific Scribe v2 batch model, while
keeping the existing transcriptionModel for other providers. Use the
makeTranscriptionService and TranscriptionService symbols to locate the change.

)
}

private func transcriptionConfigurationErrorMessage() -> String? {
if resolvedTranscriptionAPIKey.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty {
switch transcriptionProvider {
case .openAICompatible:
return "Enter an API key in Settings."
case .elevenLabs:
return "Enter an ElevenLabs API key in Settings."
}
}
return nil
}

private var resolvedTranscriptionLanguage: String? {
let normalized = Self.normalizeTranscriptionLanguage(transcriptionLanguage)
return normalized.isEmpty ? nil : normalized
Expand Down Expand Up @@ -1937,6 +1976,16 @@ final class AppState: ObservableObject, @unchecked Sendable {
: scheduledManualCommandInvocation,
startedAt: t0
) else { return }
if let configurationError = transcriptionConfigurationErrorMessage() {
errorMessage = configurationError
statusText = "Missing API Key"
activeRecordingTriggerMode = nil
currentSessionIntent = .dictation
shortcutSessionController.reset()
playAlertSound(named: "Basso")
scheduleReadyStatusReset(after: 2, matching: ["Missing API Key"])
return
}
guard ensureMicrophoneAccess() else { return }
os_log(.info, log: recordingLog, "mic access check passed: %.3fms", (CFAbsoluteTimeGetCurrent() - t0) * 1000)
applyAudioInterruptionIfNeeded()
Expand Down Expand Up @@ -2043,6 +2092,16 @@ final class AppState: ObservableObject, @unchecked Sendable {
selectionSnapshot: pendingSelectionSnapshot,
manualCommandRequested: pendingManualCommandRequested
) else { return }
if let configurationError = strongSelf.transcriptionConfigurationErrorMessage() {
strongSelf.errorMessage = configurationError
strongSelf.statusText = "Missing API Key"
strongSelf.activeRecordingTriggerMode = nil
strongSelf.currentSessionIntent = .dictation
strongSelf.shortcutSessionController.reset()
strongSelf.playAlertSound(named: "Basso")
strongSelf.scheduleReadyStatusReset(after: 2, matching: ["Missing API Key"])
return
}
strongSelf.shortcutSessionController.beginManual(mode: .toggle)
strongSelf.applyAudioInterruptionIfNeeded()
strongSelf.beginRecording(triggerMode: .toggle)
Expand Down Expand Up @@ -2505,7 +2564,7 @@ final class AppState: ObservableObject, @unchecked Sendable {
/// gets a transcript. Runs the realtime commit and file upload in that
/// strict order to avoid paying for both when realtime succeeds.
private static func resolveRawTranscript(
realtimeService: RealtimeTranscriptionService?,
realtimeService: RealtimeTranscriptionClient?,
fileService: TranscriptionService,
fileURL: URL
) async throws -> String {
Expand Down Expand Up @@ -2849,28 +2908,44 @@ final class AppState: ObservableObject, @unchecked Sendable {
os_log(.info, log: recordingLog, "realtime streaming requested but base URL is empty — skipping")
return
}
let model = realtimeStreamingModel.trimmingCharacters(in: .whitespacesAndNewlines)
let config = RealtimeTranscriptionService.Configuration(
baseURL: trimmedBase,
apiKey: resolvedTranscriptionAPIKey,
model: model,
language: resolvedTranscriptionLanguage
)
let service = RealtimeTranscriptionService(config: config)

let service: RealtimeTranscriptionClient
switch transcriptionProvider {
case .openAICompatible:
let model = realtimeStreamingModel.trimmingCharacters(in: .whitespacesAndNewlines)
let config = RealtimeTranscriptionService.Configuration(
baseURL: trimmedBase,
apiKey: resolvedTranscriptionAPIKey,
model: model,
language: resolvedTranscriptionLanguage
)
service = RealtimeTranscriptionService(config: config)
case .elevenLabs:
let config = ElevenLabsRealtimeTranscriptionService.Configuration(
baseURL: trimmedBase,
apiKey: resolvedTranscriptionAPIKey,
model: TranscriptionService.defaultElevenLabsRealtimeModel,
language: resolvedTranscriptionLanguage
)
service = ElevenLabsRealtimeTranscriptionService(config: config)
}

do {
try service.start()
} catch {
os_log(.error, log: recordingLog, "failed to start realtime service: %{public}@", error.localizedDescription)
return
}
realtimeService = service
audioRecorder.realtimePCM16SampleRate = service.pcmSampleRate
audioRecorder.onPCM16Samples = { [weak service] data in
service?.appendPCM16(data)
}
}

private func tearDownRealtimeService() {
audioRecorder.onPCM16Samples = nil
audioRecorder.realtimePCM16SampleRate = 24_000
realtimeService?.cancel()
realtimeService = nil
}
Expand Down
28 changes: 17 additions & 11 deletions Sources/AudioRecorder.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -95,12 +95,18 @@ final class AudioRecorder: NSObject, ObservableObject, AVCaptureAudioDataOutputS

var onRecordingReady: (() -> Void)?
var onRecordingFailure: ((Error) -> Void)?
/// Fires on the sample-buffer queue with a 24 kHz mono PCM16 chunk for
/// each incoming audio buffer (matching OpenAI Realtime's default PCM
/// input rate). Set before ``startRecording`` to stream audio out-of-band
/// to a realtime transcription socket. The recorder writes a normalized
/// 16 kHz mono PCM16 WAV file independently for upload-based transcription.
/// Fires on the sample-buffer queue with mono PCM16 chunks for each
/// incoming audio buffer. Set before ``startRecording`` to stream audio
/// out-of-band to a realtime transcription socket. The recorder writes a
/// normalized 16 kHz mono PCM16 WAV file independently for upload-based
/// transcription.
var onPCM16Samples: ((Data) -> Void)?
var realtimePCM16SampleRate: Double = 24_000 {
didSet {
guard realtimePCM16SampleRate != oldValue else { return }
pcm16ConverterLock.withLock { $0 = nil }
}
}
private let recordingConverterLock = OSAllocatedUnfairLock<AVAudioConverter?>(initialState: nil)
private let pcm16ConverterLock = OSAllocatedUnfairLock<AVAudioConverter?>(initialState: nil)
private let recordingTargetFormat: AVAudioFormat = {
Expand All @@ -111,14 +117,14 @@ final class AudioRecorder: NSObject, ObservableObject, AVCaptureAudioDataOutputS
interleaved: true
)!
}()
private let pcm16TargetFormat: AVAudioFormat = {
private var pcm16TargetFormat: AVAudioFormat {
AVAudioFormat(
commonFormat: .pcmFormatInt16,
sampleRate: 24_000,
sampleRate: realtimePCM16SampleRate,
channels: 1,
interleaved: true
)!
}()
}
private var readyFired = false
private var failureReported = false
private static let watchdogTimeout: TimeInterval = 2.0
Expand Down Expand Up @@ -187,7 +193,7 @@ final class AudioRecorder: NSObject, ObservableObject, AVCaptureAudioDataOutputS
removeSessionObservers()

let runtimeObserver = NotificationCenter.default.addObserver(
forName: AVCaptureSession.runtimeErrorNotification,
forName: NSNotification.Name.AVCaptureSessionRuntimeError,
object: session,
queue: nil
) { [weak self] notification in
Expand All @@ -199,7 +205,7 @@ final class AudioRecorder: NSObject, ObservableObject, AVCaptureAudioDataOutputS
sessionObservers.append(runtimeObserver)

let interruptionObserver = NotificationCenter.default.addObserver(
forName: AVCaptureSession.wasInterruptedNotification,
forName: NSNotification.Name.AVCaptureSessionWasInterrupted,
object: session,
queue: nil
) { [weak self] notification in
Expand All @@ -208,7 +214,7 @@ final class AudioRecorder: NSObject, ObservableObject, AVCaptureAudioDataOutputS
sessionObservers.append(interruptionObserver)

let interruptionEndedObserver = NotificationCenter.default.addObserver(
forName: AVCaptureSession.interruptionEndedNotification,
forName: NSNotification.Name.AVCaptureSessionInterruptionEnded,
object: session,
queue: nil
) { [weak self] notification in
Expand Down
Loading