diff --git a/CHANGELOG.md b/CHANGELOG.md index 3bc4ddb..f17d6be 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,61 @@ All notable changes to bestASR are documented here. The format follows ### Added +- **Apple Speech backend (#121)**: `apple-speech` — the OS-native backend + (Speech.framework's `SpeechAnalyzer` / `SpeechTranscriber`, macOS 26+). The + only backend in the pool with no third-party dependency and no HuggingFace + weight pin: the recognizer ships with the operating system, so **its version + IS the OS version**. That is a claim about *provenance*, not about network + traffic — a locale whose asset is absent is still downloaded from Apple on + first use, as `ja_JP` was here. Registered unconditionally; `isAvailable()` is a pure + macOS-26 gate, so older hosts report it as not installed rather than failing + to build. One grid row (`apple-speech · speechanalyzer/system`), priority 1, + **`verified: false`** — the API is live-probed but the row is not yet measured + on this project's corpora, and `estMemoryGB` 2.0 is an explicit **unmeasured + placeholder**. + + Three behaviors were established by live probing, not from the headers, and + are load-bearing enough to record here: + + - The `results` `AsyncSequence` must be consumed **before** audio is fed, or + results are lost. + - A missing locale asset surfaces as `SFSpeechErrorDomain Code=3 "Audio + format is not supported"` — a **misleading message**. Proved not to be a + format problem (three corpora byte-identical in format behaved differently; + the same ja file transcribed fine under `en_US`; installing the ja asset + fixed it immediately). The engine therefore checks `installedLocales` and + downloads on demand, and never diagnoses from the error text. + - The framework exposes **no determinism knob** (`Speech.swiftinterface` has + zero hits for temperature/greedy/beam/sampling/seed), so this backend + records `flag-not-consumed` for `decode_deterministic` (#118) rather than + claiming an enforcement it cannot perform. + + Language is **required**: `SpeechTranscriber` selects its model by locale and + has no auto-detect mode, so an unresolved language throws instead of guessing + — a ja file decoded under `en_US` yields measured garbage, and a benchmark + harness must not admit a silent guess into its evidence base. `zh` maps to + `zh_TW` to match the project's Common Voice zh-TW corpora. Per-segment + confidence is **derived** (the minimum over Apple's per-run values), so a + confident run cannot mask a weak one inside the same cue. + + **First measured numbers** (M5 Max, macOS 27, single corpus per language — + indicative, not a certification; the row stays `verified: false` until a full + sweep): + + | corpus | metric | apple-speech | whisperkit large-v3-turbo | + |---|---|---|---| + | `cv-zhtw-2` (zh) | CER | **13.9 %** @ 63–65× realtime | 12.5 % @ 7.1× | + | `fleurs-ja-1` (ja) | CER | **10.9 %** @ 85× realtime | 10.6 % @ 6.9× | + | `librispeech-testclean-1` (en) | WER | **4.3 %** @ 58× realtime | not yet measured | + | `librispeech-devclean-1` (en) | WER | **4.1 %** @ 45× realtime | not yet measured | + + So on this machine Apple lands within ~1.4 pp of Whisper large-v3-turbo on + zh and ~0.3 pp on ja, at roughly **9–12× the speed** and with no model to + download or hold resident. Reported `peak-GB` is 0.00 because recognition + runs in Apple's out-of-process daemon — the harness's in-process sampler + cannot see that footprint, so this backend's memory column is **not + comparable** with the in-process backends'. + - **Release sweep (#109)**: `scripts/release-sweep.sh` — the per-version deterministic measurement snapshot. Runs the priority-1 runnable candidates (default ceiling; `--all-grid` widens to the whole grid) over the **canonical @@ -53,6 +108,53 @@ All notable changes to bestASR are documented here. The format follows ### Fixed +- **`--backend apple-speech` was silently substituted (#121)**: `Router`'s + backend-membership list omitted the new backend, so an explicit `--backend + apple-speech` was discarded and another backend's output was written under + the user's chosen name. The accompanying `unavailable` warning was both + **false** (`isAvailable()` returns true on macOS 26+) and invisible without + `--explain`. Measured before the fix: the command produced mlx-audio Whisper + output in **Simplified** Chinese; after it, genuine Apple Speech output in + **Traditional**. `ModelRegistry.isRunnableModel` gained the same membership so + `--model system` resolves. + +- **`zh-Hans` selected the Traditional model (#121 verify)**: language subtags + were read **by position**, so the script subtag in `zh-Hans` was taken for a + region, matched no locale, and fell through to the `zh` preference — a user + who explicitly asked for **Simplified** silently received **Traditional**. + Subtags are now classified by **shape** (BCP-47: script is 4 alpha, region is + 2 alpha or 3 digits), with `Hans → CN` / `Hant → TW`; an explicit region still + outranks a contradicting script (`zh-Hans-TW` → `zh_TW`). + +- **Provenance recorded the requested language, not the one that ran (#121 + verify)**: asking for `pt-BR` on a system shipping only `pt_PT` produced a + measurement labelled `pt-BR`. The resolved locale is now recorded, in + hyphenated form — Apple's underscored `zh_TW` would otherwise defeat + `LanguageResolver.baseSubtag` (which splits on `-` only) and silently disable + the D7 Traditional/Simplified fold for this backend's zh scoring. + +- **Locale asset downloaded before the audio file was validated (#121 + verify)**: `transcribe --language ja /nonexistent.wav` could fetch a + multi-hundred-MB asset and only then report the missing file. The file is now + opened first. Additionally, a nil installation request no longer falls through + to Apple's misleading `"Audio format is not supported"`: `installedLocales` is + re-checked after the attempt and a still-absent asset fails with that + explanation. + +- **Recognized speech could vanish silently (#121 verify)**: a result with + unusable timing was dropped. Because the WER denominator is the reference word + count and does not move, a dropped correct word adds a deletion while a dropped + garbage word removes an insertion — the measured rate shifts in an unsignposted + direction. Non-empty text with unusable timing now **fails the run**. Only + truly-empty text is dropped; whitespace-only segments are kept, since + `Engine.transcribe` joins with no separator and discarding the separator would + glue neighbouring words. + +- **Orphaned collector task (#121 verify)**: the unstructured `Task` reading + `transcriber.results` was neither awaited nor cancelled when the analyzer + threw, pinning the transcriber and its buffers for the life of the process — + which accumulates across a benchmark sweep. It is now cancelled on every exit. + - **The compare stage no longer renders baseline values unescaped (#117)**: `scripts/lib/baseline-compare.py` interpolated `corpus` / `language` / `metric` straight into the pass/fail lines it prints, and that stdout is the diff --git a/README.md b/README.md index 794d8f6..c18f469 100644 --- a/README.md +++ b/README.md @@ -26,7 +26,14 @@ recommendations**, **stable fallback**, and **clear explanations**. ## Requirements - Apple Silicon Mac (arm64) — Intel Macs and Rosetta are not supported -- macOS 14 (Sonoma) or later +- macOS 14 (Sonoma) or later to **run** +- **Xcode 26 / the macOS 26 SDK to _build_** (since #121). The `apple-speech` + backend references `SpeechAnalyzer` / `SpeechTranscriber`, which exist only in + that SDK. `@available` keeps the binary running on macOS 14 — it does **not** + let an older toolchain compile the file, so building with Xcode 16 fails at + type-checking with "cannot find type 'SpeechAnalyzer' in scope". Run-time + behavior below macOS 26 is unaffected: `isAvailable()` reports the backend as + not installed and it is skipped with a note. - Swift 6.3+ (the Xcode built-in toolchain works). **Known issue**: the swiftly-managed swift-6.2.4-RELEASE toolchain crashes the compiler (signal 6, SIL specialization in the swift-transformers dependency) diff --git a/Sources/BestASRKit/CommandCore.swift b/Sources/BestASRKit/CommandCore.swift index 6d11a4e..d4997d4 100644 --- a/Sources/BestASRKit/CommandCore.swift +++ b/Sources/BestASRKit/CommandCore.swift @@ -60,6 +60,11 @@ public struct CommandCore: Sendable { var engines: [any Engine] = [ WhisperKitEngine(), WhisperCppEngine(), ParakeetEngine(), ChineseFamilyEngine.paraformer(), ChineseFamilyEngine.sensevoice(), + // #121: registered unconditionally. The struct is constructible at + // the package's macOS 14 deployment target; its isAvailable() is + // the pure macOS-26 gate, so an older host simply reports it as + // not installed instead of failing to build or link. + AppleSpeechEngine(), ] for entry in ExternalEngineRegistry().engines { engines.append(ExternalProcessEngine(id: entry.id, command: entry.command)) @@ -642,6 +647,9 @@ public struct CommandCore: Sendable { let liveFamilies = [ ModelGrid.backendFluidParakeet, ModelGrid.backendFluidParaformer, ModelGrid.backendFluidSenseVoice, + // #121: the OS-native backend is bundled too — a catalog row the + // catalog never prints is undiscoverable. + ModelGrid.backendAppleSpeech, ] for backend in liveFamilies { for row in ModelGrid.rows(backend: backend, priorityCeiling: nil) { diff --git a/Sources/BestASRKit/Engines/AppleSpeechEngine.swift b/Sources/BestASRKit/Engines/AppleSpeechEngine.swift new file mode 100644 index 0000000..42fc3d9 --- /dev/null +++ b/Sources/BestASRKit/Engines/AppleSpeechEngine.swift @@ -0,0 +1,468 @@ +import AVFoundation +import Foundation +import Speech + +/// The OS-native ASR backend (#121): Speech.framework's `SpeechAnalyzer` + +/// `SpeechTranscriber`, introduced in macOS 26. It is the only backend in the +/// pool with no third-party dependency and no HuggingFace weight pin — the +/// recognizer ships with the operating system, so its version IS the OS version. +/// +/// That is NOT the same as "downloads nothing": a locale whose asset is absent +/// is fetched from Apple on first use, and this project measured exactly that +/// (`ja_JP` was missing on the development machine and had to be installed). +/// The supply-chain claim is about provenance — Apple rather than a model hub — +/// not about network traffic. +/// +/// API mapping (live-probed on macOS 27 / Xcode 26.6 → BestASRKit): +/// +/// | Speech.framework | here | +/// |------------------------------------------------------|-----------------------------| +/// | `SpeechTranscriber(locale:…:attributeOptions:)` | per-call, keyed by language | +/// | `SpeechAnalyzer(modules:)` + `analyzeSequence(from:)` | `transcribeRaw` | +/// | `SpeechTranscriber.Result.range` (`CMTimeRange`) | `RawSegment.start`/`.end` | +/// | `Result.text` (`AttributedString`) | `RawSegment.text` | +/// | per-run `ConfidenceAttribute` | `RawSegment.confidence` (min)| +/// | `AssetInventory.assetInstallationRequest(supporting:)`| `transcribeRaw` preflight | +/// +/// Three things about this framework are load-bearing and were established by +/// live probing rather than from the headers: +/// +/// 1. **Consume `results` BEFORE feeding audio.** `transcriber.results` is an +/// `AsyncSequence`; starting the collector after `analyzeSequence` loses +/// results. The collector Task below is started first, deliberately. +/// 2. **The locale asset must be installed.** With the asset missing, Apple +/// throws `SFSpeechErrorDomain Code=3 "Audio format is not supported"` — a +/// MISLEADING message. It was proved not to be a format problem: three +/// corpora byte-identical in format (1ch/16 kHz/Int16) behaved differently, +/// and the same ja file transcribed fine under `en_US`. After installing the +/// ja asset the same file+locale worked immediately. **Never diagnose asset +/// availability from the error text — check `installedLocales`.** +/// 3. **There is no determinism knob.** The whole `Speech.swiftinterface` has +/// zero hits for temperature/greedy/beam/sampling/seed, so this backend +/// ignores `options.deterministicDecode`. `CommandCore`'s persistence gate +/// already records `decode_deterministic` only for the whisper-family +/// backends that actually consume it, so nothing here has to lie. +/// +/// Availability is a pure OS-version gate: the struct itself carries no +/// `@available`, so `CommandCore.live()` can hold it unconditionally at the +/// package's macOS 14 deployment target, and every macOS-26-only call lives in +/// an `@available` helper below. `isAvailable()` never touches the asset +/// inventory — availability detection is specified as graceful AND cheap, and +/// an asset download takes tens of seconds. +public struct AppleSpeechEngine: Engine { + public let id: BackendID = .appleSpeech + + public init() {} + + // MARK: - Diagnostics + + static let backendName = BackendID.appleSpeech.rawValue + + /// Thrown when transcription is requested below macOS 26. Held as a + /// constant so the branch this machine cannot execute is still assertable. + static let unsupportedOSMessage = + "requires macOS 26 or later — SpeechAnalyzer / SpeechTranscriber " + + "(Speech.framework) do not exist on this system. Run `bestasr " + + "list-backends` to see which backends are usable here." + + /// The `--language` requirement, stated once. SpeechTranscriber selects a + /// model by locale and has no autodetect mode, so an unresolved language + /// cannot be papered over: guessing English produced measured garbage on + /// Japanese audio (#121 probe), and a benchmark harness must not let a + /// silent guess enter its evidence base. + static let missingLanguageMessage = + "requires an explicit transcription language: SpeechTranscriber selects " + + "its model by locale and has no auto-detect mode, so a guessed locale " + + "silently degrades output (a ja file decoded under en_US yields " + + "garbage). Re-run with an explicit --language (e.g. en, ja, zh)." + + // MARK: - Locale resolution + + /// Which region to pick for a base language that Apple ships in several. + /// Policy, not fact — hence a hand-maintained table with the reasoning: + /// + /// - `en` → US and `ja` → JP and `zh` → TW are bestASR's benchmark triple. + /// zh is the load-bearing one: the zh corpora are Common Voice zh-TW + /// (`RegressionBaselineTests` requires corpus names to start `cv-zhtw-`), + /// and the lexicographic fallback would otherwise pick zh_CN. + /// - The rest resolve to the language's own national variant (de_DE, fr_FR, + /// it_IT, pt_PT, es_ES) rather than a secondary region (de_AT, fr_BE, …). + /// + /// Base languages Apple ships in exactly ONE region need no entry — the + /// resolver finds the single candidate on its own, so this table stays as + /// small as the genuine ambiguity. `AppleSpeechGridTests` locks every entry + /// against the probed supported set so a typo cannot demote silently. + static let preferredRegions: [String: String] = [ + "en": "US", "ja": "JP", "zh": "TW", + "de": "DE", "es": "ES", "fr": "FR", "it": "IT", "pt": "PT", + ] + + /// Which region carries a given script, for tags that name a script rather + /// than a country. Only entries whose mapping is unambiguous belong here. + /// + /// This table exists because a subtag's POSITION does not identify it: in + /// `zh-Hans` the second subtag is a script, in `zh-CN` it is a region. + /// Reading position-2 as "the region" made `zh-Hans` miss every candidate + /// and fall through to the `zh` preference — so a user who explicitly asked + /// for **Simplified** silently received the **Traditional** model. Verified + /// by probe before the fix: `zh-Hans` → `zh_TW`. + static let scriptRegions: [String: String] = ["hans": "CN", "hant": "TW"] + + /// A parsed language tag. `script` and `region` are distinguished by SHAPE + /// (BCP-47: script is 4 alpha, region is 2 alpha or 3 digits), never by + /// position, so `zh-Hans`, `zh-CN` and `zh-Hans-CN` each parse correctly. + struct ParsedTag { + var base: String + var script: String? + var region: String? + } + + /// Shape-directed subtag parse. Lowercased throughout — `LanguageResolver` + /// already lowercases the request, and locale identifiers are lowercased + /// on the way in, so comparisons never depend on the caller's casing. + static func parseTag(_ tag: String) -> ParsedTag? { + let parts = tag.lowercased().split(whereSeparator: { $0 == "-" || $0 == "_" }) + guard let first = parts.first, first.count >= 2, first.allSatisfy(\.isLetter) else { + return nil + } + var parsed = ParsedTag(base: String(first), script: nil, region: nil) + for part in parts.dropFirst() { + if part.count == 4, part.allSatisfy(\.isLetter) { + parsed.script = parsed.script ?? String(part) + } else if (part.count == 2 && part.allSatisfy(\.isLetter)) + || (part.count == 3 && part.allSatisfy(\.isNumber)) { + parsed.region = parsed.region ?? String(part) + } + // Variants/extensions are not selectors for this API — ignored. + } + return parsed + } + + /// Apple's `zh_TW` in the hyphenated BCP-47 form the rest of this project + /// parses. See the call site in `transcribe` for why this is load-bearing. + static func bcp47(_ localeIdentifier: String) -> String { + localeIdentifier.replacingOccurrences(of: "_", with: "-") + } + + /// Map a bestASR language tag onto one of Apple's supported locale + /// identifiers. Pure and total over its inputs (the supported set is passed + /// in, never queried here) so the mapping is testable on any OS and the + /// runtime truth still comes from `SpeechTranscriber.supportedLocales`. + /// + /// Resolution order, most specific first: + /// 1. an explicitly requested REGION that Apple supports (`zh-CN` → zh_CN) + /// 2. the region implied by an explicitly requested SCRIPT + /// (`zh-Hans` → CN → zh_CN) + /// 3. the documented preference for the base language (`zh` → zh_TW) + /// 4. the lexicographically smallest remaining candidate — deterministic, + /// so a future OS reordering its list cannot change a benchmark result + /// + /// Region outranks script because a region is the more specific claim and + /// the combination is legitimate (`zh-Hans-TW` = Simplified as written in + /// Taiwan; Apple ships no such locale, and the explicit region is the + /// better-evidenced half of the request). + /// + /// A requested region Apple does NOT ship degrades within the same language + /// (`pt-BR` → pt_PT when only pt_PT exists): the language decides whether + /// output is usable at all, the region only shades it. The caller is told + /// which locale actually ran — see `transcribe`, which records the RESOLVED + /// identifier rather than the requested tag. + /// + /// Throws — never falls back across languages. That fallback is precisely + /// the failure this backend was measured producing. + static func resolveLocaleIdentifier(language: String?, supported: [String]) throws -> String { + guard let requested = LanguageResolver.resolve(language), + let tag = parseTag(requested) + else { + throw TranscriptionError(backend: backendName, message: missingLanguageMessage) + } + + let candidates = supported.filter { Self.parseTag($0)?.base == tag.base } + guard !candidates.isEmpty else { + throw TranscriptionError( + backend: backendName, + message: "no Apple Speech locale supports language '\(requested)'. " + + "SpeechTranscriber advertises \(supported.count) locale(s) on this " + + "system; run a supported language instead — this backend will not " + + "substitute another language's model.") + } + func match(region: String?) -> String? { + guard let region = region?.lowercased() else { return nil } + return candidates.first { Self.parseTag($0)?.region == region } + } + if let exact = match(region: tag.region) { return exact } + if let script = tag.script, let hit = match(region: scriptRegions[script]) { return hit } + if let hit = match(region: preferredRegions[tag.base]) { return hit } + // `candidates` is non-empty, so `min` is total. + return candidates.min()! + } + + // MARK: - Confidence + + /// One confidence per segment, derived from Apple's PER-RUN values. + /// + /// `SpeechTranscriber` attaches `transcriptionConfidence` to runs inside a + /// result's `AttributedString`, while `RawSegment.confidence` is a single + /// number per segment — so this value is DERIVED, not reported, and the + /// aggregation is a choice worth stating: **the minimum over the runs.** + /// + /// Why min rather than a mean: + /// - It cannot overstate a cue. A mean lets a confident run mask a weak + /// one inside the SAME cue, and the cue is the unit a reader reads and + /// the hallucination filter drops. + /// - bestASR's only consumer of the field is a degradation detector + /// (`HallucinationFilter.full`), and a lower bound is the right shape + /// for a detector. + /// - It needs no weighting policy. A weighted mean would force a choice + /// between character count, run count, or audio duration per run — + /// Apple attaches confidence to runs, not to time — and any of those + /// would layer an unmeasured guess on top of a derived number. + /// + /// A run missing the attribute makes the whole segment nil: `confidence` is + /// Optional precisely so "unknown" is representable, and an invented number + /// would be worse than an absent one. + /// + /// Note the field is backend-defined across this project: WhisperKit stores + /// `avgLogprob` there, FluidAudio Parakeet stores a 0…1 confidence. This + /// backend follows Parakeet. `HallucinationFilter`'s joint silence rule + /// cannot misfire on the difference — it also requires `noSpeechProb`, + /// which is Whisper-specific and nil here. + /// The value is a MINIMUM OBSERVED RUN CONFIDENCE, not a calibrated + /// per-cue probability: more runs means more chances to hit a low one, so + /// long cues score systematically lower. Do not threshold it across + /// backends without re-establishing what it means there. + static func aggregateConfidence(_ perRun: [Double?]) -> Double? { + guard !perRun.isEmpty else { return nil } + var lowest = Double.infinity + for value in perRun { + // A missing attribute makes the whole segment nil — the field is + // Optional precisely so "unknown" is representable. + guard let value else { return nil } + // A non-finite or out-of-range value is not a low confidence, it + // is a broken one. Reporting NaN as a number would let it sort and + // compare nonsensically downstream; nil says "unknown", which is + // the truth. + guard value.isFinite, value >= 0, value <= 1 else { return nil } + lowest = Swift.min(lowest, value) + } + return lowest + } + + /// Per-run `transcriptionConfidence` values, in run order; nil for a run + /// that carries no such attribute. + @available(macOS 26.0, *) + static func runConfidences(_ text: AttributedString) -> [Double?] { + text.runs.map { run in + run.attributes[AttributeScopes.SpeechAttributes.ConfidenceAttribute.self] + } + } + + // MARK: - Engine + + public func isAvailable() async -> Bool { + // Pure version gate — no asset inventory query, no download. See the + // type doc: `list-backends` calls this for every backend. + if #available(macOS 26.0, *) { + return true + } else { + return false + } + } + + public func transcribeRaw( + audioPath: String, options: TranscribeOptions + ) async throws -> RawTranscription { + guard #available(macOS 26.0, *) else { + throw TranscriptionError(backend: id.rawValue, message: Self.unsupportedOSMessage) + } + return try await Self.transcribe(audioPath: audioPath, options: options) + } + + // MARK: - macOS 26+ implementation + + @available(macOS 26.0, *) + private static func transcribe( + audioPath: String, options: TranscribeOptions + ) async throws -> RawTranscription { + let supported = await SpeechTranscriber.supportedLocales.map(\.identifier) + let identifier = try resolveLocaleIdentifier( + language: options.language, supported: supported) + let locale = Locale(identifier: identifier) + + let transcriber = SpeechTranscriber( + locale: locale, + transcriptionOptions: [], + reportingOptions: [], + // Both attributes are required: audioTimeRange for cue times, + // transcriptionConfidence for the per-run confidence the segment + // value is derived from. Omitting either silently drops that field. + attributeOptions: [.audioTimeRange, .transcriptionConfidence]) + + // Open and validate the audio BEFORE any asset install. Installing + // first meant `transcribe --language ja /nonexistent.wav` downloaded a + // multi-hundred-MB locale asset and only then reported the missing + // file — an expensive, networked failure for a cheap, local error. + let file: AVAudioFile + do { + file = try AVAudioFile(forReading: URL(fileURLWithPath: audioPath)) + } catch { + throw TranscriptionError( + backend: backendName, + message: "\(audioPath): \(error.localizedDescription)", + underlying: error) + } + // The engine seam already normalized to 16 kHz mono (#36); this is the + // true wall-clock length of what the analyzer will read. + let duration = file.length > 0 + ? Double(file.length) / file.processingFormat.sampleRate : nil + + try await installAssetIfNeeded(locale: locale, transcriber: transcriber) + + let results: [SpeechTranscriber.Result] + do { + // `results` is an AsyncSequence — the collector MUST be running + // before audio is fed, or results are lost (see type doc, item 1). + let collector = Task { () -> [SpeechTranscriber.Result] in + var collected: [SpeechTranscriber.Result] = [] + for try await result in transcriber.results { collected.append(result) } + return collected + } + // The collector is UNSTRUCTURED, so it does not inherit + // cancellation and is not awaited on the throwing paths below. If + // the analyzer fails, an un-cancelled collector keeps iterating a + // sequence that may never terminate, pinning the transcriber and + // its buffers for the life of the process — across a benchmark + // sweep that accumulates. `defer` covers every exit, including the + // success path (where cancelling an already-finished Task is a + // no-op) and cancellation of our own parent task. + defer { collector.cancel() } + let analyzer = SpeechAnalyzer(modules: [transcriber]) + _ = try await analyzer.analyzeSequence(from: file) + try await analyzer.finalizeAndFinishThroughEndOfInput() + results = try await collector.value + } catch { + throw TranscriptionError( + backend: backendName, + message: "\(audioPath) (locale \(identifier)): \(error.localizedDescription)", + underlying: error) + } + + return RawTranscription( + segments: try segments(from: results), + // The RESOLVED locale, not the requested tag. Recording the request + // would attribute a measurement to a locale that never ran: ask for + // `pt-BR` on a system shipping only `pt_PT` and the row would claim + // pt-BR accuracy for a pt-PT model. Provenance must name what + // executed (design D2). + // + // HYPHENATED, because Apple's identifiers are underscored and every + // language predicate in this project routes through + // `LanguageResolver.baseSubtag`, which splits on "-" ONLY. Emitting + // `zh_TW` verbatim would make `baseSubtag` return "zh_tw", so + // `isChinese` turns false and the D7 Traditional/Simplified fold + // silently stops applying to this backend's zh scoring (#34). + language: Self.bcp47(identifier), + duration: duration) + } + + /// Asset preflight. Lives here and NOT in `isAvailable()` because a + /// download takes tens of seconds while availability must stay cheap. + /// + /// The check is `installedLocales`, never the error text: a missing asset + /// surfaces as `SFSpeechErrorDomain Code=3 "Audio format is not supported"`, + /// which sends the reader hunting a format bug that does not exist. + @available(macOS 26.0, *) + private static func installAssetIfNeeded( + locale: Locale, transcriber: SpeechTranscriber + ) async throws { + func isInstalled() async -> Bool { + // Compare canonically: an inventory that reports `zh-TW` while the + // request holds `zh_TW` would otherwise look permanently missing + // and re-trigger a download on every single call. + let wanted = bcp47(locale.identifier).lowercased() + return await SpeechTranscriber.installedLocales + .contains { bcp47($0.identifier).lowercased() == wanted } + } + guard await !isInstalled() else { return } + do { + if let request = try await AssetInventory.assetInstallationRequest( + supporting: [transcriber]) { + try await request.downloadAndInstall() + } + } catch { + throw TranscriptionError( + backend: backendName, + message: "failed to install the on-device speech asset for locale " + + "'\(locale.identifier)': \(error.localizedDescription)", + underlying: error) + } + // Re-check instead of trusting either branch. A nil request means "the + // system has nothing to install", which CONTRADICTS the inventory that + // just said the locale is missing; proceeding on that contradiction is + // exactly how the misleading `Code=3 "Audio format is not supported"` + // reaches the user — the error this preflight exists to prevent. A + // completed download that still leaves the locale absent is equally + // unusable. Both fail here, named for what they are. + guard await isInstalled() else { + throw TranscriptionError( + backend: backendName, + message: "the on-device speech asset for locale '\(locale.identifier)' is " + + "still not installed after the install attempt. Transcribing now would " + + "surface Apple's misleading \"Audio format is not supported\" error, " + + "which is really a missing-asset report. Check Settings ▸ General ▸ " + + "Language & Region, or free disk space, and retry.") + } + } + + /// One `SpeechTranscriber.Result` becomes one raw segment. + /// + /// Text is taken verbatim (`String(result.text.characters)`), including any + /// leading space: `Engine.transcribe` joins segment texts with NO + /// separator, so trimming here would glue words together across cues and + /// inflate this backend's measured WER — the same contract WhisperKit's + /// segments and #35's parakeet mapper follow. + @available(macOS 26.0, *) + static func segments(from results: [SpeechTranscriber.Result]) throws + -> [RawTranscription.RawSegment] + { + try results.enumerated().compactMap { index, result in + let text = String(result.text.characters) + // Drop only a TRULY empty result. A whitespace-only one is kept: + // `Engine.transcribe` joins segment texts with NO separator, so + // discarding a segment that consists of the separator would glue + // the neighbouring words together and change the hypothesis. + guard !text.isEmpty else { return nil } + // CMTime seconds are NaN/±inf for invalid or indefinite times — + // never let one into a cue's timing (#53 seam-defense discipline). + let start = result.range.start.seconds + let span = result.range.duration.seconds + let end = start + span + // Recognized text with unusable timing FAILS THE RUN rather than + // vanishing. Dropping it would edit the hypothesis invisibly: the + // WER denominator is the reference word count and does not move, + // so a dropped correct word adds a deletion while a dropped + // garbage word removes an insertion — the measured number shifts + // in an unsignposted direction. A benchmark tool must not do that + // silently; an error the operator can see is the honest failure. + // (`end` is checked separately: two finite addends can still + // overflow to infinity.) + guard start.isFinite, start >= 0, span.isFinite, span >= 0, end.isFinite else { + throw TranscriptionError( + backend: backendName, + message: "result \(index) carries unusable timing " + + "(start \(start), duration \(span)) for non-empty text — refusing to " + + "drop recognized speech, which would silently bias the measured " + + "error rate") + } + return .init( + start: start, + end: end, + text: text, + confidence: aggregateConfidence(runConfidences(result.text)), + // Whisper-specific hallucination signals; this backend + // computes neither, and nil never trips a threshold (#100). + noSpeechProb: nil, + compressionRatio: nil) + } + } +} diff --git a/Sources/BestASRKit/Models/DataModels.swift b/Sources/BestASRKit/Models/DataModels.swift index 3dce40a..b822bf2 100644 --- a/Sources/BestASRKit/Models/DataModels.swift +++ b/Sources/BestASRKit/Models/DataModels.swift @@ -17,6 +17,11 @@ public enum BackendID: String, Codable, CaseIterable, Sendable { case fluidParaformer = "fluid-paraformer" case fluidSenseVoice = "fluid-sensevoice" case mlxAudio = "mlx-audio" + // #121: the OS-native backend (Speech.framework's SpeechAnalyzer / + // SpeechTranscriber, macOS 26+). No download, no dependency — the model + // ships with the operating system. Appended at the tail for the same + // reason as #35: store enumeration order stays stable (design D2). + case appleSpeech = "apple-speech" } // MARK: - Detection diff --git a/Sources/BestASRKit/Models/ModelGrid.swift b/Sources/BestASRKit/Models/ModelGrid.swift index 97258e8..59cc8ab 100644 --- a/Sources/BestASRKit/Models/ModelGrid.swift +++ b/Sources/BestASRKit/Models/ModelGrid.swift @@ -15,13 +15,16 @@ public enum ModelGrid { public static let backendFluidParakeet = "fluid-parakeet" public static let backendFluidParaformer = "fluid-paraformer" public static let backendFluidSenseVoice = "fluid-sensevoice" + public static let backendAppleSpeech = "apple-speech" static let whisperSizes: [(size: String, memoryGB: Double)] = [ ("tiny", 1.0), ("base", 1.5), ("small", 2.5), ("medium", 5.0), ("large-v3-turbo", 6.0), ("large-v3", 10.0), ] - public static let rows: [ModelRow] = existingBackendRows + fluidParakeetRows + chineseFamilyRows + mlxAudioRows + public static let rows: [ModelRow] = + existingBackendRows + fluidParakeetRows + chineseFamilyRows + appleSpeechRows + + mlxAudioRows /// Resolve a model ADDRESS to its row (#65): mlx-audio rows are /// addressed `family/size` (sizes collide across families — canary 1b vs @@ -88,6 +91,61 @@ public enum ModelGrid { languages: ["multi"], estMemoryGB: 1.5, priority: 1, verified: true), ] + /// The OS-native Apple Speech row (#121, spec model-grid "Full-family + /// catalog"). Every field here differs in KIND from the other live rows, + /// so each choice is recorded rather than copied: + /// + /// - `family` "speechanalyzer": Apple publishes no model name — the only + /// honest handle is the framework surface that exposes it + /// (`SpeechAnalyzer` + `SpeechTranscriber`). Inventing a weight name + /// would be a guess dressed as a fact. + /// - `size` "system": there is no user-selectable size and no weight file. + /// The model is whatever the installed OS ships, so the address names + /// its provenance instead of a nonexistent parameter count. "system" is + /// unique across the grid, so it collides with nothing in the registry's + /// bare-size memory map or in `--model` resolution. + /// - `hfRepo`/`hfRevision` nil: nothing is fetched from HuggingFace. The + /// supply-chain pin for this backend is the OS version itself. + /// - `languages`: the 25 base subtags of the 45 locales + /// `SpeechTranscriber.supportedLocales` reported on macOS 27 (probed + /// 2026-08-01). NOT "multi" — that sentinel is reserved for the + /// 99+/1000+ class (ModelRow doc comment), and #105 is the standing + /// lesson about what mislabeling a bounded set costs: a 25-language + /// parakeet labeled "multi" got proposed for zh/ja/ko audio. Note `mul` + /// is ISO 639-2 "multiple languages" (Apple's `mul_IN` locale) — a real + /// advertised subtag, NOT this field's "multi" sentinel. + /// - `estMemoryGB` 2.0: **UNMEASURED PLACEHOLDER.** Apple exposes no model + /// footprint and this project has not measured one; 2.0 sits with the + /// other on-device CoreML/ANE speech rows (parakeet 2.0, paraformer 2.5) + /// and is deliberately conservative, since the value only ever gates + /// cold-start feasibility (design D2: measured data wins once benchmarked). + /// - `priority` 1: the tier gates the DEFAULT benchmark sweep, and there is + /// nothing to defer for the measured set — en_* and zh_* assets were + /// pre-installed on the development machine (ja_JP was NOT, and had to be + /// downloaded; "no download" is not a property of this backend), and on + /// a host below macOS 26 `isAvailable()` returns false so enumeration + /// skips it with a note. Paraformer sits at 2 because its output is + /// unusable, not merely because it downloads (whisperkit rows are + /// priority 1 and download on first use). A first-class competitor across + /// all three benchmark languages belongs in the first-run set. + /// - `verified` false: the flag means live-measured on this project's + /// corpora (ModelGridTests: "verified false until benchmarked"). Ad-hoc + /// probes proved the API works; they are not a `bestasr benchmark` run, + /// so the honest value is false. It also satisfies the other reading of + /// the field (hf repo checked against the hub) vacuously — there is no + /// repo, and the grid invariant forbids a repo id on an unverified row. + static let appleSpeechRows: [ModelRow] = [ + ModelRow( + backend: backendAppleSpeech, family: "speechanalyzer", size: "system", + quantization: "default", + languages: [ + "bn", "de", "en", "es", "fr", "gu", "hi", "it", "ja", "kn", "ko", "ks", + "mai", "ml", "mr", "mul", "ne", "or", "pa", "pt", "ta", "te", "ur", + "yue", "zh", + ], + estMemoryGB: 2.0, priority: 1, verified: false) + ] + /// Existing backends: live-validated all session — verified, priority 1. static let existingBackendRows: [ModelRow] = { var rows: [ModelRow] = [] diff --git a/Sources/BestASRKit/Models/ModelRegistry.swift b/Sources/BestASRKit/Models/ModelRegistry.swift index 42d0fee..65335e3 100644 --- a/Sources/BestASRKit/Models/ModelRegistry.swift +++ b/Sources/BestASRKit/Models/ModelRegistry.swift @@ -35,6 +35,12 @@ public enum ModelRegistry { || $0.backend == ModelGrid.backendFluidParakeet || $0.backend == ModelGrid.backendFluidParaformer || $0.backend == ModelGrid.backendFluidSenseVoice + // #121: the OS-native row registers here like every + // other live-engine backend. Its estimate is an + // UNMEASURED placeholder (see the grid row's comment); + // the "system" size name collides with nothing, so the + // max-uniquing below never has to arbitrate for it. + || $0.backend == ModelGrid.backendAppleSpeech } .map { ($0.size, $0.estMemoryGB) }, uniquingKeysWith: max) @@ -91,6 +97,11 @@ public enum ModelRegistry { var liveNonWhisper: Set = [ ModelGrid.backendFluidParakeet, ModelGrid.backendFluidParaformer, ModelGrid.backendFluidSenseVoice, + // #121: bundled like the fluid backends, so `--model system` + // resolves. Unconditional (not gated on includeExternal) because + // this engine ships in-process; the macOS-26 gate is availability, + // which the router checks separately. + ModelGrid.backendAppleSpeech, ] // A registered external adapter upgrades its catalog rows to // runnable (#51, spec asr-routing) — the caller passes availability. diff --git a/Sources/BestASRKit/Router/Router.swift b/Sources/BestASRKit/Router/Router.swift index 9d2dd1c..3ccf191 100644 --- a/Sources/BestASRKit/Router/Router.swift +++ b/Sources/BestASRKit/Router/Router.swift @@ -45,9 +45,15 @@ public enum Router { // the measured tier ranks across families; the cold-start prior below // still walks its whisper chain, so an unmeasured family is never // proposed without evidence. + // + // #121: membership here is what makes a backend selectable at all — + // omitting an available one discards the user's --backend and writes + // another backend's output under it. Position is inert: this is a + // membership set (contains / filter), and ColdStartPrior.selectBackend + // consults only `.whisperKit`, so appending changes no autonomous route. let availableOrdered: [BackendID] = [ .whisperKit, .whisperCpp, .fluidParakeet, .fluidParaformer, .fluidSenseVoice, - .mlxAudio, + .mlxAudio, .appleSpeech, ].filter { availability[$0] == true } diff --git a/Tests/BestASRKitTests/AppleSpeechEngineTests.swift b/Tests/BestASRKitTests/AppleSpeechEngineTests.swift new file mode 100644 index 0000000..6e13af4 --- /dev/null +++ b/Tests/BestASRKitTests/AppleSpeechEngineTests.swift @@ -0,0 +1,398 @@ +import Foundation +import Testing + +@testable import BestASRKit + +/// #121 — the OS-native Apple Speech backend (SpeechAnalyzer / SpeechTranscriber). +/// +/// Every test here is HERMETIC: no real transcription, no asset download, no +/// network. The locale table below is the live-probed ground truth from this +/// machine (macOS 27, `SpeechTranscriber.supportedLocales`, probed 2026-08-01) +/// used as a FIXTURE — the resolver takes the supported set as a parameter +/// precisely so it can be tested without calling into the OS. +struct AppleSpeechLocaleResolutionTests { + /// The 45 locales `SpeechTranscriber.supportedLocales` reported on macOS 27. + static let supported: [String] = [ + "bn_IN", "de_AT", "de_CH", "de_DE", "en_AU", "en_CA", "en_GB", "en_IE", + "en_IN", "en_NZ", "en_SG", "en_US", "en_ZA", "es_CL", "es_ES", "es_MX", + "es_US", "fr_BE", "fr_CA", "fr_CH", "fr_FR", "gu_IN", "hi_IN", "it_CH", + "it_IT", "ja_JP", "kn_IN", "ko_KR", "ks_IN", "mai_IN", "ml_IN", "mr_IN", + "mul_IN", "ne_IN", "or_IN", "pa_IN", "pt_BR", "pt_PT", "ta_IN", "te_IN", + "ur_IN", "yue_CN", "zh_CN", "zh_HK", "zh_TW", + ] + + @Test func `bestASR's three benchmark languages map to their Apple locales`() throws { + // The corpora this project measures against: en (LibriSpeech-class), + // ja, and zh — the zh suite is Common Voice zh-TW (RegressionBaseline + // asserts corpus names start `cv-zhtw-`), so bare "zh" MUST resolve to + // Traditional zh_TW, not the lexicographically-first zh_CN. + #expect( + try AppleSpeechEngine.resolveLocaleIdentifier( + language: "en", supported: Self.supported) == "en_US") + #expect( + try AppleSpeechEngine.resolveLocaleIdentifier( + language: "ja", supported: Self.supported) == "ja_JP") + #expect( + try AppleSpeechEngine.resolveLocaleIdentifier( + language: "zh", supported: Self.supported) == "zh_TW") + } + + @Test func `A script subtag selects its script, never the language default`() throws { + // VERIFY REGRESSION (Codex leg, reproduced by probe): subtags were read + // BY POSITION, so `Hans` in zh-Hans was taken for a region, matched + // nothing, and fell through to the `zh` preference — a user who asked + // for SIMPLIFIED silently received the TRADITIONAL model. Position does + // not identify a subtag; shape does. + #expect( + try AppleSpeechEngine.resolveLocaleIdentifier( + language: "zh-Hans", supported: Self.supported) == "zh_CN") + #expect( + try AppleSpeechEngine.resolveLocaleIdentifier( + language: "zh-Hant", supported: Self.supported) == "zh_TW") + // Case must not matter — the tag arrives lowercased from + // LanguageResolver, but the resolver may not depend on that. + #expect( + try AppleSpeechEngine.resolveLocaleIdentifier( + language: "ZH-HANS", supported: Self.supported) == "zh_CN") + } + + @Test func `An explicit region outranks a script that contradicts it`() throws { + // zh-Hans-TW is a legitimate tag (Simplified as written in Taiwan). + // Apple ships no such locale; the explicit region is the + // better-evidenced half of the request, so it wins. + #expect( + try AppleSpeechEngine.resolveLocaleIdentifier( + language: "zh-Hans-TW", supported: Self.supported) == "zh_TW") + #expect( + try AppleSpeechEngine.resolveLocaleIdentifier( + language: "zh-Hant-CN", supported: Self.supported) == "zh_CN") + } + + @Test func `A four-letter script is never mistaken for a region`() throws { + // en-Latn-GB: `Latn` is a script and must not consume the region slot, + // or GB is dropped and the answer degrades to en_US. + #expect( + try AppleSpeechEngine.resolveLocaleIdentifier( + language: "en-Latn-GB", supported: Self.supported) == "en_GB") + } + + @Test func `Resolution never crosses to another base language`() throws { + // The contract the whole design rests on, asserted as a property over + // every supported locale plus a batch of malformed tags — not as a + // handful of hand-picked examples. + let probes = Self.supported.map { $0.replacingOccurrences(of: "_", with: "-") } + + Self.supported + ["zh-Hans", "en-Latn-GB", "pt-US", "en-", "de_CH_x", "yue"] + for probe in probes { + guard let got = try? AppleSpeechEngine.resolveLocaleIdentifier( + language: probe, supported: Self.supported) + else { continue } // throwing is always an acceptable answer here + #expect( + AppleSpeechEngine.parseTag(got)?.base == AppleSpeechEngine.parseTag(probe)?.base, + "\(probe) resolved to \(got) — different base language") + } + } + + @Test func `Resolution is stable under reordering of the supported set`() throws { + // A future OS listing its locales in another order must not change a + // measured result. Reversal is the cheapest adversarial permutation. + for probe in ["zh", "zh-Hans", "en", "pt", "de", "yue"] { + let forward = try AppleSpeechEngine.resolveLocaleIdentifier( + language: probe, supported: Self.supported) + let reversed = try AppleSpeechEngine.resolveLocaleIdentifier( + language: probe, supported: Self.supported.reversed()) + #expect(forward == reversed, "\(probe): \(forward) vs \(reversed) under reordering") + } + } + + @Test func `Apple's underscored identifier is recorded in hyphenated form`() { + // Load-bearing: every language predicate in this project routes through + // LanguageResolver.baseSubtag, which splits on "-" ONLY. Recording + // `zh_TW` verbatim makes baseSubtag return "zh_tw", so isChinese turns + // false and the D7 Traditional/Simplified fold silently stops applying + // to this backend's zh scoring. + #expect(AppleSpeechEngine.bcp47("zh_TW") == "zh-TW") + #expect(LanguageResolver.isChinese(AppleSpeechEngine.bcp47("zh_TW"))) + #expect(LanguageResolver.metricKind(forLanguage: AppleSpeechEngine.bcp47("zh_TW")) == .cer) + #expect(LanguageResolver.metricKind(forLanguage: AppleSpeechEngine.bcp47("ja_JP")) == .cer) + #expect(LanguageResolver.metricKind(forLanguage: AppleSpeechEngine.bcp47("en_US")) == .wer) + // The un-fixed form is genuinely broken — asserted so the fix cannot be + // reverted without a red test. + #expect(!LanguageResolver.isChinese("zh_TW")) + } + + @Test func `An explicit region is honored verbatim when Apple supports it`() throws { + #expect( + try AppleSpeechEngine.resolveLocaleIdentifier( + language: "zh-cn", supported: Self.supported) == "zh_CN") + #expect( + try AppleSpeechEngine.resolveLocaleIdentifier( + language: "en-gb", supported: Self.supported) == "en_GB") + // Underscore form and case are both accepted (store rows and CLI input + // disagree on style; the resolver must not). + #expect( + try AppleSpeechEngine.resolveLocaleIdentifier( + language: "PT_br", supported: Self.supported) == "pt_BR") + } + + @Test func `A base language with exactly one region needs no preference entry`() throws { + // ko/ja/hi… each have a single supported locale — the resolver must + // find them without a hand-maintained table entry. + #expect( + try AppleSpeechEngine.resolveLocaleIdentifier( + language: "ko", supported: Self.supported) == "ko_KR") + #expect( + try AppleSpeechEngine.resolveLocaleIdentifier( + language: "hi", supported: Self.supported) == "hi_IN") + #expect( + try AppleSpeechEngine.resolveLocaleIdentifier( + language: "yue", supported: Self.supported) == "yue_CN") + } + + @Test func `Ambiguous base languages resolve to the documented preference, deterministically`() throws { + // de_AT sorts first; the preference table must win, or the choice + // would silently depend on Apple's list ordering. + #expect( + try AppleSpeechEngine.resolveLocaleIdentifier( + language: "de", supported: Self.supported) == "de_DE") + #expect( + try AppleSpeechEngine.resolveLocaleIdentifier( + language: "fr", supported: Self.supported) == "fr_FR") + #expect( + try AppleSpeechEngine.resolveLocaleIdentifier( + language: "it", supported: Self.supported) == "it_IT") + #expect( + try AppleSpeechEngine.resolveLocaleIdentifier( + language: "pt", supported: Self.supported) == "pt_PT") + #expect( + try AppleSpeechEngine.resolveLocaleIdentifier( + language: "es", supported: Self.supported) == "es_ES") + } + + @Test func `An unsupported region degrades within the same language, never across languages`() throws { + // pt_US does not exist; falling back to pt_PT keeps the LANGUAGE right, + // which is the only property that decides whether output is usable. + #expect( + try AppleSpeechEngine.resolveLocaleIdentifier( + language: "pt-us", supported: Self.supported) == "pt_PT") + // A script subtag is not a region — documented fall-through to the + // base preference (pass zh-CN / zh-TW to pick a script explicitly). + #expect( + try AppleSpeechEngine.resolveLocaleIdentifier( + language: "zh-hant", supported: Self.supported) == "zh_TW") + } + + @Test func `An unsupported language throws and names the language — never a silent English fallback`() { + // The failure mode this guards: a ja file decoded under en_US produced + // measured garbage. Falling back to English is worse than failing. + do { + _ = try AppleSpeechEngine.resolveLocaleIdentifier( + language: "sw", supported: Self.supported) + Issue.record("expected an unsupported language to throw") + } catch let error as TranscriptionError { + #expect(error.backend == "apple-speech") + #expect(error.message.contains("sw")) + #expect(!error.message.contains("en_US")) + } catch { + Issue.record("unexpected error type: \(error)") + } + } + + @Test func `A nil or auto language is refused with actionable guidance, not guessed`() { + // SpeechTranscriber has no language-autodetect mode; picking one for + // the user is exactly the silent-degradation bug above. + for requested in [nil, "auto", " "] as [String?] { + do { + _ = try AppleSpeechEngine.resolveLocaleIdentifier( + language: requested, supported: Self.supported) + Issue.record("expected a nil/auto language to throw (\(requested ?? "nil"))") + } catch let error as TranscriptionError { + #expect(error.backend == "apple-speech") + #expect(error.message.lowercased().contains("language")) + } catch { + Issue.record("unexpected error type: \(error)") + } + } + } + + @Test func `An empty supported set throws rather than inventing a locale`() { + #expect(throws: TranscriptionError.self) { + _ = try AppleSpeechEngine.resolveLocaleIdentifier(language: "en", supported: []) + } + } + + @Test func `Every preferred region names a locale Apple actually supports`() { + // Drift guard: the preference table is hand-maintained policy; a typo + // ("en"/"UK") would silently demote to the lexicographic fallback. + for (base, region) in AppleSpeechEngine.preferredRegions { + #expect( + Self.supported.contains("\(base)_\(region)"), + "preferred locale \(base)_\(region) is not in the probed supported set") + } + } +} + +struct AppleSpeechConfidenceTests { + @Test func `A broken confidence value reads as unknown, never as a number`() { + // NaN/inf/out-of-range are not low confidences, they are invalid ones. + // Passing NaN through would let it sort and compare nonsensically in + // any downstream filter; nil is the honest answer. + #expect(AppleSpeechEngine.aggregateConfidence([0.9, .nan]) == nil) + #expect(AppleSpeechEngine.aggregateConfidence([.infinity]) == nil) + #expect(AppleSpeechEngine.aggregateConfidence([-0.1]) == nil) + #expect(AppleSpeechEngine.aggregateConfidence([1.5]) == nil) + // The valid boundaries still pass through. + #expect(AppleSpeechEngine.aggregateConfidence([0.0]) == 0.0) + #expect(AppleSpeechEngine.aggregateConfidence([1.0]) == 1.0) + } + + @Test func `Segment confidence is the minimum over the result's runs`() { + // Apple reports confidence PER RUN; RawSegment carries one value per + // segment, so the segment number is DERIVED. Min is the conservative + // summary — a mean would let a confident run mask a bad one inside the + // same cue, and the cue is the unit a reader (and the filter) acts on. + #expect(AppleSpeechEngine.aggregateConfidence([0.985, 0.513]) == 0.513) + #expect(AppleSpeechEngine.aggregateConfidence([0.4]) == 0.4) + } + + @Test func `A run without the confidence attribute makes the whole segment nil`() { + // An invented number is worse than an absent one — RawSegment.confidence + // is Optional precisely so "unknown" is representable. + #expect(AppleSpeechEngine.aggregateConfidence([0.9, nil]) == nil) + #expect(AppleSpeechEngine.aggregateConfidence([nil]) == nil) + #expect(AppleSpeechEngine.aggregateConfidence([]) == nil) + } + + @available(macOS 26.0, *) + @Test func `Per-run confidences are read off a real AttributedString`() throws { + // Proves the attribute key path itself (not just the aggregation): + // a hand-built AttributedString carrying the Speech confidence + // attribute round-trips through the extractor. No ASR involved. + var first = AttributedString("And") + first[AttributeScopes.SpeechAttributes.ConfidenceAttribute.self] = 0.985 + var second = AttributedString(" Ask") + second[AttributeScopes.SpeechAttributes.ConfidenceAttribute.self] = 0.513 + let combined = first + second + #expect(AppleSpeechEngine.runConfidences(combined) == [0.985, 0.513]) + #expect(AppleSpeechEngine.aggregateConfidence( + AppleSpeechEngine.runConfidences(combined)) == 0.513) + + // A run with no attribute reads as nil rather than a default. + let bare = AttributedString("hello") + #expect(AppleSpeechEngine.runConfidences(bare) == [nil]) + } +} + +struct AppleSpeechAvailabilityTests { + @Test func `Availability is exactly the macOS 26 gate`() async { + let engine = AppleSpeechEngine() + let available = await engine.isAvailable() + // Both branches are expressed; this host runs whichever applies. + if #available(macOS 26.0, *) { + #expect(available == true) + } else { + #expect(available == false) + } + } + + @Test func `Availability probing is cheap — it must never download an asset`() async { + // Asset installation takes tens of seconds; availability detection is + // specified as graceful and cheap (spec asr-engine), and `bestasr + // list-backends` calls it for every backend. + let engine = AppleSpeechEngine() + let start = Date() + _ = await engine.isAvailable() + #expect(Date().timeIntervalSince(start) < 5.0) + } + + @Test func `The pre-macOS-26 refusal names the requirement and the framework`() { + // The path this machine cannot execute, asserted through the message + // constant the guard throws. + let message = AppleSpeechEngine.unsupportedOSMessage + #expect(message.contains("macOS 26")) + #expect(message.contains("SpeechAnalyzer")) + } + + @Test func `The engine is constructible on the package's macOS 14 deployment target`() { + // No @available on the type: CommandCore.live() holds it unconditionally + // and isAvailable() does the gating. If this ever needs @available, the + // engines array stops compiling for the deployment target. + let engine = AppleSpeechEngine() + #expect(engine.id == .appleSpeech) + #expect(engine.id.rawValue == "apple-speech") + } +} + +struct AppleSpeechGridTests { + @Test func `The apple-speech row is present and well-formed`() throws { + let rows = ModelGrid.rows(backend: ModelGrid.backendAppleSpeech, priorityCeiling: nil) + let row = try #require(rows.first) + #expect(rows.count == 1) + #expect(row.quantization == "default") + // Ships with macOS — there is no HuggingFace artifact to pin, and the + // grid invariant forbids a repo id on an unverified row. + #expect(row.hfRepo == nil) + #expect(row.hfRevision == nil) + // Never benchmarked through `bestasr benchmark` on this project's + // corpora — `verified` stays false until a measurement exists. + #expect(row.verified == false) + #expect(row.priority == 1) + #expect(row.estMemoryGB > 0) + } + + @Test func `The row enumerates in the default priority-1 sweep`() { + let sweep = ModelGrid.rows(backend: ModelGrid.backendAppleSpeech) + #expect(sweep.count == 1) + } + + @Test func `The row is reachable through the normal grid lookup, both address forms`() throws { + let bare = try #require( + ModelGrid.row(backend: ModelGrid.backendAppleSpeech, modelAddress: "system")) + let addressed = try #require( + ModelGrid.row( + backend: ModelGrid.backendAppleSpeech, + modelAddress: "\(bare.family)/\(bare.size)")) + #expect(bare.modelId == addressed.modelId) + #expect(bare.modelId == "apple-speech|speechanalyzer|system|default") + } + + @Test func `Declared languages are the probed locale set — never the multi sentinel`() throws { + let row = try #require( + ModelGrid.row(backend: ModelGrid.backendAppleSpeech, modelAddress: "system")) + // "multi" is reserved for the 99+/1000+ class (ModelRow doc comment); + // 45 locales over 25 base subtags is not that class, and #105 is the + // standing lesson about mislabeling a bounded set as multilingual. + #expect(!row.languages.contains("multi")) + // The three languages this project benchmarks must all be declared, or + // the router's declared-support gate would warn on every zh/ja run. + #expect(row.languages.contains("en")) + #expect(row.languages.contains("ja")) + #expect(row.languages.contains("zh")) + // Exactly the base subtags of the 45 probed locales. + let probed = Set( + AppleSpeechLocaleResolutionTests.supported.map { + String($0.split(separator: "_")[0]) + }) + #expect(Set(row.languages) == probed) + #expect(row.languages.count == 25) + } + + @Test func `Every declared language resolves to a supported Apple locale`() throws { + // Round-trip lock: the grid row and the engine's resolver cannot drift + // apart — anything the row advertises must be transcribable. + let row = try #require( + ModelGrid.row(backend: ModelGrid.backendAppleSpeech, modelAddress: "system")) + for language in row.languages { + let identifier = try AppleSpeechEngine.resolveLocaleIdentifier( + language: language, + supported: AppleSpeechLocaleResolutionTests.supported) + #expect(identifier.hasPrefix("\(language)_")) + } + } + + @Test func `The registry carries a memory estimate for the row`() throws { + let row = try #require( + ModelGrid.row(backend: ModelGrid.backendAppleSpeech, modelAddress: "system")) + let requirements = try ModelRegistry.requirements(for: row.size) + #expect(requirements.memoryGB == row.estMemoryGB) + } +} diff --git a/Tests/BestASRKitTests/CLITests.swift b/Tests/BestASRKitTests/CLITests.swift index fc145e8..df72523 100644 --- a/Tests/BestASRKitTests/CLITests.swift +++ b/Tests/BestASRKitTests/CLITests.swift @@ -258,11 +258,14 @@ struct ListCommandTests { // carries the bundled engines; external backends join only when the // user registry enables them, so this machine-dependent tail is // asserted by capability, not by exact list. + // #121: apple-speech is bundled too — registered unconditionally, with + // its own isAvailable() (not this list) gating pre-macOS-26 hosts. let ids = CommandCore.live().engines.map(\.id) - #expect(ids.prefix(5) == [ + #expect(ids.prefix(6) == [ .whisperKit, .whisperCpp, .fluidParakeet, .fluidParaformer, .fluidSenseVoice, + .appleSpeech, ]) - for extra in ids.dropFirst(5) { + for extra in ids.dropFirst(6) { #expect(ExternalEngineRegistry.externalCapable.contains(extra)) } } @@ -277,6 +280,22 @@ struct ListCommandTests { #expect(output.contains("0.6b-v3")) #expect(output.contains("fluid-parakeet")) } + + @Test func `list-models shows the apple-speech row like every other live family`() throws { + // #121: the catalog command's live section is documented as "every + // bundled non-whisper backend renders its own rows". A grid row the + // catalog never prints is undiscoverable — the user cannot learn that + // `--backends apple-speech` exists, or that the row is unverified. + let dir = try makeTempDir() + defer { try? FileManager.default.removeItem(at: dir) } + let core = makeCore(engines: [], cacheDir: dir) + let output = core.listModels() + #expect(output.contains("apple-speech")) + #expect(output.contains("system")) + // Unverified rows must say so, exactly as fluid-paraformer's does. + let line = output.split(separator: "\n").first { $0.contains("apple-speech") } + #expect(line?.contains("unverified") == true) + } } // MARK: - Context wiring (tasks 3.1/3.2; spec context-calibration + cli MODIFIED) diff --git a/Tests/BestASRKitTests/DataModelTests.swift b/Tests/BestASRKitTests/DataModelTests.swift index a61dd00..cc4c48c 100644 --- a/Tests/BestASRKitTests/DataModelTests.swift +++ b/Tests/BestASRKitTests/DataModelTests.swift @@ -58,11 +58,13 @@ struct DataModelTests { @Test func `BackendID enumerates the three bundled engines in stable order`() throws { // #35 (spec asr-engine "Common engine interface"): exactly the backends // with a bundled runtime — appended at the tail so store enumeration - // order stays stable (design D2). + // order stays stable (design D2). #121 appended apple-speech at the + // tail for the same reason. #expect(BackendID.allCases == [ .whisperKit, .whisperCpp, .fluidParakeet, .fluidParaformer, .fluidSenseVoice, - .mlxAudio, + .mlxAudio, .appleSpeech, ]) + #expect(BackendID.appleSpeech.rawValue == "apple-speech") #expect(BackendID.fluidParakeet.rawValue == "fluid-parakeet") // Codable round-trip through the raw value (store records use strings). let decoded = try JSONDecoder().decode( diff --git a/Tests/BestASRKitTests/RouterTests.swift b/Tests/BestASRKitTests/RouterTests.swift index 60a0ac5..1ef3289 100644 --- a/Tests/BestASRKitTests/RouterTests.swift +++ b/Tests/BestASRKitTests/RouterTests.swift @@ -112,6 +112,36 @@ struct RouterCrossFamilyTests { #expect(rec.reason.contains { $0.contains("unverified") }) } + @Test func `Locking the OS-native backend routes instead of silently substituting`() throws { + // #121 verify: an available backend that the router's membership list + // omits is WORSE than an unsupported one — the override is discarded, + // another backend's output is written under the user's chosen name, and + // the "unavailable" warning is both false (isAvailable() returns true + // on macOS 26+) and invisible without --explain. Measured on the real + // binary before this test existed: `transcribe --backend apple-speech` + // silently produced mlx-audio whisper output. + let rec = try Router.recommend( + host: Fixtures.m5Max, profile: .high, requestedLanguage: "zh", + backendOverride: "apple-speech", modelOverride: nil, + records: [], availability: [.appleSpeech: true] + ) + #expect(rec.backend == .appleSpeech) + #expect(rec.model == "system") + #expect(!rec.warnings.contains { $0.contains("unavailable") }) + } + + @Test func `An unavailable OS-native backend still warns rather than routing`() throws { + // The mirror case — below macOS 26 isAvailable() is false, and the + // membership fix must not turn that into a bogus route. + let rec = try Router.recommend( + host: Fixtures.m5Max, profile: .high, requestedLanguage: "en", + backendOverride: "apple-speech", modelOverride: nil, + records: [], availability: [.whisperKit: true, .appleSpeech: false] + ) + #expect(rec.backend == .whisperKit) + #expect(rec.warnings.contains { $0.contains("apple-speech") && $0.contains("unavailable") }) + } + @Test func `A single flattering measurement never outranks the aggregate`() throws { // #64: tiny has one 0.0 record on a short corpus but a bad mean over // its records; large-v3-turbo is broadly measured at ~0.09. The