Filter subtitle-credit hallucinations in any language - #281
Conversation
Whisper emits a subtitle-credit line on silent audio in whatever language
it guessed ("Undertekster av Ai-Media", "Untertitel von ...", "字幕by...").
The filter only matched whole strings against an English-only phrase list,
so every non-English variant landed in the user's text.
Match a credit marker as a substring instead, on diacritic-folded text and
only for short outputs, still gated behind the existing no_speech_prob
check so real speech mentioning subtitles survives.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
📝 WalkthroughWalkthroughTranscriptionService now detects localized subtitle-credit hallucinations using normalized marker matching. A standalone executable test suite covers speech-confidence cases and is integrated into the Makefile’s test target. ChangesTranscription hallucination filtering
Estimated code review effort: 2 (Simple) | ~10 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@Sources/TranscriptionService.swift`:
- Around line 315-333: Refine the hallucination filtering around the
`hallucinationMarkers` checks in `TranscriptionService` so standalone
subtitle-related words and ordinary phrases such as “subtitles by Friday” are
not rejected solely because the output is short. Require stronger credit-line or
attribution context, while preserving detection of genuine subtitle credits; add
a regression test for the short legitimate sentence alongside the existing
length-boundary coverage.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: d90b4f37-d468-403f-b509-02f1eb3ce5e8
📒 Files selected for processing (3)
MakefileSources/TranscriptionService.swiftTests/TranscriptionServiceTests.swift
| private let hallucinationMarkers = [ | ||
| "amara.org", | ||
| "subtitles by", "subtitle by", "subs by", "captions by", "captioning by", | ||
| "undertekster", "undertitel", "tekstet av", // no/da | ||
| "undertext", "textning", // sv | ||
| "untertitel", // de | ||
| "ondertitel", // nl | ||
| "sous-titr", // fr | ||
| "subtitulos", "subtitulado", // es | ||
| "sottotitoli", // it | ||
| "legendas", // pt | ||
| "napisy", // pl | ||
| "tekstitys", // fi | ||
| "altyaz", // tr | ||
| "субтитр", // ru | ||
| "字幕", // zh/ja | ||
| "자막", // ko | ||
| "ترجمة" // ar | ||
| ] |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | 🏗️ Heavy lift
Avoid classifying every short subtitle mention as a credit line.
The substring check treats standalone words such as untertitel, undertekster, and 字幕—as well as phrases like subtitles by—as hallucinations in any output up to 60 characters. For example, a legitimate utterance such as “Can you add subtitles by Friday?” is filtered whenever no_speech_prob >= 0.1. Require stronger credit-line/attribution context or narrow these markers, and add a short-sentence regression test; the current test only protects sentences exceeding the length limit.
Also applies to: 360-367
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@Sources/TranscriptionService.swift` around lines 315 - 333, Refine the
hallucination filtering around the `hallucinationMarkers` checks in
`TranscriptionService` so standalone subtitle-related words and ordinary phrases
such as “subtitles by Friday” are not rejected solely because the output is
short. Require stronger credit-line or attribution context, while preserving
detection of genuine subtitle credits; add a regression test for the short
legitimate sentence alongside the existing length-boundary coverage.
1 participant
Problem
On silent or near-silent audio, Whisper emits a subtitle-credit line in whatever language it guessed —
Undertekster av Ai-Media,Untertitel von ...,字幕by.... These land verbatim in the user's text.The existing filter in
TranscriptionService.isHallucinationmatches the transcript as a whole string against an English-only phrase list, so every non-English variant sails through. The wording varies endlessly across languages, so extending the exact-match list can't close this.Fix
Match a credit marker as a substring instead of matching the full phrase:
no_speech_prob >= 0.1checkSo a real sentence that happens to mention subtitles ("Can you add subtitles by tomorrow...") is not swallowed, and a credit line spoken deliberately on non-silent audio still survives.
Tests
New
Tests/TranscriptionServiceTests.swiftcovers both directions — credit lines in 5 languages dropped on silent audio, the same strings preserved whenno_speech_probis low, and a long sentence mentioning subtitles preserved. Wired intomake testas a second runner (the existing runner has its own@main).make test:Not covered
The realtime WebSocket path (
RealtimeTranscriptionService) has no hallucination filter at all — it gets nono_speech_probfrom the server, so it needs a different approach. Left alone here.🤖 Generated with Claude Code
Summary by CodeRabbit
Bug Fixes
Tests