fix: keep non-English dictation in its own language during cleanup - #287
Open
melnikaite wants to merge 2 commits into
Open
fix: keep non-English dictation in its own language during cleanup#287melnikaite wants to merge 2 commits into
melnikaite wants to merge 2 commits into
Conversation
The cleanup prompt says "No translation" and "Preserve the speaker's final
intended meaning, tone, and language", but every example in it is written in
English and models follow the examples over the rule. Dictating German at the
default cleanup model comes back in English: "benenne user id in user
unterstrich id um" returns as "rename user id to user_id". On smaller local
models the whole transcript flips, greeting and all.
FreeFlow already knows the language being dictated — TranscriptionService
sends it to the transcription API as the `language` field — and it already has
the machinery for naming a language inside the cleanup prompt, in
applyOutputLanguage. This adds the counterpart: when no output language is
set, name the dictation language so cleanup preserves it instead of
translating it.
Wording alone does not fix this, which is why the change rides the existing
applyOutputLanguage shape rather than adding more prompt text. A generic rule
("write the output in the language of RAW_TRANSCRIPTION") had no measurable
effect in three different placements, a clause scoped to the offending rule had
none either, and adding a counterpart example in a different non-English
language did not transfer to a third language. Naming the language explicitly
is what works.
Auto-detect and English are skipped. There is nothing to name in the first
case, and in the second the prompt's own examples are already English and
English dictation never drifted in any measurement, so English-only users get
a byte-identical request.
Measured at temperature 0 with 5 repeats, scoring the output language
separately from the rule each case also exercises, because a model that cannot
join "user underscore id" into "user_id" otherwise reads as a translation
failure:
openai/gpt-oss-20b (default cleanup model)
German developer instruction 2/5 -> 4/5 kept in German
Russian 5/5 -> 5/5, never drifted
meta-llama/llama-3.1-8b-instruct (two runs pooled; see note)
Russian greeting 3/10 -> 10/10
Russian developer instruction 3/9 -> 9/9
German developer instruction 4/5 -> 4/5, already mostly fine
qwen/qwen3.6-27b (default fallback and context model)
unaffected, 5/5 -> 5/5 on every case
Qwen3-VL-8B-Instruct via a local OpenAI-compatible server
Russian and German 0/3 -> 3/3
gpt-oss-20b improves rather than fully recovers on German, 4/5 instead of 5/5.
The effect is largest on small models, which is also where the app's support
for a custom base URL puts it.
llama-3.1-8b is pooled across two runs because it was inconsistent between
them even at temperature 0 — the Russian greeting scored 3/5 in one run and
0/5 in the other before the change, and 5/5 in both after it. The direction was
the same every time; the absolute baseline was not, so quoting a single run
would overstate the precision.
Deliberately mixed-language dictation stays mixed: "надо добавить retry на
flaky тесты и запустить с флагом дэш дэш fix" keeps retry, flaky and --fix,
which the prompt requires and a language directive could plausibly have
flattened.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
🚧 Files skipped from review as they are similar to previous changes (2)
📝 WalkthroughWalkthroughThe post-processing flow now derives a dictation language, passes it into ChangesDictation language handling
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant AppState
participant PostProcessingService
participant LanguageModel
AppState->>PostProcessingService: processTranscript passes dictationLanguage
PostProcessingService->>PostProcessingService: Propagate dictationLanguage through fallback processing
PostProcessingService->>LanguageModel: Send prompt with dictation-language instruction
Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@Sources/PostProcessingService.swift`:
- Around line 753-760: The dictation-language prompt currently forces all output
into one language, conflicting with mixed-language preservation. In
Sources/PostProcessingService.swift lines 753-760, update applyDictationLanguage
to instruct preserving spans spoken in other languages while retaining the
named-language guidance; in Tests/AppContextServiceTests.swift lines 79-98,
remove the single-language assertion and assert the mixed-language preservation
instruction instead.
In `@Tests/AppContextServiceTests.swift`:
- Around line 79-98: Update testDictationLanguageIsNamedInPrompt to remove the
assertion requiring “Output ONLY in Russian” and instead verify that Russian is
identified as the primary dictation language while mixed-language spans are
preserved in their original languages. Keep the existing prompt-prefix and
explicit-language assertions intact.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 856d1939-0d0e-4c3f-8b9f-cca68c95ed4f
📒 Files selected for processing (4)
MakefileSources/AppState.swiftSources/PostProcessingService.swiftTests/AppContextServiceTests.swift
The directive said "Output ONLY in <language>", which reads as a contradiction of the prompt's own "Preserve mixed-language text exactly as mixed" even though mixed dictation measured fine. Take the reviewer's wording: name the language as the primary one and carve out foreign spans explicitly. A/B'd the two wordings before switching, since "primarily" is a hedge and hedged language rules had already failed at 0/3 earlier in this work. They tie: Qwen3-VL-8B and Gemma 4 E4B both score 3/3 on all three drift cases and all six mixed-language patterns under either wording, and openai/gpt-oss-20b scores 5/5 on every case under both. Equal reliability, one less contradiction in the prompt. The suggested test edit is not taken verbatim: it kept an assertion for "The dictation is in Russian" alongside one for "The dictation is primarily in Russian", and the second string does not contain the first, so the test would have failed. Assertions now match the wording actually shipped and add the mixed-language carve-out the review asked for. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The cleanup prompt says "No translation" and "Preserve the speaker's final intended meaning, tone, and language", but every example in it is written in English and models follow the examples over the rule. Dictating German at the default cleanup model comes back in English: "benenne user id in user unterstrich id um" returns as "rename user id to user_id". On smaller local models the whole transcript flips, greeting and all.
FreeFlow already knows the language being dictated — TranscriptionService sends it to the transcription API as the
languagefield — and it already has the machinery for naming a language inside the cleanup prompt, in applyOutputLanguage. This adds the counterpart: when no output language is set, name the dictation language so cleanup preserves it instead of translating it.Wording alone does not fix this, which is why the change rides the existing applyOutputLanguage shape rather than adding more prompt text. A generic rule ("write the output in the language of RAW_TRANSCRIPTION") had no measurable effect in three different placements, a clause scoped to the offending rule had none either, and adding a counterpart example in a different non-English language did not transfer to a third language. Naming the language explicitly is what works.
Auto-detect and English are skipped. There is nothing to name in the first case, and in the second the prompt's own examples are already English and English dictation never drifted in any measurement, so English-only users get a byte-identical request.
Measured at temperature 0 with 5 repeats, scoring the output language separately from the rule each case also exercises, because a model that cannot join "user underscore id" into "user_id" otherwise reads as a translation failure:
openai/gpt-oss-20b (default cleanup model)
German developer instruction 2/5 -> 4/5 kept in German
Russian 5/5 -> 5/5, never drifted
meta-llama/llama-3.1-8b-instruct (two runs pooled; see note)
Russian greeting 3/10 -> 10/10
Russian developer instruction 3/9 -> 9/9
German developer instruction 4/5 -> 4/5, already mostly fine
qwen/qwen3.6-27b (default fallback and context model)
unaffected, 5/5 -> 5/5 on every case
Qwen3-VL-8B-Instruct via a local OpenAI-compatible server
Russian and German 0/3 -> 3/3
gpt-oss-20b improves rather than fully recovers on German, 4/5 instead of 5/5. The effect is largest on small models, which is also where the app's support for a custom base URL puts it.
llama-3.1-8b is pooled across two runs because it was inconsistent between them even at temperature 0 — the Russian greeting scored 3/5 in one run and 0/5 in the other before the change, and 5/5 in both after it. The direction was the same every time; the absolute baseline was not, so quoting a single run would overstate the precision.
Deliberately mixed-language dictation stays mixed: "надо добавить retry на flaky тесты и запустить с флагом дэш дэш fix" keeps retry, flaky and --fix, which the prompt requires and a language directive could plausibly have flattened.
Summary by CodeRabbit
New Features
Bug Fixes