Fix: carry_initial_prompt overflows the decoder prompt budget - #2823
Open
Yigtwxx wants to merge 1 commit into
Open
Fix: carry_initial_prompt overflows the decoder prompt budget#2823Yigtwxx wants to merge 1 commit into
Yigtwxx wants to merge 1 commit into
Conversation
When `initial_prompt` is at least `n_text_ctx // 2 - 1` tokens long, `remaining_prompt_length` becomes zero or negative. `tokens[-0:]` returns the whole list rather than an empty one, so instead of carrying nothing the sliding window carries the entire accumulated transcript. `DecodingTask._get_initial_tokens` then keeps only the last `n_text_ctx // 2 - 1` tokens of that prompt, which drops the leading `initial_prompt` tokens - the text `carry_initial_prompt` exists to preserve. Clamp the remaining length at zero and slice explicitly. Prompts that leave room to carry are unaffected.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
carry_initial_prompt=Trueexists so that a longinitial_prompt(a glossary, alist of proper nouns) keeps reaching the decoder on every sliding window. When
the prompt is long enough to fill the decoder's prompt budget, it does the
opposite: from the second window onwards the
initial_promptis droppedentirely and replaced by carried-over transcript.
Root cause
The prompt budget is
n_text_ctx // 2 - 1, which is 223 tokens for the releasedcheckpoints (
n_text_ctx = 448).remaining_prompt_lengthtracks how much ofthat budget is left once
initial_prompthas taken its share:Once
initial_promptis 223 tokens or longer,remaining_prompt_lengthis zeroor negative and the slice inverts:
So instead of carrying nothing, the window carries the entire accumulated
transcript.
DecodingTask._get_initial_tokensthen trims the combined promptfrom the left:
Left-trimming removes the leading tokens, which are exactly the
initial_prompttokens the option is meant to preserve.Fix
Clamp the remaining length at zero, and slice explicitly so the empty case
stays empty.
Evidence
tiny.enon CPU,tests/jfk.flactiled to ~66 s (3 windows),temperature=0and
condition_on_previous_text=Trueso the carry-over path runs on everywindow.
DecodingTask.__init__was wrapped to recordoptions.prompt.initial_promptof exactly 223 tokens (budget exactly full, so nothing maybe carried):
initial_promptreaches the decoderinitial_promptof 14 tokens (room to carry, i.e. every existing use of theoption):
Per-window prompt token ids are identical before and after, and the transcript
is byte-identical (
sha256 861f8b56223f5e5124744caacf57f1f87dc4b37a10472cecf9a5ab6abf355bc5in both runs).
The repository's own workflow was run on this branch before opening the PR and
is green across the whole matrix,
pre-commitincluded:https://github.com/Yigtwxx/whisper/actions/runs/30698478749
Compatibility
No API, CLI, or default behavior change. The only inputs affected are those
where
remaining_prompt_lengthwas already zero or negative, i.e. aninitial_promptof at leastn_text_ctx // 2 - 1tokens combined withcarry_initial_prompt=True. Every shorter prompt takes the same code path asbefore and produces identical output.