Skip to content

Fix: carry_initial_prompt overflows the decoder prompt budget - #2823

Open
Yigtwxx wants to merge 1 commit into
openai:mainfrom
Yigtwxx:fix-carry-initial-prompt-budget
Open

Fix: carry_initial_prompt overflows the decoder prompt budget#2823
Yigtwxx wants to merge 1 commit into
openai:mainfrom
Yigtwxx:fix-carry-initial-prompt-budget

Conversation

@Yigtwxx

@Yigtwxx Yigtwxx commented Aug 1, 2026

Copy link
Copy Markdown

Problem

carry_initial_prompt=True exists so that a long initial_prompt (a glossary, a
list of proper nouns) keeps reaching the decoder on every sliding window. When
the prompt is long enough to fill the decoder's prompt budget, it does the
opposite: from the second window onwards the initial_prompt is dropped
entirely and replaced by carried-over transcript.

Root cause

The prompt budget is n_text_ctx // 2 - 1, which is 223 tokens for the released
checkpoints (n_text_ctx = 448). remaining_prompt_length tracks how much of
that budget is left once initial_prompt has taken its share:

remaining_prompt_length = model.dims.n_text_ctx // 2 - 1
if initial_prompt is not None:
    initial_prompt_tokens = tokenizer.encode(" " + initial_prompt.strip())
    all_tokens.extend(initial_prompt_tokens)
    remaining_prompt_length -= len(initial_prompt_tokens)   # may reach 0 or go negative
if carry_initial_prompt:
    nignored = max(len(initial_prompt_tokens), prompt_reset_since)
    remaining_prompt = all_tokens[nignored:][-remaining_prompt_length:]
    decode_options["prompt"] = initial_prompt_tokens + remaining_prompt

Once initial_prompt is 223 tokens or longer, remaining_prompt_length is zero
or negative and the slice inverts:

>>> tokens = [1, 2, 3, 4, 5]
>>> tokens[-2:]     # room for 2 more tokens
[4, 5]
>>> tokens[-0:]     # no room left - returns everything
[1, 2, 3, 4, 5]
>>> tokens[-(-2):]  # negative - drops from the front instead
[3, 4, 5]

So instead of carrying nothing, the window carries the entire accumulated
transcript. DecodingTask._get_initial_tokens then trims the combined prompt
from the left:

prompt_tokens[-(self.n_ctx // 2 - 1) :]

Left-trimming removes the leading tokens, which are exactly the
initial_prompt tokens the option is meant to preserve.

Fix

Clamp the remaining length at zero, and slice explicitly so the empty case
stays empty.

Evidence

tiny.en on CPU, tests/jfk.flac tiled to ~66 s (3 windows), temperature=0
and condition_on_previous_text=True so the carry-over path runs on every
window. DecodingTask.__init__ was wrapped to record options.prompt.

initial_prompt of exactly 223 tokens (budget exactly full, so nothing may
be carried):

window prompt length tokens over budget initial_prompt reaches the decoder
0 223 → 223 0 → 0 yes → yes
1 284 → 223 61 → 0 no → yes
2 367 → 223 144 → 0 no → yes

initial_prompt of 14 tokens (room to carry, i.e. every existing use of the
option):

window prompt length before prompt length after
0 14 14
1 76 76
2 134 134

Per-window prompt token ids are identical before and after, and the transcript
is byte-identical (sha256 861f8b56223f5e5124744caacf57f1f87dc4b37a10472cecf9a5ab6abf355bc5
in both runs).

The repository's own workflow was run on this branch before opening the PR and
is green across the whole matrix, pre-commit included:
https://github.com/Yigtwxx/whisper/actions/runs/30698478749

Compatibility

No API, CLI, or default behavior change. The only inputs affected are those
where remaining_prompt_length was already zero or negative, i.e. an
initial_prompt of at least n_text_ctx // 2 - 1 tokens combined with
carry_initial_prompt=True. Every shorter prompt takes the same code path as
before and produces identical output.

When `initial_prompt` is at least `n_text_ctx // 2 - 1` tokens long,
`remaining_prompt_length` becomes zero or negative. `tokens[-0:]` returns
the whole list rather than an empty one, so instead of carrying nothing the
sliding window carries the entire accumulated transcript.

`DecodingTask._get_initial_tokens` then keeps only the last
`n_text_ctx // 2 - 1` tokens of that prompt, which drops the leading
`initial_prompt` tokens - the text `carry_initial_prompt` exists to preserve.

Clamp the remaining length at zero and slice explicitly. Prompts that leave
room to carry are unaffected.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant