[Issue]: Whisper periodic forced timestamp sampling truncates Japanese sentence endings

[tamayanaohito]

Problem Description

I found a reproducible Japanese sentence-ending truncation issue in Whisper V3 Turbo running through FastFlowLM v0.9.45 on a Ryzen AI NPU.

The strongest cause isolated in controlled A/B testing was the decoder watchdog that periodically forces sampling from the timestamp-token range after approximately 16 decoded tokens:

if (watching_dog == 0 && allow_force_time_stamp) {
    last_idx = this->_sample_in_time_stamp(logits);
    watching_dog = 16;
}

This code is also present in current main at commit 2d6c4838a02555478f0de1e57fe6428e614b67ba.

Disabling only this periodic forced timestamp selection recovered all three reproducible sentence endings in the original Japanese test set. It does not disable normal Whisper timestamp tokens.

Aggregate results:

Dataset	Upstream baseline CER	Candidate CER	Relative change
Six existing Japanese TTS clips	40.12%	23.46%	-41.5%
Ten held-out Japanese TTS clips	20.68%	8.93%	-56.8%
One 36.096-second microphone recording	2.86%	2.86%	no regression observed
Two quiet negative-sentence clips	0%	0%	maintained
Five seconds of exact digital silence	irrelevant repetition	empty result	corrected

Feeding the detected language token into the decoder KV cache was tested independently and did not improve CER in this test set. Disabling the watchdog exposed a separate exact-digital-silence repetition case, so the candidate returns an empty result only when every PCM sample is exactly 0.0f; it does not use a near-silence threshold.

Full methodology, limitations, aggregate measurements, and the v0.9.45 patch:

https://github.com/tamayanaohito/fastflowlm-whisper-japanese-decoder-fix

This is an experimental report, not a claim of general Japanese ASR improvement. Human speech coverage is limited to one speaker and one recording. Personal audio and raw transcripts are not published.

Would the maintainers consider removing, revising, or making this periodic forced timestamp watchdog configurable? I can rebase the minimal change onto current main if that direction is acceptable.

Operating System

Microsoft Windows 11 Home, version 10.0.26200, build 26200

CPU

AMD Ryzen AI 9 465 with Radeon 880M

GPU

AMD Radeon 880M and NVIDIA GeForce RTX 5060 Laptop GPU are installed, but neither was used for ASR inference. Whisper inference ran on the Ryzen AI NPU.

ROCm Version

N/A on Windows. FastFlowLM used the Ryzen AI NPU backend with XRT 2.21.75.

ROCm Component

No response

Steps to Reproduce

1. Check out FastFlowLM v0.9.45 (f43272c77bb6de4984058430b3e5fbf3e7926c61) and build an unchanged baseline.
2. Transcribe short Japanese WAV files whose reference text extends beyond approximately 16 decoded tokens.
3. Confirm that the rebuilt baseline matches the installed FastFlowLM result on identical WAV bytes.
4. Apply fastflowlm-v0.9.45-whisper-decoder.patch from the linked repository.
5. Run the candidate in a separate process with HIGHRISE_WHISPER_DECODE=natural-timestamps.
6. Compare complete output and normalized CER using identical WAV bytes and serialized requests.
7. Test exact digital silence and quiet non-zero speech separately.

(Optional for Linux users) Output of /opt/rocm/bin/rocminfo --support

rocminfo --support output

Paste output here

Additional Information

• Model: Whisper V3 Turbo / whisper-v3:turbo
• Inference device: Ryzen AI NPU
• FastFlowLM target: v0.9.45; current main source was also inspected
• Requests were serialized and each mode ran in a separate loopback-only process
• Model files were hash-checked before and after testing
• A separately loaded NPU vision model remained ready during the comparisons
• No NPU kernels, model weights, audio preprocessing, public APIs, or postprocessing were changed

[tamayanaohito]

Update: validated against FastFlowLM v1.0.0

I reran the same serialized WAV set against the official FastFlowLM v1.0.0 release and a decoder-only patch rebased onto v1.0.0 commit 39fb58cc7f9e23d97b764a1ef21ff0ade088047b.

The periodic forced timestamp watchdog is still present in v1.0.0, and the tested Japanese sentence-ending truncations still reproduce with the unmodified v1.0.0 baseline. The rebased patch recovered the same three truncated endings.

Results on the same Ryzen AI NPU:

Dataset	Official v1.0.0 CER	Patched v1.0.0 CER
Six existing Japanese TTS clips	40.12%	23.46%
Ten held-out Japanese TTS clips	20.68%	8.93%
One 36.096-second microphone recording	2.86%	2.86%

Median wall time across all 20 cases was 3.287 s for official v1.0.0 and 3.189 s for the patched build.

The updated experimental release is here:

https://github.com/tamayanaohito/fastflowlm-whisper-japanese-decoder-fix/releases/tag/v0.2.0-experimental

It includes the version-pinned v1.0.0 source patch, continued v0.9.45 support, English and Japanese documentation, SHA-256 checksums, and Windows PowerShell install/rollback scripts that verify the exact upstream commit before changing the source tree. It still contains no binaries, model weights, NPU kernels, personal audio, or private transcripts.

This v1.0.0 retest strengthens the original report: updating to v1.0.0 alone did not resolve the tested decoder behavior.