[Bugfix][Multimodal] Fix video temporal padding estimates

[labAxiaoming]

Fix Video Temporal Padding Token Estimates

Summary

Fix incorrect video token estimation when temporal_patch_size > 2.
Fix #47866

Root Cause

Several multimodal models used:

padded_num_frames = num_frames + num_frames % temporal_patch_size

This only works reliably for temporal_patch_size 1 or 2. For example:

num_frames = 17
temporal_patch_size = 4

current:  17 + 17 % 4 = 18
expected: 20

The incorrect padding can make grid_t and the estimated video token count too
small. This affects vLLM-side budget estimation, dummy input sizing, and
warmup.

Changes

Replace the formula with:

padded_num_frames = num_frames + (-num_frames % temporal_patch_size)

Updated model paths:

• Qwen2-VL
• GLM4V
• Kanana-V
• Keye
• LLaVA-OneVision2
• MiMo-V2-Omni

MiMo additionally converts the padded frame count to int, because its
effective_frames is calculated from num_frames * tokens_per_second.

No shared helper or math_utils changes are introduced.

Tests

test video and image in GLM-OCR and Qwen2-VL-7B-Instruct model

Added a CPU-only regression test covering:
python -m pytest tests/models/multimodal/processing/test_glm4_1v.py -k test_vision_info_rounds_up_temporal_frames -v

temporal_patch_size=2: 17 frames -> grid_t=9
temporal_patch_size=4: 17 frames -> grid_t=5
temporal_patch_size=8: 17 frames -> grid_t=3

The change only affects vLLM token estimation. Hugging Face preprocessing,
vision encoder computation, model weights, and generated outputs are unchanged.

Related Issues and PRs

• Fixes vllm-project/vllm#47866.
• This is a minimal alternative to draft PR
  vllm-project/vllm#47876:
  it changes only the six call sites and adds focused regression coverage,
  without introducing a shared helper or modifying common math utilities.

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[labAxiaoming]

Hello~ Could you please help review this bug fix when you have a moment? Thankyou~ @DarkLight1337, @Isotr0py, @ywang96

[Isotr0py]

QQ: I think we have padded frames in video loader?

vllm/vllm/multimodal/video.py

Lines 1636 to 1663 in 854c33f

	@classmethod
	def load_bytes(
	cls,
	data: bytes,
	num_frames: int = -1,
	fps: int = 2,
	max_duration: int = 300,
	frame_recovery: bool = False,
	*,
	backend: Literal[
	"opencv", "pyav", "torchcodec", "pynvvideocodec", "deepstream"
	] = "opencv",
	**kwargs,
	) -> tuple[npt.NDArray, dict[str, Any]]:
	frames, metadata = super().load_bytes(
	data,
	num_frames=num_frames,
	fps=fps,
	max_duration=max_duration,
	frame_recovery=frame_recovery,
	backend=backend,
	**kwargs,
	)
	# Ensure even frame count — matches HF's sample_frames even-padding
	# and _preprocess temporal_patch_size divisibility check.
	if frames.shape[0] & 1:
	frames = np.concatenate([frames, frames[-1:]], axis=0)
	return frames, metadata

[labAxiaoming]

Thank for your review

_get_vision_info is used for vLLM-side token/budget estimation, dummy input sizing, and warmup. It is not the actual image/video preprocessing path; real inputs are still processed by the HF/model image or video processor.

if frames.shape[0] & 1: 
    frames = np.concatenate([frames, frames[-1:]], axis=0) 

This code may be process real video and ensures the video has an even number of frames. If the frame count is odd, it duplicates the last frame and appends it, making the total frame count even. but if temporal_patch_size was not 2, such as 4, the origin frames was 17, paded frame was 18 after pad, 20 was expected.

the real video preprocess also on transformers huggingface/transformers#47141

[labAxiaoming]

ci/pr fail seems OOM for Zyphra/Zamba2-1.2B-instruct model , but i have not modify about this model
ValueError: Free memory on device cuda:0 (29.89/32.5 GiB) on startup is less than desired GPU memory utilization (0.92, 29.9 GiB). Decrease GPU memory utilization or reduce GPU memory used by other processes.