From a77f14040318d09847b87f3b1d920e6ff8d879b8 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Fri, 14 Aug 2026 12:22:29 +0000 Subject: [PATCH 1/3] feat(ENG-MM-INPUT-PIPELINE): the multimodal limits reach a live request, so the flags stop being decoration (#607, #686) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Wave L2 of #607. L1 ported vLLM's per-modality input limits and the refusal they produce, but nothing constructed a config on a live request, so the numbers were unreachable and the HTTP 400 arm was unproven. This wave adds the two serve flags, the C-ABI fields, and — the part that makes the other two mean anything — the CALL SITE. Three surfaces, each mirroring vLLM at the pin (5559679229bc): 1. `--[no-]language-model-only` and `--limit-mm-per-prompt ''` (arg_utils.py:555-556,1276-1279,1691-1692). Both spellings of the boolean, because _compute_kwargs gives a bool field argparse.BooleanOptionalAction (arg_utils.py:346-348) and a recipe that turns the flag off explicitly must not die on an unknown argument. The dict value is a JSON object (type=parse_type(json.loads), arg_utils.py:375-377); ParseLimitMmPerPromptJson ports _validate_limit_per_prompt (multimodal.py:212-236) and the DummyOptions dataclasses behind it (:17-43), so the legacy count-only, the configurable {"count": N, ...} and the mixed spellings all parse, while a non-object document, a negative count (Field(999, ge=0)), an unknown per-modality option (extra="forbid") or a non-positive one (Field(None, gt=0)) is REFUSED before the model load. A mistyped limit that silently became 999 is a limit that is not there. The profiling options are validated, dropped (only .count feeds get_limit_per_prompt, :335) and the drop is announced per key. 2. vllm_model_params.language_model_only + .limit_mm_per_prompt, appended, so VLLM_ABI_VERSION goes 18 -> 19 and a zero-initialised v18 struct is byte-identical. The JSON-string shape follows the v9 kv_transfer_config precedent: a dict-valued vLLM flag crosses the ABI as its own JSON rather than as a fixed struct of modalities the ABI would then owe forever. Both land on EngineParams::multimodal -> LoadedEngine::mm_config(), so the flags and the ABI resolve ONE config per engine and cannot drift. 3. chat_mm.cpp calls ValidateChatMmLimits — the port of the chat_utils.py:648-662 tracker — as step 0 of MakeQwen3VLImageChatFn, over a BaseProcessingInfo folding the engine's config with the seam's own declared ceiling, Qwen3VLChatSupportedMmLimits() == {"image": 1}. That declaration is the answer to the question #686 left open ("the model's own supported limit; today nothing declares one"): this seam locates a single image part and routes no video or audio, so its honest ceiling is one image and every other modality is absent, which context.py:414-415 reads as limit 0. Closes #686. A three-image chat request is now 400 BadRequestError "At most 1 image(s) may be provided in one prompt." One correction to that issue, found by the RED run and recorded rather than quietly fixed: through the PRODUCTION seam the pre-L2 behaviour was not the truncated 200 the issue describes but an HTTP 500 — the seam injects one placeholder marker per image part while routing only the first image, so ExpandImagePlaceholders raised "more image placeholders than grids" and the client saw a server fault carrying an internal message. The truncated 200 is real for the validate-then-build shape and is pinned as its own test leg. Both are wrong the same way, so the issue's diagnosis stands and only its consequence was understated. NO MEMORY CLAIM. Nothing gates vision-tower construction on the limits, so --language-model-only changes what the server accepts, not what it allocates. That is wave L3 and it is owed with a measured RSS reduction; the docs say so explicitly. Also still owed and named rather than assumed: the second call site, process_inputs_mm (upstream's context.py:461). It needs the per-model get_supported_mm_limits() hook L1 already recorded as absent (the M2 towers own it); wiring it first would mean inventing a supported-limits source inside the engine, which the mirror rule forbids. The chat call site is wired because it HAS a concrete source, and every path the OpenAI server reaches today goes through it. Verified on aarch64 (the build-test-cpu-arm64 CI lane), CPU-only: test_serve_mm_limits 10/10 (101 assertions) -- new test_chat_mm 11/11 (126 assertions) test_openai_api_server 56/56 (638 assertions) test_capi -- ABI v19 + the config hop RED first, behaviourally: CHECK_THROWS_AS threw "ExpandImagePlaceholders: more image placeholders than grids" instead of InputValidationError; the HTTP legs failed CHECK(500 == 400), CHECK("InternalServerError" == BadRequestError) and CHECK(200 == 400) -- the last being the truncated 200 itself. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --- .agents/engine-matrix.md | 2 +- .agents/roadmap_v1.md | 4 +- .agents/specs/mm-serving.md | 17 + .agents/specs/multimodal-track.md | 87 +++++ .agents/specs/serve-recipe-args.md | 2 +- CMakeLists.txt | 1 + README.md | 4 +- docs/FEATURES.md | 2 +- docs/STATUS.md | 12 +- docs/USAGE.md | 56 ++- include/vllm.h | 61 ++- include/vllm/config/multimodal.h | 43 +++ include/vllm/entrypoints/model_loader.h | 26 ++ include/vllm/entrypoints/openai/chat_mm.h | 72 +++- src/capi/vllm_c.cpp | 16 + src/vllm/config/multimodal.cpp | 103 +++++ src/vllm/entrypoints/model_loader.cpp | 4 + src/vllm/entrypoints/openai/chat_mm.cpp | 65 +++- src/vllm/entrypoints/openai/server_main.cpp | 114 +++++- tests/CMakeLists.txt | 7 + tests/capi/test_capi.cpp | 85 +++++ .../entrypoints/openai/test_api_server.cpp | 195 ++++++++++ .../vllm/entrypoints/openai/test_chat_mm.cpp | 257 ++++++++++++- .../openai/test_serve_mm_limits.cpp | 353 ++++++++++++++++++ 24 files changed, 1549 insertions(+), 39 deletions(-) create mode 100644 src/vllm/config/multimodal.cpp create mode 100644 tests/vllm/entrypoints/openai/test_serve_mm_limits.cpp diff --git a/.agents/engine-matrix.md b/.agents/engine-matrix.md index 8c9bdca94..2c16aece5 100644 --- a/.agents/engine-matrix.md +++ b/.agents/engine-matrix.md @@ -71,7 +71,7 @@ forensics: roadmap_v1.md and the parity ledger. | `ENG-DBO-UBATCH` | DBO and ubatch overlap | T2 | `vllm/config/parallel.py:208,524` | - | - | `planned: specs/dbo-ubatch.md` | `INVENTORIED` | - | | `ENG-MOE-SHARED-AUX` | MoE shared-expert MLP on an aux CUDA stream concurrent with the routed-expert router/align/grouped-GEMMs (mirror vLLM's decode overlap; the largest remaining 35B c1/c2 engine lever). Fork the shared MLP onto a 2nd persistent per-device stream, join before the combine → byte-identical to serial (independent shared/routed paths both complete before combine; overlap changes WHEN not WHAT). Gated `T <= threshold` decode + CUDA. The aux stream draws scratch from a SEPARATE `AuxPool` so the concurrent main-stream routed allocations never share a live block with it (the `DevicePool` reuse invariant is single-stream ordering; vLLM sidesteps this with its stream-aware caching allocator's `record_stream`). `VT_MOE_SHARED_AUX_STREAM` **DEFAULT ON** (`=0` rollback); `VT_MOE_SHARED_AUX_THRESHOLD` (default 128; GB10 48-SM calibration). Captured in the decode CUDA-graph via the fork/join event edges (`ThreadLocal` capture, no abort). Only the committed Marlin MoE decode path; wmma fallback/CPU/GGUF and 27B dense unaffected | T1 | `vllm/model_executor/layers/fused_moe/runner/shared_experts.py:99-104,125-142`; `vllm/utils/multi_stream_utils.py:20-58` (`maybe_execute_in_parallel`, TRT-LLM port); `vllm/utils/torch_utils.py:736-756` (`aux_stream`); `vllm/envs.py:260` (threshold 256) | fork/join `src/vllm/model_executor/models/qwen3_5.cpp:3999,4114` (`MoeBlockFusedMarlinCuda`); aux stream+events `src/vllm/model_executor/models/qwen3_5.cpp:3575,3581` (`MoeAuxStream`/`MoeAuxStreamFor`); predicates `:3553,3560`; aux-pool isolation `:496,3538` (`AuxPool`/`ActivePool`/`ActivePoolScope`) + `DBuf pool_` routing `:645` | **DGX (prod flags, one flock):** overlap ON==OFF BYTE-IDENTICAL — `tests/parity/test_qwen36_paged_engine.cpp:116` 35B **315/315** + `tests/parity/test_qwen27_paged_engine.cpp` 27B **235/235** under `VT_MOE_SHARED_AUX_STREAM`∈{0,1}; captured-vs-eager (`VLLM_CPP_CUDAGRAPH=0`, ON) 315/315; shipping default (no env) 315/315+235/235, rollback `=0` 315/315+235/235; `compute-sanitizer memcheck` (default ON, captured) 0 errors; in-situ interleaved TPOT A/B (drop cold rep1) c1 −5.6% / c2 −2.7% / c4 −3.7% / c8 −3.4% / c16 −1.6% / c32 −1.5% (WINS every conc, zero regression); ledger [parity-ledger.md](parity-ledger.md) 2026-07-19 row | [moe-shared-aux-stream.md](specs/moe-shared-aux-stream.md) | `ANCHOR-BACKFILL` | `CLAIM-MOE-SHARED-AUX-1` | | `ENG-RUNNER-MODELSHAPE` | **Runner is model-shape-agnostic over the KV-cache group structure** — the extensibility deliverable the first additive-model bring-up (Qwen3 dense) forced. Before W1 the `GPUModelRunner` had only ever executed the Qwen3.6 HYBRID topology and hardcoded it in two places: (#1) the KV-buffer alloc loop indexed `config_.layer_types[l]`, out-of-bounds on a pure-dense model's EMPTY `layer_types`; (#2) each `execute_model` step unconditionally built the GDN metadata (`gather_block_table(gdn_group_id_)` / `remap_gdn_state_slots` / `GDNAttentionMetadataBuilder`), which reads `block_table[-1]` when there is no mamba group. W1 drives both off the resolved KV-group structure — a model-agnostic `has_mamba_group` / `gdn_group_id_ >= 0` predicate (NOT a model-name check): empty/absent `layer_types` ⇒ all full-attention; no mamba group ⇒ the whole GDN metadata/state path is skipped and `gdn_meta` stays default-empty. A full-attention-only KV config (one FA group, no MambaSpec) now allocates + steps cleanly; the hybrid gate models keep their GDN group so their path is BYTE-IDENTICAL. This is a one-time generalization: every future dense/non-hybrid arch (Llama, Mistral) now adds new-files-only, zero further runner edits. **PER-LAYER KV head_dim extension (Gemma-4 G1b, 2026-07-28, `CLAIM-GEMMA4-G1B`):** the runner's full-attn alloc/view loops now consume an OPTIONAL `KVCacheConfig::per_layer_attn_specs` (index == layer) so a HETEROGENEOUS-head_dim model (Gemma-4: sliding 256 / global 512, same num_kv_heads) sizes each non-GDN layer's paged KV + PagedKvCache view from its OWN spec. The field is EMPTY for every uniform-KV model ⇒ the loop collapses to the single group spec ⇒ byte-identical allocation/view/indexing/dispatch (same additive-identical property as the model-shape generalization above). Block table / KV manager / scheduler stay head_dim-independent (num_blocks + block_size, uniform) so no per-group block table is introduced | T0 | model-agnostic runner drives off `kv_cache_config.kv_cache_groups` — `vllm/v1/worker/gpu/model_runner.py` `initialize_kv_cache` / attention-metadata build (per-group, no hardcoded hybrid) @ `e24d1b24` | `src/vllm/v1/worker/gpu/runner.cpp:458-470` (alloc loop: `has_mamba_group && !layer_types.empty()` gate) + `:651-680` (GDN metadata build gated on `gdn_group_id_ >= 0`, default-empty `gdn_meta` otherwise); per-layer KV head_dim: `include/vllm/v1/kv_cache_interface.h` (`KVCacheConfig::per_layer_attn_specs`) consumed in `src/vllm/v1/worker/gpu/runner.cpp` `initialize_kv_cache` (per-layer `FaDims` alloc+view), published by `src/vllm/model_executor/models/gemma4_registry.cpp` (`MakeGemma4ForConditionalGenerationKVCache`); the full-attention-only KV spec that exercises the base path `src/vllm/model_executor/models/qwen3_dense.cpp` (`MakeQwen3ForCausalLMKVCache`) | `tests/vllm/v1/worker/test_runner.cpp:1129` — "full-attention-only KV config allocates without the GDN path" + "full-attention-only step skips GDN metadata build (no OOB)" (RED→GREEN: both SIGSEGV pre-generalization; GREEN post). Behaviour-preservation gate: DGX **27B 235/235 + 35B 315/315 UNCHANGED** under the fix; per-layer-KV inertness: full CPU runner/KV suite green + **OLMo-2 SACRED GPU re-gate 16/16 UNCHANGED**; heterogeneous path proven by **Gemma-4 E4B STRICT 32/32** (`tests/parity/test_gemma4_paged_engine.cpp`); ASan/UBSan clean on the affected paths | [first-additive-model-qwen3-dense.md](specs/first-additive-model-qwen3-dense.md) §3 (seam gaps #1/#2), §6 (W1); [gemma4-multimodal.md](specs/gemma4-multimodal.md) §G1b | `ACTIVE` | `CLAIM-MODEL-QWEN3-DENSE` | -| `ENG-MM-INPUT-PIPELINE` | **Multimodal INPUT pipeline + encoder-cache engine seam (M1), INERT when no mm input.** The C++ mirror of `vllm/multimodal/`: `MultiModalKwargs`/`MultiModalFeatureSpec`/`MultiModalInputs`, the `MultiModalHasher` mm-hash (blake3), the Qwen3-VL image processor (smart_resize + fused rescale/normalize + patchify -> `pixel_values`+`image_grid_thw`) and placeholder-token expansion, plus the `EncoderCacheManager` (+`ComputeMmEncoderBudget`) and the LMCache `extra_keys` seam. Additive `mm_features` carried on `Request`/`EngineCoreRequest`; with NO mm input every field is empty and every path is byte-identical to the text engine. Processor output is BIT/BYTE-identical to the vLLM 0.25.0 oracle (M0 fixture). Does NOT build the vision tower / embed-merge (M2). **SERVING wiring (ROAD-V1-MM `MM-SERVE-ENGINE`, 2026-07-28, `CLAIM-MM-SERVING-W2`):** the OpenAI server now carries the parsed `MultiModalInputs` into the engine — additive `LLMEngine`/`AsyncLLM` `add_request(MultiModalInputs)`+`generate(MultiModalInputs)` overloads via `InputProcessor::process_inputs_mm` (mirror `input_processor.py:333-379`, empty mm_features == the tokens path), the chat-template placeholder-STRING helpers (`get_placeholder_str`/`_add_placeholder` mirror), and the serving_chat `MultiModalChatFn` seam (default unset ⇒ text byte-identical). **SEAM BODY (ROAD-V1-MM `MM-SERVE-E2E` W3, 2026-07-28, `CLAIM-MM-SERVING-E2E`):** `MakeQwen3VLImageChatFn` (chat_mm.cpp) is the seam body the server sets — messages → marker-inject → chat template → `EncodeWithSpecialTokens` (the single image_pad marker → one image_token_id) → `RouteImageRgb` EXPAND to 196 image tokens + mm_features; wired in `examples/server/main.cpp` (guarded on `preprocessor_config.json`; text-only unset ⇒ byte-identical). Gated `test_chat_mm` 8/8 + `test_openai_serving` (seam invoked + routed). **ENGINE MM-FORWARD LANDED (ROAD-V1-MM `MM-SERVE-E2E`, 2026-07-28, `CLAIM-ENGINE-MM-FORWARD`):** the engine model runner now HAS an mm forward — `ModelForwardInput` gains an ADDITIVE default-nullopt `std::optional mm` (merged inputs_embeds + 3-D MRoPE positions + DeepStack, borrowed handles; nullopt-for-text ⇒ shared runner path byte-identical BY CONSTRUCTION), `Qwen3VLForConditionalGeneration` is `REGISTER_VLLM_MODEL`-registered (`qwen3_vl_registry.cpp`), and the registered forward FOLDS the M2c decode into `ModelRegistry::Forward` via the SHARED `Qwen3VLForwardStepLastLogits` (`Qwen3VLGenerateGreedyViaRegistry` drives every step through the registry). GPU token-exact gate `test_qwen3vl_registry_e2e` (image→text THROUGH `ModelRegistry::Forward` == M2c golden 32/32 STRICT, dgx.casa GB10); text inertness `test_runner` 16/16 + `test_scheduler` 36/36 + `test_model_registry` 24/24 + `test_chat_mm` 8/8 + `test_openai_serving` 41/41 all green. RESIDUAL: the FULL in-runner scheduler-fed tower run (batched-loop mm building the field from staged encoder outputs) + the real server `/v1/chat/completions` GPU e2e — recipe in `specs/mm-serving.md`. **INPUT LIMITS L1 LANDED (#607, 2026-08-13):** the per-modality `limit_per_prompt` + `GetLimitPerPrompt` precedence (`language_model_only` ⇒ 0 BEFORE the map, else the map, else 999) and the refusal that gives those numbers effect — `AllowedMmLimits` folding by `min()` against the model's own ceiling, `ValidateNumItems` with upstream's exact message, and both call sites with the `enable_mm_embeds` escape. NO serve surface and NO live call site: nothing constructs a `MultiModalConfig` on a request yet, which is L2's. | T1 | `vllm/multimodal/{inputs.py,hasher.py:50,processing/processor.py:1663,processing/inputs.py:62}`; `vllm/model_executor/models/qwen3_vl.py:{1400,1233}`; `vllm/v1/core/encoder_cache_manager.py:17`; transformers `image_processing_qwen2_vl.py:62`, `image_processing_backends.py:327`; tests `tests/multimodal/test_processing.py`, `tests/multimodal/test_hasher.py`, `tests/v1/core/test_encoder_cache_manager.py` @ `e24d1b24` | `src/vllm/multimodal/hasher.cpp`, `src/vllm/multimodal/qwen3vl_processor.cpp`, `include/vllm/multimodal/{inputs.h,hasher.h,qwen3vl_processor.h}`; `src/vllm/v1/core/encoder_cache_manager.cpp` + `include/vllm/v1/core/encoder_cache_manager.h`; additive inert fields `include/vllm/v1/request.h` + `src/vllm/v1/request.cpp` + `include/vllm/v1/engine/types.h`; `extra_keys` seam `include/vllm/v1/kv_offload/lmcache/chunked_token_database.h` + `.cpp`; M0 `scripts/mm/m0_oracle_capture.py`; L1 limits `include/vllm/config/multimodal.h` + `include/vllm/multimodal/processing/context.h` + `src/vllm/multimodal/processing/context.cpp`, refusal type relocated to `include/vllm/v1/engine/validation_error.h` — anchor `src/vllm/multimodal/hasher.cpp:56` | `tests/vllm/multimodal/test_qwen3vl_processor.cpp` (processor-parity 23/23 BIT-identical vs the M0 oracle fixture `tests/vllm/multimodal/fixtures/qwen3vl/`, RED-first: wrong normalize shift -> 1.2M mismatches); `tests/vllm/v1/core/test_encoder_cache_manager.cpp` 32/32. Text-inertness: `test_request`/`test_engine_types`/`test_lmcache_codec`/`test_lmcache_key_agreement`/`test_openai_conformance` all green standalone; SACRED CUDA 27B/35B/Coder = GPU inertness proof; `check-device-leakage` OK — anchor `tests/vllm/multimodal/test_qwen3vl_processor.cpp:59`. L1 limits: `tests/vllm/config/test_multimodal_config.cpp` 7/7 (21 assertions) + `tests/vllm/multimodal/test_processing_limits.cpp` 19/19 (78 assertions), porting `tests/multimodal/test_processing.py:902-941,944-985`, `tests/entrypoints/multimodal/llm/test_mm_embeds_only.py:41-49` and `tests/entrypoints/unit_tests/test_chat_utils.py:1498-1560` @ `5559679229bc`; mutations proven RED: map-before-flag precedence, the dropped throw, the dropped `min()` fold | [multimodal-track.md](specs/multimodal-track.md) §3 (M0/M1) | `READY` | - | +| `ENG-MM-INPUT-PIPELINE` | **Multimodal INPUT pipeline + encoder-cache engine seam (M1), INERT when no mm input.** The C++ mirror of `vllm/multimodal/`: `MultiModalKwargs`/`MultiModalFeatureSpec`/`MultiModalInputs`, the `MultiModalHasher` mm-hash (blake3), the Qwen3-VL image processor (smart_resize + fused rescale/normalize + patchify -> `pixel_values`+`image_grid_thw`) and placeholder-token expansion, plus the `EncoderCacheManager` (+`ComputeMmEncoderBudget`) and the LMCache `extra_keys` seam. Additive `mm_features` carried on `Request`/`EngineCoreRequest`; with NO mm input every field is empty and every path is byte-identical to the text engine. Processor output is BIT/BYTE-identical to the vLLM 0.25.0 oracle (M0 fixture). Does NOT build the vision tower / embed-merge (M2). **SERVING wiring (ROAD-V1-MM `MM-SERVE-ENGINE`, 2026-07-28, `CLAIM-MM-SERVING-W2`):** the OpenAI server now carries the parsed `MultiModalInputs` into the engine — additive `LLMEngine`/`AsyncLLM` `add_request(MultiModalInputs)`+`generate(MultiModalInputs)` overloads via `InputProcessor::process_inputs_mm` (mirror `input_processor.py:333-379`, empty mm_features == the tokens path), the chat-template placeholder-STRING helpers (`get_placeholder_str`/`_add_placeholder` mirror), and the serving_chat `MultiModalChatFn` seam (default unset ⇒ text byte-identical). **SEAM BODY (ROAD-V1-MM `MM-SERVE-E2E` W3, 2026-07-28, `CLAIM-MM-SERVING-E2E`):** `MakeQwen3VLImageChatFn` (chat_mm.cpp) is the seam body the server sets — messages → marker-inject → chat template → `EncodeWithSpecialTokens` (the single image_pad marker → one image_token_id) → `RouteImageRgb` EXPAND to 196 image tokens + mm_features; wired in `examples/server/main.cpp` (guarded on `preprocessor_config.json`; text-only unset ⇒ byte-identical). Gated `test_chat_mm` 8/8 + `test_openai_serving` (seam invoked + routed). **ENGINE MM-FORWARD LANDED (ROAD-V1-MM `MM-SERVE-E2E`, 2026-07-28, `CLAIM-ENGINE-MM-FORWARD`):** the engine model runner now HAS an mm forward — `ModelForwardInput` gains an ADDITIVE default-nullopt `std::optional mm` (merged inputs_embeds + 3-D MRoPE positions + DeepStack, borrowed handles; nullopt-for-text ⇒ shared runner path byte-identical BY CONSTRUCTION), `Qwen3VLForConditionalGeneration` is `REGISTER_VLLM_MODEL`-registered (`qwen3_vl_registry.cpp`), and the registered forward FOLDS the M2c decode into `ModelRegistry::Forward` via the SHARED `Qwen3VLForwardStepLastLogits` (`Qwen3VLGenerateGreedyViaRegistry` drives every step through the registry). GPU token-exact gate `test_qwen3vl_registry_e2e` (image→text THROUGH `ModelRegistry::Forward` == M2c golden 32/32 STRICT, dgx.casa GB10); text inertness `test_runner` 16/16 + `test_scheduler` 36/36 + `test_model_registry` 24/24 + `test_chat_mm` 8/8 + `test_openai_serving` 41/41 all green. RESIDUAL: the FULL in-runner scheduler-fed tower run (batched-loop mm building the field from staged encoder outputs) + the real server `/v1/chat/completions` GPU e2e — recipe in `specs/mm-serving.md`. **INPUT LIMITS L1 LANDED (#607, 2026-08-13):** the per-modality `limit_per_prompt` + `GetLimitPerPrompt` precedence (`language_model_only` ⇒ 0 BEFORE the map, else the map, else 999) and the refusal that gives those numbers effect — `AllowedMmLimits` folding by `min()` against the model's own ceiling, `ValidateNumItems` with upstream's exact message, and both call sites with the `enable_mm_embeds` escape. NO serve surface and NO live call site: nothing constructs a `MultiModalConfig` on a request yet, which is L2's. **INPUT LIMITS L2 LANDED (#607, #686, 2026-08-14):** the flags (`--[no-]language-model-only`, `--limit-mm-per-prompt ''`; `arg_utils.py:555-556,1276-1279,1691-1692` over `ParseLimitMmPerPromptJson`, the port of `multimodal.py:212-236` + the DummyOptions dataclasses `:17-43`), the C-ABI fields (`vllm_model_params.language_model_only`/`.limit_mm_per_prompt`, **ABI v19**), and the LIVE CALL SITE: `ValidateChatMmLimits` (`chat_utils.py:648-662`) runs as step 0 of `MakeQwen3VLImageChatFn` over a `BaseProcessingInfo` folding `LoadedEngine::mm_config()` with `Qwen3VLChatSupportedMmLimits() == {"image": 1}` — the seam's own ceiling, which is the `min()` fold operand #686 recorded as undeclared. A three-image request is now HTTP 400 with upstream's message rather than an opaque 500 / a truncated answer. NOT claimed: any memory win (L3 gates tower construction, unmeasured). Still unwired: the `process_inputs_mm` call site (`context.py:461`), blocked on the per-model `get_supported_mm_limits()` hook. | T1 | `vllm/multimodal/{inputs.py,hasher.py:50,processing/processor.py:1663,processing/inputs.py:62}`; `vllm/model_executor/models/qwen3_vl.py:{1400,1233}`; `vllm/v1/core/encoder_cache_manager.py:17`; transformers `image_processing_qwen2_vl.py:62`, `image_processing_backends.py:327`; tests `tests/multimodal/test_processing.py`, `tests/multimodal/test_hasher.py`, `tests/v1/core/test_encoder_cache_manager.py` @ `e24d1b24` | `src/vllm/multimodal/hasher.cpp`, `src/vllm/multimodal/qwen3vl_processor.cpp`, `include/vllm/multimodal/{inputs.h,hasher.h,qwen3vl_processor.h}`; `src/vllm/v1/core/encoder_cache_manager.cpp` + `include/vllm/v1/core/encoder_cache_manager.h`; additive inert fields `include/vllm/v1/request.h` + `src/vllm/v1/request.cpp` + `include/vllm/v1/engine/types.h`; `extra_keys` seam `include/vllm/v1/kv_offload/lmcache/chunked_token_database.h` + `.cpp`; M0 `scripts/mm/m0_oracle_capture.py`; L1 limits `include/vllm/config/multimodal.h` + `include/vllm/multimodal/processing/context.h` + `src/vllm/multimodal/processing/context.cpp`, refusal type relocated to `include/vllm/v1/engine/validation_error.h`; L2 flags+ABI+call site `src/vllm/config/multimodal.cpp` (`ParseLimitMmPerPromptJson`) + `src/vllm/entrypoints/openai/server_main.cpp` + `include/vllm.h` (ABI v19) + `src/capi/vllm_c.cpp` + `EngineParams::multimodal`/`LoadedEngine::mm_config()` + `src/vllm/entrypoints/openai/chat_mm.cpp` (`ChatPartModality`, `ValidateChatMmLimits`, `Qwen3VLChatSupportedMmLimits`) — anchor `src/vllm/multimodal/hasher.cpp:56` | `tests/vllm/multimodal/test_qwen3vl_processor.cpp` (processor-parity 23/23 BIT-identical vs the M0 oracle fixture `tests/vllm/multimodal/fixtures/qwen3vl/`, RED-first: wrong normalize shift -> 1.2M mismatches); `tests/vllm/v1/core/test_encoder_cache_manager.cpp` 32/32. Text-inertness: `test_request`/`test_engine_types`/`test_lmcache_codec`/`test_lmcache_key_agreement`/`test_openai_conformance` all green standalone; SACRED CUDA 27B/35B/Coder = GPU inertness proof; `check-device-leakage` OK — anchor `tests/vllm/multimodal/test_qwen3vl_processor.cpp:59`. L1 limits: `tests/vllm/config/test_multimodal_config.cpp` 7/7 (21 assertions) + `tests/vllm/multimodal/test_processing_limits.cpp` 19/19 (78 assertions), porting `tests/multimodal/test_processing.py:902-941,944-985`, `tests/entrypoints/multimodal/llm/test_mm_embeds_only.py:41-49` and `tests/entrypoints/unit_tests/test_chat_utils.py:1498-1560` @ `5559679229bc`; mutations proven RED: map-before-flag precedence, the dropped throw, the dropped `min()` fold. L2 (aarch64 `build-test-cpu-arm64` lane, `-DVLLM_CPP_CUDA=OFF`): `tests/vllm/entrypoints/openai/test_serve_mm_limits.cpp` 10/10 (101 assertions, flags + the parser's upstream refusals) + `test_chat_mm` 11/11 (126) + `test_openai_api_server` 56/56 (638, the HTTP 400 arm proven against BOTH a 500 and a truncated 200); RED-first behavioural: `CHECK(500 == 400)`, `CHECK("InternalServerError" == BadRequestError)` and `CHECK(200 == 400)`; mutations proven RED: flag→config plumbing dropped, the call-site wiring dropped, and the refusal re-typed off `InputValidationError` (which lands as the 500 L1's design avoided) | [multimodal-track.md](specs/multimodal-track.md) §3 (M0/M1) | `READY` | - | | `ENG-MM-VISION-TOWER` | **Qwen3-VL vision tower `Qwen3_VisionTransformer` (M2a), proven faithful vs vLLM 0.25.0 in isolation.** The reusable vision half of the whole Qwen3-VL family + Qwen3.6 (27B/35B share this exact tower). Pure-additive C++ forward composed from public vt:: ops: patch-embed (Conv3d-as-matmul + bias), host pos-embed bilinear-interp+spatial-merge-reorder, 24 ViT blocks (LayerNorm + vision attention with partial-rotary NeoX vision RoPE via `vt::RopeFromCache` + non-causal `vt::Attention(causal=false)` + tanh-GELU MLP), patch merger (LayerNorm + exact-erf-GELU + 2 FCs), DeepStack 3 post-shuffle-norm mergers at layers 5/11/17 → `[196,10240]`. Adds 2 additive elementwise vt ops (`GeluTanh`/`GeluErf`). NO runner/model/registry edit → text engines byte-identical by construction. Proven faithful in ISOLATION; the merge into `input_embeds` + the MRoPE/DeepStack text backbone + the e2e image gate are M2b/M2c. | T1 | `vllm/model_executor/models/qwen3_vl.py` `Qwen3_VisionPatchEmbed:347`, `Qwen3_VisionBlock:413`, `Qwen3_VisionPatchMerger:467`, `Qwen3_VisionTransformer:519`, `forward:800`, `pos_embed_interpolate_native:277`, `rot_pos_emb:667`; `qwen2_5_vl.py::Qwen2_5_VisionAttention.forward:397`; `rotary_embedding/common.py::ApplyRotaryEmb.forward_static:151` @ `e24d1b24` | `src/vllm/model_executor/models/qwen3_vl_vision.{h,cpp}`; 2 vt ops `include/vt/ops.h` + `src/vt/ops.cpp` + `src/vt/cuda/cuda_layernorm.cu` + `src/vt/cpu/cpu_layernorm.cpp`; dumps `scripts/mm/m2a_tower_{ref,weight}_dump.py`; fixtures `tests/vllm/multimodal/fixtures/qwen3vl_tower/` | `tests/vllm/multimodal/test_qwen3vl_tower.cpp` — 4 RED-first tower gates vs the dumped vLLM-0.25.0 reference 348/348 (patch-embed 2.1e-3, block0 6.8e-3, merger 6.5e-2, DeepStack 1.2e-2/3.3e-2/4.4e-2, full tower 5.1e-2; pos-embed 2.5e-3 + rope 1.9e-3 TIGHT); bf16-depth envelope RCA'd; RED = rope disabled → block0 0.149/tower 0.75/6 fails; cutlass-ON+FA2 banner; clean `-Werror`; compute-sanitizer 0 — anchor `tests/vllm/multimodal/test_qwen3vl_tower.cpp:96` | [multimodal-track.md](specs/multimodal-track.md) §3 (M2a) | `ACTIVE` | `CLAIM-MULTIMODAL-M2A` | | `ENG-MM-TEXT-BACKBONE` | **Qwen3-VL text-backbone numeric contracts `Qwen3VLGetRopeIndex`/`Qwen3VLMergeMultimodal`/`Qwen3VLComputeDeepstack` (M2b/M2c), unit-green vs vLLM 0.25.0.** The deterministic pieces that fork the plain Qwen3-dense text path for a vision-conditioned decode: (1) MRoPE 3-D `get_rope_index` positions [3,T] (image tokens get (t,h,w) grid positions, text sequential); (2) the 3-section MRoPE APPLICATION — proven to be the EXISTING `vt::RopeFromCache` mrope path (positions [3,T] + `mrope_section=[24,20,20]` interleaved), faithful to `MRotaryEmbedding.forward_native` for Qwen3-VL's exact config; (3) `_compute_deepstack_embeds` scatter → [L,T,H] decoder-injection tensor; (4) `_merge_multimodal_embeddings` masked scatter of the tower's `[:,:2560]` into `input_embeds`. Pure-additive TU — NO shared dense forward / runner / registry edit → text engines byte-identical by construction. The e2e image forward (VL weight loader + forked MRoPE/DeepStack decode loop) is the remaining M2c wire-up. | T1 | `vllm/model_executor/models/qwen3_vl.py` `_get_mrope_input_positions:2567`, `_iter_mm_grid_hw:2482`, `_compute_deepstack_embeds:2761`, `Qwen3LLMModel.forward` deepstack `:1589`; `vllm/model_executor/models/utils.py::_merge_multimodal_embeddings:524`; `vllm/model_executor/layers/rotary_embedding/mrope.py` MRotaryEmbedding @ `e24d1b24` | `src/vllm/model_executor/models/qwen3_vl_text.{h,cpp}`; existing `vt::RopeFromCache` mrope path (`src/vt/{cpu,cuda}/*`); dump `scripts/mm/m2b_text_ref_dump.py`; fixtures `tests/vllm/multimodal/fixtures/qwen3vl_text/` — anchor `src/vllm/model_executor/models/qwen3_vl_text.cpp:9` | `tests/vllm/multimodal/test_qwen3vl_text.cpp` — 4 RED-first gates vs the dumped vLLM-0.25.0 reference 85/85 (get_rope_index BIT-exact [3,204], delta −182; MRoPE q rel-L2 1.5e-3 / k 1.5e-3, RED interleaved-off >5e-2; DeepStack + merge BIT-exact); CPU-only, no weights; clean CPU `-Werror` — anchor `tests/vllm/multimodal/test_qwen3vl_text.cpp:99` | [multimodal-track.md](specs/multimodal-track.md) §3 (M2b/M2c) | `ACTIVE` | `CLAIM-MULTIMODAL-M2BC` | | `ENG-MM-QWEN36-VL-FORWARD` | **Qwen3.6-27B (`Qwen3_5ForConditionalGeneration`) GDN-hybrid VL forward — IMAGE (M3-b) + VIDEO (M3d) BOTH e2e, STRICT gates PASS 32/32. Our own gate model's image+video paths now work end-to-end (speed pending).** The genuinely-new integration completing our own gate model's mm paths: fork the landed bf16 `Qwen3_5DenseModel` GDN-hybrid forward (48 GDN + 16 full-attn) on gated, default-off points so a text-only 27B request stays byte-identical — (a) `inputs_embeds` entry (embed ids + `Qwen3VLMergeMultimodal` scatter of the 27B tower merger `[N,5120]` into the visual-token rows; 27B has EMPTY `deepstack_visual_indexes` ⇒ NO DeepStack); (b) 3-section MRoPE (`mrope_section=[11,11,10]` interleaved, rotary_dim 64, theta 1e7) in the 16 full-attn layers only via the proven `vt::RopeFromCache` mrope path (GDN layers carry no rope); (c) mixed load = the M2a `Qwen3_VisionTransformer` (27B vision config, empty deepstack) bf16 tower + the bf16 GDN-hybrid LLM via the EXISTING `LoadQwen3_5Dense`. **M3d (2026-07-25) added VIDEO by REUSE:** the M3-b image driver refactored into a shared `VLGenerateCoreGdn`, image+video wrappers differ ONLY in the merge mask (`image_token` vs `video_token` across frames) + the get_rope_index (`Qwen3VLGetRopeIndex` vs `Qwen3VLGetRopeIndexVideo`); the M3c processor/windowed-tower/video-MRoPE are reused verbatim. | T1 | `vllm/model_executor/models/qwen3_5.py:389` (`Qwen3_5ForConditionalGeneration` subclasses `Qwen3VLForConditionalGeneration`; `visual = Qwen3_VisionTransformer`, modalities {"image","video"}); `qwen3_vl.py` `_process_video_input:2165`, `_get_mrope_input_positions:2567` video branch, `get_video_repl:1479`; the 27B `config.json` (`mrope_section=[11,11,10]`, empty `deepstack_visual_indexes`) @ `e24d1b24` / vLLM 0.25.0 | **M3-b + M3d BUILT + GATED 2026-07-25:** vision-only loader `LoadQwen3VLVisionWeights` (`src/vllm/model_executor/models/qwen3_vl.cpp`, 27B config) + shared `VLGenerateCoreGdn` + image driver `Qwen3_5VLGenerateGreedy` + **video driver `Qwen3_5VLGenerateGreedyVideo`** + `BuildMropeCosSinHost` + the `mrope_cos_sin` param on `DenseForwardLayers` (`src/vllm/model_executor/models/qwen3_5.cpp`, nullptr on every text caller ⇒ byte-identical; the video driver is purely additive, the shared text forward UNTOUCHED per `git diff --stat`) reusing M2a tower + `LoadQwen3_5Dense` bf16 LLM | **IMAGE:** golden `tests/vllm/multimodal/fixtures/qwen3_5_27b/` (STRICT sha256 `ead4b484…`); STRICT image gate PASS **32/32** (`test_qwen3_5_vl_e2e.cpp`, 54/54, re-run post-refactor). **VIDEO (M3d):** oracle `scripts/mm/m3d_video_oracle_capture.py` on the M3c synthetic clip (raw sha `8a111599…`, grid `[4,8,8]`, 64 video tokens) K=5 DETERMINISTIC ⇒ STRICT golden; **STRICT video gate PASS 32/32** (`test_qwen3_5_vl_video_e2e.cpp`, 27/27; near-tie gaps 0.0000 nats everywhere), fixtures `tests/vllm/multimodal/fixtures/qwen3_5_27b_video/`. Text-inertness 27B 235/235, 35B 315/315, Coder 138/138 (by construction); clean `-Werror` 0 warn; compute-sanitizer 0 on the 27B video forward. **SPEED MEASURED (2026-07-26, `CLAIM-MULTIMODAL-SPEED`): image c1 vs vLLM 0.25.0 GRAPHED — decode TPOT 225.0 ms/tok vs 226.9 = AT PARITY (0.99×), LLM prefill 326 ms vs vLLM TTFT 321 ms = at parity; vision tower WAS 2114 ms vs vLLM encode ≤~250 ms = ~10× (THE gap). TOWER LEVER EXECUTED (2026-07-26, `CLAIM-MULTIMODAL-SPEED-TOWER`, [multimodal-speed.md](specs/multimodal-speed.md) §7): nsys `cuda_gpu_kern_sum` attributed 98.9 % of the tower forward to the naive `vt::cuda::AttentionKernel` (56 ms/block; NOT QKV/FA2-routing); fixed by a warp-scoped online-softmax op `AttentionDenseFast` (separate op ⇒ `kAttention`/text byte-identical) + one-time resident-weight load ⇒ per-image tower 2114 → 148 ms (14.3×), **0.59× vs vLLM eager encode = FASTER**. STRICT image/video e2e HELD 32/32 (+4B DeepStack 32/32), `test_ops_attention` 37239/37239, 27B text SACRED 235/235, compute-sanitizer memcheck 0, clean `-Werror`. `benchmark_binding=false`, single-seq driver (no c2+/server). Remaining: batched/graphed mm serving (c2+) + audio our-side — DONE bar not yet met.** | [multimodal-track.md](specs/multimodal-track.md) §M3 + [multimodal-speed.md](specs/multimodal-speed.md) §7 + §8 (decode lever #2 CLOSED 2026-07-27: on-GPU greedy argmax + decode embed round-trip removed on `VLGenerateCoreGdn`; bit-exact — image/video STRICT 32/32 held; 27B decode NEUTRAL at the ~222 ms bandwidth floor) + §9 (lever #3 FIRST BRICK 2026-07-27, `CLAIM-MULTIMODAL-SPEED-GRAPH`: the shared `VLGenerateCoreGdn` decode step now routes through the production `Qwen3_5DenseDecodeGraph` cold→warm→replay captured decode — the mm decode is now GRAPH-CAPTURABLE, closing the un-graphed-eager-loop structural gap; S==B==1 bit-identical rebuild; token-exact HELD image/video STRICT 32/32 with 30 graph replays confirmed; A/B graphed 232.5 vs eager 233.4 ms/tok = NEUTRAL at the 27B bandwidth floor; the launch-overhead win + batched c2+ + serving ingestion are the recorded W-plan W1-W3) + §16 (vision-forward flash kernel 2026-07-28, `CLAIM-MM-SPEED-QWEN-IMAGE`: ATTRIBUTION-FIRST nsys attributed ~85% of the 148 ms tower forward to the dense attention `AttentionWarpKernel` [4.66 ms/block×27]; routed it to the §14 flash-tiled `vt::AttentionDenseFlash` [head_dim 72, byte-identical — per-warp math verbatim, only K/V from shared-mem tiles]. STRICT image/video e2e HELD 32/32 [27B+4B], `test_ops_attention` 37239/37239, goldens md5 UNCHANGED, nsys proof `AttentionDenseFlashKernel` 24 inst/zero warp, RED 30/46→46/46, sanitizer 0. A/B warp 148.3→flash 142.3 ms = 1.04× — the profile REFUTED a big lever: at t=784 the vision attention is serial-latency-bound not bandwidth-bound [audio §14 was 1.82× at t=1500], flash recovers only ~6 ms. **HONEST: the tower ALREADY BEATS vLLM — 142 ms vs ~250 ms eager encode = 0.57×**; image/video mm-forward is correctness-DONE + speed-BEATS-vLLM; residual = tensor-core MMA hd-72 attention [not needed for parity] + batched c2+/serving) | `ACTIVE` | `CLAIM-MULTIMODAL-SPEED-TOWER` + `CLAIM-MULTIMODAL-SPEED-DECODE` + `CLAIM-MULTIMODAL-SPEED-GRAPH` + `CLAIM-MM-SPEED-QWEN-IMAGE` | diff --git a/.agents/roadmap_v1.md b/.agents/roadmap_v1.md index 3e8df2f5c..66a2ec5be 100644 --- a/.agents/roadmap_v1.md +++ b/.agents/roadmap_v1.md @@ -192,7 +192,7 @@ issue is not yet placed. Keyed record: update in place, never append. | [#558](https://github.com/mudler/vllm.cpp/issues/558) | — | `tests/parity/hf_snapshot.h` has no guard against declaration-order breaks: the C++ build catches them, but the records-only lane that broke it never builds C++, and all 14 TUs that include the header are checkpoint-gated so `ctest` reports the break as `***Not Run`. `fafa16f0f` (#546, #551) fixed the ordering and carried no guard | bug | | [#603](https://github.com/mudler/vllm.cpp/issues/603) | — | `windows-msvc-cpu` / `windows-msvc-vulkan` are RED on `main`: `test_backend_cross_device.cpp` calls POSIX `setenv`/`unsetenv`, which MSVC does not provide | bug | | [#606](https://github.com/mudler/vllm.cpp/issues/606) | — | `vllm-serve` aborts on `--enable-auto-tool-choice` (89/157 recipes) and `--trust-remote-code` (82/157), both of which are no-ops for us, so a copy-pasted official recipe command never reaches model load. Needs an accepted-and-inert seam with a per-flag reason; no row owns serve CLI recipe compatibility | feature | -| [#607](https://github.com/mudler/vllm.cpp/issues/607) | `ENG-MM-INPUT-PIPELINE` | **Premise corrected 2026-08-13, see `specs/multimodal-track.md` §1.5.** Not "skip the vision encoder": `--language-model-only` sets every modality limit to **0** (`multimodal.py:78,321-327`) and is sugar over `--limit-mm-per-prompt`. Two consequences follow, and we have NEITHER — upstream then **refuses every multimodal request** (`processing/context.py:409-428` raises "At most 0 image(s) may be provided in one prompt", from `:461` and `chat_utils.py:662`), and builds the tower uninitialised (`interfaces.py:293`). This is a PORT of the limits mechanism (L1-L4), not the exposure of a boolean. **L1 LANDED 2026-08-13:** `vllm::MultiModalConfig` + `GetLimitPerPrompt` (`include/vllm/config/multimodal.h`) and the refusal it carries (`include/vllm/multimodal/processing/context.h`) are in, unit-gated, with NO serve surface — nothing constructs the config on a live request yet, so the two call sites still validate nothing. L2 (the flags + C-ABI field + wiring those call sites), L3 (tower skip) and L4 (kernel gate) are still owed. 43 of 157 recipes pass the flag and we abort on it. The flag appears in this repo only in `tools/bench/run_serve_low.py`, which passes it to the ORACLE — a grep reads as coverage and is not | feature | +| [#607](https://github.com/mudler/vllm.cpp/issues/607) | `ENG-MM-INPUT-PIPELINE` | **Premise corrected 2026-08-13, see `specs/multimodal-track.md` §1.5.** Not "skip the vision encoder": `--language-model-only` sets every modality limit to **0** (`multimodal.py:78,321-327`) and is sugar over `--limit-mm-per-prompt`. Two consequences follow, and we have NEITHER — upstream then **refuses every multimodal request** (`processing/context.py:409-428` raises "At most 0 image(s) may be provided in one prompt", from `:461` and `chat_utils.py:662`), and builds the tower uninitialised (`interfaces.py:293`). This is a PORT of the limits mechanism (L1-L4), not the exposure of a boolean. **L1 LANDED 2026-08-13:** `vllm::MultiModalConfig` + `GetLimitPerPrompt` (`include/vllm/config/multimodal.h`) and the refusal it carries (`include/vllm/multimodal/processing/context.h`) are in, unit-gated, with NO serve surface. **L2 LANDED 2026-08-14 (also closing [#686](https://github.com/mudler/vllm.cpp/issues/686)):** both serve flags (`--[no-]language-model-only`, `--limit-mm-per-prompt ''` with upstream's legacy/configurable/mixed spellings and its refusals, `src/vllm/config/multimodal.cpp`), the C-ABI fields (`vllm_model_params.language_model_only` / `.limit_mm_per_prompt`, **ABI v18 → v19**), and the CALL SITE — `MakeQwen3VLImageChatFn` validates through `ValidateChatMmLimits` (port of `chat_utils.py:648-662`) over the seam's declared ceiling `{"image": 1}`, so a three-image request is `400 "At most 1 image(s) may be provided in one prompt."` instead of a truncated answer. The 43 recipes now reach model load. Gated on aarch64: `test_serve_mm_limits` 10/10, `test_chat_mm` 11/11, `test_openai_api_server` 56/56 (the HTTP 400 arm proven e2e against both 500 and a truncated 200). Still owed: L3 (tower skip — **no memory claim is made by L2**), L4 (kernel gate), and the second call site `process_inputs_mm`, which is blocked on the per-model `get_supported_mm_limits()` hook the M2 towers own. The flag appears in this repo only in `tools/bench/run_serve_low.py`, which passes it to the ORACLE — a grep reads as coverage and is not | feature | | [#651](https://github.com/mudler/vllm.cpp/issues/651) | — | `test_agent_record`'s MODEL-ratchet docstring is two contradictory paragraphs spliced together, and the surviving half records a pin transition that never happened | bug | | [#652](https://github.com/mudler/vllm.cpp/issues/652) | — | `model-matrix.md` prose counters drifted: LTX-2.5 reached the rows and the CI-enforced rollup but none of the five sentences that count them | bug | | [#659](https://github.com/mudler/vllm.cpp/issues/659) | — | LTX-2.5 device select adopts M3a's platform seam but not its companion capability guard: `ltx2_video.cpp` asks `CurrentPlatform().device_type()` and `TryGetBackend(...)` but never `supports_model_architecture`, so a PARTIAL backend (Metal 15/75 ops, Tenstorrent) is handed a queue and dies in a kernel bind where it used to be refused BY NAME (found while reviewing #553 for landing) | bug | @@ -618,7 +618,7 @@ degraded run — it is no run at all. | `--tool-call-parser` | 42 names, **84/90 recipe uses (93%)** — the healthy axis; `inkling` (2 uses) landed 2026-08-13 under #608 W1 | [#608](https://github.com/mudler/vllm.cpp/issues/608) for the last 6 | | `--reasoning-parser` | 10 of 28 names, **15/76 uses (20%)**; `qwen3` (18) rejected on our own gate models | [#605](https://github.com/mudler/vllm.cpp/issues/605) | | `--enable-auto-tool-choice`, `--trust-remote-code` | no-ops for us, yet **abort startup** on 89 and 82 recipes | [#606](https://github.com/mudler/vllm.cpp/issues/606) | -| `--language-model-only` | absent; 43 recipes use it. It zeroes every modality limit — the freed encoder VRAM **and** the refusal of every mm request both follow from that | [#607](https://github.com/mudler/vllm.cpp/issues/607) | +| `--language-model-only`, `--limit-mm-per-prompt` | **ACCEPTED + ENFORCED 2026-08-14** (#607 L2): the 43 recipes that pass the flag reach model load, and it zeroes every modality limit so mm requests are REFUSED with upstream's message and HTTP 400. The freed encoder VRAM is the half still owed (wave L3), and the flag must not be described as freeing memory until that lands and is measured | [#607](https://github.com/mudler/vllm.cpp/issues/607) | | `--kv-cache-dtype` | not a serve flag; residual on the `KV-FP8` row | — | | `--speculative-config` | MTP + DFlash land; `eagle`/`eagle3` (7 uses) do not | — | | TP / EP / multi-node (`--tensor-parallel-size`, `--enable-expert-parallel`, `--mm-encoder-tp-mode`) | absent by scope, not by defect — single-box engine | see the TP W-plan above | diff --git a/.agents/specs/mm-serving.md b/.agents/specs/mm-serving.md index ab7bdc215..7b8c44936 100644 --- a/.agents/specs/mm-serving.md +++ b/.agents/specs/mm-serving.md @@ -140,6 +140,23 @@ chat request content-part array [MM-SERVE-PARSE — CPU, THIS BRICK] #607's L1 has now ported that refusal (`BaseProcessingInfo::ValidateNumItems`); wiring this call site to it is L2, and this is the reason L2 exists. + **CLOSED 2026-08-14 by #607 wave L2** (`row/mm-limits-l2`), with one correction + to the description above that the RED run produced and that is recorded rather + than quietly dropped. Through the PRODUCTION seam the pre-L2 behaviour was not + a truncated 200: `MakeQwen3VLImageChatFn` injects one placeholder marker per + image part while routing only the first image, so `ExpandImagePlaceholders` + raised *"more image placeholders than grids"* and the client got an **HTTP 500** + carrying an internal message. The silent truncation is real for the + validate-then-build seam shape (pinned as its own leg in + `test_api_server.cpp`), so the issue's diagnosis — first image, `break`, no + refusal — stands; only its stated consequence was understated. Both cases are + now the same answer: `ValidateChatMmLimits` runs first and a request over the + seam's declared ceiling (`Qwen3VLChatSupportedMmLimits() == {"image": 1}`, + the `min()` fold operand the issue asked for) is refused with + **HTTP 400 "At most 1 image(s) may be provided in one prompt."** Video and + audio parts are refused the same way rather than dropped, because a modality + absent from the seam's supported limits reads as limit 0 (`context.py:414-415`). + ## Brick 2 (`MM-SERVE-ENGINE`) — landed - `include/vllm/v1/engine/input_processor.{h,cpp}` — `process_inputs_mm`: the diff --git a/.agents/specs/multimodal-track.md b/.agents/specs/multimodal-track.md index 2558f6491..ed70d7026 100644 --- a/.agents/specs/multimodal-track.md +++ b/.agents/specs/multimodal-track.md @@ -313,6 +313,93 @@ comparing the two arms must set the flag on both sides or state that it did not. - **L2** — `--limit-mm-per-prompt` and `--language-model-only` serve flags plus the C-ABI field, over L1. This is the point at which the 43 recipes stop aborting. + + **LANDED 2026-08-14 (#607, #686, `row/mm-limits-l2`).** Three surfaces, and + the third is the one that makes the other two mean anything: + + 1. **The flags** (`server_main.cpp`). `--language-model-only` / + `--no-language-model-only`, mirroring `arg_utils.py:555,1276,1691` — both + spellings, because `_compute_kwargs` gives a bool field + `argparse.BooleanOptionalAction` (`arg_utils.py:346-348`) and a recipe that + turns the flag off explicitly must not die on an unknown argument. + `--limit-mm-per-prompt ''`, mirroring `arg_utils.py:556,1279,1692`, + whose dict type resolves to `type=parse_type(json.loads)` + (`arg_utils.py:375-377`) — so the value is a JSON object. + `ParseLimitMmPerPromptJson` (`src/vllm/config/multimodal.cpp`) ports + `_validate_limit_per_prompt` (`multimodal.py:212-236`) and the DummyOptions + dataclasses behind it (`:17-43`): the legacy count-only, the configurable + `{"count": N, …}` and the mixed spellings all parse; a non-object document, + a negative count (`count: int = Field(999, ge=0)`, `:20`), an unknown + per-modality option (`extra="forbid"`) or a non-positive one + (`Field(None, gt=0)`) is REFUSED before the model load rather than + defaulted, because a mistyped limit that silently became 999 is a limit + that is not there. The profiling options are validated and then dropped + (only `.count` feeds `get_limit_per_prompt`, `:335`) and the drop is + ANNOUNCED per key. **NAMED RESIDUAL:** upstream's dotted spelling + (`--limit-mm-per-prompt.image 2`) is a `FlexibleArgumentParser` feature + (`argparse_utils.py:389-425`) that applies equally to `--kv-transfer-config` + and `--speculative-config`, which this server also takes as JSON only. + Adding it for one flag would be the bespoke path; it belongs to a parser + brick covering all three. + 2. **The C-ABI field** — `vllm_model_params.language_model_only` + + `.limit_mm_per_prompt`, `VLLM_ABI_VERSION` 18 → **19**, appended so a + zero-initialised v18 struct is byte-identical. `limit_mm_per_prompt` is the + same JSON object the flag takes, following the v9 `kv_transfer_config` + precedent that a dict-valued vLLM flag crosses the ABI as its own JSON + rather than as a fixed struct of modalities the ABI would then owe forever. + Malformed input fails `vllm_engine_load` with `VLLM_ERR_INVALID_ARGUMENT`. + Both land on `EngineParams::multimodal` → `LoadedEngine::mm_config()`, so + the server flags and the ABI resolve ONE config object per engine and + cannot drift. + 3. **The call site — this is what L2 is for.** `chat_mm.cpp` now calls + `ValidateChatMmLimits` (the port of the `chat_utils.py:648-662` tracker) + as step 0 of `MakeQwen3VLImageChatFn`, over a `BaseProcessingInfo` folding + the engine's config with the seam's own declared ceiling, + `Qwen3VLChatSupportedMmLimits() == {"image": 1}`. That declaration is the + answer to the open question #686 left ("the model's own supported limit — + today nothing declares one"): this seam locates a single image part and + handles no video or audio, so its honest ceiling is one image and every + other modality is absent, which `context.py:414-415` reads as limit 0. A + user limit can only LOWER it (the fold is a `min`), so + `--limit-mm-per-prompt image=99` still refuses the second image, and the + refusal then carries no `--limit-mm-per-prompt` hint because raising the + user's limit would not help. + + **#686 is CLOSED by this.** A three-image request is answered + `400 BadRequestError "At most 1 image(s) may be provided in one prompt."` + instead of being served with its first image. One correction to the issue's + own text, found by the RED run and recorded rather than quietly fixed: against + the production seam the pre-L2 behaviour was not the truncated 200 the issue + describes but an **HTTP 500** — `MakeQwen3VLImageChatFn` injects one + placeholder marker per image part while routing only the first image, so + `ExpandImagePlaceholders` raised "more image placeholders than grids" and the + client saw a server fault carrying an internal message. The truncated 200 is + real for the validate-then-build shape and is pinned as its own leg. Both are + wrong the same way — neither is upstream's refusal — so the issue's diagnosis + stands and only its consequence was understated. + + **Gates** (aarch64, the `build-test-cpu-arm64` lane; `-DVLLM_CPP_CUDA=OFF`): + `test_chat_mm` 11/11 (126 assertions), `test_serve_mm_limits` 10/10 (101), + `test_openai_api_server` 56/56 (638). RED before the change: `test_chat_mm` + 2 cases failing (`CHECK_THROWS_AS ... threw a DIFFERENT exception: "Expand + ImagePlaceholders: more image placeholders than grids"` and `FATAL ERROR: + expected --language-model-only to refuse an image request`), and the HTTP legs + failing `CHECK(500 == 400)`, `CHECK("InternalServerError" == BadRequestError)` + and `CHECK(200 == 400)` — the last of those being the truncated 200 itself. + + **What L2 does NOT do, so L3 knows what it inherits.** No memory claim is made + or implied: nothing gates tower CONSTRUCTION on the limits, so + `--language-model-only` today refuses multimodal requests and frees nothing. + That is L3 and it is owed with a MEASURED RSS reduction. Separately owed and + named rather than assumed: the SECOND call site, + `process_inputs_mm` (`input_processor.cpp:321,352`, upstream's + `context.py:461`), is still unwired. It needs the per-model + `get_supported_mm_limits()` hook that L1 already recorded as absent (the models + that would implement it are the M2 towers); wiring it before that hook exists + would mean inventing a supported-limits source inside the engine, which is the + bespoke path the mirror rule forbids. The chat call site is wired because it + HAS a concrete source — the seam's own implemented arm. Every path the OpenAI + server can reach today goes through the wired one. - **L3** — the tower skip: construct-without-initialising when every limit is 0, gated on **measured** RSS reduction against a multimodal checkpoint, plus token-exactness of the text path with and without the flag. diff --git a/.agents/specs/serve-recipe-args.md b/.agents/specs/serve-recipe-args.md index 44086d666..ae6f60cb1 100644 --- a/.agents/specs/serve-recipe-args.md +++ b/.agents/specs/serve-recipe-args.md @@ -21,7 +21,7 @@ Explicitly NOT in scope, each for its own reason: | Excluded | Why | |---|---| -| `--language-model-only` | a real capability gap, tracked as [#607](https://github.com/mudler/vllm.cpp/issues/607) — accepting it would hide missing behaviour | +| `--language-model-only` | a real capability gap, tracked as [#607](https://github.com/mudler/vllm.cpp/issues/607) — accepting it would hide missing behaviour. **Resolved the right way 2026-08-14**: #607 wave L2 IMPLEMENTED it (and `--limit-mm-per-prompt`) rather than adding it here, so it is a real flag with real enforcement and `kAcceptedInertArgs` still holds only genuine no-ops | | the `--tool-call-parser` default (`hermes` vs upstream `None`) | pre-existing divergence, worth its own issue; changing it here would mix a behaviour change into a parsing change | | `--tensor-parallel-size`, the EP flags, `--mm-encoder-tp-mode` | inert only because the CAPABILITY is missing; they must keep aborting | | any change to a forward pass, kernel, or model path | none is reached — this is argument parsing | diff --git a/CMakeLists.txt b/CMakeLists.txt index 2683039a1..5ada76dae 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -702,6 +702,7 @@ add_library(vllm STATIC src/vllm/config/scheduler.cpp src/vllm/config/device.cpp src/vllm/config/kv_transfer.cpp + src/vllm/config/multimodal.cpp src/vllm/config/speculative.cpp src/vllm/outputs.cpp src/vllm/transformers_utils/hf_config.cpp diff --git a/README.md b/README.md index eb8997e2b..a01338ac0 100644 --- a/README.md +++ b/README.md @@ -186,7 +186,7 @@ configs, token-for-token the same output. Switching to it should be boring. Ever you get on top, most of it borrowed from whichever engine does it best: - **One 66 MiB binary instead of a 9.1 GiB install.** A flat, exception-free, llama.cpp-style C ABI - ([`include/vllm.h`](include/vllm.h), ABI v18, 36 functions) for C, C++, Go, or Rust. No Python + ([`include/vllm.h`](include/vllm.h), ABI v19, 36 functions) for C, C++, Go, or Rust. No Python interpreter in the process. - **GGUF as a first-class citizen.** Load the same quantized files llama.cpp uses, and on CPU **compute directly on the compressed blocks** (Q4_0/Q8_0/Q3_K/Q4_K/Q5_K/Q6_K) with no BF16 @@ -385,7 +385,7 @@ behind a model gallery, multi-model serving, the full OpenAI API surface, auth, ## Use it as a library (C API) Link `libvllm` and include [`include/vllm.h`](include/vllm.h): a flat, exception-free, -llama.cpp-style C ABI (`VLLM_ABI_VERSION 18`, 36 exported functions) suitable for `dlopen` / FFI. +llama.cpp-style C ABI (`VLLM_ABI_VERSION 19`, 36 exported functions) suitable for `dlopen` / FFI. ```c vllm_model_params mp = vllm_model_params_default(); diff --git a/docs/FEATURES.md b/docs/FEATURES.md index af667dd24..16c1899e3 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -330,7 +330,7 @@ CPU elementwise GEMM (f32/f16/bf16) runs AVX2 and AVX-512 tiers on x86 where the | Muse Glimmer 30B (Meta) | Text gated at **reduced depth 4/52** only; vision wired but never reference-checked | [spec](../.agents/specs/muse-glimmer.md) / [#268](https://github.com/mudler/vllm.cpp/issues/268). Full depth, multi-step decode, image/video, server path and parser scoping open. vLLM speed OPEN GAP; llama.cpp bar #333 | | Multi-GPU execution | Hardware-blocked | TP proven equal to tp=1 on CPU; no 2-GPU box to run it | | LoRA end to end | CPU brick landed | Unwired standalone; not usable through the server | -| Multimodal over HTTP | Image request path wired; forward + codec pending | `ROAD-V1-MM` W1-W3 landed (`server_main.cpp:826`). Open: no mm-forward consuming `Request.mm_features`; no image codec vendored (raw RGB only); video/audio/multi-image not started | +| Multimodal over HTTP | Image request path wired; forward + codec pending | `ROAD-V1-MM` W1-W3 landed. Open: no mm-forward on `Request.mm_features`; no image codec. Video/audio/multi-image now **refuse** with HTTP 400 rather than drop ([#686](https://github.com/mudler/vllm.cpp/issues/686)) | | Reranking / classify models | Engine side only | Embeddings are LIVE (`LlamaModel`, `vllm_embed`, `/v1/embeddings`); the classify/score heads are landed ops with no registered arch | | ROCm | W0 community-verified on 5 gfx archs; classic-dense and GDN-hybrid e2e run all-native; correctness gaps remain | 44 registered ops including the GDN state/conv/postconv/recurrence set; APU managed-allocation branch remains unverified. [ROCM.md](ROCM.md) | | XPU, TPU | Not started | CUDA, CPU, Metal and Vulkan are the built backends | diff --git a/docs/STATUS.md b/docs/STATUS.md index 3038b35f9..e67cf2cfe 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -1780,8 +1780,16 @@ runtime-verified yet. validation still fires — `--enable-auto-tool-choice` with `--tool-call-parser none` is refused, mirroring `vllm/entrypoints/openai/cli_args.py:395`. Not yet reviewed or gate-rerun by - the operator; `--language-model-only` (#607) is a real capability gap and is - deliberately NOT in this list. + the operator; `--language-model-only` was excluded from this list because it + is a real capability, not a no-op — and as of 2026-08-14 (#607 wave L2) it is + **implemented** rather than accepted-and-inert, alongside + `--limit-mm-per-prompt`. Both set `vllm::MultiModalConfig` and are ENFORCED: a + server started with `--language-model-only` answers a multimodal request with + HTTP 400 `At most 0 image(s) may be provided in one prompt.`, which is what + upstream's flag does. The 43 recipes that pass it now reach model load. It + does **not** free memory yet — nothing gates vision-tower construction on the + limits (wave L3, owed with a measured RSS reduction), so the flag must not be + described as a VRAM knob until that lands. - **Surface coverage (ONE SURFACE, `ARCH-ONE-SURFACE`, `.agents/specs/surface-coverage-2026-08-07.md`).** 21/30 text archs on-framework; the recurring defect (a capability in a per-model CLI) is in seven diff --git a/docs/USAGE.md b/docs/USAGE.md index d16855c12..11cb81e59 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -1155,6 +1155,8 @@ a stop token early. | `--reasoning-parser ` | `none` | Reasoning parser (`think_auto`, `deepseek_r1`, `deepseek_v3`, `holo2`, `mistral`, `minimax_m2`, `minimax_m2_append_think`, `step3`, `olmo3`, `muse_glimmer`, `qwen3`, `mimo`). `auto` detects, `none` disables. `qwen3` and its `mimo` alias are the engine-backed adapter (one upstream class, two registry names): thinking is ON, so a marker-less stream is reasoning and a `` ends reasoning with no ``. `auto` never selects it — a generic `` template resolves to `think_auto`, which is the right default for hybrid-thinking models that may answer with no think block at all | | `--kv-transfer-config ''` | (unset) | External KV connector, same JSON as vLLM's flag. See [docs/KV-OFFLOAD.md](KV-OFFLOAD.md) | | `--speculative-config ''` | (unset) | Speculative decoding (`mtp`, `dflash`, `ngram`), same JSON as vLLM's flag. `dspark` speculates on the Qwen3.6 gate models (native + Speculators drafts), token-identically to speculative-off, but is not gated on speed (currently ~2% behind at c1). A GGUF target, or a target with no aux multi-tap, is refused by name (`SPEC-DSPARK`). Its sequential Markov sampling runs on device by default; `VT_DSPARK_DEVICE_SAMPLE=0` restores the host loop (token-identical, cost only). The speculative verify runs from a captured CUDA graph, worth +12.2%/+3.5% on the 35B cells; `VT_SPEC_DECODE_GRAPH=0` restores the eager verify (also token-identical). See [docs/SPECULATIVE-DECODING.md](SPECULATIVE-DECODING.md) | +| `--language-model-only` / `--no-language-model-only` | off | Disable all multimodal input by setting **every** modality limit to 0, mirroring vLLM's flag of the same name. It is not a "skip the encoder" switch: the server then **refuses** a multimodal request with `400 At most 0 image(s) may be provided in one prompt. Set --limit-mm-per-prompt to increase this limit.` It does **not** free VRAM yet — nothing gates tower construction on it ([#607](https://github.com/mudler/vllm.cpp/issues/607) wave L3) | +| `--limit-mm-per-prompt ''` | (unset ⇒ 999 per modality) | Maximum multimodal input items per prompt, per modality, as the same JSON object vLLM's flag takes: `'{"image": 2, "video": 0}'`, or with profiling options `'{"video": {"count": 1, "num_frames": 32}}'` (the options are validated and ignored — they size dummy inputs for memory profiling, which this engine does not do). A limit can only **lower** what the model/seam supports, never raise it. Malformed JSON, a negative count or an unknown per-modality option is refused at startup rather than defaulted. Upstream's dotted spelling (`--limit-mm-per-prompt.image 2`) is not accepted here, as for `--kv-transfer-config` and `--speculative-config` | | `--enable-log-requests` / `--disable-log-requests` | on | Log each incoming request. Mirrors vLLM's flag of the same name | | `--enable-log-outputs` | off | Also log the generated output, not just the request | | `--max-log-len N` | `256` | Truncate logged prompts and outputs to N characters | @@ -1588,7 +1590,7 @@ vllm-server --model /path/to/text-model \ ## Consuming it as a library (C ABI) Link `libvllm` (static or shared) and include [`include/vllm.h`](../include/vllm.h). -It exposes a flat, exception-free, llama.cpp-style C ABI (`VLLM_ABI_VERSION 18`, +It exposes a flat, exception-free, llama.cpp-style C ABI (`VLLM_ABI_VERSION 19`, 36 exported functions) suitable for `dlopen` / FFI / LocalAI integration. ```c @@ -1830,25 +1832,45 @@ Accepted part types (`src/vllm/entrypoints/openai/chat_mm.cpp`): | `video_url` | video | | `input_audio` / `audio_url` | audio | -### Per-prompt input limits — the mechanism exists, the flags do not yet +### Per-prompt input limits vLLM caps how many items of each modality one prompt may carry (`--limit-mm-per-prompt`), and `--language-model-only` is sugar for setting every -one of those limits to 0, which makes the server refuse multimodal requests -outright. **Neither flag is accepted yet** — `vllm-server` still exits on both, -and there is no config key or C ABI field for them. - -What landed (#607, wave L1) is the mechanism underneath, as library-internal -headers only: `vllm::MultiModalConfig::GetLimitPerPrompt` -([`config/multimodal.h`](../include/vllm/config/multimodal.h)) resolving -upstream's precedence, and the refusal it carries -([`multimodal/processing/context.h`](../include/vllm/multimodal/processing/context.h)), -which raises `vllm::v1::InputValidationError` — the same type the API server -answers with HTTP 400. Nothing constructs that config on a live request, so -**today no request is limited or refused on item count**, and a chat request -carrying several images still has all but the first silently dropped by -`chat_mm.cpp`. Wave L2 adds the two flags, the C ABI field, and the call-site -wiring that makes the limits take effect. +one of those limits to 0. Both flags are accepted (#607, waves L1+L2), both are +enforced, and both are also C ABI fields +(`vllm_model_params.language_model_only` / `.limit_mm_per_prompt`, ABI v19). + +The limits are the mechanism and the flag is the sugar, so it is worth stating +what the flag actually does: it does not "skip the encoder", it makes the server +**refuse** multimodal requests. + +```console +$ curl -s localhost:8000/v1/chat/completions -d '{... three image_url parts ...}' +{"error":{"type":"BadRequestError", + "message":"At most 1 image(s) may be provided in one prompt."}} # HTTP 400 + +$ vllm-server --model … --language-model-only # then any image request: +{"error":{"type":"BadRequestError", + "message":"At most 0 image(s) may be provided in one prompt. Set `--limit-mm-per-prompt` to increase this limit."}} +``` + +Two things follow from how the limit is computed +(`min(user limit, what the model/seam supports)`): + +- A user limit can only **lower** the ceiling. `--limit-mm-per-prompt + '{"image": 99}'` on this server still refuses a second image, because the + OpenAI chat seam handles exactly one image today (video and audio parts are + not routed at all, so their limit is 0 and they are refused by name rather + than dropped — this is what closed + [#686](https://github.com/mudler/vllm.cpp/issues/686)). +- The `Set --limit-mm-per-prompt to increase this limit.` hint appears only when + raising the limit would actually help — that is, when the seam could take the + items and the configuration is what refused them. + +**Not yet:** `--language-model-only` frees no memory. Nothing gates vision-tower +construction on the limits, so the flag today changes what the server accepts, +not what it allocates ([#607](https://github.com/mudler/vllm.cpp/issues/607) +wave L3, owed with a measured RSS reduction). ## MiniMax-H3 browser console (`vllm-video-studio`) diff --git a/include/vllm.h b/include/vllm.h index f372c9fd5..540cd6992 100644 --- a/include/vllm.h +++ b/include/vllm.h @@ -167,7 +167,34 @@ extern "C" { * stays NULL (detect), n_extras stays 0, `partition` keeps its exact v12 * meaning, and every v12 status/message contract is unchanged (the * text-checkpoint refusal still names vllm_engine_load). */ -#define VLLM_ABI_VERSION 18 +/* v19 — MULTIMODAL INPUT LIMITS on vllm_model_params (ENG-MM-INPUT-PIPELINE + * wave L2, issue #607). Two APPENDED fields mirroring vLLM's MultiModalConfig + * (vllm/config/multimodal.py:78,81) as its own two serve flags expose it + * (vllm/engine/arg_utils.py:555-556,1276-1279,1691-1692): + * - vllm_model_params.language_model_only — the flag whose name misleads and + * whose docstring does not: it "disables all multimodal inputs by setting + * all modality limits to 0" (multimodal.py:78-80). It is sugar; the limits + * are the mechanism. + * - vllm_model_params.limit_mm_per_prompt — the per-modality input-count + * limits, as the SAME JSON object the flag takes ('{"image": 2, + * "video": 0}', or the option form '{"video": {"count": 1}}'), following + * the v9 precedent that a dict-valued vLLM flag crosses this ABI as its own + * JSON rather than as a fixed struct of modalities the ABI would then owe + * forever. A malformed document, an unknown per-modality option, or a + * negative count fails vllm_engine_load with VLLM_ERR_INVALID_ARGUMENT + * rather than defaulting — mirroring the pydantic validation upstream does + * at parse time (multimodal.py:17-43,212-236). + * Both are ENFORCED, not recorded: a modality's limit is what + * BaseProcessingInfo::ValidateNumItems refuses against, so an engine loaded + * with language_model_only answers a multimodal request with + * "At most 0 image(s) may be provided in one prompt." rather than serving it. + * The memory win upstream also gets from zero limits (skipping the vision tower + * weights, interfaces.py:293) is NOT in this version — it is wave L3, and until + * it lands and is MEASURED this field must not be described as freeing VRAM. + * Appended at the END of vllm_model_params, so a zero-initialized v18 struct is + * byte-identical: language_model_only 0 (off) and limit_mm_per_prompt NULL (no + * limits configured => the 999-per-modality default, multimodal.py:331-333). */ +#define VLLM_ABI_VERSION 19 /* ── Export macro ───────────────────────────────────────────────────────────── * Marks the symbols that make up the stable ABI. Default visibility now; Task 3 @@ -359,6 +386,38 @@ typedef struct vllm_model_params { * (cache.py:182,189). 0 => unset. A budget smaller than a single KV block * fails vllm_engine_load with VLLM_ERR_INVALID_ARGUMENT. */ int64_t kv_cache_memory_bytes; + /* ── Multimodal input limits (ABI v19) ──────────────────────────────────── + * The mirror of vLLM's --language-model-only / --limit-mm-per-prompt + * (arg_utils.py:555-556,1276-1279,1691-1692) over MultiModalConfig + * (multimodal.py:78,81). These are ONE mechanism, not two: the flag is + * defined as "disables all multimodal inputs by setting all modality limits + * to 0" (:78-80), so it is checked BEFORE the map and an explicit non-zero + * entry does not survive it (get_limit_per_prompt, :321-336). + * + * language_model_only: 0 => off (the zero value, byte-identical to pre-v19); + * nonzero => every modality limit becomes 0, which makes the engine REFUSE + * every multimodal request with "At most 0 (s) may be provided in + * one prompt." That refusal is the flag's main observable effect and it is + * live at v19; the tower-skip memory win it also produces upstream is not + * (wave L3). */ + int32_t language_model_only; + /* limit_mm_per_prompt: the per-modality maximum input-item count, as the same + * JSON object the flag takes. NULL/empty (the zero value) => no limit + * configured => 999 per modality (:331-333), NOT zero — an empty map is "no + * limits", not "nothing allowed". Accepted spellings, all upstream's own + * (:87-96,212-236): + * {"image": 16, "video": 2} (count only) + * {"video": {"count": 1, "num_frames": 32}} (with options) + * {"image": 16, "video": {"count": 1}} (mixed) + * The option keys are validated exactly as upstream's per-modality + * dataclasses do (video: num_frames/width/height, image: width/height, audio: + * length; each an integer > 0; anything else is refused, `extra="forbid"`) + * and then DROPPED: they size dummy inputs for memory profiling, which this + * engine does not do, and only `count` feeds the limit (:335). + * Invalid JSON, a non-object document, a negative count, an unknown option, + * or a non-integer value fails vllm_engine_load with + * VLLM_ERR_INVALID_ARGUMENT. Borrowed for the call only. */ + const char* limit_mm_per_prompt; } vllm_model_params; /* ── Custom logits processor (ABI v8) ───────────────────────────────────────── diff --git a/include/vllm/config/multimodal.h b/include/vllm/config/multimodal.h index cc4b76979..435625dbf 100644 --- a/include/vllm/config/multimodal.h +++ b/include/vllm/config/multimodal.h @@ -39,6 +39,7 @@ #include #include +#include namespace vllm { @@ -87,6 +88,48 @@ struct MultiModalConfig { } }; +// Parse the value of `--limit-mm-per-prompt` (arg_utils.py:556,1279) / the C-ABI +// `vllm_model_params.limit_mm_per_prompt` into `limit_per_prompt`. +// +// Ported from `MultiModalConfig._validate_limit_per_prompt` +// (multimodal.py:212-236) together with the DummyOptions dataclasses it feeds +// (:17-43). Upstream reaches this through argparse `type=parse_type(json.loads)` +// (_compute_kwargs, arg_utils.py:375-377), so the flag's value is a JSON OBJECT +// and every spelling below is the object's, not the flag's: +// +// * the LEGACY format, count only — {"image": 16, "video": 2} (:87-88,220-222, +// which rewrites a bare int to {"count": }); +// * the CONFIGURABLE format — {"video": {"count": 1, "num_frames": 32, +// "width": 512, "height": 512}} (:90-92,224-232); +// * the two MIXED in one object (:94-96). +// +// Refusals, all of them upstream's own, none of them invented here: +// * a non-object document, or a value that is neither an int nor an object — +// upstream's json.loads/pydantic pair rejects both; +// * `count` absent-and-not-an-int, or NEGATIVE — `count: int = Field(999, +// ge=0)` (:20); +// * an option key the modality does not define, or an option <= 0 — every +// DummyOptions dataclass is `extra="forbid"` with `Field(None, gt=0)` +// (:23-43). video takes num_frames/width/height, image width/height, audio +// length, and any OTHER modality takes `count` alone (BaseDummyOptions, +// :233-234). +// Refusing rather than defaulting is the point: a typo'd limit that silently +// became 999 is a limit that is not there, which is exactly the failure this +// wave exists to remove. +// +// DEVIATION, recorded: the option keys are parsed and VALIDATED, then DROPPED. +// They size DUMMY multimodal inputs during memory profiling, a surface we do not +// have (see the header comment above on `limit_per_prompt`'s mapped type), and +// only `.count` participates in get_limit_per_prompt (:335). `ignored_options`, +// when non-null, collects "." for every option dropped, so the +// caller can ANNOUNCE the drop rather than let a user infer that `num_frames` +// took effect. +// +// Throws std::invalid_argument, carrying the offending key/value, on anything +// above. +std::map ParseLimitMmPerPromptJson( + const std::string& json, std::vector* ignored_options); + } // namespace vllm #endif // VLLM_CONFIG_MULTIMODAL_H_ diff --git a/include/vllm/entrypoints/model_loader.h b/include/vllm/entrypoints/model_loader.h index 8aa372a12..6a2b2815d 100644 --- a/include/vllm/entrypoints/model_loader.h +++ b/include/vllm/entrypoints/model_loader.h @@ -17,6 +17,7 @@ #include "vllm/config/device.h" #include "vllm/config/kv_transfer.h" +#include "vllm/config/multimodal.h" #include "vllm/config/scheduler.h" #include "vllm/config/speculative.h" #include "vllm/model_executor/models/model_registry.h" @@ -168,6 +169,19 @@ struct EngineParams { // vllm_model_params.device (ABI v14: 0=auto, 1=cpu, 2=cuda) and on the // server as --device. vllm::Device device = vllm::Device::kAuto; + + // ENG-MM-INPUT-PIPELINE wave L2 (#607): the per-modality multimodal input + // limits, mirroring vLLM threading MultiModalConfig onto the model config + // (arg_utils.py:1691-1692 -> ModelConfig -> multimodal_config). Exposed on the + // server as --language-model-only / --limit-mm-per-prompt and on the C ABI as + // vllm_model_params.language_model_only / .limit_mm_per_prompt (ABI v19). + // + // The DEFAULT is the pre-L2 behaviour byte for byte: an empty map with the + // flag off resolves to 999 per modality (multimodal.py:331-333), which no + // real request reaches. It is deliberately a VALUE and not an optional — a + // "no config" state distinct from "the default config" would be a second + // spelling of the same thing, and upstream has one. + vllm::MultiModalConfig multimodal; }; // The shared queue-selection seam used by every LoadedEngine construction @@ -271,6 +285,13 @@ class LoadedEngine { std::optional named_platform_type); vllm::v1::LLMEngine& engine() { return engine_; } + // The multimodal input limits this engine was loaded with (#607 L2). ONE + // config object per engine, whether it was set by the server's + // --language-model-only / --limit-mm-per-prompt or by the C ABI's + // vllm_model_params, so the two entry points cannot resolve a limit + // differently. It outlives every consumer that borrows it (declared before + // input_processor_), which is what lets the OpenAI chat seam hold a reference. + const vllm::MultiModalConfig& mm_config() const { return mm_config_; } // ARCH-ONE-SURFACE ROW 6: whether the loaded model registration declares the // POOLING task class (is_pooling_model). The entrypoints dispatch BY TASK on // this — text-generation refuses on a pooling engine (naming vllm_embed / @@ -415,6 +436,11 @@ class LoadedEngine { // the ctor body once runner_ geometry is known; see model_loader.cpp. std::unique_ptr kv_connector_; tok::Tokenizer tokenizer_; + // #607 L2: the engine's multimodal input limits, copied from EngineParams. + // Declared here — before input_processor_ and before anything that can borrow + // it — because the OpenAI chat seam holds a reference for the process + // lifetime. A default-constructed one is the pre-L2 behaviour exactly. + vllm::MultiModalConfig mm_config_; // kv_cfg_ is declared BEFORE max_model_len_: the serving length is resolved // AGAINST the KV pool (ResolveMaxModelLen auto-fits it down to what the pool // holds, or refuses an explicit --max-model-len the pool cannot serve), and diff --git a/include/vllm/entrypoints/openai/chat_mm.h b/include/vllm/entrypoints/openai/chat_mm.h index cf1a1e795..75d60b390 100644 --- a/include/vllm/entrypoints/openai/chat_mm.h +++ b/include/vllm/entrypoints/openai/chat_mm.h @@ -25,6 +25,7 @@ #include #include +#include #include #include #include @@ -32,6 +33,7 @@ #include "vllm/entrypoints/openai/protocol.h" #include "vllm/multimodal/audio_processor.h" #include "vllm/multimodal/inputs.h" +#include "vllm/multimodal/processing/context.h" #include "vllm/multimodal/qwen3vl_processor.h" namespace vllm { @@ -137,6 +139,58 @@ std::vector CollectChatPlaceholders(const ChatMessage& message); // content unchanged (byte-identical text path). std::string BuildMarkerInjectedContent(const ChatMessage& message); +// ── The per-item LIMIT check on the chat path (#607 wave L2, #686) ───────── +// +// Ported from: vllm/entrypoints/chat_utils.py:630-662 — the tracker that counts +// every multimodal content part it parses and validates the RUNNING count +// against the configured limit before the request reaches the engine. +// `BaseProcessingInfo::ValidateTrackedChatItem` (#607 L1) is the check itself; +// what follows is the WALK that reaches it, which is the half L1 deliberately +// left out because nothing constructed a config on a live request yet. + +// The tracked mm modality for one content part — the modality +// `validate_num_items` is asked about, or nullopt for a part that is not +// multimodal input at all. Mirrors MM_PARSER_MAP (chat_utils.py:1478) reduced to +// the part types this server's protocol.h parses: +// image_url -> "image" (chat_utils.py:1479) +// video_url -> "video" (chat_utils.py:1483) +// input_audio/audio_url -> "audio" (chat_utils.py:1481-1482) +// *_embeds -> passed THROUGH verbatim, because +// ValidateTrackedChatItem owns the suffix strip and +// the enable_mm_embeds escape (chat_utils.py:635,653-660) +// text (and anything else) -> nullopt +std::optional ChatPartModality(const ChatContentPart& part); + +// The WALK (chat_utils.py:648-662): every mm content part of every message, IN +// ORDER, maintaining the running per-modality count and validating each item as +// it is counted. The count is cumulative ACROSS messages, exactly as upstream's +// tracker is, so a limit cannot be evaded by splitting the items over two turns. +// +// Throws vllm::v1::InputValidationError — the type api_server.cpp:185,252 maps +// to HTTP 400 — carrying upstream's own message text. This is what turns the +// silent truncation of #686 into upstream's refusal: before it, chat_mm.cpp took +// the FIRST image part and dropped the rest without a word. +void ValidateChatMmLimits(const multimodal::BaseProcessingInfo& info, + const std::vector& messages); + +// The Qwen3-VL IMAGE chat seam's OWN supported limits — the `min()` fold's other +// operand (context.py:392-405), which #686 recorded as undeclared. +// +// Upstream's Qwen3-VL declares image and video UNLIMITED +// (get_supported_mm_limits, qwen3_vl.py) because its processor handles N of +// each. Ours handles exactly ONE image (MakeQwen3VLImageChatFn locates a single +// image part) and no video or audio at all, so the honest ceiling is +// {"image": 1} and every other modality is ABSENT — which context.py:414-415 +// reads as "not supported", limit 0. That is not a policy choice, it is this +// seam's implemented arm stated as a number, and AGENTS.md requires exactly +// that: an unimplemented arm is refused with a message naming the missing piece, +// never left to be discovered. A user limit can only LOWER it (the fold is a +// min), so `--limit-mm-per-prompt image=99` still refuses the second image. +// +// When the multi-image / video arms land they raise these numbers here, and +// nothing else changes. +std::map> Qwen3VLChatSupportedMmLimits(); + // ── The multimodal chat SEAM BODY (MM-SERVE-E2E) ─────────────────────────── // // A decoded RGB image: raw HWC uint8 (height*width*3) + dims. Turning the @@ -177,9 +231,18 @@ using ChatPromptRenderFn = std::function( const std::vector&)> MakeQwen3VLImageChatFn(const multimodal::Qwen3VLImageProcessor& proc, const vllm::tok::Tokenizer& tokenizer, - ChatPromptRenderFn prompt_fn, ImageCodecFn codec); + ChatPromptRenderFn prompt_fn, ImageCodecFn codec, + const multimodal::BaseProcessingInfo& info); } // namespace vllm::entrypoints::openai diff --git a/src/capi/vllm_c.cpp b/src/capi/vllm_c.cpp index 2d1b214a4..9d4d7a16f 100644 --- a/src/capi/vllm_c.cpp +++ b/src/capi/vllm_c.cpp @@ -30,6 +30,7 @@ #include "capi/chat_prompt.h" #include "capi/engine_handle.h" #include "vllm/config/kv_transfer.h" // ParseKVTransferConfigJson (ABI v9) +#include "vllm/config/multimodal.h" // ParseLimitMmPerPromptJson (ABI v19) #include "vllm/config/scheduler.h" // SchedulerPolicyFromString (ABI v9) #include "vllm/v1/kv_offload/kv_connector.h" // KVConnectorFactory (ABI v9) #include "vllm/entrypoints/chat_template.h" @@ -502,6 +503,8 @@ VLLM_API vllm_model_params vllm_model_params_default(void) { p.device = 0; // 0 => auto: the accelerator-first probe (ABI v14). p.gpu_memory_utilization = 0.92; // vLLM default fraction (ABI v16). p.kv_cache_memory_bytes = 0; // 0 => unset (ABI v16). + p.language_model_only = 0; // 0 => off; limits stay at 999 (ABI v19). + p.limit_mm_per_prompt = nullptr; // NULL => no limits configured (ABI v19). return p; } @@ -604,6 +607,19 @@ VLLM_API vllm_status vllm_engine_load(const vllm_model_params* params, } ep.kv_transfer_config = std::move(kv_cfg); } + // ABI v19: the multimodal input limits (#607 L2). Parsed HERE so a + // malformed document is a CALLER error reported before any model I/O, + // exactly as the server refuses --limit-mm-per-prompt before the load. + // The flag half is deliberately not a second knob: language_model_only + // zeroes every limit inside GetLimitPerPrompt, ahead of the map + // (multimodal.py:326-327), so setting both is well-defined and the flag + // wins — mirrored rather than reimplemented here. + ep.multimodal.language_model_only = params->language_model_only != 0; + if (params->limit_mm_per_prompt != nullptr && + params->limit_mm_per_prompt[0] != '\0') { + ep.multimodal.limit_per_prompt = vllm::ParseLimitMmPerPromptJson( + params->limit_mm_per_prompt, /*ignored_options=*/nullptr); + } } catch (const std::invalid_argument& e) { SetError(std::string("vllm_engine_load: ") + e.what()); return VLLM_ERR_INVALID_ARGUMENT; diff --git a/src/vllm/config/multimodal.cpp b/src/vllm/config/multimodal.cpp new file mode 100644 index 000000000..a720e1c5e --- /dev/null +++ b/src/vllm/config/multimodal.cpp @@ -0,0 +1,103 @@ +// Ported from: vllm/config/multimodal.py:17-43,212-236 @ 5559679229bc +// (see include/vllm/config/multimodal.h for scope and deviations). +#include "vllm/config/multimodal.h" + +#include +#include +#include + +#include + +namespace vllm { +namespace { + +// The option keys each DummyOptions subclass declares, beyond `count` +// (multimodal.py:23-43). `extra="forbid"` is what makes an unlisted key an +// error rather than a silently-kept extra, so the lookup below is exhaustive by +// construction: a modality with no entry here is BaseDummyOptions, which +// declares NOTHING but `count` (:233-234). +bool IsKnownOption(const std::string& modality, const std::string& key) { + if (modality == "video") { + return key == "num_frames" || key == "width" || key == "height"; + } + if (modality == "image") return key == "width" || key == "height"; + if (modality == "audio") return key == "length"; + return false; +} + +[[noreturn]] void Refuse(const std::string& detail) { + throw std::invalid_argument("limit_mm_per_prompt: " + detail); +} + +// multimodal.py:220-222 — the legacy spelling is an int, rewritten to +// {"count": } before the dataclass sees it. +int ParseCount(const std::string& modality, const nlohmann::json& value) { + if (!value.is_number_integer()) { + Refuse("\"" + modality + "\".count must be an integer, got " + value.dump()); + } + const int64_t count = value.get(); + // count: int = Field(999, ge=0) (multimodal.py:20). + if (count < 0) { + Refuse("\"" + modality + "\".count must be >= 0, got " + + std::to_string(count)); + } + return static_cast(count); +} + +} // namespace + +std::map ParseLimitMmPerPromptJson( + const std::string& json, std::vector* ignored_options) { + nlohmann::json doc; + try { + doc = nlohmann::json::parse(json); + } catch (const std::exception& e) { + Refuse(std::string("value is not valid JSON (") + e.what() + ")"); + } + if (!doc.is_object()) { + Refuse("value must be a JSON object mapping a modality to its limit, e.g. " + "'{\"image\": 2, \"video\": 0}'; got " + + doc.dump()); + } + + std::map limits; + for (const auto& [modality, value] : doc.items()) { + if (modality.empty()) Refuse("a modality name may not be empty"); + + // The LEGACY format: a bare count (multimodal.py:87-88,220-222). + if (!value.is_object()) { + limits[modality] = ParseCount(modality, value); + continue; + } + + // The CONFIGURABLE format: {"count": N, } + // (multimodal.py:90-92,224-232). `count` keeps its 999 default when the + // object omits it — the object exists to carry the OPTIONS, so omitting the + // count is not "no limit configured", it is the dataclass default. + int count = kDefaultLimitPerPrompt; + for (const auto& [key, option] : value.items()) { + if (key == "count") { + count = ParseCount(modality, option); + continue; + } + if (!IsKnownOption(modality, key)) { + // extra="forbid" (multimodal.py:23,32,39). Naming the key is the whole + // value of the refusal: a dropped `num_frame` typo is invisible. + Refuse("\"" + modality + "\" has no option \"" + key + "\""); + } + // Field(None, gt=0) on every option (multimodal.py:26-28,35-36,42). + if (!option.is_number_integer() || option.get() <= 0) { + Refuse("\"" + modality + "\".\"" + key + "\" must be an integer > 0, " + "got " + option.dump()); + } + // Validated, then dropped — see the header's recorded deviation. + if (ignored_options != nullptr) { + ignored_options->push_back(modality + "." + key); + } + } + limits[modality] = count; + } + return limits; +} + +} // namespace vllm diff --git a/src/vllm/entrypoints/model_loader.cpp b/src/vllm/entrypoints/model_loader.cpp index d99c1d770..a21f20e29 100644 --- a/src/vllm/entrypoints/model_loader.cpp +++ b/src/vllm/entrypoints/model_loader.cpp @@ -972,6 +972,10 @@ LoadedEngine::LoadedEngine(HfConfig config, dflash_draft_(std::move(dflash_draft)), model_(std::move(model)), tokenizer_(std::move(tokenizer)), + // #607 L2: carry the multimodal input limits onto the engine, so the ONE + // config object every consumer asks (GetLimitPerPrompt) is the one the + // server flags / the C ABI set. Default-constructed == the pre-L2 999. + mm_config_(params.multimodal), // ROAD-V1-MEM M1: resolve the block count from the sizing knobs // (num_blocks override > kv_cache_memory_bytes > util fallback) against the // model's own per-block byte geometry. FIRST, because max_model_len_ is diff --git a/src/vllm/entrypoints/openai/chat_mm.cpp b/src/vllm/entrypoints/openai/chat_mm.cpp index c6f47285b..9b1210bb9 100644 --- a/src/vllm/entrypoints/openai/chat_mm.cpp +++ b/src/vllm/entrypoints/openai/chat_mm.cpp @@ -4,6 +4,7 @@ #include "vllm/entrypoints/openai/chat_mm.h" #include +#include #include #include #include @@ -246,16 +247,72 @@ std::string BuildMarkerInjectedContent(const ChatMessage& message) { return out; } +// chat_utils.py:1478-1483 (MM_PARSER_MAP), reduced to the part types protocol.h +// parses. See the header for why `*_embeds` passes straight through. +std::optional ChatPartModality(const ChatContentPart& part) { + if (part.type == "image_url") return std::string("image"); + if (part.type == "video_url") return std::string("video"); + if (part.type == "input_audio" || part.type == "audio_url") { + return std::string("audio"); + } + // "text" is not multimodal input; anything else that names an embeds kind is + // handed to ValidateTrackedChatItem verbatim so it owns the suffix strip. + if (part.type.size() > 7 && + part.type.compare(part.type.size() - 7, 7, "_embeds") == 0) { + return part.type; + } + return std::nullopt; +} + +// chat_utils.py:648-662. +void ValidateChatMmLimits(const multimodal::BaseProcessingInfo& info, + const std::vector& messages) { + // The running item count, keyed by the ORIGINAL (as-written) modality — + // `self._items_by_modality[original_modality]` (chat_utils.py:625,652). Note + // it is NOT keyed by the input modality: `image` and `image_embeds` keep + // separate counters upstream, and only the LIMIT is looked up under the + // stripped name (:633,662). Keying on the stripped name here would refuse a + // request upstream serves. + // + // The map lives for the whole `messages` walk, not per message, because + // upstream's tracker does: two images across two turns of one request are two + // images, so the limit cannot be evaded by splitting them up. + std::map items_by_modality; + for (const ChatMessage& message : messages) { + if (!message.content_parts.has_value()) continue; + for (const ChatContentPart& part : *message.content_parts) { + const std::optional modality = ChatPartModality(part); + if (!modality.has_value()) continue; + // The count INCLUDES this item (`len(...) + 1`, :652): upstream validates + // the length the item WOULD make, so the refusal names the limit the + // request crossed rather than the one it was already at. + const int num_items = ++items_by_modality[*modality]; + info.ValidateTrackedChatItem(*modality, num_items); + } + } +} + +std::map> Qwen3VLChatSupportedMmLimits() { + return {{"image", std::optional(1)}}; +} + std::function( const std::vector&)> MakeQwen3VLImageChatFn(const multimodal::Qwen3VLImageProcessor& proc, const vllm::tok::Tokenizer& tokenizer, - ChatPromptRenderFn prompt_fn, ImageCodecFn codec) { - return [&proc, &tokenizer, prompt_fn = std::move(prompt_fn), + ChatPromptRenderFn prompt_fn, ImageCodecFn codec, + const multimodal::BaseProcessingInfo& info) { + return [&proc, &tokenizer, &info, prompt_fn = std::move(prompt_fn), codec = std::move(codec)](const std::vector& messages) -> std::optional { - // Locate the FIRST image part across the messages (single-image; multiple - // images / video / audio are named residuals). + // STEP 0 (#607 L2, #686): the per-item limit check, BEFORE anything is + // decoded or dropped. chat_utils.py:662 validates as it tracks, for the same + // reason: refusing costs nothing, and truncating is invisible. + ValidateChatMmLimits(info, messages); + + // Locate the image part across the messages. At most ONE survives the check + // above (Qwen3VLChatSupportedMmLimits caps image at 1), so this loop no + // longer silently drops a second one — there cannot be one. const ChatContentPart* image_part = nullptr; for (const ChatMessage& m : messages) { if (!m.content_parts.has_value()) continue; diff --git a/src/vllm/entrypoints/openai/server_main.cpp b/src/vllm/entrypoints/openai/server_main.cpp index 18003a777..ef73b92a4 100644 --- a/src/vllm/entrypoints/openai/server_main.cpp +++ b/src/vllm/entrypoints/openai/server_main.cpp @@ -63,6 +63,7 @@ #include "vllm.h" #include "vllm/config/device.h" #include "vllm/config/kv_transfer.h" +#include "vllm/config/multimodal.h" #include "vllm/config/scheduler.h" #include "vllm/entrypoints/chat_template.h" #include "vllm/entrypoints/model_loader.h" @@ -269,6 +270,17 @@ struct Args { // JSON object vLLM takes (e.g. '{"method":"mtp","num_speculative_tokens":1}'). // Empty (default) == no speculation == the inert production path (SPEC-MTP I5d). std::string speculative_config; + // ── Multimodal input limits (ENG-MM-INPUT-PIPELINE wave L2, #607) ───────── + // --language-model-only (arg_utils.py:555,1276,1691) and --limit-mm-per-prompt + // (arg_utils.py:556,1279,1692), the two flags 43 of the 157 official recipes + // need. They are sugar over vllm::MultiModalConfig, which is the mechanism: + // the flag sets every modality limit to 0, and the REFUSAL those zeros produce + // (BaseProcessingInfo::ValidateNumItems, #607 L1) is the observable effect. + // + // Defaults reproduce today's behaviour exactly: language_model_only false and + // an empty map resolve to the 999-per-modality default, so a server started + // without either flag refuses nothing it used to serve. + vllm::MultiModalConfig multimodal; }; // ── Accepted-and-inert serve arguments (SERVE-RECIPE-ARGS, #606) ──────────── @@ -360,6 +372,8 @@ const InertArg* FindAcceptedInertArg(const std::string& flag) { " [--reasoning-parser |auto|none]\n" " [--kv-transfer-config '']\n" " [--speculative-config '']\n" + " [--[no-]language-model-only]\n" + " [--limit-mm-per-prompt '']\n" " [--version]\n" " accepted for published-recipe compatibility, NO effect: " "--enable-auto-tool-choice, --trust-remote-code\n"; @@ -524,6 +538,48 @@ Args ParseArgs(int argc, char** argv) { a.kv_transfer_config = NextArg(argc, argv, i, argv[0]); } else if (flag == "--speculative-config") { a.speculative_config = NextArg(argc, argv, i, argv[0]); + } else if (flag == "--language-model-only" || + flag == "--no-language-model-only") { + // arg_utils.py:1276 over a bool field, which _compute_kwargs gives + // argparse.BooleanOptionalAction (arg_utils.py:346-348) — so upstream + // accepts BOTH spellings and the negative one is the default. Mirrored, + // rather than accepting only the positive: a recipe that switches the flag + // off explicitly must not die on an unknown argument. + a.multimodal.language_model_only = flag == "--language-model-only"; + } else if (flag == "--limit-mm-per-prompt") { + // arg_utils.py:1279 over a dict field => type=parse_type(json.loads) + // (arg_utils.py:375-377): the value is a JSON OBJECT. Parsed HERE, before + // the multi-GB model load, for the same reason the parser dialects are: + // a malformed limit costs a second rather than a full load, and it is + // REFUSED rather than defaulted — a typo that silently became 999 is a + // limit that is not there. + // + // NAMED RESIDUAL, and it is not this flag's: upstream's dotted spelling + // (`--limit-mm-per-prompt.image 2`) is a FlexibleArgumentParser feature + // that rewrites any dotted key into the JSON form before argparse sees it + // (argparse_utils.py:389-425). It applies equally to --kv-transfer-config + // and --speculative-config, which this server also takes as JSON only, so + // adding it for one flag would be the bespoke path. It belongs to a parser + // brick covering all three. + const std::string value = NextArg(argc, argv, i, argv[0]); + std::vector ignored_options; + try { + a.multimodal.limit_per_prompt = + vllm::ParseLimitMmPerPromptJson(value, &ignored_options); + } catch (const std::exception& e) { + std::cerr << "server: --limit-mm-per-prompt: " << e.what() << "\n"; + Usage(argv[0], 2); + } + // Accepting is ANNOUNCED (the kAcceptedInertArgs rule, applied to a + // partially-inert VALUE): the dummy-profiling options are validated and + // then dropped, so a reader learns `num_frames` did nothing here instead + // of inferring that it worked. + for (const std::string& key : ignored_options) { + std::cerr << "server: --limit-mm-per-prompt " << key + << " accepted and IGNORED: it sizes dummy inputs for memory " + "profiling, a surface this engine does not have; only the " + "count is read\n"; + } } else if (flag == "--version") { std::cout << "vllm.cpp " << vllm::Version() << " c-abi=" << VLLM_ABI_VERSION << "\n"; @@ -656,6 +712,37 @@ int VllmServerMain(int argc, char** argv) { << "\n"; } + // The RESOLVED multimodal limits, printed before the model load so the + // startup log records what the two flags actually produced rather than what + // was typed. GetLimitPerPrompt is the single accessor every consumer asks + // (multimodal.py:321-336), so printing through it is what makes the flag's + // PRECEDENCE visible: --language-model-only alongside an explicit + // `image=4` prints image=0, because the flag is checked before the map. + { + std::cerr << "server: multimodal limits language-model-only=" + << (args.multimodal.language_model_only ? "ON" : "OFF"); + // Print every modality the user named, plus the ones the flag zeroes even + // though nobody named them, so "--language-model-only" is not a line that + // says only "ON" with no consequence next to it. + std::vector shown; + for (const auto& [modality, unused] : args.multimodal.limit_per_prompt) { + (void)unused; + shown.push_back(modality); + } + if (shown.empty() && args.multimodal.language_model_only) { + shown = {"audio", "image", "video"}; + } + for (const std::string& modality : shown) { + std::cerr << " " << modality << "=" + << args.multimodal.GetLimitPerPrompt(modality); + } + if (shown.empty()) { + std::cerr << " (no per-modality limit set; default " + << vllm::kDefaultLimitPerPrompt << ")"; + } + std::cerr << "\n"; + } + const fs::path dir = NativeUtf8Path(args.model_dir); const std::string config_path = PathUtf8(dir / "config.json"); const std::string tokenizer_path = PathUtf8(dir / "tokenizer.json"); @@ -805,6 +892,10 @@ int VllmServerMain(int argc, char** argv) { engine_params.max_num_seqs = args.max_num_seqs; engine_params.max_num_batched_tokens = args.max_num_batched_tokens; engine_params.enable_prefix_caching = args.enable_prefix_caching; + // #607 L2: the multimodal input limits go onto the ENGINE, not into a + // server-local variable, so the C ABI and this server resolve one limit the + // same way and the chat seam can borrow the engine's copy. + engine_params.multimodal = args.multimodal; // --device: explicit device selection (ARCH-ONE-SURFACE ROW 8). "auto" // (default) keeps the accelerator-first probe byte-identical; an unknown // name throws HERE (a startup error), and an explicitly named ABSENT @@ -967,6 +1058,19 @@ int VllmServerMain(int argc, char** argv) { // tower + merge + MRoPE/DeepStack on the GPU worker consuming // Request.mm_features) is the remaining MM-SERVE-E2E residual — the engine // model runner has no mm-forward path yet. Kept alive for the server loop. + // + // #607 L2 / #686: the seam now REFUSES rather than truncates. It is + // constructed with a BaseProcessingInfo folding the engine's limits + // (loaded->mm_config(), where --limit-mm-per-prompt / --language-model-only + // landed) against this seam's own ceiling (Qwen3VLChatSupportedMmLimits — + // one image, no video, no audio), so a three-image request is answered with + // HTTP 400 "At most 1 image(s) may be provided in one prompt." instead of + // being served with its first image. Declared AFTER `loaded` so it is + // destroyed BEFORE the MultiModalConfig it references. It shares + // `mm_image_proc`'s lifetime shape exactly — both are borrowed by the + // closure `chat` holds and both outlive the server loop, which is the only + // time the closure runs. + std::unique_ptr mm_proc_info; std::unique_ptr mm_image_proc; const std::string preprocessor_config_path = PathUtf8(dir / "preprocessor_config.json"); @@ -1002,8 +1106,16 @@ int VllmServerMain(int argc, char** argv) { "multimodal image: container-format decode (PNG/JPEG -> RGB) is a " "named MM-SERVE residual; supply raw RGB (image/x-raw-rgb)"); }; + mm_proc_info = + std::make_unique( + loaded->mm_config(), oai::Qwen3VLChatSupportedMmLimits()); chat.set_multimodal_chat_fn(oai::MakeQwen3VLImageChatFn( - *mm_image_proc, tokenizer, chat_prompt_fn, std::move(codec))); + *mm_image_proc, tokenizer, chat_prompt_fn, std::move(codec), + *mm_proc_info)); + for (const auto& [modality, limit] : mm_proc_info->AllowedMmLimits()) { + std::cerr << "server: multimodal limit " << modality << "=" << limit + << " (over the request limit for this seam)\n"; + } std::cerr << "server: multimodal image seam wired (Qwen3-VL processor " "from " << preprocessor_config_path << ")\n"; diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index 3ccb9228f..e0a964adf 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -1056,6 +1056,13 @@ if(VLLM_CPP_SERVER) vllm_cpp_add_test(test_serve_recipe_args vllm/entrypoints/openai/test_serve_recipe_args.cpp) target_include_directories(test_serve_recipe_args PRIVATE ${CMAKE_SOURCE_DIR}/src) + # ENG-MM-INPUT-PIPELINE wave L2 (#607): --language-model-only / + # --limit-mm-per-prompt. Same shape and the same reason as the row above — it + # re-execs the REAL VllmServerMain to observe the flags reaching the config and + # a malformed limit aborting before the load. + vllm_cpp_add_test(test_serve_mm_limits + vllm/entrypoints/openai/test_serve_mm_limits.cpp) + target_include_directories(test_serve_mm_limits PRIVATE ${CMAKE_SOURCE_DIR}/src) endif() vllm_cpp_add_test(test_outputs vllm/test_outputs.cpp) diff --git a/tests/capi/test_capi.cpp b/tests/capi/test_capi.cpp index b118384e6..6eea48ff8 100644 --- a/tests/capi/test_capi.cpp +++ b/tests/capi/test_capi.cpp @@ -30,6 +30,7 @@ #include "capi/engine_handle.h" #include "vllm/config/device.h" +#include "vllm/config/multimodal.h" #include "vllm/entrypoints/model_loader.h" #include "vllm/platforms/interface.h" #include "vllm/entrypoints/openai/serving_utils.h" @@ -1245,6 +1246,90 @@ TEST_CASE("capi: enable_jump_forward=on reaches the engine; default is inert (AB } } +// ── ABI v19: the multimodal input limits (#607 wave L2) ──────────────────── +// +// Ported from vllm/config/multimodal.py:78,81,212-236,321-336 and the two flags +// that set them (vllm/engine/arg_utils.py:555-556,1276-1279,1691-1692) @ +// 5559679229bc. Two fields, and the load-bearing property is that they and the +// SERVER FLAGS land on ONE config object — EngineParams::multimodal -> +// LoadedEngine::mm_config() — so the two entry points cannot resolve a limit +// differently. +TEST_CASE("capi: multimodal input limits (ABI v19)") { + // The zero values keep a pre-v19 caller byte-identical: the flag off and NO + // limits configured, which resolves to 999 per modality (multimodal.py:331-333) + // — NOT 0. An empty map is "no limits", not "nothing allowed". + vllm_model_params def = vllm_model_params_default(); + CHECK(def.language_model_only == 0); + CHECK(def.limit_mm_per_prompt == nullptr); + + // A well-formed limit reaches model load (which then fails on the missing + // directory) rather than being rejected as a caller error. + vllm_model_params ok = vllm_model_params_default(); + ok.model_path = "/nonexistent/vllm-cpp/model/dir"; + ok.limit_mm_per_prompt = "{\"image\": 2, \"video\": 0}"; + vllm_engine* eng = nullptr; + CHECK(vllm_engine_load(&ok, &eng) == VLLM_ERR_MODEL_LOAD); + CHECK(eng == nullptr); + + // ...and so does the flag, which needs no value at all. + vllm_model_params lm_only = vllm_model_params_default(); + lm_only.model_path = "/nonexistent/vllm-cpp/model/dir"; + lm_only.language_model_only = 1; + vllm_engine* eng_lm = nullptr; + CHECK(vllm_engine_load(&lm_only, &eng_lm) == VLLM_ERR_MODEL_LOAD); + + // Every upstream validation is a CALLER error here, reported before any model + // I/O — never a silent default to 999, which would leave the caller believing + // they set a limit they did not. + for (const char* bad_value : + {"{not json", "[1,2]", "{\"image\": \"two\"}", "{\"image\": -1}", + "{\"video\": {\"fps\": 2}}", "{\"video\": {\"num_frames\": 0}}"}) { + CAPTURE(bad_value); + vllm_model_params bad = vllm_model_params_default(); + bad.model_path = "/nonexistent/vllm-cpp/model/dir"; + bad.limit_mm_per_prompt = bad_value; + vllm_engine* eng_bad = reinterpret_cast(0x1); + CHECK(vllm_engine_load(&bad, &eng_bad) == VLLM_ERR_INVALID_ARGUMENT); + CHECK(eng_bad == nullptr); + } + CHECK(std::string(vllm_last_error()).find("limit_mm_per_prompt") != + std::string::npos); +} + +TEST_CASE("capi: EngineParams::multimodal reaches LoadedEngine::mm_config()") { + // The hop the ABI field and the two server flags SHARE. Without it the flags + // would be recorded on a struct nobody reads, which is exactly the + // "accepted and inert" failure the limits wave exists to avoid. + const HfConfig c = MakeConfig(); + { + EngineParams p = SyntheticParams(); + LoadedEngine e(c, MakeWeights(c), BuildFixture(), p); + // Default: no limits configured => 999 per modality, the pre-L2 behaviour. + CHECK(e.mm_config().GetLimitPerPrompt("image") == + vllm::kDefaultLimitPerPrompt); + } + { + EngineParams p = SyntheticParams(); + p.multimodal.limit_per_prompt = {{"image", 2}}; + LoadedEngine e(c, MakeWeights(c), BuildFixture(), p); + CHECK(e.mm_config().GetLimitPerPrompt("image") == 2); + // A modality nobody named keeps the default — an explicit limit for one + // modality is not a global switch. + CHECK(e.mm_config().GetLimitPerPrompt("video") == + vllm::kDefaultLimitPerPrompt); + } + { + // The precedence that matters (multimodal.py:326-327): the flag is checked + // BEFORE the map, so an explicit non-zero entry does not survive it. + EngineParams p = SyntheticParams(); + p.multimodal.language_model_only = true; + p.multimodal.limit_per_prompt = {{"image", 4}}; + LoadedEngine e(c, MakeWeights(c), BuildFixture(), p); + CHECK(e.mm_config().GetLimitPerPrompt("image") == 0); + CHECK(e.mm_config().GetLimitPerPrompt("video") == 0); + } +} + TEST_CASE("capi: kv_transfer_config parses and validates the connector name") { // A well-formed config naming a REGISTERED connector passes the gate and // reaches model load. diff --git a/tests/vllm/entrypoints/openai/test_api_server.cpp b/tests/vllm/entrypoints/openai/test_api_server.cpp index 8015de896..29fb6c1af 100644 --- a/tests/vllm/entrypoints/openai/test_api_server.cpp +++ b/tests/vllm/entrypoints/openai/test_api_server.cpp @@ -55,7 +55,9 @@ #include "vllm/config/device.h" #include "vllm/config/scheduler.h" +#include "vllm/config/multimodal.h" #include "vllm/entrypoints/model_loader.h" +#include "vllm/entrypoints/openai/chat_mm.h" #include "vllm/entrypoints/openai/serving_chat.h" #include "vllm/entrypoints/openai/serving_completion.h" #include "vllm/entrypoints/openai/serving_models.h" @@ -2663,3 +2665,196 @@ TEST_CASE("platform shutdown: teardown drains an acquired console handler") { CloseHandle(acquired); } #endif + +// ── MULTIMODAL INPUT LIMITS, END TO END (#607 wave L2, #686) ──────────────── +// +// The gate PR #685's reviewer owed to this wave: L1 ported the refusal and +// proved it throws, but nothing called it on a live request, so the claim "and +// it becomes HTTP 400" was unproven end to end. This is that proof, and it is +// deliberately written to distinguish 400 from BOTH of the wrong answers: +// +// * NOT 500. `InputValidationError` is caught at api_server.cpp:252, AHEAD of +// the generic `std::exception -> InternalServerError` arm. A refusal thrown +// as any other type would land as a 500 — a client mistake reported as a +// server fault — which is exactly why L1 reused the ONE validation type +// instead of adding a multimodal-only one. Leg 3 below pins that by +// throwing a bespoke type through the same seam and observing the 500. +// * NOT a truncated 200. Before this wave the seam took the first image_url +// part and `break`ed, so THREE images produced a perfectly ordinary 200 +// about one of them (#686). Leg 2 shows a within-limit request still +// answering 200, so the 400 is a LIMIT decision, not "multimodal is off". +// +// RECORDED, because the RED run says something #686 does not: against the code +// before this wave, leg 2 (the validate-then-build shape) returned the truncated +// 200 exactly as #686 describes, but leg 1 — the REAL MakeQwen3VLImageChatFn — +// returned a FIVE HUNDRED. It injects one placeholder marker per image part but +// routes only the first image, so ExpandImagePlaceholders raised "more image +// placeholders than grids" and the client saw a server fault carrying an +// internal message. Both are wrong in the same way (neither is upstream's +// refusal) and both become the 400 below, but the issue's "served with one, +// silently" understates the production seam's case rather than overstating it. +// +// Upstream chain: chat_utils.py:662 (the per-item check) -> +// multimodal/processing/context.py:409-428 (VLLMValidationError) -> +// serve/utils/error_response.py:62-65 (BadRequestError / 400), all at +// 5559679229bc. +TEST_CASE("api_server: an over-limit multimodal chat request is HTTP 400") { + namespace oai = vllm::entrypoints::openai; + const HfConfig c = MakeConfig(); + const Qwen3_5MoeWeights w = MakeWeights(c); + ServerHarness h(c, w, Fixture()); + + // A chat body carrying `n` image parts, in the OpenAI array-content form. + auto ImageBody = [](int n) { + json parts = json::array(); + for (int i = 0; i < n; ++i) { + parts.push_back({{"type", "image_url"}, + {"image_url", + {{"url", "data:image/x-raw-rgb;base64,AAAA"}}}}); + } + parts.push_back({{"type", "text"}, {"text", "hello"}}); + const json body = { + {"messages", json::array({{{"role", "user"}, {"content", parts}}})}, + {"max_completion_tokens", 4}, + {"temperature", 0.0}}; + return body.dump(); + }; + + // The seam's own limits, folded with a DEFAULT MultiModalConfig (no + // --limit-mm-per-prompt, no --language-model-only): image=1. + const vllm::MultiModalConfig default_cfg; + const vllm::multimodal::BaseProcessingInfo info( + default_cfg, oai::Qwen3VLChatSupportedMmLimits()); + + // ── LEG 1: the REAL production seam. MakeQwen3VLImageChatFn validates before + // it decodes anything, so the processor and codec below are never reached on + // this path — which is why a synthetic processor config is honest here: the + // refusal is the whole code path under test. + vllm::multimodal::Qwen3VLProcessorConfig pcfg; + pcfg.image_token_id = 3; // inside the fixture vocab (ids 0..21) + const vllm::multimodal::Qwen3VLImageProcessor proc(pcfg); + oai::ImageCodecFn never_reached = + [](const oai::DecodedMedia&) -> oai::DecodedImageRgb { + FAIL("the codec must not run: the limit check refuses first"); + return {}; + }; + h.chat.set_multimodal_chat_fn(oai::MakeQwen3VLImageChatFn( + proc, Fixture(), InVocabChatPrompt, never_reached, info)); + + ApiServer::DispatchResult refused = + h.server.handle_chat_completions(ImageBody(3)); + CHECK(refused.status == 400); + // Explicitly NOT the 500 arm, and explicitly not a 200 body. + CHECK(refused.status != 500); + CHECK(refused.status != 200); + { + const json j = json::parse(refused.body); + CHECK(j.at("error").at("type") == "BadRequestError"); + // Upstream's message text VERBATIM (context.py:421-423), reaching the wire. + CHECK(j.at("error").at("message") == + "At most 1 image(s) may be provided in one prompt."); + // A truncated 200 would have carried a completion instead. + CHECK_FALSE(j.contains("choices")); + } + + // ── LEG 2: the SAME server, a WITHIN-limit request, still 200. This is what + // separates "the limit refused you" from "multimodal requests are off": the + // seam here is the production SHAPE (validate, then build the engine input), + // with the synthetic engine's in-vocab ids standing in for the real + // processor's expansion, because the fixture vocab is 22 tokens wide and a + // real Qwen image id (151655) would run off the end of it. + h.chat.set_multimodal_chat_fn( + [&info](const std::vector& messages) + -> std::optional { + oai::ValidateChatMmLimits(info, messages); + vllm::multimodal::MultiModalInputs mm; + mm.prompt_token_ids = {13, 17}; // "hello", " world" + vllm::multimodal::MultiModalFeatureSpec spec; + spec.modality = "image"; + spec.offset = 0; + spec.length = 1; + mm.mm_features.push_back(std::move(spec)); + return mm; + }); + ApiServer::DispatchResult served = + h.server.handle_chat_completions(ImageBody(1)); + CHECK(served.status == 200); + { + const json j = json::parse(served.body); + CHECK(j.at("object") == "chat.completion"); + CHECK(j.at("choices").at(0).at("message").at("role") == "assistant"); + } + // ...and the same seam refuses three, so leg 1's 400 was not an artifact of + // the production seam's own decode path. + CHECK(h.server.handle_chat_completions(ImageBody(3)).status == 400); + + // ── LEG 3: the DISCRIMINATOR. A seam that throws anything OTHER than + // InputValidationError lands as a 500. Without this leg, "status == 400" + // could be satisfied by a handler that answered 400 to every seam failure, + // and the type L1 chose would be doing no work. + h.chat.set_multimodal_chat_fn( + [](const std::vector&) + -> std::optional { + throw std::runtime_error("a seam failure that is NOT a validation error"); + }); + ApiServer::DispatchResult faulted = + h.server.handle_chat_completions(ImageBody(1)); + CHECK(faulted.status == 500); + CHECK(json::parse(faulted.body).at("error").at("type") == + "InternalServerError"); +} + +TEST_CASE("api_server: --language-model-only answers an image request with 400") { + // The flag's main observable effect, at the HTTP boundary. Upstream's + // --language-model-only is not "the same server, minus some VRAM" — it is a + // server that answers an image request with "At most 0 image(s) may be + // provided in one prompt." (multimodal.py:78-80 -> :326-327 -> + // context.py:409-428). The VRAM half is wave L3 and is NOT claimed here. + namespace oai = vllm::entrypoints::openai; + const HfConfig c = MakeConfig(); + const Qwen3_5MoeWeights w = MakeWeights(c); + ServerHarness h(c, w, Fixture()); + + vllm::MultiModalConfig lm_only; + lm_only.language_model_only = true; + const vllm::multimodal::BaseProcessingInfo info( + lm_only, oai::Qwen3VLChatSupportedMmLimits()); + + vllm::multimodal::Qwen3VLProcessorConfig pcfg; + pcfg.image_token_id = 3; + const vllm::multimodal::Qwen3VLImageProcessor proc(pcfg); + oai::ImageCodecFn never_reached = + [](const oai::DecodedMedia&) -> oai::DecodedImageRgb { + FAIL("the codec must not run under --language-model-only"); + return {}; + }; + h.chat.set_multimodal_chat_fn(oai::MakeQwen3VLImageChatFn( + proc, Fixture(), InVocabChatPrompt, never_reached, info)); + + const json body = { + {"messages", + json::array({{{"role", "user"}, + {"content", + json::array({{{"type", "image_url"}, + {"image_url", + {{"url", + "data:image/x-raw-rgb;base64,AAAA"}}}}, + {{"type", "text"}, {"text", "hello"}}})}}})}, + {"max_completion_tokens", 4}, + {"temperature", 0.0}}; + const ApiServer::DispatchResult r = + h.server.handle_chat_completions(body.dump()); + CHECK(r.status == 400); + const json j = json::parse(r.body); + CHECK(j.at("error").at("type") == "BadRequestError"); + CHECK(j.at("error").at("message") == + "At most 0 image(s) may be provided in one prompt. " + "Set `--limit-mm-per-prompt` to increase this limit."); + + // A TEXT-only request on the same server is unaffected — the flag limits + // multimodal INPUT, it does not turn the server off. + const ApiServer::DispatchResult text = h.server.handle_chat_completions( + R"({"messages":[{"role":"user","content":"hello"}],)" + R"("max_completion_tokens":4,"temperature":0.0})"); + CHECK(text.status == 200); +} diff --git a/tests/vllm/entrypoints/openai/test_chat_mm.cpp b/tests/vllm/entrypoints/openai/test_chat_mm.cpp index 74f7abe70..e54cab406 100644 --- a/tests/vllm/entrypoints/openai/test_chat_mm.cpp +++ b/tests/vllm/entrypoints/openai/test_chat_mm.cpp @@ -507,10 +507,15 @@ TEST_CASE("chat-mm seam body: MakeQwen3VLImageChatFn -> expanded engine input") return out; }; - // The seam body, wired exactly as examples/server/main.cpp does (real - // tokenizer + the chat-prompt renderer). + // The seam body, wired exactly as server_main.cpp does (real tokenizer + the + // chat-prompt renderer + the seam's own processing info, #607 L2). The config + // is DEFAULT here — 999 per modality — so this pre-existing single-image case + // is byte-identical to before the limits landed. + const vllm::MultiModalConfig default_mm_config; + const vllm::multimodal::BaseProcessingInfo info( + default_mm_config, oai::Qwen3VLChatSupportedMmLimits()); auto mm_fn = oai::MakeQwen3VLImageChatFn( - proc, tok, oai::DefaultChatPromptFallback, std::move(codec)); + proc, tok, oai::DefaultChatPromptFallback, std::move(codec), info); // An OpenAI image chat request (image then text, matching the M0 golden order). const std::string uri = "data:image/x-raw-rgb;base64," + EncodeBase64(rgb); @@ -554,3 +559,249 @@ TEST_CASE("chat-mm seam body: MakeQwen3VLImageChatFn -> expanded engine input") const ChatCompletionRequest text_req = jt.get(); CHECK_FALSE(mm_fn(text_req.messages).has_value()); } + +// --------------------------------------------------------------------------- +// 8. THE LIMIT CHECK ON THE CHAT PATH (#607 wave L2, closing #686). +// +// Ported from vllm/entrypoints/chat_utils.py:630-662 @ 5559679229bc — the +// tracker that counts every parsed multimodal content part and validates the +// RUNNING count before the request reaches the engine — over +// BaseProcessingInfo::{AllowedMmLimits,ValidateNumItems,ValidateTrackedChatItem} +// (#607 L1, multimodal/processing/context.py:392-405,409-428). +// +// RED line, and it is a BEHAVIOURAL one rather than an absence: before this +// wave the seam located the FIRST image_url part and `break`ed +// (chat_mm.cpp:256-268), so a three-image request was SERVED WITH ONE — no +// error, no warning, a confident answer about a subset of the input (#686). +// Every CHECK_THROWS below fails against that code, because it does not +// throw; it returns the first image's 196 tokens and discards two. +// --------------------------------------------------------------------------- +namespace { + +// One `image_url` content part carrying the committed raw-RGB fixture. The +// payload is only decoded AFTER the limit check passes, so the refusal cases +// never reach the codec — which is the point of validating first. +ChatContentPart ImagePart(const std::string& uri) { + ChatContentPart p; + p.type = "image_url"; + p.url = uri; + return p; +} + +ChatContentPart TypedPart(const std::string& type) { + ChatContentPart p; + p.type = type; + p.url = "data:application/octet-stream;base64,AAAA"; + return p; +} + +// N image parts in ONE user message. +std::vector ImageMessages(int n, const std::string& uri) { + ChatMessage m; + m.role = "user"; + m.content = std::string("what is in these?"); + std::vector parts; + for (int i = 0; i < n; ++i) parts.push_back(ImagePart(uri)); + m.content_parts = std::move(parts); + std::vector out; + out.push_back(std::move(m)); + return out; +} + +} // namespace + +TEST_CASE("chat mm limits: ChatPartModality mirrors MM_PARSER_MAP") { + namespace oai = vllm::entrypoints::openai; + // chat_utils.py:1478-1483. + CHECK(oai::ChatPartModality(ImagePart("data:image/x-raw-rgb;base64,AA")) == + std::optional("image")); + CHECK(oai::ChatPartModality(TypedPart("video_url")) == + std::optional("video")); + CHECK(oai::ChatPartModality(TypedPart("audio_url")) == + std::optional("audio")); + CHECK(oai::ChatPartModality(TypedPart("input_audio")) == + std::optional("audio")); + // Text is not multimodal input; it never reaches a limit. + CHECK_FALSE(oai::ChatPartModality(TypedPart("text")).has_value()); + CHECK_FALSE(oai::ChatPartModality(TypedPart("refusal")).has_value()); + // `*_embeds` passes THROUGH verbatim (chat_utils.py:635): the suffix strip and + // the enable_mm_embeds escape belong to ValidateTrackedChatItem, not here. + CHECK(oai::ChatPartModality(TypedPart("image_embeds")) == + std::optional("image_embeds")); +} + +TEST_CASE("chat mm limits: THREE images are REFUSED, not truncated (#686)") { + namespace oai = vllm::entrypoints::openai; + const std::string dir = ImgFixDir(); + const nlohmann::json manifest = ReadJson(dir + "/manifest.json"); + auto cfg = ImageConfigFromManifest(manifest); + const int64_t H = manifest.at("image").at("shape")[0].get(); + const int64_t W = manifest.at("image").at("shape")[1].get(); + const std::vector rgb = + ReadBytes(dir + "/image_rgb_uint8_448x448x3.bin"); + static const vllm::tok::Tokenizer tok = vllm::tok::Tokenizer::FromHfJson( + std::string(PARITY_GOLDENS_DIR) + "/tokenizer_qwen36/tokenizer.json"); + const std::vector pad_ids = + tok.EncodeWithSpecialTokens("<|image_pad|>"); + REQUIRE(pad_ids.size() == 1u); + cfg.image_token_id = pad_ids[0]; + vllm::multimodal::Qwen3VLImageProcessor proc(cfg); + + oai::ImageCodecFn codec = + [&](const oai::DecodedMedia& media) -> oai::DecodedImageRgb { + oai::DecodedImageRgb out; + out.rgb = media.bytes; + out.height = H; + out.width = W; + return out; + }; + const std::string uri = "data:image/x-raw-rgb;base64," + EncodeBase64(rgb); + + // The DEFAULT config: no --limit-mm-per-prompt, no --language-model-only. The + // refusal below therefore comes from the SEAM's own ceiling + // (Qwen3VLChatSupportedMmLimits: one image), folded by min() against the + // user's 999 (context.py:392-405). #686 asked exactly this question — "the + // model's own supported limit, the min() fold's other operand; today nothing + // declares one" — and this is the declaration. + const vllm::MultiModalConfig default_cfg; + const vllm::multimodal::BaseProcessingInfo info( + default_cfg, oai::Qwen3VLChatSupportedMmLimits()); + CHECK(info.AllowedMmLimits().at("image") == 1); + + auto mm_fn = oai::MakeQwen3VLImageChatFn( + proc, tok, oai::DefaultChatPromptFallback, codec, info); + + // ONE image still works, byte-identically: 196 expanded tokens. + const std::optional one = + mm_fn(ImageMessages(1, uri)); + REQUIRE(one.has_value()); + CHECK(one->mm_features.size() == 1u); + + // THREE images are REFUSED. Before L2 this returned the first image's 196 + // tokens and dropped two, which is the defect #686 records. + CHECK_THROWS_AS(mm_fn(ImageMessages(3, uri)), vllm::v1::InputValidationError); + try { + mm_fn(ImageMessages(3, uri)); + FAIL("expected the seam to refuse three images"); + } catch (const vllm::v1::InputValidationError& e) { + // Upstream's message VERBATIM (context.py:421-423). + CHECK(std::string(e.what()) == + "At most 1 image(s) may be provided in one prompt."); + // ...and WITHOUT the "--limit-mm-per-prompt" hint (:425-426): raising the + // user's limit would not help, because it is the seam that caps at one, so + // a hint would send the user to a flag that cannot fix their request. + CHECK(std::string(e.what()).find("--limit-mm-per-prompt") == + std::string::npos); + } + + // TWO images across TWO messages are refused too: the tracker's count is + // cumulative (chat_utils.py:648-652), so splitting the turn does not evade it. + std::vector split = ImageMessages(1, uri); + split.push_back(ImageMessages(1, uri)[0]); + CHECK_THROWS_AS(mm_fn(split), vllm::v1::InputValidationError); + + // A VIDEO part through the IMAGE seam is refused rather than dropped: video is + // absent from the seam's supported limits, which context.py:414-415 reads as + // "not supported", limit 0. + { + ChatMessage m; + m.role = "user"; + m.content = std::string("describe"); + m.content_parts = std::vector{TypedPart("video_url")}; + std::vector msgs; + msgs.push_back(std::move(m)); + try { + mm_fn(msgs); + FAIL("expected the seam to refuse a video part"); + } catch (const vllm::v1::InputValidationError& e) { + CHECK(std::string(e.what()) == + "At most 0 video(s) may be provided in one prompt."); + } + } +} + +TEST_CASE("chat mm limits: --limit-mm-per-prompt and --language-model-only bite") { + namespace oai = vllm::entrypoints::openai; + const std::string dir = ImgFixDir(); + const nlohmann::json manifest = ReadJson(dir + "/manifest.json"); + auto cfg = ImageConfigFromManifest(manifest); + const int64_t H = manifest.at("image").at("shape")[0].get(); + const int64_t W = manifest.at("image").at("shape")[1].get(); + const std::vector rgb = + ReadBytes(dir + "/image_rgb_uint8_448x448x3.bin"); + static const vllm::tok::Tokenizer tok = vllm::tok::Tokenizer::FromHfJson( + std::string(PARITY_GOLDENS_DIR) + "/tokenizer_qwen36/tokenizer.json"); + cfg.image_token_id = tok.EncodeWithSpecialTokens("<|image_pad|>")[0]; + vllm::multimodal::Qwen3VLImageProcessor proc(cfg); + oai::ImageCodecFn codec = + [&](const oai::DecodedMedia& media) -> oai::DecodedImageRgb { + oai::DecodedImageRgb out; + out.rgb = media.bytes; + out.height = H; + out.width = W; + return out; + }; + const std::string uri = "data:image/x-raw-rgb;base64," + EncodeBase64(rgb); + + // (a) --language-model-only: EVERY modality limit becomes 0 + // (multimodal.py:78-80,326-327), so the ONE image a default server serves is + // refused. This is the flag's main observable effect and the half a port most + // easily leaves out, because omitting it breaks nothing a text-only workload + // would notice. + { + vllm::MultiModalConfig lm_only; + lm_only.language_model_only = true; + const vllm::multimodal::BaseProcessingInfo info( + lm_only, oai::Qwen3VLChatSupportedMmLimits()); + auto mm_fn = oai::MakeQwen3VLImageChatFn( + proc, tok, oai::DefaultChatPromptFallback, codec, info); + try { + mm_fn(ImageMessages(1, uri)); + FAIL("expected --language-model-only to refuse an image request"); + } catch (const vllm::v1::InputValidationError& e) { + // HERE the hint IS appended (context.py:425-426): the seam CAN take this + // image, the configuration is what refused it, so raising the limit helps. + CHECK(std::string(e.what()) == + "At most 0 image(s) may be provided in one prompt. " + "Set `--limit-mm-per-prompt` to increase this limit."); + } + // A text-only request is untouched by the flag — it limits multimodal INPUT, + // it does not refuse the server. + ChatMessage text; + text.role = "user"; + text.content = std::string("hello there"); + std::vector text_msgs; + text_msgs.push_back(std::move(text)); + CHECK_FALSE(mm_fn(text_msgs).has_value()); + } + + // (b) --limit-mm-per-prompt '{"image": 0}' reaches the same refusal by the + // other route, which is the whole reason L1 ported the limits before the flag: + // the flag is sugar, the limits are the mechanism. + { + vllm::MultiModalConfig zeroed; + zeroed.limit_per_prompt = {{"image", 0}}; + const vllm::multimodal::BaseProcessingInfo info( + zeroed, oai::Qwen3VLChatSupportedMmLimits()); + auto mm_fn = oai::MakeQwen3VLImageChatFn( + proc, tok, oai::DefaultChatPromptFallback, codec, info); + CHECK_THROWS_AS(mm_fn(ImageMessages(1, uri)), + vllm::v1::InputValidationError); + } + + // (c) A user limit can only LOWER the seam's ceiling, never raise it + // (context.py:392-405 folds by min). `image=99` still refuses the second + // image. + { + vllm::MultiModalConfig raised; + raised.limit_per_prompt = {{"image", 99}}; + const vllm::multimodal::BaseProcessingInfo info( + raised, oai::Qwen3VLChatSupportedMmLimits()); + CHECK(info.AllowedMmLimits().at("image") == 1); + auto mm_fn = oai::MakeQwen3VLImageChatFn( + proc, tok, oai::DefaultChatPromptFallback, codec, info); + CHECK(mm_fn(ImageMessages(1, uri)).has_value()); + CHECK_THROWS_AS(mm_fn(ImageMessages(2, uri)), + vllm::v1::InputValidationError); + } +} diff --git a/tests/vllm/entrypoints/openai/test_serve_mm_limits.cpp b/tests/vllm/entrypoints/openai/test_serve_mm_limits.cpp new file mode 100644 index 000000000..9b893310d --- /dev/null +++ b/tests/vllm/entrypoints/openai/test_serve_mm_limits.cpp @@ -0,0 +1,353 @@ +// ENG-MM-INPUT-PIPELINE wave L2 (#607) — the SERVE FLAGS half. +// +// Ported from / grounded in: +// - vllm/engine/arg_utils.py:555 (`language_model_only: bool = +// MultiModalConfig.language_model_only`), :1276 (the argparse registration, +// which _compute_kwargs gives BooleanOptionalAction, :346-348), :1691 (it +// reaching the config); +// - vllm/engine/arg_utils.py:556 (`limit_mm_per_prompt: dict[...] = +// get_field(MultiModalConfig, "limit_per_prompt")`), :1279, :1692, whose +// dict type resolves to `type=parse_type(json.loads)` (:375-377) — so the +// value is a JSON OBJECT; +// - vllm/config/multimodal.py:17-43 (the DummyOptions dataclasses: `count: +// int = Field(999, ge=0)`, `extra="forbid"`, every option `Field(None, +// gt=0)`) and :212-236 (`_validate_limit_per_prompt`, which rewrites a bare +// int to {"count": } and routes each modality to its own dataclass); +// - vllm/config/multimodal.py:321-336 (`get_limit_per_prompt`, the precedence +// the startup banner prints THROUGH so the flag ordering is observable). +// All at 5559679229bc, the pinned oracle in .agents/upstream-sync.md. +// +// THE GATE: 43 of the 157 official vllm-project/recipes commands pass +// `--language-model-only` and this server used to stop at the unknown-argument +// guard before reading a weight. Both flags must now parse, must REACH THE +// CONFIG (not merely be swallowed — that is the failure mode #606's own spec +// names), and a malformed limit must ABORT rather than silently default to 999, +// because a limit that quietly became 999 is a limit that is not there. +// +// WHY A SUBPROCESS for the flag half. `ParseArgs` lives in an anonymous +// namespace and reports a bad argument through `Usage()`, which calls +// `std::exit`; an in-process call would take the whole test binary with it. Each +// case therefore RE-EXECS THIS TEST BINARY into a skip-decorated child that +// calls the REAL `VllmServerMain`, exactly as +// tests/vllm/entrypoints/openai/test_serve_recipe_args.cpp does (#606), and +// asserts on the child's combined output and exit status. The model directory is +// deliberately nonexistent, so "parsing succeeded" is observable without a +// checkpoint or a bound port. +// +// The PARSER half needs no subprocess: ParseLimitMmPerPromptJson is a library +// function, so its refusals are asserted directly. +#include + +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include + +#include "vllm/config/multimodal.h" +#include "vllm/entrypoints/openai/server_main.h" + +namespace { + +// A path that cannot exist, so the child always fails at load rather than +// serving forever. +constexpr const char* kMissingModel = + "--model /nonexistent/vllm-cpp/serve-mm-limits"; + +// Printed by VllmServerMain AFTER ParseArgs returns. Its presence proves that +// argument parsing succeeded and control reached engine construction. +constexpr const char* kPostParseBanner = "server: request logging"; +constexpr const char* kLoadBanner = "server: loading model from"; +constexpr const char* kUnknownArgument = "server: unknown argument"; +// The RESOLVED-limits banner. It prints through GetLimitPerPrompt, so it is a +// statement about the CONFIG, not about the argv. +constexpr const char* kLimitsBanner = "server: multimodal limits"; + +struct ChildRun { + std::string output; + int status = -1; +}; + +bool Contains(const std::string& haystack, const std::string& needle) { + return haystack.find(needle) != std::string::npos; +} + +ChildRun RunServer(const std::string& serve_args) { + // Resolve our own path HERE, in the parent: popen runs the command under + // /bin/sh, so a literal /proc/self/exe inside it would resolve to the shell. + char exe[4096]; + const ssize_t n = ::readlink("/proc/self/exe", exe, sizeof(exe) - 1); + REQUIRE(n > 0); + exe[n] = '\0'; + // Single quotes around the args would break the JSON values, which carry + // double quotes; the child reads them from the environment instead, and the + // shell only ever sees the assignment. Escape any single quote for the shell. + std::string quoted; + for (const char c : serve_args) { + if (c == '\'') { + quoted += "'\\''"; + } else { + quoted.push_back(c); + } + } + const std::string cmd = "VLLM_TEST_MM_SERVE_ARGS='" + quoted + "' " + + std::string(exe) + + " --no-skip --test-case='serve_mm_limits_child'" + " 2>&1"; + FILE* pipe = ::popen(cmd.c_str(), "r"); + REQUIRE(pipe != nullptr); + ChildRun run; + std::array buf{}; + while (std::fgets(buf.data(), static_cast(buf.size()), pipe) != nullptr) { + run.output += buf.data(); + } + const int closed = ::pclose(pipe); + REQUIRE(closed != -1); + run.status = WIFEXITED(closed) ? WEXITSTATUS(closed) : -1; + return run; +} + +// Split on spaces EXCEPT inside single quotes, so a JSON value survives as one +// argv entry the way a shell would hand it over. +std::vector SplitArgs(const std::string& text) { + std::vector out; + std::string current; + bool in_quote = false; + bool started = false; + for (const char c : text) { + if (c == '\'') { + in_quote = !in_quote; + started = true; + continue; + } + if (c == ' ' && !in_quote) { + if (started || !current.empty()) out.push_back(current); + current.clear(); + started = false; + continue; + } + current.push_back(c); + } + if (started || !current.empty()) out.push_back(current); + return out; +} + +} // namespace + +// The CHILD case, filtered out of a normal run. +TEST_CASE("serve_mm_limits_child" * doctest::skip()) { + const char* raw = std::getenv("VLLM_TEST_MM_SERVE_ARGS"); + REQUIRE(raw != nullptr); + std::vector args{"vllm-server"}; + for (std::string& token : SplitArgs(raw)) args.push_back(std::move(token)); + std::vector argv; + argv.reserve(args.size()); + for (std::string& arg : args) argv.push_back(arg.data()); + const int rc = vllm::entrypoints::openai::VllmServerMain( + static_cast(argv.size()), argv.data()); + std::cout << "SERVE_RC=" << rc << "\n" << std::flush; + std::exit(0); +} + +// ── The PARSER (vllm/config/multimodal.py:212-236 + the dataclasses :17-43) ── + +TEST_CASE("limit-mm-per-prompt: the LEGACY count-only object (multimodal.py:87-88)") { + std::vector ignored; + // Upstream's own example (:88). + const std::map limits = + vllm::ParseLimitMmPerPromptJson(R"({"image": 16, "video": 2})", &ignored); + CHECK(limits.at("image") == 16); + CHECK(limits.at("video") == 2); + CHECK(ignored.empty()); + + // 0 is a LEGAL limit, and it is the one that matters: it is what + // --language-model-only is defined as being equivalent to (multimodal.py:79). + const std::map zeroed = + vllm::ParseLimitMmPerPromptJson(R"({"image": 0})", nullptr); + CHECK(zeroed.at("image") == 0); + + vllm::MultiModalConfig cfg; + cfg.limit_per_prompt = zeroed; + CHECK(cfg.GetLimitPerPrompt("image") == 0); + // A modality the map does not mention keeps the 999 default (:331-333) — an + // explicit zero for ONE modality is not a global off switch. + CHECK(cfg.GetLimitPerPrompt("video") == vllm::kDefaultLimitPerPrompt); +} + +TEST_CASE("limit-mm-per-prompt: the CONFIGURABLE + MIXED objects (multimodal.py:90-96)") { + std::vector ignored; + // Upstream's own configurable example (:91-92). + const std::map limits = vllm::ParseLimitMmPerPromptJson( + R"({"video": {"count": 1, "num_frames": 32, "width": 512, "height": 512}, + "image": {"count": 5, "width": 512, "height": 512}})", + &ignored); + CHECK(limits.at("video") == 1); + CHECK(limits.at("image") == 5); + // Only `count` participates in get_limit_per_prompt (:335). The options are + // validated and dropped, and the drop is REPORTED so the server can announce + // it rather than let a user infer that num_frames took effect. + CHECK(ignored.size() == 5); + + // The MIXED spelling (:94-96): one modality as a count, one as an object. + const std::map mixed = vllm::ParseLimitMmPerPromptJson( + R"({"image": 16, "video": {"count": 1, "num_frames": 32}})", nullptr); + CHECK(mixed.at("image") == 16); + CHECK(mixed.at("video") == 1); + + // An object with no `count` keeps the dataclass default (:20), not 0 — the + // object exists to carry OPTIONS, so omitting the count is not a refusal. + const std::map options_only = + vllm::ParseLimitMmPerPromptJson(R"({"image": {"width": 512}})", nullptr); + CHECK(options_only.at("image") == vllm::kDefaultLimitPerPrompt); +} + +TEST_CASE("limit-mm-per-prompt: malformed input is REFUSED, never defaulted") { + // Each of these is a distinct upstream validation, and every one of them must + // throw rather than yield a map that silently reads 999. + const std::vector refused = { + "not json at all", // json.loads raises + "[1, 2, 3]", // not an object + "\"image\"", // not an object + R"({"image": "two"})", // count is not an int + R"({"image": 2.5})", // count is not an int + R"({"image": -1})", // count: Field(999, ge=0), :20 + R"({"image": {"count": -1}})", // same, through the object form + R"({"image": {"count": "two"}})", // same, wrong type + R"({"image": {"num_frames": 32}})", // image has no num_frames, :32-37 + R"({"video": {"fps": 2}})", // extra="forbid", :23 + R"({"audio": {"width": 2}})", // audio takes `length`, :39-43 + R"({"tactile": {"count": 1, "width": 2}})", // BaseDummyOptions: count ONLY + R"({"video": {"num_frames": 0}})", // Field(None, gt=0), :26 + R"({"video": {"num_frames": -4}})", // same + R"({"image": {"width": "big"}})", // option is not an int + }; + for (const std::string& doc : refused) { + CAPTURE(doc); + CHECK_THROWS_AS(vllm::ParseLimitMmPerPromptJson(doc, nullptr), + std::invalid_argument); + } + + // ...and the message NAMES what was wrong, because a dropped `num_frame` typo + // is invisible otherwise. + try { + vllm::ParseLimitMmPerPromptJson(R"({"video": {"num_frame": 32}})", nullptr); + FAIL("expected a refusal"); + } catch (const std::invalid_argument& e) { + const std::string msg = e.what(); + CHECK(Contains(msg, "num_frame")); + CHECK(Contains(msg, "video")); + } + + // An accepted modality name is NOT enumerated: upstream routes an unknown + // modality to BaseDummyOptions (:233-234) rather than rejecting it, because + // the modality set is the model's, not the config's. + CHECK(vllm::ParseLimitMmPerPromptJson(R"({"tactile": 3})", nullptr) + .at("tactile") == 3); +} + +// ── The FLAGS (arg_utils.py:1276,1279) reaching the CONFIG ────────────────── + +TEST_CASE("serve flags: --language-model-only parses and ZEROES every limit") { + const ChildRun run = + RunServer(std::string(kMissingModel) + " --language-model-only"); + INFO("child output:\n" << run.output); + CHECK_FALSE(Contains(run.output, kUnknownArgument)); + CHECK(Contains(run.output, kPostParseBanner)); + // It reached MODEL LOAD, which failed on the deliberately missing checkpoint + // — "parsing succeeded, the engine tried to load". + CHECK(Contains(run.output, kLoadBanner)); + CHECK_FALSE(Contains(run.output, "SERVE_RC=0")); + CHECK(run.status == 0); + // THE LOAD-BEARING ASSERTION: the flag reached the CONFIG. The banner prints + // through GetLimitPerPrompt, so this is the resolved limit, not the argv. + CHECK(Contains(run.output, "server: multimodal limits language-model-only=ON")); + CHECK(Contains(run.output, "image=0")); + CHECK(Contains(run.output, "video=0")); + CHECK(Contains(run.output, "audio=0")); +} + +TEST_CASE("serve flags: --no-language-model-only is accepted (BooleanOptionalAction)") { + // arg_utils.py:1276 registers the field with argparse.BooleanOptionalAction + // (arg_utils.py:346-348), which defines BOTH spellings. A recipe that turns + // the flag off explicitly must not die on an unknown argument. + const ChildRun run = + RunServer(std::string(kMissingModel) + " --no-language-model-only"); + INFO("child output:\n" << run.output); + CHECK_FALSE(Contains(run.output, kUnknownArgument)); + CHECK(Contains(run.output, "server: multimodal limits language-model-only=OFF")); + CHECK(Contains(run.output, kLoadBanner)); +} + +TEST_CASE("serve flags: --limit-mm-per-prompt reaches the config") { + const ChildRun run = RunServer(std::string(kMissingModel) + + " --limit-mm-per-prompt '{\"image\": 2, " + "\"video\": 0}'"); + INFO("child output:\n" << run.output); + CHECK_FALSE(Contains(run.output, kUnknownArgument)); + CHECK(Contains(run.output, kLimitsBanner)); + CHECK(Contains(run.output, "image=2")); + CHECK(Contains(run.output, "video=0")); + CHECK(Contains(run.output, kLoadBanner)); + CHECK(run.status == 0); +} + +TEST_CASE("serve flags: --language-model-only WINS over an explicit limit") { + // get_limit_per_prompt checks the flag BEFORE reading the map + // (multimodal.py:326-327). Reading the map first would be indistinguishable + // on every configuration except this one, which is why this case exists. + const ChildRun run = + RunServer(std::string(kMissingModel) + + " --limit-mm-per-prompt '{\"image\": 4}' --language-model-only"); + INFO("child output:\n" << run.output); + CHECK(Contains(run.output, "language-model-only=ON")); + CHECK(Contains(run.output, "image=0")); + CHECK_FALSE(Contains(run.output, "image=4")); +} + +TEST_CASE("serve flags: a malformed --limit-mm-per-prompt ABORTS before model load") { + // THE OTHER LOAD-BEARING ONE. Silently defaulting a mistyped limit to 999 is + // strictly worse than aborting: the server then runs with no limit at all, + // and the user believes they set one. + for (const char* value : + {"'not json'", "'[1,2]'", "'{\"image\": -1}'", + "'{\"video\": {\"fps\": 2}}'"}) { + CAPTURE(value); + const ChildRun run = RunServer(std::string(kMissingModel) + + " --limit-mm-per-prompt " + value); + INFO("child output:\n" << run.output); + CHECK(Contains(run.output, "server: --limit-mm-per-prompt")); + // Aborted inside ParseArgs: the model load never started. + CHECK_FALSE(Contains(run.output, kLoadBanner)); + CHECK_FALSE(Contains(run.output, "SERVE_RC=")); + } +} + +TEST_CASE("serve flags: the dropped profiling options are ANNOUNCED, not silent") { + const ChildRun run = + RunServer(std::string(kMissingModel) + + " --limit-mm-per-prompt '{\"video\": {\"count\": 1, " + "\"num_frames\": 32}}'"); + INFO("child output:\n" << run.output); + CHECK(Contains(run.output, "video=1")); + CHECK(Contains(run.output, "video.num_frames")); + CHECK(Contains(run.output, "IGNORED")); + CHECK(Contains(run.output, kLoadBanner)); +} + +TEST_CASE("serve flags: neither flag changes anything when neither is passed") { + // The RED line for the whole wave: a server started without either flag must + // be byte-identical to the pre-L2 one. 999 per modality is what + // multimodal.py:331-333 resolves, and no request reaches it. + const ChildRun run = RunServer(kMissingModel); + INFO("child output:\n" << run.output); + CHECK(Contains(run.output, "language-model-only=OFF")); + CHECK(Contains(run.output, "no per-modality limit set")); + CHECK(Contains(run.output, "default 999")); + CHECK(Contains(run.output, kLoadBanner)); +} From 75046265c4a1d6f6cc58388617a61ea758d2a5dc Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Fri, 14 Aug 2026 12:32:41 +0000 Subject: [PATCH 2/3] record(ENG-MM-INPUT-PIPELINE): re-derive the L2 anchors at the final head (#607, #686) Anchors are gate-unvalidated (#632) and rot on rebase, and this branch rebased twice while the gate ran. Every line reference the wave records was re-read at the final head rather than trusted: * api_server.cpp:185,252 -- both ARE `catch (const vllm::v1::InputValidationError& e)`, which is the claim the whole 400-not-500 argument rests on. * input_processor.cpp:321,352 -- the `mm_features` parameter and the `request.mm_features = std::move(mm_features);` assignment, i.e. the second call site this wave deliberately leaves unwired. * chat_mm.cpp -- the first-image loop MOVED. It was :256-268 before this wave and is :313-326 now, preceded by the new check at :311. The test comment and mm-serving.md said only the old numbers, which after L2 point at neither the code they describe nor the code that replaced it; both now say which is which. The engine-matrix row also gains the anchors the L2 code actually introduces (config/multimodal.cpp:49, chat_mm.cpp:295,311, vllm.h:197,403), so the record names the new surface rather than only the pre-existing hasher anchor. Records only. No behaviour, no test assertion, and no build input changes: the diff is two record files and one comment block. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --- .agents/engine-matrix.md | 2 +- .agents/specs/mm-serving.md | 3 ++- tests/vllm/entrypoints/openai/test_chat_mm.cpp | 4 +++- 3 files changed, 6 insertions(+), 3 deletions(-) diff --git a/.agents/engine-matrix.md b/.agents/engine-matrix.md index 2c16aece5..64bbd9a9d 100644 --- a/.agents/engine-matrix.md +++ b/.agents/engine-matrix.md @@ -71,7 +71,7 @@ forensics: roadmap_v1.md and the parity ledger. | `ENG-DBO-UBATCH` | DBO and ubatch overlap | T2 | `vllm/config/parallel.py:208,524` | - | - | `planned: specs/dbo-ubatch.md` | `INVENTORIED` | - | | `ENG-MOE-SHARED-AUX` | MoE shared-expert MLP on an aux CUDA stream concurrent with the routed-expert router/align/grouped-GEMMs (mirror vLLM's decode overlap; the largest remaining 35B c1/c2 engine lever). Fork the shared MLP onto a 2nd persistent per-device stream, join before the combine → byte-identical to serial (independent shared/routed paths both complete before combine; overlap changes WHEN not WHAT). Gated `T <= threshold` decode + CUDA. The aux stream draws scratch from a SEPARATE `AuxPool` so the concurrent main-stream routed allocations never share a live block with it (the `DevicePool` reuse invariant is single-stream ordering; vLLM sidesteps this with its stream-aware caching allocator's `record_stream`). `VT_MOE_SHARED_AUX_STREAM` **DEFAULT ON** (`=0` rollback); `VT_MOE_SHARED_AUX_THRESHOLD` (default 128; GB10 48-SM calibration). Captured in the decode CUDA-graph via the fork/join event edges (`ThreadLocal` capture, no abort). Only the committed Marlin MoE decode path; wmma fallback/CPU/GGUF and 27B dense unaffected | T1 | `vllm/model_executor/layers/fused_moe/runner/shared_experts.py:99-104,125-142`; `vllm/utils/multi_stream_utils.py:20-58` (`maybe_execute_in_parallel`, TRT-LLM port); `vllm/utils/torch_utils.py:736-756` (`aux_stream`); `vllm/envs.py:260` (threshold 256) | fork/join `src/vllm/model_executor/models/qwen3_5.cpp:3999,4114` (`MoeBlockFusedMarlinCuda`); aux stream+events `src/vllm/model_executor/models/qwen3_5.cpp:3575,3581` (`MoeAuxStream`/`MoeAuxStreamFor`); predicates `:3553,3560`; aux-pool isolation `:496,3538` (`AuxPool`/`ActivePool`/`ActivePoolScope`) + `DBuf pool_` routing `:645` | **DGX (prod flags, one flock):** overlap ON==OFF BYTE-IDENTICAL — `tests/parity/test_qwen36_paged_engine.cpp:116` 35B **315/315** + `tests/parity/test_qwen27_paged_engine.cpp` 27B **235/235** under `VT_MOE_SHARED_AUX_STREAM`∈{0,1}; captured-vs-eager (`VLLM_CPP_CUDAGRAPH=0`, ON) 315/315; shipping default (no env) 315/315+235/235, rollback `=0` 315/315+235/235; `compute-sanitizer memcheck` (default ON, captured) 0 errors; in-situ interleaved TPOT A/B (drop cold rep1) c1 −5.6% / c2 −2.7% / c4 −3.7% / c8 −3.4% / c16 −1.6% / c32 −1.5% (WINS every conc, zero regression); ledger [parity-ledger.md](parity-ledger.md) 2026-07-19 row | [moe-shared-aux-stream.md](specs/moe-shared-aux-stream.md) | `ANCHOR-BACKFILL` | `CLAIM-MOE-SHARED-AUX-1` | | `ENG-RUNNER-MODELSHAPE` | **Runner is model-shape-agnostic over the KV-cache group structure** — the extensibility deliverable the first additive-model bring-up (Qwen3 dense) forced. Before W1 the `GPUModelRunner` had only ever executed the Qwen3.6 HYBRID topology and hardcoded it in two places: (#1) the KV-buffer alloc loop indexed `config_.layer_types[l]`, out-of-bounds on a pure-dense model's EMPTY `layer_types`; (#2) each `execute_model` step unconditionally built the GDN metadata (`gather_block_table(gdn_group_id_)` / `remap_gdn_state_slots` / `GDNAttentionMetadataBuilder`), which reads `block_table[-1]` when there is no mamba group. W1 drives both off the resolved KV-group structure — a model-agnostic `has_mamba_group` / `gdn_group_id_ >= 0` predicate (NOT a model-name check): empty/absent `layer_types` ⇒ all full-attention; no mamba group ⇒ the whole GDN metadata/state path is skipped and `gdn_meta` stays default-empty. A full-attention-only KV config (one FA group, no MambaSpec) now allocates + steps cleanly; the hybrid gate models keep their GDN group so their path is BYTE-IDENTICAL. This is a one-time generalization: every future dense/non-hybrid arch (Llama, Mistral) now adds new-files-only, zero further runner edits. **PER-LAYER KV head_dim extension (Gemma-4 G1b, 2026-07-28, `CLAIM-GEMMA4-G1B`):** the runner's full-attn alloc/view loops now consume an OPTIONAL `KVCacheConfig::per_layer_attn_specs` (index == layer) so a HETEROGENEOUS-head_dim model (Gemma-4: sliding 256 / global 512, same num_kv_heads) sizes each non-GDN layer's paged KV + PagedKvCache view from its OWN spec. The field is EMPTY for every uniform-KV model ⇒ the loop collapses to the single group spec ⇒ byte-identical allocation/view/indexing/dispatch (same additive-identical property as the model-shape generalization above). Block table / KV manager / scheduler stay head_dim-independent (num_blocks + block_size, uniform) so no per-group block table is introduced | T0 | model-agnostic runner drives off `kv_cache_config.kv_cache_groups` — `vllm/v1/worker/gpu/model_runner.py` `initialize_kv_cache` / attention-metadata build (per-group, no hardcoded hybrid) @ `e24d1b24` | `src/vllm/v1/worker/gpu/runner.cpp:458-470` (alloc loop: `has_mamba_group && !layer_types.empty()` gate) + `:651-680` (GDN metadata build gated on `gdn_group_id_ >= 0`, default-empty `gdn_meta` otherwise); per-layer KV head_dim: `include/vllm/v1/kv_cache_interface.h` (`KVCacheConfig::per_layer_attn_specs`) consumed in `src/vllm/v1/worker/gpu/runner.cpp` `initialize_kv_cache` (per-layer `FaDims` alloc+view), published by `src/vllm/model_executor/models/gemma4_registry.cpp` (`MakeGemma4ForConditionalGenerationKVCache`); the full-attention-only KV spec that exercises the base path `src/vllm/model_executor/models/qwen3_dense.cpp` (`MakeQwen3ForCausalLMKVCache`) | `tests/vllm/v1/worker/test_runner.cpp:1129` — "full-attention-only KV config allocates without the GDN path" + "full-attention-only step skips GDN metadata build (no OOB)" (RED→GREEN: both SIGSEGV pre-generalization; GREEN post). Behaviour-preservation gate: DGX **27B 235/235 + 35B 315/315 UNCHANGED** under the fix; per-layer-KV inertness: full CPU runner/KV suite green + **OLMo-2 SACRED GPU re-gate 16/16 UNCHANGED**; heterogeneous path proven by **Gemma-4 E4B STRICT 32/32** (`tests/parity/test_gemma4_paged_engine.cpp`); ASan/UBSan clean on the affected paths | [first-additive-model-qwen3-dense.md](specs/first-additive-model-qwen3-dense.md) §3 (seam gaps #1/#2), §6 (W1); [gemma4-multimodal.md](specs/gemma4-multimodal.md) §G1b | `ACTIVE` | `CLAIM-MODEL-QWEN3-DENSE` | -| `ENG-MM-INPUT-PIPELINE` | **Multimodal INPUT pipeline + encoder-cache engine seam (M1), INERT when no mm input.** The C++ mirror of `vllm/multimodal/`: `MultiModalKwargs`/`MultiModalFeatureSpec`/`MultiModalInputs`, the `MultiModalHasher` mm-hash (blake3), the Qwen3-VL image processor (smart_resize + fused rescale/normalize + patchify -> `pixel_values`+`image_grid_thw`) and placeholder-token expansion, plus the `EncoderCacheManager` (+`ComputeMmEncoderBudget`) and the LMCache `extra_keys` seam. Additive `mm_features` carried on `Request`/`EngineCoreRequest`; with NO mm input every field is empty and every path is byte-identical to the text engine. Processor output is BIT/BYTE-identical to the vLLM 0.25.0 oracle (M0 fixture). Does NOT build the vision tower / embed-merge (M2). **SERVING wiring (ROAD-V1-MM `MM-SERVE-ENGINE`, 2026-07-28, `CLAIM-MM-SERVING-W2`):** the OpenAI server now carries the parsed `MultiModalInputs` into the engine — additive `LLMEngine`/`AsyncLLM` `add_request(MultiModalInputs)`+`generate(MultiModalInputs)` overloads via `InputProcessor::process_inputs_mm` (mirror `input_processor.py:333-379`, empty mm_features == the tokens path), the chat-template placeholder-STRING helpers (`get_placeholder_str`/`_add_placeholder` mirror), and the serving_chat `MultiModalChatFn` seam (default unset ⇒ text byte-identical). **SEAM BODY (ROAD-V1-MM `MM-SERVE-E2E` W3, 2026-07-28, `CLAIM-MM-SERVING-E2E`):** `MakeQwen3VLImageChatFn` (chat_mm.cpp) is the seam body the server sets — messages → marker-inject → chat template → `EncodeWithSpecialTokens` (the single image_pad marker → one image_token_id) → `RouteImageRgb` EXPAND to 196 image tokens + mm_features; wired in `examples/server/main.cpp` (guarded on `preprocessor_config.json`; text-only unset ⇒ byte-identical). Gated `test_chat_mm` 8/8 + `test_openai_serving` (seam invoked + routed). **ENGINE MM-FORWARD LANDED (ROAD-V1-MM `MM-SERVE-E2E`, 2026-07-28, `CLAIM-ENGINE-MM-FORWARD`):** the engine model runner now HAS an mm forward — `ModelForwardInput` gains an ADDITIVE default-nullopt `std::optional mm` (merged inputs_embeds + 3-D MRoPE positions + DeepStack, borrowed handles; nullopt-for-text ⇒ shared runner path byte-identical BY CONSTRUCTION), `Qwen3VLForConditionalGeneration` is `REGISTER_VLLM_MODEL`-registered (`qwen3_vl_registry.cpp`), and the registered forward FOLDS the M2c decode into `ModelRegistry::Forward` via the SHARED `Qwen3VLForwardStepLastLogits` (`Qwen3VLGenerateGreedyViaRegistry` drives every step through the registry). GPU token-exact gate `test_qwen3vl_registry_e2e` (image→text THROUGH `ModelRegistry::Forward` == M2c golden 32/32 STRICT, dgx.casa GB10); text inertness `test_runner` 16/16 + `test_scheduler` 36/36 + `test_model_registry` 24/24 + `test_chat_mm` 8/8 + `test_openai_serving` 41/41 all green. RESIDUAL: the FULL in-runner scheduler-fed tower run (batched-loop mm building the field from staged encoder outputs) + the real server `/v1/chat/completions` GPU e2e — recipe in `specs/mm-serving.md`. **INPUT LIMITS L1 LANDED (#607, 2026-08-13):** the per-modality `limit_per_prompt` + `GetLimitPerPrompt` precedence (`language_model_only` ⇒ 0 BEFORE the map, else the map, else 999) and the refusal that gives those numbers effect — `AllowedMmLimits` folding by `min()` against the model's own ceiling, `ValidateNumItems` with upstream's exact message, and both call sites with the `enable_mm_embeds` escape. NO serve surface and NO live call site: nothing constructs a `MultiModalConfig` on a request yet, which is L2's. **INPUT LIMITS L2 LANDED (#607, #686, 2026-08-14):** the flags (`--[no-]language-model-only`, `--limit-mm-per-prompt ''`; `arg_utils.py:555-556,1276-1279,1691-1692` over `ParseLimitMmPerPromptJson`, the port of `multimodal.py:212-236` + the DummyOptions dataclasses `:17-43`), the C-ABI fields (`vllm_model_params.language_model_only`/`.limit_mm_per_prompt`, **ABI v19**), and the LIVE CALL SITE: `ValidateChatMmLimits` (`chat_utils.py:648-662`) runs as step 0 of `MakeQwen3VLImageChatFn` over a `BaseProcessingInfo` folding `LoadedEngine::mm_config()` with `Qwen3VLChatSupportedMmLimits() == {"image": 1}` — the seam's own ceiling, which is the `min()` fold operand #686 recorded as undeclared. A three-image request is now HTTP 400 with upstream's message rather than an opaque 500 / a truncated answer. NOT claimed: any memory win (L3 gates tower construction, unmeasured). Still unwired: the `process_inputs_mm` call site (`context.py:461`), blocked on the per-model `get_supported_mm_limits()` hook. | T1 | `vllm/multimodal/{inputs.py,hasher.py:50,processing/processor.py:1663,processing/inputs.py:62}`; `vllm/model_executor/models/qwen3_vl.py:{1400,1233}`; `vllm/v1/core/encoder_cache_manager.py:17`; transformers `image_processing_qwen2_vl.py:62`, `image_processing_backends.py:327`; tests `tests/multimodal/test_processing.py`, `tests/multimodal/test_hasher.py`, `tests/v1/core/test_encoder_cache_manager.py` @ `e24d1b24` | `src/vllm/multimodal/hasher.cpp`, `src/vllm/multimodal/qwen3vl_processor.cpp`, `include/vllm/multimodal/{inputs.h,hasher.h,qwen3vl_processor.h}`; `src/vllm/v1/core/encoder_cache_manager.cpp` + `include/vllm/v1/core/encoder_cache_manager.h`; additive inert fields `include/vllm/v1/request.h` + `src/vllm/v1/request.cpp` + `include/vllm/v1/engine/types.h`; `extra_keys` seam `include/vllm/v1/kv_offload/lmcache/chunked_token_database.h` + `.cpp`; M0 `scripts/mm/m0_oracle_capture.py`; L1 limits `include/vllm/config/multimodal.h` + `include/vllm/multimodal/processing/context.h` + `src/vllm/multimodal/processing/context.cpp`, refusal type relocated to `include/vllm/v1/engine/validation_error.h`; L2 flags+ABI+call site `src/vllm/config/multimodal.cpp` (`ParseLimitMmPerPromptJson`) + `src/vllm/entrypoints/openai/server_main.cpp` + `include/vllm.h` (ABI v19) + `src/capi/vllm_c.cpp` + `EngineParams::multimodal`/`LoadedEngine::mm_config()` + `src/vllm/entrypoints/openai/chat_mm.cpp` (`ChatPartModality`, `ValidateChatMmLimits`, `Qwen3VLChatSupportedMmLimits`) — anchor `src/vllm/multimodal/hasher.cpp:56` | `tests/vllm/multimodal/test_qwen3vl_processor.cpp` (processor-parity 23/23 BIT-identical vs the M0 oracle fixture `tests/vllm/multimodal/fixtures/qwen3vl/`, RED-first: wrong normalize shift -> 1.2M mismatches); `tests/vllm/v1/core/test_encoder_cache_manager.cpp` 32/32. Text-inertness: `test_request`/`test_engine_types`/`test_lmcache_codec`/`test_lmcache_key_agreement`/`test_openai_conformance` all green standalone; SACRED CUDA 27B/35B/Coder = GPU inertness proof; `check-device-leakage` OK — anchor `tests/vllm/multimodal/test_qwen3vl_processor.cpp:59`. L1 limits: `tests/vllm/config/test_multimodal_config.cpp` 7/7 (21 assertions) + `tests/vllm/multimodal/test_processing_limits.cpp` 19/19 (78 assertions), porting `tests/multimodal/test_processing.py:902-941,944-985`, `tests/entrypoints/multimodal/llm/test_mm_embeds_only.py:41-49` and `tests/entrypoints/unit_tests/test_chat_utils.py:1498-1560` @ `5559679229bc`; mutations proven RED: map-before-flag precedence, the dropped throw, the dropped `min()` fold. L2 (aarch64 `build-test-cpu-arm64` lane, `-DVLLM_CPP_CUDA=OFF`): `tests/vllm/entrypoints/openai/test_serve_mm_limits.cpp` 10/10 (101 assertions, flags + the parser's upstream refusals) + `test_chat_mm` 11/11 (126) + `test_openai_api_server` 56/56 (638, the HTTP 400 arm proven against BOTH a 500 and a truncated 200); RED-first behavioural: `CHECK(500 == 400)`, `CHECK("InternalServerError" == BadRequestError)` and `CHECK(200 == 400)`; mutations proven RED: flag→config plumbing dropped, the call-site wiring dropped, and the refusal re-typed off `InputValidationError` (which lands as the 500 L1's design avoided) | [multimodal-track.md](specs/multimodal-track.md) §3 (M0/M1) | `READY` | - | +| `ENG-MM-INPUT-PIPELINE` | **Multimodal INPUT pipeline + encoder-cache engine seam (M1), INERT when no mm input.** The C++ mirror of `vllm/multimodal/`: `MultiModalKwargs`/`MultiModalFeatureSpec`/`MultiModalInputs`, the `MultiModalHasher` mm-hash (blake3), the Qwen3-VL image processor (smart_resize + fused rescale/normalize + patchify -> `pixel_values`+`image_grid_thw`) and placeholder-token expansion, plus the `EncoderCacheManager` (+`ComputeMmEncoderBudget`) and the LMCache `extra_keys` seam. Additive `mm_features` carried on `Request`/`EngineCoreRequest`; with NO mm input every field is empty and every path is byte-identical to the text engine. Processor output is BIT/BYTE-identical to the vLLM 0.25.0 oracle (M0 fixture). Does NOT build the vision tower / embed-merge (M2). **SERVING wiring (ROAD-V1-MM `MM-SERVE-ENGINE`, 2026-07-28, `CLAIM-MM-SERVING-W2`):** the OpenAI server now carries the parsed `MultiModalInputs` into the engine — additive `LLMEngine`/`AsyncLLM` `add_request(MultiModalInputs)`+`generate(MultiModalInputs)` overloads via `InputProcessor::process_inputs_mm` (mirror `input_processor.py:333-379`, empty mm_features == the tokens path), the chat-template placeholder-STRING helpers (`get_placeholder_str`/`_add_placeholder` mirror), and the serving_chat `MultiModalChatFn` seam (default unset ⇒ text byte-identical). **SEAM BODY (ROAD-V1-MM `MM-SERVE-E2E` W3, 2026-07-28, `CLAIM-MM-SERVING-E2E`):** `MakeQwen3VLImageChatFn` (chat_mm.cpp) is the seam body the server sets — messages → marker-inject → chat template → `EncodeWithSpecialTokens` (the single image_pad marker → one image_token_id) → `RouteImageRgb` EXPAND to 196 image tokens + mm_features; wired in `examples/server/main.cpp` (guarded on `preprocessor_config.json`; text-only unset ⇒ byte-identical). Gated `test_chat_mm` 8/8 + `test_openai_serving` (seam invoked + routed). **ENGINE MM-FORWARD LANDED (ROAD-V1-MM `MM-SERVE-E2E`, 2026-07-28, `CLAIM-ENGINE-MM-FORWARD`):** the engine model runner now HAS an mm forward — `ModelForwardInput` gains an ADDITIVE default-nullopt `std::optional mm` (merged inputs_embeds + 3-D MRoPE positions + DeepStack, borrowed handles; nullopt-for-text ⇒ shared runner path byte-identical BY CONSTRUCTION), `Qwen3VLForConditionalGeneration` is `REGISTER_VLLM_MODEL`-registered (`qwen3_vl_registry.cpp`), and the registered forward FOLDS the M2c decode into `ModelRegistry::Forward` via the SHARED `Qwen3VLForwardStepLastLogits` (`Qwen3VLGenerateGreedyViaRegistry` drives every step through the registry). GPU token-exact gate `test_qwen3vl_registry_e2e` (image→text THROUGH `ModelRegistry::Forward` == M2c golden 32/32 STRICT, dgx.casa GB10); text inertness `test_runner` 16/16 + `test_scheduler` 36/36 + `test_model_registry` 24/24 + `test_chat_mm` 8/8 + `test_openai_serving` 41/41 all green. RESIDUAL: the FULL in-runner scheduler-fed tower run (batched-loop mm building the field from staged encoder outputs) + the real server `/v1/chat/completions` GPU e2e — recipe in `specs/mm-serving.md`. **INPUT LIMITS L1 LANDED (#607, 2026-08-13):** the per-modality `limit_per_prompt` + `GetLimitPerPrompt` precedence (`language_model_only` ⇒ 0 BEFORE the map, else the map, else 999) and the refusal that gives those numbers effect — `AllowedMmLimits` folding by `min()` against the model's own ceiling, `ValidateNumItems` with upstream's exact message, and both call sites with the `enable_mm_embeds` escape. NO serve surface and NO live call site: nothing constructs a `MultiModalConfig` on a request yet, which is L2's. **INPUT LIMITS L2 LANDED (#607, #686, 2026-08-14):** the flags (`--[no-]language-model-only`, `--limit-mm-per-prompt ''`; `arg_utils.py:555-556,1276-1279,1691-1692` over `ParseLimitMmPerPromptJson`, the port of `multimodal.py:212-236` + the DummyOptions dataclasses `:17-43`), the C-ABI fields (`vllm_model_params.language_model_only`/`.limit_mm_per_prompt`, **ABI v19**), and the LIVE CALL SITE: `ValidateChatMmLimits` (`chat_utils.py:648-662`) runs as step 0 of `MakeQwen3VLImageChatFn` over a `BaseProcessingInfo` folding `LoadedEngine::mm_config()` with `Qwen3VLChatSupportedMmLimits() == {"image": 1}` — the seam's own ceiling, which is the `min()` fold operand #686 recorded as undeclared. A three-image request is now HTTP 400 with upstream's message rather than an opaque 500 / a truncated answer. NOT claimed: any memory win (L3 gates tower construction, unmeasured). Still unwired: the `process_inputs_mm` call site (`context.py:461`), blocked on the per-model `get_supported_mm_limits()` hook. | T1 | `vllm/multimodal/{inputs.py,hasher.py:50,processing/processor.py:1663,processing/inputs.py:62}`; `vllm/model_executor/models/qwen3_vl.py:{1400,1233}`; `vllm/v1/core/encoder_cache_manager.py:17`; transformers `image_processing_qwen2_vl.py:62`, `image_processing_backends.py:327`; tests `tests/multimodal/test_processing.py`, `tests/multimodal/test_hasher.py`, `tests/v1/core/test_encoder_cache_manager.py` @ `e24d1b24` | `src/vllm/multimodal/hasher.cpp`, `src/vllm/multimodal/qwen3vl_processor.cpp`, `include/vllm/multimodal/{inputs.h,hasher.h,qwen3vl_processor.h}`; `src/vllm/v1/core/encoder_cache_manager.cpp` + `include/vllm/v1/core/encoder_cache_manager.h`; additive inert fields `include/vllm/v1/request.h` + `src/vllm/v1/request.cpp` + `include/vllm/v1/engine/types.h`; `extra_keys` seam `include/vllm/v1/kv_offload/lmcache/chunked_token_database.h` + `.cpp`; M0 `scripts/mm/m0_oracle_capture.py`; L1 limits `include/vllm/config/multimodal.h` + `include/vllm/multimodal/processing/context.h` + `src/vllm/multimodal/processing/context.cpp`, refusal type relocated to `include/vllm/v1/engine/validation_error.h`; L2 flags+ABI+call site `src/vllm/config/multimodal.cpp` (`ParseLimitMmPerPromptJson`) + `src/vllm/entrypoints/openai/server_main.cpp` + `include/vllm.h` (ABI v19) + `src/capi/vllm_c.cpp` + `EngineParams::multimodal`/`LoadedEngine::mm_config()` + `src/vllm/entrypoints/openai/chat_mm.cpp` (`ChatPartModality`, `ValidateChatMmLimits`, `Qwen3VLChatSupportedMmLimits`) — anchors `src/vllm/multimodal/hasher.cpp:56`, `src/vllm/config/multimodal.cpp:49`, `src/vllm/entrypoints/openai/chat_mm.cpp:295,311`, `include/vllm.h:197,403` | `tests/vllm/multimodal/test_qwen3vl_processor.cpp` (processor-parity 23/23 BIT-identical vs the M0 oracle fixture `tests/vllm/multimodal/fixtures/qwen3vl/`, RED-first: wrong normalize shift -> 1.2M mismatches); `tests/vllm/v1/core/test_encoder_cache_manager.cpp` 32/32. Text-inertness: `test_request`/`test_engine_types`/`test_lmcache_codec`/`test_lmcache_key_agreement`/`test_openai_conformance` all green standalone; SACRED CUDA 27B/35B/Coder = GPU inertness proof; `check-device-leakage` OK — anchor `tests/vllm/multimodal/test_qwen3vl_processor.cpp:59`. L1 limits: `tests/vllm/config/test_multimodal_config.cpp` 7/7 (21 assertions) + `tests/vllm/multimodal/test_processing_limits.cpp` 19/19 (78 assertions), porting `tests/multimodal/test_processing.py:902-941,944-985`, `tests/entrypoints/multimodal/llm/test_mm_embeds_only.py:41-49` and `tests/entrypoints/unit_tests/test_chat_utils.py:1498-1560` @ `5559679229bc`; mutations proven RED: map-before-flag precedence, the dropped throw, the dropped `min()` fold. L2 (aarch64 `build-test-cpu-arm64` lane, `-DVLLM_CPP_CUDA=OFF`): `tests/vllm/entrypoints/openai/test_serve_mm_limits.cpp` 10/10 (101 assertions, flags + the parser's upstream refusals) + `test_chat_mm` 11/11 (126) + `test_openai_api_server` 56/56 (638, the HTTP 400 arm proven against BOTH a 500 and a truncated 200); RED-first behavioural: `CHECK(500 == 400)`, `CHECK("InternalServerError" == BadRequestError)` and `CHECK(200 == 400)`; mutations proven RED: flag→config plumbing dropped, the call-site wiring dropped, and the refusal re-typed off `InputValidationError` (which lands as the 500 L1's design avoided) | [multimodal-track.md](specs/multimodal-track.md) §3 (M0/M1) | `READY` | - | | `ENG-MM-VISION-TOWER` | **Qwen3-VL vision tower `Qwen3_VisionTransformer` (M2a), proven faithful vs vLLM 0.25.0 in isolation.** The reusable vision half of the whole Qwen3-VL family + Qwen3.6 (27B/35B share this exact tower). Pure-additive C++ forward composed from public vt:: ops: patch-embed (Conv3d-as-matmul + bias), host pos-embed bilinear-interp+spatial-merge-reorder, 24 ViT blocks (LayerNorm + vision attention with partial-rotary NeoX vision RoPE via `vt::RopeFromCache` + non-causal `vt::Attention(causal=false)` + tanh-GELU MLP), patch merger (LayerNorm + exact-erf-GELU + 2 FCs), DeepStack 3 post-shuffle-norm mergers at layers 5/11/17 → `[196,10240]`. Adds 2 additive elementwise vt ops (`GeluTanh`/`GeluErf`). NO runner/model/registry edit → text engines byte-identical by construction. Proven faithful in ISOLATION; the merge into `input_embeds` + the MRoPE/DeepStack text backbone + the e2e image gate are M2b/M2c. | T1 | `vllm/model_executor/models/qwen3_vl.py` `Qwen3_VisionPatchEmbed:347`, `Qwen3_VisionBlock:413`, `Qwen3_VisionPatchMerger:467`, `Qwen3_VisionTransformer:519`, `forward:800`, `pos_embed_interpolate_native:277`, `rot_pos_emb:667`; `qwen2_5_vl.py::Qwen2_5_VisionAttention.forward:397`; `rotary_embedding/common.py::ApplyRotaryEmb.forward_static:151` @ `e24d1b24` | `src/vllm/model_executor/models/qwen3_vl_vision.{h,cpp}`; 2 vt ops `include/vt/ops.h` + `src/vt/ops.cpp` + `src/vt/cuda/cuda_layernorm.cu` + `src/vt/cpu/cpu_layernorm.cpp`; dumps `scripts/mm/m2a_tower_{ref,weight}_dump.py`; fixtures `tests/vllm/multimodal/fixtures/qwen3vl_tower/` | `tests/vllm/multimodal/test_qwen3vl_tower.cpp` — 4 RED-first tower gates vs the dumped vLLM-0.25.0 reference 348/348 (patch-embed 2.1e-3, block0 6.8e-3, merger 6.5e-2, DeepStack 1.2e-2/3.3e-2/4.4e-2, full tower 5.1e-2; pos-embed 2.5e-3 + rope 1.9e-3 TIGHT); bf16-depth envelope RCA'd; RED = rope disabled → block0 0.149/tower 0.75/6 fails; cutlass-ON+FA2 banner; clean `-Werror`; compute-sanitizer 0 — anchor `tests/vllm/multimodal/test_qwen3vl_tower.cpp:96` | [multimodal-track.md](specs/multimodal-track.md) §3 (M2a) | `ACTIVE` | `CLAIM-MULTIMODAL-M2A` | | `ENG-MM-TEXT-BACKBONE` | **Qwen3-VL text-backbone numeric contracts `Qwen3VLGetRopeIndex`/`Qwen3VLMergeMultimodal`/`Qwen3VLComputeDeepstack` (M2b/M2c), unit-green vs vLLM 0.25.0.** The deterministic pieces that fork the plain Qwen3-dense text path for a vision-conditioned decode: (1) MRoPE 3-D `get_rope_index` positions [3,T] (image tokens get (t,h,w) grid positions, text sequential); (2) the 3-section MRoPE APPLICATION — proven to be the EXISTING `vt::RopeFromCache` mrope path (positions [3,T] + `mrope_section=[24,20,20]` interleaved), faithful to `MRotaryEmbedding.forward_native` for Qwen3-VL's exact config; (3) `_compute_deepstack_embeds` scatter → [L,T,H] decoder-injection tensor; (4) `_merge_multimodal_embeddings` masked scatter of the tower's `[:,:2560]` into `input_embeds`. Pure-additive TU — NO shared dense forward / runner / registry edit → text engines byte-identical by construction. The e2e image forward (VL weight loader + forked MRoPE/DeepStack decode loop) is the remaining M2c wire-up. | T1 | `vllm/model_executor/models/qwen3_vl.py` `_get_mrope_input_positions:2567`, `_iter_mm_grid_hw:2482`, `_compute_deepstack_embeds:2761`, `Qwen3LLMModel.forward` deepstack `:1589`; `vllm/model_executor/models/utils.py::_merge_multimodal_embeddings:524`; `vllm/model_executor/layers/rotary_embedding/mrope.py` MRotaryEmbedding @ `e24d1b24` | `src/vllm/model_executor/models/qwen3_vl_text.{h,cpp}`; existing `vt::RopeFromCache` mrope path (`src/vt/{cpu,cuda}/*`); dump `scripts/mm/m2b_text_ref_dump.py`; fixtures `tests/vllm/multimodal/fixtures/qwen3vl_text/` — anchor `src/vllm/model_executor/models/qwen3_vl_text.cpp:9` | `tests/vllm/multimodal/test_qwen3vl_text.cpp` — 4 RED-first gates vs the dumped vLLM-0.25.0 reference 85/85 (get_rope_index BIT-exact [3,204], delta −182; MRoPE q rel-L2 1.5e-3 / k 1.5e-3, RED interleaved-off >5e-2; DeepStack + merge BIT-exact); CPU-only, no weights; clean CPU `-Werror` — anchor `tests/vllm/multimodal/test_qwen3vl_text.cpp:99` | [multimodal-track.md](specs/multimodal-track.md) §3 (M2b/M2c) | `ACTIVE` | `CLAIM-MULTIMODAL-M2BC` | | `ENG-MM-QWEN36-VL-FORWARD` | **Qwen3.6-27B (`Qwen3_5ForConditionalGeneration`) GDN-hybrid VL forward — IMAGE (M3-b) + VIDEO (M3d) BOTH e2e, STRICT gates PASS 32/32. Our own gate model's image+video paths now work end-to-end (speed pending).** The genuinely-new integration completing our own gate model's mm paths: fork the landed bf16 `Qwen3_5DenseModel` GDN-hybrid forward (48 GDN + 16 full-attn) on gated, default-off points so a text-only 27B request stays byte-identical — (a) `inputs_embeds` entry (embed ids + `Qwen3VLMergeMultimodal` scatter of the 27B tower merger `[N,5120]` into the visual-token rows; 27B has EMPTY `deepstack_visual_indexes` ⇒ NO DeepStack); (b) 3-section MRoPE (`mrope_section=[11,11,10]` interleaved, rotary_dim 64, theta 1e7) in the 16 full-attn layers only via the proven `vt::RopeFromCache` mrope path (GDN layers carry no rope); (c) mixed load = the M2a `Qwen3_VisionTransformer` (27B vision config, empty deepstack) bf16 tower + the bf16 GDN-hybrid LLM via the EXISTING `LoadQwen3_5Dense`. **M3d (2026-07-25) added VIDEO by REUSE:** the M3-b image driver refactored into a shared `VLGenerateCoreGdn`, image+video wrappers differ ONLY in the merge mask (`image_token` vs `video_token` across frames) + the get_rope_index (`Qwen3VLGetRopeIndex` vs `Qwen3VLGetRopeIndexVideo`); the M3c processor/windowed-tower/video-MRoPE are reused verbatim. | T1 | `vllm/model_executor/models/qwen3_5.py:389` (`Qwen3_5ForConditionalGeneration` subclasses `Qwen3VLForConditionalGeneration`; `visual = Qwen3_VisionTransformer`, modalities {"image","video"}); `qwen3_vl.py` `_process_video_input:2165`, `_get_mrope_input_positions:2567` video branch, `get_video_repl:1479`; the 27B `config.json` (`mrope_section=[11,11,10]`, empty `deepstack_visual_indexes`) @ `e24d1b24` / vLLM 0.25.0 | **M3-b + M3d BUILT + GATED 2026-07-25:** vision-only loader `LoadQwen3VLVisionWeights` (`src/vllm/model_executor/models/qwen3_vl.cpp`, 27B config) + shared `VLGenerateCoreGdn` + image driver `Qwen3_5VLGenerateGreedy` + **video driver `Qwen3_5VLGenerateGreedyVideo`** + `BuildMropeCosSinHost` + the `mrope_cos_sin` param on `DenseForwardLayers` (`src/vllm/model_executor/models/qwen3_5.cpp`, nullptr on every text caller ⇒ byte-identical; the video driver is purely additive, the shared text forward UNTOUCHED per `git diff --stat`) reusing M2a tower + `LoadQwen3_5Dense` bf16 LLM | **IMAGE:** golden `tests/vllm/multimodal/fixtures/qwen3_5_27b/` (STRICT sha256 `ead4b484…`); STRICT image gate PASS **32/32** (`test_qwen3_5_vl_e2e.cpp`, 54/54, re-run post-refactor). **VIDEO (M3d):** oracle `scripts/mm/m3d_video_oracle_capture.py` on the M3c synthetic clip (raw sha `8a111599…`, grid `[4,8,8]`, 64 video tokens) K=5 DETERMINISTIC ⇒ STRICT golden; **STRICT video gate PASS 32/32** (`test_qwen3_5_vl_video_e2e.cpp`, 27/27; near-tie gaps 0.0000 nats everywhere), fixtures `tests/vllm/multimodal/fixtures/qwen3_5_27b_video/`. Text-inertness 27B 235/235, 35B 315/315, Coder 138/138 (by construction); clean `-Werror` 0 warn; compute-sanitizer 0 on the 27B video forward. **SPEED MEASURED (2026-07-26, `CLAIM-MULTIMODAL-SPEED`): image c1 vs vLLM 0.25.0 GRAPHED — decode TPOT 225.0 ms/tok vs 226.9 = AT PARITY (0.99×), LLM prefill 326 ms vs vLLM TTFT 321 ms = at parity; vision tower WAS 2114 ms vs vLLM encode ≤~250 ms = ~10× (THE gap). TOWER LEVER EXECUTED (2026-07-26, `CLAIM-MULTIMODAL-SPEED-TOWER`, [multimodal-speed.md](specs/multimodal-speed.md) §7): nsys `cuda_gpu_kern_sum` attributed 98.9 % of the tower forward to the naive `vt::cuda::AttentionKernel` (56 ms/block; NOT QKV/FA2-routing); fixed by a warp-scoped online-softmax op `AttentionDenseFast` (separate op ⇒ `kAttention`/text byte-identical) + one-time resident-weight load ⇒ per-image tower 2114 → 148 ms (14.3×), **0.59× vs vLLM eager encode = FASTER**. STRICT image/video e2e HELD 32/32 (+4B DeepStack 32/32), `test_ops_attention` 37239/37239, 27B text SACRED 235/235, compute-sanitizer memcheck 0, clean `-Werror`. `benchmark_binding=false`, single-seq driver (no c2+/server). Remaining: batched/graphed mm serving (c2+) + audio our-side — DONE bar not yet met.** | [multimodal-track.md](specs/multimodal-track.md) §M3 + [multimodal-speed.md](specs/multimodal-speed.md) §7 + §8 (decode lever #2 CLOSED 2026-07-27: on-GPU greedy argmax + decode embed round-trip removed on `VLGenerateCoreGdn`; bit-exact — image/video STRICT 32/32 held; 27B decode NEUTRAL at the ~222 ms bandwidth floor) + §9 (lever #3 FIRST BRICK 2026-07-27, `CLAIM-MULTIMODAL-SPEED-GRAPH`: the shared `VLGenerateCoreGdn` decode step now routes through the production `Qwen3_5DenseDecodeGraph` cold→warm→replay captured decode — the mm decode is now GRAPH-CAPTURABLE, closing the un-graphed-eager-loop structural gap; S==B==1 bit-identical rebuild; token-exact HELD image/video STRICT 32/32 with 30 graph replays confirmed; A/B graphed 232.5 vs eager 233.4 ms/tok = NEUTRAL at the 27B bandwidth floor; the launch-overhead win + batched c2+ + serving ingestion are the recorded W-plan W1-W3) + §16 (vision-forward flash kernel 2026-07-28, `CLAIM-MM-SPEED-QWEN-IMAGE`: ATTRIBUTION-FIRST nsys attributed ~85% of the 148 ms tower forward to the dense attention `AttentionWarpKernel` [4.66 ms/block×27]; routed it to the §14 flash-tiled `vt::AttentionDenseFlash` [head_dim 72, byte-identical — per-warp math verbatim, only K/V from shared-mem tiles]. STRICT image/video e2e HELD 32/32 [27B+4B], `test_ops_attention` 37239/37239, goldens md5 UNCHANGED, nsys proof `AttentionDenseFlashKernel` 24 inst/zero warp, RED 30/46→46/46, sanitizer 0. A/B warp 148.3→flash 142.3 ms = 1.04× — the profile REFUTED a big lever: at t=784 the vision attention is serial-latency-bound not bandwidth-bound [audio §14 was 1.82× at t=1500], flash recovers only ~6 ms. **HONEST: the tower ALREADY BEATS vLLM — 142 ms vs ~250 ms eager encode = 0.57×**; image/video mm-forward is correctness-DONE + speed-BEATS-vLLM; residual = tensor-core MMA hd-72 attention [not needed for parity] + batched c2+/serving) | `ACTIVE` | `CLAIM-MULTIMODAL-SPEED-TOWER` + `CLAIM-MULTIMODAL-SPEED-DECODE` + `CLAIM-MULTIMODAL-SPEED-GRAPH` + `CLAIM-MM-SPEED-QWEN-IMAGE` | diff --git a/.agents/specs/mm-serving.md b/.agents/specs/mm-serving.md index 7b8c44936..305806dfa 100644 --- a/.agents/specs/mm-serving.md +++ b/.agents/specs/mm-serving.md @@ -150,7 +150,8 @@ chat request content-part array [MM-SERVE-PARSE — CPU, THIS BRICK] validate-then-build seam shape (pinned as its own leg in `test_api_server.cpp`), so the issue's diagnosis — first image, `break`, no refusal — stands; only its stated consequence was understated. Both cases are - now the same answer: `ValidateChatMmLimits` runs first and a request over the + now the same answer: `ValidateChatMmLimits` (`chat_mm.cpp:311`, ahead of the + unchanged first-image loop at `:313-326`) runs first, and a request over the seam's declared ceiling (`Qwen3VLChatSupportedMmLimits() == {"image": 1}`, the `min()` fold operand the issue asked for) is refused with **HTTP 400 "At most 1 image(s) may be provided in one prompt."** Video and diff --git a/tests/vllm/entrypoints/openai/test_chat_mm.cpp b/tests/vllm/entrypoints/openai/test_chat_mm.cpp index e54cab406..9c1b67cc7 100644 --- a/tests/vllm/entrypoints/openai/test_chat_mm.cpp +++ b/tests/vllm/entrypoints/openai/test_chat_mm.cpp @@ -571,7 +571,9 @@ TEST_CASE("chat-mm seam body: MakeQwen3VLImageChatFn -> expanded engine input") // // RED line, and it is a BEHAVIOURAL one rather than an absence: before this // wave the seam located the FIRST image_url part and `break`ed -// (chat_mm.cpp:256-268), so a three-image request was SERVED WITH ONE — no +// (PRE-L2 chat_mm.cpp:256-268; the same loop is chat_mm.cpp:313-323 today, +// now preceded by the check at :311), so a three-image request was neither +// served nor refused, it was quietly reduced to one — no // error, no warning, a confident answer about a subset of the input (#686). // Every CHECK_THROWS below fails against that code, because it does not // throw; it returns the first image's 196 tokens and discards two. From e8e6c891ac5593489a92d626ade8fdf85b3f292e Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Fri, 14 Aug 2026 13:42:13 +0000 Subject: [PATCH 3/3] fix(ENG-MM-INPUT-PIPELINE): the v19 ABI claimed an enforcement the C ABI cannot reach, and refused a document vLLM accepts (#607, #686, #758) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review repairs for #749. The L2 deliverable itself is untouched — three surfaces wired, the e2e 400 proven, seven mutations caught. What changed is one false claim on a PERMANENT contract, one real mirror divergence, and three citations. ## 1. include/vllm.h said the v19 fields are "ENFORCED, not recorded" They are not, on the surface that header describes. Verified three ways at this head: `grep -c set_multimodal_chat_fn src/capi/vllm_c.cpp` is 0; the seam is installed exactly once, at `server_main.cpp:1117`; and `serving_chat.cpp:677` gates the whole multimodal branch on `if (mm_chat_fn_)`. So on a C-ABI engine `mm_inputs` stays empty, `ValidateChatMmLimits` never runs, an `image_url` content part is dropped, and `language_model_only=1` changes nothing a `vllm_chat` caller can observe. `test_capi`'s existing case only asserted the config LANDED. Remediation (a) of the two offered: correct the wording. (b) — adding ABI-level enforcement — means building a C-ABI multimodal REQUEST path, which is a new capability and not L2's scope; the header now says that in as many words. The version log and the field block state where enforcement lives (the OpenAI-server path, the one caller that installs the seam), that this ABI has no multimodal request path yet, and exactly what a C-ABI caller gets today: the part parses, is dropped, and the answer is text. So the claim cannot silently become false again, it is PINNED behaviourally, not just reworded: a new `test_capi` case builds a `language_model_only` engine, sends a real `image_url` body through `vllm_chat`, and asserts `VLLM_OK` + a `chat.completion` + no `error` — the opposite of the HTTP 400 the same config produces on the server. Wire the seam into the ABI and it goes red. This is the failure #685's review named — inheriting an unproven end-to-end claim — restated on a contract that cannot be revised later. ## 2. `IsKnownOption` refused unknown option keys for EVERY modality Upstream's `BaseDummyOptions` (`multimodal.py:17-21`), which `_validate_limit_per_prompt`'s `else` at `:233` builds for any modality outside image/video/audio, is the ONE dummy-options dataclass declared WITHOUT `config=ConfigDict(extra="forbid")` — unlike `:24,33,41`. Re-derived rather than taken on trust: I fetched `multimodal.py` at the pinned `5559679229bc`, read the four declarations, and ran them under the same pydantic (2.12.5) the oracle has. BaseDummyOptions(**{"count": 2, "foo": 3}) -> BaseDummyOptions(count=2) ImageDummyOptions(**{"count": 2, "foo": 3}) -> ValidationError BaseDummyOptions(**{"count": -1}) -> ValidationError So `'{"pointcloud": {"count": 2, "foo": 3}}'` is accepted upstream and was refused here, and the code comment asserted the opposite reasoning. Now mirrored: a non-builtin modality's unlisted key is dropped (and ANNOUNCED, since we announce every drop) with its value unvalidated, while the three builtin modalities keep `extra="forbid"` and `Field(None, gt=0)`, and `count` — declared on BaseDummyOptions itself — stays validated for all of them. Pinned by a new `test_serve_mm_limits` case covering both halves, so the repair cannot decay into "accept everything". The `{"tactile": {"count": 1, "width": 2}}` row of the refusal list, which encoded the divergence, is gone. No ABI version bump: v19 is unreleased, landing in this same PR. ## 3-5. Citations and one rendering - `arg_utils.py:375-377` was the `union_dict_and_str` branch. `limit_per_prompt` takes the plain-`dict` branch, `parse_type(json.loads)`, at `:379-381` — `dict[str, BaseDummyOptions]` has no `str` arm and `dict[...]` reports `__module__ == "builtins"`, so `is_not_builtin` is False and `:374-378` is not taken. Fixed in all five places it appeared (the reviewer found four; the fifth is `include/vllm/config/multimodal.h:97`), plus the PR body. - `get_supported_mm_limits` is not on `Qwen3VLProcessingInfo`. It is inherited from `Qwen2VLProcessingInfo` at `qwen2_vl.py:851-852` (`{"image": None, "video": None}`); `qwen3_vl.py:848` subclasses it. - The `chat_mm.h` block claimed the `{"image": 1}` ceiling satisfies AGENTS.md's "an unimplemented arm is refused with a message naming the missing piece". It does not: the message is upstream's generic "At most 0 video(s) may be provided in one prompt.", indistinguishable from a configured limit. The claim is withdrawn and the behaviour is owed to #758, filed for it — naming the arm diverges from a verbatim-ported message three suites assert byte-for-byte, so it takes its own spec rather than a review repair. - While re-deriving, the dataclass anchors themselves were off by one to four lines (`:20` for `count`, `:23,32,39` for the forbid decorators, `:26-28,...` for the option fields, `:17-43` for the block). Corrected everywhere against the lines actually read. - `docs/USAGE.md`'s flag table rendered the hint without the backticks `context.py:426` carries; the console example below it had them. Both now match the real string. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --- .agents/engine-matrix.md | 2 +- .agents/roadmap_v1.md | 3 +- .agents/specs/multimodal-track.md | 75 +++++++++++++--- docs/USAGE.md | 25 ++++-- include/vllm.h | 65 ++++++++++---- include/vllm/config/multimodal.h | 48 +++++++---- include/vllm/entrypoints/openai/chat_mm.h | 33 ++++--- src/vllm/config/multimodal.cpp | 41 +++++++-- src/vllm/entrypoints/openai/server_main.cpp | 13 ++- tests/capi/test_capi.cpp | 62 ++++++++++++- .../openai/test_serve_mm_limits.cpp | 86 ++++++++++++++++--- 11 files changed, 361 insertions(+), 92 deletions(-) diff --git a/.agents/engine-matrix.md b/.agents/engine-matrix.md index 64bbd9a9d..7d570f0ba 100644 --- a/.agents/engine-matrix.md +++ b/.agents/engine-matrix.md @@ -71,7 +71,7 @@ forensics: roadmap_v1.md and the parity ledger. | `ENG-DBO-UBATCH` | DBO and ubatch overlap | T2 | `vllm/config/parallel.py:208,524` | - | - | `planned: specs/dbo-ubatch.md` | `INVENTORIED` | - | | `ENG-MOE-SHARED-AUX` | MoE shared-expert MLP on an aux CUDA stream concurrent with the routed-expert router/align/grouped-GEMMs (mirror vLLM's decode overlap; the largest remaining 35B c1/c2 engine lever). Fork the shared MLP onto a 2nd persistent per-device stream, join before the combine → byte-identical to serial (independent shared/routed paths both complete before combine; overlap changes WHEN not WHAT). Gated `T <= threshold` decode + CUDA. The aux stream draws scratch from a SEPARATE `AuxPool` so the concurrent main-stream routed allocations never share a live block with it (the `DevicePool` reuse invariant is single-stream ordering; vLLM sidesteps this with its stream-aware caching allocator's `record_stream`). `VT_MOE_SHARED_AUX_STREAM` **DEFAULT ON** (`=0` rollback); `VT_MOE_SHARED_AUX_THRESHOLD` (default 128; GB10 48-SM calibration). Captured in the decode CUDA-graph via the fork/join event edges (`ThreadLocal` capture, no abort). Only the committed Marlin MoE decode path; wmma fallback/CPU/GGUF and 27B dense unaffected | T1 | `vllm/model_executor/layers/fused_moe/runner/shared_experts.py:99-104,125-142`; `vllm/utils/multi_stream_utils.py:20-58` (`maybe_execute_in_parallel`, TRT-LLM port); `vllm/utils/torch_utils.py:736-756` (`aux_stream`); `vllm/envs.py:260` (threshold 256) | fork/join `src/vllm/model_executor/models/qwen3_5.cpp:3999,4114` (`MoeBlockFusedMarlinCuda`); aux stream+events `src/vllm/model_executor/models/qwen3_5.cpp:3575,3581` (`MoeAuxStream`/`MoeAuxStreamFor`); predicates `:3553,3560`; aux-pool isolation `:496,3538` (`AuxPool`/`ActivePool`/`ActivePoolScope`) + `DBuf pool_` routing `:645` | **DGX (prod flags, one flock):** overlap ON==OFF BYTE-IDENTICAL — `tests/parity/test_qwen36_paged_engine.cpp:116` 35B **315/315** + `tests/parity/test_qwen27_paged_engine.cpp` 27B **235/235** under `VT_MOE_SHARED_AUX_STREAM`∈{0,1}; captured-vs-eager (`VLLM_CPP_CUDAGRAPH=0`, ON) 315/315; shipping default (no env) 315/315+235/235, rollback `=0` 315/315+235/235; `compute-sanitizer memcheck` (default ON, captured) 0 errors; in-situ interleaved TPOT A/B (drop cold rep1) c1 −5.6% / c2 −2.7% / c4 −3.7% / c8 −3.4% / c16 −1.6% / c32 −1.5% (WINS every conc, zero regression); ledger [parity-ledger.md](parity-ledger.md) 2026-07-19 row | [moe-shared-aux-stream.md](specs/moe-shared-aux-stream.md) | `ANCHOR-BACKFILL` | `CLAIM-MOE-SHARED-AUX-1` | | `ENG-RUNNER-MODELSHAPE` | **Runner is model-shape-agnostic over the KV-cache group structure** — the extensibility deliverable the first additive-model bring-up (Qwen3 dense) forced. Before W1 the `GPUModelRunner` had only ever executed the Qwen3.6 HYBRID topology and hardcoded it in two places: (#1) the KV-buffer alloc loop indexed `config_.layer_types[l]`, out-of-bounds on a pure-dense model's EMPTY `layer_types`; (#2) each `execute_model` step unconditionally built the GDN metadata (`gather_block_table(gdn_group_id_)` / `remap_gdn_state_slots` / `GDNAttentionMetadataBuilder`), which reads `block_table[-1]` when there is no mamba group. W1 drives both off the resolved KV-group structure — a model-agnostic `has_mamba_group` / `gdn_group_id_ >= 0` predicate (NOT a model-name check): empty/absent `layer_types` ⇒ all full-attention; no mamba group ⇒ the whole GDN metadata/state path is skipped and `gdn_meta` stays default-empty. A full-attention-only KV config (one FA group, no MambaSpec) now allocates + steps cleanly; the hybrid gate models keep their GDN group so their path is BYTE-IDENTICAL. This is a one-time generalization: every future dense/non-hybrid arch (Llama, Mistral) now adds new-files-only, zero further runner edits. **PER-LAYER KV head_dim extension (Gemma-4 G1b, 2026-07-28, `CLAIM-GEMMA4-G1B`):** the runner's full-attn alloc/view loops now consume an OPTIONAL `KVCacheConfig::per_layer_attn_specs` (index == layer) so a HETEROGENEOUS-head_dim model (Gemma-4: sliding 256 / global 512, same num_kv_heads) sizes each non-GDN layer's paged KV + PagedKvCache view from its OWN spec. The field is EMPTY for every uniform-KV model ⇒ the loop collapses to the single group spec ⇒ byte-identical allocation/view/indexing/dispatch (same additive-identical property as the model-shape generalization above). Block table / KV manager / scheduler stay head_dim-independent (num_blocks + block_size, uniform) so no per-group block table is introduced | T0 | model-agnostic runner drives off `kv_cache_config.kv_cache_groups` — `vllm/v1/worker/gpu/model_runner.py` `initialize_kv_cache` / attention-metadata build (per-group, no hardcoded hybrid) @ `e24d1b24` | `src/vllm/v1/worker/gpu/runner.cpp:458-470` (alloc loop: `has_mamba_group && !layer_types.empty()` gate) + `:651-680` (GDN metadata build gated on `gdn_group_id_ >= 0`, default-empty `gdn_meta` otherwise); per-layer KV head_dim: `include/vllm/v1/kv_cache_interface.h` (`KVCacheConfig::per_layer_attn_specs`) consumed in `src/vllm/v1/worker/gpu/runner.cpp` `initialize_kv_cache` (per-layer `FaDims` alloc+view), published by `src/vllm/model_executor/models/gemma4_registry.cpp` (`MakeGemma4ForConditionalGenerationKVCache`); the full-attention-only KV spec that exercises the base path `src/vllm/model_executor/models/qwen3_dense.cpp` (`MakeQwen3ForCausalLMKVCache`) | `tests/vllm/v1/worker/test_runner.cpp:1129` — "full-attention-only KV config allocates without the GDN path" + "full-attention-only step skips GDN metadata build (no OOB)" (RED→GREEN: both SIGSEGV pre-generalization; GREEN post). Behaviour-preservation gate: DGX **27B 235/235 + 35B 315/315 UNCHANGED** under the fix; per-layer-KV inertness: full CPU runner/KV suite green + **OLMo-2 SACRED GPU re-gate 16/16 UNCHANGED**; heterogeneous path proven by **Gemma-4 E4B STRICT 32/32** (`tests/parity/test_gemma4_paged_engine.cpp`); ASan/UBSan clean on the affected paths | [first-additive-model-qwen3-dense.md](specs/first-additive-model-qwen3-dense.md) §3 (seam gaps #1/#2), §6 (W1); [gemma4-multimodal.md](specs/gemma4-multimodal.md) §G1b | `ACTIVE` | `CLAIM-MODEL-QWEN3-DENSE` | -| `ENG-MM-INPUT-PIPELINE` | **Multimodal INPUT pipeline + encoder-cache engine seam (M1), INERT when no mm input.** The C++ mirror of `vllm/multimodal/`: `MultiModalKwargs`/`MultiModalFeatureSpec`/`MultiModalInputs`, the `MultiModalHasher` mm-hash (blake3), the Qwen3-VL image processor (smart_resize + fused rescale/normalize + patchify -> `pixel_values`+`image_grid_thw`) and placeholder-token expansion, plus the `EncoderCacheManager` (+`ComputeMmEncoderBudget`) and the LMCache `extra_keys` seam. Additive `mm_features` carried on `Request`/`EngineCoreRequest`; with NO mm input every field is empty and every path is byte-identical to the text engine. Processor output is BIT/BYTE-identical to the vLLM 0.25.0 oracle (M0 fixture). Does NOT build the vision tower / embed-merge (M2). **SERVING wiring (ROAD-V1-MM `MM-SERVE-ENGINE`, 2026-07-28, `CLAIM-MM-SERVING-W2`):** the OpenAI server now carries the parsed `MultiModalInputs` into the engine — additive `LLMEngine`/`AsyncLLM` `add_request(MultiModalInputs)`+`generate(MultiModalInputs)` overloads via `InputProcessor::process_inputs_mm` (mirror `input_processor.py:333-379`, empty mm_features == the tokens path), the chat-template placeholder-STRING helpers (`get_placeholder_str`/`_add_placeholder` mirror), and the serving_chat `MultiModalChatFn` seam (default unset ⇒ text byte-identical). **SEAM BODY (ROAD-V1-MM `MM-SERVE-E2E` W3, 2026-07-28, `CLAIM-MM-SERVING-E2E`):** `MakeQwen3VLImageChatFn` (chat_mm.cpp) is the seam body the server sets — messages → marker-inject → chat template → `EncodeWithSpecialTokens` (the single image_pad marker → one image_token_id) → `RouteImageRgb` EXPAND to 196 image tokens + mm_features; wired in `examples/server/main.cpp` (guarded on `preprocessor_config.json`; text-only unset ⇒ byte-identical). Gated `test_chat_mm` 8/8 + `test_openai_serving` (seam invoked + routed). **ENGINE MM-FORWARD LANDED (ROAD-V1-MM `MM-SERVE-E2E`, 2026-07-28, `CLAIM-ENGINE-MM-FORWARD`):** the engine model runner now HAS an mm forward — `ModelForwardInput` gains an ADDITIVE default-nullopt `std::optional mm` (merged inputs_embeds + 3-D MRoPE positions + DeepStack, borrowed handles; nullopt-for-text ⇒ shared runner path byte-identical BY CONSTRUCTION), `Qwen3VLForConditionalGeneration` is `REGISTER_VLLM_MODEL`-registered (`qwen3_vl_registry.cpp`), and the registered forward FOLDS the M2c decode into `ModelRegistry::Forward` via the SHARED `Qwen3VLForwardStepLastLogits` (`Qwen3VLGenerateGreedyViaRegistry` drives every step through the registry). GPU token-exact gate `test_qwen3vl_registry_e2e` (image→text THROUGH `ModelRegistry::Forward` == M2c golden 32/32 STRICT, dgx.casa GB10); text inertness `test_runner` 16/16 + `test_scheduler` 36/36 + `test_model_registry` 24/24 + `test_chat_mm` 8/8 + `test_openai_serving` 41/41 all green. RESIDUAL: the FULL in-runner scheduler-fed tower run (batched-loop mm building the field from staged encoder outputs) + the real server `/v1/chat/completions` GPU e2e — recipe in `specs/mm-serving.md`. **INPUT LIMITS L1 LANDED (#607, 2026-08-13):** the per-modality `limit_per_prompt` + `GetLimitPerPrompt` precedence (`language_model_only` ⇒ 0 BEFORE the map, else the map, else 999) and the refusal that gives those numbers effect — `AllowedMmLimits` folding by `min()` against the model's own ceiling, `ValidateNumItems` with upstream's exact message, and both call sites with the `enable_mm_embeds` escape. NO serve surface and NO live call site: nothing constructs a `MultiModalConfig` on a request yet, which is L2's. **INPUT LIMITS L2 LANDED (#607, #686, 2026-08-14):** the flags (`--[no-]language-model-only`, `--limit-mm-per-prompt ''`; `arg_utils.py:555-556,1276-1279,1691-1692` over `ParseLimitMmPerPromptJson`, the port of `multimodal.py:212-236` + the DummyOptions dataclasses `:17-43`), the C-ABI fields (`vllm_model_params.language_model_only`/`.limit_mm_per_prompt`, **ABI v19**), and the LIVE CALL SITE: `ValidateChatMmLimits` (`chat_utils.py:648-662`) runs as step 0 of `MakeQwen3VLImageChatFn` over a `BaseProcessingInfo` folding `LoadedEngine::mm_config()` with `Qwen3VLChatSupportedMmLimits() == {"image": 1}` — the seam's own ceiling, which is the `min()` fold operand #686 recorded as undeclared. A three-image request is now HTTP 400 with upstream's message rather than an opaque 500 / a truncated answer. NOT claimed: any memory win (L3 gates tower construction, unmeasured). Still unwired: the `process_inputs_mm` call site (`context.py:461`), blocked on the per-model `get_supported_mm_limits()` hook. | T1 | `vllm/multimodal/{inputs.py,hasher.py:50,processing/processor.py:1663,processing/inputs.py:62}`; `vllm/model_executor/models/qwen3_vl.py:{1400,1233}`; `vllm/v1/core/encoder_cache_manager.py:17`; transformers `image_processing_qwen2_vl.py:62`, `image_processing_backends.py:327`; tests `tests/multimodal/test_processing.py`, `tests/multimodal/test_hasher.py`, `tests/v1/core/test_encoder_cache_manager.py` @ `e24d1b24` | `src/vllm/multimodal/hasher.cpp`, `src/vllm/multimodal/qwen3vl_processor.cpp`, `include/vllm/multimodal/{inputs.h,hasher.h,qwen3vl_processor.h}`; `src/vllm/v1/core/encoder_cache_manager.cpp` + `include/vllm/v1/core/encoder_cache_manager.h`; additive inert fields `include/vllm/v1/request.h` + `src/vllm/v1/request.cpp` + `include/vllm/v1/engine/types.h`; `extra_keys` seam `include/vllm/v1/kv_offload/lmcache/chunked_token_database.h` + `.cpp`; M0 `scripts/mm/m0_oracle_capture.py`; L1 limits `include/vllm/config/multimodal.h` + `include/vllm/multimodal/processing/context.h` + `src/vllm/multimodal/processing/context.cpp`, refusal type relocated to `include/vllm/v1/engine/validation_error.h`; L2 flags+ABI+call site `src/vllm/config/multimodal.cpp` (`ParseLimitMmPerPromptJson`) + `src/vllm/entrypoints/openai/server_main.cpp` + `include/vllm.h` (ABI v19) + `src/capi/vllm_c.cpp` + `EngineParams::multimodal`/`LoadedEngine::mm_config()` + `src/vllm/entrypoints/openai/chat_mm.cpp` (`ChatPartModality`, `ValidateChatMmLimits`, `Qwen3VLChatSupportedMmLimits`) — anchors `src/vllm/multimodal/hasher.cpp:56`, `src/vllm/config/multimodal.cpp:49`, `src/vllm/entrypoints/openai/chat_mm.cpp:295,311`, `include/vllm.h:197,403` | `tests/vllm/multimodal/test_qwen3vl_processor.cpp` (processor-parity 23/23 BIT-identical vs the M0 oracle fixture `tests/vllm/multimodal/fixtures/qwen3vl/`, RED-first: wrong normalize shift -> 1.2M mismatches); `tests/vllm/v1/core/test_encoder_cache_manager.cpp` 32/32. Text-inertness: `test_request`/`test_engine_types`/`test_lmcache_codec`/`test_lmcache_key_agreement`/`test_openai_conformance` all green standalone; SACRED CUDA 27B/35B/Coder = GPU inertness proof; `check-device-leakage` OK — anchor `tests/vllm/multimodal/test_qwen3vl_processor.cpp:59`. L1 limits: `tests/vllm/config/test_multimodal_config.cpp` 7/7 (21 assertions) + `tests/vllm/multimodal/test_processing_limits.cpp` 19/19 (78 assertions), porting `tests/multimodal/test_processing.py:902-941,944-985`, `tests/entrypoints/multimodal/llm/test_mm_embeds_only.py:41-49` and `tests/entrypoints/unit_tests/test_chat_utils.py:1498-1560` @ `5559679229bc`; mutations proven RED: map-before-flag precedence, the dropped throw, the dropped `min()` fold. L2 (aarch64 `build-test-cpu-arm64` lane, `-DVLLM_CPP_CUDA=OFF`): `tests/vllm/entrypoints/openai/test_serve_mm_limits.cpp` 10/10 (101 assertions, flags + the parser's upstream refusals) + `test_chat_mm` 11/11 (126) + `test_openai_api_server` 56/56 (638, the HTTP 400 arm proven against BOTH a 500 and a truncated 200); RED-first behavioural: `CHECK(500 == 400)`, `CHECK("InternalServerError" == BadRequestError)` and `CHECK(200 == 400)`; mutations proven RED: flag→config plumbing dropped, the call-site wiring dropped, and the refusal re-typed off `InputValidationError` (which lands as the 500 L1's design avoided) | [multimodal-track.md](specs/multimodal-track.md) §3 (M0/M1) | `READY` | - | +| `ENG-MM-INPUT-PIPELINE` | **Multimodal INPUT pipeline + encoder-cache engine seam (M1), INERT when no mm input.** The C++ mirror of `vllm/multimodal/`: `MultiModalKwargs`/`MultiModalFeatureSpec`/`MultiModalInputs`, the `MultiModalHasher` mm-hash (blake3), the Qwen3-VL image processor (smart_resize + fused rescale/normalize + patchify -> `pixel_values`+`image_grid_thw`) and placeholder-token expansion, plus the `EncoderCacheManager` (+`ComputeMmEncoderBudget`) and the LMCache `extra_keys` seam. Additive `mm_features` carried on `Request`/`EngineCoreRequest`; with NO mm input every field is empty and every path is byte-identical to the text engine. Processor output is BIT/BYTE-identical to the vLLM 0.25.0 oracle (M0 fixture). Does NOT build the vision tower / embed-merge (M2). **SERVING wiring (ROAD-V1-MM `MM-SERVE-ENGINE`, 2026-07-28, `CLAIM-MM-SERVING-W2`):** the OpenAI server now carries the parsed `MultiModalInputs` into the engine — additive `LLMEngine`/`AsyncLLM` `add_request(MultiModalInputs)`+`generate(MultiModalInputs)` overloads via `InputProcessor::process_inputs_mm` (mirror `input_processor.py:333-379`, empty mm_features == the tokens path), the chat-template placeholder-STRING helpers (`get_placeholder_str`/`_add_placeholder` mirror), and the serving_chat `MultiModalChatFn` seam (default unset ⇒ text byte-identical). **SEAM BODY (ROAD-V1-MM `MM-SERVE-E2E` W3, 2026-07-28, `CLAIM-MM-SERVING-E2E`):** `MakeQwen3VLImageChatFn` (chat_mm.cpp) is the seam body the server sets — messages → marker-inject → chat template → `EncodeWithSpecialTokens` (the single image_pad marker → one image_token_id) → `RouteImageRgb` EXPAND to 196 image tokens + mm_features; wired in `examples/server/main.cpp` (guarded on `preprocessor_config.json`; text-only unset ⇒ byte-identical). Gated `test_chat_mm` 8/8 + `test_openai_serving` (seam invoked + routed). **ENGINE MM-FORWARD LANDED (ROAD-V1-MM `MM-SERVE-E2E`, 2026-07-28, `CLAIM-ENGINE-MM-FORWARD`):** the engine model runner now HAS an mm forward — `ModelForwardInput` gains an ADDITIVE default-nullopt `std::optional mm` (merged inputs_embeds + 3-D MRoPE positions + DeepStack, borrowed handles; nullopt-for-text ⇒ shared runner path byte-identical BY CONSTRUCTION), `Qwen3VLForConditionalGeneration` is `REGISTER_VLLM_MODEL`-registered (`qwen3_vl_registry.cpp`), and the registered forward FOLDS the M2c decode into `ModelRegistry::Forward` via the SHARED `Qwen3VLForwardStepLastLogits` (`Qwen3VLGenerateGreedyViaRegistry` drives every step through the registry). GPU token-exact gate `test_qwen3vl_registry_e2e` (image→text THROUGH `ModelRegistry::Forward` == M2c golden 32/32 STRICT, dgx.casa GB10); text inertness `test_runner` 16/16 + `test_scheduler` 36/36 + `test_model_registry` 24/24 + `test_chat_mm` 8/8 + `test_openai_serving` 41/41 all green. RESIDUAL: the FULL in-runner scheduler-fed tower run (batched-loop mm building the field from staged encoder outputs) + the real server `/v1/chat/completions` GPU e2e — recipe in `specs/mm-serving.md`. **INPUT LIMITS L1 LANDED (#607, 2026-08-13):** the per-modality `limit_per_prompt` + `GetLimitPerPrompt` precedence (`language_model_only` ⇒ 0 BEFORE the map, else the map, else 999) and the refusal that gives those numbers effect — `AllowedMmLimits` folding by `min()` against the model's own ceiling, `ValidateNumItems` with upstream's exact message, and both call sites with the `enable_mm_embeds` escape. NO serve surface and NO live call site: nothing constructs a `MultiModalConfig` on a request yet, which is L2's. **INPUT LIMITS L2 LANDED (#607, #686, 2026-08-14):** the flags (`--[no-]language-model-only`, `--limit-mm-per-prompt ''`; `arg_utils.py:555-556,1276-1279,1691-1692` over `ParseLimitMmPerPromptJson`, the port of `multimodal.py:212-236` + the DummyOptions dataclasses `:17-43`), the C-ABI fields (`vllm_model_params.language_model_only`/`.limit_mm_per_prompt`, **ABI v19**), and the LIVE CALL SITE: `ValidateChatMmLimits` (`chat_utils.py:648-662`) runs as step 0 of `MakeQwen3VLImageChatFn` over a `BaseProcessingInfo` folding `LoadedEngine::mm_config()` with `Qwen3VLChatSupportedMmLimits() == {"image": 1}` — the seam's own ceiling, which is the `min()` fold operand #686 recorded as undeclared. A three-image request is now HTTP 400 with upstream's message rather than an opaque 500 / a truncated answer. NOT claimed: any memory win (L3 gates tower construction, unmeasured). Still unwired: the `process_inputs_mm` call site (`context.py:461`), blocked on the per-model `get_supported_mm_limits()` hook. | T1 | `vllm/multimodal/{inputs.py,hasher.py:50,processing/processor.py:1663,processing/inputs.py:62}`; `vllm/model_executor/models/qwen3_vl.py:{1400,1233}`; `vllm/v1/core/encoder_cache_manager.py:17`; transformers `image_processing_qwen2_vl.py:62`, `image_processing_backends.py:327`; tests `tests/multimodal/test_processing.py`, `tests/multimodal/test_hasher.py`, `tests/v1/core/test_encoder_cache_manager.py` @ `e24d1b24` | `src/vllm/multimodal/hasher.cpp`, `src/vllm/multimodal/qwen3vl_processor.cpp`, `include/vllm/multimodal/{inputs.h,hasher.h,qwen3vl_processor.h}`; `src/vllm/v1/core/encoder_cache_manager.cpp` + `include/vllm/v1/core/encoder_cache_manager.h`; additive inert fields `include/vllm/v1/request.h` + `src/vllm/v1/request.cpp` + `include/vllm/v1/engine/types.h`; `extra_keys` seam `include/vllm/v1/kv_offload/lmcache/chunked_token_database.h` + `.cpp`; M0 `scripts/mm/m0_oracle_capture.py`; L1 limits `include/vllm/config/multimodal.h` + `include/vllm/multimodal/processing/context.h` + `src/vllm/multimodal/processing/context.cpp`, refusal type relocated to `include/vllm/v1/engine/validation_error.h`; L2 flags+ABI+call site `src/vllm/config/multimodal.cpp` (`ParseLimitMmPerPromptJson`) + `src/vllm/entrypoints/openai/server_main.cpp` + `include/vllm.h` (ABI v19) + `src/capi/vllm_c.cpp` + `EngineParams::multimodal`/`LoadedEngine::mm_config()` + `src/vllm/entrypoints/openai/chat_mm.cpp` (`ChatPartModality`, `ValidateChatMmLimits`, `Qwen3VLChatSupportedMmLimits`) — anchors `src/vllm/multimodal/hasher.cpp:56`, `src/vllm/config/multimodal.cpp:49`, `src/vllm/entrypoints/openai/chat_mm.cpp:295,311`, `include/vllm.h:197,403` | `tests/vllm/multimodal/test_qwen3vl_processor.cpp` (processor-parity 23/23 BIT-identical vs the M0 oracle fixture `tests/vllm/multimodal/fixtures/qwen3vl/`, RED-first: wrong normalize shift -> 1.2M mismatches); `tests/vllm/v1/core/test_encoder_cache_manager.cpp` 32/32. Text-inertness: `test_request`/`test_engine_types`/`test_lmcache_codec`/`test_lmcache_key_agreement`/`test_openai_conformance` all green standalone; SACRED CUDA 27B/35B/Coder = GPU inertness proof; `check-device-leakage` OK — anchor `tests/vllm/multimodal/test_qwen3vl_processor.cpp:59`. L1 limits: `tests/vllm/config/test_multimodal_config.cpp` 7/7 (21 assertions) + `tests/vllm/multimodal/test_processing_limits.cpp` 19/19 (78 assertions), porting `tests/multimodal/test_processing.py:902-941,944-985`, `tests/entrypoints/multimodal/llm/test_mm_embeds_only.py:41-49` and `tests/entrypoints/unit_tests/test_chat_utils.py:1498-1560` @ `5559679229bc`; mutations proven RED: map-before-flag precedence, the dropped throw, the dropped `min()` fold. L2 (aarch64 `build-test-cpu-arm64` lane, `-DVLLM_CPP_CUDA=OFF`): `tests/vllm/entrypoints/openai/test_serve_mm_limits.cpp` 11/11 (109 assertions, flags + the parser's upstream refusals + the builtin-only reach of `extra="forbid"`) + `test_chat_mm` 11/11 (126) + `test_openai_api_server` 56/56 (CASES; its assertion count is timing-dependent — 632/648/651 across three runs of one binary, so only the case count is quotable — the HTTP 400 arm proven against BOTH a 500 and a truncated 200); RED-first behavioural: `CHECK(500 == 400)`, `CHECK("InternalServerError" == BadRequestError)` and `CHECK(200 == 400)`; mutations proven RED: flag→config plumbing dropped, the call-site wiring dropped, and the refusal re-typed off `InputValidationError` (which lands as the 500 L1's design avoided) | [multimodal-track.md](specs/multimodal-track.md) §3 (M0/M1) | `READY` | - | | `ENG-MM-VISION-TOWER` | **Qwen3-VL vision tower `Qwen3_VisionTransformer` (M2a), proven faithful vs vLLM 0.25.0 in isolation.** The reusable vision half of the whole Qwen3-VL family + Qwen3.6 (27B/35B share this exact tower). Pure-additive C++ forward composed from public vt:: ops: patch-embed (Conv3d-as-matmul + bias), host pos-embed bilinear-interp+spatial-merge-reorder, 24 ViT blocks (LayerNorm + vision attention with partial-rotary NeoX vision RoPE via `vt::RopeFromCache` + non-causal `vt::Attention(causal=false)` + tanh-GELU MLP), patch merger (LayerNorm + exact-erf-GELU + 2 FCs), DeepStack 3 post-shuffle-norm mergers at layers 5/11/17 → `[196,10240]`. Adds 2 additive elementwise vt ops (`GeluTanh`/`GeluErf`). NO runner/model/registry edit → text engines byte-identical by construction. Proven faithful in ISOLATION; the merge into `input_embeds` + the MRoPE/DeepStack text backbone + the e2e image gate are M2b/M2c. | T1 | `vllm/model_executor/models/qwen3_vl.py` `Qwen3_VisionPatchEmbed:347`, `Qwen3_VisionBlock:413`, `Qwen3_VisionPatchMerger:467`, `Qwen3_VisionTransformer:519`, `forward:800`, `pos_embed_interpolate_native:277`, `rot_pos_emb:667`; `qwen2_5_vl.py::Qwen2_5_VisionAttention.forward:397`; `rotary_embedding/common.py::ApplyRotaryEmb.forward_static:151` @ `e24d1b24` | `src/vllm/model_executor/models/qwen3_vl_vision.{h,cpp}`; 2 vt ops `include/vt/ops.h` + `src/vt/ops.cpp` + `src/vt/cuda/cuda_layernorm.cu` + `src/vt/cpu/cpu_layernorm.cpp`; dumps `scripts/mm/m2a_tower_{ref,weight}_dump.py`; fixtures `tests/vllm/multimodal/fixtures/qwen3vl_tower/` | `tests/vllm/multimodal/test_qwen3vl_tower.cpp` — 4 RED-first tower gates vs the dumped vLLM-0.25.0 reference 348/348 (patch-embed 2.1e-3, block0 6.8e-3, merger 6.5e-2, DeepStack 1.2e-2/3.3e-2/4.4e-2, full tower 5.1e-2; pos-embed 2.5e-3 + rope 1.9e-3 TIGHT); bf16-depth envelope RCA'd; RED = rope disabled → block0 0.149/tower 0.75/6 fails; cutlass-ON+FA2 banner; clean `-Werror`; compute-sanitizer 0 — anchor `tests/vllm/multimodal/test_qwen3vl_tower.cpp:96` | [multimodal-track.md](specs/multimodal-track.md) §3 (M2a) | `ACTIVE` | `CLAIM-MULTIMODAL-M2A` | | `ENG-MM-TEXT-BACKBONE` | **Qwen3-VL text-backbone numeric contracts `Qwen3VLGetRopeIndex`/`Qwen3VLMergeMultimodal`/`Qwen3VLComputeDeepstack` (M2b/M2c), unit-green vs vLLM 0.25.0.** The deterministic pieces that fork the plain Qwen3-dense text path for a vision-conditioned decode: (1) MRoPE 3-D `get_rope_index` positions [3,T] (image tokens get (t,h,w) grid positions, text sequential); (2) the 3-section MRoPE APPLICATION — proven to be the EXISTING `vt::RopeFromCache` mrope path (positions [3,T] + `mrope_section=[24,20,20]` interleaved), faithful to `MRotaryEmbedding.forward_native` for Qwen3-VL's exact config; (3) `_compute_deepstack_embeds` scatter → [L,T,H] decoder-injection tensor; (4) `_merge_multimodal_embeddings` masked scatter of the tower's `[:,:2560]` into `input_embeds`. Pure-additive TU — NO shared dense forward / runner / registry edit → text engines byte-identical by construction. The e2e image forward (VL weight loader + forked MRoPE/DeepStack decode loop) is the remaining M2c wire-up. | T1 | `vllm/model_executor/models/qwen3_vl.py` `_get_mrope_input_positions:2567`, `_iter_mm_grid_hw:2482`, `_compute_deepstack_embeds:2761`, `Qwen3LLMModel.forward` deepstack `:1589`; `vllm/model_executor/models/utils.py::_merge_multimodal_embeddings:524`; `vllm/model_executor/layers/rotary_embedding/mrope.py` MRotaryEmbedding @ `e24d1b24` | `src/vllm/model_executor/models/qwen3_vl_text.{h,cpp}`; existing `vt::RopeFromCache` mrope path (`src/vt/{cpu,cuda}/*`); dump `scripts/mm/m2b_text_ref_dump.py`; fixtures `tests/vllm/multimodal/fixtures/qwen3vl_text/` — anchor `src/vllm/model_executor/models/qwen3_vl_text.cpp:9` | `tests/vllm/multimodal/test_qwen3vl_text.cpp` — 4 RED-first gates vs the dumped vLLM-0.25.0 reference 85/85 (get_rope_index BIT-exact [3,204], delta −182; MRoPE q rel-L2 1.5e-3 / k 1.5e-3, RED interleaved-off >5e-2; DeepStack + merge BIT-exact); CPU-only, no weights; clean CPU `-Werror` — anchor `tests/vllm/multimodal/test_qwen3vl_text.cpp:99` | [multimodal-track.md](specs/multimodal-track.md) §3 (M2b/M2c) | `ACTIVE` | `CLAIM-MULTIMODAL-M2BC` | | `ENG-MM-QWEN36-VL-FORWARD` | **Qwen3.6-27B (`Qwen3_5ForConditionalGeneration`) GDN-hybrid VL forward — IMAGE (M3-b) + VIDEO (M3d) BOTH e2e, STRICT gates PASS 32/32. Our own gate model's image+video paths now work end-to-end (speed pending).** The genuinely-new integration completing our own gate model's mm paths: fork the landed bf16 `Qwen3_5DenseModel` GDN-hybrid forward (48 GDN + 16 full-attn) on gated, default-off points so a text-only 27B request stays byte-identical — (a) `inputs_embeds` entry (embed ids + `Qwen3VLMergeMultimodal` scatter of the 27B tower merger `[N,5120]` into the visual-token rows; 27B has EMPTY `deepstack_visual_indexes` ⇒ NO DeepStack); (b) 3-section MRoPE (`mrope_section=[11,11,10]` interleaved, rotary_dim 64, theta 1e7) in the 16 full-attn layers only via the proven `vt::RopeFromCache` mrope path (GDN layers carry no rope); (c) mixed load = the M2a `Qwen3_VisionTransformer` (27B vision config, empty deepstack) bf16 tower + the bf16 GDN-hybrid LLM via the EXISTING `LoadQwen3_5Dense`. **M3d (2026-07-25) added VIDEO by REUSE:** the M3-b image driver refactored into a shared `VLGenerateCoreGdn`, image+video wrappers differ ONLY in the merge mask (`image_token` vs `video_token` across frames) + the get_rope_index (`Qwen3VLGetRopeIndex` vs `Qwen3VLGetRopeIndexVideo`); the M3c processor/windowed-tower/video-MRoPE are reused verbatim. | T1 | `vllm/model_executor/models/qwen3_5.py:389` (`Qwen3_5ForConditionalGeneration` subclasses `Qwen3VLForConditionalGeneration`; `visual = Qwen3_VisionTransformer`, modalities {"image","video"}); `qwen3_vl.py` `_process_video_input:2165`, `_get_mrope_input_positions:2567` video branch, `get_video_repl:1479`; the 27B `config.json` (`mrope_section=[11,11,10]`, empty `deepstack_visual_indexes`) @ `e24d1b24` / vLLM 0.25.0 | **M3-b + M3d BUILT + GATED 2026-07-25:** vision-only loader `LoadQwen3VLVisionWeights` (`src/vllm/model_executor/models/qwen3_vl.cpp`, 27B config) + shared `VLGenerateCoreGdn` + image driver `Qwen3_5VLGenerateGreedy` + **video driver `Qwen3_5VLGenerateGreedyVideo`** + `BuildMropeCosSinHost` + the `mrope_cos_sin` param on `DenseForwardLayers` (`src/vllm/model_executor/models/qwen3_5.cpp`, nullptr on every text caller ⇒ byte-identical; the video driver is purely additive, the shared text forward UNTOUCHED per `git diff --stat`) reusing M2a tower + `LoadQwen3_5Dense` bf16 LLM | **IMAGE:** golden `tests/vllm/multimodal/fixtures/qwen3_5_27b/` (STRICT sha256 `ead4b484…`); STRICT image gate PASS **32/32** (`test_qwen3_5_vl_e2e.cpp`, 54/54, re-run post-refactor). **VIDEO (M3d):** oracle `scripts/mm/m3d_video_oracle_capture.py` on the M3c synthetic clip (raw sha `8a111599…`, grid `[4,8,8]`, 64 video tokens) K=5 DETERMINISTIC ⇒ STRICT golden; **STRICT video gate PASS 32/32** (`test_qwen3_5_vl_video_e2e.cpp`, 27/27; near-tie gaps 0.0000 nats everywhere), fixtures `tests/vllm/multimodal/fixtures/qwen3_5_27b_video/`. Text-inertness 27B 235/235, 35B 315/315, Coder 138/138 (by construction); clean `-Werror` 0 warn; compute-sanitizer 0 on the 27B video forward. **SPEED MEASURED (2026-07-26, `CLAIM-MULTIMODAL-SPEED`): image c1 vs vLLM 0.25.0 GRAPHED — decode TPOT 225.0 ms/tok vs 226.9 = AT PARITY (0.99×), LLM prefill 326 ms vs vLLM TTFT 321 ms = at parity; vision tower WAS 2114 ms vs vLLM encode ≤~250 ms = ~10× (THE gap). TOWER LEVER EXECUTED (2026-07-26, `CLAIM-MULTIMODAL-SPEED-TOWER`, [multimodal-speed.md](specs/multimodal-speed.md) §7): nsys `cuda_gpu_kern_sum` attributed 98.9 % of the tower forward to the naive `vt::cuda::AttentionKernel` (56 ms/block; NOT QKV/FA2-routing); fixed by a warp-scoped online-softmax op `AttentionDenseFast` (separate op ⇒ `kAttention`/text byte-identical) + one-time resident-weight load ⇒ per-image tower 2114 → 148 ms (14.3×), **0.59× vs vLLM eager encode = FASTER**. STRICT image/video e2e HELD 32/32 (+4B DeepStack 32/32), `test_ops_attention` 37239/37239, 27B text SACRED 235/235, compute-sanitizer memcheck 0, clean `-Werror`. `benchmark_binding=false`, single-seq driver (no c2+/server). Remaining: batched/graphed mm serving (c2+) + audio our-side — DONE bar not yet met.** | [multimodal-track.md](specs/multimodal-track.md) §M3 + [multimodal-speed.md](specs/multimodal-speed.md) §7 + §8 (decode lever #2 CLOSED 2026-07-27: on-GPU greedy argmax + decode embed round-trip removed on `VLGenerateCoreGdn`; bit-exact — image/video STRICT 32/32 held; 27B decode NEUTRAL at the ~222 ms bandwidth floor) + §9 (lever #3 FIRST BRICK 2026-07-27, `CLAIM-MULTIMODAL-SPEED-GRAPH`: the shared `VLGenerateCoreGdn` decode step now routes through the production `Qwen3_5DenseDecodeGraph` cold→warm→replay captured decode — the mm decode is now GRAPH-CAPTURABLE, closing the un-graphed-eager-loop structural gap; S==B==1 bit-identical rebuild; token-exact HELD image/video STRICT 32/32 with 30 graph replays confirmed; A/B graphed 232.5 vs eager 233.4 ms/tok = NEUTRAL at the 27B bandwidth floor; the launch-overhead win + batched c2+ + serving ingestion are the recorded W-plan W1-W3) + §16 (vision-forward flash kernel 2026-07-28, `CLAIM-MM-SPEED-QWEN-IMAGE`: ATTRIBUTION-FIRST nsys attributed ~85% of the 148 ms tower forward to the dense attention `AttentionWarpKernel` [4.66 ms/block×27]; routed it to the §14 flash-tiled `vt::AttentionDenseFlash` [head_dim 72, byte-identical — per-warp math verbatim, only K/V from shared-mem tiles]. STRICT image/video e2e HELD 32/32 [27B+4B], `test_ops_attention` 37239/37239, goldens md5 UNCHANGED, nsys proof `AttentionDenseFlashKernel` 24 inst/zero warp, RED 30/46→46/46, sanitizer 0. A/B warp 148.3→flash 142.3 ms = 1.04× — the profile REFUTED a big lever: at t=784 the vision attention is serial-latency-bound not bandwidth-bound [audio §14 was 1.82× at t=1500], flash recovers only ~6 ms. **HONEST: the tower ALREADY BEATS vLLM — 142 ms vs ~250 ms eager encode = 0.57×**; image/video mm-forward is correctness-DONE + speed-BEATS-vLLM; residual = tensor-core MMA hd-72 attention [not needed for parity] + batched c2+/serving) | `ACTIVE` | `CLAIM-MULTIMODAL-SPEED-TOWER` + `CLAIM-MULTIMODAL-SPEED-DECODE` + `CLAIM-MULTIMODAL-SPEED-GRAPH` + `CLAIM-MM-SPEED-QWEN-IMAGE` | diff --git a/.agents/roadmap_v1.md b/.agents/roadmap_v1.md index fdad088ea..dfe6eae71 100644 --- a/.agents/roadmap_v1.md +++ b/.agents/roadmap_v1.md @@ -192,7 +192,8 @@ issue is not yet placed. Keyed record: update in place, never append. | [#558](https://github.com/mudler/vllm.cpp/issues/558) | — | `tests/parity/hf_snapshot.h` has no guard against declaration-order breaks: the C++ build catches them, but the records-only lane that broke it never builds C++, and all 14 TUs that include the header are checkpoint-gated so `ctest` reports the break as `***Not Run`. `fafa16f0f` (#546, #551) fixed the ordering and carried no guard | bug | | [#603](https://github.com/mudler/vllm.cpp/issues/603) | — | `windows-msvc-cpu` / `windows-msvc-vulkan` are RED on `main`: `test_backend_cross_device.cpp` calls POSIX `setenv`/`unsetenv`, which MSVC does not provide | bug | | [#606](https://github.com/mudler/vllm.cpp/issues/606) | — | `vllm-serve` aborts on `--enable-auto-tool-choice` (89/157 recipes) and `--trust-remote-code` (82/157), both of which are no-ops for us, so a copy-pasted official recipe command never reaches model load. Needs an accepted-and-inert seam with a per-flag reason; no row owns serve CLI recipe compatibility | feature | -| [#607](https://github.com/mudler/vllm.cpp/issues/607) | `ENG-MM-INPUT-PIPELINE` | **Premise corrected 2026-08-13, see `specs/multimodal-track.md` §1.5.** Not "skip the vision encoder": `--language-model-only` sets every modality limit to **0** (`multimodal.py:78,321-327`) and is sugar over `--limit-mm-per-prompt`. Two consequences follow, and we have NEITHER — upstream then **refuses every multimodal request** (`processing/context.py:409-428` raises "At most 0 image(s) may be provided in one prompt", from `:461` and `chat_utils.py:662`), and builds the tower uninitialised (`interfaces.py:293`). This is a PORT of the limits mechanism (L1-L4), not the exposure of a boolean. **L1 LANDED 2026-08-13:** `vllm::MultiModalConfig` + `GetLimitPerPrompt` (`include/vllm/config/multimodal.h`) and the refusal it carries (`include/vllm/multimodal/processing/context.h`) are in, unit-gated, with NO serve surface. **L2 LANDED 2026-08-14 (also closing [#686](https://github.com/mudler/vllm.cpp/issues/686)):** both serve flags (`--[no-]language-model-only`, `--limit-mm-per-prompt ''` with upstream's legacy/configurable/mixed spellings and its refusals, `src/vllm/config/multimodal.cpp`), the C-ABI fields (`vllm_model_params.language_model_only` / `.limit_mm_per_prompt`, **ABI v18 → v19**), and the CALL SITE — `MakeQwen3VLImageChatFn` validates through `ValidateChatMmLimits` (port of `chat_utils.py:648-662`) over the seam's declared ceiling `{"image": 1}`, so a three-image request is `400 "At most 1 image(s) may be provided in one prompt."` instead of a truncated answer. The 43 recipes now reach model load. Gated on aarch64: `test_serve_mm_limits` 10/10, `test_chat_mm` 11/11, `test_openai_api_server` 56/56 (the HTTP 400 arm proven e2e against both 500 and a truncated 200). Still owed: L3 (tower skip — **no memory claim is made by L2**), L4 (kernel gate), and the second call site `process_inputs_mm`, which is blocked on the per-model `get_supported_mm_limits()` hook the M2 towers own. The flag appears in this repo only in `tools/bench/run_serve_low.py`, which passes it to the ORACLE — a grep reads as coverage and is not | feature | +| [#607](https://github.com/mudler/vllm.cpp/issues/607) | `ENG-MM-INPUT-PIPELINE` | **Premise corrected 2026-08-13, see `specs/multimodal-track.md` §1.5.** Not "skip the vision encoder": `--language-model-only` sets every modality limit to **0** (`multimodal.py:78,321-327`) and is sugar over `--limit-mm-per-prompt`. Two consequences follow, and we have NEITHER — upstream then **refuses every multimodal request** (`processing/context.py:409-428` raises "At most 0 image(s) may be provided in one prompt", from `:461` and `chat_utils.py:662`), and builds the tower uninitialised (`interfaces.py:293`). This is a PORT of the limits mechanism (L1-L4), not the exposure of a boolean. **L1 LANDED 2026-08-13:** `vllm::MultiModalConfig` + `GetLimitPerPrompt` (`include/vllm/config/multimodal.h`) and the refusal it carries (`include/vllm/multimodal/processing/context.h`) are in, unit-gated, with NO serve surface. **L2 LANDED 2026-08-14 (also closing [#686](https://github.com/mudler/vllm.cpp/issues/686)):** both serve flags (`--[no-]language-model-only`, `--limit-mm-per-prompt ''` with upstream's legacy/configurable/mixed spellings and its refusals, `src/vllm/config/multimodal.cpp`), the C-ABI fields (`vllm_model_params.language_model_only` / `.limit_mm_per_prompt`, **ABI v18 → v19**), and the CALL SITE — `MakeQwen3VLImageChatFn` validates through `ValidateChatMmLimits` (port of `chat_utils.py:648-662`) over the seam's declared ceiling `{"image": 1}`, so a three-image request is `400 "At most 1 image(s) may be provided in one prompt."` instead of a truncated answer. The 43 recipes now reach model load. Gated on aarch64: `test_serve_mm_limits` 11/11, `test_chat_mm` 11/11, `test_capi` 58/58, `test_openai_api_server` 56/56 (the HTTP 400 arm proven e2e against both 500 and a truncated 200). Repaired in review (#758 filed): the v19 ABI header claimed an enforcement no C-ABI caller can reach — `set_multimodal_chat_fn` has one caller, `server_main.cpp` — and `ParseLimitMmPerPromptJson` refused unknown option keys for EVERY modality, where upstream's `BaseDummyOptions` fallback has no `extra="forbid"` and drops them. Still owed: L3 (tower skip — **no memory claim is made by L2**), L4 (kernel gate), and the second call site `process_inputs_mm`, which is blocked on the per-model `get_supported_mm_limits()` hook the M2 towers own. The flag appears in this repo only in `tools/bench/run_serve_low.py`, which passes it to the ORACLE — a grep reads as coverage and is not | feature | +| [#758](https://github.com/mudler/vllm.cpp/issues/758) | `ENG-MM-INPUT-PIPELINE` | A multimodal refusal cannot distinguish a configured limit from an UNIMPLEMENTED arm. `Qwen3VLChatSupportedMmLimits()` declares the seam's honest ceiling `{"image": 1}` with video/audio absent, but the message a client gets is upstream's generic `At most 0 video(s) may be provided in one prompt.` — identical to what `--limit-mm-per-prompt '{"video": 0}'` produces. AGENTS.md requires an unimplemented arm be refused "with a message naming the missing piece"; #749 claimed the ceiling satisfies that and its review found it does not. The only present signal is by OMISSION (`ValidateNumItems` withholds the `--limit-mm-per-prompt` hint when raising the limit would not help). Not fixed in flow: naming the arm diverges from a verbatim-ported message three suites assert byte-for-byte, so it needs its own spec and fresh review. Found in the #749 review round (#607 wave L2, #686) | bug | | [#651](https://github.com/mudler/vllm.cpp/issues/651) | — | `test_agent_record`'s MODEL-ratchet docstring is two contradictory paragraphs spliced together, and the surviving half records a pin transition that never happened | bug | | [#652](https://github.com/mudler/vllm.cpp/issues/652) | — | `model-matrix.md` prose counters drifted: LTX-2.5 reached the rows and the CI-enforced rollup but none of the five sentences that count them | bug | | [#659](https://github.com/mudler/vllm.cpp/issues/659) | — | LTX-2.5 device select adopts M3a's platform seam but not its companion capability guard: `ltx2_video.cpp` asks `CurrentPlatform().device_type()` and `TryGetBackend(...)` but never `supports_model_architecture`, so a PARTIAL backend (Metal 15/75 ops, Tenstorrent) is handed a queue and dies in a kernel bind where it used to be refused BY NAME (found while reviewing #553 for landing) | bug | diff --git a/.agents/specs/multimodal-track.md b/.agents/specs/multimodal-track.md index ed70d7026..26ec3b65e 100644 --- a/.agents/specs/multimodal-track.md +++ b/.agents/specs/multimodal-track.md @@ -324,16 +324,30 @@ comparing the two arms must set the flag on both sides or state that it did not. turns the flag off explicitly must not die on an unknown argument. `--limit-mm-per-prompt ''`, mirroring `arg_utils.py:556,1279,1692`, whose dict type resolves to `type=parse_type(json.loads)` - (`arg_utils.py:375-377`) — so the value is a JSON object. + (`arg_utils.py:379-381` — the plain-`dict` branch; the `union_dict_and_str` + branch immediately above at `:374-378` is a different rule and needs a `str` + arm or a non-builtin type hint, which `limit_per_prompt: dict[str, + BaseDummyOptions]` does not have because `dict[…]` reports + `__module__ == "builtins"`) — so the value is a JSON object. `ParseLimitMmPerPromptJson` (`src/vllm/config/multimodal.cpp`) ports `_validate_limit_per_prompt` (`multimodal.py:212-236`) and the DummyOptions - dataclasses behind it (`:17-43`): the legacy count-only, the configurable + dataclasses behind it (`:17-45`): the legacy count-only, the configurable `{"count": N, …}` and the mixed spellings all parse; a non-object document, - a negative count (`count: int = Field(999, ge=0)`, `:20`), an unknown - per-modality option (`extra="forbid"`) or a non-positive one - (`Field(None, gt=0)`) is REFUSED before the model load rather than - defaulted, because a mistyped limit that silently became 999 is a limit - that is not there. The profiling options are validated and then dropped + a negative count (`count: int = Field(999, ge=0)`, `:21`), and — **for the + three builtin modalities only** — an unknown per-modality option + (`extra="forbid"`, `:24,33,41`) or a non-positive one (`Field(None, gt=0)`, + `:28-30,37-38,45`) is REFUSED before the model load rather than defaulted, + because a mistyped limit that silently became 999 is a limit that is not + there. A modality OUTSIDE image/video/audio falls to the bare + `BaseDummyOptions` at `:233`, the one dummy-options dataclass declared + without `extra="forbid"` (`:17-21`), so pydantic's default `extra='ignore'` + applies and its unknown keys are dropped rather than refused — re-derived + under pydantic 2.12.5 against the pinned declarations: + `BaseDummyOptions(count=2, foo=3)` → `BaseDummyOptions(count=2)`, while + `ImageDummyOptions(count=2, foo=3)` raises. We mirror both halves; refusing + the second would refuse a document upstream accepts (repaired in the #749 + review round — the original L2 landing refused it for every modality). + The profiling options are validated and then dropped (only `.count` feeds `get_limit_per_prompt`, `:335`) and the drop is ANNOUNCED per key. **NAMED RESIDUAL:** upstream's dotted spelling (`--limit-mm-per-prompt.image 2`) is a `FlexibleArgumentParser` feature @@ -351,6 +365,20 @@ comparing the two arms must set the flag on both sides or state that it did not. Both land on `EngineParams::multimodal` → `LoadedEngine::mm_config()`, so the server flags and the ABI resolve ONE config object per engine and cannot drift. + **What the ABI fields do NOT do, corrected in the #749 review round.** The + v19 header initially said the fields are "ENFORCED, not recorded" and that + an engine loaded with `language_model_only` answers a multimodal request + with `At most 0 image(s)…`. That is true of the OpenAI-server path and false + of the C ABI: `set_multimodal_chat_fn` has exactly one caller, + `server_main.cpp`, and `serving_chat.cpp` gates the whole multimodal branch + on that seam being set — `vllm_chat` / `vllm_chat_stream` never install one. + On a C-ABI engine the two fields are therefore RECORDED on the config and + consulted by nothing the ABI can reach: an `image_url` content part parses, + is dropped, and the request is answered as text. The header now says so, and + `tests/capi/test_capi.cpp` pins it behaviourally (a `language_model_only` + engine + an `image_url` body → `VLLM_OK` and a `chat.completion`) so the + wording cannot go stale on a permanent public contract. Adding a C-ABI + multimodal REQUEST path is a new capability, not an L2 repair. 3. **The call site — this is what L2 is for.** `chat_mm.cpp` now calls `ValidateChatMmLimits` (the port of the `chat_utils.py:648-662` tracker) as step 0 of `MakeQwen3VLImageChatFn`, over a `BaseProcessingInfo` folding @@ -364,6 +392,19 @@ comparing the two arms must set the flag on both sides or state that it did not. `--limit-mm-per-prompt image=99` still refuses the second image, and the refusal then carries no `--limit-mm-per-prompt` hint because raising the user's limit would not help. + **The ceiling is a number, not yet a message (#758).** The L2 landing + claimed this declaration satisfies AGENTS.md's "an unimplemented arm is + refused with a message naming the missing piece"; the #749 review found it + does not. The text a client receives is upstream's generic `At most 0 + video(s) may be provided in one prompt.`, which is indistinguishable from an + operator having configured that limit. The only present signal is by + OMISSION — the withheld `--limit-mm-per-prompt` hint. Naming the arm means + diverging from a verbatim-ported message that three suites assert + byte-for-byte, so it is owed to #758 with its own spec rather than repaired + in review. Also corrected there: `get_supported_mm_limits` is not defined on + `Qwen3VLProcessingInfo`; it is inherited from `Qwen2VLProcessingInfo` + (`qwen2_vl.py:851-852`, `{"image": None, "video": None}`), which + `qwen3_vl.py:848` subclasses. **#686 is CLOSED by this.** A three-image request is answered `400 BadRequestError "At most 1 image(s) may be provided in one prompt."` @@ -378,9 +419,23 @@ comparing the two arms must set the flag on both sides or state that it did not. wrong the same way — neither is upstream's refusal — so the issue's diagnosis stands and only its consequence was understated. - **Gates** (aarch64, the `build-test-cpu-arm64` lane; `-DVLLM_CPP_CUDA=OFF`): - `test_chat_mm` 11/11 (126 assertions), `test_serve_mm_limits` 10/10 (101), - `test_openai_api_server` 56/56 (638). RED before the change: `test_chat_mm` + **Gates** (aarch64, the `build-test-cpu-arm64` lane; `-DVLLM_CPP_CUDA=OFF`). + At the #749 REVIEW-REPAIR head, on `kairos-4db2`: clean rebuild 1355/1355 + targets, 0 warnings under `-Werror`; `ctest -j 6` 456/457 with 2 skipped, the + single failure being `test_op_parity` (#737, reproduced from pristine main); + `test_serve_mm_limits` 11/11 (109 assertions), `test_chat_mm` 11/11 (126), + `test_openai_api_server` 56/56, `test_capi` 58/58 (536), + `test_processing_limits` 19/19 (78), `test_multimodal_config` 7/7 (21). + Mutations, `cp` + `touch` + rebuild + re-verified green between each and both + files md5-identical afterwards: `IsBuiltinModality` forced to `true` (the + pre-repair "forbid extras everywhere") takes `test_serve_mm_limits` to + 10 passed / 1 failed, and wiring an image refusal into `vllm_chat` takes + `test_capi` to 57 passed / 1 failed on `REQUIRE(1 == 0)`. + **`test_openai_api_server`'s ASSERTION count is not a pin** — measured 632, + 648 and 651 across three runs of one binary, because SSE-chunk loops assert + per chunk received. Its CASE count, 56/56, is stable and is the number to + quote. At the L2 landing the assertion count was recorded as 638. + RED before the L2 change: `test_chat_mm` 2 cases failing (`CHECK_THROWS_AS ... threw a DIFFERENT exception: "Expand ImagePlaceholders: more image placeholders than grids"` and `FATAL ERROR: expected --language-model-only to refuse an image request`), and the HTTP legs diff --git a/docs/USAGE.md b/docs/USAGE.md index 916ff0801..d40dd8807 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -1155,8 +1155,8 @@ a stop token early. | `--reasoning-parser ` | `none` | Reasoning parser (`think_auto`, `deepseek_r1`, `deepseek_v3`, `holo2`, `mistral`, `minimax_m2`, `minimax_m2_append_think`, `step3`, `olmo3`, `muse_glimmer`, `qwen3`, `mimo`). `auto` detects, `none` disables. `qwen3` and its `mimo` alias are the engine-backed adapter (one upstream class, two registry names): thinking is ON, so a marker-less stream is reasoning and a `` ends reasoning with no ``. `auto` never selects it — a generic `` template resolves to `think_auto`, which is the right default for hybrid-thinking models that may answer with no think block at all | | `--kv-transfer-config ''` | (unset) | External KV connector, same JSON as vLLM's flag. See [docs/KV-OFFLOAD.md](KV-OFFLOAD.md) | | `--speculative-config ''` | (unset) | Speculative decoding (`mtp`, `dflash`, `ngram`), same JSON as vLLM's flag. `dspark` speculates on the Qwen3.6 gate models (native + Speculators drafts), token-identically to speculative-off, but is not gated on speed (currently ~2% behind at c1). A GGUF target, or a target with no aux multi-tap, is refused by name (`SPEC-DSPARK`). Its sequential Markov sampling runs on device by default; `VT_DSPARK_DEVICE_SAMPLE=0` restores the host loop (token-identical, cost only). The speculative verify runs from a captured CUDA graph, worth +12.2%/+3.5% on the 35B cells; `VT_SPEC_DECODE_GRAPH=0` restores the eager verify (also token-identical). See [docs/SPECULATIVE-DECODING.md](SPECULATIVE-DECODING.md) | -| `--language-model-only` / `--no-language-model-only` | off | Disable all multimodal input by setting **every** modality limit to 0, mirroring vLLM's flag of the same name. It is not a "skip the encoder" switch: the server then **refuses** a multimodal request with `400 At most 0 image(s) may be provided in one prompt. Set --limit-mm-per-prompt to increase this limit.` It does **not** free VRAM yet — nothing gates tower construction on it ([#607](https://github.com/mudler/vllm.cpp/issues/607) wave L3) | -| `--limit-mm-per-prompt ''` | (unset ⇒ 999 per modality) | Maximum multimodal input items per prompt, per modality, as the same JSON object vLLM's flag takes: `'{"image": 2, "video": 0}'`, or with profiling options `'{"video": {"count": 1, "num_frames": 32}}'` (the options are validated and ignored — they size dummy inputs for memory profiling, which this engine does not do). A limit can only **lower** what the model/seam supports, never raise it. Malformed JSON, a negative count or an unknown per-modality option is refused at startup rather than defaulted. Upstream's dotted spelling (`--limit-mm-per-prompt.image 2`) is not accepted here, as for `--kv-transfer-config` and `--speculative-config` | +| `--language-model-only` / `--no-language-model-only` | off | Disable all multimodal input by setting **every** modality limit to 0, mirroring vLLM's flag of the same name. It is not a "skip the encoder" switch: the server then **refuses** a multimodal request with ``400 At most 0 image(s) may be provided in one prompt. Set `--limit-mm-per-prompt` to increase this limit.`` It does **not** free VRAM yet — nothing gates tower construction on it ([#607](https://github.com/mudler/vllm.cpp/issues/607) wave L3) | +| `--limit-mm-per-prompt ''` | (unset ⇒ 999 per modality) | Maximum multimodal input items per prompt, per modality, as the same JSON object vLLM's flag takes: `'{"image": 2, "video": 0}'`, or with profiling options `'{"video": {"count": 1, "num_frames": 32}}'` (the options are validated and ignored — they size dummy inputs for memory profiling, which this engine does not do). A limit can only **lower** what the model/seam supports, never raise it. Malformed JSON, a negative count, or an unknown option on `image` / `video` / `audio` is refused at startup rather than defaulted. An unknown option on any other modality name is dropped rather than refused, mirroring upstream, whose fallback `BaseDummyOptions` is the one such dataclass without `extra="forbid"`. Upstream's dotted spelling (`--limit-mm-per-prompt.image 2`) is not accepted here, as for `--kv-transfer-config` and `--speculative-config` | | `--enable-log-requests` / `--disable-log-requests` | on | Log each incoming request. Mirrors vLLM's flag of the same name | | `--enable-log-outputs` | off | Also log the generated output, not just the request | | `--max-log-len N` | `256` | Truncate logged prompts and outputs to N characters | @@ -1836,9 +1836,15 @@ Accepted part types (`src/vllm/entrypoints/openai/chat_mm.cpp`): vLLM caps how many items of each modality one prompt may carry (`--limit-mm-per-prompt`), and `--language-model-only` is sugar for setting every -one of those limits to 0. Both flags are accepted (#607, waves L1+L2), both are -enforced, and both are also C ABI fields -(`vllm_model_params.language_model_only` / `.limit_mm_per_prompt`, ABI v19). +one of those limits to 0. Both flags are accepted (#607, waves L1+L2) and both +are enforced **on this server's chat path**, which is the one place that installs +the multimodal chat seam the check runs behind. + +Both are also C ABI fields (`vllm_model_params.language_model_only` / +`.limit_mm_per_prompt`, ABI v19), and there they configure the engine — including +a server built on it — but they do not change what a `vllm_chat` call returns: +the C ABI has no multimodal request path yet, so an `image_url` content part sent +through it is dropped and answered as text. The refusals below are the server's. The limits are the mechanism and the flag is the sugar, so it is worth stating what the flag actually does: it does not "skip the encoder", it makes the server @@ -1863,9 +1869,12 @@ Two things follow from how the limit is computed not routed at all, so their limit is 0 and they are refused by name rather than dropped — this is what closed [#686](https://github.com/mudler/vllm.cpp/issues/686)). -- The `Set --limit-mm-per-prompt to increase this limit.` hint appears only when - raising the limit would actually help — that is, when the seam could take the - items and the configuration is what refused them. +- The ``Set `--limit-mm-per-prompt` to increase this limit.`` hint appears only + when raising the limit would actually help — that is, when the seam could take + the items and the configuration is what refused them. Its absence is currently + the only way to tell an unimplemented arm from a configured limit; the + refusal message itself does not say which + ([#758](https://github.com/mudler/vllm.cpp/issues/758)). **Not yet:** `--language-model-only` frees no memory. Nothing gates vision-tower construction on the limits, so the flag today changes what the server accepts, diff --git a/include/vllm.h b/include/vllm.h index 540cd6992..24fd02683 100644 --- a/include/vllm.h +++ b/include/vllm.h @@ -180,14 +180,38 @@ extern "C" { * "video": 0}', or the option form '{"video": {"count": 1}}'), following * the v9 precedent that a dict-valued vLLM flag crosses this ABI as its own * JSON rather than as a fixed struct of modalities the ABI would then owe - * forever. A malformed document, an unknown per-modality option, or a - * negative count fails vllm_engine_load with VLLM_ERR_INVALID_ARGUMENT - * rather than defaulting — mirroring the pydantic validation upstream does - * at parse time (multimodal.py:17-43,212-236). - * Both are ENFORCED, not recorded: a modality's limit is what - * BaseProcessingInfo::ValidateNumItems refuses against, so an engine loaded - * with language_model_only answers a multimodal request with + * forever. A malformed document, a negative count, or an unknown option on + * one of the three modalities upstream gives an `extra="forbid"` dataclass + * (image/video/audio) fails vllm_engine_load with + * VLLM_ERR_INVALID_ARGUMENT rather than defaulting — mirroring the pydantic + * validation upstream does at parse time (multimodal.py:17-45,212-236). + * An unknown option on any OTHER modality is dropped, not refused, because + * the BaseDummyOptions it falls back to (:17-21,233) is the one such + * dataclass declared without extra="forbid". + * WHERE THEY BITE, stated exactly, because this contract is permanent. Both + * fields land on the engine's ONE MultiModalConfig + * (vllm_engine_load -> EngineParams::multimodal -> LoadedEngine::mm_config()), + * and that config is what BaseProcessingInfo::ValidateNumItems refuses against. + * The caller that reaches ValidateNumItems on a live request is the OPENAI + * SERVER: it is the one place that installs the multimodal chat seam + * (server_main.cpp `chat.set_multimodal_chat_fn(...)`), and serving_chat.cpp + * gates the whole multimodal branch on that seam being set. So a server started + * with --language-model-only answers a multimodal chat request with HTTP 400 * "At most 0 image(s) may be provided in one prompt." rather than serving it. + * + * THIS ABI HAS NO MULTIMODAL CHAT REQUEST PATH YET, so on a C-ABI engine the + * two fields are RECORDED and read by nothing the ABI itself can reach. + * vllm_chat / vllm_chat_stream never install that seam. A chat request whose + * content array carries an `image_url` part is therefore answered as TEXT: the + * part is dropped, its text siblings still form the prompt, no limit is + * consulted, and language_model_only changes neither the status nor the body. + * Setting these fields configures the ENGINE — including an OpenAI server built + * on one — but it does not make a C-ABI chat call refuse an image. Carrying + * media across this ABI is a later version, and the refusal arm becomes + * reachable from here only when it lands. That is pinned behaviourally by + * tests/capi/test_capi.cpp ("capi: the v19 limits are RECORDED on a C-ABI + * engine; there is no multimodal request path to enforce them on"), so this + * paragraph cannot silently become false. * The memory win upstream also gets from zero limits (skipping the vision tower * weights, interfaces.py:293) is NOT in this version — it is wave L3, and until * it lands and is MEASURED this field must not be described as freeing VRAM. @@ -395,11 +419,15 @@ typedef struct vllm_model_params { * entry does not survive it (get_limit_per_prompt, :321-336). * * language_model_only: 0 => off (the zero value, byte-identical to pre-v19); - * nonzero => every modality limit becomes 0, which makes the engine REFUSE - * every multimodal request with "At most 0 (s) may be provided in - * one prompt." That refusal is the flag's main observable effect and it is - * live at v19; the tower-skip memory win it also produces upstream is not - * (wave L3). */ + * nonzero => every modality limit resolves to 0 on this engine's + * MultiModalConfig. On the OPENAI-SERVER path that is a refusal — HTTP 400 + * "At most 0 (s) may be provided in one prompt." — because the + * server installs the multimodal chat seam that reaches ValidateNumItems. On + * this ABI's own vllm_chat there is no multimodal request to refuse yet, so + * the field configures the engine without changing any C-ABI call's result; + * see the v19 note in the version log above for exactly what a C-ABI caller + * gets today. The tower-skip memory win upstream also produces is not here + * either (wave L3). */ int32_t language_model_only; /* limit_mm_per_prompt: the per-modality maximum input-item count, as the same * JSON object the flag takes. NULL/empty (the zero value) => no limit @@ -411,11 +439,14 @@ typedef struct vllm_model_params { * {"image": 16, "video": {"count": 1}} (mixed) * The option keys are validated exactly as upstream's per-modality * dataclasses do (video: num_frames/width/height, image: width/height, audio: - * length; each an integer > 0; anything else is refused, `extra="forbid"`) - * and then DROPPED: they size dummy inputs for memory profiling, which this - * engine does not do, and only `count` feeds the limit (:335). - * Invalid JSON, a non-object document, a negative count, an unknown option, - * or a non-integer value fails vllm_engine_load with + * length; each an integer > 0; anything else on those three is refused, + * `extra="forbid"`, :24,33,41) and then DROPPED: they size dummy inputs for + * memory profiling, which this engine does not do, and only `count` feeds the + * limit (:335). A modality outside those three is upstream's bare + * BaseDummyOptions (:233), which has no `extra="forbid"`, so its unknown keys + * are dropped rather than refused — mirrored, not invented. + * Invalid JSON, a non-object document, a negative count, or a refused option + * per the paragraph above fails vllm_engine_load with * VLLM_ERR_INVALID_ARGUMENT. Borrowed for the call only. */ const char* limit_mm_per_prompt; } vllm_model_params; diff --git a/include/vllm/config/multimodal.h b/include/vllm/config/multimodal.h index 435625dbf..68d4dc5f1 100644 --- a/include/vllm/config/multimodal.h +++ b/include/vllm/config/multimodal.h @@ -93,9 +93,13 @@ struct MultiModalConfig { // // Ported from `MultiModalConfig._validate_limit_per_prompt` // (multimodal.py:212-236) together with the DummyOptions dataclasses it feeds -// (:17-43). Upstream reaches this through argparse `type=parse_type(json.loads)` -// (_compute_kwargs, arg_utils.py:375-377), so the flag's value is a JSON OBJECT -// and every spelling below is the object's, not the flag's: +// (:17-45). Upstream reaches this through argparse `type=parse_type(json.loads)` +// (_compute_kwargs, arg_utils.py:379-381 — the plain-dict branch; the +// `union_dict_and_str` branch immediately above it at :374-378 is a DIFFERENT +// rule and is not the one `limit_per_prompt: dict[str, BaseDummyOptions]` takes, +// because that annotation contains no `str` arm and `dict[...]` reports +// `__module__ == "builtins"`, so `is_not_builtin` is False), so the flag's value +// is a JSON OBJECT and every spelling below is the object's, not the flag's: // // * the LEGACY format, count only — {"image": 16, "video": 2} (:87-88,220-222, // which rewrites a bare int to {"count": }); @@ -107,23 +111,37 @@ struct MultiModalConfig { // * a non-object document, or a value that is neither an int nor an object — // upstream's json.loads/pydantic pair rejects both; // * `count` absent-and-not-an-int, or NEGATIVE — `count: int = Field(999, -// ge=0)` (:20); -// * an option key the modality does not define, or an option <= 0 — every -// DummyOptions dataclass is `extra="forbid"` with `Field(None, gt=0)` -// (:23-43). video takes num_frames/width/height, image width/height, audio -// length, and any OTHER modality takes `count` alone (BaseDummyOptions, -// :233-234). -// Refusing rather than defaulting is the point: a typo'd limit that silently -// became 999 is a limit that is not there, which is exactly the failure this -// wave exists to remove. +// ge=0)` (:21), declared on BaseDummyOptions and therefore checked for EVERY +// modality; +// * for a BUILTIN modality only — video, image, audio, the three +// `_validate_limit_per_prompt` routes to a dataclass of their own +// (:226-232) — an option key the modality does not define, or an option +// <= 0: those three are the dataclasses carrying +// `config=ConfigDict(extra="forbid")` (:24,33,41) over `Field(None, gt=0)` +// (:28-30,37-38,45). video takes num_frames/width/height, image +// width/height, audio length. +// +// And the one that is deliberately NOT a refusal: any OTHER modality falls to +// the `else` at :233 and is built as a bare `BaseDummyOptions`, which is the ONE +// dummy-options dataclass declared WITHOUT `extra="forbid"` (:17-21). Pydantic's +// default `extra='ignore'` therefore applies, so an unknown option key on a +// non-builtin modality is DROPPED, not refused, and its value is never +// validated: upstream accepts `{"pointcloud": {"count": 2, "foo": 3}}` and +// yields `BaseDummyOptions(count=2)`. We mirror that — refusing it would refuse +// a document the reference accepts. Only `count` is validated there. +// +// Refusing rather than defaulting is the point wherever upstream refuses: a +// typo'd limit that silently became 999 is a limit that is not there, which is +// exactly the failure this wave exists to remove. // // DEVIATION, recorded: the option keys are parsed and VALIDATED, then DROPPED. // They size DUMMY multimodal inputs during memory profiling, a surface we do not // have (see the header comment above on `limit_per_prompt`'s mapped type), and // only `.count` participates in get_limit_per_prompt (:335). `ignored_options`, -// when non-null, collects "." for every option dropped, so the -// caller can ANNOUNCE the drop rather than let a user infer that `num_frames` -// took effect. +// when non-null, collects "." for every option dropped — both the +// builtin options validated above and the non-builtin extras upstream itself +// discards — so the caller can ANNOUNCE the drop rather than let a user infer +// that `num_frames` took effect. // // Throws std::invalid_argument, carrying the offending key/value, on anything // above. diff --git a/include/vllm/entrypoints/openai/chat_mm.h b/include/vllm/entrypoints/openai/chat_mm.h index 75d60b390..e0bf6f2d2 100644 --- a/include/vllm/entrypoints/openai/chat_mm.h +++ b/include/vllm/entrypoints/openai/chat_mm.h @@ -176,16 +176,29 @@ void ValidateChatMmLimits(const multimodal::BaseProcessingInfo& info, // The Qwen3-VL IMAGE chat seam's OWN supported limits — the `min()` fold's other // operand (context.py:392-405), which #686 recorded as undeclared. // -// Upstream's Qwen3-VL declares image and video UNLIMITED -// (get_supported_mm_limits, qwen3_vl.py) because its processor handles N of -// each. Ours handles exactly ONE image (MakeQwen3VLImageChatFn locates a single -// image part) and no video or audio at all, so the honest ceiling is -// {"image": 1} and every other modality is ABSENT — which context.py:414-415 -// reads as "not supported", limit 0. That is not a policy choice, it is this -// seam's implemented arm stated as a number, and AGENTS.md requires exactly -// that: an unimplemented arm is refused with a message naming the missing piece, -// never left to be discovered. A user limit can only LOWER it (the fold is a -// min), so `--limit-mm-per-prompt image=99` still refuses the second image. +// Upstream's Qwen3-VL declares image and video UNLIMITED — +// `get_supported_mm_limits` is not defined on Qwen3VLProcessingInfo at all +// (qwen3_vl.py:848 subclasses Qwen2VLProcessingInfo); it is INHERITED from +// qwen2_vl.py:851-852, `return {"image": None, "video": None}` — because its +// processor handles N of each. Ours handles exactly ONE image +// (MakeQwen3VLImageChatFn locates a single image part) and no video or audio at +// all, so the honest ceiling is {"image": 1} and every other modality is +// ABSENT — which context.py:414-415 reads as "not supported", limit 0. That is +// not a policy choice, it is this seam's implemented arm stated as a NUMBER. A +// user limit can only LOWER it (the fold is a min), so +// `--limit-mm-per-prompt image=99` still refuses the second image. +// +// WHAT THIS DOES NOT YET SATISFY (#758, found in the #749 review). AGENTS.md +// asks that an unimplemented arm be "refused with a message naming the missing +// piece". The number is stated here, but the message a client receives is +// upstream's generic "At most 0 video(s) may be provided in one prompt." — which +// names nothing, and which a user cannot tell apart from an operator having set +// `--limit-mm-per-prompt '{"video": 0}'`. The only signal today is by OMISSION: +// ValidateNumItems withholds the "Set `--limit-mm-per-prompt` to increase this +// limit." hint when raising the configured limit would not help. Changing the +// text is a deliberate divergence from a verbatim-ported message that three +// suites assert byte-for-byte, so it is owed to #758 with its own spec rather +// than folded in here. // // When the multi-image / video arms land they raise these numbers here, and // nothing else changes. diff --git a/src/vllm/config/multimodal.cpp b/src/vllm/config/multimodal.cpp index a720e1c5e..e67848fe3 100644 --- a/src/vllm/config/multimodal.cpp +++ b/src/vllm/config/multimodal.cpp @@ -1,4 +1,4 @@ -// Ported from: vllm/config/multimodal.py:17-43,212-236 @ 5559679229bc +// Ported from: vllm/config/multimodal.py:17-45,212-236 @ 5559679229bc // (see include/vllm/config/multimodal.h for scope and deviations). #include "vllm/config/multimodal.h" @@ -11,11 +11,18 @@ namespace vllm { namespace { -// The option keys each DummyOptions subclass declares, beyond `count` -// (multimodal.py:23-43). `extra="forbid"` is what makes an unlisted key an -// error rather than a silently-kept extra, so the lookup below is exhaustive by -// construction: a modality with no entry here is BaseDummyOptions, which -// declares NOTHING but `count` (:233-234). +// The modalities `_validate_limit_per_prompt` routes to a dataclass of their own +// (multimodal.py:226-231). Everything else falls to the `else` at :232-233 and +// is built as a bare BaseDummyOptions. +bool IsBuiltinModality(const std::string& modality) { + return modality == "video" || modality == "image" || modality == "audio"; +} + +// The option keys each BUILTIN DummyOptions subclass declares, beyond `count` +// (multimodal.py:24-45). Only those three carry +// `@dataclass(config=ConfigDict(extra="forbid"))` (:24,33,41), which is what +// makes an unlisted key an error rather than a silently-kept extra, so this +// lookup is exhaustive for exactly the modalities IsBuiltinModality names. bool IsKnownOption(const std::string& modality, const std::string& key) { if (modality == "video") { return key == "num_frames" || key == "width" || key == "height"; @@ -36,7 +43,8 @@ int ParseCount(const std::string& modality, const nlohmann::json& value) { Refuse("\"" + modality + "\".count must be an integer, got " + value.dump()); } const int64_t count = value.get(); - // count: int = Field(999, ge=0) (multimodal.py:20). + // count: int = Field(999, ge=0) (multimodal.py:21) — declared on + // BaseDummyOptions, so it is validated for EVERY modality, builtin or not. if (count < 0) { Refuse("\"" + modality + "\".count must be >= 0, got " + std::to_string(count)); @@ -80,12 +88,27 @@ std::map ParseLimitMmPerPromptJson( count = ParseCount(modality, option); continue; } + if (!IsBuiltinModality(modality)) { + // BaseDummyOptions (multimodal.py:17-21) is the ONE dummy-options + // dataclass declared WITHOUT `config=ConfigDict(extra="forbid")`, unlike + // the three at :24,33,41 — so pydantic's default `extra='ignore'` + // applies and an unlisted key on a modality outside image/video/audio is + // DROPPED, not refused, with its value never validated. Re-derived + // against the pinned oracle's own declarations under pydantic 2.12.5: + // `BaseDummyOptions(count=2, foo=3)` returns `BaseDummyOptions(count=2)` + // while `ImageDummyOptions(count=2, foo=3)` raises ValidationError. + // Refusing here instead would refuse a document upstream accepts. + if (ignored_options != nullptr) { + ignored_options->push_back(modality + "." + key); + } + continue; + } if (!IsKnownOption(modality, key)) { - // extra="forbid" (multimodal.py:23,32,39). Naming the key is the whole + // extra="forbid" (multimodal.py:24,33,41). Naming the key is the whole // value of the refusal: a dropped `num_frame` typo is invisible. Refuse("\"" + modality + "\" has no option \"" + key + "\""); } - // Field(None, gt=0) on every option (multimodal.py:26-28,35-36,42). + // Field(None, gt=0) on every option (multimodal.py:28-30,37-38,45). if (!option.is_number_integer() || option.get() <= 0) { Refuse("\"" + modality + "\".\"" + key + "\" must be an integer > 0, " "got " + option.dump()); diff --git a/src/vllm/entrypoints/openai/server_main.cpp b/src/vllm/entrypoints/openai/server_main.cpp index ef73b92a4..5d5a571dd 100644 --- a/src/vllm/entrypoints/openai/server_main.cpp +++ b/src/vllm/entrypoints/openai/server_main.cpp @@ -548,7 +548,10 @@ Args ParseArgs(int argc, char** argv) { a.multimodal.language_model_only = flag == "--language-model-only"; } else if (flag == "--limit-mm-per-prompt") { // arg_utils.py:1279 over a dict field => type=parse_type(json.loads) - // (arg_utils.py:375-377): the value is a JSON OBJECT. Parsed HERE, before + // (arg_utils.py:379-381 — the plain-dict branch, NOT the + // `union_dict_and_str` one at :374-378, which needs a `str` arm or a + // non-builtin hint that `dict[str, BaseDummyOptions]` does not have): + // the value is a JSON OBJECT. Parsed HERE, before // the multi-GB model load, for the same reason the parser dialects are: // a malformed limit costs a second rather than a full load, and it is // REFUSED rather than defaulted — a typo that silently became 999 is a @@ -576,9 +579,11 @@ Args ParseArgs(int argc, char** argv) { // of inferring that it worked. for (const std::string& key : ignored_options) { std::cerr << "server: --limit-mm-per-prompt " << key - << " accepted and IGNORED: it sizes dummy inputs for memory " - "profiling, a surface this engine does not have; only the " - "count is read\n"; + << " accepted and IGNORED: only the modality's count is read " + "here. A key upstream DECLARES sizes dummy inputs for " + "memory profiling, a surface this engine does not have; a " + "key it does not declare is one its BaseDummyOptions " + "fallback drops too\n"; } } else if (flag == "--version") { std::cout << "vllm.cpp " << vllm::Version() diff --git a/tests/capi/test_capi.cpp b/tests/capi/test_capi.cpp index 6eea48ff8..4a838d02c 100644 --- a/tests/capi/test_capi.cpp +++ b/tests/capi/test_capi.cpp @@ -253,10 +253,10 @@ vllm_engine* MakeSyntheticEngine() { // Chat-capable synthetic engine: same stack, but the chat serving is built with // an IN-VOCAB prompt seam (the tiny fixture vocab cannot spell a real chat // template), mirroring the api-server harness's InVocabChatPrompt. -vllm_engine* MakeSyntheticChatEngine() { +vllm_engine* MakeSyntheticChatEngine(EngineParams p) { const HfConfig c = MakeConfig(); - auto loaded = std::make_unique(c, MakeWeights(c), BuildFixture(), - SyntheticParams()); + auto loaded = + std::make_unique(c, MakeWeights(c), BuildFixture(), p); return vllm::capi::MakeEngineHandle( std::move(loaded), [](const std::vector& messages, @@ -270,6 +270,10 @@ vllm_engine* MakeSyntheticChatEngine() { }); } +vllm_engine* MakeSyntheticChatEngine() { + return MakeSyntheticChatEngine(SyntheticParams()); +} + vllm_sampling_params GreedyParams(int32_t max_tokens) { vllm_sampling_params sp = vllm_sampling_params_default(); sp.temperature = 0.0f; // greedy (argmax) -> deterministic. @@ -1330,6 +1334,58 @@ TEST_CASE("capi: EngineParams::multimodal reaches LoadedEngine::mm_config()") { } } +// The PIN for the ABI v19 paragraph in include/vllm.h. That paragraph is a +// permanent public contract, and the thing it must not claim is that setting +// these fields makes a C-ABI call REFUSE a multimodal request. It does not: +// ValidateNumItems is reached only behind the multimodal chat seam, and +// server_main.cpp is the sole caller of set_multimodal_chat_fn — vllm_chat and +// vllm_chat_stream never install one, so serving_chat.cpp's `if (mm_chat_fn_)` +// gate is never taken on this path and no MultiModalInputs is ever built. +// +// What a C-ABI caller gets today, asserted rather than described: the request +// PARSES (protocol.cpp does read `image_url` content parts), the image part is +// DROPPED, its text siblings still form the prompt, and the answer is an +// ordinary 200-shaped chat.completion — with language_model_only set, which on +// the server path would be an HTTP 400. Wire the seam into the ABI without +// revisiting that paragraph and this case goes red, which is the point. +TEST_CASE("capi: the v19 limits are RECORDED on a C-ABI engine; there is no " + "multimodal request path to enforce them on") { + EngineParams p = SyntheticParams(); + p.multimodal.language_model_only = true; // every modality limit => 0 + vllm_engine* eng = MakeSyntheticChatEngine(p); + REQUIRE(eng != nullptr); + + // A real OpenAI multimodal chat body: one text part, one image_url part. + const char* request = + "{\"messages\":[{\"role\":\"user\",\"content\":[" + "{\"type\":\"text\",\"text\":\"hello\"}," + "{\"type\":\"image_url\",\"image_url\":{\"url\":\"data:image/png;base64," + "iVBORw0KGgo=\"}}" + "]}],\"temperature\":0,\"max_tokens\":6}"; + char* response = nullptr; + const vllm_status st = vllm_chat(eng, request, &response); + CAPTURE(std::string(vllm_last_error() == nullptr ? "" : vllm_last_error())); + REQUIRE(st == VLLM_OK); + REQUIRE(response != nullptr); + const json body = json::parse(response); + CAPTURE(std::string(response)); + // NOT a refusal: served as text, exactly as if the image part were absent. + CHECK(body.at("object") == "chat.completion"); + CHECK(body.at("choices").size() == 1); + CHECK(!body.at("choices").at(0).at("message").at("content") + .get() + .empty()); + CHECK(body.count("error") == 0); + vllm_string_free(response); + vllm_engine_free(eng); + + // The limits ARE on the config all the same — recorded, just not consulted by + // anything this ABI can reach. That is the exact wording include/vllm.h owes. + const HfConfig c = MakeConfig(); + LoadedEngine e(c, MakeWeights(c), BuildFixture(), p); + CHECK(e.mm_config().GetLimitPerPrompt("image") == 0); +} + TEST_CASE("capi: kv_transfer_config parses and validates the connector name") { // A well-formed config naming a REGISTERED connector passes the gate and // reaches model load. diff --git a/tests/vllm/entrypoints/openai/test_serve_mm_limits.cpp b/tests/vllm/entrypoints/openai/test_serve_mm_limits.cpp index 9b893310d..87f884fb8 100644 --- a/tests/vllm/entrypoints/openai/test_serve_mm_limits.cpp +++ b/tests/vllm/entrypoints/openai/test_serve_mm_limits.cpp @@ -7,12 +7,17 @@ // reaching the config); // - vllm/engine/arg_utils.py:556 (`limit_mm_per_prompt: dict[...] = // get_field(MultiModalConfig, "limit_per_prompt")`), :1279, :1692, whose -// dict type resolves to `type=parse_type(json.loads)` (:375-377) — so the -// value is a JSON OBJECT; -// - vllm/config/multimodal.py:17-43 (the DummyOptions dataclasses: `count: -// int = Field(999, ge=0)`, `extra="forbid"`, every option `Field(None, -// gt=0)`) and :212-236 (`_validate_limit_per_prompt`, which rewrites a bare -// int to {"count": } and routes each modality to its own dataclass); +// dict type resolves to `type=parse_type(json.loads)` (:379-381, the +// plain-dict branch — the `union_dict_and_str` branch at :374-378 is a +// different rule and needs a `str` arm the annotation does not have) — so +// the value is a JSON OBJECT; +// - vllm/config/multimodal.py:17-45 (the DummyOptions dataclasses: `count: +// int = Field(999, ge=0)` on BaseDummyOptions :21; `extra="forbid"` on the +// video/image/audio subclasses ONLY, :24,33,41; every declared option +// `Field(None, gt=0)`, :28-30,37-38,45) and :212-236 +// (`_validate_limit_per_prompt`, which rewrites a bare int to +// {"count": } and routes each modality to its own dataclass, falling to +// the bare BaseDummyOptions at :233 for anything outside those three); // - vllm/config/multimodal.py:321-336 (`get_limit_per_prompt`, the precedence // the startup banner prints THROUGH so the flag ordering is observable). // All at 5559679229bc, the pinned oracle in .agents/upstream-sync.md. @@ -200,7 +205,7 @@ TEST_CASE("limit-mm-per-prompt: the CONFIGURABLE + MIXED objects (multimodal.py: CHECK(mixed.at("image") == 16); CHECK(mixed.at("video") == 1); - // An object with no `count` keeps the dataclass default (:20), not 0 — the + // An object with no `count` keeps the dataclass default (:21), not 0 — the // object exists to carry OPTIONS, so omitting the count is not a refusal. const std::map options_only = vllm::ParseLimitMmPerPromptJson(R"({"image": {"width": 512}})", nullptr); @@ -216,14 +221,13 @@ TEST_CASE("limit-mm-per-prompt: malformed input is REFUSED, never defaulted") { "\"image\"", // not an object R"({"image": "two"})", // count is not an int R"({"image": 2.5})", // count is not an int - R"({"image": -1})", // count: Field(999, ge=0), :20 + R"({"image": -1})", // count: Field(999, ge=0), :21 R"({"image": {"count": -1}})", // same, through the object form R"({"image": {"count": "two"}})", // same, wrong type - R"({"image": {"num_frames": 32}})", // image has no num_frames, :32-37 - R"({"video": {"fps": 2}})", // extra="forbid", :23 - R"({"audio": {"width": 2}})", // audio takes `length`, :39-43 - R"({"tactile": {"count": 1, "width": 2}})", // BaseDummyOptions: count ONLY - R"({"video": {"num_frames": 0}})", // Field(None, gt=0), :26 + R"({"image": {"num_frames": 32}})", // image has no num_frames, :33-38 + R"({"video": {"fps": 2}})", // extra="forbid", :24 + R"({"audio": {"width": 2}})", // audio takes `length`, :41-45 + R"({"video": {"num_frames": 0}})", // Field(None, gt=0), :28 R"({"video": {"num_frames": -4}})", // same R"({"image": {"width": "big"}})", // option is not an int }; @@ -245,12 +249,66 @@ TEST_CASE("limit-mm-per-prompt: malformed input is REFUSED, never defaulted") { } // An accepted modality name is NOT enumerated: upstream routes an unknown - // modality to BaseDummyOptions (:233-234) rather than rejecting it, because + // modality to BaseDummyOptions (:233) rather than rejecting it, because // the modality set is the model's, not the config's. CHECK(vllm::ParseLimitMmPerPromptJson(R"({"tactile": 3})", nullptr) .at("tactile") == 3); } +TEST_CASE("limit-mm-per-prompt: forbidding extras is the BUILTIN modalities' " + "rule, not a global one") { + // The divergence the #749 review caught. `extra="forbid"` is declared on + // VideoDummyOptions / ImageDummyOptions / AudioDummyOptions ONLY + // (multimodal.py:24,33,41). BaseDummyOptions — what :233's `else` builds for + // every OTHER modality name — is a plain `@dataclass` (:17-21), so pydantic's + // default `extra='ignore'` applies and an unlisted key is DROPPED with its + // value never validated. + // + // Re-derived at the pinned oracle 5559679229bc under pydantic 2.12.5, by + // declaring those two dataclasses verbatim and calling them: + // BaseDummyOptions(**{"count": 2, "foo": 3}) -> BaseDummyOptions(count=2) + // ImageDummyOptions(**{"count": 2, "foo": 3}) -> ValidationError + // Refusing the first would refuse a document the reference accepts, which is + // what this port did before the repair. + std::vector ignored; + const std::map limits = vllm::ParseLimitMmPerPromptJson( + R"({"pointcloud": {"count": 2, "foo": 3}})", &ignored); + CHECK(limits.at("pointcloud") == 2); + // Dropped, but ANNOUNCED — upstream discards it silently; we say so, exactly + // as we do for the builtin options we also drop. + REQUIRE(ignored.size() == 1); + CHECK(ignored.at(0) == "pointcloud.foo"); + + // The extra's VALUE is not validated either: `Field(None, gt=0)` guards + // declared fields, and `foo` is not one. `0` and a string both survive on a + // non-builtin modality... + CHECK(vllm::ParseLimitMmPerPromptJson(R"({"tactile": {"count": 1, "foo": 0}})", + nullptr) + .at("tactile") == 1); + CHECK(vllm::ParseLimitMmPerPromptJson( + R"({"tactile": {"count": 1, "foo": "big"}})", nullptr) + .at("tactile") == 1); + // ...while the same shapes on a BUILTIN modality are still refused, because + // there the key either is declared (and then gt=0 applies) or extra="forbid" + // rejects it. This half is what keeps the repair from becoming "accept + // everything". + const std::vector still_refused = { + R"({"image": {"count": 1, "foo": 3}})", + R"({"image": {"count": 1, "width": 0}})", + R"({"video": {"count": 1, "num_frames": "big"}})"}; + for (const std::string& doc : still_refused) { + CAPTURE(doc); + CHECK_THROWS_AS(vllm::ParseLimitMmPerPromptJson(doc, nullptr), + std::invalid_argument); + } + + // `count` is declared on BaseDummyOptions itself (:21), so ITS validation is + // global — a non-builtin modality does not escape `Field(999, ge=0)`. + CHECK_THROWS_AS( + vllm::ParseLimitMmPerPromptJson(R"({"tactile": {"count": -1}})", nullptr), + std::invalid_argument); +} + // ── The FLAGS (arg_utils.py:1276,1279) reaching the CONFIG ────────────────── TEST_CASE("serve flags: --language-model-only parses and ZEROES every limit") {