feat(ENG-MM-INPUT-PIPELINE): the multimodal limits reach a live request, so the flags stop being decoration (#607, #686)

[localai-bot]

Wave L2 of #607, and it closes #686.

L1 ported vLLM's per-modality input limits and the refusal they produce, but
nothing constructed a config on a live request — so the numbers were
unreachable, and #685's reviewer correctly recorded that the HTTP 400 arm was
unproven end to end and that "L2's reviewer should treat that as an owed gate,
not an inherited pass." This wave pays that.

Spec: .agents/specs/multimodal-track.md §1.5
(the L2 block now carries the outcome). Issues: #607, #686.

The three surfaces

1. Serve flags — src/vllm/entrypoints/openai/server_main.cpp

ours	upstream @ 5559679229bc
--language-model-only / --no-language-model-only	arg_utils.py:555,1276,1691
--limit-mm-per-prompt '<json>'	arg_utils.py:556,1279,1692
ParseLimitMmPerPromptJson (src/vllm/config/multimodal.cpp)	multimodal.py:212-236 + the DummyOptions dataclasses :17-43

Both spellings of the boolean, because _compute_kwargs gives a bool field
argparse.BooleanOptionalAction (arg_utils.py:346-348) — a recipe that turns
the flag off explicitly must not die on an unknown argument. The dict value is a
JSON object (type=parse_type(json.loads), arg_utils.py:379-381 — the plain-dict branch, corrected from :375-377 in the review round), and all
three of upstream's object spellings parse: legacy count-only, configurable
{"count": N, …}, and the two mixed.

Malformed input is refused before the model load, never defaulted — a
mistyped limit that silently became 999 is a limit that is not there. Every
refusal is upstream's own: a non-object document, a negative count
(count: int = Field(999, ge=0)), an unknown per-modality option on
image/video/audio (extra="forbid"; corrected in the review round — every OTHER
modality falls to BaseDummyOptions, which has no extra="forbid", so its
unknown keys are DROPPED, not refused), a non-positive option (Field(None, gt=0)). The profiling
options are validated, dropped (only .count feeds get_limit_per_prompt,
:335) and the drop is announced per key.

kAcceptedInertArgs was checked first and is untouched: #606 deliberately
excluded --language-model-only as "a real capability gap", and L2 resolves that
by implementing it rather than by adding an entry, so that table still holds only
genuine no-ops. .agents/specs/serve-recipe-args.md is reconciled to say so.

NAMED RESIDUAL: upstream's dotted spelling (--limit-mm-per-prompt.image 2)
is a FlexibleArgumentParser feature (argparse_utils.py:389-425) that applies
equally to --kv-transfer-config and --speculative-config, which this server
also takes as JSON only. Adding it for one flag would be the bespoke path; it
belongs to a parser brick covering all three.

2. C-ABI field — include/vllm.h, VLLM_ABI_VERSION 18 → 19

vllm_model_params.language_model_only (int32_t) and .limit_mm_per_prompt
(const char*, the same JSON object the flag takes). Appended at the end of the
struct, following the convention v14/v16/v18 set: a versioned /* ── … (ABI v19) ── */ block on the field, a v19 — … paragraph in the header's version log
stating what a zero-initialised v18 struct gets, and the JSON-string shape
following the v9 kv_transfer_config precedent — a dict-valued vLLM flag
crosses this ABI as its own JSON rather than as a fixed struct of modalities the
ABI would then owe forever. Malformed input fails vllm_engine_load with
VLLM_ERR_INVALID_ARGUMENT.

Both surfaces land on EngineParams::multimodal → LoadedEngine::mm_config(),
so the flags and the ABI resolve one config object per engine and cannot
drift.

3. The call site — src/vllm/entrypoints/openai/chat_mm.cpp

MakeQwen3VLImageChatFn now runs ValidateChatMmLimits (the port of the
chat_utils.py:648-662 tracker) as step 0, over a BaseProcessingInfo folding
the engine's config with the seam's own declared ceiling,
Qwen3VLChatSupportedMmLimits() == {"image": 1}.

That declaration answers the question #686 left open — "the model's own
supported limit (the min() fold's other operand) — today nothing declares
one". This seam locates a single image part and routes no video or audio, so its
honest ceiling is one image and every other modality is absent, which
context.py:414-415 reads as limit 0. A user limit can only lower it, so
--limit-mm-per-prompt '{"image": 99}' still refuses the second image — and then
correctly omits the Set --limit-mm-per-prompt to increase this limit. hint,
because raising the user's limit would not help.

The e2e 400 test, and exactly what it distinguishes

tests/vllm/entrypoints/openai/test_api_server.cpp, three legs through the real
ApiServer::handle_chat_completions:

1. the real production seam (MakeQwen3VLImageChatFn) + a three-image body
   → 400 / BadRequestError / "At most 1 image(s) may be provided in one prompt.", and asserted != 500 and != 200 and no choices key;
2. the same server, a within-limit request → 200 with a real completion,
   so the 400 is a limit decision and not "multimodal is off"; the same seam
   then refuses three;
3. a seam throwing a non-validation exception → 500 /
   InternalServerError, so "status == 400" cannot be satisfied by a handler
   that answers 400 to every seam failure.

Plus --language-model-only → 400 "At most 0 image(s) … Set --limit-mm-per-prompt to increase this limit." with a text request on the same
server still 200.

RED first, behaviourally (these are verbatim from the pre-implementation
run):

CHECK_THROWS_AS( mm_fn(ImageMessages(3, uri)), vllm::v1::InputValidationError )
  threw a DIFFERENT exception: "ExpandImagePlaceholders: more image placeholders than grids"
FATAL ERROR: expected --language-model-only to refuse an image request
CHECK( refused.status == 400 )                       values: CHECK( 500 == 400 )
CHECK( j.at("error").at("type") == "BadRequestError" ) values: CHECK( "InternalServerError" == BadRequestError )
CHECK( h.server.handle_chat_completions(ImageBody(3)).status == 400 ) values: CHECK( 200 == 400 )

A correction to #686, recorded rather than quietly fixed

The RED run says something the issue does not. Against the production seam
the pre-L2 behaviour was not the truncated 200 #686 describes — it was an
HTTP 500: MakeQwen3VLImageChatFn injects one placeholder marker per image
part while routing only the first image, so ExpandImagePlaceholders raised
"more image placeholders than grids" and the client got a server fault carrying
an internal message. The truncated 200 is real for the validate-then-build seam
shape and is pinned as its own leg (the CHECK( 200 == 400 ) above). Both are
wrong the same way — neither is upstream's refusal — so the issue's diagnosis
(first image, break, no refusal) stands and only its stated consequence was
understated. .agents/specs/mm-serving.md is corrected accordingly, as the issue
asked.

Mutation proof

Restored with cp + touch (not cp -a, which preserves mtime and makes ninja
skip the relink), rebuilt and re-verified green between every mutation. All four
mutated files verified byte-for-byte identical to the working tree afterwards by
md5.

mutation	result	caught by
(a1) flag→config: a.multimodal.language_model_only = false;	test_serve_mm_limits 8/10, 7 assertions	CHECK(Contains(run.output, "server: multimodal limits language-model-only=ON")), image=0, and CHECK_FALSE(Contains(run.output, "image=4")) in the precedence case
(a2) config→engine: mm_config_() instead of mm_config_(params.multimodal)	test_capi 56/57, 3 assertions	CHECK(e.mm_config().GetLimitPerPrompt("image") == 2)
(b) call-site wiring removed from the seam	test_openai_api_server 54/56 and test_chat_mm 9/11	CHECK(refused.status == 400) → 500; CHECK(j.at("error").at("message") == "At most 1 image(s)…")
(c) refusal re-typed to std::runtime_error	test_openai_api_server 54/56	CHECK(refused.status == 400) → CHECK(500 == 400) and CHECK("InternalServerError" == BadRequestError)

(c) is the one worth reading twice: it is exactly the failure mode L1's
single-refusal-type design avoided, and it is now pinned by a live call site. Note
the message text survived unchanged — a test asserting only the message would
have passed it. The status assertion is what caught it. An existing guard also
catches it earlier: L1's test_processing_limits goes 5/19 under the same
mutation, so the unit layer sees it too; the new e2e leg is what proves the HTTP
consequence.

Gates — verified on aarch64, not x86

Ran on kairos-4db2 (Thor, 14 cores) in the baked vllmcpp-build:aarch64
container, CPU-only (-DVLLM_CPP_CUDA=OFF). aarch64 is a CI-supported lane
(build-test-cpu-arm64), so this is legitimate coverage, but it is not an
x86 run.

• Clean rebuild (required: include/vllm.h changed) — 1326/1326 targets,
  0 warnings, 0 errors under -Werror.
• Full ctest -j 6: 446/447 passed, 2 skipped.
• test_serve_mm_limits 10/10 (101 assertions) — new
• test_chat_mm 11/11 (126 assertions)
• test_openai_api_server 56/56 (638 assertions)
• test_capi 57/57 (528 assertions)

The one failure is not mine and is already tracked. test_op_parity throws
[json.exception.type_error.302] type must be string, but is null at
tests/parity/test_op_parity.cpp:1989. I built and ran it from a pristine
origin/main tree on the same host: identical failure, same file, same line.
That is #737 ("main red: test_op_parity throws on the MiniMax-Music3 golden
manifest"), filed earlier today.

Checkers, run explicitly:

• check-windows-portability.py — the only errors are
  src/vllm/multimodal/video_engine.cpp:21,59,64 (VideoEngine uses unguarded POSIX stat in native Windows build #648, pre-existing).
  None of my files appear.
• check-device-leakage.py — OK (DSR 32 == baseline 32, ratchet holds). My
  files contribute zero DSR entries; the server_main.cpp DSR-ALLOWs in
  the listing are the pre-existing VT_BENCH_PROFILE_CONTROL guards whose line
  numbers shifted.
• check-surface-coverage.py — OK.
• check-doc-checkpoint.py --commit HEAD — OK.
• check-commit-trailers.py --range $(git merge-base origin/main HEAD)..HEAD —
  OK: commit trailer contract (run explicitly, per agent-preflight silently SKIPS commit-trailers when the branch is behind main, and still prints "All gates green" #653).

What this does NOT claim

No memory win. Nothing gates vision-tower construction on the limits, so
--language-model-only changes what the server accepts, not what it
allocates. That is wave L3 and it is owed with a measured RSS reduction.
docs/USAGE.md, docs/STATUS.md and the roadmap row all say so explicitly.

One call site of two. process_inputs_mm (input_processor.cpp:321,352,
upstream's context.py:461) is still unwired, and this is a scope judgement a
reviewer should rule on rather than a silent omission. It needs the per-model
get_supported_mm_limits() hook that L1 already recorded as absent (the models
that would implement it are the M2 towers). Wiring it first would mean inventing
a supported-limits source inside the engine, which the mirror rule forbids. The
chat call site is wired because it has a concrete source — the seam's own
implemented arm — and every path the OpenAI server can reach today goes through
it.

L4 (the qwen3_next.py:325 kernel gate) is untouched, as scoped.

🤖 Generated with Claude Code

---

Review repairs (e8e6c891a, aarch64)

The fresh review returned FAIL on finding 1 with finding 2 a real mirror
divergence, and confirmed the core L2 deliverable sound. The implementation is
unchanged apart from one parser rule; everything else was a false or imprecise
claim.

1 (MEDIUM, blocking) — include/vllm.h claimed an enforcement the C ABI
cannot reach. The v19 block said the two fields are "ENFORCED, not recorded"
and that an engine loaded with language_model_only answers a multimodal
request with At most 0 image(s)…. Re-verified at this head:
grep -c set_multimodal_chat_fn src/capi/vllm_c.cpp is 0, the seam is
installed exactly once at server_main.cpp:1117, and serving_chat.cpp:677
gates the whole multimodal branch on if (mm_chat_fn_). So on a C-ABI engine
mm_inputs stays empty, ValidateChatMmLimits never runs, an image_url part
is dropped, and the flag changes nothing a vllm_chat caller can observe.

Took remediation (a), correct the wording, not (b). (b) means building a
C-ABI multimodal request path — a new capability, not an L2 repair — and the
header now says exactly that, alongside where enforcement does live and what a
C-ABI caller actually gets. It is pinned behaviourally, not just reworded: a
new test_capi case builds a language_model_only engine, sends a real
image_url chat body through vllm_chat, and asserts VLLM_OK + a
chat.completion + no error — the opposite of the 400 the same config
produces on the server path.

2 (LOW, real divergence) — extra="forbid" is the builtin modalities' rule,
not a global one. Re-derived rather than inherited: fetched multimodal.py at
the pinned 5559679229bc, read the four declarations, and ran them under
pydantic 2.12.5.

BaseDummyOptions(**{"count": 2, "foo": 3})  -> BaseDummyOptions(count=2)
ImageDummyOptions(**{"count": 2, "foo": 3}) -> ValidationError
BaseDummyOptions(**{"count": -1})           -> ValidationError

BaseDummyOptions (:17-21) — what _validate_limit_per_prompt's else at
:233 builds for any modality outside image/video/audio — is the only one of
the four declared without config=ConfigDict(extra="forbid") (cf. :24,33,41).
So '{"pointcloud": {"count": 2, "foo": 3}}' is accepted upstream and was
refused here. Now mirrored: a non-builtin modality's unlisted key is dropped
(and announced, as every drop is) with its value unvalidated; the three builtin
modalities keep extra="forbid" + Field(None, gt=0); count — declared on
BaseDummyOptions itself — stays validated for all of them. No ABI bump: v19 is
unreleased and lands in this PR.

3 — the arg_utils.py citation. :375-377 is inside the
union_dict_and_str branch (:374-378). limit_per_prompt: dict[str, BaseDummyOptions] takes the plain-dict branch, parse_type(json.loads), at
:379-381 — the annotation has no str arm, and dict[…] reports
__module__ == "builtins", so is_not_builtin is False. Fixed in all five
places it appeared (the review named four; the fifth was
include/vllm/config/multimodal.h:97) and above in this body.

4 — the ceiling is a number, not yet a message. chat_mm.h claimed
Qwen3VLChatSupportedMmLimits() satisfies AGENTS.md's "an unimplemented arm is
refused with a message naming the missing piece". It does not: the client gets
upstream's generic At most 0 video(s) may be provided in one prompt.,
indistinguishable from a configured limit — the only present signal is the
withheld --limit-mm-per-prompt hint. Claim withdrawn, behaviour owed to
#758, filed for it. Not fixed in flow deliberately: naming the arm diverges
from a verbatim-ported message three suites assert byte-for-byte, so it takes
its own spec and fresh review. Also corrected there —
get_supported_mm_limits is not on Qwen3VLProcessingInfo; it is inherited
from Qwen2VLProcessingInfo at qwen2_vl.py:851-852 ({"image": None, "video": None}), which qwen3_vl.py:848 subclasses.

5 — docs/USAGE.md rendered the hint without the backticks
context.py:426 carries; the console example below it had them. Both match the
real string now.

While re-deriving, the dataclass anchors themselves proved off by one to four
lines (:20 for count, :23,32,39 for the forbid decorators, :26-28,… for
the option fields, :17-43 for the block). Corrected everywhere against the
lines actually read.

Repair-round gates — aarch64, not x86

kairos-4db2 (Thor, 14 cores), baked vllmcpp-build:aarch64, CPU-only
(-DVLLM_CPP_CUDA=OFF).

• Clean rebuild (required — include/vllm.h changed): 1355/1355 targets,
  0 warnings, 0 errors under -Werror.
• ctest -j 6: 456/457, 2 skipped. The one failure is test_op_parity
  ([json.exception.type_error.302] at test_op_parity.cpp:1989) — main red: test_op_parity throws on the MiniMax-Music3 golden manifest — null where a string is required #737,
  already reproduced from pristine main.
• test_serve_mm_limits 11/11 (109 assertions) · test_chat_mm 11/11
  (126) · test_capi 58/58 (536) · test_openai_api_server 56/56 ·
  test_processing_limits 19/19 (78) · test_multimodal_config 7/7 (21).

A correction to this PR's own gate line. test_openai_api_server's
assertion count is not a pin: three runs of one binary gave 632 / 648 / 651,
because SSE-chunk loops assert per chunk received. Its case count, 56/56, is
stable and is the number to quote. The 638 recorded at the L2 landing was one
sample of that spread, not a target.

Checkers, run explicitly: check-commit-trailers --range $(git merge-base origin/main HEAD)..HEAD OK (per #653) · check-doc-checkpoint --commit HEAD
OK · check-device-leakage OK (DSR 32 == baseline 32) ·
check-windows-portability OK (verified, not assumed — #648 fixed it) ·
check-public-doc-tables / check-surface-coverage / check-agent-record OK.
Preflight's remaining reds are audit-live-rows / test_audit_live_rows
(#731; the stale ACTIVE row it names is
MODEL-MUSIC-minimax-music3-…, not mine) and test_cpu_x86_llamacpp_floor
(#618, NO_QUIET_WINDOW at load 209 on a contended box).

Mutations

cp + touch + rebuild, re-verified green between each, both files md5-identical
to the working tree afterwards.

mutation	result	caught by
IsBuiltinModality forced to true — the pre-repair "forbid extras everywhere"	test_serve_mm_limits 10 passed / 1 failed, Status FAILURE	the new case THREW limit_mm_per_prompt: "pointcloud" has no option "foo"
an image-part refusal wired into vllm_chat — the change that would make the v19 paragraph false	test_capi 57 passed / 1 failed	REQUIRE( st == VLLM_OK ) → REQUIRE( 1 == 0 ), logging vllm_chat: At most 0 image(s) may be provided in one prompt.

Worth reading twice: under the first mutation doctest printed
assertions: 100 | 100 passed | 0 failed — a thrown case contributes no
failed assertion. Only the case count and Status: FAILURE! caught it.

An earlier form of that mutation (deleting the escape block outright) failed to
build on -Werror=unused-function, and the stale binary re-ran and printed
Status: SUCCESS! — the mutation was re-shaped to keep the function used
rather than accepted as "not caught".