[Router] Require classifier runtime weights so partial downloads heal - #2820
Open
olifarhaan wants to merge 2 commits into
Open
[Router] Require classifier runtime weights so partial downloads heal#2820olifarhaan wants to merge 2 commits into
olifarhaan wants to merge 2 commits into
Conversation
On the mmBERT-32K backend the category, PII, and jailbreak classifiers load through TraditionalModernBertTokenClassifier, which hard-reads config.json, tokenizer.json, and model.safetensors from the model root. Completeness only demanded config.json plus any weight-shaped file anywhere in the tree, so an interrupted download that left a companion mapping and a nested adapter blob behind read as complete and was never re-fetched, while init kept failing. Mirror addEmbeddingModelRequiredFiles and record the root weights and tokenizer for these three models. Restrict it to mmBERT-32K, which initialises with no fallback; the other backend auto-detects LoRA directories that legitimately carry adapter weights instead, and requiring root weights there would strand a valid model in a permanent re-download loop. Signed-off-by: Mohammad Ali Farhan <alifarhan231087@gmail.com>
Pin the required-file contract for all three classifiers, the partial-download directory that used to read as complete, the fully downloaded control, and the LoRA backend that must keep the looser heuristic. Signed-off-by: Mohammad Ali Farhan <alifarhan231087@gmail.com>
✅ Deploy Preview for vllm-semantic-router ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
github-actions
Bot
assigned
AayushSaini101,
drivebyer,
FAUST-BENCHOU,
ramkrishs,
shraderdm and
WUKUNTAI-0211
Contributor
👥 vLLM Semantic Team NotificationThe following members have been identified for the changed files in this PR and have been automatically assigned when their GitHub accounts are assignable in this repository: 📁
|
Contributor
✅ Supply Chain Security Report — All Clear
Scanned at |
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #2669
Purpose
On the mmBERT-32K backend, which is the default for all three, the category, PII, and jailbreak classifiers load through
TraditionalModernBertTokenClassifier::new_with_variant. It hard-readsconfig.json,tokenizer.json, andmodel.safetensorsfrom the model root and fails outright if any is missingCompleteness only demanded
config.jsonplus any weight-shaped file, andhasModelWeightsmatches*.safetensors,*.bin, and*.onnxrecursively. So an interrupted download that left the companion mapping and a nested adapter blob behind read as complete, was never re-fetched, and since these classifiers register withbestEffort: false, every restart failed the same wayaddClassifierModelRequiredFilesmirrors the existingaddEmbeddingModelRequiredFilesfrom #2172 and records the root weights and tokenizer for the three models. Module affected:RouterOn the LoRA caveat raised in the issue: this is gated on
use_mmbert_32k, which is exactly the flag selecting the initialiser that has no LoRA fallback. With it off, PII and jailbreak init auto-detects LoRA directories that legitimately carry adapter weights instead of root weights, so those keep the looser heuristic. A test pins that carve-outTest Plan
To confirm the tests are not vacuous, revert each part of the fix and re-run
Test Result
All four new tests fail on the base for the right reason, for example:
Reverting each part of the change in turn fails at least one test every time:
addClassifierModelRequiredFilescall droppedTestBuildModelSpecsRequiresClassifierRuntimeWeights,TestPartialClassifierDirReportedIncompleteuse_mmbert_32kgate dropped, so LoRA also demands root weightsTestLoRAClassifierKeepsHeuristicCompletenessmake go-lintreports 0 issues,make check-go-mod-tidyandmake agent-ci-lintpass, and every applicable pre-commit hook passes.make test-semantic-routerreports one failure,TestHybridCachePendingRequestinpkg/cache, which is a known flake unrelated to this change. It fails on other contributors' PRs that do not touchpkg/cacheeither (#2530, #2507, #2810), and it passed on my own PR #2819. The test writes to the Milvus-backed hybrid cache, sleeps a fixed 100ms "for indexing", then asserts the entry is findable, so a loaded runner fails it. Locally it cannot run at all without a Milvus instance, which is whymake test-semantic-routerdefaultsSKIP_MILVUS_TESTS=true. It is currently failing on several unrelated PRs and onmainitselfOne thing I did not change:
DefaultRequiredFilesstill allows the loose nested-weight heuristic for every other model. Tightening that globally would be a much wider change than this issueThe
models/prefix is now a named constant, since the linter flags the third repetition this change would have introduced