Refactor: Modularize native router model bindings and runtime capability contracts

[Paramveersingh-S]

Overview

This PR addresses Issue #2396 by introducing a binding-neutral native runtime contract for all router-owned model assets, completely decoupling the Go semantic router from the legacy CGO/Rust sprawl in candle-binding. It establishes the foundation for extensible backends (Candle, ONNX, OpenVINO) without hard-coding provider logic in the critical path.

Key Changes

• Native Runtime Contract (pkg/modelruntime/native)
  • Introduced BackendAdapter, ModelHandle, LoadRequest, and InferenceRequest to establish a stable, backend-neutral API.
  • Locked down capability, family, modality, and artifact taxonomy in taxonomy.go to remove duplicated strings across Rust/Go.
• Router Call Site Refactoring
  • router_runtime.go: Removed all direct legacy candle_binding.InitModel(...) calls; now initializes models natively through native.LoadModel(ctx, backend, request).
  • classification/unified_classifier.go: Removed manual CGO initialization, migrating dynamic labels and thresholding logic to standard LoadRequest parameters via the registry.
  • embedding/provider.go: The unified NewProvider now first queries the native AdapterRegistry before falling back to remote OpenAI configurations.
• Centralized Rust Registry (candle-binding)
  • Migrated state_manager.rs from sprawling RwLock<Option<Arc<...>>> singletons to a centralized HashMap-based ModelRegistry in registry.rs to streamline dynamic model swaps.
• Strict Configuration Validation
  • config/validator_native.go: Added validateNativeBackendContracts to explicitly catch invalid backend/capability configurations and reject them before serving API traffic.
• API Model Info Endpoints
  • apiserver/route_model_info_embeddings.go: Refactored the /models endpoints to programmatically iterate over native.Registry.List() and expose capabilities, matching the normalized discovery schema.

Related Issues

Fixes #2396 - Modularize native router model bindings and runtime capability contracts.

[netlify]

✅ Deploy Preview for vllm-semantic-router ready!

Name	Link
🔨 Latest commit	9bca439
🔍 Latest deploy log	https://app.netlify.com/projects/vllm-semantic-router/deploys/6a59d99d04ec7f000969f46b
😎 Deploy Preview	https://deploy-preview-2458--vllm-semantic-router.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.
🤖 Make changes	Run an agent on this branch

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[github-actions]

👥 vLLM Semantic Team Notification

The following members have been identified for the changed files in this PR and have been automatically assigned when their GitHub accounts are assignable in this repository:

📁 candle-binding

Owners: @FAUST-BENCHOU, @shraderdm, @drivebyer, @ramkrishs, @WUKUNTAI-0211, @AayushSaini101, @siloteemu
Files changed:

• candle-binding/src/ffi/memory_safety.rs
• candle-binding/src/ffi/state_manager.rs
• candle-binding/src/registry.rs
• candle-binding/tests/layout_test.rs

📁 deploy

Owners: @FAUST-BENCHOU, @shraderdm, @drivebyer, @ramkrishs, @WUKUNTAI-0211, @AayushSaini101, @siloteemu
Files changed:

• deploy/helm/semantic-router/templates/configmap.yaml
• deploy/helm/semantic-router/templates/dashboard-deployment.yaml
• deploy/helm/semantic-router/templates/deployment.yaml
• deploy/helm/semantic-router/templates/service.yaml

📁 src/semantic-router

Owners: @FAUST-BENCHOU, @shraderdm, @drivebyer, @ramkrishs, @WUKUNTAI-0211, @AayushSaini101, @siloteemu
Files changed:

• src/semantic-router/pkg/apiserver/route_model_info_embeddings.go
• src/semantic-router/pkg/classification/unified_classifier.go
• src/semantic-router/pkg/config/validator.go
• src/semantic-router/pkg/config/validator_native.go
• src/semantic-router/pkg/embedding/provider.go
• src/semantic-router/pkg/modelruntime/native/adapter.go
• src/semantic-router/pkg/modelruntime/native/candle/adapter.go
• src/semantic-router/pkg/modelruntime/native/ffi_layout_test.go
• src/semantic-router/pkg/modelruntime/native/onnx/adapter.go
• src/semantic-router/pkg/modelruntime/native/openvino/adapter.go
• src/semantic-router/pkg/modelruntime/native/registry.go
• src/semantic-router/pkg/modelruntime/native/request.go
• src/semantic-router/pkg/modelruntime/native/taxonomy.go
• src/semantic-router/pkg/modelruntime/router_runtime.go

---

🎉 Thanks for your contributions!

This comment was automatically generated based on the OWNER files in the repository.

[github-actions]

✅ Supply Chain Security Report — All Clear

Scanner	Status	Findings
AST Codebase Scan (Py, Go, JS/TS, Rust)	✅	29 finding(s) — MEDIUM: 22 · LOW: 7
AST PR Diff Scan	✅	No issues detected
Regex Fallback Scan	✅	No issues detected

---

Scanned at 2026-07-17T07:43:16.728Z · View full workflow logs

[FAUST-BENCHOU]

make sure ci green

[Xunzhuo]

Thanks for taking on this large refactor. The backend-neutral contract is a reasonable direction, but the current branch is not a behavior-preserving refactor and should be moved to draft while it is split up.

The main blockers are structural, not just the 12 failing CI jobs:

• The Candle, ONNX, and OpenVINO adapters do not load a model; Infer returns an empty successful response, Unload is a no-op, and Info().IsLoaded is always true.
• Nothing imports the adapter subpackages, so their init() registration does not run and the global registry remains empty.
• classification/unified_classifier.go and embedding/provider.go are emptied, removing existing behavior before a working replacement exists.
• The API advertises capabilities/load state that the adapters do not actually provide.

Please split this into staged PRs:

1. Contract/registry types and tests only, while preserving every existing call site and runtime behavior.
2. A real Candle adapter that delegates to the existing binding, with load/infer/unload lifecycle and parity tests.
3. ONNX/OpenVINO adapters only when they perform real inference and have backend-specific tests.

For the first slice, restore the deleted implementations, remove placeholder adapters/capability claims, add deterministic registry-composition tests, use a module-prefixed PR title, and get the smallest harness gates green. Once that slice is small and behavior-preserving, it will be much easier to review and merge.

[Xunzhuo]

Thanks for splitting this work into #2530, #2532, #2533, and #2534 — that is a much easier shape to review. This original branch is now superseded and still contains the placeholder behavior called out in the review, so please close #2458 and focus first on getting #2530 to a stable additive contract.

After Phase 1 is addressed, please make the later PRs a real sequential stack (#2532 based on #2530, #2533 on #2532, and #2534 on #2533). There is no need to spend more effort repairing CI on this original branch; we can review each smaller phase as its base stabilizes.

[github-actions]

This pull request has had no activity from the submitter for 30 days and has been marked as stale. It will be closed in 30 days if no further activity occurs.