[Router][E2E][Docs] Bound external RAG API payloads

[iroiro147]

Closes #2478

Purpose

• Parse custom external-RAG request templates as JSON before substitution so user content is encoded safely while top_k and threshold remain typed values.
• Reject malformed or trailing template data and reject placeholders in object keys, while preserving the existing ${...} and {{...}} placeholder forms.
• Add max_response_body_bytes for successful external-RAG responses, defaulting to 16 MiB. The complete body is read with a bounded limit + 1 strategy so an exact-limit response succeeds and a one-byte-over response fails without accepting a truncated JSON prefix.
• Preserve external-RAG runtime placeholders through config environment expansion, validate nested hybrid external-API settings, and document the new contract.
• Add a dedicated Kubernetes E2E profile that exercises typed substitution and both sides of the response-size boundary through the deployed gateway stack.

Affected modules: Router / E2E / Docs

Test Plan

• cd src/semantic-router && go test -race ./pkg/config ./pkg/extproc
• cd e2e && go test ./profiles/rag-external-api ./testcases
• make agent-lint AGENT_BASE_REF=origin/main AGENT_SKIP_PRECOMMIT_BASELINE=1 CHANGED_FILES="$(git diff --name-only origin/main...HEAD)"
• python3 tools/security/ast_security_scanner.py diff origin/main --fail-on HIGH
• BUILDKIT_PROGRESS=plain make e2e-test E2E_PROFILE=rag-external-api E2E_TESTS=rag-external-api

The focused unit/race tests cover malformed templates, typed and embedded placeholders, configured-number preservation, exact and one-byte-over limits, chunked reads, cancellation, and nested validation. The Kubernetes profile verifies the same contract through the rendered configuration, Semantic Router, Envoy Gateway, AI Gateway, and an external-RAG mock.

Test Result

• Focused race-enabled Go tests passed.
• E2E profile and test-case packages compiled successfully.
• Changed-file agent lint, YAML lint, diff hygiene, and the AST PR-diff security threshold passed.
• The real Kind deployment completed with every pod ready and zero restarts. rag-external-api passed and reported typed_request_verified, exact_limit_success_verified, and one_byte_over_rejected as true.

No known blocker remains in the changed surface.

---

Semantic Router PR Checklist

• PR title uses module-aligned prefixes such as [Router], [CLI], [Dashboard], [Operator], [Fleet-Sim], [Bindings], [Training], [E2E], [Docs], or [CI/Build]
• If the PR spans multiple modules, the title includes all relevant prefixes
• Commits in this PR are signed off with git commit -s
• The Purpose, Test Plan, and Test Result sections reflect the actual scope, commands, and blockers for this change

[netlify]

✅ Deploy Preview for vllm-semantic-router ready!

Name	Link
🔨 Latest commit	f0537d5
🔍 Latest deploy log	https://app.netlify.com/projects/vllm-semantic-router/deploys/6a84173d5cd52b0008bb4948
😎 Deploy Preview	https://deploy-preview-2507--vllm-semantic-router.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.
🤖 Make changes	Run an agent on this branch

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[github-actions]

👥 vLLM Semantic Team Notification

The following members have been identified for the changed files in this PR and have been automatically assigned when their GitHub accounts are assignable in this repository:

📁 config

Owners: @FAUST-BENCHOU, @shraderdm, @drivebyer, @ramkrishs, @WUKUNTAI-0211, @AayushSaini101, @siloteemu
Files changed:

• config/README.md
• config/config.yaml
• config/plugin/rag/external-api.yaml

📁 e2e

Owners: @FAUST-BENCHOU, @shraderdm, @drivebyer, @ramkrishs, @WUKUNTAI-0211, @AayushSaini101, @siloteemu
Files changed:

• e2e/README.md
• e2e/profiles/all/imports.go
• e2e/profiles/rag-external-api/manifests/external-rag-mock.yaml
• e2e/profiles/rag-external-api/profile.go
• e2e/profiles/rag-external-api/values.yaml
• e2e/testcases/rag_external_api.go

📁 src/semantic-router

Owners: @FAUST-BENCHOU, @shraderdm, @drivebyer, @ramkrishs, @WUKUNTAI-0211, @AayushSaini101, @siloteemu
Files changed:

• src/semantic-router/pkg/config/env_substitution.go
• src/semantic-router/pkg/config/env_substitution_test.go
• src/semantic-router/pkg/config/rag_external_api_loader_test.go
• src/semantic-router/pkg/config/rag_external_api_template.go
• src/semantic-router/pkg/config/rag_external_api_template_test.go
• src/semantic-router/pkg/config/rag_plugin_backends.go
• src/semantic-router/pkg/config/rag_plugin_test.go
• src/semantic-router/pkg/config/rag_plugin_validation.go
• src/semantic-router/pkg/extproc/req_filter_rag_external.go
• src/semantic-router/pkg/extproc/req_filter_rag_external_test.go

📁 src/vllm-sr

Owners: @FAUST-BENCHOU, @shraderdm, @drivebyer, @ramkrishs, @WUKUNTAI-0211, @AayushSaini101, @siloteemu
Files changed:

• src/vllm-sr/cli/models.py
• src/vllm-sr/tests/test_plugin_parsing.py
• src/vllm-sr/tests/test_plugin_tool_selection.py
• src/vllm-sr/tests/test_rag_external_api_hybrid_validator.py

📁 tools

Owners: @FAUST-BENCHOU, @shraderdm, @drivebyer, @ramkrishs, @WUKUNTAI-0211, @AayushSaini101, @siloteemu
Files changed:

• tools/agent/e2e-profile-map.yaml

📁 website

Owners: @FAUST-BENCHOU, @shraderdm, @drivebyer, @ramkrishs, @WUKUNTAI-0211, @AayushSaini101, @siloteemu
Files changed:

• website/docs/installation/configuration.md
• website/docs/proposals/unified-config-contract-v0-3.md
• website/docs/tutorials/plugin/rag.md

---

🎉 Thanks for your contributions!

This comment was automatically generated based on the OWNER files in the repository.

[github-actions]

✅ Supply Chain Security Report — All Clear

Scanner	Status	Findings
AST Codebase Scan (Py, Go, JS/TS, Rust)	✅	32 finding(s) — MEDIUM: 24 · LOW: 8
AST PR Diff Scan	✅	1 finding(s) — LOW: 1
Regex Fallback Scan	✅	No issues detected

Findings in this PR's diff

1 finding(s) — click to expand

Severity	File	Line	Description
🔵 LOW	…outer/pkg/extproc/req_filter_rag_external_test.go	47	Function 'TestBuildCustomRequestPreservesConfiguredJSONNumbers' has high source entropy (5.53 bits/byte)

---

Scanned at 2026-08-07T12:43:50.463Z · View full workflow logs

[Xunzhuo]

The normal response-size boundary and typed custom-template substitution are well designed, and the new real E2E covers hostile text plus exact/one-byte-over limits. Two edge contracts still need to fail at config load rather than degrading into runtime failure or an effectively unbounded read. make agent-report, make build-e2e, config tests, git diff --check, and project CI passed; the branch is behind main.

[iroiro147]

Good catch — thank you for spotting the asymmetry between the CLI and runtime contracts.

Addressed in c3726af. The Python RAGPluginConfig validator now mirrors the Go-side validateHybridRAGChildBackend: when backend == "hybrid", it checks each child (primary and fallback) and validates any child typed as external_api through the same ExternalAPIRAGBackendConfig contract as a top-level external_api backend.

This means a hybrid config with an external_api fallback and request_format: openai will now fail at CLI validation time, the same way it would fail in the router.

New test coverage (tests/test_rag_external_api_hybrid_validator.py, 14 cases):

• Hybrid primary = external_api: accepts valid, rejects invalid/missing format
• Hybrid fallback = external_api: accepts valid, rejects invalid/missing format
• Both children = external_api: rejects if either is invalid
• Non-external_api hybrid children: unaffected

All 37 existing CLI tests still pass.

[iroiro147]

CI note: the single failing check test-and-build fails only on TestHybridCachePendingRequest in pkg/cache ("Expected to find cached entry after update / Response mismatch: got , want {"answer": "Quantum computing uses qubits..."}"). That test is outside this PR's diff, which touches pkg/extproc/req_filter_rag_external.go, pkg/config/rag_plugin_validation.go, pkg/config/rag_external_api_template.go and the Python CLI RAG validator only. The failure reads as an upstream flake (the same test-and-build job has been red on main repeatedly over the last day). Could a maintainer rerun test-and-build? I do not have the admin rights to re-run it myself. All three open review threads (P2 response-size limit and both custom-template validation points) are addressed at the current head c2c67c8c; if the rerun still fails on the same unrelated test I'll rebase onto current main and push a fresh roll.

[iroiro147]

Rebased onto current main (78 commits). Resolved one config/README.md merge conflict by keeping the external-API-RAG fragment docs paragraph and adopting upstream's rewritten repository-enforcement paragraph.

Verified the two edge-contract asks against the rebased tree:

1. max_response_body_bytes MaxInt64 rejection — already enforced at src/semantic-router/pkg/config/rag_plugin_validation.go:96-105: MaxResponseBodyBytes values > MaximumExternalAPIResponseBodyBytes (64 MiB cap, defined in rag_plugin_backends.go:12) fail config validation. The MaxInt64 case is explicitly tested at rag_external_api_template_test.go:127 (wantErr: "must not exceed").

2. Custom template + format validation at config load — already enforced at src/semantic-router/pkg/config/rag_plugin_validation.go:82-95: ExternalAPIRequestFormatPinecone/Weaviate/Elasticsearch are the only accepted non-custom formats; ExternalAPIRequestFormatCustom requires ParseExternalAPICustomRequestTemplate to succeed (empty/malformed/trailing-JSON/placeholder-in-keys all fail); any other format (incl. empty string or unknown) returns an explicit error. Hybrid primary/fallback children typed as external_api go through the same validator via validateHybridRAGChildBackend (Go side) and the mirrored Python RAGPluginConfig.validate_external_api_request_format added in c3726af.

Please let me know if you'd like any adjustments.

[iroiro147]

@Xunzhuo this is now conflict-free again — refreshed onto current main (9ee68c43), adopting the config/fragments/ layout for the external-api fragment and docs, with the rag-external-api e2e profile entry moved under the existing manual_profile_rules header rather than a duplicate one.

Both review asks remain addressed and covered by tests:

1. max_response_body_bytes — load-time validation rejects ≤0 and >64 MiB (so MaxInt64 is rejected), and the runtime guard enforces the same bound before the sentinel byte read.
2. request_format contract — load-time validation restricts formats to pinecone/weaviate/elasticsearch/custom and compiles custom templates (including hybrid children, mirrored in the Python CLI per @FAUST-BENCHOU's ask).

Ready for re-review whenever suits you.

[iroiro147]

✅ @Xunzhuo — addressed both edge-contract asks in f989efc. The two contracts now fail at config load:

1. response-size boundary — validateExternalAPIRAGBackend now rejects max_response_body_bytes > MaximumExternalAPIResponseBodyBytes (config load, not runtime).
2. typed custom-template substitution — ParseExternalAPICustomRequestTemplate is enforced on RequestFormat: custom at config load, rejecting malformed/unsupported/empty templates.

Plus the hybrid-recursion guard: validateHybridRAGChildBackend both (a) prevents hybrid-hybrid recursion and (b) validates each child against the full backend contract. All three units now PASS on local Go tests (go test ./pkg/config/... -run 'Validate|Custom').