feat(filter): add grid route inference routing#802
Conversation
|
PR too large: 1145 lines added (limit: 750, excludes Cargo files, tests, docs, examples, and benchmarks). Please split into smaller PRs. Add |
888f077 to
44e41fe
Compare
Signed-off-by: Brent Salisbury <bsalisbu@redhat.com>
44e41fe to
3bf742e
Compare
praxis-bot
left a comment
There was a problem hiding this comment.
Review: grid_route inference routing filter
Purpose: Adds a grid_route HTTP filter that selects an upstream cluster from a static candidate list by matching the inference model name from a configured request header.
Overall: Well-structured implementation with clean separation between descriptor validation (descriptor.rs) and route selection logic (route.rs). Candidate scoring is deterministic, well-documented, and thoroughly unit-tested. Five medium findings to address, primarily around a missing trait implementation and test coverage gaps.
Rebase required: filter/src/builtins/http/ai/mod.rs is created from scratch on this branch (the diff shows @@ -0,0), but the file already exists on current main with existing AI filter modules (agentic, inference, openai, prompt_enrich). Similarly, http/mod.rs and builtins/mod.rs have diverged. Merging without rebasing will produce conflicts that could break existing AI filters if resolved incorrectly.
| Severity | Count |
|---|---|
| Critical | 0 |
| Large | 0 |
| Medium | 5 |
|
Missing Signed-off-by: 4609115. All commits require sign-off (via |
Signed-off-by: Brent Salisbury <bsalisbu@redhat.com>
4609115 to
441cfc3
Compare
|
Closing to move to the AI repo |
Summary
This PR adds the
grid_routeHTTP filter for static AI Grid inference routing.grid_routeselects an upstream cluster from a configured list of route candidates. It is intended to sit after a request-inspection filter such asjson_body_field, which extracts the requested model into a header, and beforeload_balancer, which forwards to the selected cluster.The filter provides the first Praxis-side routing primitive needed for Grid gateway-to-gateway inference routing:
X-Modelinference_modelroute candidates withname,site,cluster, andfreshlocal_sitewhen freshness is equalctx.clusterfor downstream filtersctx.clusterif an earlier filter already selected onegrid.route.*namespaceThis PR also adds:
filter/src/builtins/http/ai/grid/module structuregrid_routeregistration in the built-in filter registrygrid_routejson_body_field -> grid_route -> load_balancerMCP tool routing and A2A routing are intentionally out of scope for this PR. This change only accepts and routes
inference_modelcandidates.Behavior
Routing is deterministic and local to the request.
Selection flow:
ctx.clusteris already set,grid_routedoes not override it.400.inference_modelcandidates are scored.ctx.cluster.404.Candidate scoring:
Validation
grid_routeunit coverage for descriptor parsing, candidate validation, model-header handling, route scoring,selected-cluster behavior, and metadata emission.
examples/configs/ai/grid-route-inference.yaml.nerdalert/praxis-research-spikes.