Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 29 additions & 0 deletions docs/filters/http/ai/grid_route.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
<!-- Generated by: cargo xtask generate-filter-docs -->
<!-- Do not edit manually -->

# `grid_route`

Selects an upstream cluster from a static site/capability descriptor by matching the inference model name from a configured request header.

## Configuration Notes

**Behavior:** - If `ctx.cluster` is already set by an earlier filter, the selection is preserved and no metadata is written. - If the model header is absent, the filter returns `Continue` without routing. - If the header is blank, oversized, or non-UTF-8, the filter rejects with 400. - If a matching candidate is found, `ctx.cluster` is set and bounded route-decision metadata is written. - If no matching candidate is found, the filter rejects with 404.

**Scoring:** candidates are scored deterministically. Fresh candidates score 0; stale candidates receive −100. Candidates on `local_site` receive +10. Freshness is stronger than local preference. First configured candidate wins on equal scores.

**Metadata:** on successful selection, bounded in-process filter metadata is written under the `grid.route.` namespace (`kind`, `name`, `site`, `cluster`, `local_site`). No HTTP forwarding headers are written. No request-time database, control-plane, or metrics lookups are performed.

**Scope:** only `inference_model` candidates are matched in this release. MCP tool routing and A2A agent routing are separate PRs.

## Configuration

| Field | Type | Required | Description |
|-------|------|---------|-------------|
| `candidates` | CandidateConfig[] | yes | Static list of route candidates. |
| `candidates[].cluster` | string | yes | Cluster name to select when this candidate is chosen. |
| `candidates[].fresh` | bool | no | Whether this candidate is fresh (default: `true`). |
| `candidates[].kind` | `inference_model` | yes | Capability kind. |
| `candidates[].name` | string | yes | Capability name (model name, tool name, or agent name). |
| `candidates[].site` | string | yes | Site that owns this capability. |
| `local_site` | string | yes | Name of the local site. |
| `model_header` | string | no | Header name that carries the model name (default: `X-Model`). |
6 changes: 6 additions & 0 deletions docs/filters/reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,12 @@

Built-in filters organized by protocol and category.

## HTTP / AI

| Filter | Feature | Description |
|--------|---------|-------------|
| [`grid_route`](http/ai/grid_route.md) | - | Selects an upstream cluster from a static site/capability descriptor by matching the inference model name from a configured request header. |

## HTTP / Observability

| Filter | Feature | Description |
Expand Down
6 changes: 6 additions & 0 deletions examples/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,12 @@ page.

## Configs

### Ai

| File | Description |
| ------ | ------------- |
| [grid-route-inference.yaml](configs/ai/grid-route-inference.yaml) | Routes requests to different upstream clusters based on the inference model name extracted from a configured request header. The header value is set by an earlier filter such as `json_body_field` |

### Branching

| File | Description |
Expand Down
60 changes: 60 additions & 0 deletions examples/configs/ai/grid-route-inference.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
# Grid Route: Inference Model Routing
#
# Routes requests to different upstream clusters based on the inference
# model name extracted from a configured request header. The header
# value is set by an earlier filter such as `json_body_field`.
#
# `grid_route` selects a candidate by static scoring and writes the
# cluster name into `ctx.cluster`. The downstream `load_balancer` reads
# that cluster. No `router` filter is required between them: if a
# `router` filter is present in the chain, it will see that
# `ctx.cluster` is already set and skip its own selection.
#
# Usage:
# cargo run -p praxis-proxy -- -c examples/configs/ai/grid-route-inference.yaml

listeners:
- name: proxy
address: "0.0.0.0:8080"
filter_chains:
- main

filter_chains:
- name: main
filters:
# Extract model name from the JSON body into a request header.
- filter: json_body_field
field: model
header: X-Model

# Route to the cluster that serves the requested model.
# Selects by highest score: fresh > stale, local > remote.
- filter: grid_route
local_site: site-a
model_header: X-Model
candidates:
- kind: inference_model
name: granite-3.3-8b
site: site-a
cluster: granite-local
fresh: true

- kind: inference_model
name: llama-3.2-8b
site: site-b
cluster: llama-remote
fresh: true

# Forward to whichever cluster grid_route selected.
- filter: load_balancer
clusters:
- name: granite-local
endpoints:
- "127.0.0.1:8001"
- name: llama-remote
endpoints:
- "127.0.0.1:8002"

admin:
address: "127.0.0.1:9901"
shutdown_timeout_secs: 5
Loading
Loading