Routing

Routing

Nenya's routing system dynamically selects the optimal upstream provider for each request based on multiple factors including latency, cost, and model capabilities.

Multi-Account Routing

When a provider has multiple API keys configured via accounts, Nenya selects the optimal account per request during target building:

• AccountSelector — The SelectCredentialForModel() interface decouples account selection from routing, enabling testable pluggable strategies
• Credential on target — The selected Credential and AccountName are stored directly on UpstreamTarget, avoiding double-selection at dispatch time
• Per-account billing — Spend tracking, quota polling, and exhaustion filtering all use the selected account ID

The AccountPool (LRU-based selection) is the built-in implementation:

Feature	Behavior
Selection	Least-recently-used strategy with mutex-protected access
Error classification	6 error classes (auth, rate_limit, quota, capacity, server, unknown) with semantic cooldowns
Exponential backoff	Failed accounts receive exponentially increasing backoff (±10% jitter) up to level 15
Model locks	Individual model+provider combinations locked during cooldown
State persistence	Account state persisted in <provider>.accounts.json files
Unified response writer	Passthrough responses use response_writer.go (writeUpstreamResponse / writeUpstreamBytesResponse) with consistent header/body handling

Agent-Based Routing

When a client sends a request with model: "agent-name", Nenya resolves the agent and routes through its model list. Agents define:

• Strategy: "fallback" (try first, then next on failure) or "round-robin" (distribute across targets)
• Model list: ordered list of models to try
• Circuit breaker: per target, protects against cascading failures
• Cooldown: seconds to skip a model after a retryable error

Fallback Chain

Request → Model A → fails → Model B → fails → Model C → succeeds
         ↓
     Circuit breaker trips A, B → tries next

Round-Robin

Request 1 → Model A
Request 2 → Model B
Request 3 → Model C
Request 4 → Model A (wraps)

Billing-Aware Routing

The routing pipeline integrates billing at four stages:

Stage 1: Target Building

• Free-only stripping: For providers with free_only: true, paid models are removed via isPaidModelOnFreeOnlyProvider() using a conservative detection priority (config list → pricing → name suffix)
• Account selection: Each target gets a credential and account ID from AccountSelector, enabling per-account tracking downstream

Stage 2: Exhaustion Filtering

• filterExhaustedTargets() removes all targets where BillingTracker.IsExhausted(provider, account) returns true
• Runs before scoring, preventing routing to spent accounts
• Logs a debug message per skipped target

Stage 3: Billing-Aware Scoring

When routing_strategy is "balanced", billing weights are applied:

• Free-only providers: Forced economy mode (×1.5 cost weight)
• Mixed providers: Free models get +0.4 scoring bonus via isModelFreeInProvider() using an optimistic detection priority (config list → name suffix → pricing)
• Subscription/credit: Use configured routing_cost_weight

Stage 4: Budget Enforcement

• Before dispatching, prepareAndSend() checks agent.BudgetLimitUSD
• If the agent's total spend across all targets ≥ limit, the target is skipped
• Emits nenya_budget_limit_rejected metric
• Independent of provider-level exhaustion

Configuration

{
  "agents": {
    "build": {
      "models": ["gemini-3-flash", "deepseek-chat"],
      "budget_limit_usd": 50.0
    }
  },
  "governance": {
    "routing_strategy": "balanced",
    "routing_latency_weight": 1.0,
    "routing_cost_weight": 0.0
  }
}

Balanced Scoring Algorithm

When auto_reorder_by_latency is enabled and routing_strategy is "balanced", targets are scored using a multi-dimensional formula:

score = (latency_normalized * latency_weight)
      - (cost_normalized * cost_weight)
      + capability_boost
      + score_bonus

• latency_normalized: (maxLat - modelLatency) / (maxLat - minLat) — higher = faster
• cost_normalized: (modelCost - minCost) / (maxCost - minCost) — lower = cheaper
• score_bonus: Per-model override from static registry or discovered metadata
• capability_boost: +0.1 per matching capability, -0.1 per mismatch

Normalization

Scores are normalized using min-max normalization:

• Latency: normalized = (max_lat - current_lat) / (max_lat - min_lat)
• Cost: normalized = (current_cost - min_cost) / (max_cost - min_cost)

Cost Tracking

When OpenRouter is configured, pricing data is fetched from its /v1/models endpoint. Per-request costs calculated from usage data via PricingEntry.CalculateCost() and recorded in CostTracker (microUSD precision). Cost data exposed via /statsz and /v1/models.

Capability Matching

Models scored based on ModelMetadata capabilities:

• CapToolCalls: function/tool calling
• CapReasoning: reasoning tokens
• CapVision: image inputs
• CapContentArrays: complex content arrays
• CapStreamOptions: stream_options.include_usage
• CapAutoToolChoice: tool_choice: "auto"

Latency-Aware Reordering

When governance.auto_reorder_by_latency is enabled, targets are sorted by historical median latency (fastest first). A ±5% random jitter is applied to prevent thundering herd.

Configuration

{
  "governance": {
    "auto_reorder_by_latency": true,
    "routing_strategy": "balanced",
    "routing_latency_weight": 1.0,
    "routing_cost_weight": 0.0
  }
}

Latency Tracker

infra.LatencyTracker maintains per-model sorted sample buffers:

• Incremental binary-search insertion — O(n) per Record()
• At most 100 samples per model
• Stale entries (no updates for 1 hour) evicted automatically
• ±5% jitter injectable for deterministic testing

Circuit Breaker

Each agent+provider+model combination is tracked independently:

State	Behavior
Closed	Normal operation. Semantic error classification. Trips to Open after failure_threshold failures (immediately for auth errors).
Open	All requests skipped. After cooldown_seconds, transitions to HalfOpen.
HalfOpen	Allows up to half_open_max_requests probe requests (default 3). All succeed → Closed. Any fail → Open.
ForceOpen	Immediately opened (HTTP 429). Extends cooldown for quota exhaustion.

Checked twice per target: during BuildTargetList and before sending.

Semantic Error Classification

6 error classes for circuit breaker and backoff:

Class	HTTP Status	Backoff	Lock
auth	401, 403	5 min cooldown	Immediate trip
rate_limit	429	Exponential (500ms base)	Model-level lock
quota	400 (quota msg)	Exponential (60s base)	Long cooldown
capacity	503 (overload)	Exponential (30s base)	Model-level lock
server	500, 502, 504	5s cooldown	Short backoff
unknown	Other 4xx/5xx	No backoff	No lock

Exponential Backoff

Per-model backoff levels (0–15 capped):

• Base delays: rate_limit=500ms, quota=60s, capacity=30s
• ±10% jitter per level
• Reset on RecordSuccess

Model-Level Locks

Locked models are skipped during target list construction. Auto-clear on success.

Check state via /statsz:

{
  "circuit_breakers": {
    "build:gemini:gemini-3-flash": "closed",
    "build:deepseek:deepseek-reasoner": "open"
  }
}

See Also

• Configuration — Agent model selectors and regex patterns
• Providers — Provider capabilities
• Architecture — Request lifecycle and graceful degradation