[Discussion] Model Failover Policy

[Thenujan-Nagaratnam]

TL;DR

We need model failover to support three cases: changing only the model, changing the backend or region, and changing to a different provider. When a fallback uses another provider, the gateway must transform the original request again and apply the fallback provider's authentication before sending it.

I have come up with the following feasible approaches:

1. Let the policy engine (external processor) make the provider calls and control the complete failover loop (including doing transformations and authentications).
2. Let Envoy retry through priority targets, with an upstream external processor applying the correct transformation and authentication for every attempt.
3. Re-enter provider-specific routes through an internal loopback (policy engine acting as a client and calling the gateway), so each fallback uses the existing policies on that route.
4. Let the first request fail, suspend that target for a period, and send later requests to its fallbacks.

Approach 2 is the strongest option because Envoy continues to handle retries, timeouts, connections, and health while the upstream processor handles provider-specific transformation and authentication.

• But it introduces a heavy architectural change
  • Adding a new external processor
  • Moving upstream authentication to upstream ext_proc
  • Moving all transformation policies to upstream ext_proc

Approach 3 is the main alternative when reusing the existing policies is more important than keeping the request path simple.

• But it will rerun all policies including rate-limit, analytics and etc
  • We can avoid it by adding execution conditions

Approach 4 is simpler, but it provides a weaker guarantee: the request that discovers the problem still fails. Only later requests use a fallback. (All other gateways support retrying in the same failing request)

Summary

The AI Gateway needs to retry a failed model invocation against the ordered fallback list configured for its primary target. A target is more than a model name: it identifies the provider, model, endpoint, credentials and request/response adapter required for one invocation.

The design must support all of these cases:

1. Same provider and same endpoint, different model. For example, retry gpt-4.1 as gpt-4.1-mini using the same OpenAI endpoint and credentials.
2. Same provider, different endpoint. For example, retry an Azure OpenAI deployment in a backup region represented by another upstreamDefinition.
3. Different provider. For example, retry an OpenAI request using Anthropic, including request transformation, response normalization, endpoint selection, and different authentication.

Ordinary Envoy retries are sufficient only when every attempt sends effectively the same upstream request. They do not, by themselves, change a model in the body, translate the payload to another provider's schema, switch authentication, or normalize a different response format. The central design question is therefore:

Where should the gateway build the provider-specific request and apply authentication whenever a primary or fallback target is selected, while retaining Envoy's mature timeout, connection-pooling, cancellation, health, and telemetry behavior?

Feasible architecture approaches

The following four approaches fit our gateway architecture. Approaches 1, 2, and 3 can transparently apply the three required failover forms to the active request. Approach 4 can route later requests across the same three target combinations after allowing the detecting request to fail. All four can reuse, extract, or invoke the existing provider transformation and upstream-authentication logic. Approaches that move responsibility to clients, rely on hedged duplicate calls, or depend on unreliable route-reselection tricks are intentionally excluded.

Approach 1: Policy engine calls each fallback provider

The policy engine receives the canonical request, performs each outbound provider call itself, transforms and authenticates each attempt, and returns the first accepted response to Envoy.

How it covers the requirements

• Changing only the model is a body transformation before the next call.
• Changing endpoints selects a different HTTP client destination.
• Changing providers selects a different adapter, credentials, and destination.

Advantages

• Full control over target selection, error classification, transformation, authentication, and response validation.
• Straightforward access to structured LLM data rather than proxy-filter lifecycle state.
• Does not depend on Envoy's currently alpha upstream HTTP filter support.
• Easy to implement provider-specific rules and semantic checks.

Disadvantages

• Model failover policy needs to handle all authentication and transformation in its logic.
• Since the policy invokes and sends the response back to the policy chain, the response chain before this policy will be entirely skipped (there might be some response guardrails and etc)
• The policy engine becomes an HTTP reverse proxy and must implement pooling, DNS, TLS, proxying, cancellation, backpressure, streaming, deadlines, and resource limits correctly.
• Envoy no longer sees each provider as its actual upstream, reducing the usefulness of native health, circuit-breaking, and per-cluster telemetry.
• Request and response bodies cross the Envoy-to-policy-engine boundary and then another network boundary.
• A slow or overloaded policy engine becomes the bottleneck for all inference traffic.
• It mixes policy evaluation with transport orchestration and increases the blast radius of policy-engine failures.

---

Approach 2: Envoy retries while an upstream processor prepares each attempt

Compile every failover target into Envoy priority levels. Envoy's router owns the failover decision, retries, and advancement through priorities. A target-specific upstream HTTP filter or upstream ext_proc runs after backend selection, restores the canonical request, performs that backend's transformation, and injects its credentials.

1. A listener/router-level processor runs once, discovers the requested model, preserves the unmodified request body, and can cause route rematching.
2. A per-cluster upstream processor runs again for every upstream attempt. It resolves the translator for the backend Envoy selected and translates again from the preserved original request. Response translation belongs to the processor instance for the winning attempt.

Failover is still native Envoy behavior. The policy engine does not maintain the retry loop or choose the next fallback. This separation is important: it keeps the failover state machine in Envoy while making provider transformation attempt-scoped.

For our gateway, two targets may resolve to the same physical address but carry different target identities and model adapters. This represents “same endpoint, different model” while allowing each attempt to rebuild the body correctly.

Envoy router
  priority 0 -> endpoint A + OpenAI/gpt-4.1 metadata
  priority 1 -> endpoint A + OpenAI/gpt-4.1-mini metadata
  priority 2 -> endpoint B + OpenAI/gpt-4.1-mini metadata
  priority 3 -> endpoint C + Anthropic/Claude metadata
          |
          v
per-backend upstream adapter -> authentication -> network

Advantages

• Envoy retains ownership of retries, timeouts, cancellation, connection pools, health, circuit breakers, and upstream telemetry.
• All three required failover modes use the same abstraction.
• Backend selection and transformation occur at the right lifecycle point (during each retry attempt): Envoy chooses a priority target, then the selected backend's transformation runs before bytes are sent upstream.
• Common downstream policies run once, while transformation and authentication run for every attempt.
• Ordered priorities give deterministic fallback behavior.

Disadvantages and risks

• Envoy documents upstream HTTP filters as alpha. This is the most material production-readiness concern.
• It introduces a new upstream external processor and have to move all upstream authentication and transformation policies to this ext_proc.
• Needs a downstream-to-upstream state contract for the canonical request and plan identity. That state must remain request-scoped, bounded, and valid across retries.
• Envoy notes that local replies generated by upstream filters do not trigger router retries. Adapter and authentication failures need an explicit error path rather than relying on ordinary retry behavior.

---

Approach 3: Policy engine retries through the same Envoy route

The policy engine invokes the same Envoy route again after an eligible upstream failure. The request re-enters through a protected loopback listener or internal address with trusted attempt context indicating that the route should select the next fallback target. This lets the same route execute the provider-specific transformation, authentication, and upstream configuration required for that fallback.

Advantages

• Reuses existing provider transformation and upstream-authentication policies with limited refactoring.
• Supports ordered chains and all three required failover forms.
• Avoids depending on the alpha upstream HTTP filter lifecycle.
• The orchestrator can inspect normalized errors and apply target-specific retry rules.

Disadvantages and risks

• Adds an internal proxy hop for every fallback attempt.
• Common policies, analytics, quotas, and tracing may run more than once unless policies are explicitly scoped as logical-request or attempt policies.
• Requires strict recursion limits and a protected internal routing mechanism that clients cannot invoke or influence.
• The original body must remain replayable, and the total deadline and cancellation signal must span every re-entry.
• One logical request appears as several internal requests unless correlation and accounting are designed explicitly.
• Streaming still cannot fail over after response commitment.
• Rate limiting and billing can be applied twice if internal attempts are not distinguished correctly.

---

Approach 4: Fail the first request and temporarily skip that target

Allow the request that discovers a failure to fail normally. Record the selected target as suspended for a configured duration. While it is suspended, new requests that select the same primary target start from the first available entry in that target's fallbacks list.

request 1 -> primary target -> failure returned to client
                              -> suspend primary for cooldown

request 2 -> primary is suspended -> fallback[0]
request 3 -> primary is suspended -> fallback[0]

cooldown expires -> one probe request tries primary
                 -> success: restore primary
                 -> failure: return failure and suspend again

Suspension applies to the resolved target identity, which includes provider, model, and upstreamDefinition. The implementation may also maintain endpoint-level health so a transport outage can suspend every model target sharing an endpoint, while a model-specific error suspends only that model target.

Advantages

• Does not replay the triggering POST, avoiding duplicate inference, duplicate billing, and request-body replay complexity.
• Works with existing routing, transformation, authentication, and upstream definitions because each new request is routed normally to its selected target.
• Does not require an upstream ext_proc to run again within one request.
• Naturally supports long-running and streaming requests because the selected target never changes during a request.
• Closely matches passive outlier-detection and circuit-breaker patterns already used in Envoy.
• Can use Retry-After as the suspension duration for provider rate limiting.

Disadvantages and risks

• The request that detects each outage fails. This is availability steering for subsequent requests, not transparent per-request failover.
• With sparse traffic, failure is discovered late. With concurrent traffic, several requests may fail before suspension becomes visible.
• Every gateway replica needs consistent enough suspension state. Proxy-local state may continue sending traffic from other replicas; shared state adds a distributed dependency.
• Recovery needs a half-open state so only a limited probe reaches the primary after cooldown. Without it, many requests can retry the primary simultaneously.
• Each fallback can cause its own first-client failure before being suspended, producing several visible failures during a wider outage.
• Error classification must distinguish model, endpoint, authentication, configuration, and client failures.
• A single transient failure can suspend a healthy target unless failure thresholds are configurable.
• It does not meet a strict requirement that the original client request succeed through a fallback.

---

Comparison

Approach	Same endpoint, different model	Different endpoint	Different provider	Envoy remains provider-facing transport	Reuse of existing transformation policies	Production readiness for WSO2	Implementation complexity
1. Policy engine calls each provider	Yes: policy engine changes the model	Yes: policy engine selects the endpoint	Yes: policy engine selects transformation, auth, and provider	No: the policy engine becomes the provider-facing HTTP client	Medium: policy implementations are reusable, but require a new direct-call execution path	Low: WSO2 must build and operate provider-grade HTTP transport in the policy engine	Very high
2. Envoy retries; upstream processor prepares attempts	Yes: requires per-attempt model rewriting	Yes: requires a separately configured upstream target	Yes: requires per-target transformation, auth, and transport configuration	Yes: Envoy owns retries, pools, TLS, timeouts, and cancellation	Medium: transformation logic is reusable after it is exposed to the upstream per-attempt phase	Medium: Envoy upstream HTTP filters are alpha and WSO2 has not yet implemented this execution scope	High
3. Policy engine retries through the same route	Yes: retry context selects the next model	Yes: retry context selects the next upstream	Yes: the re-entered route selects the provider policies and upstream	Yes, with an extra loopback pass: Envoy still makes every provider call	High: every loopback request enters the current route-level policy pipeline again	Medium: built from known components, but recursion, policy scoping, correlation, and replay are new	High
4. Fail once and temporarily skip the target	Later requests only	Later requests only	Later requests only	Yes: each request follows the normal Envoy upstream path	High: later requests use the existing policy pipeline for the selected target	Medium: passive health is established, but model-aware suspension, replica consistency, and half-open recovery are new	Medium locally; high with shared state

Recommended direction

Two approaches deserve a focused prototype for transparent per-request failover:

1. Approach 2: Envoy retries while an upstream processor prepares each attempt. This best preserves Envoy's data-plane strengths and places transformation/authentication at the correct lifecycle point.
2. Approach 3: Policy engine retries through the same Envoy route. This avoids upstream-filter maturity concerns and maximizes reuse of the existing transformation and authentication policies, at the cost of internal proxy passes and stricter policy-scope handling.

Background material

• Envoy router retry behavior
• Envoy retry predicates and priorities
• Envoy external processing filter
• Envoy upstream HTTP filters
• Envoy aggregate clusters
• Envoy Gateway retry policy
• Kong AI Proxy Advanced
• LiteLLM proxy and router

Questions for the discussion

The decisions should be made in this order:

1. Should we retry within the same client request, or let that request fail?

   • Retry within the same request: choose between Approaches 1, 2, and 3. The gateway may make several billable upstream calls and must preserve the original request for replay.
   • Let it fail: choose Approach 4. Suspend the failed target and route only later requests to its fallbacks.

2. If we retry within the same request, should Envoy or the policy engine control the fallback loop?

   • Envoy controls it: approach 2.
   • The policy engine controls it: choose between Approaches 1 and 3.

3. If the policy engine controls failover, should it call providers directly or invoke the same Envoy route again?

   • Call providers directly: Approach 1. This gives the policy engine full control but turns it into part of the provider-facing HTTP data plane.
   • Invoke the same Envoy route again: Approach 3. This reuses the gateway's routing, transformation, authentication, and upstream handling, but adds loopback calls.

4. Are we prepared to run transformation and authentication in an upstream processor for every attempt?

   • Yes: Approach 2 is the strongest fit because Envoy retains ownership of retries and upstream transport.
   • No, we want to reuse the current route-level policies: Approach 3 is the stronger fit.

5. Are we willing to depend on Envoy's alpha upstream HTTP filter capability?

   • Yes: keep Approach 2 as the preferred design and validate it with the exact Envoy version WSO2 ships.
   • No: choose Approach 3.