Add haproxy.router.openshift.io/timeout annotation to kagenti-api route and expose configurable knob

[jordigilh]

Summary

The kagenti-api OpenShift route requires a haproxy.router.openshift.io/timeout annotation to support long-lived SSE streaming connections (e.g., kubernaut_watch during multi-minute workflow executions). Without it, HAProxy's default tunnelTimeout (5 minutes on most OCP clusters) kills active streams mid-flight, causing the browser to receive: Connection error: peer closed connection without sending complete message body (incomplete chunked read).

Root Cause Analysis

The SSE streaming path is:

Browser ←→ HAProxy (kagenti-api route, edge TLS) ←→ Kagenti Backend ←→ Apifrontend

• Apifrontend has no server-side write timeout (removed in kubernaut PR #1346) — streams run indefinitely.
• Kagenti Backend uses httpx.AsyncClient(timeout=120.0) — inter-byte timeout only, satisfied by AF's 5-second keepalive dots.
• Uvicorn has no active-stream timeout.
• HAProxy applies tunnelTimeout (default 300s / 5 minutes) to the route when no annotation is present, killing the connection regardless of data flow.

Evidence

• AF logs show the stream continued gracefully for 5:20 minutes (error: false), while the browser error appeared at exactly 5:00.
• Adding haproxy.router.openshift.io/timeout: 900s manually to the route resolved the issue immediately.
• The annotation was previously removed because its necessity wasn't understood — we now have proof it's required.

Request

1. Add haproxy.router.openshift.io/timeout annotation to the kagenti-api route template in the Helm chart with a sensible default (e.g., 900s / 15 minutes).
2. Expose a configurable Helm value (e.g., route.timeout or route.annotations) so deployers can tune the timeout based on their workload characteristics (some workflows like disk-pressure remediation can exceed 10 minutes).

Suggested Helm values structure

route:
  timeout: "900s"  # Default: 15 minutes for SSE streaming
  # OR more generic:
  annotations:
    haproxy.router.openshift.io/timeout: "900s"

Impact

Without this annotation, any SSE stream exceeding the cluster's default tunnelTimeout (typically 5 minutes) will be terminated mid-flight. This affects:

• kubernaut_watch (monitoring workflow execution progress)
• Any long-running A2A streaming interaction
• Future features that rely on sustained SSE connections

Workaround

Manual annotation (lost on next helm upgrade):

kubectl -n kagenti-system annotate route kagenti-api haproxy.router.openshift.io/timeout=900s

Related

• kubernaut PR #1346: Removed http.Server.WriteTimeout from apifrontend (server-side fix)
• OpenShift docs: Route timeouts