fix(llm-api-gateway): preserve long-lived SSE responses

[mikeyrcamp]

Summary

• disable the route-level request deadline for the llm-api-gateway HTTPRoute so
  SSE responses are not truncated by the ingress default
• add a render regression assertion for the disabled route deadline
• preserve request-router routing, TLS, worker configuration, and application
  cancellation behavior

This draft currently addresses the confirmed 15-second ingress boundary. The
additional evidence below identifies a second, application-level boundary that
must also be addressed before the PR fully resolves #1060.

Root cause

The llm-api-gateway route did not specify a request timeout, so Envoy Gateway
applied its 15-second default. When that deadline expired, ingress returned 504
and disconnected llm-api-gateway; the resulting downstream cancellation then
propagated to the upstream request.

Direct upstream, request-router, and gateway-process requests all carried the
same 16-second SSE response successfully outside ingress. The failure occurred
only through the HTTPRoute.

Validation of the route fix

• make test in deploy/helm/gateway-routes
• existing llm-api-gateway upstream-cancellation tests
• before: caller 504 at 15.006 seconds, no [DONE]
• after: caller 200 at 16.007 seconds, [DONE] received, upstream completed in
  16.000 seconds, no caller or upstream error

Additional long-stream evidence

A public-safe backend emitted valid SSE chunks every 10 seconds and sent
[DONE] at 75 seconds. The active stream still ended at 60.004 seconds under
the gateway's current Go HTTP server write deadline, after HTTP 200 headers but
before [DONE], producing an incomplete caller response.

With only that write deadline disabled in an isolated test harness, the same
request returned HTTP 200 in 75.004 seconds with [DONE]; the upstream
completed in 75.000 seconds without a disconnect.

This confirms that disabling the HTTPRoute deadline is necessary but not
sufficient for streams longer than 60 seconds.

Remaining fix

Manage the write deadline at the SSE endpoint so streaming responses are not
limited by the server-wide total response-write duration. The implementation
must retain the server-wide deadline for ordinary requests, preserve prompt
upstream cancellation when the caller disconnects, and include a
shortened-duration regression test.

This is a focused prerequisite for completing the end-to-end validation in
#999.

Fixes #1060

Summary by CodeRabbit

• Bug Fixes

  • Updated the LLM API gateway route to use a zero-second request timeout, preventing unintended request time limits.

• Tests

  • Added validation to ensure the gateway route consistently renders with the expected timeout setting.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 3de1efae-a1c4-446f-a486-36aed47a8c88

📥 Commits

Reviewing files that changed from the base of the PR and between 356d8ff and fd8e779.

📒 Files selected for processing (2)

• deploy/helm/gateway-routes/chart/templates/httproute-llm-api-gateway.yaml
• deploy/helm/gateway-routes/scripts/test-render-routes.sh

Included review availability: Your plan provides up to 12 included reviews per hour; 11 remain after this review.

---

📝 Walkthrough

Walkthrough

The LLM API gateway HTTPRoute now sets a zero-second request timeout. The route rendering test verifies this default configuration.

Changes

LLM gateway timeout

Layer / File(s)	Summary
Disable and validate route timeout deploy/helm/gateway-routes/chart/templates/httproute-llm-api-gateway.yaml, deploy/helm/gateway-routes/scripts/test-render-routes.sh	The HTTPRoute sets timeouts.request to 0s. The render test verifies the default value.

Estimated code review effort: 1 (Trivial) | ~5 minutes

Merge Risk: 🟡 Moderate · up to fd8e7

The route-level timeout is removed, but SSE responses longer than 60 seconds can still be cut off before completion, leaving callers with incomplete responses. Merge should wait until the endpoint-level streaming deadline is addressed and covered by a regression test.

Suggested reviewers: shelleyshen-0

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Linked Issues check	⚠️ Warning	The PR fixes and tests the 15-second HTTPRoute deadline but omits the SSE write-deadline change and shortened-duration regression test required by issue #1060.	Implement endpoint-level write-deadline management and add the shortened-duration active-stream regression test while preserving ordinary-request deadlines and cancellation.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title uses the required Conventional Commits format and accurately describes the HTTPRoute fix for long-lived SSE responses.
Out of Scope Changes check	✅ Passed	The route timeout change and render assertion directly support the linked issue objective to preserve long-lived SSE responses.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 1 files. (1 skipped: 1 unsupported.)

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/issue-1060-sse-completion

---

_{Comment @coderabbitai help to get the list of available commands.}

[balajinvda]

🎉 This PR is included in version nvcf-gateway-routes-v1.15.1 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀