[release-1.0] Add session boost queue wait timeout (HTTP 504 on excessive wait)

charts/kthena/charts/networking/README.md

@@ -52,18 +52,19 @@ kthenaRouter:
 
 #### Configuration Parameters
 
-| Parameter                                  | Type    | Default          | Description                                                        |
-| ------------------------------------------ | ------- | ---------------- | ------------------------------------------------------------------ |
-| `kthenaRouter.fairness.enabled`            | boolean | `false`          | Enable user-fairness scheduling (mutually exclusive with boost)    |
-| `kthenaRouter.fairness.windowSize`         | string  | `"1h"`           | Fairness: sliding window duration (1m-1h)                          |
-| `kthenaRouter.fairness.inputTokenWeight`   | float   | `1.0`            | Fairness: weight for input tokens (≥0)                             |
-| `kthenaRouter.fairness.outputTokenWeight`  | float   | `2.0`            | Fairness: weight for output tokens (≥0)                            |
-| `kthenaRouter.fairness.maxConcurrent`      | int     | `0`              | Fairness: global inflight limit (`0` falls back to QPS mode)       |
-| `kthenaRouter.sessionBoost.enabled`        | boolean | `false`          | Enable session-boost scheduling (mutually exclusive with fairness) |
-| `kthenaRouter.sessionBoost.header`         | string  | `"X-Session-ID"` | HTTP header used to identify conversation sessions                 |
-| `kthenaRouter.sessionBoost.maxSessions`    | int     | `4096`           | Max recently-completed sessions kept warm (LRU-evicted)            |
-| `kthenaRouter.sessionBoost.inflightPerPod` | int     | `16`             | Inflight requests per backend pod; total = perPod x pod count      |
-| `kthenaRouter.sessionBoost.gracePeriod`    | string  | `"0s"`           | Wait time for a same-session follow-up (disabled by default)       |
+| Parameter                                  | Type    | Default          | Description                                                                   |
+| ------------------------------------------ | ------- | ---------------- | ----------------------------------------------------------------------------- |
+| `kthenaRouter.fairness.enabled`            | boolean | `false`          | Enable user-fairness scheduling (mutually exclusive with boost)               |
+| `kthenaRouter.fairness.windowSize`         | string  | `"1h"`           | Fairness: sliding window duration (1m-1h)                                     |
+| `kthenaRouter.fairness.inputTokenWeight`   | float   | `1.0`            | Fairness: weight for input tokens (≥0)                                        |
+| `kthenaRouter.fairness.outputTokenWeight`  | float   | `2.0`            | Fairness: weight for output tokens (≥0)                                       |
+| `kthenaRouter.fairness.maxConcurrent`      | int     | `0`              | Fairness: global inflight limit (`0` falls back to QPS mode)                  |
+| `kthenaRouter.sessionBoost.enabled`        | boolean | `false`          | Enable session-boost scheduling (mutually exclusive with fairness)            |
+| `kthenaRouter.sessionBoost.header`         | string  | `"X-Session-ID"` | HTTP header used to identify conversation sessions                            |
+| `kthenaRouter.sessionBoost.maxSessions`    | int     | `4096`           | Max recently-completed sessions kept warm (LRU-evicted)                       |
+| `kthenaRouter.sessionBoost.inflightPerPod` | int     | `16`             | Inflight requests per backend pod; total = perPod x pod count                 |
+| `kthenaRouter.sessionBoost.gracePeriod`    | string  | `"0s"`           | Wait time for a same-session follow-up (disabled by default)                  |
+| `kthenaRouter.sessionBoost.timeout`        | string  | `"30s"`          | Max queue wait before 504; set a non-positive duration (e.g. `0s`) to disable |
 
 #### Session Boost Configuration
 

charts/kthena/charts/networking/templates/kthena-router/component/deployment.yaml

@@ -106,6 +106,10 @@ spec:
             {{- end }}
             - name: SESSION_BOOST_GRACE_PERIOD
               value: {{ .gracePeriod | quote }}
+            {{- if .timeout }}
+            - name: SESSION_BOOST_TIMEOUT
+              value: {{ .timeout | quote }}
+            {{- end }}
             {{- end }}
             {{- end }}
             # Access log configuration

charts/kthena/charts/networking/values.yaml

@@ -70,6 +70,11 @@ kthenaRouter:
     # follow-up to arrive. Disabled by default (0s); enable only if you understand
     # the latency trade-off for non-boosted requests.
     gracePeriod: "0s"
+    # timeout is the maximum time a request may wait in the session-boost queue
+    # before it is rejected with HTTP 504. Defaults to 30s; set it to a non-positive
+    # duration (e.g. "0s") to disable the timeout, in which case a request is bounded
+    # only by client disconnect.
+    timeout: "30s"
   # accessLog configuration for request logging
   accessLog:
     # enabled controls whether access logging is active

charts/kthena/values.yaml

@@ -100,6 +100,10 @@ networking:
       # -- Wait time after a request completes for a same-session follow-up.<br/>
       # Disabled by default (`0s`).
       gracePeriod: "0s"
+      # -- Maximum time a request may wait in the session-boost queue before it is<br/>
+      # rejected with HTTP 504. Defaults to `30s`; set a non-positive duration<br/>
+      # (e.g. `0s`) to disable it (bounded only by client disconnect).
+      timeout: "30s"
     gatewayAPI:
       # -- Enable Gateway API related features.
       enabled: false

docs/kthena/docs/reference/helm-chart-values.md

@@ -37,6 +37,7 @@ A Helm chart for deploying Kthena
 | networking.kthenaRouter.sessionBoost.header | string | `"X-Session-ID"` | HTTP header used to identify conversation sessions. |
 | networking.kthenaRouter.sessionBoost.inflightPerPod | int | `16` | Maximum inflight requests admitted per backend pod.<br/> Total inflight limit is this value times the number of backend pods.<br/> `0` or unset uses the router default (16). |
 | networking.kthenaRouter.sessionBoost.maxSessions | int | `4096` | Maximum number of recently-completed sessions kept warm for boosting.<br/> Bounds an LRU cache; the least-recently-used session is evicted automatically.<br/> Size it by the number of concurrent conversations to keep boosted. |
+| networking.kthenaRouter.sessionBoost.timeout | string | `"30s"` | Maximum time a request may wait in the session-boost queue before it is<br/> rejected with HTTP 504. Defaults to `30s`; set a non-positive duration<br/> (e.g. `0s`) to disable it (bounded only by client disconnect). |
 | networking.kthenaRouter.terminationGracePeriodSeconds | int | `330` | The router will drain all in-flight requests before forcefully closing connections. |
 | networking.kthenaRouter.tls.dnsName | string | `"your-domain.com"` | DNS name to use for the certificate. |
 | networking.kthenaRouter.tls.enabled | bool | `false` | Enable TLS for Kthena Router server. |

docs/kthena/docs/user-guide/fairness-scheduling.md

@@ -121,13 +121,13 @@ env:
 
 ### Core Settings
 
-| Environment Variable           | Purpose                                               | Default              | Notes                                                                 |
-| ------------------------------ | ----------------------------------------------------- | -------------------- | --------------------------------------------------------------------- |
-| `ENABLE_FAIRNESS_SCHEDULING`   | Enables fairness scheduling in the router             | `false`              | Global feature switch. Mutually exclusive with `ENABLE_SESSION_BOOST` |
-| `FAIRNESS_WINDOW_SIZE`         | Sliding window used to track recent usage             | runtime default `5m` | The Helm chart default sets this to `1h` when fairness is enabled     |
-| `FAIRNESS_INPUT_TOKEN_WEIGHT`  | Weight applied to input tokens when recording usage   | `1.0`                | Used by the token tracker                                             |
-| `FAIRNESS_OUTPUT_TOKEN_WEIGHT` | Weight applied to output tokens when recording usage  | `2.0`                | Used by the token tracker                                             |
-| `FAIRNESS_QUEUE_TIMEOUT`       | Maximum time a request may wait in the fairness queue | `60s`                | Waiting longer returns a timeout to the client                        |
+| Environment Variable           | Purpose                                               | Default              | Notes                                                                                                            |
+| ------------------------------ | ----------------------------------------------------- | -------------------- | ---------------------------------------------------------------------------------------------------------------- |
+| `ENABLE_FAIRNESS_SCHEDULING`   | Enables fairness scheduling in the router             | `false`              | Global feature switch. Mutually exclusive with `ENABLE_SESSION_BOOST`                                            |
+| `FAIRNESS_WINDOW_SIZE`         | Sliding window used to track recent usage             | runtime default `5m` | The Helm chart default sets this to `1h` when fairness is enabled                                                |
+| `FAIRNESS_INPUT_TOKEN_WEIGHT`  | Weight applied to input tokens when recording usage   | `1.0`                | Used by the token tracker                                                                                        |
+| `FAIRNESS_OUTPUT_TOKEN_WEIGHT` | Weight applied to output tokens when recording usage  | `2.0`                | Used by the token tracker                                                                                        |
+| `FAIRNESS_QUEUE_TIMEOUT`       | Maximum time a request may wait in the fairness queue | `60s`                | Waiting longer returns a timeout (504) to the client. Applies only to the user-fairness queue, not session boost |
 
 ### Queue Policy Settings
 

docs/kthena/docs/user-guide/session-boost.md

@@ -30,7 +30,9 @@ flowchart TD
     E --> F
     F -- Yes --> G[Dispatch to backend]
     F -- No --> H[Wait / backpressure]
-    H --> F
+    H --> K{Waited longer than<br/>SESSION_BOOST_TIMEOUT?}
+    K -- Yes --> R[Reject with HTTP 504]
+    K -- No --> F
     G --> I[Request completes]
     I --> J[Mark session completed<br/>promote in LRU cache]
     J --> M[Dequeue next request]
@@ -73,6 +75,7 @@ networking:
       header: "X-Session-ID"
       maxSessions: 4096          # LRU cache of recently-completed sessions kept warm
       inflightPerPod: 16         # total inflight = inflightPerPod x backend pod count
+      timeout: "30s"             # reject requests that wait longer than this with HTTP 504 (set "0s" to disable)
 ```
 
 Apply with Helm:
@@ -98,16 +101,19 @@ env:
   value: "4096"
 - name: SESSION_BOOST_INFLIGHT_PER_POD
   value: "16"                 # total inflight = perPod x backend pod count
+- name: SESSION_BOOST_TIMEOUT
+  value: "30s"               # reject requests that wait longer than this with HTTP 504 (set "0s" to disable)
 ```
 
 ## Configuration Reference
 
-| Environment Variable             | Purpose                                                                          | Default        | Notes                                                                                                                                                                                      |
-| -------------------------------- | -------------------------------------------------------------------------------- | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
-| `ENABLE_SESSION_BOOST`           | Enable the session-boost scheduling strategy                                     | `false`        | Mutually exclusive with `ENABLE_FAIRNESS_SCHEDULING`                                                                                                                                       |
-| `SESSION_BOOST_HEADER`           | HTTP header used to identify conversation sessions                               | `X-Session-ID` | Must match what your clients send                                                                                                                                                          |
-| `SESSION_BOOST_MAX_SESSIONS`     | Max number of recently-completed sessions kept "warm" for boosting (LRU-bounded) | `4096`         | When the cache is full, the least-recently-used session is evicted automatically. Size it by the number of concurrent conversations you want to keep boosted—no time-based tuning required |
-| `SESSION_BOOST_INFLIGHT_PER_POD` | Inflight requests admitted per backend pod                                       | `16`           | The total inflight limit is this value multiplied by the number of backend pods. Size it from the per-pod concurrency (e.g., vLLM's `--max-num-seqs`)                                      |
+| Environment Variable             | Purpose                                                                          | Default        | Notes                                                                                                                                                                                                                                                             |
+| -------------------------------- | -------------------------------------------------------------------------------- | -------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `ENABLE_SESSION_BOOST`           | Enable the session-boost scheduling strategy                                     | `false`        | Mutually exclusive with `ENABLE_FAIRNESS_SCHEDULING`                                                                                                                                                                                                              |
+| `SESSION_BOOST_HEADER`           | HTTP header used to identify conversation sessions                               | `X-Session-ID` | Must match what your clients send                                                                                                                                                                                                                                 |
+| `SESSION_BOOST_MAX_SESSIONS`     | Max number of recently-completed sessions kept "warm" for boosting (LRU-bounded) | `4096`         | When the cache is full, the least-recently-used session is evicted automatically. Size it by the number of concurrent conversations you want to keep boosted—no time-based tuning required                                                                        |
+| `SESSION_BOOST_INFLIGHT_PER_POD` | Inflight requests admitted per backend pod                                       | `16`           | The total inflight limit is this value multiplied by the number of backend pods. Size it from the per-pod concurrency (e.g., vLLM's `--max-num-seqs`)                                                                                                             |
+| `SESSION_BOOST_TIMEOUT`          | Maximum time a request may wait in the queue before it is rejected with 504      | `30s`          | Enabled by default. A request queued longer than this is rejected with `504 Gateway Timeout`. Set a non-positive duration (e.g. `0s`) to disable it. This is the only server-side queue-wait bound in session-boost mode; `FAIRNESS_QUEUE_TIMEOUT` does not apply |
 
 > `SESSION_BOOST_GRACE_PERIOD` is intentionally omitted from the table above. It is an advanced, scenario-specific knob that is disabled by default; see [Advanced: Grace Period](#advanced-grace-period-use-with-caution).
 
@@ -174,6 +180,19 @@ The queue uses two-level admission control to avoid flooding backends:
 
 When a request completes, the queue immediately attempts to dequeue the next request (release-driven dequeue) rather than waiting for the next metrics refresh.
 
+### Queue Wait Timeout (504 Rejection)
+
+Under heavy load a request may sit in the queue for a long time before backend capacity frees up. Session boost does **not** apply the fairness queue timeout (`FAIRNESS_QUEUE_TIMEOUT` only governs the user-fairness queue). Instead, session boost bounds the wait with its own **queue wait timeout** so latency-sensitive front ends **fail fast** and can retry or shed load rather than waiting indefinitely.
+
+The wait timeout rejects over-queued requests with `504 Gateway Timeout`:
+
+- It is controlled by `SESSION_BOOST_TIMEOUT` and **enabled by default** at `30s`.
+- Set a non-positive duration (e.g. `0s`) to disable it; a session-boost request is then bounded only by client disconnect.
+
+If a request waits in the queue longer than `SESSION_BOOST_TIMEOUT`, the router removes it from the queue and responds with `504 Gateway Timeout`. The 504 signals that the request *timed out* waiting for backend capacity (the queue is saturated); the client can retry later or shed load.
+
+> `SESSION_BOOST_TIMEOUT` is the only server-side queue-wait bound in session-boost mode. `FAIRNESS_QUEUE_TIMEOUT` has no effect here—it applies exclusively to the user-fairness strategy. When `SESSION_BOOST_TIMEOUT` is disabled (a non-positive value), a session-boost request is bounded only by client disconnect.
+
 ## Session Boost vs User Fairness
 
 Session boost and user fairness are two mutually exclusive scheduling strategies for the per-model request queue; they configure that queue for different goals:
@@ -249,6 +268,14 @@ Session boost only controls queue ordering. If the boosted request is routed to
 
 Each tracked session consumes minimal memory (just a session ID). The tracker is a bounded LRU cache, so total memory is capped at `SESSION_BOOST_MAX_SESSIONS` entries regardless of how much traffic flows through the router. If memory is a concern, reduce `SESSION_BOOST_MAX_SESSIONS`.
 
+### Clients receive HTTP 504 responses
+
+When `SESSION_BOOST_TIMEOUT` is set (enabled by default at `30s`), the router returns `504 Gateway Timeout` for requests that wait in the queue longer than it. This is expected load-shedding under sustained overload—it means the backends cannot keep up with demand. If you see more 504s than desired:
+
+1. Confirm the backends are healthy and scaled appropriately; add capacity if they are saturated.
+2. Increase `SESSION_BOOST_TIMEOUT` to allow requests to wait longer before rejection.
+3. Disable the timeout by setting `SESSION_BOOST_TIMEOUT` to a non-positive value (e.g. `0s`) so requests are no longer rejected on a wait timeout (they then wait until backend capacity frees up or the client disconnects).
+
 ## Advanced: Grace Period (Use With Caution)
 
 :::warning