Gate capacity reservation leaks on retry → inFlight ratchets up → queue wedges permanently

[shimib]

Summary

In the Redis sorted-set flow, the per-queue gate's capacity reservation (a LocalConcurrencyGate slot or a redis-quota INCR) is released only when a ResultMessage for the request flows back through resultWorker. Every retry path re-enqueues the request and emits a RetryMessage with no result, so the reservation is never released. Because the release closure is stored with a plain activeReleases.Store(reqID, …), a later re-dispatch of the same id overwrites the prior closure, orphaning it permanently.

Under ordinary retry conditions (a gateway 503, an ActionRefuse, an ActionWait timeout), inFlight climbs monotonically until Budget() returns 0, at which point batchSize floors to 0 and the queue stops dispatching entirely. With LocalConcurrencyGate (no TTL) this is permanent; with redis-quota it is masked only by the key's TTL reset (itself incorrect — it causes over-admission when the counter resets while requests are still in flight).

Affected code

• pkg/redis/sortedset_impl.go:527 — r.activeReleases.Store(rview.ReqID(), releases) (plain store; overwrites any existing closure for the id).
• pkg/redis/sortedset_impl.go:648 — r.activeReleases.LoadAndDelete(m.ID) in resultWorker — the only non-shutdown release site; runs solely on a ResultMessage.
• pkg/asyncworker/worker.go:67,129,169,183,282,427,491 — retry/re-enqueue branches, each emitting a RetryMessage and no ResultMessage (so no release).
• pkg/async/inference/flowcontrol/local_concurrency_gate.go:59-61 — Budget() returns 0 when inFlight >= limit; no TTL on the counter.

Steps to reproduce (conceptual)

1. Configure a queue with a local-concurrency (or redis-quota concurrency) gate, limit = N.
2. Submit a request whose dispatch hits a retriable condition (inference 503, pool-gate ActionRefuse, or an ActionWait timeout).
3. The worker re-enqueues it via retryChannel — no result is produced, so activeReleases[id] is never released; inFlight stays elevated.
4. Repeat under normal backpressure. Each leaked reservation is permanent (no TTL on LocalConcurrencyGate). Once inFlight reaches limit, Budget()=0 → batchSize=0 → the worker stops pulling messages. The queue is wedged with no in-flight work.

Two distinct failure modes, both reachable:

• Accounting drift (limit >= 2): dispatch a request (inFlight=1, reserved) → it retries (inFlight still 1, still reserved — not released) → it is re-dispatched (inFlight=2, but only that one id is reserved; the earlier closure is orphaned by the Store overwrite). inFlight now over-counts real concurrency.
• Permanent wedge (limit = 1): a request that retries can never be re-dispatched, because its own un-released reservation occupies the only slot. It never produces a result (it is queued, not dispatched), so the reservation never releases — the request, and the whole queue, wedge forever.

Impact

• Liveness / availability (high): a queue can permanently stop dispatching under routine retry load, with no crash and no error surfaced — requests pile up until their deadlines and clients time out.
• Correctness: inFlight (and the redis-quota counter) over-count real concurrency; capacity is silently lost.

Suggested fix

Release the reservation on the retry paths — either invoke the stored release before re-enqueuing, or LoadAndDelete the closure before any re-Store, or carry the reservation across the retry so it is not double-counted. Add a regression test asserting inFlight returns to baseline after a request is retried and later completed. Separately, consider a TTL/lease refresh on the redis-quota concurrency counter to bound leaks from genuine crashes.