You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Add a new service that exposes the OpenAI REST API (e.g. /v1/chat/completions, /v1/completions) as a drop-in front end for the async queues. It accepts a normal synchronous OpenAI request, submits it to the queue via the producer library, waits for the correlated result, and returns it to the caller. The async processor behind the queue applies gates (quota, saturation, priority) and merge policies before forwarding to the inference gateway.
The goal is frictionless migration: an operator points their OpenAI client's base URL at this service instead of the llm-d router and transparently gains async's fairness/back-pressure/prioritization — with no client code changes.
Motivation
Today async is only reachable by writing directly to the queue (via the producer library or raw Redis/PubSub). That's a real adoption barrier for teams already speaking the OpenAI API to the llm-d router. This service closes that gap:
Migration path: swap the base URL; keep using the OpenAI SDK.
Translates each request into an api.RequestMessage (payload = the body, endpoint = the path, deadline = now + request timeout) and submits it via producer.SubmitRequest.
Blocks until the correlated api.ResultMessage arrives, maps it back to an HTTP response, and returns it.
On client disconnect / timeout, calls producer.CancelRequests.
It should use the producer library internally rather than re-implementing queue plumbing.
Central design challenge — result correlation
producer.GetResult is a shared BRPOP on one result queue (producer/redis_sortedset_producer.go), so a single caller draining it receives all results, not the one it's waiting for. A synchronous, multi-request, possibly multi-replica front end needs per-request correlation. Options:
A — per-request result queue. Set RedisRequest.ResultQueueName to a unique key per request (e.g. resp:<instance>:<reqid>) and BRPOP exactly that. Simplest correlation; cost is many short-lived Redis keys (needs TTL/cleanup).
B — per-instance result queue + in-memory demux. Each replica owns one result queue; a single drain goroutine matches results to waiting handlers by id (the pattern pkg/pubsub already uses with its resultChannels map). Fewer keys, but each replica must consume only its own results (route results back to the submitting instance).
C — shared queue + distributed demux. Any replica can pop any result and must forward it to the replica holding the caller (pub/sub fan-out). Most complex; probably not v1.
This likely means a small producer enhancement: a correlated/awaitable get (e.g. SubmitAndAwait, or a per-request result-queue helper) so this logic lives in the library, not each caller. Recommendation: start with A for correctness/simplicity, revisit B for efficiency.
Other design considerations
Streaming (stream: true). Async results are captured as a single completed upstream body, so true SSE pass-through isn't available. v1 likely rejects stream: true (or buffers then replays a single chunk). Real streaming would need the processor/result format to carry a stream — out of scope for v1; call it out explicitly.
Deadlines / timeouts. Map a configurable request timeout to RequestMessage.deadline; if it passes with no result, return 504. This interacts with the gate/queue wait time.
Endpoint mapping. OpenAI path → RequestMessage.Endpoint; the processor joins it with igw_base_url. Decide the supported path set for v1.
Tenant / priority conveyance. Gates and the tier-priority merge policy key on metadata.team and a tier label. Define how the caller supplies these — e.g. request headers → metadata/labels, or derived from the API key/tenant. Without this, all traffic lands in one lane.
Auth.Authorization: Bearer <key> → optionally map keys to tenant/quota; or pass through. Decide v1 behavior.
Error → HTTP mapping. Use the structured ResultMessage (Preserve HTTP status and surface non-HTTP failures in async results #300): status_code > 0 → return that status + payload (raw upstream body); otherwise map error_code → HTTP (GATE_DROPPED → 429, DEADLINE_EXCEEDED → 504, CANCELLED → 499/client-closed, INFERENCE_ERROR/INVALID_REQUEST → 4xx/5xx).
Backends. The producer currently implements Redis SortedSet; the service inherits whatever backends the producer supports.
Deployment. New binary + Helm chart in this repo (alongside the processor), reusing the producer module. Metrics (request rate, queue-wait, end-to-end latency, cancellations).
Does this belong in this repo (new cmd/ + chart, reusing producer/) or its own repo? (Compare with llm-d-batch-gateway, which is the OpenAI Batch API front end — this proposal is its real-time/synchronous counterpart over the same queue.)
Which correlation approach for v1 (A vs B)?
How should tenant/tier/priority be supplied — headers, API-key mapping, or both?
Is buffered "fake streaming" acceptable for v1, or is non-streaming-only clearer?
Summary
Add a new service that exposes the OpenAI REST API (e.g.
/v1/chat/completions,/v1/completions) as a drop-in front end for the async queues. It accepts a normal synchronous OpenAI request, submits it to the queue via theproducerlibrary, waits for the correlated result, and returns it to the caller. The async processor behind the queue applies gates (quota, saturation, priority) and merge policies before forwarding to the inference gateway.The goal is frictionless migration: an operator points their OpenAI client's base URL at this service instead of the llm-d router and transparently gains async's fairness/back-pressure/prioritization — with no client code changes.
Motivation
Today async is only reachable by writing directly to the queue (via the
producerlibrary or raw Redis/PubSub). That's a real adoption barrier for teams already speaking the OpenAI API to the llm-d router. This service closes that gap:CancelRequests([Feature]: Support per-request cancellation from Producer side #299/add generation-aware request cancellation with pre-dispatch drop checks #307) so abandoned requests are dropped before dispatch instead of wasting inference.Note it does not replace the router — it sits in front of the queue; the async processor still forwards to the gateway/EPP/vLLM behind it:
Proposed solution
A new service/binary that:
api.RequestMessage(payload= the body,endpoint= the path,deadline= now + request timeout) and submits it viaproducer.SubmitRequest.api.ResultMessagearrives, maps it back to an HTTP response, and returns it.producer.CancelRequests.It should use the
producerlibrary internally rather than re-implementing queue plumbing.Central design challenge — result correlation
producer.GetResultis a sharedBRPOPon one result queue (producer/redis_sortedset_producer.go), so a single caller draining it receives all results, not the one it's waiting for. A synchronous, multi-request, possibly multi-replica front end needs per-request correlation. Options:RedisRequest.ResultQueueNameto a unique key per request (e.g.resp:<instance>:<reqid>) andBRPOPexactly that. Simplest correlation; cost is many short-lived Redis keys (needs TTL/cleanup).id(the patternpkg/pubsubalready uses with itsresultChannelsmap). Fewer keys, but each replica must consume only its own results (route results back to the submitting instance).This likely means a small producer enhancement: a correlated/awaitable get (e.g.
SubmitAndAwait, or a per-request result-queue helper) so this logic lives in the library, not each caller. Recommendation: start with A for correctness/simplicity, revisit B for efficiency.Other design considerations
stream: true). Async results are captured as a single completed upstream body, so true SSE pass-through isn't available. v1 likely rejectsstream: true(or buffers then replays a single chunk). Real streaming would need the processor/result format to carry a stream — out of scope for v1; call it out explicitly.RequestMessage.deadline; if it passes with no result, return504. This interacts with the gate/queue wait time.RequestMessage.Endpoint; the processor joins it withigw_base_url. Decide the supported path set for v1.metadata.teamand atierlabel. Define how the caller supplies these — e.g. request headers →metadata/labels, or derived from the API key/tenant. Without this, all traffic lands in one lane.Authorization: Bearer <key>→ optionally map keys to tenant/quota; or pass through. Decide v1 behavior.ResultMessage(Preserve HTTP status and surface non-HTTP failures in async results #300):status_code > 0→ return that status +payload(raw upstream body); otherwise maperror_code→ HTTP (GATE_DROPPED→ 429,DEADLINE_EXCEEDED→ 504,CANCELLED→ 499/client-closed,INFERENCE_ERROR/INVALID_REQUEST→ 4xx/5xx).producercurrently implements Redis SortedSet; the service inherits whatever backends the producer supports.Suggested phasing
/v1/chat/completions+/v1/completions, non-streaming, per-request result queue (Option A), header→metadata/tier mapping, deadline/timeout, disconnect→cancel, structured-result→HTTP mapping, chart + metrics./v1/embeddings//v1/models, in-memory demux (Option B), API-key→tenant/quota, additional backends.Open questions
cmd/+ chart, reusingproducer/) or its own repo? (Compare withllm-d-batch-gateway, which is the OpenAI Batch API front end — this proposal is its real-time/synchronous counterpart over the same queue.)Related
ResultMessage(status_code / error_code) for HTTP mappingproducer/— the library this service builds onllm-d-batch-gateway(OpenAI Batch API),llm-d-router(the sync path being migrated from)