Support OpenAI-compatible Batch API (/v1/batches and /v1/files)

[YaoZengzeng]

What would you like to be added:

Add first-class support for an OpenAI-compatible Batch API to Kthena, so that users can submit, track, and manage large-scale offline/asynchronous inference jobs against models served by Kthena, in addition to the existing interactive (synchronous) serving path.

Rather than introducing a separate standalone service, this capability should be built into the kthena-router itself, extending the existing data-plane component with a batch-serving path alongside its interactive routing.

Concretely, this would include:

• OpenAI-compatible endpoints with schema parity, served by the router:
  • /v1/files — upload/download/list/delete the JSONL input and output files used by batch jobs.
  • /v1/batches — create, retrieve, list, and cancel batch jobs, with real-time status/progress fields matching the OpenAI Batch object (status, request_counts, output_file_id, error_file_id, etc.).
• Large-scale job processing — support many thousands of requests per job (e.g. tens of thousands), reading requests from an input file and writing results to an output file.
• Asynchronous execution decoupled from interactive traffic — the router dispatches batch requests to Kthena-served backends (vLLM/SGLang/Triton) reusing its existing routing and load-balancing logic, in a way that minimizes interference with latency-sensitive interactive workloads, ideally with adaptive concurrency/flow control so batch throughput backs off when interactive traffic or backend pressure rises.
• Router-integrated design — the batch API, job queue, and dispatch loop live inside the router process (or as an optional sub-module of it) rather than as separate deployables, so operators get batch support without deploying and scaling an additional component. It should be exposed through the existing router deployment and Helm charts, with liveness/readiness probes and Prometheus metrics.
• Pluggable storage backends — for job/file metadata, the job queue, and file storage (e.g. object storage such as S3, plus a relational/queue backend), so operators can choose backends that fit their environment.
• Job lifecycle management — cancellation during processing, fault tolerance/recovery if the router restarts, and garbage collection of expired jobs and files.

Kthena, being a Kubernetes-native LLM serving platform with a control plane and an intelligent router, is a natural place to offer this capability. Because the router already owns request classification, model-aware routing (KV-cache aware, model-load aware), and traffic control, embedding the batch path there lets batch dispatch directly reuse those mechanisms — e.g. grouping/ordering requests by model and system-prompt prefix to keep KV-cache hot, and sharing a single view of backend load between interactive and batch traffic.

Why is this needed:

• Large offline/batch workloads are a first-class LLM use case — dataset inference, embedding generation for large corpora, model evaluation/testing, and offline analysis. These are asynchronous and throughput-oriented rather than latency-sensitive, and are awkward to run over a purely synchronous serving API.
• OpenAI compatibility lowers adoption friction — many teams already build against the OpenAI /v1/batches and /v1/files API. Exposing the same schema lets them move batch workloads to Kthena without rewriting client code.
• Better cluster utilization and cost efficiency — batch jobs can soak up spare capacity and be dispatched with adaptive concurrency control so they yield to interactive traffic, enabling differentiated SLOs and cost-optimized inference (batch vs. interactive) on shared infrastructure.
• Completes the serving story with no extra moving parts — Kthena already provides production-grade interactive serving, autoscaling, and intelligent routing. Building batch support into the router lets a single component and deployment cover both interactive and batch inference, instead of requiring a separate system for offline jobs.

[avinxshKD]

/assign

[avinxshKD]

@YaoZengzeng makes sense to expose this through the router, but I would not treat it as just two new HTTP handlers.

Batch needs durable state: uploaded files, job metadata, output/error files, cancellation, retries, and recovery after router restart. The current router path is mostly synchronous proxying, so the important design point is the job store + dispatcher, not the endpoint shape.

I’d start with OpenAI-shaped /v1/files and /v1/batches, backed by pluggable storage, and have the dispatcher reuse the existing routing/rate-limit/scheduling path. In-memory queue only for tests/dev; not as the real implementation.

Let me know, love to know your thoughts on this.

[hzxuzhonghu]

Please double check whether this api is supported in inference engine first, otherwise we can only rely on the thirdparty maas

[avinxshKD]

Good point, to verify the backend side first.

My read is that Kthena should not require engine-native /v1/batches. The router can own the batch lifecycle and dispatch each JSONL request through the normal inference path. But that only works for request types the backend already supports, such as /v1/chat/completions, /v1/completions, or /v1/embeddings.

I’ll check vLLM/SGLang support for those paths before starting the design.

[katara-Jayprakash]

/assign
this seems interesting i would love to work on this issue! is it ok? @avinxshKD

[avinxshKD]

@katara-Jayprakash sure, pls go ahead.

[YaoZengzeng]

Good point, to verify the backend side first.

My read is that Kthena should not require engine-native /v1/batches. The router can own the batch lifecycle and dispatch each JSONL request through the normal inference path. But that only works for request types the backend already supports, such as /v1/chat/completions, /v1/completions, or /v1/embeddings.

I’ll check vLLM/SGLang support for those paths before starting the design.

Good, this is way I prefer.

[avinxshKD]

Hey @katara-Jayprakash, are you working on this one or should I take it forward? happy to proceed if you haven't started.

[katara-Jayprakash]

don't worry, i am on it