Make Redis optional: Postgres-only mode

[wseaton]

Support running the batch gateway without Redis by implementing the exchange interfaces (priority queue, in-flight tracking, volatile status, event channels) on PostgreSQL, selected via a new exchange_client.type config (default redis, no behavior change).

PR: #564
Design doc: docs/design/postgres-only-mode.md (in the PR).

Notes:

• llm-d-async has no Postgres backend, so postgres-only mode v1 is sync dispatch only (validated at startup).
• Real-DB integration tests + local benchmarks included in the PR.

[lioraron]

Thanks @wseaton for looking into this.

Several points:

Performance risk

The exchange interfaces aren't all job-granularity. UpdateProgressCounts is called per-request within a batch job. With multiple concurrent batch jobs, the status path could saturate the shared Postgres connection pool, and unlike Redis - these writes compete with the persistent DB layer for the same pool and disk I/O. The UNLOGGED optimization helps, but Postgres still carries buffer pool management, MVCC overhead, and row-level locking that an in-memory store doesn't. This is the path most likely to become a bottleneck under load.

Postgres-specific, not DB-agnostic

The persistent DB layer is pluggable (not necessarily Postgres), but this exchange implementation depends on Postgres-specific features (LISTEN/NOTIFY and UNLOGGED). If someone deploys with a different SQL database, they'd still probably need Redis or similar for the exchange.

Is there a customer need driving this?

PostgreSQL and Redis have equivalent deployment strategies (e.g. in OpenShift - both are OperatorHub add-ons). They serve well-defined, different roles in the current architecture — Postgres for persistent metadata, Redis for in-memory exchange. The current split plays to each system's strengths.
The associated PR adds a second implementation of ~3,400 lines, which is a permanent maintenance and testing surface.
Before taking on that cost, is there a specific customer requesting Redis-free operation?
I'd prefer to defer this until there is a concrete demand, and if there is demand - understanding their constraints would help us validate that the Postgres performance characteristics meet their needs.

[wseaton]

UpdateProgressCounts is called per-request within a batch job

Isn't UpdateProgressCountsthrottled here? It shouldn't write more than once per-second per worker (so 20/s)

Task granularity would be solveable via microbatching (eg. batch 100 requets at once to limit DB I/O), definitely a future phase optimization because it dovetails with job recovery semantics.

and unlike Redis - these writes compete with the persistent DB layer for the same pool and disk I/O

Not necessarily, NewPostgresExchangeClient as proposed in the PR opens it's own connection pool.

The UNLOGGED optimization helps, but Postgres still carries buffer pool management, MVCC overhead, and row-level locking that an in-memory store doesn't. This is the path most likely to become a bottleneck under load.

The benchmark numbers in the PR on laptop grade hardware shows performance that is roughly two orders of magnitude faster than the needs for the batch gateway, can we try to ground the conversation in a real benchmark scenario that proves a postgres queue implementation can't keep up?