Description
Kthena router already has a scheduler plugin E2E framework (prefix-cache / least-request / least-latency / lora-affinity / random / gpu-usage), and KVCache-aware scheduling plugin implementation exists in the codebase.
However, KVCache-aware currently lacks robust end-to-end coverage that validates the full behavior in a real cluster environment (router + mock backends + Redis/tokenization interactions + observable signals).
This proposal aims to add KVCache-aware plugin E2E tests on top of the existing plugin E2E framework, improving confidence and preventing regressions.
Background / Prior work
Expected outcome
1.Add a KVCache-aware E2E case under test/e2e/router/, reusing the existing plugin fixtures while wiring llm-d-sim with a ZMQ bridge and Redis so cache-hit differences across replicas are deterministically seedable.
2.Simulate runtime KV-cache block ownership via ZMQ event dispatch and verify the router writes/reads the expected Redis mappings before scoring pods.
3.Assert strong routing behavior (e.g. ≥90% traffic to cache-hit replicas) using stable polling on router access logs and plugin metrics, not fixed sleeps.
4.Document and harden the llm-d-sim + ZMQ + Redis test path so CI can catch regressions in tokenization, Redis integration, and scheduling decisions end-to-end.
Recommended skills
go, Kubernetes, controller/router E2E testing (Kind), basic Redis, observability/metrics.
Mentor(s)
Jinyu Zhou(@FAUST-BENCHOU , 2319109590@qq.co)
Description
Kthena router already has a scheduler plugin E2E framework (prefix-cache / least-request / least-latency / lora-affinity / random / gpu-usage), and KVCache-aware scheduling plugin implementation exists in the codebase.
However, KVCache-aware currently lacks robust end-to-end coverage that validates the full behavior in a real cluster environment (router + mock backends + Redis/tokenization interactions + observable signals).
This proposal aims to add KVCache-aware plugin E2E tests on top of the existing plugin E2E framework, improving confidence and preventing regressions.
Background / Prior work
Expected outcome
1.Add a KVCache-aware E2E case under test/e2e/router/, reusing the existing plugin fixtures while wiring llm-d-sim with a ZMQ bridge and Redis so cache-hit differences across replicas are deterministically seedable.
2.Simulate runtime KV-cache block ownership via ZMQ event dispatch and verify the router writes/reads the expected Redis mappings before scoring pods.
3.Assert strong routing behavior (e.g. ≥90% traffic to cache-hit replicas) using stable polling on router access logs and plugin metrics, not fixed sleeps.
4.Document and harden the llm-d-sim + ZMQ + Redis test path so CI can catch regressions in tokenization, Redis integration, and scheduling decisions end-to-end.
Recommended skills
go, Kubernetes, controller/router E2E testing (Kind), basic Redis, observability/metrics.
Mentor(s)
Jinyu Zhou(@FAUST-BENCHOU , 2319109590@qq.co)