Skip to content

Releases: volcano-sh/kthena

Release list

v1.0.0-rc.0

v1.0.0-rc.0 Pre-release
Pre-release

Choose a tag to compare

View all tags
@LiZhenCheng9527 LiZhenCheng9527 released this 03 Jul 09:36
v1.0.0-rc.0
e15f7f9
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

This release includes the following artifacts:

  • Helm Chart: kthena.tgz (OCI artifact pushed to GHCR)
  • Installation YAML: kthena-install.yaml
  • CLI Binaries: Pre-built binaries for Linux (amd64/arm64), macOS (amd64/arm64), and Windows (amd64)

Documentation

Release notes are automatically generated from commits.

What's Changed

Exciting New Features 🎉

Bug fixes 🐛

  • Replace the kubeclient.Update method in autoScaler with kubeclient.Patch by @LiZhenCheng9527 in #915
  • test: wait for router webhook readiness by @acsoto in #930
  • fix(autoscaler): use maximum sliding window for scale-down stabilization by @kube-gopher in #946
  • fix(nixl): rebuild prefill/decode request bodies on every Proxy call by @Sanchit2662 in #947
  • fix: replace testify/assert/yaml with sigs.k8s.io/yaml in scheduler p… by @anirudh240 in #967
  • fix: use GetGauge() instead of GetCounter() for sglang metrics by @anirudh240 in #976
  • fix(router): retry sends empty body to fallback pods in aggregated proxy path by @Sanchit2662 in #1031
  • fix(autoscaler/metric_collector): exit decode loop on non-EOF error by @Kernel-9 in #952
  • fix: require bearer scheme for JWT auth by @pm-ju in #1035
  • fix: reject empty router model requests by @pm-ju in #1036
  • test/e2e: fix flaky TestKthenaRouterValidatingWebhook caused by webhook race after pod restart by @nXtCyberNet in #1065
  • model-booster: fix data race and concurrent-map-write panic in ModelBoosterController by @kube-gopher in #1085
  • fix: use controller owner refs for modelbooster children by @pm-ju in #1054
  • fix(autoscaler/metric_collector): Use the target-referenced namespace first by @Kernel-9 in #1068
  • remove unused GetModelRoutesByGateway by @anirudh240 in #1078
  • fix: support ipv6 pod backend URLs by @pm-ju in #1071
  • Fix HTTPRoute PathPrefix matching by @Monti-27 in #1119
  • fix(router): return mid-stream and copy errors from proxyRequest by @Sanchit2662 in #1049
  • metrics: check HTTP status before parsing response by @Sri-Varshith in #1142
  • test(e2e): add self-healing coverage for ModelBooster by @katara-Jayprakash in #1143
  • fix(model-booster): wire ModelWorker.Affinity to pod spec templates by @Abirdcfly in #1146
  • Fix concurrent graceMap check/store race in handleErrorPod by @nXtCyberNet in #1157
  • fix: guard PodInfo access against data races by @zhy76 in #1167
  • Fix router auth panic when JWKS cache is empty by @avinxshKD in #1219
  • Validate ModelRoute rule fields by @avinxshKD in #1201
  • fix: remove unnecessary goroutine in ModelPrefixStore LRU eviction callback by @nabrahma in #1243
  • fix(scheduler): treat zero TTFT/TPOT as uninitialized in least-latency plugin by @Sanchit2662 in #1040
  • fix: aggregate labeled autoscaler metrics by @pm-ju in #1064
  • Fix stale KV cache ownership by @avinxshKD in #1224
  • fix(router): respect Gateway allowedRoutes by @avinxshKD in #1263
  • fix: parallelize pod metrics scraping loop with bounded concurrency by @nabrahma in #1255
  • Fixed an issue where a PG was created by mistake when scaling up a Se… by @LiZhenCheng9527 in #1268

Documentation Updates 📚

Other Changes 🔄

Read more

Contributors

Abirdcfly, lx1036, and 33 other contributors
Loading

Kthena v0.4.0

Kthena v0.4.0 Latest
Latest

Choose a tag to compare

View all tags
@LiZhenCheng9527 LiZhenCheng9527 released this 21 Apr 02:36
v0.4.0
2916fb7
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Thanks to the incredible dedication and collective efforts of our contributors over the past two months, Kthena’s stability has reached new heights. We want to express our deepest gratitude to everyone who contributed to this milestone. Today, we are thrilled to announce the official release of Kthena v0.4.0—our most robust and feature-rich version yet!

Beyond rock-solid stability, Kthena v0.4.0 introduces a wave of exciting new features designed to streamline your LLM workloads and empower your AI infrastructure.

Improved Observability

Role Status Visibility

To minimize kube-apiserver load, Kthena's ModelServing utilizes a local store to cache the status of ServingGroups and Roles. While highly efficient, we realized this limited our observability during debugging.

In v0.4.0, we've broken the black box. We now expose role status directly via Kubernetes Events, dramatically enhancing the observability of ModelServing. Looking ahead, we plan to directly embed this crucial Role information into the ModelServing Status, giving you complete, transparent control over your deployments.

Comprehensive Access Logging

Router observability is now easier than ever. The Router now generates a detailed access log, capturing rich routing metadata for every single request. Here is an example of a Router log:

[2026-04-16T07:33:08.435627146Z] "POST /v1/chat/completions HTTP/1.1" 200 model_name=deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B model_router=deepseek-r1-1-1.5b model_server=deepseek-r1 selected_pod=deepseek-r1-1-1.5b-6989c66877-p6jvv request_id=ad683d1b-6011-4b0f-b9b5-cbb18d43c57b gateway=dev/default http_route=kthena-e2e-gie-8eoas/llm-route inference_pool=kthena-e2e-gie-8eoas/deepseek-r1-1-1.5b tokens=10/38 timings=3ms(0+2+0)

Compared to previous versions, we have introduced the gateway, http_route, and inference_pool fields to provide deeper visibility into Gateway and Gateway Inference Extension traffic.

A Faster, Smarter Router

Deterministic & Efficient Model Selection

Previously, mapping multiple ModelRoute resources to a single model could trigger route conflicts—leading to ambiguous rule matching and inconsistent target selection. Because Kubernetes' built-in CRD validation cannot enforce global cross-object uniqueness, we tackled this gracefully at the routing layer.

Kthena v0.4.0 introduces a robust conflict-resolution mechanism. When duplicate ModelRoutes exist, the router deterministically prioritizes the oldest (typically prebuilt) route, treating newer duplicates as lower priority. Predictable, rock-solid routing every time.

Configurable Prefix-Caching

One size doesn't fit all. That's why Kthena replaced hardcoded prefix-cache parameters with a fully configurable prefix-matching system. You now have granular control over prefix-cache behavior with the following parameters:

  • Block Size (for hash processing): Controls the granularity of the prefix match. Smaller blocks offer more precise matching but increase CPU overhead, while larger blocks process faster.
  • Max Block Limits: Sets a ceiling on how much of a given prompt is hashed. This protects the router from computational bottlenecks and latency spikes when handling excessively long incoming prompts.
  • Cache Capacity: Defines how many prefix entries the router can remember. Increasing this improves cache hit rates for highly diverse workloads, at the cost of slightly higher memory footprint.
  • Top-K Results: Determines how many candidate instances are considered when a match is found. Tuning this allows for better load balancing, ensuring traffic is distributed smoothly across multiple nodes instead of overwhelming a single active instance.

By fine-tuning these settings, you can tailor Kthena's routing performance to match your specific models and business LLM workloads.

Granular, Resource-Efficient Rolling Updates

Historically, Kthena executed rolling updates at the entire ServingGroup level. For massive LLM applications, completely rebuilding a ServingGroup is an incredibly resource-intensive and time-consuming process.

To solve this, we are introducing role-based rolling updates. You no longer need to update an entire ServingGroup when only a specific Role requires changes (which is also why we introduced RoleRecreate in the recovery policy). Starting from v0.4.0, you can dynamically adjust your rolloutStrategy—drastically cutting down resource consumption, speeding up deployment times.

An Open Ecosystem

We are deeply committed to building Kthena as an open, inclusive, and thriving project alongside the wider open-source community.

ModelScope Support

In v0.4.0, we have extended Kthena's model downloader with ModelScope protocol support. This allows users and operators to choose a model repository that suits their needs.

Support for A Variety of Inference Engines

Furthermore, we are proud that ModelServing is now thoroughly verified to support leading inference engines like vLLM and SGLang.

By seamlessly integrating with diverse AI technologies rather than locking you into a single solution, Kthena continues to deepen its roots within the vibrant Cloud Native AI landscape. We warmly invite developers worldwide to join us in building this inclusive and dynamic future together!

What's Changed

Exciting New Features 🎉

Bug fixes 🐛

  • fix table overflow in LWS integration documentation by @thisisharsh7 in #751
  • fix: hide navigation on docs lightboxImage by @thisisharsh7 in #750
  • fix roling update block by always unavailable servingGroup by @LiZhenCheng9527 in #744
  • Fix Prefix Cache Plugin: Candidate Filtering, Score Completeness, and API Refactor by @YaoZengzeng in #783
  • fix data race in model serving controller by @YaoZengzeng in #800
  • fix controller panic when create modelserving with pod annotations by @VanderChen in #808
  • allow partition to be greater than or equal to replicas by @VanderChen in #810
  • [bug] Fixed an error where the headless service was not pointing solely to the entry pod by @LiZhenCheng9527 in #821
  • [BUG] fix bug about servingGroup partition in role scale up by @LiZhenCheng9527 in #860
  • fix: panic when workerTemplate is nil by @anirudh240 in #746
  • Bug fix: When deleting a Pod associated with a single Pod role, the role is not recreated by @LiZhenCheng9527 in #891
  • test/e2e: fix flake in WaitForModelServingReady by @vyagh in #876

Documentation Updates 📚

Other Changes 🔄

  • fix an erro...
Read more

Contributors

YaoZengzeng, VanderChen, and 18 other contributors
Assets 9
Loading

v0.4.0-rc.0

v0.4.0-rc.0 Pre-release
Pre-release

Choose a tag to compare

View all tags
@hzxuzhonghu hzxuzhonghu released this 17 Apr 08:41
v0.4.0-rc.0
2916fb7
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

This release includes the following artifacts:

  • Helm Chart: kthena.tgz (OCI artifact pushed to GHCR)
  • Installation YAML: kthena-install.yaml
  • CLI Binaries: Pre-built binaries for Linux (amd64/arm64), macOS (amd64/arm64), and Windows (amd64)

Documentation

Release notes are automatically generated from commits.

What's Changed

Exciting New Features 🎉

Bug fixes 🐛

Documentation Updates 📚

Other Changes 🔄

Read more

Contributors

YaoZengzeng, git-malu, and 26 other contributors
Loading

Kthena v0.3.0

Kthena v0.3.0

Choose a tag to compare

View all tags
@hzxuzhonghu hzxuzhonghu released this 31 Jan 10:49
v0.3.0
5564493
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Summary

Release v0.3.0 establishes Kthena as a more robust and scalable platform for AI inference workloads. This release introduces significant enhancements in ModelServing, Router, and ModelBooster. Key highlights include seamless integration with LeaderWorkerSet, advanced network topology-aware scheduling for PD disaggregation, and a comprehensive Router Observability framework. Additionally, this version brings native ModelServing version control, support for vLLM data parallel deployment, and a complete E2E test suite for the router, ensuring high stability and reliability for production environments.

What's New

Key Features Overview

  • LeaderWorkerSet Support: Integration with the LeaderWorkerSet (LWS) API allows for sophisticated management of distributed inference workloads.
  • Role-Level Gang Scheduling & Topology Awareness: Leverages Volcano's new subGroupPolicy feature to enable fine-grained, role-based gang scheduling and network topology awareness.
  • ModelServing Partition Revision Control: Introduced a native revision-based version control system for ModelServing.
  • Router Observability & Debugging: Comprehensive documentation and framework for router observability, plus a dedicated debug port.
  • Enhanced Rolling Updates: Support for maxUnavailable allows tuning the velocity of updates for faster rollouts.
  • Plugin Support: Flexible plugin architecture for ModelServing to inject custom configuration logic.

LeaderWorkerSet Support for ModelServing Role

Background and Motivation:
Distributed inference workloads often require complex topologies where a leader pod manages multiple worker pods. Configuring these relationships manually can be error-prone. By integrating with the Kubernetes LeaderWorkerSet (LWS) API, Kthena simplify the deployment and management of these workloads.

Key Capabilities:

  • Direct Integration: ModelServing Roles can now leverage LWS to automatically manage leader-worker groups.
  • Simplified Topology: Reduces the complexity of defining distributed inference services requiring strict coordination.

Related:

Role-Level Gang Scheduling & Topology Awareness

Background and Motivation:
In Prefill-Decode (PD) separation scenarios, the communication overhead between prefill and decode instances is critical. Ensuring these instances are scheduled closer together (e.g., on the same switch or rack) significantly improves performance. Kthena now enables fine-grained, role-level control over both gang scheduling and network topology awareness by leveraging Volcano's subGroupPolicy.

Key Capabilities:

  • Declarative Topology Policies: Configure distinct network topology constraints for the entire ServingGroup (groupPolicy) and for individual Roles (rolePolicy) directly in the ModelServing spec.
  • Automatic Pod Grouping: The controller automatically labels Pods with modelserving.volcano.sh/role and modelserving.volcano.sh/role-id, enabling Volcano to form subGroups for precise topology-aware placement.
  • Performance Optimization: Minimizes inter-role communication latency and maximizes bandwidth utilization for intensive distributed inference jobs by co-locating related tasks on network-proximal nodes.
  • Role-Level Gang Scheduling: The subGroupPolicy also enforces gang scheduling at the role level, ensuring that all Pods belonging to a specific role (e.g., all prefill-0 Pods) are scheduled together as an atomic unit. This guarantees that partial deployments of a role do not occur, which is critical for correctness in distributed inference workloads.

Note: This feature requires Volcano v1.14+ for subGroupPolicy support.

Related:

ModelServing Partition Revision Control

Background and Motivation:
The partition field in a Kthena ModelServing defines a boundary for rolling updates, allowing you to partition the update process so that only a subset of ServingGroups are updated while others remain on the previous version. It is primarily used for canary deployments, phased rollouts, and staging updates in stateful applications where strict control over update order is necessary.

Key Capabilities:

  • Revision Tracking: Automatically tracks changes to ModelServing configurations.
  • Partition Protection: Supports partition-based updates to ensure service continuity during rollouts.
  • Rollback: Easily revert to a previous stable revision.

Related:

Router Observability & Debugging

Background and Motivation:
Deep visibility into the inference router is essential for diagnosing latency issues and ensuring SLA compliance. The new observability framework and debug port provide the necessary tools for operators.

Key Capabilities:

  • Debug Port: A dedicated port (default 15000) for real-time inspection of routing tables and upstream health.
  • Comprehensive Metrics: Detailed documentation and setup for monitoring request latency, throughput, and error rates.
  • E2E Testing: A robust E2E test framework covering most routing scenarios ensures reliability.

Related:

Other Notable Changes

Features and Improvements

  • [ModelServing] Support maxUnavailable in modelserving rolling update #640 (@LiZhenCheng9527)
  • [ModelServing] Implement extension plugin framework #588 (@hzxuzhonghu)
  • [ModelServing] Support vLLM data parallel deployment and Expert Parallel modes
  • [CLI] Add templates for PD disaggregation use cases #571 (@huntersman)
  • [Client] Make client QPS and Burst customizable #686 (@FAUST-BENCHOU)
  • [Webhooks] Enable ModelServing webhooks by default in Helm charts #694 (@VanderChen)
  • [Infra] One-click deploy from source via hack/local-up-kthena.sh #613 (@FAUST-BENCHOU)

Bug Fixes

  • [Scheduler] Fix divide-by-zero in LeastRequest scoring #723 (@WHOIM1205)
  • [Controller] Fix role status transition to Running to restore scale-down protection #706 (@WHOIM1205)
  • [Controller] Fix panic in PD scheduler when no prefill pods are available #714 (@WHOIM1205)
  • [Controller] Fix silent recovery of failed pods after ModelServing controller restart #697 (@WHOIM1205)
  • [Controller] Fix recovering headless services after deletion #598 (@LiZhenCheng9527)
  • [Controller] Fix validate gangpolicy minRoleReplicas #699 (@VanderChen)
  • [Controller] Fix controllerrevision data warping #698 (@VanderChen)
  • [Controller] Fix modelserving controller panic #688 (@LiZhenCheng9527)
  • [Controller] Fix restart during modelserving create: pod number mismatch #689 (@hzxuzhonghu)
  • [Controller] Check role.Name in ModelServing validator #684 (@FAUST-BENCHOU)
  • [Controller] Fix bug where role deletion did not trigger reconstruction #629 (@LiZhenCheng9527)
  • [Router] Protect Headless Services Created by ModelServing #598 (@LiZhenCheng9527)

Contributors

Thank you to all contributors who made this release possible:

@hzxuzhonghu, @LiZhenCheng9527, @YaoZengzeng, @git-malu, @FAUST-BENCHOU, @katara-Jayprakash, @zhiweideren, [@aaradhychinche-alt...

Read more
Loading

Kthena v0.2.0

Kthena v0.2.0

Choose a tag to compare

View all tags
@hzxuzhonghu hzxuzhonghu released this 10 Dec 09:48
v0.2.0
6a69061
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

After a month of dedicated effort from the Kthena community contributors, we are excited to announce the official release of Kthena v0.2.0!

This release brings a range of practical features and significant improvements, specifically designed to empower AI workloads and meet the needs of our users. Notably, Kthena v0.2.0 delivers comprehensive optimizations across its router, modelBooster, and modelserving. These enhancements enable Kthena to foster a richer open source ecosystem and provide users with more diverse services. We sincerely thank all contributors for their hard work and feedback, which have made this release possible.

Key Features and Enhancements

Solve model name conflicts by supporting the gateway api:

Kthena Router supports Kubernetes Gateway API, providing more flexible traffic management and routing capabilities for model inference services.In traditional routing configurations, the modelName field in ModelRoute resources is global. When multiple ModelRoute resources use the same modelName, conflicts occur, leading to undefined routing behavior.
By using Gateway API, you can bind different ModelRoute resources to different Gateway resources. Even if they use the same modelName, there will be no conflicts. Each Gateway can listen on different ports, creating independent routing spaces.
Also gateway API Inference Extension depends on the foundational features of Gateway API. In order to support Gateway API Inference Extension, Kthena Router needs to support Gateway API first.

Router Supports Gateway API Inference Extension:

Gateway Inference Extension provides a standardized way to expose AI/ML inference services through Kubernetes Gateway API. In release v0.1.0, we have already added support for it through third-party projects such as Istio. Now we have native support for it in Kthena Router which means we support both the native ModelRoute/ModelServer and the upstream Gateway API Inference Extension. Users can easily migrate traffic from other Gateway API Inference Extension implementations. Additionally, users could try advanced features offered by ModelRouter/ModelServer, such as PD disaggregation, without needing to restart or redeploy Kthena router. We could manage the traffic of both API simultaneously!

Enhancements of kthena CLI:

The Kthena CLI provides kubectl‑style commands for managing AI inference workloads on Kubernetes, featuring quick deployment via curated templates and optional integration with kubectl‑ai for natural‑language command generation.

AutoScale Supports Scaling PD Instances Proportionally:

A new subTarget field is added to the Autoscaling Policy Binding, which supports Homogeneous Scaling at the granularity of the ModelServing Role. This feature can be used for SLO-based dynamic scaling in the deployment mode where Prefill and Decode are separated.

Network Topology Aware Scheduling:

Kthena leverages Volcano's new subGroupPolicy feature to enable role-based, network topology-aware scheduling. With this enhancement, users can configure sophisticated network topology constraints not only for pods within a ServingGroup, but also for specific Roles. By defining both group policies and role policies, users gain fine-grained control over how workloads are distributed across nodes, ensuring that frequently communicating pods are scheduled closer together and network bandwidth is utilized efficiently. This capability helps optimize performance and resource utilization for large-scale AI inference scenarios.

NOTE: This feature relies on Volcano's new API and requires the Volcano latest image. Even the volcano v1.12.0 is not enough

Support Binpack Scale Down:

Binpack scaling down is designed to maximize available node capacity, helping the cluster efficiently prepare for upcoming resource-intensive tasks. In Kthena, the Binpack Scale Down feature utilizes the controller.kubernetes.io/pod-deletion-cost annotation on each pod to intelligently manage capacity. Kthena evaluates the deletion cost assigned to each group or role based on the value of this annotation, ranks them in order from lowest to highest cost, and selectively removes the group or role with the lowest deletion cost. This targeted approach ensures that cluster resources are freed in a way that minimizes disruption, avoids unnecessary loss of high-priority workloads, and maintains optimal readiness for scaling large AI jobs.

Streamline Model Booster Configuration:

The configuration of Model Booster has been significantly streamlined to align with its goal of efficient and simple deployment. We have removed redundant features such as heterogeneous scaling between backends, routing weights, and dynamic Lora mounting to reduce complexity. Meanwhile, core capabilities like ModelMatch, custom schedulers, and multi-node deployment scenarios (e.g., Ray) are fully retained to ensure robust inference performance.

NOTE: These changes apply exclusively to Model Booster to ensure a streamlined user experience. Advanced features, including heterogeneous backend scaling, routing weights, and dynamic LoRA mounting, remain fully available in Model Serving.

This release includes the following artifacts:

  • Helm Chart: kthena.tgz (OCI artifact pushed to GHCR)
  • Installation YAML: kthena-install.yaml
  • CLI Binaries: Pre-built binaries for Linux (amd64/arm64), macOS (amd64/arm64), and Windows (amd64)

Documentation

Release notes are automatically generated from commits.

What's Changed

Other Changes 🔄

Full Changelog: v0.2.0-rc.0...v0.2.0

Contributors

volcano-sh-bot
Loading

v0.2.0-rc.0

v0.2.0-rc.0 Pre-release
Pre-release

Choose a tag to compare

View all tags
@hzxuzhonghu hzxuzhonghu released this 09 Dec 13:15
v0.2.0-rc.0
e785f8e
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

This release includes the following artifacts:

  • Helm Chart: kthena.tgz (OCI artifact pushed to GHCR)
  • Installation YAML: kthena-install.yaml
  • CLI Binaries: Pre-built binaries for Linux (amd64/arm64), macOS (amd64/arm64), and Windows (amd64)

Documentation

Release notes are automatically generated from commits.

What's Changed

Exciting New Features 🎉

Bug fixes 🐛

Documentation Updates 📚

Other Changes 🔄

Read more

Contributors

kevin-wangzefeng, YaoZengzeng, and 13 other contributors
Loading

Kthena v0.1.0

Kthena v0.1.0

Choose a tag to compare

View all tags
@github-actions github-actions released this 31 Oct 07:06
v0.1.0
f430562
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Kthena v0.1.0 Release Notes

Thanks to the efforts of all Kthena contributors, we are pleased to announce the official release of Kthena v0.1.0!

Kthena is a Kubernetes-native LLM inference platform designed for efficient deployment and management of Large Language Models in production. It uses declarative model lifecycle management and intelligent request routing for high performance and scalability, enabling cost-effective autoscaling and seamless integration with cloud-native workflows.

Highlights

  • Intelligent, model-aware routing purpose-built for LLM serving workloads
  • Simplified LLM lifecycle management via Kubernetes-native CRDs
  • One-step LLM deployment and autoscaling with ModelBooster
  • Ready-to-use templates for popular open LLMs

Intelligent Routing

Kthena Router is a Kubernetes-native, standalone inference router purpose-built for LLM serving workloads. Unlike generic proxies or load balancers, Kthena Router is model-aware and makes intelligent routing decisions based on real-time metrics from inference engines. This enables sophisticated traffic management strategies that significantly improve throughput, reduce latency, and lower operational costs. The router integrates seamlessly with existing API gateway infrastructure while providing advanced capabilities specifically designed for AI workloads:

  • Model-Aware Routing: Leverages real-time metrics from inference engines (vLLM, SGLang, TGI) to make intelligent routing decisions.
  • LoRA-Aware Load Balancing: Intelligently routes to pods that have already loaded the desired LoRA adapter to reduce adapter swap latency from hundreds of milliseconds to near-zero.
  • Advanced Scheduling Algorithms: Includes Prefix Cache Aware, KV Cache Aware, and Fairness Scheduling strategies.
  • Prefill-Decode Disaggregation: Native support for xPyD (x-prefill/y-decode) deployment patterns.

Simplified LLM Management

Kthena introduces the ModelServing custom resource definition (CRD) to represent LLMs with a three-layer architecture: ModelServing → ServingGroup → Role. This structure makes it easy to express and manage various deployment strategies for LLMs, such as PD-disaggregation or native deployments.

  • Each ServingGroup represents an LLM and supports gang scheduling and network topology-aware scheduling for the pods within a ServingGroup.
  • ModelServing supports rolling updates, scaling, and restart policies at different granularities.
  • Enables flexible definition and management of complex model service topologies for efficient resource utilization and seamless operations.

One-Step LLM Deployment

Kthena’s ModelBooster streamlines end-to-end model deployment in Kubernetes:

  • Create a ModelBooster CR and let the modelbooster-controller handle the rest.
  • The controller automatically creates all required resources—ModelServing, ModelServer, and ModelRoute—to build a complete inference system.
  • Built-in autoscaling adjusts resource allocation based on system metrics to maximize efficiency.
  • When the ModelBooster’s condition is “Active,” the model is ready to serve.
  • Updates and deletes follow the same simple workflow.

Refer to the Quick Start guide to get up and running in minutes.

Getting Started

New Contributors

Full Changelog: https://github.com/volcano-sh/kthena/commits/v0.1.0

Acknowledgements

A big thank you to all contributors and users who provided feedback and helped shape this first official release of Kthena.

Contributors

kevin-wangzefeng, YaoZengzeng, and 12 other contributors
Loading

v0.1.0-rc.1

v0.1.0-rc.1 Pre-release
Pre-release

Choose a tag to compare

View all tags
@github-actions github-actions released this 29 Oct 03:26
v0.1.0-rc.1
4aec881
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

What's Changed

Other Changes 🔄

Read more

Contributors

kevin-wangzefeng, YaoZengzeng, and 12 other contributors
Loading