Release blog for 0.3 #105
Merged
Release blog for 0.3 #105
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
✅ Deploy Preview for elaborate-kangaroo-25e1ee ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
smarterclayton
force-pushed
the
blog
branch
2 times, most recently
from
1b2442e to
2cf16af
Compare
kfswain
reviewed
Oct 10, 2025
|
|
||
| # llm-d 0.3: Wider Well-Lit Paths for Scalable Inference | ||
|
|
||
| In our [0.2 release](https://llm-d.ai/blog/llm-d-v0.2-our-first-well-lit-paths), we introduced the first *well-lit paths*, tested blueprints for scaling inference on Kubernetes. With 0.3 release, we double down on the mission: to provide a fast path to deploying high performance, hardware-agnostic, easy to operationalize, at scale inference. |
There was a problem hiding this comment.
These are definitely nits/opinions, but:
Suggested change
| In our [0.2 release](https://llm-d.ai/blog/llm-d-v0.2-our-first-well-lit-paths), we introduced the first *well-lit paths*, tested blueprints for scaling inference on Kubernetes. With 0.3 release, we double down on the mission: to provide a fast path to deploying high performance, hardware-agnostic, easy to operationalize, at scale inference. | |
| In our [0.2 release](https://llm-d.ai/blog/llm-d-v0.2-our-first-well-lit-paths), we introduced the first *well-lit paths*, tested blueprints for scaling inference on Kubernetes. With 0.3 release, we double down on the [mission](#commit-to-the-mission): to provide a fast path to deploying high performance, hardware-agnostic, easy to operationalize, at-scale inference. |
blog/2025-10-10_llm-d-v0.3-expanded-hardware-faster-perf-and-igw-ga.md
Outdated
Show resolved
Hide resolved
blog/2025-10-10_llm-d-v0.3-expanded-hardware-faster-perf-and-igw-ga.md
Outdated
Show resolved
Hide resolved
blog/2025-10-10_llm-d-v0.3-expanded-hardware-faster-perf-and-igw-ga.md
Outdated
Show resolved
Hide resolved
|
|
||
| Serving LLM is complex \- our documentation and configuration should be simple. Quickstarts have been streamlined and renamed to guides, with fewer options and more context around the key decisions you need to make. They are now located in the main repository and treated as living documents alongside our growing documentation for common scenarios. Since llm-d is about exposing the key tradeoffs and exposing useful patterns, we’ve split out the key prerequisites for each guide \- cluster configuration, client setup, and gateway choice \- into their own sections, and replaced our all-in-one installer scripts with better step by step instructions. | ||
|
|
||
| As more cluster providers integrate into llm-d we’ve expanded the documentation for infrastructure with provider specific troubleshooting, configuration, and testing. This release adds documentation and steps for CoreWeave, Digital Ocean, Google Kubernetes Engine, and OpenShift. |
There was a problem hiding this comment.
Suggested change
| As more cluster providers integrate into llm-d we’ve expanded the documentation for infrastructure with provider specific troubleshooting, configuration, and testing. This release adds documentation and steps for CoreWeave, Digital Ocean, Google Kubernetes Engine, and OpenShift. | |
| To support more cluster providers integrating into llm-d, we’ve expanded the documentation for infrastructure with provider specific troubleshooting, configuration, and testing. This release adds documentation and steps for CoreWeave, Digital Ocean, Google Kubernetes Engine, and OpenShift. |
blog/2025-10-10_llm-d-v0.3-expanded-hardware-faster-perf-and-igw-ga.md
Outdated
Show resolved
Hide resolved
| As more cluster providers integrate into llm-d we’ve expanded the documentation for infrastructure with provider specific troubleshooting, configuration, and testing. This release adds documentation and steps for CoreWeave, Digital Ocean, Google Kubernetes Engine, and OpenShift. | ||
| Guides now include curated Inference Gateway installs and static manifests for clarity, with overlays available for benchmarking sweeps. RBAC patterns were refactored toward namespace scope for smoother multi-tenancy | ||
|
|
||
| **Why it matters:** With 0.3, experimenting with intelligent scheduling or disaggregation is as simple as running a documented guide. The control plane is more transparent, reproducible, and extensible, independently of the platform you are running. |
There was a problem hiding this comment.
Suggested change
| **Why it matters:** With 0.3, experimenting with intelligent scheduling or disaggregation is as simple as running a documented guide. The control plane is more transparent, reproducible, and extensible, independently of the platform you are running. | |
| **Why it matters:** With 0.3, experimenting with intelligent scheduling or disaggregation is as simple as running a documented guide. The control plane is more transparent, reproducible, and extensible; independent of the platform you are running. |
blog/2025-10-10_llm-d-v0.3-expanded-hardware-faster-perf-and-igw-ga.md
Outdated
Show resolved
Hide resolved
blog/2025-10-10_llm-d-v0.3-expanded-hardware-faster-perf-and-igw-ga.md
Outdated
Show resolved
Hide resolved
kfswain
reviewed
Oct 10, 2025
blog/2025-10-10_llm-d-v0.3-expanded-hardware-faster-perf-and-igw-ga.md
Outdated
Show resolved
Hide resolved
blog/2025-10-10_llm-d-v0.3-expanded-hardware-faster-perf-and-igw-ga.md
Outdated
Show resolved
Hide resolved
|
/lgtm |
|
|
||
| ### **Wide-EP Performance** | ||
|
|
||
| The wide-EP path, which parallelizes across experts to maximize throughput, has reached **2.7k tokens/s per GPU** in community benchmarks on H200 clusters. |
blog/2025-10-10_llm-d-v0.3-expanded-hardware-faster-perf-and-igw-ga.md
Outdated
Show resolved
Hide resolved
|
LGTM, pending nit on the topline performance number |
|
Added a truncation comment, but feel free to adjust the location, this just ensures the main /blog landing page isn't just the entire blog but a list of them. |
petecheslock
approved these changes
Oct 10, 2025
|
Link failure build can be ignored, that's just the link for this blog and will be "fixed" when it goes live. |
Signed-off-by: Clayton Coleman <[email protected]>
Link to release Signed-off-by: Clayton Coleman <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.