[DO NOT MERGE] Add llm-d v0.8 release blog post#405
Open
chcost wants to merge 1 commit into
Open
Conversation
Signed-off-by: Carlos H. Andrade Costa <chcost@us.ibm.com>
chcost
requested review from
Gregory-Pereira,
clubanderson,
davidgs,
jjasghar,
petecheslock and
robertgshaw2-redhat
as code owners
July 2, 2026 00:01
✅ Deploy Preview for llm-d ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
chcost
changed the title
ahg-g
reviewed
Jul 6, 2026
| @@ -0,0 +1,118 @@ | |||
| --- | |||
| title: "llm-d v0.8: From Platform to Control Plane" | |||
| description: "llm-d v0.8 graduates Flow Control, Batch Gateway, and multi-modal serving to production, extends beyond Kubernetes with RL/Slurm support, and aligns with upstream vLLM — sharpening the project's identity as an inference control plane for any environment." | |||
Collaborator
There was a problem hiding this comment.
Another two important features:
- The first agentic serving guide
- Expanding model server coverage to sglang across most guides (the only one not covered is wide-ep right now).
|
|
||
| **Predicted latency scheduling** receives a complete rewrite with a new production-ready guide. The latency predictor provides model-specific latency estimates that the endpoint picker uses for routing decisions, moving beyond queue-depth heuristics to predictions grounded in actual model serving characteristics. Nightly CI now validates that the predictor is serving real predictions rather than falling back to heuristics — asserting that the optimization path operators configured is actually running. | ||
|
|
||
| **SGLang support** lands in the precise prefix cache routing path. The prefix cache scorer, previously vLLM-only, now works with SGLang backends through an [updated well-lit path guide](https://github.com/llm-d/llm-d/tree/release-0.8/guides/precise-prefix-cache-routing) with optimal configuration and benchmark numbers. This makes llm-d's most impactful routing optimization — prefix-cache-aware request placement — engine-agnostic. |
Collaborator
There was a problem hiding this comment.
I think this deserves its own section, it is important to highlight that the project is not vllm specific anymore
|
|
||
| Each release expands the set of tested, documented deployment patterns — what we call well-lit paths. v0.8 adds several that reflect the project's broadening ecosystem. | ||
|
|
||
| **TensorRT-LLM** receives a [full recipe and optimized baseline guide](https://github.com/llm-d/llm-d/tree/release-0.8/guides/optimized-baseline/modelserver/gpu/trtllm). This is the first well-lit path using a non-vLLM inference engine, demonstrating that llm-d's control plane works independently of the model server underneath. The recipe uses `trtllm-serve` with llm-d's router handling prefix-cache-aware scheduling on top. |
Collaborator
There was a problem hiding this comment.
is it the first non-vllm inference engine? we had sglang for the optimized-baseline and expanded its coverage in this release.
|
|
||
| **TensorRT-LLM** receives a [full recipe and optimized baseline guide](https://github.com/llm-d/llm-d/tree/release-0.8/guides/optimized-baseline/modelserver/gpu/trtllm). This is the first well-lit path using a non-vLLM inference engine, demonstrating that llm-d's control plane works independently of the model server underneath. The recipe uses `trtllm-serve` with llm-d's router handling prefix-cache-aware scheduling on top. | ||
|
|
||
| **DeepSeek-V4 on GB200 NVL72** gets a dedicated deployment guide, running a frontier MoE model on NVIDIA's latest hardware with WideEP expert parallelism. The guide includes API server scaling (bumped from 1 to 4 replicas) and NVSHMEM configuration for the GB200's NVLink domain. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
release-0.8branch (not PR URLs)Test plan
npm run buildpasses with no new broken linksnpm run serve