[Blog] Refine heterogeneous inference blog positioning

blog/2025-05-20_announce.md

@@ -188,7 +188,7 @@ llm-d builds brings together the performance of vLLM with the operationalizabili
 We welcome AI engineers and researchers to join the llm-d community and contribute:
 
 * Check out our repository on Github: [https://github.com/llm-d/llm-d](https://github.com/llm-d/llm-d)   
-* Join our developer slack: <a href="/slack" target="_self">/slack</a>  
+* Join our developer slack: <a href="https://llm-d.ai/slack" target="_self">/slack</a>
 * Try out our quick starts to deploy llm-d on your Kubernetes cluster: [https://github.com/llm-d/llm-d-deployer/tree/main/quickstart](https://github.com/llm-d/llm-d-deployer/tree/main/quickstart)
 
 Please join us. The future of AI is open.

blog/2025-06-03_week_1_round_up.md

@@ -45,7 +45,7 @@ We use Google Groups to share architecture diagrams and other content. Please jo
     - All SIG meetings
     - Weekly Community Meeting, every Wednesday @ 12:30pm Eastern Time
 * Join the [llm-d Google group](http://red.ht/llm-d-google-group) - this is where all engineering drafts are shared. You must join the group to have access to project google docs.
-* Join <a href="/slack" target="_self">Slack</a>
+* Join <a href="https://llm-d.ai/slack" target="_self">Slack</a>
 * [llm-d Roadmap for Q2](https://github.com/llm-d/llm-d/issues/26)
 
 **Where to follow us on Social Media**

blog/2025-06-25_community_update.md

@@ -71,7 +71,7 @@ There are many ways to contribute to llm-d:
 2. Review the [project public calendar](https://red.ht/llm-d-public-calendar) and join an upcoming community meeting
 3. Subscribe to our [YouTube channel](https://youtube.com/@llm-d-project) for tutorials and meeting recordings
 4. Fill out the [Use Case Survey](https://docs.google.com/forms/d/e/1FAIpQLScENfBNrN1q8XpugCmDUjy8sSn1cMzi_V0mCWTRwzixcPAQkw/viewform) to help shape our benchmarks
-5. Join our <a href="/slack" target="_self">Slack workspace</a> for real-time discussions
+5. Join our <a href="https://llm-d.ai/slack" target="_self">Slack workspace</a> for real-time discussions
 6. Check out our [Contributor Guidelines](https://llm-d.ai/community/contribute) to start contributing code
 
 We're looking forward to hearing from you and working together to make llm-d even better!

blog/2025-09-24_kvcache-wins-you-can-see.md

@@ -18,7 +18,7 @@ tags: [blog, updates, llm-d]
 
 The llm-d project provides a series of “well-lit paths” \- tested, benchmarked solutions for deploying large language models in production. Our first path, [**Intelligent Inference Scheduling**](/blog/intelligent-inference-scheduling-with-llm-d), established a baseline for AI-aware routing by balancing both cluster load and prefix-cache affinities. The default configuration for that path uses an *approximate* method for the latter, predicting cache locality based on request traffic.
 
-This blog illuminates a more advanced and powerful path: [**precise prefix-cache aware scheduling**](/docs/dev/guides/precise-prefix-cache-routing).
+This blog illuminates a more advanced and powerful path: [**precise prefix-cache aware scheduling**](https://github.com/llm-d/llm-d/tree/main/guides/precise-prefix-cache-routing).
 
 We take a deep dive into the next generation of this feature, which moves beyond prediction and gives the scheduler direct introspection into distributed vLLM caches. This precision is key to maximizing cache hit rates and achieving a new level of performance and maximizing cost-efficiency in your distributed deployments.
 
@@ -284,7 +284,7 @@ The optimal scheduler depends on the complexity of the workload. Below is a hier
 The llm-d project thrives on community contributions, and there are many ways to get involved:
 
 * Explore the llm-d Community Quickstart Guide → [Start here](/community) to learn more about getting involved in the llm-d project.
-* Join our Slack → [Get your invite](/slack) and connect with maintainers and contributors
+* Join our Slack → [Get your invite](https://llm-d.ai/slack) and connect with maintainers and contributors
 * Explore the code → Browse our [GitHub organization](https://github.com/llm-d) and find issues that interest you
 * Attend meetings → All meetings are open\! Add our [public calendar](/community#public-meeting-calendar) and join discussions\`
 
@@ -324,4 +324,4 @@ The overhead for this global bookkeeping is that of storing KV-block hashes, and
 
 A full KV-cache for the **DeepSeek R1** model running with FP8 on **8x** **NVIDIA H200 GPUs** with a total of **45.7 GB \* 8 \= 365 GB VRAM** dedicated to the KV-cache pool would be composed of thousands of individual memory blocks. Each block, representing **128 tokens**, consumes about **8.6 MB** of VRAM. However, the metadata needed to track each of these blocks in the global index is just a single 64-bit hash, which is **8 bytes**. The additional overhead of maintaining a data structure is trivial if efficient. 
 
-This means that managing the entire **365 GB** cache pool requires only about **339 KB** of memory for the scheduler's index \- a data-to-metadata ratio of over **1,000,000-to-1**. The index's memory footprint is several orders of magnitude smaller than the GPU VRAM it tracks, making this a highly efficient, low-overhead solution.
+This means that managing the entire **365 GB** cache pool requires only about **339 KB** of memory for the scheduler's index \- a data-to-metadata ratio of over **1,000,000-to-1**. The index's memory footprint is several orders of magnitude smaller than the GPU VRAM it tracks, making this a highly efficient, low-overhead solution.

blog/2026-04-21_production-grade-llm-inference-at-scale-kserve-llm-d-vllm.md

@@ -73,7 +73,7 @@ We’d like to thank everyone from the community who has contributed to the succ
 The work described here is just one example of what becomes possible when a community of engineers tackles hard problems together in the open. If you're running LLMs at scale and wrestling with the same challenges — storage, routing, efficiency, day-2 operations — we'd love to have you involved.
 
 * **Explore the code** → Browse our [GitHub organization](https://github.com/llm-d) and dig into the projects powering this stack
-* **Join our Slack** → [Get your invite](/slack) and connect directly with maintainers and contributors from Red Hat, Tesla, and beyond
+* **Join our Slack** → [Get your invite](https://llm-d.ai/slack) and connect directly with maintainers and contributors from Red Hat, Tesla, and beyond
 * **Attend community calls** → All meetings are open! Add our [public calendar](https://red.ht/llm-d-public-calendar) (Wednesdays 12:30pm ET) and join the conversation
 * **Follow project updates** → Stay current on [Twitter/X](https://twitter.com/_llm_d_), [Bluesky](https://bsky.app/profile/llm-d.ai), and [LinkedIn](https://www.linkedin.com/company/llm-d)
 * **Watch demos and recordings** → Subscribe to the [llm-d YouTube channel](https://www.youtube.com/@llm-d-project) for community call recordings and feature walkthroughs

blog/2026-06-09_heterogeneous-inference-3-vendor-sovereign-cluster.md

@@ -0,0 +1,135 @@
+---
+title: "Heterogeneous inference serving across three GPU vendors with llm-d"
+description: "Benchmarking llm-d's prefix-cache-aware routing across anonymized GPU pools on the NxtGen sovereign cloud, showing how one routing layer improves throughput and TTFT across single-vendor and heterogeneous fleets."
+slug: heterogeneous-inference-3-vendor-sovereign-cluster
+date: 2026-06-09T09:00
+
+authors:
+  - praveinkannan
+  - praveenjayachandran
+  - jaikhari
+  - varunraste
+  - prasadmukhedkar
+  - vinodpathangay
+  - jayanthreddy
+  - abhisyant
+
+tags: [blog, inference, scheduling, kv-cache, sig-benchmarking]
+---
+# Heterogeneous inference serving across three GPU vendors with llm-d
+
+Most production inference clusters today are single-vendor because that is often the simplest way to configure and operate a cluster.
+
+That is starting to change. Procurement cycles bring new generations alongside older ones, supply planning spans multiple accelerator options, and cost/performance profiles differ by workload. Real production fleets are accumulating heterogeneity whether or not the architecture planned for it.
+
+This is an opportunity to unlock real value: different accelerator classes can be matched to workload requirements, stranded capacity gets reclaimed, and operators gain more flexibility in capacity planning. The case is stronger still for sovereign and on-premise deployments, where data residency, regulatory alignment, and the long-term economics of high-volume inference make local fleet optimization especially important.
+
+Making that work in practice is a non-trivial systems problem. Each accelerator stack brings its own optimized drivers, firmware, container images, runtime settings, and attention kernels. A coherent serving layer needs to preserve those platform-specific optimizations while still giving operators one control plane for routing, observability, and policy.
+
+<!-- truncate -->
+
+## Setup
+
+To evaluate llm-d on a heterogeneous environment, we ran experiments on the **NxtGen sovereign cloud's** mixed GPU environment, with the following accelerator pools within a single OpenShift AI cluster:
+
+| Pool | Hardware | Count |
+| :---- | :---- | :---- |
+| Vendor A | 2 nodes, each with 2 GPUs | 4 |
+| Vendor B | 1 node with 8 GPUs | 8 |
+| Vendor C | 1 node with 8 GPUs | 8 |
+
+We use anonymized labels to keep the focus on the serving-layer behavior: how llm-d routes the same workload across the accelerator pools available in this test environment. The pools differ in generation, memory capacity, runtime characteristics, and expected per-GPU performance, so the results should be read as routing-policy comparisons within each pool and across the combined fleet.
+
+All nodes are connected over a shared **100 G RoCE** network. We pinned each vLLM replica to a single accelerator card (TP = 1) to maximize the number of independent serving instances and exercise the routing layer.
+
+Models served:
+
+* `ibm-granite/granite-4.1-8b` — 8 B parameter, decoder-only dense transformer model
+* `sarvamai/sarvam-30b` — 30 B MoE, Indic-multilingual model
+
+The workload is the prefill-heavy `shared_prefix_synthetic` from [inference-perf](https://github.com/kubernetes-sigs/inference-perf): a long shared system prompt + short question + decode-tolerant output (~7.2K input tokens + 1K output tokens). This matches production RAG, chat, and citizen-services traffic profiles where prefix-cache routing is most beneficial.
+
+## What this benchmark shows
+
+This benchmark evaluates llm-d as a routing and control-plane layer over a mixed accelerator fleet. It does not compare accelerator vendors directly. Instead, it asks whether a cache-aware, saturation-aware router can make single-vendor and heterogeneous pools behave like one inference service, using the same vLLM pods for both the llm-d run and the Kubernetes round-robin baseline within each pool.
+
+The results are most relevant to operators who already have mixed capacity: sovereign cloud teams, enterprise AI platform teams, on-prem clusters, and organizations with staggered hardware refresh cycles. For these teams, the question is not a standalone accelerator ranking; it is how to route production traffic across the fleet they actually operate.
+
+## Prefix-aware caching
+
+We deployed llm-d v0.0.7 with prefix-cache-aware routing in two flavours, picked per-model rather than per-pool: **`granite-4.1-8b` runs against the [precise prefix-cache scorer](https://github.com/llm-d/llm-d/tree/main/guides/precise-prefix-cache-routing)** (tokenizer-backed, exact prefix matching), and **`sarvam-30b` runs against the approximate prefix-cache scorer** (xxhash over raw prompt bytes, no tokenizer required). The precise scorer needs a tokenizer, but sarvam's custom HF code requires `trust_remote_code=True` — a flag the v0.8.0 llm-d-router images does not pass through the UDS tokenizer sidecar. The hash-based approximate scorer bridges that gap.
+
+Each vendor's pods are deployed as a separate Helm release in the same namespace; only the `nodeSelector` and a small set of vendor-specific tuning flags (e.g. Vendor C's `--block-size 128`, `--max-num-seqs 256`, `VLLM_BUILD` pin) vary between releases. All pods carry the same selector labels and register with a single InferencePool maintained by llm-d's router. For the baseline, we use a ClusterIP service over the same set of pods to drive Kubernetes round-robin scheduling. Within each pool, the llm-d and baseline runs use the same pods, vLLM image, and serving flags; only the routing layer differs.
+
+Across every pool we tested — single-vendor (A-only, B-only, C-only) and heterogeneous (A+B, A+B+C) — **llm-d's prefix-cache-aware routing improved throughput and time-to-first-token (TTFT) compared with the k8s round-robin baseline**. The improvement grows with pool size and heterogeneity.
+
+| Pool | Pods | Model | Throughput improvement (llm-d vs k8s) | TTFT reduction |
+| :---- | :---- | :---- | :---- | :---- |
+| Vendor A only | 4 GPUs | granite-4.1-8b | +25–36% | 16× |
+| Vendor A only | 4 GPUs | sarvam-30b | 2× | 22× |
+| Vendor B only | 8 GPUs | granite-4.1-8b | +79% | 21× |
+| Vendor B only | 8 GPUs | sarvam-30b | +83% @ 200 qps (28.6 K vs 15.6 K out tok/s) | 5× |
+| Vendor C only | 8 GPUs | granite-4.1-8b | +34% | 18× |
+| Vendor A + B | 12 GPUs | granite-4.1-8b | +85% (19.4 K vs 10–11 K) | 3.4–5.6× |
+| Vendor A + B | 12 GPUs | sarvam-30b | ~3× @ 200 qps | 2.85–4.54× |
+| **Vendor A + B + C** | **20 GPUs** | **granite-4.1-8b** | **+91% @ 85 qps** | **5.4×** |
+
+**Why heterogeneous pools benefit from llm-d:** k8s round-robin spreads requests evenly regardless of current pod capacity, so lower-capacity or saturated pods can accumulate queue depth and reduce aggregate throughput. llm-d's prefix-cache-aware EPP accounts for saturation and concentrates cache hits on warm pods, so heterogeneous capacity can be used more effectively.
+
+:::note Workload caveat
+
+`shared_prefix_synthetic` is a favourable regime for prefix-cache routing: all requests share the same long system prompt, so cache hit rate on llm-d approaches 100% once a prefix is warm on a pod. Results are strongest for workloads with high prefix reuse; gains can vary with prefix diversity.
+
+:::
+
+### Single-vendor pools — granite-4.1-8b
+
+We start with the per-vendor baselines so the heterogeneous results below have a reference point. All runs use ~7.2K ISL + 1K OSL.
+
+**4× Vendor A GPUs.** llm-d improves TTFT by up to 16× compared to k8s, and output throughput by 25–36%.
+
+![Vendor A granite-4.1-8b: llm-d vs k8s round-robin](/img/blogs/heterogeneous-3vendor/vendor-a-granite.png)
+
+**8× Vendor B GPUs.** llm-d delivers up to 21× better TTFT and +79% throughput vs k8s round-robin on this Vendor B-only granite deployment.
+
+![Vendor B granite-4.1-8b: llm-d vs k8s round-robin](/img/blogs/heterogeneous-3vendor/vendor-b-granite.png)
+
+**8× Vendor C GPUs.** At saturation (rate 25), llm-d delivers +34% throughput and ~18× better TTFT compared with the k8s round-robin baseline.
+
+![Vendor C granite-4.1-8b: llm-d vs k8s round-robin](/img/blogs/heterogeneous-3vendor/vendor-c-granite.png)
+
+### Single-vendor pools — sarvam-30b (multilingual MoE)
+
+**4× Vendor A GPUs.** llm-d delivers 2× the throughput and 22× better TTFT. The k8s round-robin baseline reaches saturation between rate 15-25, while llm-d continues to scale across that range.
+
+![Vendor A sarvam-30b: llm-d vs k8s round-robin](/img/blogs/heterogeneous-3vendor/vendor-a-sarvam.png)
+
+**8× Vendor B GPUs.** k8s throughput plateaus at ~17 K out tok/s (peak at rate 175) and declines to 15.6 K at rate 200, while llm-d keeps scaling to **28.6 K out tok/s at rate 200 — +83% over k8s at the same rate** (or +65% peak-vs-peak). TTFT-wise llm-d is up to 5× faster at lower rates.
+
+![Vendor B sarvam-30b: llm-d vs k8s round-robin](/img/blogs/heterogeneous-3vendor/vendor-b-sarvam.png)
+
+We did not include sarvam-30b results on Vendor C because the model integration and the tested runtime image depended on different vLLM API versions: sarvam's `hotpatch_vllm.py` pins `vllm==0.15.0`, while the tested Vendor C image was based on vLLM 0.16. We plan to revisit with the llm-d community once the version mismatch is resolved.
+
+### Heterogeneous pool results
+
+**Vendor A + B (12 pods, granite-4.1-8b).** While k8s throughput plateaus at 10–11 K tok/s, llm-d goes up to 19.4 K — 85% higher throughput. TTFT-wise llm-d is 3.4–5.6× faster at higher rates.
+
+![Vendor A + B mixed pool granite-4.1-8b: llm-d vs k8s round-robin](/img/blogs/heterogeneous-3vendor/vendor-ab-granite.png)
+
+**Vendor A + B (12 pods, sarvam-30b).** llm-d brings down TTFT by 2.85–4.54× and increases throughput by close to 3× at rate 200. The mixed pool shows the largest gain for this model because round-robin is least aligned with heterogeneous capacity, while llm-d's prefix-aware routing adapts to the observed serving state.
+
+![Vendor A + B mixed pool sarvam-30b: llm-d vs k8s round-robin](/img/blogs/heterogeneous-3vendor/vendor-ab-sarvam.png)
+
+**Vendor A + B + C (20 pods, granite-4.1-8b).** The 20-pod 3-vendor (A + B + C) pool delivers **14.2 K out tok/s peak with llm-d vs 9.6 K with k8s round-robin**. k8s saturates at rate 25 and *declines* to 7.5 K at rate 85 (queue depth dominates) — llm-d delivers **+91% throughput at the same load**. TTFT at rate 85: llm-d 6.8 s, k8s 36.4 s (**5.4× better**).
+
+![3-vendor (A + B + C) granite-4.1-8b: llm-d vs k8s round-robin](/img/blogs/heterogeneous-3vendor/vendor-abc-granite.png)
+
+The rate ladder stopped at 85 QPS because the single-replica EPP became CPU-bound, not because the pool saturated. EPP work scales with `pod_count × QPS`, so the 20-pod 3-vendor (A + B + C) pool hits the CPU ceiling well before a smaller single-vendor pool at the same QPS. Per-pod throughput here sums to ~25 K, so the true pool ceiling is likely higher; EPP horizontal scaling (replicas: 1 → 2) or a higher CPU limit would unblock it further.
+
+## Takeaway
+
+Heterogeneous serving is useful when the routing layer understands more than endpoint count. In this workload, llm-d improves throughput and TTFT by routing with prefix-cache locality and saturation signals, while preserving the platform-specific settings each accelerator pool needs. That is the practical value for mixed fleets: operators can expose diverse capacity through one inference pool and let policy, cache state, and load shape placement decisions.
+
+## What's next
+
+**Cross-accelerator P/D disaggregation.** We plan to take heterogeneous inference to the next level by enabling prefill and decode to run on mixed accelerator types within the same cluster — for example, routing compute-heavy prefill to one vendor's nodes and memory-bandwidth-intensive decode to another's (or vice versa), based on where each phase runs most efficiently. This requires the KV cache transfer library to work across different GPU vendors on each end, an active area of development in the llm-d community.