From 9c4955ba3fbd1f7d22c739b7729215071d63643c Mon Sep 17 00:00:00 2001 From: The Anh Nguyen Date: Fri, 24 Jul 2026 14:21:07 +0700 Subject: [PATCH 1/4] slides(part3): presenter notes on every slide + slidr notes fix Add condensed teleprompter notes (time cue, key numbers, the land line, stage directions) as HTML comments on each Part 3 slide - shown in the presenter view, hidden from the audience. Bump slidr to the notes-escaping fix (slidr-cli/slidr#3) so multi-line notes render instead of blanking the deck. Signed-off-by: The Anh Nguyen --- kcd_vietnam.md | 101 +++++++++++++++++++++++++++++++++++++++++++++++++ slidr | 2 +- 2 files changed, 102 insertions(+), 1 deletion(-) diff --git a/kcd_vietnam.md b/kcd_vietnam.md index 78072a2..b1c69b2 100644 --- a/kcd_vietnam.md +++ b/kcd_vietnam.md @@ -478,6 +478,14 @@ GPU memory automatically swapped to host RAM for idle tasks. Typical scenario: m @subtitle Viettel Cloud, the AI Platform, and the GPU-sharing slice we'll cover + + ::: grid {cols=2} ::: card {tag=cyan} ### The Anh Nguyen @@ -508,6 +516,13 @@ Cloud platform of Viettel Group, Vietnam's largest telco. @subtitle 31 platforms in the world have it. We are one. + + ![Certified Kubernetes - AI Platform](assets/kcd_vietnam/certified-k8s-ai-platform.png) ![badge](assets/kcd_vietnam/ai-conformance-badge.png) Our **AI Platform** - **first & only in Vietnam**, #**20** worldwide. @@ -518,6 +533,16 @@ Cloud platform of Viettel Group, Vietnam's largest telco. @subtitle HAMi in our clusters + + - {icon:server cls=accent-primary} **HAMi v2.9** on **both clusters**: the H200 pool, and the mixed-GPU one - {icon:users cls=accent-primary} Our own engineers use it every day - {icon:lock cls=accent-primary} The memory limit is real: pod sees **2 GB, not 141 GB** @@ -544,6 +569,17 @@ Numbers in this talk: mostly the **H200 pool**, a few from **L40**. @subtitle Asking for a GPU is not the same as using it + + Two real jobs. Each one held **a whole H200, 141 GB**: ```seaborn @@ -595,6 +631,14 @@ ax.set_xticks([]) @subtitle Fix the SLA, scale replicas, fill the empty GPU + + ```seaborn import matplotlib.pyplot as plt @@ -647,6 +691,16 @@ One pod uses only **18% of the card**. Pack **10 replicas** on it and the GPU hi @subtitle One slice guaranteed, on purpose - and what it costs + + **Workload:** an AI Notebook training YOLO11 - ~17k images, batch 64 → **39 GB** used. Batch is sized for **accuracy**, not to fill the card - so the spare **~100 GB is genuinely free**, not just idle. We fence **~30%** and cap the notebook to **70% of the cores**. ```seaborn @@ -685,6 +739,15 @@ ax.set_ylim(-20, 112) @subtitle One notebook fixed all day - inference flexes by the hour + + ```seaborn import matplotlib.pyplot as plt @@ -729,6 +792,14 @@ ax.set_ylim(-0.6, 3.2) @subtitle Does sharing make training jittery? We measured it + + The card is now *truly* shared - the notebook trains while **bursty inference** flexes beside it. Does that shake the training? We measured epoch-time **CV** - standard deviation ÷ mean, lower is steadier. ```seaborn @@ -769,6 +840,19 @@ ax.set_ylim(-4, 24.5) @subtitle HAMi fills the *empty* part of a card - a full one has nothing to give + + ::: grid {cols=2} ::: card {tag=green} ### {icon:check cls=accent-primary} It helps when the card is empty @@ -801,6 +885,15 @@ ax.set_ylim(-4, 24.5) @subtitle The hard part is not installing HAMi + + | What surprised us | What we did | |---|---| | **Kyverno** blocks HAMi's device plugin - needs **privileged + hostPath** | Exclude `hami-system` from the **pod-security** policies | @@ -816,6 +909,14 @@ ax.set_ylim(-4, 24.5) @subtitle Useful to know before you debug for two days + + HAMi sits in front of every CUDA call. **Go around CUDA, and HAMi cannot see you.** ```seaborn diff --git a/slidr b/slidr index 727a611..7f42054 160000 --- a/slidr +++ b/slidr @@ -1 +1 @@ -Subproject commit 727a61109785a46c5a3874849c73649a116decc2 +Subproject commit 7f4205426de4b1bfb406d7b0f433f6ab514fd851 From 9d45a261371242ea57b6271193ac8dc4db2be26a Mon Sep 17 00:00:00 2001 From: The Anh Nguyen Date: Fri, 24 Jul 2026 17:47:15 +0700 Subject: [PATCH 2/4] slides(part3): sharpen What It Really Took table Rename table headers to Obstacle/Solution, reword the obstacle column in precise technical terms (privileged access vs Kyverno, nvidia.com/gpu ambiguity, resource-pool overlap, 2.9 metrics refactor), and turn the closing line into a quote-styled pull-quote. Colleague review (Hung). Signed-off-by: The Anh Nguyen --- kcd_vietnam.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/kcd_vietnam.md b/kcd_vietnam.md index b1c69b2..8086304 100644 --- a/kcd_vietnam.md +++ b/kcd_vietnam.md @@ -894,14 +894,14 @@ install = a Helm chart. NOT the hard part. LAND (SLOW): this is an operations project, not a one-line install --> -| What surprised us | What we did | +| Obstacle | Solution | |---|---| -| **Kyverno** blocks HAMi's device plugin - needs **privileged + hostPath** | Exclude `hami-system` from the **pod-security** policies | -| GPU Operator **also registers** `nvidia.com/gpu` - clashes with HAMi | Operator device plugin off, CDI off | -| Fractional, full-GPU, Slurm (Slinky) **all want the same nodes** | Own pool, own scheduler, and a taint | -| HAMi 2.9 **renamed all metrics** - dashboards empty | Use `hami_*`. We fixed the docs upstream | +| HAMi's **privileged access** conflicts with **Kyverno** policies | Exclude `hami-system` from the **pod-security** policies | +| **`nvidia.com/gpu`** is ambiguous - claimed by both the **NVIDIA device plugin** and HAMi | Operator device plugin off, CDI off | +| **Resource-pool overlap** - fractional, full-GPU, and **Slurm** (Slinky) | Own pool, own scheduler, and a taint | +| HAMi 2.9 **metrics refactor** - Grafana dashboards mismatch, go **empty** | Use `hami_*`. We fixed the docs upstream | -**Remember this: it is an operations project, not a one-line install.** +> **It is an operations project, not a one-line install.** --- From 37bcdec968e519214e1a5e9c576eb5ed4610516c Mon Sep 17 00:00:00 2001 From: The Anh Nguyen Date: Fri, 24 Jul 2026 17:52:45 +0700 Subject: [PATCH 3/4] chore(slidr): rebase submodule onto master (--dist + notes-escape fix) slidr-cli master had advanced to the --dist flag while the notes-escape fix branched from an older commit. Rebased the fix onto current master so the submodule carries both, making the bump a clean forward move (slidr-cli/slidr#3). Signed-off-by: The Anh Nguyen --- slidr | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/slidr b/slidr index 7f42054..e13e0e9 160000 --- a/slidr +++ b/slidr @@ -1 +1 @@ -Subproject commit 7f4205426de4b1bfb406d7b0f433f6ab514fd851 +Subproject commit e13e0e99976bfd508464ed5a3388500d0726d226 From 52fbd89b7c69a7c7c4059f317a05eda1e57cc94f Mon Sep 17 00:00:00 2001 From: The Anh Nguyen Date: Fri, 24 Jul 2026 17:55:27 +0700 Subject: [PATCH 4/4] slides(part3): align What It Really Took note with slide wording Update the presenter note to say 'NVIDIA device plugin' for the nvidia.com/gpu clash, matching the reworded Obstacle column. Signed-off-by: The Anh Nguyen --- kcd_vietnam.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kcd_vietnam.md b/kcd_vietnam.md index 8086304..cf58f84 100644 --- a/kcd_vietnam.md +++ b/kcd_vietnam.md @@ -888,7 +888,7 @@ cut-for-time: drop LLM/training detail, NEVER the gpumem/gpucores table