Add simple options for enabling Gremlin GPU access by ZachNo · Pull Request #147 · gremlin/helm

ZachNo · 2026-07-21T19:13:21Z

To allow customers who install gremlin via helm chart access to the host-level GPU Gremlin, additional config is needed to pull in access to files and drivers for the GPU.

Tested using an EC2 instance with Nvidia drivers, docker, minikube, and NVIDIA Container Toolkit.

philgebhardt

Nice work. I only took a partial first pass here. I've got some more thoughts on the organization of our vendor items, but wanted to publish these first.

philgebhardt · 2026-07-22T21:46:23Z

Do we need unit tests for this document?

philgebhardt

I've proposed an alternative layout for the vendor configuration but I'm not convinced it's complete, as it's related to one open question I have: If I have a cluster with 90% no GPU hardware and 10% nvidia, what happens to the gremlin pods on the non-GPU nodes?

philgebhardt

This is looking good. I think there's one more conceptual change we should make here to align address the upgrade issues (re: --force), and repeatability/testing:

When gremlin.gpu.enabled=false, we keep the existing single daemonset and name:

<release>-gremlin

When gremlin.gpu.enabled=true, we unconditionally produce two daemonsets:

<release>-gremlin-gpu-<vendor>
<release>-gremlin-gpu-none (matching the non-gpu hardware nodes; which may be 0 at runtime)

(on helm upgrade the old <release>-gremlin is cleanly deleted, replaced with the two new ones)

This allows our daemonset system to react to changes in the target cluster's node topology, as it relates to their chosen vendor while keeping the construction of our daemonset YAMLs a pure function. This should also give a test-time win, getting rid of lookup.

philgebhardt · 2026-07-31T18:17:53Z

+On a cluster where only some nodes have GPUs and different nodes may have different GPU vendors, a single GPU DaemonSet cannot run cluster-wide: nodes lacking the vendor's RuntimeClass or device mounts would fail to start the Gremlin pod. So whenever `gremlin.gpu.enabled` is set, the chart inspects the cluster's Nodes at install/upgrade time and renders:
+
+- one GPU DaemonSet per detected vendor (`<release>-gremlin-gpu-<vendor>`), scheduled via node affinity onto that vendor's nodes and carrying that vendor's GPU configuration, and
+- one plain DaemonSet (`<release>-gremlin`) for every remaining non-GPU node.


Specifically for when gremlin.gpu.enabled=true, we could instead name the non-gpu daemonset something different from our existing naming scheme. I think this should remove the need for the --force based upgrade mentioned below and let the cluster upgrade the pods cleaning.

philgebhardt · 2026-07-31T19:09:19Z

+                                   fallback when the cluster's nodes are not visible.
+*/ -}}
+{{- if .Values.gremlin.gpu.enabled -}}
+{{- if include "gremlinGpuNodesVisible" . -}}


Because these *Visible functions depend on node state, the same helm upgrade on the same cluster can yield different results depending on the cluster's node topology at the time of upgrade.

Given a user has declared gremlin.gpu.enabled=true, and selected a vendor. Is there any reason we shouldn't always create a daemonset, regardless of whether nodes exist?

philgebhardt · 2026-07-31T19:37:41Z

+
+This chart does not create RuntimeClass objects; any RuntimeClass named by a vendor block must already exist on the cluster.
+
+_note_: The per-node DaemonSets carry an extra `gremlin.com/gpu` pod-selector label so each only manages its own pods. Because a DaemonSet's selector is immutable, upgrading an existing GPU-less Gremlin install to a GPU one may require deleting the old DaemonSet first (or `helm upgrade --force`).


I think first time upgrading from an existing cluster happens, when gremlin.gpu.enabled is flipped to true, and --force is not supplied: the old daemonset fails to be updated, and the new daemonset is created, which lands two agent pods on the same node until --force is run.

philgebhardt

~~We'll need a chart bump here~~

Copilot

Pull request overview

Adds Helm-chart options to enable host GPU/OpenCL access for the Gremlin agent, supporting mixed-vendor clusters by rendering separate DaemonSets per GPU vendor (plus a non-GPU DaemonSet) and optionally projecting OpenCL ICD registry files.

Changes:

Introduces gremlin.gpu.* values (vendor presets, CDI annotation support, OpenCL ICD projection toggle).
Refactors DaemonSet rendering into a dispatcher (daemonset.yaml) plus a reusable DaemonSet body template (_daemonset.tpl) to support per-vendor variants.
Adds Helm unittest coverage for GPU DaemonSet behavior and OpenCL ICD ConfigMap rendering; documents GPU options in the README.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
`gremlin/values.yaml`	Adds configurable GPU/OpenCL options and vendor presets (NVIDIA/AMD/custom).
`gremlin/templates/daemonset.yaml`	Switches to rendering one DS per vendor + `gpu-none` when GPU is enabled.
`gremlin/templates/_daemonset.tpl`	New shared DaemonSet body template that applies per-vendor runtime/env/volumes and optional ICD projection.
`gremlin/templates/opencl-icd-configmap.yaml`	New ConfigMap template to project vendor OpenCL ICD registry files when needed.
`gremlin/templates/_helpers.tpl`	Adds helpers for ICD projection gating and for AND-ing vendor node affinity into user affinity.
`gremlin/tests/daemonset_gpu_test.yaml`	Adds unit tests validating multi-DS rendering, vendor scoping, CDI annotation, and ICD projection toggles.
`gremlin/tests/opencl_icd_configmap_test.yaml`	Adds unit tests validating conditional rendering and contents of the ICD ConfigMap.
`gremlin/README.md`	Documents the new GPU values and adds a GPU support installation example.
`gremlin/Chart.yaml`	Bumps chart version for the new feature.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+{{- $vendors := default (list) .Values.gremlin.gpu.vendors -}}
+{{- range $vendor := $vendors }}
+{{- include "gremlin.daemonset" (dict
+      "root" $
+      "name" (printf "%s-gpu-%s" $fullname $vendor)
+      "selectorLabels" (dict "gremlin.com/gpu" $vendor)
+      "gpu" (default (dict) (index $.Values.gremlin.gpu $vendor))
+      "affinity" (include "gremlinGpuNodeAffinity" (dict "root" $ "vendors" (list $vendor) "operator" "In"))) }}
+---
+{{ end }}



+### With GPU Support
+
+To let the GPU attack enumerate and target GPUs, enable `gremlin.gpu` and list the vendors your cluster has in `gremlin.gpu.vendors` (both `nvidia` and `amd` by default).


ZachNo added 5 commits July 20, 2026 07:40

Add easy GPU driver setup

70bb987

Add runtimeclass creation

0281768

Fix ICD not mounting

c5e802d

Simplify ICD loader config

a5526c8

Support Nvidia CDI config, clean up docs

ed9b656

ZachNo requested review from a team as code owners July 21, 2026 19:13

rowezuniga reviewed Jul 22, 2026

View reviewed changes

Comment thread gremlin/values.yaml Outdated

Remove orphaned variable

cd043d5

rowezuniga approved these changes Jul 22, 2026

View reviewed changes

ZachNo added 2 commits July 22, 2026 12:18

Add README changes

9048fe0

bump chart version

7be99b5

philgebhardt reviewed Jul 22, 2026

View reviewed changes

Comment thread gremlin/values.yaml

fred-gremlin approved these changes Jul 23, 2026

View reviewed changes

Address comments, add multi-daemon mode for GPU

9bbff57

philgebhardt reviewed Jul 31, 2026

View reviewed changes

Address comments about upgrade path

457a84c

philgebhardt requested a lite review from Copilot August 4, 2026 19:21

philgebhardt approved these changes Aug 4, 2026

View reviewed changes

Copilot started reviewing on behalf of philgebhardt August 4, 2026 19:22 View session

ZachNo merged commit a403271 into master Aug 4, 2026
4 checks passed

Copilot AI reviewed Aug 4, 2026

View reviewed changes

Navigation Menu

Uh oh!

Add simple options for enabling Gremlin GPU access - #147

Add simple options for enabling Gremlin GPU access#147
ZachNo merged 10 commits into
masterfrom
allow-nvidia-driver

ZachNo commented Jul 21, 2026

Uh oh!

Uh oh!

philgebhardt left a comment

Uh oh!

Uh oh!

philgebhardt Jul 22, 2026

Uh oh!

philgebhardt left a comment

Uh oh!

Uh oh!

philgebhardt left a comment

Uh oh!

philgebhardt Jul 31, 2026

Uh oh!

philgebhardt Jul 31, 2026

Uh oh!

philgebhardt Jul 31, 2026

Uh oh!

philgebhardt left a comment •

edited

Loading

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants


		This chart does not create RuntimeClass objects; any RuntimeClass named by a vendor block must already exist on the cluster.

		_note_: The per-node DaemonSets carry an extra `gremlin.com/gpu` pod-selector label so each only manages its own pods. Because a DaemonSet's selector is immutable, upgrading an existing GPU-less Gremlin install to a GPU one may require deleting the old DaemonSet first (or `helm upgrade --force`).


		### With GPU Support

		To let the GPU attack enumerate and target GPUs, enable `gremlin.gpu` and list the vendors your cluster has in `gremlin.gpu.vendors` (both `nvidia` and `amd` by default).

Uh oh!

Conversation

ZachNo commented Jul 21, 2026

Uh oh!

Uh oh!

philgebhardt left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

philgebhardt Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

philgebhardt left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

philgebhardt left a comment

Choose a reason for hiding this comment

Uh oh!

philgebhardt Jul 31, 2026

Choose a reason for hiding this comment

Uh oh!

philgebhardt Jul 31, 2026

Choose a reason for hiding this comment

Uh oh!

philgebhardt Jul 31, 2026

Choose a reason for hiding this comment

Uh oh!

philgebhardt left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

philgebhardt left a comment •

edited

Loading