For some reason, the ubuntu24.04 daemonset is selecting a 22.04 binary driver image (reopens #722)

[doctorpangloss]

Reopens #722

Describe the bug

Name:                 nvidia-driver-daemonset-6.8.0-59-generic-ubuntu24.04-lzpsx
Namespace:            gpu-operator
...
helm.sh/chart=gpu-operator-v25.3.0
...
Controlled By:  DaemonSet/nvidia-driver-daemonset-6.8.0-59-generic-ubuntu24.04
...
Containers:
  nvidia-driver-ctr:
    Container ID:
    Image:         nvcr.io/nvidia/driver:570-6.8.0-59-generic-ubuntu22.04

Well, that's wrong.

To Reproduce
Install Ubuntu 24.04 on your cluster. Observe the wrong driver is selected..

Expected behavior
You have the 24.04 images. Why would the daemonset be deployed with the wrong image? A bug is why.

Environment (please provide the following information):

• GPU Operator Version: 25.3.0
• OS: Ubuntu24.04
• Kernel Version: 6.8.0-59-generic
• Container Runtime Version: containerd 1.7.22
• Kubernetes Distro and Version: k0s 1.30

[doctorpangloss]

kubectl logs pod/nvidia-driver-daemonset-q42pc -n gpu-operator --follow
DRIVER_ARCH is x86_64
Creating directory NVIDIA-Linux-x86_64-570.124.06
Verifying archive integrity... OK
Uncompressing NVIDIA Accelerated Graphics Driver for Linux-x86_64 570.124.06.......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................

WARNING: Unable to determine the default X library path. The path /tmp/null/lib will be used, but this path was not detected in the ldconfig(8) cache, and no directory exists at this path, so it is likely that libraries installed there will not be found by the loader.


WARNING: You specified the '--no-kernel-modules' command line option, nvidia-installer will not install any kernel modules as part of this driver installation, and it will not remove existing NVIDIA kernel modules not part of an earlier NVIDIA driver installation.  Please ensure that NVIDIA kernel modules matching this driver version are installed separately.


WARNING: This NVIDIA driver package includes Vulkan components, but no Vulkan ICD loader was detected on this system. The NVIDIA Vulkan ICD will not function without the loader. Most distributions package the Vulkan loader; try installing the "vulkan-loader", "vulkan-icd-loader", or "libvulkan1" package.


========== NVIDIA Software Installer ==========

Starting installation of NVIDIA driver version 570.124.06 for Linux kernel version 6.8.0-60-generic

Stopping NVIDIA persistence daemon...
Unloading NVIDIA driver kernel modules...
Unmounting NVIDIA driver rootfs...
Checking NVIDIA driver packages...
Updating the package cache...
Resolving Linux kernel version...
Could not resolve Linux kernel version
Stopping NVIDIA persistence daemon...
Unloading NVIDIA driver kernel modules...
Unmounting NVIDIA driver rootfs...

Issue persists, what is going on? Why is this giving me a 22 image on 24?

[peterezzo]

i have this same issue, the ds pod shows 24.04 but the driver-ctr container is still 22.04

Events:
  Type     Reason     Age                  From               Message
  ----     ------     ----                 ----               -------
  Normal   Pulled     25m                  kubelet            Container image "nvcr.io/nvidia/cloud-native/k8s-driver-manager:v0.8.0" already present on machine
  Normal   Created    25m                  kubelet            Created container: k8s-driver-manager
  Normal   Started    25m                  kubelet            Started container k8s-driver-manager
  Normal   Scheduled  24m                  default-scheduler  Successfully assigned gpu-operator/nvidia-driver-daemonset-6.8.0-60-generic-ubuntu24.04-6h794 to gpu5
  Normal   Pulling    23m (x4 over 25m)    kubelet            Pulling image "nvcr.io/nvidia/driver:570-6.8.0-60-generic-ubuntu22.04"
  Warning  Failed     23m (x4 over 25m)    kubelet            Failed to pull image "nvcr.io/nvidia/driver:570-6.8.0-60-generic-ubuntu22.04": rpc error: code = NotFound desc = failed to pull and unpack image "nvcr.io/nvidia/driver:570-6.8.0-60-generic-ubuntu22.04": failed to resolve reference "nvcr.io/nvidia/driver:570-6.8.0-60-generic-ubuntu22.04": nvcr.io/nvidia/driver:570-6.8.0-60-generic-ubuntu22.04: not found
  Warning  Failed     23m (x4 over 25m)    kubelet            Error: ErrImagePull
  Warning  Failed     23m (x5 over 24m)    kubelet            Error: ImagePullBackOff
  Normal   BackOff    16s (x105 over 24m)  kubelet            Back-off pulling image "nvcr.io/nvidia/driver:570-6.8.0-60-generic-ubuntu22.04"

[github-actions]

This issue is stale because it has been open 90 days with no activity. This issue will be closed in 30 days unless new comments are made or the stale label is removed. To skip these checks, apply the "lifecycle/frozen" label.

[rajathagasthya]

Can you try the latest GPU Operator and confirm if you're still hitting these issues?

[NurullahMorshed]

I'm still seeing this issue in 25.10.1 when trying to run on ubuntu24.04
with the default values except for

operator:
  defaultRuntime: crio

My workaround for the moment was to retag the 24.04 image as 22.04 and put it on a custom repo

[NurullahMorshed]

gpu-operator/cmd/gpuop-cfg/validate/clusterpolicy/images.go

Line 38 in 35f404e

path += "-ubuntu22.04"

I suspect this line is the issue

[rahulait]

@NurullahMorshed are you sure you are using gpu-operator version v25.10.1? The values you mentioned were in 24.9.2 and earlier versions of gpu-operator. They were removed going forward and v25.10.1 doesn't have them.

Something similar (#1917) was fixed before and is in release v25.10.1 . We are not aware of your setup, so kindly do share details of your setup if you are still seeing this issue with v25.10.1 version of gpu-operator. You can provide us the must-gather output from your cluster so that we can analyze how your cluster is setup and see pod logs to identify if there is something wrong going on there.

[NurullahMorshed]

apiVersion: v2
appVersion: v25.10.1
dependencies:
- condition: nfd.enabled
  name: node-feature-discovery
  repository: https://kubernetes-sigs.github.io/node-feature-discovery/charts
  version: v0.18.2
description: NVIDIA GPU Operator creates/configures/manages GPUs atop Kubernetes
home: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/overview.html
icon: https://assets.nvidiagrid.net/ngc/logos/GPUoperator.png
keywords:
- gpu
- cuda
- compute
- operator
- deep learning
- monitoring
- tesla
kubeVersion: '>= 1.16.0-0'
name: gpu-operator
sources:
- https://github.com/NVIDIA/gpu-operator
version: v25.10.1

That's my chart.yaml.

We were using an older version in the past. I did not realize that value is now dead, so I can remove that as well. Based on that we were just running the default values for the gpu operator. Now we are running it with

driver:
  repository: <private repo>
  imagePullSecrets:
    - <key>

so we can trick it into downloading the 24.04 image by retagging it as the 22.04 image.

[rahulait]

Can you run this script and attach the generated logs in a zipped file to this ticket? If you don't want to attach here, you can also send them to operator_feedback@nvidia.com. Just curious what your cluster is and why its behaving incorrectly there. You can run it from any node from where you have kubectl access to the cluster.

curl -fsSL https://raw.githubusercontent.com/NVIDIA/gpu-operator/refs/heads/main/hack/must-gather.sh | bash