Can Multiple Containers Share the Same GPU?

[kristian-lemurian]

Hey! I’m running a single-node Kubernetes cluster with 8 MI300X GPUs and using the AMD GPU Device Plugin. Right now, when I request amd.com/gpu: 1 in my pod spec, I’m limited to just 8 containers – one per GPU.

I’d love to be able to launch additional containers that share the same GPU devices. Is there a way to enable GPU sharing or over-allocation with the plugin?

Thanks in advance!

[MartinLoeper]

I wondered the same thing as I only have one gpu in my cluster and it is used by a devcontainer, so no pods left for testing the actual app

[bhatnitish]

Can you please elaborate on your usecase? Would the containers sharing the GPU simultaneously use the device? Or will this be exclusive access?

[ivan-sirosh]

@bhatnitish Would be nice to run n+1 containers on the same gpu in the same time. My system has only one gpu and it means that I limited to run only one container right now. Intel and Nvidia already have such slicing, you can just put in resources intel.com/gpu: 0.5 and it will allow you to use your one gpu by 1/2 scale.

[ivan-sirosh]

there are similar requests:
#111
#69

[AdamNowotny]

A possible workaround is NOT to use resource limits for "amd.com/gpu" (as time slicing is not implemented in the plugin), but instead mount device volume for direct GPU access:

volumes:
  - name: kfd
    hostPath:
      path: /dev/kfd
  - name: dri
    hostPath:
      path: /dev/dri
containers:
  volumeMounts:
    - name: kfd
      mountPath: /dev/kfd
    - name: dri
      mountPath: /dev/dri
  securityContext:
    seccompProfile:
      type: Unconfined

Make sure you use ROCm image for AMD GPU detection.

Also applies to #111 and #69

[ivan-sirosh]

A possible workaround is NOT to use resource limits for "amd.com/gpu" (as time slicing is not implemented in the plugin), but instead mount device volume for direct GPU access

This solution requires you to force use a root container, otherwise GPU devices can not be mounted!

I would rather wait the developers answer (if it would come sometime 😄)

meanwhile a full working solution, tested on k3s:

---
apiVersion: v1
kind: Namespace
metadata:
  name: gpu-direct-test
  labels:
    pod-security.kubernetes.io/enforce: privileged
    pod-security.kubernetes.io/audit: privileged
    pod-security.kubernetes.io/warn: privileged
---
apiVersion: v1
kind: Pod
metadata:
  name: gpu-direct-test
  namespace: gpu-direct-test
spec:
  securityContext:
    runAsUser: 1000
    runAsGroup: 1000
    supplementalGroups: [ 44, 105 ] # video
    seccompProfile:
      type: Unconfined
  containers:
    - name: rocm-test
      image: rocm/dev-ubuntu-22.04:5.7
      securityContext:
        privileged: true
      volumeMounts:
        - name: dri-card
          mountPath: /dev/dri/card0
          readOnly: false
        - name: dri-render
          mountPath: /dev/dri/renderD128
          readOnly: false
        - name: kfd
          mountPath: /dev/kfd
          readOnly: false
      command: [ "sh", "-c", "rocminfo || sleep 3600" ]
    - name: busybox
      image: busybox
      securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop: [ "ALL" ]
      volumeMounts:
        - name: dri-card
          mountPath: /dev/dri/card0
          readOnly: false
        - name: dri-render
          mountPath: /dev/dri/renderD128
          readOnly: false
        - name: kfd
          mountPath: /dev/kfd
          readOnly: false
      command: [ "sh", "-c", "ls -l /dev/dri || sleep 3600" ]
  volumes:
    - name: dri-card
      hostPath:
        path: /dev/dri/card0
        type: CharDevice
    - name: dri-render
      hostPath:
        path: /dev/dri/renderD128
        type: CharDevice
    - name: kfd
      hostPath:
        path: /dev/kfd
        type: CharDevice

[AdamNowotny]

@ivan-sirosh it's not required to use privileged container, I found it's enough to just add container security context with seccompProfile (updated my comment).
But you're absolutely right, it's just a hack that's applicable in self-hosting scenarios but not a recommended solution in production setups.