multus-cni-network fails to assign IP: `error waiting for pod: Unauthorized`

[surajssd]

What happened:

I have a Kubernetes cluster deployed on AKS. I am running network-operator. After an hour of the deployment the multus plugin fails to assign IP to the pods. The event I get is like this:

0s          Warning   FailedCreatePodSandBox   pod/ibtest-57599dd56f-45ksq
Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox 
"26ed6556afa0ebd0407fa6db67b308e56524733d73304a2611eacdc1408a27aa": plugin type="multus" name="multus-cni-network"
failed (add): Multus: [default/ibtest-57599dd56f-45ksq/811508e0-b5d0-4126-9b00-16def7acfd6f]: error waiting for pod: Unauthorized

Here is full list of events:

$ kubectl get events
LAST SEEN   TYPE      REASON                   OBJECT                                  MESSAGE
6s          Normal    Scheduled                pod/ibtest-57599dd56f-45ksq             Successfully assigned default/ibtest-57599dd56f-45ksq to aks-gpunodes-17372139-vmss000000
6s          Warning   FailedCreatePodSandBox   pod/ibtest-57599dd56f-45ksq             Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "2e21cf024a4aab49665572445258e58c8ba1f53ed6e55dbae18816c339872b1d": plugin type="multus" name="multus-cni-network" failed (add): Multus: [default/ibtest-57599dd56f-45ksq/811508e0-b5d0-4126-9b00-16def7acfd6f]: error waiting for pod: Unauthorized
6s          Normal    Scheduled                pod/ibtest-57599dd56f-k2xmt             Successfully assigned default/ibtest-57599dd56f-k2xmt to aks-gpunodes-17372139-vmss000001
6s          Warning   FailedCreatePodSandBox   pod/ibtest-57599dd56f-k2xmt             Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "449727243cb4f36d9ab9e55a5cd7df86675b1346facc3b1e48ce3cf50f2ed7c5": plugin type="multus" name="multus-cni-network" failed (add): Multus: [default/ibtest-57599dd56f-k2xmt/43887458-6c67-4217-8e47-e08b09b114ce]: error waiting for pod: Unauthorized
6s          Normal    SuccessfulCreate         replicaset/ibtest-57599dd56f            Created pod: ibtest-57599dd56f-k2xmt
6s          Normal    SuccessfulCreate         replicaset/ibtest-57599dd56f            Created pod: ibtest-57599dd56f-45ksq
6s          Normal    ScalingReplicaSet        deployment/ibtest                       Scaled up replica set ibtest-57599dd56f to 2
0s          Warning   FailedCreatePodSandBox   pod/ibtest-57599dd56f-k2xmt             Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "1d4aec3c086b534f216d6ca1e2d0c84d16ccfdd7283fadb50c96f3cc544014ae": plugin type="multus" name="multus-cni-network" failed (add): Multus: [default/ibtest-57599dd56f-k2xmt/43887458-6c67-4217-8e47-e08b09b114ce]: error waiting for pod: Unauthorized
0s          Warning   FailedCreatePodSandBox   pod/ibtest-57599dd56f-45ksq             Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "26ed6556afa0ebd0407fa6db67b308e56524733d73304a2611eacdc1408a27aa": plugin type="multus" name="multus-cni-network" failed (add): Multus: [default/ibtest-57599dd56f-45ksq/811508e0-b5d0-4126-9b00-16def7acfd6f]: error waiting for pod: Unauthorized

What you expected to happen:

I expected the pod to be started with the secondary IP assigned by the multus plugin.

How to reproduce it (as minimally and precisely as possible):

Steps here: https://gist.github.com/surajssd/a0596ca7785228f025be5c3ac177219f

But here are the relevant steps:

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update

helm upgrade -i \
    --wait \
    --create-namespace \
    -n network-operator \
    network-operator \
    nvidia/network-operator \
    --set nfd.deployNodeFeatureRules=false

Anything else we need to know?:

I am using the machine type: Standard_HB120-16rs_v3 on Azure.

Logs:

• NicClusterPolicy CR spec and state:

---
apiVersion: nfd.k8s-sigs.io/v1alpha1
kind: NodeFeatureRule
metadata:
  name: nfd-network-rule
spec:
   rules:
   - name: "nfd-network-rule"
     labels:
        "feature.node.kubernetes.io/pci-15b3.present": "true"
     matchFeatures:
        - feature: pci.device
          matchExpressions:
            device: {op: In, value: ["101c", "101e"]}
---
# Try to match the versions from: https://github.com/Mellanox/network-operator/blob/master/example/crs/mellanox.com_v1alpha1_nicclusterpolicy_cr-full.yaml
apiVersion: mellanox.com/v1alpha1
kind: NicClusterPolicy
metadata:
  name: nic-cluster-policy
spec:
  nvIpam:
    enableWebhook: false
    repository: ghcr.io/mellanox
    image: nvidia-k8s-ipam
    # Latest tag: https://github.com/mellanox/nvidia-k8s-ipam/pkgs/container/nvidia-k8s-ipam
    version: v0.2.0

  ofedDriver:
    forcePrecompiled: false
    repository: nvcr.io/nvidia/mellanox
    image: doca-driver
    # Latest tag: https://catalog.ngc.nvidia.com/orgs/nvidia/teams/mellanox/containers/doca-driver/tags
    # When this is deployed a suffix is added in the format "-<os><os version>-<cpu arch>" for e.g.: "-ubuntu22.04-amd64".
    version: 24.10-0.7.0.0-0

    upgradePolicy:
      autoUpgrade: true
      drain:
        deleteEmptyDir: true
        enable: true
        force: true
        timeoutSeconds: 300
      maxParallelUpgrades: 1
    startupProbe:
      initialDelaySeconds: 10
      periodSeconds: 20
    livenessProbe:
      initialDelaySeconds: 30
      periodSeconds: 30
    readinessProbe:
      initialDelaySeconds: 10
      periodSeconds: 30

  rdmaSharedDevicePlugin:
    repository: ghcr.io/mellanox
    image: k8s-rdma-shared-dev-plugin
    # Latest tag: https://github.com/mellanox/k8s-rdma-shared-dev-plugin/pkgs/container/k8s-rdma-shared-dev-plugin
    version: v1.5.2
    useCdi: true

    # The config below directly propagates to k8s-rdma-shared-device-plugin configuration.
    # Replace 'devices' with your (RDMA capable) netdevice name.
    config: |
      {
        "configList": [
          {
            "resourceName": "rdma_shared_device_a",
            "rdmaHcaMax": 63,
            "selectors": {
              "vendors": ["15b3"]
            }
          }
        ]
      }

  secondaryNetwork:
    cniPlugins:
      repository: ghcr.io/k8snetworkplumbingwg
      image: plugins
      # Latest tag: https://github.com/k8snetworkplumbingwg/plugins/pkgs/container/plugins
      version: v1.5.0

    multus:
      repository: ghcr.io/k8snetworkplumbingwg
      image: multus-cni
      # Latest tag: https://github.com/k8snetworkplumbingwg/plugins/pkgs/container/multus-cni
      version: v4.1.0

    ipamPlugin:
      repository: ghcr.io/k8snetworkplumbingwg
      image: whereabouts
      # Latest tag: https://github.com/k8snetworkplumbingwg/whereabouts/pkgs/container/whereabouts
      version: v0.7.0

    ipoib:
      repository: ghcr.io/mellanox
      image: ipoib-cni
      # Latest tag: https://github.com/mellanox/ipoib-cni/pkgs/container/ipoib-cni
      version: v1.2.1

  # This is needed so that you don't schedule pods on the non-infiniband machines.
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        # This node label is added by NFD.
        - key: feature.node.kubernetes.io/pci-15b3.present
          operator: In
          values:
          - "true"
---
apiVersion: mellanox.com/v1alpha1
kind: IPoIBNetwork
metadata:
  name: aks-infiniband
spec:
  networkNamespace: "default"
  # This is an alt interface name for the IPoIB interface. As per this blog it
  # seems to be common across the same VM SKU:
  # https://techcommunity.microsoft.com/blog/azurehighperformancecomputingblog/running-tightly-coupled-hpcai-workloads-with-infiniband-using-nvidia-network-ope/4117209
  # It has been consistent on the machine types: Standard_HB120rs_v3 and Standard_ND96asr_v4.
  # TODO: Figure out how to get this generic name?
  master: "ibP257p0s0"
  ipam: |
    {
      "type": "whereabouts",
      "datastore": "kubernetes",
      "kubernetes": {
        "kubeconfig": "/etc/cni/net.d/whereabouts.d/whereabouts.kubeconfig"
      },
      "range": "192.168.0.0/16",
      "exclude": [
       "192.168.0.0/32",
       "192.168.255.255/32"
      ],
      "log_file" : "/var/log/whereabouts.log",
      "log_level" : "info",
      "gateway": "192.168.0.1"
    }

• Output of: kubectl get -n network-operator all:

✗ kubectl get -n network-operator all
NAME                                                                  READY   STATUS    RESTARTS      AGE
pod/cni-plugins-ds-j5v8k                                              1/1     Running   0             101m
pod/cni-plugins-ds-x9h28                                              1/1     Running   0             101m
pod/kube-ipoib-cni-ds-p4lfq                                           1/1     Running   0             101m
pod/kube-ipoib-cni-ds-pwwmh                                           1/1     Running   0             101m
pod/kube-multus-ds-bt6h6                                              1/1     Running   0             101m
pod/kube-multus-ds-ss8dn                                              1/1     Running   0             101m
pod/mofed-ubuntu22.04-6fd94b4c6b-ds-8d2k6                             1/1     Running   5 (94m ago)   101m
pod/mofed-ubuntu22.04-6fd94b4c6b-ds-s5sp9                             1/1     Running   4 (95m ago)   101m
pod/network-operator-5ff6ff9559-qvqg8                                 1/1     Running   0             101m
pod/network-operator-node-feature-discovery-gc-6d48649f49-wqwmd       1/1     Running   0             101m
pod/network-operator-node-feature-discovery-master-57648d678f-cmw7v   1/1     Running   0             101m
pod/network-operator-node-feature-discovery-worker-9n887              1/1     Running   0             101m
pod/network-operator-node-feature-discovery-worker-qtlpm              1/1     Running   0             101m
pod/network-operator-node-feature-discovery-worker-rtpmp              1/1     Running   0             101m
pod/nv-ipam-controller-67556c846b-c5ljf                               1/1     Running   0             101m
pod/nv-ipam-controller-67556c846b-zb2nw                               1/1     Running   0             101m
pod/nv-ipam-node-4bqhh                                                1/1     Running   0             101m
pod/nv-ipam-node-t4l47                                                1/1     Running   0             101m
pod/rdma-shared-dp-ds-kvfhb                                           1/1     Running   0             92m
pod/rdma-shared-dp-ds-l442j                                           1/1     Running   0             93m
pod/whereabouts-x5nql                                                 1/1     Running   0             101m
pod/whereabouts-xwqzt                                                 1/1     Running   0             101m

NAME                                                            DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                                                                                                                                                                                                                            AGE
daemonset.apps/cni-plugins-ds                                   2         2         2       2            2           <none>                                                                                                                                                                                                                                   101m
daemonset.apps/kube-ipoib-cni-ds                                2         2         2       2            2           <none>                                                                                                                                                                                                                                   101m
daemonset.apps/kube-multus-ds                                   2         2         2       2            2           <none>                                                                                                                                                                                                                                   101m
daemonset.apps/mofed-ubuntu22.04-6fd94b4c6b-ds                  2         2         2       2            2           feature.node.kubernetes.io/kernel-version.full=5.15.0-1079-azure,feature.node.kubernetes.io/pci-15b3.present=true,feature.node.kubernetes.io/system-os_release.ID=ubuntu,feature.node.kubernetes.io/system-os_release.VERSION_ID=22.04   101m
daemonset.apps/network-operator-node-feature-discovery-worker   3         3         3       3            3           <none>                                                                                                                                                                                                                                   101m
daemonset.apps/nv-ipam-node                                     2         2         2       2            2           <none>                                                                                                                                                                                                                                   101m
daemonset.apps/rdma-shared-dp-ds                                2         2         2       2            2           feature.node.kubernetes.io/pci-15b3.present=true,network.nvidia.com/operator.mofed.wait=false                                                                                                                                            101m
daemonset.apps/whereabouts                                      2         2         2       2            2           <none>                                                                                                                                                                                                                                   101m

NAME                                                             READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/network-operator                                 1/1     1            1           101m
deployment.apps/network-operator-node-feature-discovery-gc       1/1     1            1           101m
deployment.apps/network-operator-node-feature-discovery-master   1/1     1            1           101m
deployment.apps/nv-ipam-controller                               2/2     2            2           101m

NAME                                                                        DESIRED   CURRENT   READY   AGE
replicaset.apps/network-operator-5ff6ff9559                                 1         1         1       101m
replicaset.apps/network-operator-node-feature-discovery-gc-6d48649f49       1         1         1       101m
replicaset.apps/network-operator-node-feature-discovery-master-57648d678f   1         1         1       101m
replicaset.apps/nv-ipam-controller-67556c846b                               2         2         2       101m

• Network Operator version:

✗ helm -n network-operator ls
NAME                    NAMESPACE               REVISION        UPDATED                                 STATUS          CHART                   APP VERSION
network-operator        network-operator        1               2025-03-05 16:37:15.858515 -0800 PST    deployed        network-operator-25.1.0 v25.1.0

• Logs of Network Operator controller:

✗ kubectl logs -n network-operator network-operator-5ff6ff9559-qvqg8

full log here: https://gist.github.com/surajssd/3c89a7aed0a57ca40d7b371bf84940a9

Environment:

• Kubernetes version (use kubectl version):

✗ kubectl version
Client Version: v1.32.2
Kustomize Version: v5.5.0
Server Version: v1.30.9
WARNING: version difference between client (1.32) and server (1.30) exceeds the supported minor version skew of +/-1

• Hardware configuration:
  • Network adapter model and firmware version: mlx5_core driver version: 24.10-0.7.0

Logs from the mofed pods: https://gist.github.com/surajssd/1521f546efe4612c2cd9b1280d1dc654

• OS (e.g: cat /etc/os-release):

# cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.5 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.5 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

• Kernel (e.g. uname -a):

# uname -a
Linux aks-gpunodes-17372139-vmss000000 5.15.0-1079-azure #88-Ubuntu SMP Thu Jan 16 19:18:54 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

[surajssd]

Is there a possibility that the multus thick plugin becomes default? I think that the thick plugin can solve the problem: https://github.com/k8snetworkplumbingwg/multus-cni/blob/master/docs/thick-plugin.md

I see that there is a PR to fix this: #975 but not sure what's the status of it.

[thuffmlx]

please provide the network-operator values.yaml file.
are you running GPU-operator as well?
If you are the network-operator node-feature discovery should be disabled.

[thuffmlx]

please attach the output of:
"kubectl describe node {worker node}" for each worker node
"kubectl logs -n network-operator kube-multus-ds-bt6h6"
"kubectl logs -n network-operator kube-multus-ds-ss8dn"

[thuffmlx]

I am surprised to see both nv-ipam and whereabouts running
pod/nv-ipam-controller-67556c846b-c5ljf 1/1 Running 0 101m
pod/nv-ipam-controller-67556c846b-zb2nw 1/1 Running 0 101m
pod/nv-ipam-node-4bqhh 1/1 Running 0 101m
pod/nv-ipam-node-t4l47 1/1 Running 0 101m
pod/whereabouts-x5nql 1/1 Running 0 101m
pod/whereabouts-xwqzt 1/1 Running 0 101m

[chewong]

I am suspecting it's caused by --skip-config-watch being enabled in https://github.com/Mellanox/network-operator/blob/master/manifests/state-multus-cni/0050-multus-ds.yml#L77, which caused multus not to rotate the kubeconfig in https://github.com/k8snetworkplumbingwg/multus-cni/blob/master/cmd/thin_entrypoint/main.go#L634. Could we remove that flag?

[surajssd]

@thuffmlx here are the details that you asked for:

are you running GPU-operator as well?

No, this machine doesn't have any GPUs, just a Mellanox NIC.

Reproduce the setup instructions: https://gist.github.com/surajssd/2a848ed8f8f99b5a45c5c48f9933adf4

Network Operator Configs

The chart is installed like this:

helm upgrade -i \
    --wait \
    --create-namespace \
    -n network-operator \
    --values values.yaml \
    network-operator \
    nvidia/network-operator \
    --version v25.1.0

values.yaml file contents:

~ helm get values -n network-operator network-operator
USER-SUPPLIED VALUES:
nfd:
  deployNodeFeatureRules: false

Additional configs deployed after the chart's installation:

---
apiVersion: nfd.k8s-sigs.io/v1alpha1
kind: NodeFeatureRule
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"nfd.k8s-sigs.io/v1alpha1","kind":"NodeFeatureRule","metadata":{"annotations":{},"name":"nfd-network-rule"},"spec":{"rules":[{"labels":{"feature.node.kubernetes.io/pci-15b3.present":"true"},"matchFeatures":[{"feature":"pci.device","matchExpressions":{"device":{"op":"In","value":["101c","101e"]}}}],"name":"nfd-network-rule"}]}}
  creationTimestamp: "2025-05-01T17:17:38Z"
  generation: 1
  name: nfd-network-rule
  resourceVersion: "5223"
  uid: 9a4ad51d-363a-46f5-9f3e-f85d7b9d35a5
spec:
  rules:
  - labels:
      feature.node.kubernetes.io/pci-15b3.present: "true"
    matchFeatures:
    - feature: pci.device
      matchExpressions:
        device:
          op: In
          value:
          - 101c
          - 101e
    name: nfd-network-rule
---
apiVersion: mellanox.com/v1alpha1
kind: NicClusterPolicy
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"mellanox.com/v1alpha1","kind":"NicClusterPolicy","metadata":{"annotations":{},"name":"nic-cluster-policy"},"spec":{"docaTelemetryService":{"image":"doca_telemetry","repository":"nvcr.io/nvidia/doca","version":"1.16.5-doca2.6.0-host"},"nodeAffinity":{"requiredDuringSchedulingIgnoredDuringExecution":{"nodeSelectorTerms":[{"matchExpressions":[{"key":"feature.node.kubernetes.io/pci-15b3.present","operator":"In","values":["true"]}]}]}},"ofedDriver":{"forcePrecompiled":false,"image":"doca-driver","livenessProbe":{"initialDelaySeconds":30,"periodSeconds":30},"readinessProbe":{"initialDelaySeconds":10,"periodSeconds":30},"repository":"nvcr.io/nvidia/mellanox","startupProbe":{"initialDelaySeconds":10,"periodSeconds":20},"upgradePolicy":{"autoUpgrade":true,"drain":{"deleteEmptyDir":true,"enable":true,"force":true,"timeoutSeconds":300},"maxParallelUpgrades":1},"version":"25.01-0.6.0.0-0"},"rdmaSharedDevicePlugin":{"config":"{\n  \"configList\": [\n    {\n      \"resourceName\": \"shared_ib\",\n      \"rdmaHcaMax\": 63,\n      \"selectors\": {\n        \"vendors\": [\"15b3\"],\n        \"linkTypes\": [\"infiniband\"]\n      }\n    }\n  ]\n}\n","image":"k8s-rdma-shared-dev-plugin","repository":"ghcr.io/mellanox","version":"v1.5.2"},"secondaryNetwork":{"cniPlugins":{"image":"plugins","repository":"ghcr.io/k8snetworkplumbingwg","version":"v1.5.0"},"ipamPlugin":{"image":"whereabouts","repository":"ghcr.io/k8snetworkplumbingwg","version":"v0.7.0"},"ipoib":{"image":"ipoib-cni","repository":"ghcr.io/mellanox","version":"v1.2.1"},"multus":{"image":"multus-cni","repository":"ghcr.io/k8snetworkplumbingwg","version":"v4.1.0"}}}}
  creationTimestamp: "2025-05-01T17:17:39Z"
  generation: 2
  name: nic-cluster-policy
  resourceVersion: "10653"
  uid: 4e1e0187-ceae-499d-b613-01c2554fc515
spec:
  docaTelemetryService:
    image: doca_telemetry
    imagePullSecrets: []
    repository: nvcr.io/nvidia/doca
    version: 1.16.5-doca2.6.0-host
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: feature.node.kubernetes.io/pci-15b3.present
          operator: In
          values:
          - "true"
  ofedDriver:
    forcePrecompiled: false
    image: doca-driver
    imagePullSecrets: []
    livenessProbe:
      initialDelaySeconds: 30
      periodSeconds: 30
    readinessProbe:
      initialDelaySeconds: 10
      periodSeconds: 30
    repository: nvcr.io/nvidia/mellanox
    startupProbe:
      initialDelaySeconds: 10
      periodSeconds: 20
    terminationGracePeriodSeconds: 300
    upgradePolicy:
      autoUpgrade: true
      drain:
        deleteEmptyDir: true
        enable: true
        force: true
        timeoutSeconds: 300
      maxParallelUpgrades: 1
      safeLoad: false
    version: 25.01-0.6.0.0-0
  rdmaSharedDevicePlugin:
    config: |
      {
        "configList": [
          {
            "resourceName": "shared_ib",
            "rdmaHcaMax": 63,
            "selectors": {
              "vendors": ["15b3"],
              "linkTypes": ["infiniband"]
            }
          }
        ]
      }
    image: k8s-rdma-shared-dev-plugin
    imagePullSecrets: []
    repository: ghcr.io/mellanox
    version: v1.5.2
  secondaryNetwork:
    cniPlugins:
      image: plugins
      imagePullSecrets: []
      repository: ghcr.io/k8snetworkplumbingwg
      version: v1.5.0
    ipamPlugin:
      image: whereabouts
      imagePullSecrets: []
      repository: ghcr.io/k8snetworkplumbingwg
      version: v0.7.0
    ipoib:
      image: ipoib-cni
      imagePullSecrets: []
      repository: ghcr.io/mellanox
      version: v1.2.1
    multus:
      image: multus-cni
      imagePullSecrets: []
      repository: ghcr.io/k8snetworkplumbingwg
      version: v4.1.0
status:
  appliedStates:
  - name: state-multus-cni
    state: ready
  - name: state-container-networking-plugins
    state: ready
  - name: state-ipoib-cni
    state: ready
  - name: state-whereabouts-cni
    state: ready
  - name: state-OFED
    state: ready
  - name: state-SRIOV-device-plugin
    state: ignore
  - name: state-RDMA-device-plugin
    state: ready
  - name: state-ib-kubernetes
    state: ignore
  - name: state-nv-ipam-cni
    state: ignore
  - name: state-nic-feature-discovery
    state: ignore
  - name: state-doca-telemetry-service
    state: ready
  state: ready
---
apiVersion: mellanox.com/v1alpha1
kind: IPoIBNetwork
metadata:
  annotations:
    operator.ipoibnetwork.mellanox.com/last-network-namespace: default
  creationTimestamp: "2025-05-01T17:31:10Z"
  generation: 1
  name: aks-infiniband
  resourceVersion: "10422"
  uid: d8557fe4-6070-4367-a7d7-64df8c6e3757
spec:
  ipam: |
    {
      "type": "whereabouts",
      "datastore": "kubernetes",
      "kubernetes": {
        "kubeconfig": "/etc/cni/net.d/whereabouts.d/whereabouts.kubeconfig"
      },
      "range": "192.168.0.0/16",
      "exclude": [
       "192.168.0.0/32",
       "192.168.255.255/32"
      ],
      "log_file" : "/var/log/whereabouts.log",
      "log_level" : "info",
      "gateway": "192.168.0.1"
    }
  master: ibP257p0s0
  networkNamespace: default
status:
  ipoibNetworkAttachmentDef: k8s.cni.cncf.io/v1/namespaces/default/NetworkAttachmentDefinition/aks-infiniband
  state: ready

Events

Events:

~ kubectl get events
0s          Warning   FailedCreatePodSandBox   pod/ibtest-7bcb79db58-ssdsp               Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "c1dbf1d229d550be97df0a23973f9827ed6101062cad98b739dc6ea0daf2c339": plugin type="multus" name="multus-cni-network" failed (add): Multus: [default/ibtest-7bcb79db58-ssdsp/4573f52c-7caa-4161-9025-8553985a537a]: error waiting for pod: Unauthorized
0s          Warning   FailedCreatePodSandBox   pod/ibtest-7bcb79db58-hq44x               Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "0e875e1d9d6f6ff9e2b1f064800a2bb1799253c237967d4015a8c2326786311d": plugin type="multus" name="multus-cni-network" failed (add): Multus: [default/ibtest-7bcb79db58-hq44x/d900e9ce-99a4-4e02-90f1-c9380bba6b17]: error waiting for pod: Unauthorized
0s          Warning   FailedCreatePodSandBox   pod/ibtest-7bcb79db58-ssdsp               Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "5af0fb421e981d64427a3e121b3f353be2368924fa2c047e6c8c955fc43e006a": plugin type="multus" name="multus-cni-network" failed (add): Multus: [default/ibtest-7bcb79db58-ssdsp/4573f52c-7caa-4161-9025-8553985a537a]: error waiting for pod: Unauthorized
0s          Warning   FailedCreatePodSandBox   pod/ibtest-7bcb79db58-hq44x               Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "11ee4f02169163bd055c2a1f0a07b5127d7a5a025b517e0387802b97c0863f5a": plugin type="multus" name="multus-cni-network" failed (add): Multus: [default/ibtest-7bcb79db58-hq44x/d900e9ce-99a4-4e02-90f1-c9380bba6b17]: error waiting for pod: Unauthorized

Pods:

~ kubectl get pods -n network-operator -o wide
NAME                                                              READY   STATUS    RESTARTS      AGE   IP             NODE                                 NOMINATED NODE   READINESS GATES
cni-plugins-ds-5tdz7                                              1/1     Running   0             71m   10.224.0.6     aks-ibnodepool-73651064-vmss000001   <none>           <none>
cni-plugins-ds-zhn9m                                              1/1     Running   0             71m   10.224.0.5     aks-ibnodepool-73651064-vmss000000   <none>           <none>
doca-telemetry-service-7sq9h                                      1/1     Running   0             72m   10.224.0.6     aks-ibnodepool-73651064-vmss000001   <none>           <none>
doca-telemetry-service-sjtft                                      1/1     Running   0             73m   10.224.0.5     aks-ibnodepool-73651064-vmss000000   <none>           <none>
kube-ipoib-cni-ds-b2q25                                           1/1     Running   0             71m   10.224.0.6     aks-ibnodepool-73651064-vmss000001   <none>           <none>
kube-ipoib-cni-ds-gwj5p                                           1/1     Running   0             71m   10.224.0.5     aks-ibnodepool-73651064-vmss000000   <none>           <none>
kube-multus-ds-5w2p5                                              1/1     Running   0             71m   10.224.0.6     aks-ibnodepool-73651064-vmss000001   <none>           <none>
kube-multus-ds-t4xgn                                              1/1     Running   0             71m   10.224.0.5     aks-ibnodepool-73651064-vmss000000   <none>           <none>
mofed-ubuntu22.04-65f8c4747b-ds-hdk5p                             1/1     Running   6 (77m ago)   84m   10.224.0.6     aks-ibnodepool-73651064-vmss000001   <none>           <none>
mofed-ubuntu22.04-65f8c4747b-ds-xxxtd                             1/1     Running   6 (77m ago)   84m   10.224.0.5     aks-ibnodepool-73651064-vmss000000   <none>           <none>
network-operator-84c4dc4746-r6ljz                                 1/1     Running   1 (73m ago)   88m   10.244.1.200   aks-ibnodepool-73651064-vmss000001   <none>           <none>
network-operator-node-feature-discovery-gc-76ff7b66d6-cbqjm       1/1     Running   1 (74m ago)   88m   10.244.2.11    aks-ibnodepool-73651064-vmss000000   <none>           <none>
network-operator-node-feature-discovery-master-7fb7c58f86-hqn6m   1/1     Running   1 (73m ago)   88m   10.244.1.149   aks-ibnodepool-73651064-vmss000001   <none>           <none>
network-operator-node-feature-discovery-worker-6gd66              1/1     Running   1 (73m ago)   88m   10.244.1.62    aks-ibnodepool-73651064-vmss000001   <none>           <none>
network-operator-node-feature-discovery-worker-bt9m9              1/1     Running   3 (74m ago)   88m   10.244.2.226   aks-ibnodepool-73651064-vmss000000   <none>           <none>
network-operator-node-feature-discovery-worker-j85qt              1/1     Running   0             88m   10.244.0.153   aks-nodepool1-88695946-vmss000000    <none>           <none>
rdma-shared-dp-ds-bfkrn                                           1/1     Running   0             71m   10.224.0.5     aks-ibnodepool-73651064-vmss000000   <none>           <none>
rdma-shared-dp-ds-qcrtz                                           1/1     Running   0             71m   10.224.0.6     aks-ibnodepool-73651064-vmss000001   <none>           <none>
whereabouts-gk7gk                                                 1/1     Running   0             71m   10.224.0.6     aks-ibnodepool-73651064-vmss000001   <none>           <none>
whereabouts-gkd8b                                                 1/1     Running   0             71m   10.224.0.5     aks-ibnodepool-73651064-vmss000000   <none>           <none>

Pods

~ kubectl get pods -o wide
NAME                      READY   STATUS              RESTARTS   AGE   IP       NODE                                 NOMINATED NODE   READINESS GATES
ibtest-7bcb79db58-hq44x   0/1     ContainerCreating   0          20m   <none>   aks-ibnodepool-73651064-vmss000001   <none>           <none>
ibtest-7bcb79db58-ssdsp   0/1     ContainerCreating   0          20m   <none>   aks-ibnodepool-73651064-vmss000000   <none>           <none>

Node Descriptions

~ kubectl describe node aks-ibnodepool-73651064-vmss000000
Name:               aks-ibnodepool-73651064-vmss000000
Roles:              <none>
Labels:             agentpool=ibnodepool
                    beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/instance-type=Standard_HB120rs_v3
                    beta.kubernetes.io/os=linux
                    failure-domain.beta.kubernetes.io/region=southcentralus
                    failure-domain.beta.kubernetes.io/zone=0
                    feature.node.kubernetes.io/cpu-cpuid.ADX=true
                    feature.node.kubernetes.io/cpu-cpuid.AESNI=true
                    feature.node.kubernetes.io/cpu-cpuid.AVX=true
                    feature.node.kubernetes.io/cpu-cpuid.AVX2=true
                    feature.node.kubernetes.io/cpu-cpuid.CETSS=true
                    feature.node.kubernetes.io/cpu-cpuid.CLZERO=true
                    feature.node.kubernetes.io/cpu-cpuid.CMPXCHG8=true
                    feature.node.kubernetes.io/cpu-cpuid.FMA3=true
                    feature.node.kubernetes.io/cpu-cpuid.FSRM=true
                    feature.node.kubernetes.io/cpu-cpuid.FXSR=true
                    feature.node.kubernetes.io/cpu-cpuid.FXSROPT=true
                    feature.node.kubernetes.io/cpu-cpuid.HYPERVISOR=true
                    feature.node.kubernetes.io/cpu-cpuid.LAHF=true
                    feature.node.kubernetes.io/cpu-cpuid.MOVBE=true
                    feature.node.kubernetes.io/cpu-cpuid.MOVU=true
                    feature.node.kubernetes.io/cpu-cpuid.OSXSAVE=true
                    feature.node.kubernetes.io/cpu-cpuid.PSFD=true
                    feature.node.kubernetes.io/cpu-cpuid.RDPRU=true
                    feature.node.kubernetes.io/cpu-cpuid.SHA=true
                    feature.node.kubernetes.io/cpu-cpuid.SSE4A=true
                    feature.node.kubernetes.io/cpu-cpuid.SYSCALL=true
                    feature.node.kubernetes.io/cpu-cpuid.SYSEE=true
                    feature.node.kubernetes.io/cpu-cpuid.TOPEXT=true
                    feature.node.kubernetes.io/cpu-cpuid.VAES=true
                    feature.node.kubernetes.io/cpu-cpuid.VPCLMULQDQ=true
                    feature.node.kubernetes.io/cpu-cpuid.X87=true
                    feature.node.kubernetes.io/cpu-cpuid.XGETBV1=true
                    feature.node.kubernetes.io/cpu-cpuid.XSAVE=true
                    feature.node.kubernetes.io/cpu-cpuid.XSAVEC=true
                    feature.node.kubernetes.io/cpu-cpuid.XSAVEOPT=true
                    feature.node.kubernetes.io/cpu-cpuid.XSAVES=true
                    feature.node.kubernetes.io/cpu-hardware_multithreading=false
                    feature.node.kubernetes.io/cpu-model.family=25
                    feature.node.kubernetes.io/cpu-model.id=1
                    feature.node.kubernetes.io/cpu-model.vendor_id=AMD
                    feature.node.kubernetes.io/kernel-config.NO_HZ=true
                    feature.node.kubernetes.io/kernel-config.NO_HZ_FULL=true
                    feature.node.kubernetes.io/kernel-version.full=5.15.0-1086-azure
                    feature.node.kubernetes.io/kernel-version.major=5
                    feature.node.kubernetes.io/kernel-version.minor=15
                    feature.node.kubernetes.io/kernel-version.revision=0
                    feature.node.kubernetes.io/memory-numa=true
                    feature.node.kubernetes.io/pci-15b3.present=true
                    feature.node.kubernetes.io/rdma.available=true
                    feature.node.kubernetes.io/rdma.capable=true
                    feature.node.kubernetes.io/storage-nonrotationaldisk=true
                    feature.node.kubernetes.io/system-os_release.ID=ubuntu
                    feature.node.kubernetes.io/system-os_release.VERSION_ID=22.04
                    feature.node.kubernetes.io/system-os_release.VERSION_ID.major=22
                    feature.node.kubernetes.io/system-os_release.VERSION_ID.minor=04
                    kubernetes.azure.com/agentpool=ibnodepool
                    kubernetes.azure.com/azure-cni-overlay=true
                    kubernetes.azure.com/consolidated-additional-properties=3c0dd82a-26af-11f0-9220-1ec8ba1e8a16
                    kubernetes.azure.com/kubelet-identity-client-id=be74854c-8cb8-455e-9bed-ae3887877590
                    kubernetes.azure.com/mode=user
                    kubernetes.azure.com/network-name=aks-vnet-35137910
                    kubernetes.azure.com/network-subnet=aks-subnet
                    kubernetes.azure.com/node-image-version=AKSUbuntu-2204gen2containerd-202504.06.0
                    kubernetes.azure.com/nodenetwork-vnetguid=18139c70-55e3-4050-8cce-bae41a18f9ae
                    kubernetes.azure.com/nodepool-type=VirtualMachineScaleSets
                    kubernetes.azure.com/os-sku=Ubuntu
                    kubernetes.azure.com/podnetwork-type=overlay
                    kubernetes.azure.com/role=agent
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=aks-ibnodepool-73651064-vmss000000
                    kubernetes.io/os=linux
                    network.nvidia.com/operator.mofed.wait=false
                    node.kubernetes.io/instance-type=Standard_HB120rs_v3
                    nvidia.com/ofed-driver-upgrade-state=upgrade-done
                    topology.disk.csi.azure.com/zone=
                    topology.kubernetes.io/region=southcentralus
                    topology.kubernetes.io/zone=0
Annotations:        alpha.kubernetes.io/provided-node-ip: 10.224.0.5
                    csi.volume.kubernetes.io/nodeid:
                      {"disk.csi.azure.com":"aks-ibnodepool-73651064-vmss000000","file.csi.azure.com":"aks-ibnodepool-73651064-vmss000000"}
                    nfd.node.kubernetes.io/feature-labels:
                      cpu-cpuid.ADX,cpu-cpuid.AESNI,cpu-cpuid.AVX,cpu-cpuid.AVX2,cpu-cpuid.CETSS,cpu-cpuid.CLZERO,cpu-cpuid.CMPXCHG8,cpu-cpuid.FMA3,cpu-cpuid.FS...
                    node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Thu, 01 May 2025 10:12:37 -0700
Taints:             <none>
Unschedulable:      false
Lease:
  HolderIdentity:  aks-ibnodepool-73651064-vmss000000
  AcquireTime:     <unset>
  RenewTime:       Thu, 01 May 2025 11:35:08 -0700
Conditions:
  Type                          Status  LastHeartbeatTime                 LastTransitionTime                Reason                          Message
  ----                          ------  -----------------                 ------------------                ------                          -------
  FrequentKubeletRestart        False   Thu, 01 May 2025 11:30:48 -0700   Thu, 01 May 2025 10:40:42 -0700   NoFrequentKubeletRestart        kubelet is functioning properly
  FrequentDockerRestart         False   Thu, 01 May 2025 11:30:48 -0700   Thu, 01 May 2025 10:40:42 -0700   NoFrequentDockerRestart         docker is functioning properly
  KubeletProblem                False   Thu, 01 May 2025 11:30:48 -0700   Thu, 01 May 2025 10:40:42 -0700   KubeletIsUp                     kubelet service is up
  FrequentContainerdRestart     False   Thu, 01 May 2025 11:30:48 -0700   Thu, 01 May 2025 10:40:42 -0700   NoFrequentContainerdRestart     containerd is functioning properly
  VMEventScheduled              False   Thu, 01 May 2025 11:30:48 -0700   Thu, 01 May 2025 10:40:42 -0700   NoVMEventScheduled              VM has no scheduled event
  ContainerRuntimeProblem       False   Thu, 01 May 2025 11:30:48 -0700   Thu, 01 May 2025 10:40:42 -0700   ContainerRuntimeIsUp            container runtime service is up
  FrequentUnregisterNetDevice   False   Thu, 01 May 2025 11:30:48 -0700   Thu, 01 May 2025 10:40:42 -0700   NoFrequentUnregisterNetDevice   node is functioning properly
  FilesystemCorruptionProblem   False   Thu, 01 May 2025 11:30:48 -0700   Thu, 01 May 2025 10:40:42 -0700   FilesystemIsOK                  Filesystem is healthy
  KernelDeadlock                False   Thu, 01 May 2025 11:30:48 -0700   Thu, 01 May 2025 10:40:42 -0700   KernelHasNoDeadlock             kernel has no deadlock
  ReadonlyFilesystem            False   Thu, 01 May 2025 11:30:48 -0700   Thu, 01 May 2025 10:40:42 -0700   FilesystemIsNotReadOnly         Filesystem is not read-only
  MemoryPressure                False   Thu, 01 May 2025 11:31:08 -0700   Thu, 01 May 2025 10:27:30 -0700   KubeletHasSufficientMemory      kubelet has sufficient memory available
  DiskPressure                  False   Thu, 01 May 2025 11:31:08 -0700   Thu, 01 May 2025 10:27:30 -0700   KubeletHasNoDiskPressure        kubelet has no disk pressure
  PIDPressure                   False   Thu, 01 May 2025 11:31:08 -0700   Thu, 01 May 2025 10:27:30 -0700   KubeletHasSufficientPID         kubelet has sufficient PID available
  Ready                         True    Thu, 01 May 2025 11:31:08 -0700   Thu, 01 May 2025 10:27:30 -0700   KubeletReady                    kubelet is posting ready status
Addresses:
  InternalIP:  10.224.0.5
  Hostname:    aks-ibnodepool-73651064-vmss000000
Capacity:
  cpu:                120
  ephemeral-storage:  129886128Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             470529224Ki
  pods:               250
  rdma/shared_ib:     63
Allocatable:
  cpu:                118701m
  ephemeral-storage:  119703055367
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             465255624Ki
  pods:               250
  rdma/shared_ib:     63
System Info:
  Machine ID:                 7e1c1192ee9a4cb98da5e0c0ffe30da5
  System UUID:                20b106c6-1052-4ab2-ab73-535855f842d9
  Boot ID:                    9cac8e27-627e-450b-a731-71dd4e897e13
  Kernel Version:             5.15.0-1086-azure
  OS Image:                   Ubuntu 22.04.5 LTS
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  containerd://1.7.27-1
  Kubelet Version:            v1.31.7
  Kube-Proxy Version:         v1.31.7
Non-terminated Pods:          (20 in total)
  Namespace                   Name                                                           CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
  ---------                   ----                                                           ------------  ----------  ---------------  -------------  ---
  default                     ibtest-7bcb79db58-ssdsp                                        0 (0%)        0 (0%)      0 (0%)           0 (0%)         2m33s
  gatekeeper-system           gatekeeper-controller-7bf46f844f-z7qxs                         100m (0%)     2 (1%)      256Mi (0%)       2Gi (0%)       79m
  kube-system                 azure-cns-bbll5                                                40m (0%)      40m (0%)    250Mi (0%)       250Mi (0%)     82m
  kube-system                 azure-ip-masq-agent-6wdlh                                      100m (0%)     500m (0%)   50Mi (0%)        250Mi (0%)     82m
  kube-system                 cloud-node-manager-z522k                                       50m (0%)      0 (0%)      50Mi (0%)        512Mi (0%)     82m
  kube-system                 csi-azuredisk-node-2xvwq                                       30m (0%)      0 (0%)      60Mi (0%)        2800Mi (0%)    82m
  kube-system                 csi-azurefile-node-75kdw                                       30m (0%)      0 (0%)      60Mi (0%)        600Mi (0%)     82m
  kube-system                 konnectivity-agent-84b85b848c-kxcn5                            20m (0%)      1 (0%)      20Mi (0%)        1Gi (0%)       67m
  kube-system                 kube-proxy-5jzvd                                               100m (0%)     0 (0%)      0 (0%)           0 (0%)         82m
  kube-system                 metrics-server-67cc7fb655-b5zpd                                157m (0%)     252m (0%)   142Mi (0%)       412Mi (0%)     67m
  monitoring                  kube-prometheus-prometheus-node-exporter-2c8vj                 0 (0%)        0 (0%)      0 (0%)           0 (0%)         82m
  network-operator            cni-plugins-ds-zhn9m                                           100m (0%)     100m (0%)   50Mi (0%)        50Mi (0%)      64m
  network-operator            doca-telemetry-service-sjtft                                   0 (0%)        0 (0%)      0 (0%)           0 (0%)         66m
  network-operator            kube-ipoib-cni-ds-gwj5p                                        100m (0%)     100m (0%)   50Mi (0%)        50Mi (0%)      64m
  network-operator            kube-multus-ds-t4xgn                                           100m (0%)     100m (0%)   50Mi (0%)        50Mi (0%)      64m
  network-operator            mofed-ubuntu22.04-65f8c4747b-ds-xxxtd                          0 (0%)        0 (0%)      0 (0%)           0 (0%)         77m
  network-operator            network-operator-node-feature-discovery-gc-76ff7b66d6-cbqjm    10m (0%)      0 (0%)      128Mi (0%)       1Gi (0%)       81m
  network-operator            network-operator-node-feature-discovery-worker-bt9m9           5m (0%)       0 (0%)      64Mi (0%)        512Mi (0%)     81m
  network-operator            rdma-shared-dp-ds-bfkrn                                        0 (0%)        0 (0%)      0 (0%)           0 (0%)         64m
  network-operator            whereabouts-gkd8b                                              100m (0%)     100m (0%)   100Mi (0%)       200Mi (0%)     64m
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests     Limits
  --------           --------     ------
  cpu                1042m (0%)   4192m (3%)
  memory             1330Mi (0%)  9782Mi (2%)
  ephemeral-storage  0 (0%)       0 (0%)
  hugepages-1Gi      0 (0%)       0 (0%)
  hugepages-2Mi      0 (0%)       0 (0%)
  rdma/shared_ib     1            1
Events:              <none>

~ kubectl describe node aks-ibnodepool-73651064-vmss000001
Name:               aks-ibnodepool-73651064-vmss000001
Roles:              <none>
Labels:             agentpool=ibnodepool
                    beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/instance-type=Standard_HB120rs_v3
                    beta.kubernetes.io/os=linux
                    failure-domain.beta.kubernetes.io/region=southcentralus
                    failure-domain.beta.kubernetes.io/zone=0
                    feature.node.kubernetes.io/cpu-cpuid.ADX=true
                    feature.node.kubernetes.io/cpu-cpuid.AESNI=true
                    feature.node.kubernetes.io/cpu-cpuid.AVX=true
                    feature.node.kubernetes.io/cpu-cpuid.AVX2=true
                    feature.node.kubernetes.io/cpu-cpuid.CETSS=true
                    feature.node.kubernetes.io/cpu-cpuid.CLZERO=true
                    feature.node.kubernetes.io/cpu-cpuid.CMPXCHG8=true
                    feature.node.kubernetes.io/cpu-cpuid.FMA3=true
                    feature.node.kubernetes.io/cpu-cpuid.FSRM=true
                    feature.node.kubernetes.io/cpu-cpuid.FXSR=true
                    feature.node.kubernetes.io/cpu-cpuid.FXSROPT=true
                    feature.node.kubernetes.io/cpu-cpuid.HYPERVISOR=true
                    feature.node.kubernetes.io/cpu-cpuid.LAHF=true
                    feature.node.kubernetes.io/cpu-cpuid.MOVBE=true
                    feature.node.kubernetes.io/cpu-cpuid.MOVU=true
                    feature.node.kubernetes.io/cpu-cpuid.OSXSAVE=true
                    feature.node.kubernetes.io/cpu-cpuid.PSFD=true
                    feature.node.kubernetes.io/cpu-cpuid.RDPRU=true
                    feature.node.kubernetes.io/cpu-cpuid.SHA=true
                    feature.node.kubernetes.io/cpu-cpuid.SSE4A=true
                    feature.node.kubernetes.io/cpu-cpuid.SYSCALL=true
                    feature.node.kubernetes.io/cpu-cpuid.SYSEE=true
                    feature.node.kubernetes.io/cpu-cpuid.TOPEXT=true
                    feature.node.kubernetes.io/cpu-cpuid.VAES=true
                    feature.node.kubernetes.io/cpu-cpuid.VPCLMULQDQ=true
                    feature.node.kubernetes.io/cpu-cpuid.X87=true
                    feature.node.kubernetes.io/cpu-cpuid.XGETBV1=true
                    feature.node.kubernetes.io/cpu-cpuid.XSAVE=true
                    feature.node.kubernetes.io/cpu-cpuid.XSAVEC=true
                    feature.node.kubernetes.io/cpu-cpuid.XSAVEOPT=true
                    feature.node.kubernetes.io/cpu-cpuid.XSAVES=true
                    feature.node.kubernetes.io/cpu-hardware_multithreading=false
                    feature.node.kubernetes.io/cpu-model.family=25
                    feature.node.kubernetes.io/cpu-model.id=1
                    feature.node.kubernetes.io/cpu-model.vendor_id=AMD
                    feature.node.kubernetes.io/kernel-config.NO_HZ=true
                    feature.node.kubernetes.io/kernel-config.NO_HZ_FULL=true
                    feature.node.kubernetes.io/kernel-version.full=5.15.0-1086-azure
                    feature.node.kubernetes.io/kernel-version.major=5
                    feature.node.kubernetes.io/kernel-version.minor=15
                    feature.node.kubernetes.io/kernel-version.revision=0
                    feature.node.kubernetes.io/memory-numa=true
                    feature.node.kubernetes.io/pci-15b3.present=true
                    feature.node.kubernetes.io/rdma.available=true
                    feature.node.kubernetes.io/rdma.capable=true
                    feature.node.kubernetes.io/storage-nonrotationaldisk=true
                    feature.node.kubernetes.io/system-os_release.ID=ubuntu
                    feature.node.kubernetes.io/system-os_release.VERSION_ID=22.04
                    feature.node.kubernetes.io/system-os_release.VERSION_ID.major=22
                    feature.node.kubernetes.io/system-os_release.VERSION_ID.minor=04
                    kubernetes.azure.com/agentpool=ibnodepool
                    kubernetes.azure.com/azure-cni-overlay=true
                    kubernetes.azure.com/consolidated-additional-properties=3c0dd82a-26af-11f0-9220-1ec8ba1e8a16
                    kubernetes.azure.com/kubelet-identity-client-id=be74854c-8cb8-455e-9bed-ae3887877590
                    kubernetes.azure.com/mode=user
                    kubernetes.azure.com/network-name=aks-vnet-35137910
                    kubernetes.azure.com/network-subnet=aks-subnet
                    kubernetes.azure.com/node-image-version=AKSUbuntu-2204gen2containerd-202504.06.0
                    kubernetes.azure.com/nodenetwork-vnetguid=18139c70-55e3-4050-8cce-bae41a18f9ae
                    kubernetes.azure.com/nodepool-type=VirtualMachineScaleSets
                    kubernetes.azure.com/os-sku=Ubuntu
                    kubernetes.azure.com/podnetwork-type=overlay
                    kubernetes.azure.com/role=agent
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=aks-ibnodepool-73651064-vmss000001
                    kubernetes.io/os=linux
                    network.nvidia.com/operator.mofed.wait=false
                    node.kubernetes.io/instance-type=Standard_HB120rs_v3
                    nvidia.com/ofed-driver-upgrade-state=upgrade-done
                    topology.disk.csi.azure.com/zone=
                    topology.kubernetes.io/region=southcentralus
                    topology.kubernetes.io/zone=0
Annotations:        alpha.kubernetes.io/provided-node-ip: 10.224.0.6
                    csi.volume.kubernetes.io/nodeid:
                      {"disk.csi.azure.com":"aks-ibnodepool-73651064-vmss000001","file.csi.azure.com":"aks-ibnodepool-73651064-vmss000001"}
                    nfd.node.kubernetes.io/feature-labels:
                      cpu-cpuid.ADX,cpu-cpuid.AESNI,cpu-cpuid.AVX,cpu-cpuid.AVX2,cpu-cpuid.CETSS,cpu-cpuid.CLZERO,cpu-cpuid.CMPXCHG8,cpu-cpuid.FMA3,cpu-cpuid.FS...
                    node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Thu, 01 May 2025 10:12:37 -0700
Taints:             <none>
Unschedulable:      false
Lease:
  HolderIdentity:  aks-ibnodepool-73651064-vmss000001
  AcquireTime:     <unset>
  RenewTime:       Thu, 01 May 2025 11:34:43 -0700
Conditions:
  Type                          Status  LastHeartbeatTime                 LastTransitionTime                Reason                          Message
  ----                          ------  -----------------                 ------------------                ------                          -------
  VMEventScheduled              False   Thu, 01 May 2025 11:30:43 -0700   Thu, 01 May 2025 10:40:34 -0700   NoVMEventScheduled              VM has no scheduled event
  FrequentUnregisterNetDevice   False   Thu, 01 May 2025 11:30:43 -0700   Thu, 01 May 2025 10:39:41 -0700   NoFrequentUnregisterNetDevice   node is functioning properly
  ContainerRuntimeProblem       False   Thu, 01 May 2025 11:30:43 -0700   Thu, 01 May 2025 10:39:41 -0700   ContainerRuntimeIsUp            container runtime service is up
  FrequentKubeletRestart        False   Thu, 01 May 2025 11:30:43 -0700   Thu, 01 May 2025 10:39:41 -0700   NoFrequentKubeletRestart        kubelet is functioning properly
  FrequentContainerdRestart     False   Thu, 01 May 2025 11:30:43 -0700   Thu, 01 May 2025 10:39:41 -0700   NoFrequentContainerdRestart     containerd is functioning properly
  KubeletProblem                False   Thu, 01 May 2025 11:30:43 -0700   Thu, 01 May 2025 10:39:41 -0700   KubeletIsUp                     kubelet service is up
  FilesystemCorruptionProblem   False   Thu, 01 May 2025 11:30:43 -0700   Thu, 01 May 2025 10:39:41 -0700   FilesystemIsOK                  Filesystem is healthy
  ReadonlyFilesystem            False   Thu, 01 May 2025 11:30:43 -0700   Thu, 01 May 2025 10:39:41 -0700   FilesystemIsNotReadOnly         Filesystem is not read-only
  FrequentDockerRestart         False   Thu, 01 May 2025 11:30:43 -0700   Thu, 01 May 2025 10:39:41 -0700   NoFrequentDockerRestart         docker is functioning properly
  KernelDeadlock                False   Thu, 01 May 2025 11:30:43 -0700   Thu, 01 May 2025 10:39:41 -0700   KernelHasNoDeadlock             kernel has no deadlock
  MemoryPressure                False   Thu, 01 May 2025 11:31:15 -0700   Thu, 01 May 2025 10:28:28 -0700   KubeletHasSufficientMemory      kubelet has sufficient memory available
  DiskPressure                  False   Thu, 01 May 2025 11:31:15 -0700   Thu, 01 May 2025 10:28:28 -0700   KubeletHasNoDiskPressure        kubelet has no disk pressure
  PIDPressure                   False   Thu, 01 May 2025 11:31:15 -0700   Thu, 01 May 2025 10:28:28 -0700   KubeletHasSufficientPID         kubelet has sufficient PID available
  Ready                         True    Thu, 01 May 2025 11:31:15 -0700   Thu, 01 May 2025 10:28:28 -0700   KubeletReady                    kubelet is posting ready status
Addresses:
  InternalIP:  10.224.0.6
  Hostname:    aks-ibnodepool-73651064-vmss000001
Capacity:
  cpu:                120
  ephemeral-storage:  129886128Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             470529208Ki
  pods:               250
  rdma/shared_ib:     63
Allocatable:
  cpu:                118701m
  ephemeral-storage:  119703055367
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             465255608Ki
  pods:               250
  rdma/shared_ib:     63
System Info:
  Machine ID:                 4186122f79bd4663ac32b613b7cab07c
  System UUID:                34010c0b-1ea2-44ea-b081-d0f570bc1889
  Boot ID:                    b1eae85f-312b-4383-8551-4a5cdffa11e9
  Kernel Version:             5.15.0-1086-azure
  OS Image:                   Ubuntu 22.04.5 LTS
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  containerd://1.7.27-1
  Kubelet Version:            v1.31.7
  Kube-Proxy Version:         v1.31.7
Non-terminated Pods:          (19 in total)
  Namespace                   Name                                                               CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
  ---------                   ----                                                               ------------  ----------  ---------------  -------------  ---
  default                     ibtest-7bcb79db58-hq44x                                            0 (0%)        0 (0%)      0 (0%)           0 (0%)         2m5s
  gatekeeper-system           gatekeeper-controller-7bf46f844f-258nc                             100m (0%)     2 (1%)      256Mi (0%)       2Gi (0%)       78m
  kube-system                 azure-cns-h5xlf                                                    40m (0%)      40m (0%)    250Mi (0%)       250Mi (0%)     82m
  kube-system                 azure-ip-masq-agent-mkb6m                                          100m (0%)     500m (0%)   50Mi (0%)        250Mi (0%)     82m
  kube-system                 cloud-node-manager-rxfzn                                           50m (0%)      0 (0%)      50Mi (0%)        512Mi (0%)     82m
  kube-system                 csi-azuredisk-node-m68np                                           30m (0%)      0 (0%)      60Mi (0%)        2800Mi (0%)    82m
  kube-system                 csi-azurefile-node-mv7lc                                           30m (0%)      0 (0%)      60Mi (0%)        600Mi (0%)     82m
  kube-system                 kube-proxy-frf5r                                                   100m (0%)     0 (0%)      0 (0%)           0 (0%)         82m
  monitoring                  kube-prometheus-prometheus-node-exporter-mtnkq                     0 (0%)        0 (0%)      0 (0%)           0 (0%)         82m
  network-operator            cni-plugins-ds-5tdz7                                               100m (0%)     100m (0%)   50Mi (0%)        50Mi (0%)      63m
  network-operator            doca-telemetry-service-7sq9h                                       0 (0%)        0 (0%)      0 (0%)           0 (0%)         65m
  network-operator            kube-ipoib-cni-ds-b2q25                                            100m (0%)     100m (0%)   50Mi (0%)        50Mi (0%)      63m
  network-operator            kube-multus-ds-5w2p5                                               100m (0%)     100m (0%)   50Mi (0%)        50Mi (0%)      63m
  network-operator            mofed-ubuntu22.04-65f8c4747b-ds-hdk5p                              0 (0%)        0 (0%)      0 (0%)           0 (0%)         77m
  network-operator            network-operator-84c4dc4746-r6ljz                                  5m (0%)       500m (0%)   64Mi (0%)        128Mi (0%)     80m
  network-operator            network-operator-node-feature-discovery-master-7fb7c58f86-hqn6m    100m (0%)     0 (0%)      128Mi (0%)       4Gi (0%)       80m
  network-operator            network-operator-node-feature-discovery-worker-6gd66               5m (0%)       0 (0%)      64Mi (0%)        512Mi (0%)     80m
  network-operator            rdma-shared-dp-ds-qcrtz                                            0 (0%)        0 (0%)      0 (0%)           0 (0%)         63m
  network-operator            whereabouts-gk7gk                                                  100m (0%)     100m (0%)   100Mi (0%)       200Mi (0%)     63m
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests     Limits
  --------           --------     ------
  cpu                960m (0%)    3440m (2%)
  memory             1232Mi (0%)  11546Mi (2%)
  ephemeral-storage  0 (0%)       0 (0%)
  hugepages-1Gi      0 (0%)       0 (0%)
  hugepages-2Mi      0 (0%)       0 (0%)
  rdma/shared_ib     1            1
Events:
  Type     Reason              Age                From                                                          Message
  ----     ------              ----               ----                                                          -------
  Normal   NoVMEventScheduled  54m                custom-scheduledevents-consolidated-condition-plugin-monitor  Node condition VMEventScheduled is now: Unknown, reason: NoVMEventScheduled, message: "IMDS query failed, exit code: 28\nConnection timed out after 24 seconds."
  Warning  PreemptScheduled    54m (x2 over 54m)  custom-scheduledevents-consolidated-preempt-plugin-monitor    IMDS query failed, exit code: 28
Connection timed out after 24 seconds.
  Normal  NoVMEventScheduled  54m  custom-scheduledevents-consolidated-condition-plugin-monitor  Node condition VMEventScheduled is now: False, reason: NoVMEventScheduled, message: "VM has no scheduled event"

Multus DS pods logs

~ kubectl logs -n network-operator kube-multus-ds-5w2p5 -c kube-multus
kubeconfig is created in /host/etc/cni/net.d/multus.d/multus.kubeconfig
kubeconfig file is created.
master capabilities is get from conflist
multus config file is created.

~ kubectl logs -n network-operator kube-multus-ds-t4xgn -c kube-multus
kubeconfig is created in /host/etc/cni/net.d/multus.d/multus.kubeconfig
kubeconfig file is created.
master capabilities is get from conflist
multus config file is created.

[thuffmlx]

is this the same problem that Herb Woodruff contacted me on today?
he bounced multus, which recreated the certs and fixed the auth problem.

[surajssd]

is this the same problem that Herb Woodruff contacted me on today? he bounced multus, which recreated the certs and fixed the auth problem.

@thuffmlx I am not sure I understand what you mean here?

[thuffmlx]

multus has an authorization process.
certs can expire and the authorization will fail.
restarting the multus pod can recover from this:

How Multus Authentication Works
Multus CNI is a meta-plugin for Kubernetes that enables pods to have multiple network interfaces by coordinating multiple CNI plugins. To function properly, Multus needs to interact with the Kubernetes API-primarily to read NetworkAttachmentDefinition custom resources that specify additional network configurations for pods.

Authentication Mechanism
Multus authenticates to the Kubernetes API using a kubeconfig file. This kubeconfig contains credentials and cluster information, allowing the Multus CNI plugin running on each node to securely communicate with the Kubernetes API server.

Key Steps in Multus Authentication:
Service Account Creation:
When Multus is deployed (typically via a DaemonSet), it uses a Kubernetes Service Account (often named multus) in the kube-system namespace. This Service Account is granted permissions to read NetworkAttachmentDefinition resources.

Kubeconfig Generation:
The Multus deployment process creates a kubeconfig file-usually at /etc/cni/net.d/multus.d/multus.kubeconfig-on each node. This file contains:

The API server endpoint

The cluster CA certificate

The Service Account token for authentication

Kubeconfig Details:
The kubeconfig is generated using the Service Account’s token and CA certificate, which are extracted from Kubernetes secrets. Here is a simplified process:

bash
mkdir -p /etc/cni/net.d/multus.d

Extract CA and token from ServiceAccount secret

SERVICEACCOUNT_CA=$(kubectl get secrets -n=kube-system -o json | jq -r '.items[]|select(.metadata.annotations."kubernetes.io/service-account.name"=="multus")| .data."ca.crt"')
SERVICEACCOUNT_TOKEN=$(kubectl get secrets -n=kube-system -o json | jq -r '.items[]|select(.metadata.annotations."kubernetes.io/service-account.name"=="multus")| .data.token' | base64 -d )

Compose kubeconfig

cat > /etc/cni/net.d/multus.d/multus.kubeconfig <<EOF
apiVersion: v1
kind: Config
clusters:

• name: local
  cluster:
  server: https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}
  certificate-authority-data: ${SERVICEACCOUNT_CA}
  users:
• name: multus
  user:
  token: "${SERVICEACCOUNT_TOKEN}"
  contexts:
• name: multus-context
  context:
  cluster: local
  user: multus
  current-context: multus-context
  EOF

[surajssd]

I am suspecting it's caused by --skip-config-watch being enabled in https://github.com/Mellanox/network-operator/blob/master/manifests/state-multus-cni/0050-multus-ds.yml#L77, which caused multus not to rotate the kubeconfig in https://github.com/k8snetworkplumbingwg/multus-cni/blob/master/cmd/thin_entrypoint/main.go#L634. Could we remove that flag?

@chewong So even after removing the flag as you've mentioned the pods stay stuck in the ContainerCreating state:

~ kubectl get pods -o wide
NAME                      READY   STATUS              RESTARTS   AGE     IP       NODE                                 NOMINATED NODE   READINESS GATES
ibtest-7bcb79db58-62f9k   0/1     ContainerCreating   0          6m21s   <none>   aks-ibnodepool-32725015-vmss000000   <none>           <none>
ibtest-7bcb79db58-7kdt9   0/1     ContainerCreating   0          6m21s   <none>   aks-ibnodepool-32725015-vmss000001   <none>           <none>

Although I am seeing a different event this time error adding container to network "aks-infiniband": error at storage engine: time limit exceeded while waiting to become leader, earlier it was error waiting for pod: Unauthorized:

~ kubectl get events
LAST SEEN   TYPE      REASON                   OBJECT                            MESSAGE
6m33s       Normal    Scheduled                pod/ibtest-7bcb79db58-62f9k       Successfully assigned default/ibtest-7bcb79db58-62f9k to aks-ibnodepool-32725015-vmss000000
6m32s       Normal    AddedInterface           pod/ibtest-7bcb79db58-62f9k       Add eth0 [10.244.2.25/16] from azure
2m32s       Warning   FailedCreatePodSandBox   pod/ibtest-7bcb79db58-62f9k       Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "ab3f1914bf837f53b5aeea7624b845f7594e62650a817f445ef2726fd9614be7": plugin type="multus" name="multus-cni-network" failed (add): [default/ibtest-7bcb79db58-62f9k/288e2a28-c69f-4f11-9606-c4ddf30fde36:aks-infiniband]: error adding container to network "aks-infiniband": error at storage engine: time limit exceeded while waiting to become leader
2m31s       Normal    SandboxChanged           pod/ibtest-7bcb79db58-62f9k       Pod sandbox changed, it will be killed and re-created.
2m31s       Normal    AddedInterface           pod/ibtest-7bcb79db58-62f9k       Add eth0 [10.244.2.163/16] from azure
6m33s       Normal    Scheduled                pod/ibtest-7bcb79db58-7kdt9       Successfully assigned default/ibtest-7bcb79db58-7kdt9 to aks-ibnodepool-32725015-vmss000001
6m32s       Normal    AddedInterface           pod/ibtest-7bcb79db58-7kdt9       Add eth0 [10.244.1.20/16] from azure
2m32s       Warning   FailedCreatePodSandBox   pod/ibtest-7bcb79db58-7kdt9       Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "81f2a82e7506326148ee37818eca6960bb78e783d4af8310f2dbebf1d84327d5": plugin type="multus" name="multus-cni-network" failed (add): [default/ibtest-7bcb79db58-7kdt9/ffc59238-581d-42e8-98b9-c7f95ca07c7c:aks-infiniband]: error adding container to network "aks-infiniband": error at storage engine: time limit exceeded while waiting to become leader
2m31s       Normal    SandboxChanged           pod/ibtest-7bcb79db58-7kdt9       Pod sandbox changed, it will be killed and re-created.
2m31s       Normal    AddedInterface           pod/ibtest-7bcb79db58-7kdt9       Add eth0 [10.244.1.164/16] from azure
6m33s       Normal    SuccessfulCreate         replicaset/ibtest-7bcb79db58      Created pod: ibtest-7bcb79db58-7kdt9
6m33s       Normal    SuccessfulCreate         replicaset/ibtest-7bcb79db58      Created pod: ibtest-7bcb79db58-62f9k
6m33s       Normal    ScalingReplicaSet        deployment/ibtest                 Scaled up replica set ibtest-7bcb79db58 to 2

The daemonset without the --skip-config-watch flag:

~ kubectl describe ds kube-multus-ds -n network-operator
Name:           kube-multus-ds
Namespace:      network-operator
Selector:       name=multus
Node-Selector:  <none>
Labels:         app=multus
                name=multus
                nvidia.network-operator.state=state-multus-cni
                tier=node
Annotations:    deprecated.daemonset.template.generation: 2
                nvidia.network-operator.revision: 656303421
Desired Number of Nodes Scheduled: 2
Current Number of Nodes Scheduled: 2
Number of Nodes Scheduled with Up-to-date Pods: 2
Number of Nodes Scheduled with Available Pods: 2
Number of Nodes Misscheduled: 0
Pods Status:  2 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:           app=multus
                    name=multus
                    tier=node
  Service Account:  multus
  Init Containers:
   install-multus-binary:
    Image:      ghcr.io/k8snetworkplumbingwg/multus-cni:v4.1.0
    Port:       <none>
    Host Port:  <none>
    Command:
      /install_multus
    Args:
      --type
      thin
    Requests:
      cpu:        10m
      memory:     15Mi
    Environment:  <none>
    Mounts:
      /host/opt/cni/bin from cnibin (rw)
  Containers:
   kube-multus:
    Image:      ghcr.io/k8snetworkplumbingwg/multus-cni:v4.1.0
    Port:       <none>
    Host Port:  <none>
    Command:
      /thin_entrypoint
    Args:
      --multus-conf-file=auto
      --multus-autoconfig-dir=/host/etc/cni/net.d
      --cni-conf-dir=/host/etc/cni/net.d
      --cleanup-config-on-exit
    Limits:
      cpu:     100m
      memory:  50Mi
    Requests:
      cpu:        100m
      memory:     50Mi
    Environment:  <none>
    Mounts:
      /host/etc/cni/net.d from cni (rw)
      /host/opt/cni/bin from cnibin (rw)
  Volumes:
   cni:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/cni/net.d
    HostPathType:
   cnibin:
    Type:               HostPath (bare host directory volume)
    Path:               /opt/cni/bin
    HostPathType:
  Priority Class Name:  system-node-critical
  Node-Selectors:       <none>
  Tolerations:          nvidia.com/gpu:NoSchedule op=Exists
Events:                 <none>

And the multus pods are up for almost an hour:

~ kubectl get pods -n network-operator -o wide | grep multus
kube-multus-ds-ll52m   1/1     Running   0             80m   10.224.0.6     aks-ibnodepool-32725015-vmss000001   <none>           <none>
kube-multus-ds-z7qn7   1/1     Running   0             80m   10.224.0.5     aks-ibnodepool-32725015-vmss000000   <none>           <none>

And the new logs look like this (it is the same for both the pods):

~ kubectl logs -n network-operator kube-multus-ds-ll52m
Defaulted container "kube-multus" out of: kube-multus, install-multus-binary (init)
kubeconfig is created in /host/etc/cni/net.d/multus.d/multus.kubeconfig
kubeconfig file is created.
master capabilities is get from conflist
multus config file is created.
Entering watch loop...
CA (false) or SA token (true) changed - recreating kubeconfig
kubeconfig is created in /host/etc/cni/net.d/multus.d/multus.kubeconfig

[surajssd]

@thuffmlx yes, recreating pods from the network-operator namespace does help. But I wish if the multus was able to recreate the certificates on it own.