The Gpu-operator labels nodes with gpu where no gpu present

[urbaman]

Hi, I have a 6-nodes microk8s cluster, VMs on a three-nodes proxmox environment.

• mk8s1, mk8s4 are on proxmox-node1, mk8s1 has GPU passed-through
• mk8s2, mk8s5 are on proxmox-node1
• mk8s3, mk8s6 are on proxmox-node1, mk8s3 has GPU passed-through

I do expect nfd/gpu-operator to find and label mk8s1 and mk8s3 with gpu, not mk8s4 and mk8s6 (on the same proxmox nodes)
mk8s2 and mk8s5, on a proxmox node without GPU, are not labeled.

Thing is that mk8s4 has no gpu but gets labeled anyway:

ubuntu@mk8s4:~$ modinfo nvidia | grep ^version
modinfo: ERROR: Module nvidia not found.
ubuntu@mk8s4:~$ nvidia-smi
Command 'nvidia-smi' not found, but can be installed with:
sudo apt install nvidia-utils-525         # version 525.147.05-0ubuntu1, or
sudo apt install nvidia-utils-525-server  # version 525.147.05-0ubuntu1
sudo apt install nvidia-utils-470         # version 470.256.02-0ubuntu0.24.04.1
sudo apt install nvidia-utils-470-server  # version 470.256.02-0ubuntu0.24.04.1
sudo apt install nvidia-utils-535         # version 535.183.01-0ubuntu0.24.04.1
sudo apt install nvidia-utils-535-server  # version 535.216.01-0ubuntu0.24.04.1
sudo apt install nvidia-utils-550         # version 550.120-0ubuntu0.24.04.1
sudo apt install nvidia-utils-550-server  # version 550.127.05-0ubuntu0.24.04.1
ubuntu@mk8s4:~$ kubectl get nodes mk8s4 --show-labels
NAME    STATUS   ROLES    AGE   VERSION   LABELS
mk8s4   Ready    <none>   27d   v1.31.5   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,feature.node.kubernetes.io/cpu-cpuid.ADX=true,feature.node.kubernetes.io/cpu-cpuid.AESNI=true,feature.node.kubernetes.io/cpu-cpuid.AVX2=true,feature.node.kubernetes.io/cpu-cpuid.AVX=true,feature.node.kubernetes.io/cpu-cpuid.AVXVNNI=true,feature.node.kubernetes.io/cpu-cpuid.CMPXCHG8=true,feature.node.kubernetes.io/cpu-cpuid.FLUSH_L1D=true,feature.node.kubernetes.io/cpu-cpuid.FMA3=true,feature.node.kubernetes.io/cpu-cpuid.FSRM=true,feature.node.kubernetes.io/cpu-cpuid.FXSR=true,feature.node.kubernetes.io/cpu-cpuid.FXSROPT=true,feature.node.kubernetes.io/cpu-cpuid.GFNI=true,feature.node.kubernetes.io/cpu-cpuid.HYPERVISOR=true,feature.node.kubernetes.io/cpu-cpuid.IA32_ARCH_CAP=true,feature.node.kubernetes.io/cpu-cpuid.IBPB=true,feature.node.kubernetes.io/cpu-cpuid.IBRS=true,feature.node.kubernetes.io/cpu-cpuid.LAHF=true,feature.node.kubernetes.io/cpu-cpuid.MD_CLEAR=true,feature.node.kubernetes.io/cpu-cpuid.MOVBE=true,feature.node.kubernetes.io/cpu-cpuid.MOVDIR64B=true,feature.node.kubernetes.io/cpu-cpuid.MOVDIRI=true,feature.node.kubernetes.io/cpu-cpuid.OSXSAVE=true,feature.node.kubernetes.io/cpu-cpuid.SERIALIZE=true,feature.node.kubernetes.io/cpu-cpuid.SHA=true,feature.node.kubernetes.io/cpu-cpuid.SPEC_CTRL_SSBD=true,feature.node.kubernetes.io/cpu-cpuid.STIBP=true,feature.node.kubernetes.io/cpu-cpuid.STOSB_SHORT=true,feature.node.kubernetes.io/cpu-cpuid.SYSCALL=true,feature.node.kubernetes.io/cpu-cpuid.SYSEE=true,feature.node.kubernetes.io/cpu-cpuid.VAES=true,feature.node.kubernetes.io/cpu-cpuid.VMX=true,feature.node.kubernetes.io/cpu-cpuid.VPCLMULQDQ=true,feature.node.kubernetes.io/cpu-cpuid.WAITPKG=true,feature.node.kubernetes.io/cpu-cpuid.X87=true,feature.node.kubernetes.io/cpu-cpuid.XGETBV1=true,feature.node.kubernetes.io/cpu-cpuid.XSAVE=true,feature.node.kubernetes.io/cpu-cpuid.XSAVEC=true,feature.node.kubernetes.io/cpu-cpuid.XSAVEOPT=true,feature.node.kubernetes.io/cpu-cpuid.XSAVES=true,feature.node.kubernetes.io/cpu-hardware_multithreading=false,feature.node.kubernetes.io/cpu-model.family=6,feature.node.kubernetes.io/cpu-model.id=186,feature.node.kubernetes.io/cpu-model.vendor_id=Intel,feature.node.kubernetes.io/kernel-config.NO_HZ=true,feature.node.kubernetes.io/kernel-config.NO_HZ_FULL=true,feature.node.kubernetes.io/kernel-version.full=6.8.0-53-generic,feature.node.kubernetes.io/kernel-version.major=6,feature.node.kubernetes.io/kernel-version.minor=8,feature.node.kubernetes.io/kernel-version.revision=0,feature.node.kubernetes.io/pci-1234.present=true,feature.node.kubernetes.io/pci-1af4.present=true,feature.node.kubernetes.io/storage-nonrotationaldisk=true,feature.node.kubernetes.io/system-os_release.ID=ubuntu,feature.node.kubernetes.io/system-os_release.VERSION_ID.major=24,feature.node.kubernetes.io/system-os_release.VERSION_ID.minor=04,feature.node.kubernetes.io/system-os_release.VERSION_ID=24.04,kubernetes.io/arch=amd64,kubernetes.io/hostname=mk8s4,kubernetes.io/os=linux,microk8s.io/cluster=true,node.kubernetes.io/microk8s-controlplane=microk8s-controlplane,nvidia.com/cuda.driver-version.full=550.127.08,nvidia.com/cuda.driver-version.major=550,nvidia.com/cuda.driver-version.minor=127,nvidia.com/cuda.driver-version.revision=08,nvidia.com/cuda.driver.major=550,nvidia.com/cuda.driver.minor=127,nvidia.com/cuda.driver.rev=08,nvidia.com/cuda.runtime-version.full=12.4,nvidia.com/cuda.runtime-version.major=12,nvidia.com/cuda.runtime-version.minor=4,nvidia.com/cuda.runtime.major=12,nvidia.com/cuda.runtime.minor=4,nvidia.com/gfd.timestamp=1737098640,nvidia.com/gpu.compute.major=7,nvidia.com/gpu.compute.minor=5,nvidia.com/gpu.count=1,nvidia.com/gpu.family=turing,nvidia.com/gpu.machine=Standard-PC-i440FX-PIIX-1996,nvidia.com/gpu.memory=8192,nvidia.com/gpu.mode=graphics,nvidia.com/gpu.product=NVIDIA-T1000-8GB,nvidia.com/gpu.replicas=4,nvidia.com/gpu.sharing-strategy=time-slicing,nvidia.com/mig.capable=false,nvidia.com/mig.strategy=single,nvidia.com/mps.capable=false,nvidia.com/vgpu.present=false

The actual gpu-operator pods only run on the right nodes (mk8s1 and mk8s3):

ubuntu@mk8s4:~$ kubectl get pods -n gpu-operator -o wide
NAME                                                          READY   STATUS      RESTARTS      AGE    IP             NODE    NOMINATED NODE   READINESS GATES
gpu-feature-discovery-s2sqs                                   2/2     Running     0             34m    10.1.238.191   mk8s1   <none>           <none>
gpu-feature-discovery-z6bpm                                   2/2     Running     0             34m    10.1.217.196   mk8s3   <none>           <none>
gpu-operator-58dcc865fd-76x9l                                 1/1     Running     2 (77m ago)   3d9h   10.1.238.130   mk8s1   <none>           <none>
gpu-operator-node-feature-discovery-gc-7f546fd4bc-mxqsd       1/1     Running     0             34m    10.1.115.180   mk8s2   <none>           <none>
gpu-operator-node-feature-discovery-master-8448c8896c-pjdll   1/1     Running     0             34m    10.1.115.158   mk8s2   <none>           <none>
gpu-operator-node-feature-discovery-worker-49bmr              1/1     Running     0             34m    10.1.236.228   mk8s4   <none>           <none>
gpu-operator-node-feature-discovery-worker-8rb46              1/1     Running     0             34m    10.1.236.167   mk8s6   <none>           <none>
gpu-operator-node-feature-discovery-worker-9tb8x              1/1     Running     0             34m    10.1.238.156   mk8s1   <none>           <none>
gpu-operator-node-feature-discovery-worker-hcrsf              1/1     Running     0             34m    10.1.217.240   mk8s3   <none>           <none>
gpu-operator-node-feature-discovery-worker-kh8nf              1/1     Running     0             34m    10.1.115.170   mk8s2   <none>           <none>
gpu-operator-node-feature-discovery-worker-ndx7j              1/1     Running     0             34m    10.1.177.108   mk8s5   <none>           <none>
nvidia-container-toolkit-daemonset-k9fdh                      1/1     Running     0             34m    10.1.217.244   mk8s3   <none>           <none>
nvidia-container-toolkit-daemonset-qn2dr                      1/1     Running     0             34m    10.1.238.154   mk8s1   <none>           <none>
nvidia-cuda-validator-bxhsg                                   0/1     Completed   0             33m    10.1.238.150   mk8s1   <none>           <none>
nvidia-cuda-validator-k56p4                                   0/1     Completed   0             33m    10.1.217.212   mk8s3   <none>           <none>
nvidia-dcgm-exporter-kx44r                                    1/1     Running     0             34m    10.1.238.134   mk8s1   <none>           <none>
nvidia-dcgm-exporter-t55qr                                    1/1     Running     0             34m    10.1.217.255   mk8s3   <none>           <none>
nvidia-device-plugin-daemonset-9qw62                          2/2     Running     0             34m    10.1.217.223   mk8s3   <none>           <none>
nvidia-device-plugin-daemonset-vd748                          2/2     Running     0             34m    10.1.238.153   mk8s1   <none>           <none>
nvidia-node-status-exporter-7x2hm                             1/1     Running     0             34m    10.1.217.214   mk8s3   <none>           <none>
nvidia-node-status-exporter-hn2g6                             1/1     Running     0             34m    10.1.238.131   mk8s1   <none>           <none>
nvidia-operator-validator-f85dz                               1/1     Running     0             34m    10.1.238.176   mk8s1   <none>           <none>
nvidia-operator-validator-xsrfg                               1/1     Running     0             34m    10.1.217.234   mk8s3   <none>           <none>

Actually, there's no real effect on the functionality (only the right microk8s nodes get scheduled for the gpu, it seems), but I do not understand why the labels get attached to nodes without the gpu if they run on proxmox nodes with gpu, as if the nfd/gpu-operator see the host's gpu.

Thank you.

[chipzoller]

GFD will respond based upon the label added by NFD. NFD will label based on the discovered physical PCI devices. VMs typically surface the PCI devices even if one hasn't been explicitly passed through. This is probably why you're seeing that result.

[Johannesm299]

It should be possible to have the node-feature-discovery only label the gpu nodes, if you have it run only on the gpu nodes like so:

node-feature-discovery:
  worker:
    nodeSelector:
      label-that-describes: gpu-nodes