We are experiencing a peculiar issue with the new nvidia-drivers in our kubernetes setup on the NVIDIA A100-NVLINK nodes.
During provisioning our new NVIDIA H100-NVLINK nodes, we upgraded the following from 535.XXX to 550.XXX versions.
nvidia-open-gpu-kernel-modules
nvidia-container-toolkit
nvidia-fabricmanager
The H100s are working fine, but on the k8s node with A100-nvlink, CUDA initialization fails in the pod unless we set less than 6 devices in the CUDA_VISIBLE_DEVICES. The issue is persistent on 570.XXX versions as well.
Since our k8s is based on talos, we are using system extensions to install those drivers.
The issue is not visible when testing the A100-nvlink node as a baremetal ubuntu server and above new drivers and docker containers.
root@a100-nvlink-host:/workspace# nvidia-smi
Mon Jun 16 14:40:24 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.148.08 Driver Version: 570.148.08 CUDA Version: 12.8 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA A100-SXM4-80GB On | 00000000:07:00.0 Off | 0 |
| N/A 34C P0 64W / 400W | 0MiB / 81920MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+
| 1 NVIDIA A100-SXM4-80GB On | 00000000:0A:00.0 Off | 0 |
| N/A 32C P0 66W / 400W | 0MiB / 81920MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+
| 2 NVIDIA A100-SXM4-80GB On | 00000000:45:00.0 Off | 0 |
| N/A 32C P0 63W / 400W | 0MiB / 81920MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+
| 3 NVIDIA A100-SXM4-80GB On | 00000000:4B:00.0 Off | 0 |
| N/A 34C P0 65W / 400W | 0MiB / 81920MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+
| 4 NVIDIA A100-SXM4-80GB On | 00000000:83:00.0 Off | 0 |
| N/A 33C P0 66W / 400W | 0MiB / 81920MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+
| 5 NVIDIA A100-SXM4-80GB On | 00000000:89:00.0 Off | 0 |
| N/A 32C P0 64W / 400W | 0MiB / 81920MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+
| 6 NVIDIA A100-SXM4-80GB On | 00000000:C0:00.0 Off | 0 |
| N/A 32C P0 64W / 400W | 0MiB / 81920MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+
| 7 NVIDIA A100-SXM4-80GB On | 00000000:C3:00.0 Off | 0 |
| N/A 34C P0 66W / 400W | 0MiB / 81920MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
+-----------------------------------------------------------------------------------------+
root@a100-nvlink-host:/workspace#
root@a100-nvlink-host:/workspace#
root@a100-nvlink-host:/workspace# export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
root@a100-nvlink-host:/workspace#
root@a100-nvlink-host:/workspace# python -c "import torch; [print(f'cuda:{i}: {torch.cuda.get_device_name(i)}') for i in range(torch.cuda.device_count())]"
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "<string>", line 1, in <listcomp>
File "/usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py", line 419, in get_device_name
return get_device_properties(device).name
File "/usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py", line 449, in get_device_properties
_lazy_init() # will define _get_device_properties
File "/usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py", line 298, in _lazy_init
torch._C._cuda_init()
RuntimeError: CUDA driver initialization failed, you might not have a CUDA gpu.
root@a100-nvlink-host:/workspace#
root@a100-nvlink-host:/workspace# export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5
root@a100-nvlink-host:/workspace#
root@a100-nvlink-host:/workspace# python -c "import torch; [print(f'cuda:{i}: {torch.cuda.get_device_name(i)}') for i in range(torch.cuda.device_count())]"
cuda:0: NVIDIA A100-SXM4-80GB
cuda:1: NVIDIA A100-SXM4-80GB
cuda:2: NVIDIA A100-SXM4-80GB
cuda:3: NVIDIA A100-SXM4-80GB
cuda:4: NVIDIA A100-SXM4-80GB
cuda:5: NVIDIA A100-SXM4-80GB
root@a100-nvlink-host:/workspace#
Any ideas what is causing it?