DL380 Gen12 nvidia-smi cannot recognize L40s but exists in lspci; CUDA 12.9, driver version 570, 575, 580

nvidia-bug-report.log_570.gz (1.4 MB)

nvidia-bug-report.log_575.gz (817.7 KB)

nvidia-drier570-575-580-log.txt (11.9 KB)

hello,

Basically, one L40S.
- Add-on L40S GPU. Due to a specific issue, the device appeared in nvidia-smi. The device was recognized in iLO and BIOS/lspci. Ubuntu 24.04.
- After removing the existing L40S, the additional L40S GPU was recognized.
After connecting only the added GPU (removing the initial GPU), the device appeared to be working normally. Therefore, the device connection appears to have been successful.
sudo vim /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="nouveau.modeset=0 pci=realloc=off"

sudo update-grub
sudo update-initramfs -u
sudo reboot now

I tried applying this command, but it didn't resolve the issue where only one nvidia-smi entry was visible.

root@jbgpu:~# dmesg | grep nvidia
[ 5.089419] nvidia: loading out-of-tree module taints kernel.
[ 5.089427] nvidia: module license ‘NVIDIA’ taints kernel.
[ 5.089431] nvidia: module verification failed: signature and/or required key missing - tainting kernel
[ 5.089432] nvidia: module license taints kernel.
[ 5.206778] nvidia-nvlink: Nvlink Core is being initialized, major device number 509
[ 5.212301] nvidia 0000:0b:00.0: enabling device (0140 → 0142)
[ 5.271798] nvidia 0001:5c:00.0: enabling device (0140 → 0142)
[ 5.343616] nvidia-modeset: Loading NVIDIA Kernel Mode Setting Driver for UNIX platforms 575.57.08 Sat May 24 06:52:56 UTC 2025
[ 5.364510] [drm] [nvidia-drm] [GPU ID 0x00000b00] Loading driver
[ 5.364514] [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:0b:00.0 on minor 1
[ 5.364610] [drm] [nvidia-drm] [GPU ID 0x00015c00] Loading driver
[ 5.364611] [drm] Initialized nvidia-drm 0.0.0 20160202 for 0001:5c:00.0 on minor 2
[ 63.921323] caller os_map_kernel_space+0x120/0x130 [nvidia] mapping multiple BARs
[ 64.153246] nvidia_uvm: module uses symbols nvUvmInterfaceDisableAccessCntr from proprietary module nvidia, inheriting taint.

root@ubuntu :~# nvidia-smi
Fri Dec 26 13:56:46 2025
±----------------------------------------------------------------------------------------+
| NVIDIA-SMI 575.57.08 Driver Version: 575.57.08 CUDA Version: 12.9 |
|-----------------------------------------±-----------------------±---------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA L40S Off | 00000001:5C:00.0 Off | 0 |
| N/A 34C P8 24W / 350W | 0MiB / 46068MiB | 0% Default |
| | | N/A |
±----------------------------------------±-----------------------±---------------------+

±----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
±----------------------------------------------------------------------------------------+

root@gpu:~$ lspci | grep -i nvidia
0000:0b:00.0 3D controller: NVIDIA Corporation AD102GL [L40S] (rev a1)
0001:5c:00.0 3D controller: NVIDIA Corporation AD102GL [L40S] (rev a1)