A100 not recognized by nvidia-smi but recognized by lspci and working with Windows correctly

Graphics / Linux Linux
1

Hello,

A100 not recognized by nvidia-smi but recognized by lspci and working with Windows correctly:

lspci | grep -i nvidia
12:00.0 3D controller: NVIDIA Corporation GA100 [A100 PCIe 40GB] (rev a1)
86:00.0 3D controller: NVIDIA Corporation GA100 [A100 PCIe 40GB] (rev a1)

nvdia-smi:

nvidia-smi
Thu Sep 18 08:28:10 2025
±----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.172.08 Driver Version: 570.172.08 CUDA Version: 12.8 |
|-----------------------------------------±-----------------------±---------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA A100-PCIE-40GB Off | 00000000:86:00.0 Off | Off |
| N/A 36C P0 31W / 250W | 0MiB / 40960MiB | 0% Default |
| | | Disabled |
±----------------------------------------±-----------------------±---------------------+

±----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
±----------------------------------------------------------------------------------------+

12:00.0 missing from nvidia-smi.

dmesg | grep nvidia :

[ 191.956094] nvidia: loading out-of-tree module taints kernel.
[ 191.956102] nvidia: module license ‘NVIDIA’ taints kernel.
[ 191.956106] nvidia: module license taints kernel.
[ 192.095293] nvidia-nvlink: Nvlink Core is being initialized, major device number 235
[ 192.097929] nvidia 0000:12:00.0: enabling device (0140 → 0142)
[ 192.155744] nvidia 0000:86:00.0: enabling device (0140 → 0142)
[ 192.272334] nvidia-modeset: Loading NVIDIA Kernel Mode Setting Driver for UNIX platforms 570.172.08 Tue Jul 8 17:57:10 UTC 2025
[ 192.486430] [drm] [nvidia-drm] [GPU ID 0x00001200] Loading driver
[ 199.667726] os_dump_stack+0xe/0x20 [nvidia]
[ 199.668443] _nv013205rm+0x508/0x5b0 [nvidia]
[ 199.669122] 0000000036bad3cc: ffffffffc1bf6b98 (_nv013205rm+0x508/0x5b0 [nvidia])
[ 199.669768] 00000000fce75b4c: ffffffffc152668e (os_dump_stack+0xe/0x20 [nvidia])
[ 199.670103] 00000000c44a5708: ffffffffc1bf6b98 (_nv013205rm+0x508/0x5b0 [nvidia])
[ 199.670732] 000000002227acb6: ffffffffc1bf4b8c (_nv039286rm+0x2c/0xe6 [nvidia])
[ 199.671354] 00000000aa06082a: ffffffffc1bf9b4d (_nv039095rm+0x19d/0x375 [nvidia])
[ 199.671974] 000000007c4c192c: ffffffffc1bf4201 (_nv039218rm+0xc01/0x1560 [nvidia])
[ 199.672595] 0000000065af40fc: ffffffffc21e4ff9 (_nv000766rm+0x1369/0x20f0 [nvidia])
[ 199.672997] 00000000d1be629d: ffffffffc21db1dd (rm_init_adapter+0xcd/0xf0 [nvidia])
[ 199.673437] 00000000fed98d38: ffffffffc2474b30 (_nv049941rm+0x90/0xffffffffffe76560 [nvidia])
[ 199.673811] 0000000009a8d2b2: ffffffffc1512929 (nv_open_device+0x209/0xa60 [nvidia])
[ 199.674149] 00000000bc77bba3: ffffffffc1515fe8 (nvidia_dev_get+0x68/0xd0 [nvidia])
[ 199.674478] 00000000c4185d94: ffffffffc14f39c0 (nv_linux_drm_exit+0x1ce0/0x320 [nvidia_drm])
[ 199.674494] 00000000f5acb20e: ffffffffc6b38e4f (nvkms_open_gpu+0x4f/0x90 [nvidia_modeset])
[ 199.674546] 00000000825a8797: ffffffffc6c0171d (_nv000013kms+0x4d/0x700 [nvidia_modeset])
[ 199.674582] 00000000c83db2ee: ffffffffc14f3c80 (nv_drm_suspend_mutex.1+0x20/0xffffffffffffe3a0 [nvidia_drm])
[ 199.674615] 000000002de45037: ffffffffc14f39c0 (nv_linux_drm_exit+0x1ce0/0x320 [nvidia_drm])
[ 199.674624] 0000000037ac23ed: ffffffffc14e2902 (nv_drm_load+0x102/0x920 [nvidia_drm])
[ 199.674646] 000000009e758ee0: ffffffffc14e25d0 (__pfx_nv_drm_event_callback+0x10/0x10 [nvidia_drm])
[ 199.674680] 0000000013661f24: ffffffffc14f3c80 (nv_drm_suspend_mutex.1+0x20/0xffffffffffffe3a0 [nvidia_drm])
[ 199.674707] 000000003de7a7c1: ffffffffc14e4407 (nv_drm_register_drm_device+0xa7/0x1c0 [nvidia_drm])
[ 199.674723] 00000000ba4dfb10: ffffffffc14e45d2 (nv_drm_probe_devices+0xa2/0xf0 [nvidia_drm])
[ 199.674729] 00000000b579f25e: ffffffffc1303010 (__pfx_nv_linux_drm_init+0x10/0x10 [nvidia_drm])
[ 199.674738] 0000000029759f79: ffffffffc14e2041 (nv_drm_init+0x31/0x70 [nvidia_drm])
[ 199.674745] 00000000beb3bb5e: ffffffffc130301e (nv_linux_drm_init+0xe/0xff0 [nvidia_drm])
[ 199.674759] 00000000b75b66b7: ffffffffc14f5040 (__UNIQUE_ID___addressable_cleanup_module259+0x38/0xffffffffffffcff8 [nvidia_drm])
[ 199.674784] 00000000d1803f2a: ffffffffc14f5040 (__UNIQUE_ID___addressable_cleanup_module259+0x38/0xffffffffffffcff8 [nvidia_drm])
[ 199.674791] 000000005ed8383b: ffffffffc14f5040 (__UNIQUE_ID___addressable_cleanup_module259+0x38/0xffffffffffffcff8 [nvidia_drm])
[ 199.674796] 00000000714a19d3: ffffffffc14f5180 (__this_module+0x140/0xffffffffffffcfc0 [nvidia_drm])
[ 199.674802] 00000000557e7a93: ffffffffc14f5040 (__UNIQUE_ID___addressable_cleanup_module259+0x38/0xffffffffffffcff8 [nvidia_drm])
[ 199.675111] ? _nv039286rm+0x2c/0xe6 [nvidia]
[ 199.675757] ? _nv039095rm+0x19d/0x375 [nvidia]
[ 199.676375] ? _nv039218rm+0xc01/0x1560 [nvidia]
[ 199.676988] ? _nv000766rm+0x1369/0x20f0 [nvidia]
[ 199.677402] ? rm_init_adapter+0xcd/0xf0 [nvidia]
[ 199.677836] ? nv_open_device+0x209/0xa60 [nvidia]
[ 199.678156] ? nvidia_dev_get+0x68/0xd0 [nvidia]
[ 199.678479] ? nvkms_open_gpu+0x4f/0x90 [nvidia_modeset]
[ 199.678515] ? _nv000013kms+0x4d/0x700 [nvidia_modeset]
[ 199.678555] ? nv_drm_load+0x102/0x920 [nvidia_drm]
[ 199.678570] ? __pfx_nv_drm_event_callback+0x10/0x10 [nvidia_drm]
[ 199.678588] ? nv_drm_register_drm_device+0xa7/0x1c0 [nvidia_drm]
[ 199.678594] ? nv_drm_probe_devices+0xa2/0xf0 [nvidia_drm]
[ 199.678600] ? __pfx_nv_linux_drm_init+0x10/0x10 [nvidia_drm]
[ 199.678606] ? nv_drm_init+0x31/0x70 [nvidia_drm]
[ 199.678611] ? nv_linux_drm_init+0xe/0xff0 [nvidia_drm]
[ 199.680783] [drm:nv_drm_load [nvidia_drm]] ERROR [nvidia-drm] [GPU ID 0x00001200] Failed to allocate NvKmsKapiDevice
[ 199.680909] [drm:nv_drm_register_drm_device [nvidia_drm]] ERROR [nvidia-drm] [GPU ID 0x00001200] Failed to register device
[ 199.681051] [drm] [nvidia-drm] [GPU ID 0x00008600] Loading driver
[ 200.823734] [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:86:00.0 on minor 0
[ 200.823758] nvidia 0000:86:00.0: [drm] No compatible format found
[ 200.823760] nvidia 0000:86:00.0: [drm] Cannot find any crtc or sizes
[ 201.680559] nvidia_uvm: module uses symbols nvUvmInterfaceDisableAccessCntr from proprietary module nvidia, inheriting taint.

2

The two GPU was swapped.

Input:

journalctl -b | grep -i “drm|nvidia|NVRM|12:00.0”

Result:

Oct 21 06:19:32 computer kernel: NVRM: _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
Oct 21 06:19:32 computer kernel: NVRM: _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
Oct 21 06:19:32 computer kernel: NVRM: crashcatWayfinderGetReportQueue_V1: insufficiently-sized L1 wayfinder scratch location 0
Oct 21 06:19:32 computer kernel: NVRM: RmInitAdapter: Cannot initialize GSP firmware RM
Oct 21 06:19:32 computer kernel: NVRM: GPU 0000:12:00.0: RmInitAdapter failed! (0x62:0x40:2015)
Oct 21 06:19:32 computer kernel: NVRM: GPU 0000:12:00.0: rm_init_adapter failed, device minor number 0
Oct 21 06:19:32 computer kernel: NVRM: _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
Oct 21 06:19:32 computer kernel: NVRM: _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
Oct 21 06:19:32 computer kernel: NVRM: crashcatWayfinderGetReportQueue_V1: insufficiently-sized L1 wayfinder scratch location 0
Oct 21 06:19:32 computer kernel: NVRM: RmInitAdapter: Cannot initialize GSP firmware RM
Oct 21 06:19:32 computer kernel: NVRM: GPU 0000:12:00.0: RmInitAdapter failed! (0x62:0x40:2015)
Oct 21 06:19:32 computer kernel: NVRM: GPU 0000:12:00.0: rm_init_adapter failed, device minor number 0
Oct 21 06:19:32 computer kernel: NVRM: _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
Oct 21 06:19:32 computer kernel: NVRM: _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
Oct 21 06:19:32 computer kernel: NVRM: crashcatWayfinderGetReportQueue_V1: insufficiently-sized L1 wayfinder scratch location 0
Oct 21 06:19:32 computer kernel: NVRM: RmInitAdapter: Cannot initialize GSP firmware RM
Oct 21 06:19:32 computer kernel: NVRM: GPU 0000:12:00.0: RmInitAdapter failed! (0x62:0x40:2015)
Oct 21 06:19:32 computer kernel: NVRM: GPU 0000:12:00.0: rm_init_adapter failed, device minor number 0
Oct 21 06:19:32 computer kernel: NVRM: _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
Oct 21 06:19:32 computer kernel: NVRM: _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
Oct 21 06:19:32 computer kernel: NVRM: crashcatWayfinderGetReportQueue_V1: insufficiently-sized L1 wayfinder scratch location 0
Oct 21 06:19:32 computer kernel: NVRM: RmInitAdapter: Cannot initialize GSP firmware RM
Oct 21 06:19:32 computer kernel: NVRM: GPU 0000:12:00.0: RmInitAdapter failed! (0x62:0x40:2015)
Oct 21 06:19:32 computer kernel: NVRM: GPU 0000:12:00.0: rm_init_adapter failed, device minor number 0
Oct 21 06:19:33 computer kernel: NVRM: _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
Oct 21 06:19:33 computer kernel: NVRM: _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
Oct 21 06:19:33 computer kernel: NVRM: crashcatWayfinderGetReportQueue_V1: insufficiently-sized L1 wayfinder scratch location 0
Oct 21 06:19:33 computer kernel: NVRM: RmInitAdapter: Cannot initialize GSP firmware RM
Oct 21 06:19:33 computer kernel: NVRM: GPU 0000:12:00.0: RmInitAdapter failed! (0x62:0x40:2015)
Oct 21 06:19:33 computer kernel: NVRM: GPU 0000:12:00.0: rm_init_adapter failed, device minor number 0
Oct 21 06:19:33 computer kernel: NVRM: _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
Oct 21 06:19:33 computer kernel: NVRM: _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
Oct 21 06:19:33 computer kernel: NVRM: crashcatWayfinderGetReportQueue_V1: insufficiently-sized L1 wayfinder scratch location 0
Oct 21 06:19:33 computer kernel: NVRM: RmInitAdapter: Cannot initialize GSP firmware RM
Oct 21 06:19:33 computer kernel: NVRM: GPU 0000:12:00.0: RmInitAdapter failed! (0x62:0x40:2015)
Oct 21 06:19:33 computer kernel: NVRM: GPU 0000:12:00.0: rm_init_adapter failed, device minor number 0
Oct 21 06:19:33 computer kernel: NVRM: _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
Oct 21 06:19:33 computer kernel: NVRM: _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
Oct 21 06:19:33 computer kernel: NVRM: crashcatWayfinderGetReportQueue_V1: insufficiently-sized L1 wayfinder scratch location 0
Oct 21 06:19:33 computer kernel: NVRM: RmInitAdapter: Cannot initialize GSP firmware RM
Oct 21 06:19:33 computer kernel: NVRM: GPU 0000:12:00.0: RmInitAdapter failed! (0x62:0x40:2015)
Oct 21 06:19:33 computer kernel: NVRM: GPU 0000:12:00.0: rm_init_adapter failed, device minor number 0
Oct 21 06:19:34 computer kernel: NVRM: _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
Oct 21 06:19:34 computer kernel: NVRM: _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
Oct 21 06:19:34 computer kernel: NVRM: crashcatWayfinderGetReportQueue_V1: insufficiently-sized L1 wayfinder scratch location 0
Oct 21 06:19:34 computer kernel: NVRM: RmInitAdapter: Cannot initialize GSP firmware RM
Oct 21 06:19:34 computer kernel: NVRM: GPU 0000:12:00.0: RmInitAdapter failed! (0x62:0x40:2015)
Oct 21 06:19:34 computer kernel: NVRM: GPU 0000:12:00.0: rm_init_adapter failed, device minor number 0
Oct 21 06:19:34 computer kernel: NVRM: _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
Oct 21 06:19:34 computer kernel: NVRM: _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
Oct 21 06:19:34 computer kernel: NVRM: crashcatWayfinderGetReportQueue_V1: insufficiently-sized L1 wayfinder scratch location 0
Oct 21 06:19:34 computer kernel: NVRM: RmInitAdapter: Cannot initialize GSP firmware RM
Oct 21 06:19:34 computer kernel: NVRM: GPU 0000:12:00.0: RmInitAdapter failed! (0x62:0x40:2015)
Oct 21 06:19:34 computer kernel: NVRM: GPU 0000:12:00.0: rm_init_adapter failed, device minor number 0
Oct 21 06:19:34 computer kernel: NVRM: _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
Oct 21 06:19:34 computer kernel: NVRM: _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
Oct 21 06:19:34 computer kernel: NVRM: crashcatWayfinderGetReportQueue_V1: insufficiently-sized L1 wayfinder scratch location 0
Oct 21 06:19:34 computer kernel: NVRM: RmInitAdapter: Cannot initialize GSP firmware RM
Oct 21 06:19:34 computer kernel: NVRM: GPU 0000:12:00.0: RmInitAdapter failed! (0x62:0x40:2015)
Oct 21 06:19:34 computer kernel: NVRM: GPU 0000:12:00.0: rm_init_adapter failed, device minor number 0
Oct 21 06:19:36 computer kernel: NVRM: _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
Oct 21 06:19:36 computer kernel: NVRM: _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
Oct 21 06:19:36 computer kernel: NVRM: crashcatWayfinderGetReportQueue_V1: insufficiently-sized L1 wayfinder scratch location 0
Oct 21 06:19:36 computer kernel: NVRM: RmInitAdapter: Cannot initialize GSP firmware RM
Oct 21 06:19:36 computer kernel: NVRM: GPU 0000:12:00.0: RmInitAdapter failed! (0x62:0x40:2015)
Oct 21 06:19:36 computer kernel: NVRM: GPU 0000:12:00.0: rm_init_adapter failed, device minor number 0
Oct 21 06:19:36 computer kernel: NVRM: _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
Oct 21 06:19:36 computer kernel: NVRM: _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
Oct 21 06:19:36 computer kernel: NVRM: crashcatWayfinderGetReportQueue_V1: insufficiently-sized L1 wayfinder scratch location 0
Oct 21 06:19:36 computer kernel: NVRM: RmInitAdapter: Cannot initialize GSP firmware RM
Oct 21 06:19:36 computer kernel: NVRM: GPU 0000:12:00.0: RmInitAdapter failed! (0x62:0x40:2015)
Oct 21 06:19:36 computer kernel: NVRM: GPU 0000:12:00.0: rm_init_adapter failed, device minor number 0
Oct 21 06:19:37 computer kernel: NVRM: _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
Oct 21 06:19:37 computer kernel: NVRM: _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
Oct 21 06:19:37 computer kernel: NVRM: crashcatWayfinderGetReportQueue_V1: insufficiently-sized L1 wayfinder scratch location 0
Oct 21 06:19:37 computer kernel: NVRM: RmInitAdapter: Cannot initialize GSP firmware RM
Oct 21 06:19:37 computer kernel: NVRM: GPU 0000:12:00.0: RmInitAdapter failed! (0x62:0x40:2015)
Oct 21 06:19:37 computer kernel: NVRM: GPU 0000:12:00.0: rm_init_adapter failed, device minor number 0
Oct 21 06:19:38 computer kernel: NVRM: _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
Oct 21 06:19:38 computer kernel: NVRM: _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
Oct 21 06:19:38 computer kernel: NVRM: crashcatWayfinderGetReportQueue_V1: insufficiently-sized L1 wayfinder scratch location 0
Oct 21 06:19:38 computer kernel: NVRM: RmInitAdapter: Cannot initialize GSP firmware RM
Oct 21 06:19:38 computer kernel: NVRM: GPU 0000:12:00.0: RmInitAdapter failed! (0x62:0x40:2015)
Oct 21 06:19:38 computer kernel: NVRM: GPU 0000:12:00.0: rm_init_adapter failed, device minor number 0
Oct 21 06:19:39 computer kernel: NVRM: _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
Oct 21 06:19:39 computer kernel: NVRM: _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
Oct 21 06:19:39 computer kernel: NVRM: crashcatWayfinderGetReportQueue_V1: insufficiently-sized L1 wayfinder scratch location 0
Oct 21 06:19:39 computer kernel: NVRM: RmInitAdapter: Cannot initialize GSP firmware RM
Oct 21 06:19:39 computer kernel: NVRM: GPU 0000:12:00.0: RmInitAdapter failed! (0x62:0x40:2015)
Oct 21 06:19:39 computer kernel: NVRM: GPU 0000:12:00.0: rm_init_adapter failed, device minor number 0
Oct 21 06:19:40 computer kernel: NVRM: _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
Oct 21 06:19:40 computer kernel: NVRM: _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
Oct 21 06:19:40 computer kernel: NVRM: crashcatWayfinderGetReportQueue_V1: insufficiently-sized L1 wayfinder scratch location 0
Oct 21 06:19:40 computer kernel: NVRM: RmInitAdapter: Cannot initialize GSP firmware RM
Oct 21 06:19:40 computer kernel: NVRM: GPU 0000:12:00.0: RmInitAdapter failed! (0x62:0x40:2015)
Oct 21 06:19:40 computer kernel: NVRM: GPU 0000:12:00.0: rm_init_adapter failed, device minor number 0
Oct 21 06:19:40 computer k3s[3628]: I1021 06:19:40.959772 3628 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume “kube-api-access-ttjcn” (UniqueName: “kubernetes.io/projected/b14d8144-902a-4eac-9ef3-d51fbead7a97-kube-api-access-ttjcn”) pod “nvidia-cuda-validator-lrkrc” (UID: “b14d8144-902a-4eac-9ef3-d51fbead7a97”) " pod=“gpu-operator/nvidia-cuda-validator-lrkrc”
Oct 21 06:19:41 computer kernel: NVRM: _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
Oct 21 06:19:41 computer kernel: NVRM: _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
Oct 21 06:19:41 computer kernel: NVRM: crashcatWayfinderGetReportQueue_V1: insufficiently-sized L1 wayfinder scratch location 0
Oct 21 06:19:41 computer kernel: NVRM: RmInitAdapter: Cannot initialize GSP firmware RM
Oct 21 06:19:41 computer kernel: NVRM: GPU 0000:12:00.0: RmInitAdapter failed! (0x62:0x40:2015)
Oct 21 06:19:41 computer kernel: NVRM: GPU 0000:12:00.0: rm_init_adapter failed, device minor number 0
Oct 21 06:19:41 computer kernel: NVRM: _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
Oct 21 06:19:41 computer kernel: NVRM: _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
Oct 21 06:19:41 computer kernel: NVRM: crashcatWayfinderGetReportQueue_V1: insufficiently-sized L1 wayfinder scratch location 0
Oct 21 06:19:41 computer kernel: NVRM: RmInitAdapter: Cannot initialize GSP firmware RM
Oct 21 06:19:41 computer kernel: NVRM: GPU 0000:12:00.0: RmInitAdapter failed! (0x62:0x40:2015)
Oct 21 06:19:41 computer kernel: NVRM: GPU 0000:12:00.0: rm_init_adapter failed, device minor number 0
Oct 21 06:19:41 computer kernel: NVRM: _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
Oct 21 06:19:41 computer kernel: NVRM: _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
Oct 21 06:19:41 computer kernel: NVRM: crashcatWayfinderGetReportQueue_V1: insufficiently-sized L1 wayfinder scratch location 0
Oct 21 06:19:41 computer kernel: NVRM: RmInitAdapter: Cannot initialize GSP firmware RM
Oct 21 06:19:41 computer kernel: NVRM: GPU 0000:12:00.0: RmInitAdapter failed! (0x62:0x40:2015)
Oct 21 06:19:41 computer kernel: NVRM: GPU 0000:12:00.0: rm_init_adapter failed, device minor number 0
Oct 21 06:19:41 computer kernel: NVRM: _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
Oct 21 06:19:41 computer kernel: NVRM: _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
Oct 21 06:19:41 computer kernel: NVRM: crashcatWayfinderGetReportQueue_V1: insufficiently-sized L1 wayfinder scratch location 0
Oct 21 06:19:41 computer kernel: NVRM: RmInitAdapter: Cannot initialize GSP firmware RM
Oct 21 06:19:41 computer kernel: NVRM: GPU 0000:12:00.0: RmInitAdapter failed! (0x62:0x40:2015)
Oct 21 06:19:41 computer kernel: NVRM: GPU 0000:12:00.0: rm_init_adapter failed, device minor number 0
Oct 21 06:19:42 computer kernel: NVRM: _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
Oct 21 06:19:42 computer kernel: NVRM: _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
Oct 21 06:19:42 computer kernel: NVRM: crashcatWayfinderGetReportQueue_V1: insufficiently-sized L1 wayfinder scratch location 0
Oct 21 06:19:42 computer kernel: NVRM: RmInitAdapter: Cannot initialize GSP firmware RM
Oct 21 06:19:42 computer kernel: NVRM: GPU 0000:12:00.0: RmInitAdapter failed! (0x62:0x40:2015)
Oct 21 06:19:42 computer kernel: NVRM: GPU 0000:12:00.0: rm_init_adapter failed, device minor number 0
Oct 21 06:19:42 computer kernel: NVRM: _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
Oct 21 06:19:42 computer kernel: NVRM: _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
Oct 21 06:19:42 computer kernel: NVRM: crashcatWayfinderGetReportQueue_V1: insufficiently-sized L1 wayfinder scratch location 0
Oct 21 06:19:42 computer kernel: NVRM: RmInitAdapter: Cannot initialize GSP firmware RM
Oct 21 06:19:42 computer kernel: NVRM: GPU 0000:12:00.0: RmInitAdapter failed! (0x62:0x40:2015)
Oct 21 06:19:42 computer kernel: NVRM: GPU 0000:12:00.0: rm_init_adapter failed, device minor number 0
Oct 21 06:19:42 computer kernel: NVRM: _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
Oct 21 06:19:42 computer kernel: NVRM: _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
Oct 21 06:19:42 computer kernel: NVRM: crashcatWayfinderGetReportQueue_V1: insufficiently-sized L1 wayfinder scratch location 0
Oct 21 06:19:42 computer kernel: NVRM: RmInitAdapter: Cannot initialize GSP firmware RM
Oct 21 06:19:42 computer kernel: NVRM: GPU 0000:12:00.0: RmInitAdapter failed! (0x62:0x40:2015)
Oct 21 06:19:42 computer kernel: NVRM: GPU 0000:12:00.0: rm_init_adapter failed, device minor number 0
Oct 21 06:19:42 computer kernel: NVRM: _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
Oct 21 06:19:42 computer kernel: NVRM: _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
Oct 21 06:19:42 computer kernel: NVRM: crashcatWayfinderGetReportQueue_V1: insufficiently-sized L1 wayfinder scratch location 0
Oct 21 06:19:42 computer kernel: NVRM: RmInitAdapter: Cannot initialize GSP firmware RM
Oct 21 06:19:42 computer kernel: NVRM: GPU 0000:12:00.0: RmInitAdapter failed! (0x62:0x40:2015)
Oct 21 06:19:42 computer kernel: NVRM: GPU 0000:12:00.0: rm_init_adapter failed, device minor number 0
Oct 21 06:19:42 computer kernel: NVRM: _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
Oct 21 06:19:42 computer kernel: NVRM: _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
Oct 21 06:19:42 computer kernel: NVRM: crashcatWayfinderGetReportQueue_V1: insufficiently-sized L1 wayfinder scratch location 0
Oct 21 06:19:42 computer kernel: NVRM: RmInitAdapter: Cannot initialize GSP firmware RM
Oct 21 06:19:42 computer kernel: NVRM: GPU 0000:12:00.0: RmInitAdapter failed! (0x62:0x40:2015)
Oct 21 06:19:42 computer kernel: NVRM: GPU 0000:12:00.0: rm_init_adapter failed, device minor number 0
Oct 21 06:19:42 computer kernel: NVRM: _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
Oct 21 06:19:42 computer kernel: NVRM: _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
Oct 21 06:19:42 computer kernel: NVRM: crashcatWayfinderGetReportQueue_V1: insufficiently-sized L1 wayfinder scratch location 0
Oct 21 06:19:42 computer kernel: NVRM: RmInitAdapter: Cannot initialize GSP firmware RM
Oct 21 06:19:42 computer kernel: NVRM: GPU 0000:12:00.0: RmInitAdapter failed! (0x62:0x40:2015)
Oct 21 06:19:42 computer kernel: NVRM: GPU 0000:12:00.0: rm_init_adapter failed, device minor number 0
Oct 21 06:19:43 computer kernel: NVRM: _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
Oct 21 06:19:43 computer kernel: NVRM: _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
Oct 21 06:19:43 computer kernel: NVRM: crashcatWayfinderGetReportQueue_V1: insufficiently-sized L1 wayfinder scratch location 0
Oct 21 06:19:43 computer kernel: NVRM: RmInitAdapter: Cannot initialize GSP firmware RM
Oct 21 06:19:43 computer kernel: NVRM: GPU 0000:12:00.0: RmInitAdapter failed! (0x62:0x40:2015)
Oct 21 06:19:43 computer kernel: NVRM: GPU 0000:12:00.0: rm_init_adapter failed, device minor number 0
Oct 21 06:19:43 computer kernel: NVRM: _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
Oct 21 06:19:43 computer kernel: NVRM: _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
Oct 21 06:19:43 computer kernel: NVRM: crashcatWayfinderGetReportQueue_V1: insufficiently-sized L1 wayfinder scratch location 0
Oct 21 06:19:43 computer kernel: NVRM: RmInitAdapter: Cannot initialize GSP firmware RM
Oct 21 06:19:43 computer kernel: NVRM: GPU 0000:12:00.0: RmInitAdapter failed! (0x62:0x40:2015)
Oct 21 06:19:43 computer kernel: NVRM: GPU 0000:12:00.0: rm_init_adapter failed, device minor number 0
Oct 21 06:19:43 computer kernel: NVRM: _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
Oct 21 06:19:43 computer kernel: NVRM: _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
Oct 21 06:19:43 computer kernel: NVRM: crashcatWayfinderGetReportQueue_V1: insufficiently-sized L1 wayfinder scratch location 0
Oct 21 06:19:43 computer kernel: NVRM: RmInitAdapter: Cannot initialize GSP firmware RM
Oct 21 06:19:43 computer kernel: NVRM: GPU 0000:12:00.0: RmInitAdapter failed! (0x62:0x40:2015)
Oct 21 06:19:43 computer kernel: NVRM: GPU 0000:12:00.0: rm_init_adapter failed, device minor number 0
Oct 21 06:19:45 computer kernel: NVRM: _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
Oct 21 06:19:45 computer kernel: NVRM: _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
Oct 21 06:19:45 computer kernel: NVRM: crashcatWayfinderGetReportQueue_V1: insufficiently-sized L1 wayfinder scratch location 0
Oct 21 06:19:45 computer kernel: NVRM: RmInitAdapter: Cannot initialize GSP firmware RM
Oct 21 06:19:45 computer kernel: NVRM: GPU 0000:12:00.0: RmInitAdapter failed! (0x62:0x40:2015)
Oct 21 06:19:45 computer kernel: NVRM: GPU 0000:12:00.0: rm_init_adapter failed, device minor number 0
Oct 21 06:19:46 computer kernel: NVRM: _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
Oct 21 06:19:46 computer kernel: NVRM: _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
Oct 21 06:19:46 computer kernel: NVRM: crashcatWayfinderGetReportQueue_V1: insufficiently-sized L1 wayfinder scratch location 0
Oct 21 06:19:46 computer kernel: NVRM: RmInitAdapter: Cannot initialize GSP firmware RM
Oct 21 06:19:46 computer kernel: NVRM: GPU 0000:12:00.0: RmInitAdapter failed! (0x62:0x40:2015)
Oct 21 06:19:46 computer kernel: NVRM: GPU 0000:12:00.0: rm_init_adapter failed, device minor number 0
Oct 21 06:19:46 computer kernel: NVRM: _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
Oct 21 06:19:46 computer kernel: NVRM: _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
Oct 21 06:19:46 computer kernel: NVRM: crashcatWayfinderGetReportQueue_V1: insufficiently-sized L1 wayfinder scratch location 0
Oct 21 06:19:46 computer kernel: NVRM: RmInitAdapter: Cannot initialize GSP firmware RM
Oct 21 06:19:46 computer kernel: NVRM: GPU 0000:12:00.0: RmInitAdapter failed! (0x62:0x40:2015)
Oct 21 06:19:46 computer kernel: NVRM: GPU 0000:12:00.0: rm_init_adapter failed, device minor number 0
Oct 21 06:19:47 computer kernel: NVRM: _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
Oct 21 06:19:47 computer kernel: NVRM: _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
Oct 21 06:19:47 computer kernel: NVRM: crashcatWayfinderGetReportQueue_V1: insufficiently-sized L1 wayfinder scratch location 0
Oct 21 06:19:47 computer kernel: NVRM: RmInitAdapter: Cannot initialize GSP firmware RM
Oct 21 06:19:47 computer kernel: NVRM: GPU 0000:12:00.0: RmInitAdapter failed! (0x62:0x40:2015)
Oct 21 06:19:47 computer kernel: NVRM: GPU 0000:12:00.0: rm_init_adapter failed, device minor number 0
Oct 21 06:19:47 computer kernel: NVRM: _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
Oct 21 06:19:47 computer kernel: NVRM: _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
Oct 21 06:19:47 computer kernel: NVRM: crashcatWayfinderGetReportQueue_V1: insufficiently-sized L1 wayfinder scratch location 0
Oct 21 06:19:47 computer kernel: NVRM: RmInitAdapter: Cannot initialize GSP firmware RM
Oct 21 06:19:47 computer kernel: NVRM: GPU 0000:12:00.0: RmInitAdapter failed! (0x62:0x40:2015)
Oct 21 06:19:47 computer kernel: NVRM: GPU 0000:12:00.0: rm_init_adapter failed, device minor number 0
Oct 21 06:19:47 computer kernel: NVRM: _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
Oct 21 06:19:47 computer kernel: NVRM: _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
Oct 21 06:19:47 computer kernel: NVRM: crashcatWayfinderGetReportQueue_V1: insufficiently-sized L1 wayfinder scratch location 0
Oct 21 06:19:47 computer kernel: NVRM: RmInitAdapter: Cannot initialize GSP firmware RM
Oct 21 06:19:47 computer kernel: NVRM: GPU 0000:12:00.0: RmInitAdapter failed! (0x62:0x40:2015)
Oct 21 06:19:47 computer kernel: NVRM: GPU 0000:12:00.0: rm_init_adapter failed, device minor number 0
Oct 21 06:19:47 computer k3s[3628]: I1021 06:19:47.502981 3628 server.go:158] “Got registration request from device plugin with resource” resourceName=“nvidia.com/gpu”
Oct 21 06:19:47 computer kernel: NVRM: _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
Oct 21 06:19:47 computer kernel: NVRM: _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
Oct 21 06:19:47 computer kernel: NVRM: crashcatWayfinderGetReportQueue_V1: insufficiently-sized L1 wayfinder scratch location 0
Oct 21 06:19:47 computer kernel: NVRM: RmInitAdapter: Cannot initialize GSP firmware RM
Oct 21 06:19:47 computer kernel: NVRM: GPU 0000:12:00.0: RmInitAdapter failed! (0x62:0x40:2015)
Oct 21 06:19:47 computer kernel: NVRM: GPU 0000:12:00.0: rm_init_adapter failed, device minor number 0
Oct 21 06:19:48 computer kernel: NVRM: _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
Oct 21 06:19:48 computer kernel: NVRM: _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
Oct 21 06:19:48 computer kernel: NVRM: crashcatWayfinderGetReportQueue_V1: insufficiently-sized L1 wayfinder scratch location 0
Oct 21 06:19:48 computer kernel: NVRM: RmInitAdapter: Cannot initialize GSP firmware RM
Oct 21 06:19:48 computer kernel: NVRM: GPU 0000:12:00.0: RmInitAdapter failed! (0x62:0x40:2015)
Oct 21 06:19:48 computer kernel: NVRM: GPU 0000:12:00.0: rm_init_adapter failed, device minor number 0
Oct 21 06:19:48 computer kernel: NVRM: _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
Oct 21 06:19:48 computer kernel: NVRM: _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
Oct 21 06:19:48 computer kernel: NVRM: crashcatWayfinderGetReportQueue_V1: insufficiently-sized L1 wayfinder scratch location 0
Oct 21 06:19:48 computer kernel: NVRM: RmInitAdapter: Cannot initialize GSP firmware RM
Oct 21 06:19:48 computer kernel: NVRM: GPU 0000:12:00.0: RmInitAdapter failed! (0x62:0x40:2015)
Oct 21 06:19:48 computer kernel: NVRM: GPU 0000:12:00.0: rm_init_adapter failed, device minor number 0
Oct 21 06:19:49 computer kernel: NVRM: _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
Oct 21 06:19:49 computer kernel: NVRM: _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
Oct 21 06:19:49 computer kernel: NVRM: crashcatWayfinderGetReportQueue_V1: insufficiently-sized L1 wayfinder scratch location 0
Oct 21 06:19:49 computer kernel: NVRM: RmInitAdapter: Cannot initialize GSP firmware RM
Oct 21 06:19:49 computer kernel: NVRM: GPU 0000:12:00.0: RmInitAdapter failed! (0x62:0x40:2015)
Oct 21 06:19:49 computer kernel: NVRM: GPU 0000:12:00.0: rm_init_adapter failed, device minor number 0
Oct 21 06:19:50 computer containerd[12267]: time=“2025-10-21T06:19:50.001419314Z” level=info msg=“Start cri plugin with config {PluginConfig:{ContainerdConfig:{Snapshotter:overlayfs DefaultRuntimeName:nvidia DefaultRuntime:{Type: Path: Engine: PodAnnotations: ContainerAnnotations: Root: Options:map PrivilegedWithoutHostDevices:false PrivilegedWithoutHostDevicesAllDevicesAllowed:false BaseRuntimeSpec: NetworkPluginConfDir: NetworkPluginMaxConfNum:0 Snapshotter: SandboxMode:} UntrustedWorkloadRuntime:{Type: Path: Engine: PodAnnotations: ContainerAnnotations: Root: Options:map PrivilegedWithoutHostDevices:false PrivilegedWithoutHostDevicesAllDevicesAllowed:false BaseRuntimeSpec: NetworkPluginConfDir: NetworkPluginMaxConfNum:0 Snapshotter: SandboxMode:} Runtimes:map[nvidia:{Type:io.containerd.runc.v2 Path: Engine: PodAnnotations: ContainerAnnotations: Root: Options:map[BinaryName:/usr/local/nvidia/toolkit/nvidia-container-runtime CriuImagePath: CriuPath: CriuWorkPath: IoGid:0 IoUid:0 NoNewKeyring:false NoPivotRoot:false Root: ShimCgroup: SystemdCgroup:false] PrivilegedWithoutHostDevices:false PrivilegedWithoutHostDevicesAllDevicesAllowed:false BaseRuntimeSpec: NetworkPluginConfDir: NetworkPluginMaxConfNum:0 Snapshotter: SandboxMode:podsandbox} nvidia-cdi:{Type:io.containerd.runc.v2 Path: Engine: PodAnnotations: ContainerAnnotations: Root: Options:map[BinaryName:/usr/local/nvidia/toolkit/nvidia-container-runtime.cdi CriuImagePath: CriuPath: CriuWorkPath: IoGid:0 IoUid:0 NoNewKeyring:false NoPivotRoot:false Root: ShimCgroup: SystemdCgroup:false] PrivilegedWithoutHostDevices:false PrivilegedWithoutHostDevicesAllDevicesAllowed:false BaseRuntimeSpec: NetworkPluginConfDir: NetworkPluginMaxConfNum:0 Snapshotter: SandboxMode:podsandbox} nvidia-legacy:{Type:io.containerd.runc.v2 Path: Engine: PodAnnotations: ContainerAnnotations: Root: Options:map[BinaryName:/usr/local/nvidia/toolkit/nvidia-container-runtime.legacy CriuImagePath: CriuPath: CriuWorkPath: IoGid:0 IoUid:0 NoNewKeyring:false NoPivotRoot:false Root: ShimCgroup: SystemdCgroup:false] PrivilegedWithoutHostDevices:false PrivilegedWithoutHostDevicesAllDevicesAllowed:false BaseRuntimeSpec: NetworkPluginConfDir: NetworkPluginMaxConfNum:0 Snapshotter: SandboxMode:podsandbox} runc:{Type:io.containerd.runc.v2 Path: Engine: PodAnnotations: ContainerAnnotations: Root: Options:map[BinaryName: CriuImagePath: CriuPath: CriuWorkPath: IoGid:0 IoUid:0 NoNewKeyring:false NoPivotRoot:false Root: ShimCgroup: SystemdCgroup:false] PrivilegedWithoutHostDevices:false PrivilegedWithoutHostDevicesAllDevicesAllowed:false BaseRuntimeSpec: NetworkPluginConfDir: NetworkPluginMaxConfNum:0 Snapshotter: SandboxMode:podsandbox}] NoPivot:false DisableSnapshotAnnotations:true DiscardUnpackedLayers:false IgnoreBlockIONotEnabledErrors:false IgnoreRdtNotEnabledErrors:false} CniConfig:{NetworkPluginBinDir:/opt/cni/bin NetworkPluginConfDir:/etc/cni/net.d NetworkPluginMaxConfNum:1 NetworkPluginSetupSerially:false NetworkPluginConfTemplate: IPPreference:} Registry:{ConfigPath: Mirrors:map Configs:map Auths:map Headers:map} ImageDecryption:{KeyModel:node} DisableTCPService:true StreamServerAddress:127.0.0.1 StreamServerPort:0 StreamIdleTimeout:4h0m0s EnableSelinux:false SelinuxCategoryRange:1024 SandboxImage:registry.k8s.io/pause:3.8 StatsCollectPeriod:10 SystemdCgroup:false EnableTLSStreaming:false X509KeyPairStreaming:{TLSCertFile: TLSKeyFile:} MaxContainerLogLineSize:16384 DisableCgroup:false DisableApparmor:false RestrictOOMScoreAdj:false MaxConcurrentDownloads:3 DisableProcMount:false UnsetSeccompProfile: TolerateMissingHugetlbController:true DisableHugetlbController:true DeviceOwnershipFromSecurityContext:false IgnoreImageDefinedVolumes:false NetNSMountsUnderStateDir:false EnableUnprivilegedPorts:false EnableUnprivilegedICMP:false EnableCDI:false CDISpecDirs:[/etc/cdi /var/run/cdi] ImagePullProgressTimeout:5m0s DrainExecSyncIOTimeout:0s ImagePullWithSyncFs:false IgnoreDeprecationWarnings:} ContainerdRootDir:/var/lib/containerd ContainerdEndpoint:/run/containerd/containerd.sock RootDir:/var/lib/containerd/io.containerd.grpc.v1.cri StateDir:/run/containerd/io.containerd.grpc.v1.cri}”
Oct 21 06:19:50 computer kernel: NVRM: _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
Oct 21 06:19:50 computer kernel: NVRM: _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
Oct 21 06:19:50 computer kernel: NVRM: crashcatWayfinderGetReportQueue_V1: insufficiently-sized L1 wayfinder scratch location 0
Oct 21 06:19:50 computer kernel: NVRM: RmInitAdapter: Cannot initialize GSP firmware RM
Oct 21 06:19:50 computer kernel: NVRM: GPU 0000:12:00.0: RmInitAdapter failed! (0x62:0x40:2015)
Oct 21 06:19:50 computer kernel: NVRM: GPU 0000:12:00.0: rm_init_adapter failed, device minor number 0
Oct 21 06:19:51 computer kernel: NVRM: _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
Oct 21 06:19:51 computer kernel: NVRM: _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
Oct 21 06:19:51 computer kernel: NVRM: crashcatWayfinderGetReportQueue_V1: insufficiently-sized L1 wayfinder scratch location 0
Oct 21 06:19:51 computer kernel: NVRM: RmInitAdapter: Cannot initialize GSP firmware RM
Oct 21 06:19:51 computer kernel: NVRM: GPU 0000:12:00.0: RmInitAdapter failed! (0x62:0x40:2015)
Oct 21 06:19:51 computer kernel: NVRM: GPU 0000:12:00.0: rm_init_adapter failed, device minor number 0
Oct 21 06:19:57 computer kernel: NVRM: _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
Oct 21 06:19:57 computer kernel: NVRM: _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
Oct 21 06:19:57 computer kernel: NVRM: crashcatWayfinderGetReportQueue_V1: insufficiently-sized L1 wayfinder scratch location 0
Oct 21 06:19:57 computer kernel: NVRM: RmInitAdapter: Cannot initialize GSP firmware RM
Oct 21 06:19:57 computer kernel: NVRM: GPU 0000:12:00.0: RmInitAdapter failed! (0x62:0x40:2015)
Oct 21 06:19:57 computer kernel: NVRM: GPU 0000:12:00.0: rm_init_adapter failed, device minor number 0
Oct 21 06:20:30 computer kernel: NVRM: _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
Oct 21 06:20:30 computer kernel: NVRM: _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
Oct 21 06:20:30 computer kernel: NVRM: crashcatWayfinderGetReportQueue_V1: insufficiently-sized L1 wayfinder scratch location 0
Oct 21 06:20:30 computer kernel: NVRM: RmInitAdapter: Cannot initialize GSP firmware RM
Oct 21 06:20:30 computer kernel: NVRM: GPU 0000:12:00.0: RmInitAdapter failed! (0x62:0x40:2015)
Oct 21 06:20:30 computer kernel: NVRM: GPU 0000:12:00.0: rm_init_adapter failed, device minor number 0
Oct 21 06:20:35 computer k3s[3628]: E1021 06:20:35.762794 3628 pod_workers.go:1301] “Error syncing pod, skipping” err=“failed to “StartContainer” for “worker” with CrashLoopBackOff: “back-off 10s restarting failed container=worker pod=network-operator-node-feature-discovery-worker-kg2m6_nvidia-network-operator(bab0c4c2-a79d-43ac-87ef-9220b52d9e23)”” pod=“nvidia-network-operator/network-operator-node-feature-discovery-worker-kg2m6” podUID=“bab0c4c2-a79d-43ac-87ef-9220b52d9e23”