We’re having an issue with one of our systems with dual RTX Pro 6000 GPUs. We were doing some profiling using Nsight and all of the Nvidia related processes hung at 100% CPU. After killing them, one of the GPUs was no longer shown in nvidia-smi, which persisted even after power cycling and reinstalling the drivers. It appears the VBIOS was corrupted given that is shows ??.??.??.??.??
$ lspci | grep -i nvidia
43:00.0 VGA compatible controller: NVIDIA Corporation Device 2bb1 (rev a1)
43:00.1 Audio device: NVIDIA Corporation Device 22e8 (rev a1)
6f:00.0 VGA compatible controller: NVIDIA Corporation Device 2bb1 (rev a1)
6f:00.1 Audio device: NVIDIA Corporation Device 22e8 (rev a1)
$ nvidia-smi -L
GPU 0: NVIDIA RTX PRO 6000 Blackwell Workstation Edition (UUID: GPU-c681a258-86c7-673f-3f2e-77c50fd2ae0c)
Failed GPU procfs shows VBIOS unreadable and no firmware:
$ cat /proc/driver/nvidia/gpus/0000:43:00.0/information
Model: NVIDIA RTX PRO 6000 Blackwell Workstation Edition
IRQ: 149
GPU UUID: GPU-e0be9369-843f-a56c-2def-efc2848da030
Video BIOS: ??.??.??.??.??
Bus Type: PCIe
DMA Size: 52 bits
DMA Mask: 0xfffffffffffff
Bus Location: 0000:43:00.0
Device Minor: 0
GPU Firmware: N/A
GPU Excluded: No
Working GPU for comparison:
$ cat /proc/driver/nvidia/gpus/0000:6f:00.0/information
Model: NVIDIA RTX PRO 6000 Blackwell Workstation Edition
IRQ: 148
GPU UUID: GPU-c681a258-86c7-673f-3f2e-77c50fd2ae0c
Video BIOS: 98.02.52.00.00
Bus Type: PCIe
DMA Size: 52 bits
DMA Mask: 0xfffffffffffff
Bus Location: 0000:6f:00.0
Device Minor: 1
GPU Firmware: 590.48.01
GPU Excluded: No
Partial dmesg excertp:
[ 12.325275] NVRM: GPU at PCI:0000:43:00: GPU-e0be9369-843f-a56c-2def-efc2848da030
[ 12.325280] NVRM: Xid (PCI:0000:43:00): 143, Error status 0x65 while polling for FSP boot complete, 0x24, 0x3e, 0x0, 0x0, 0x2
[ 12.325287] NVRM: GPU0 kfspDumpDebugState_GB100: FSP microcode v4.76
[ 12.325288] NVRM: GPU0 kfspDumpDebugState_GB100: GPU 0000:43:00
[ 12.325291] NVRM: GPU0 kfspDumpDebugState_GB100: NV_PFSP_FALCON_COMMON_SCRATCH_GROUP_2(0) = 0x3e
[ 12.325293] NVRM: GPU0 kfspDumpDebugState_GB100: NV_PFSP_FALCON_COMMON_SCRATCH_GROUP_2(1) = 0x0
[ 12.325295] NVRM: GPU0 kfspDumpDebugState_GB100: NV_PFSP_FALCON_COMMON_SCRATCH_GROUP_2(2) = 0x0
[ 12.325297] NVRM: GPU0 kfspDumpDebugState_GB100: NV_PFSP_FALCON_COMMON_SCRATCH_GROUP_2(3) = 0x2
[ 12.325300] NVRM: GPU0 gpuHandleSanityCheckRegReadError_GH100: Possible bad register read: addr: 0x110040, regvalue: 0xbadf4100, error code: Unknown SYS_PRI_ERROR_CODE
[ 12.325301] NVRM: GPU0 kfspDumpDebugState_GB100: NV_PGSP_FALCON_MAILBOX0 = 0xbadf4100
[ 12.325305] NVRM: GPU0 gpuHandleSanityCheckRegReadError_GH100: Possible bad register read: addr: 0x110044, regvalue: 0xbadf4100, error code: Unknown SYS_PRI_ERROR_CODE
[ 12.325306] NVRM: GPU0 kfspDumpDebugState_GB100: NV_PGSP_FALCON_MAILBOX1 = 0xbadf4100
[ 12.325309] NVRM: GPU0 gpuHandleSanityCheckRegReadError_GH100: Possible bad register read: addr: 0x110804, regvalue: 0xbadf4100, error code: Unknown SYS_PRI_ERROR_CODE
[ 12.325310] NVRM: GPU0 kfspDumpDebugState_GB100: NV_PGSP_MAILBOX(0) = 0xbadf4100
[ 12.325313] NVRM: GPU0 gpuHandleSanityCheckRegReadError_GH100: Possible bad register read: addr: 0x110808, regvalue: 0xbadf4100, error code: Unknown SYS_PRI_ERROR_CODE
[ 12.325314] NVRM: GPU0 kfspDumpDebugState_GB100: NV_PGSP_MAILBOX(1) = 0xbadf4100
[ 12.325317] NVRM: GPU0 gpuHandleSanityCheckRegReadError_GH100: Possible bad register read: addr: 0x11080c, regvalue: 0xbadf4100, error code: Unknown SYS_PRI_ERROR_CODE
[ 12.325318] NVRM: GPU0 kfspDumpDebugState_GB100: NV_PGSP_MAILBOX(2) = 0xbadf4100
[ 12.325321] NVRM: GPU0 gpuHandleSanityCheckRegReadError_GH100: Possible bad register read: addr: 0x110810, regvalue: 0xbadf4100, error code: Unknown SYS_PRI_ERROR_CODE
[ 12.325322] NVRM: GPU0 kfspDumpDebugState_GB100: NV_PGSP_MAILBOX(3) = 0xbadf4100
[ 12.325396] NVRM: GPU0 _kfspPrintCms2Log_GB100: CMS2 Log:
[ 12.325397] NVRM: GPU0 nvDbgDumpBufferBytes: x0 x1 x2 x3 x4 x5 x6 x7 x8 x9 xa xb xc xd xe xf
[ 12.325398] NVRM: GPU0 nvDbgDumpBufferBytes: FF751FC90130EE88 00 00 14 00 3e 00 00 00 02 00 00 00 00 00 00 00
I’m hoping there’s a linux CLI tool somewhere to reflash the firmware on the device so we can get it back up and running.
Hello @daniel400, welcome to the NVIDIA developer forums.
Before trying to re-flash, I think some more troubleshooting is in order.
Can you run nvidia-bug-report.sh and attach the resulting file here in this post?
Also, did you try running the suspicious GPU standalone in a test system to verify you get similar results? Ideally in fresh OS installation with the latest GPU drivers and only this one GPU.
Right no I have trouble understanding how using NSight would in any way modify or “break” the VBIOS. This seems rather like a system issue.
nvidia-bug-report.log.gz (1.1 MB)
We’ll try shuffling the GPU around to see if that makes any difference.
I tried moving both GPUs to another Windows test system (one at a time). The good GPU boots up fine and shows up in task manager / nvidia-smi. The bad GPU refuses to boot - stuck in POST without any monitor output. When I tried to add a different GPU as primary and the bad GPU as secondary, it’s stuck with a single “_” in the upper left corner on the primary GPU and never boots.
Note: the failed GPU is a dev sample provided to us by NVIDIA which had been working fine prior to the Nsight profiling session that went bad.
Hello @daniel400 did you find a solution yet? I am facing a similar issue and want to rule out all possible approaches on my end before trying to RMA this card.
The card (RTX Pro 6000) was working fine with a 1000W PSU and an older AsRock X570 Pro4 mainboard (DDR4) for the past three months. The system became a bit unresponsive and I did a system update + reboot (Arch Linux). Afterwards, the system showed trouble POSTing (only worked after repeated attempts), and when the system finally came up the GPU was either absent from lspci or it was present but couldn’t be used via nvidia-smi. The output of dmesg looks like this:
[Sun Apr 5 12:12:24 2026] NVRM: GPU at PCI:0000:03:00: GPU-89acf266-dbe7-ea62-e3a7-37080dd78115
[Sun Apr 5 12:12:24 2026] NVRM: Xid (PCI:0000:03:00): 143, Error status 0x65 while polling for FSP boot complete, 0x13, 0x667, 0x0, 0x0, 0x2
[Sun Apr 5 12:12:24 2026] NVRM: GPU0 kfspDumpDebugState_GB100: FSP microcode v4.76
[Sun Apr 5 12:12:24 2026] NVRM: GPU0 kfspDumpDebugState_GB100: GPU 0000:03:00
[Sun Apr 5 12:12:24 2026] NVRM: GPU0 kfspDumpDebugState_GB100: NV_PFSP_FALCON_COMMON_SCRATCH_GROUP_2(0) = 0x667
[Sun Apr 5 12:12:24 2026] NVRM: GPU0 kfspDumpDebugState_GB100: NV_PFSP_FALCON_COMMON_SCRATCH_GROUP_2(1) = 0x0
[Sun Apr 5 12:12:24 2026] NVRM: GPU0 kfspDumpDebugState_GB100: NV_PFSP_FALCON_COMMON_SCRATCH_GROUP_2(2) = 0x0
[Sun Apr 5 12:12:24 2026] NVRM: GPU0 kfspDumpDebugState_GB100: NV_PFSP_FALCON_COMMON_SCRATCH_GROUP_2(3) = 0x2
[Sun Apr 5 12:12:24 2026] NVRM: GPU0 gpuHandleSanityCheckRegReadError_GH100: Possible bad register read: addr: 0x110040, regvalue: 0xbadf4100, error code: Unknown SYS_PRI_ERROR_CODE
[Sun Apr 5 12:12:24 2026] NVRM: GPU0 kfspDumpDebugState_GB100: NV_PGSP_FALCON_MAILBOX0 = 0xbadf4100
[Sun Apr 5 12:12:24 2026] NVRM: GPU0 gpuHandleSanityCheckRegReadError_GH100: Possible bad register read: addr: 0x110044, regvalue: 0xbadf4100, error code: Unknown SYS_PRI_ERROR_CODE
[Sun Apr 5 12:12:24 2026] NVRM: GPU0 kfspDumpDebugState_GB100: NV_PGSP_FALCON_MAILBOX1 = 0xbadf4100
[Sun Apr 5 12:12:24 2026] NVRM: GPU0 gpuHandleSanityCheckRegReadError_GH100: Possible bad register read: addr: 0x110804, regvalue: 0xbadf4100, error code: Unknown SYS_PRI_ERROR_CODE
[Sun Apr 5 12:12:24 2026] NVRM: GPU0 kfspDumpDebugState_GB100: NV_PGSP_MAILBOX(0) = 0xbadf4100
[Sun Apr 5 12:12:24 2026] NVRM: GPU0 gpuHandleSanityCheckRegReadError_GH100: Possible bad register read: addr: 0x110808, regvalue: 0xbadf4100, error code: Unknown SYS_PRI_ERROR_CODE
[Sun Apr 5 12:12:24 2026] NVRM: GPU0 kfspDumpDebugState_GB100: NV_PGSP_MAILBOX(1) = 0xbadf4100
[Sun Apr 5 12:12:24 2026] NVRM: GPU0 gpuHandleSanityCheckRegReadError_GH100: Possible bad register read: addr: 0x11080c, regvalue: 0xbadf4100, error code: Unknown SYS_PRI_ERROR_CODE
[Sun Apr 5 12:12:24 2026] NVRM: GPU0 kfspDumpDebugState_GB100: NV_PGSP_MAILBOX(2) = 0xbadf4100
[Sun Apr 5 12:12:24 2026] NVRM: GPU0 gpuHandleSanityCheckRegReadError_GH100: Possible bad register read: addr: 0x110810, regvalue: 0xbadf4100, error code: Unknown SYS_PRI_ERROR_CODE
[Sun Apr 5 12:12:24 2026] NVRM: GPU0 kfspDumpDebugState_GB100: NV_PGSP_MAILBOX(3) = 0xbadf4100
[Sun Apr 5 12:12:24 2026] NVRM: GPU0 _kfspPrintCms2Log_GB100: CMS2 Log:
[Sun Apr 5 12:12:24 2026] NVRM: GPU0 nvDbgDumpBufferBytes: x0 x1 x2 x3 x4 x5 x6 x7 x8 x9 xa xb xc xd xe xf
[Sun Apr 5 12:12:24 2026] NVRM: GPU0 nvDbgDumpBufferBytes: FFFFCF460185F9D0 00 00 14 00 67 06 00 00 02 00 00 00 00 00 00 00
[Sun Apr 5 12:12:24 2026] NVRM: GPU0 nvDbgDumpBufferBytes: FFFFCF460185F9E0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[Sun Apr 5 12:12:24 2026] NVRM: GPU0 nvDbgDumpBufferBytes: FFFFCF460185F9F0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[Sun Apr 5 12:12:24 2026] NVRM: GPU0 nvDbgDumpBufferBytes: FFFFCF460185FA00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[Sun Apr 5 12:12:24 2026] NVRM: GPU0 nvDbgDumpBufferBytes: FFFFCF460185FA10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[Sun Apr 5 12:12:24 2026] NVRM: GPU0 nvDbgDumpBufferBytes: FFFFCF460185FA20 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[Sun Apr 5 12:12:24 2026] NVRM: GPU0 nvDbgDumpBufferBytes: FFFFCF460185FA30 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[Sun Apr 5 12:12:24 2026] NVRM: GPU0 nvDbgDumpBufferBytes: FFFFCF460185FA40 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[Sun Apr 5 12:12:24 2026] NVRM: GPU0 nvDbgDumpBufferBytes: FFFFCF460185FA50 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[Sun Apr 5 12:12:24 2026] NVRM: GPU0 nvDbgDumpBufferBytes: FFFFCF460185FA60 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[Sun Apr 5 12:12:24 2026] NVRM: GPU0 nvDbgDumpBufferBytes: FFFFCF460185FA70 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[Sun Apr 5 12:12:24 2026] NVRM: GPU0 nvDbgDumpBufferBytes: FFFFCF460185FA80 00 00 00 00 00 00 00 00 00 00 00 00 13 00 00 00
[Sun Apr 5 12:12:24 2026] NVRM: GPU0 nvCheckOkFailedNoLog: Check failed: Call timed out [NV_ERR_TIMEOUT] (0x00000065) returned from kgspWaitForGfwBootOk_HAL(pGpu, pKernelGsp) @ kernel_gsp.c:4904
[Sun Apr 5 12:12:24 2026] NVRM: GPU0 RmInitAdapter: Cannot initialize GSP firmware RM
[Sun Apr 5 12:12:24 2026] NVRM: GPU 0000:03:00.0: RmInitAdapter failed! (0x62:0x65:2168)
[Sun Apr 5 12:12:24 2026] NVRM: GPU 0000:03:00.0: rm_init_adapter failed, device minor number 0%
I did everything I could think of, BIOS upgrade, BIOS downgrade, PCI slot swap, reseating GPU, booting with another dedicated GPU, no change. I finally mounted the offending RTX Pro 6000 into another system which usually runs an RTX 5090, but after mounting it the other system doesn’t POST itself.
@MarkusHoHo any idea what could be causing this? Should I try to RMA the card?
I think at this point an RMA is well worth the effort.