Unable to determine the device handle for GPU0000:86:00.0: Unknown Error

Graphics / Linux Linux
,
1

OS: Ubuntu 20.04.1 LTS
Cuda compilation tools, release 11.8, V11.8.89
Build cuda_11.8.r11.8/compiler.31833905_0
GPUs: 4 4090
Hi,
I keep encountering ‘Unable to determine the device handle for GPU 0000:xx:00.0: Unknown Error’, where xx could be varied IDs representing different GPU cards.

Here’s more information:

Found 4 NVIDIA devices
        Device ID:              0
        Device name:            NVIDIA GeForce RTX 4090
        GPU internal ID:        GPU-924e60e2-b2c9-5636-bf3b-f087607114d3

        Device ID:              1
        Device name:            NVIDIA GeForce RTX 4090
        GPU internal ID:        GPU-874803ee-1638-1b6c-c636-69ef379ce0aa

Error: nvmlDeviceGetHandleByIndex(): Unknown Error
FAILED to get details on GPU (0x2): Unknown Error

Additionally, here’s nvidia-bug-report.sh output:
nvidia-bug-report.log (2.8 MB)

I started the Lora training task for stable diffusion on four 4090 cards. Two of them completed smoothly, while the other two failed after running for a short while.

2

“fallen off the bus”
Please check for overheating or lack of power.