Unable to determine the device handle for GPU 0000:05:00.0: GPU is lost. Reboot the system to recover this GPU

Graphics / Linux Linux
,
1

Our server crashes while running a pytorch model after we moved our server to a new rack.
The program get stuck after a few iterations, and when we run nvidia-smi, it reports: “Unable to determine the device handle for GPU 0000:05:00.0: GPU is lost. Reboot the system to recover this GPU”.
The same code runs perfectly before we moved the server, and we did not change anything but the position of the server, so we have no idea what is wrong.

Here’s what we have tried:

  1. reboot - can recover the GPU, but when we run any code, the “GPU is lost” problem occurs after a while.
  2. run with other 3 GPUs - no problem.
  3. check the temperature - the temperature only gose to 70C before the GPU is lost.
  4. check the power supply - we have not changed any hardware or software of the server, we only moved it, so the power supply should be the same with the previous.
  5. run command “nvidia-smi -pm 1” to turn on perssistance mode - still have the problem.

Here’s the bug log:
nvidia-bug-report.log (3.1 MB)

The server have 4 2080Ti.

If you know anything related to this problem, Please help.

2

It’s always the same gpu so I guess it came loose in its slot when you physically moved the server. Please try reseating the gpus in their pcie slots.

1 Like
3

Thanks, we will try. But we did open the case and re-plug those power lines, the GPU seems to be well fixed with screws, so we did not reseat them.