I try my model on GPU and all of a sudden I can’t access to GPU
After rebooting, I can access to GPU but when I train the model, the problem happened again.
nvidia-smi gives me: Unable to determine the device handle for GPU0000:01:00.0: Unknown Error
‘dmesg’ gives me:
[Fri Sep 13 21:46:59 2024] NVRM: GPU at PCI:0000:01:00: GPU-f9d991dc-dd1b-5207-5db2-83a6f0de5053
[Fri Sep 13 21:46:59 2024] NVRM: Xid (PCI:0000:01:00): 79, pid=‘’, name=, GPU has fallen off the bus.
[Fri Sep 13 21:46:59 2024] NVRM: GPU 0000:01:00.0: GPU has fallen off the bus.
[Fri Sep 13 21:46:59 2024] NVRM: GPU0 GSP RPC buffer contains function 78 (DUMP_PROTOBUF_COMPONENT) and data 0x0000000000000000 0x0000000000000000.
Bug report
nvidia-bug-report.log.gz (819.1 KB)