I have two GPUs, a P40 and a 2080, on a small hobby server. I use this server to train neural networks for hobby projects. I want to run two different projects on the two machines, but unfortunately, whenever I do this, the 2080 crashes after several minutes.
nvidia-smi returns
Unable to determine the device handle for GPU0000:04:00.0: Unknown Error
while “nvidia-debugdump --list” returns
Found 2 NVIDIA devices
Error: nvmlDeviceGetHandleByIndex(): Unknown Error
FAILED to get details on GPU (0x0): Unknown Error
Here is the detailed bug report:
nvidia-bug-report.log.gz (1.8 MB)
Has anyone seen this sort of issue before? Some other threads with similar errors suggest this may be a PCIe bus issue, like a loose connection, but I don’t think that’s the case: this only happens when both GPUs are used for intensive training.