Hi, I installed a 2080 Ti and run several DL jobs on it. It turns out that after 20 minutes or so it always froze the system and have ERR! shown in both the Fan and PowerUsage from the nvidia-smi. I have driver version 415.18 and running on CUDA 9.2. Any idea what’s going on? My training first show…

I also have problem with this. Mine is the supermicro with 8 GTX 1080 Ti GPUs running on driver 410.78 Please advise what I should do to fix this. The Gpus report Fan as ERR! and watt also ERR! It sometimes happens and the only way to fix at the moment is to reboot the machine, but we have a lot of…

Hi, have you solved this problem? Many thanks!

Hi, I still have that issue. Please help. It happens sometimes.

Hi how frequent does this happen? For me, it happens almost every week. I spoke with the specialist. And he told me to reseat the GPUs. Since I have 3 cards, he told me to do it for all of them. I will try tomorrow. But in the meantime if you also try, and see any difference could you please let m…

The issue still exists. I tried reseating the card, it doesn’t solve the problem.

This issue is due to the higher temperature. First, you should reseat the question card to the coolest location of your workstation. Second, set the power limitation [1] and fan speed [2] to ensure the peak temperature does not exceed 75C. [1] Change them to 150W-to-200W sudo nvidia-smi -pm 1 s…

Hi mb55407, thanks for your message. In my case the problem is not due to the card, it is due to the location. The one in the middle always shows this error. Also the temperature seems to be fine. When you were getting this error, was the temperature higher than 75C?

Yes, after running a task with 90C for 2 days, my second card which has the highest temperature got this error message. I switch the location of my second and fourth card and sets the above temperature control solutions, and then this message never appears again.

The same thing happens on my Titan V, but only after ~6 hours of training at ~85C. I’ll try underwatting it to 150W and holding it at ~77C. I’m using driver version 415.27 and CUDA 10, along with a 2080 Ti and Titan RTX in the same system.

NVIDIA-SMI Shows ERR! on both Fan and Power Usage

Graphics / Linux Linux

generix February 15, 2019, 7:03pm 13

Make sure to disable iommu in bios. If issue persists, use gpu-burn to test your hardware and post the results.

Topic		Replies	Views
2080Ti got ERR soon after starting DL training Linux	11	1911	February 2, 2019
Nvidia-smi show ERR! on FAN Linux	0	1037	July 13, 2023
The GPU FAN runs heavily after the process is done. CUDA Setup and Installation	19	5534	July 20, 2017
GPUs give ERR! with NVRM: Xid (PCI:0000:b5:00): 61 Linux	2	1278	July 22, 2019
RTX 2080 Ti always Power Cap and low utilization Linux	14	4989	November 6, 2019
only one RTX 2080Ti can be found after reboot when there is fan err and voltage err by nvidia-smi Linux	2	569	April 24, 2019
ERR in nvidia-smi Linux	0	521	August 29, 2023
NVIDIA-SMI Shows ERR! on both Fan and Power Usage General Discussion ubuntu	0	1389	October 25, 2022
Is my card burned? CUDA Setup and Installation	15	2848	June 27, 2013
ERR! on Fan in nvidia-smi with driver 410.93 Linux	1	1960	June 28, 2019

NVIDIA-SMI Shows ERR! on both Fan and Power Usage

Related topics