I have three cards with cuda on my ubuntu 14.04. The three cards run well most of the time. But sometimes, the nvidia-smi loss one card, showing only two. This happens many times on the history, everytime restart the ubuntu will solve the problem. Everyone of the three has met this problem os tha…

If this happens under load, maybe a problem with air flow so the card overheats and shuts down. Monitor the temperatures and power draw while it’s working until it shuts down.

This always hanppens at the free time, so no relation with overheats. When on working，the card never shutdown .

Please run nvidia-bug-report.sh when this happens and attach output file to your post.

How to download nvidia-bug-report.sh?

It comes bundled with the driver so should be installed. Icon for attaching files is next to editing your post. Do you have the persistence daemon started? If not, the gpus would go offline when idle.

Just now it happens again !!! This is the log content. https://gist.github.com/zhejiangyyf/fff63ba11c65432dffeaf4bc33efc795

In log I see : [184611.868266] NVRM: RmInitAdapter failed! (0x26:0x65:1097) [184611.868293] NVRM: rm_init_adapter failed for device bearing minor number 0 [184669.238992] NVRM: RmInitAdapter failed! (0x24:0x65:1060) [184669.239024] NVRM: rm_init_adapter failed for device bearing minor number 0 …

This issue happens on all of the cards not only special one, so there is no faulty gpu. I always run training task on this card for longtime. The card never hit this issue when it is running. The issue always happens on the other free card. I have no change about the system recently. I can try th…

Again! The issue hit me Again! One of my 1070 card is missing. I have to restart the system again. The log content: https://gist.github.com/zhejiangyyf/ad3c3842e6895a26b17047118ab5800e ±------------------------------------------------------------------------…

Please connected one and then two cards in system. I then check if this issue still reproduce? Is the issue reproduce with only one cards? Is the issue reproduce with only two cards? >>I always run training task on this card for longtime. May I know how to run this task so we can reproduce this i…

nvidia-smi suddenly loss one of three cards

Graphics / Linux Linux

zhejiangyyf June 8, 2017, 2:45am 7

Just now it happens again !!!
This is the log content.

Topic		Replies	Views
Nvidia-smi loss one of four cards Linux nvbugs	1	980	March 29, 2022
One of two GPUs disappears from nvidia-smi CUDA Setup and Installation	4	6695	January 13, 2018
Could not recognize one A100 card of two cards by nvidia-smi Linux ubuntu	2	2567	October 12, 2021
Nvidia-smi failed to detect all GPU cards CUDA Setup and Installation	11	14040	December 14, 2018
Under CentOS 7.1 system, eight GPU cards are checked with nvidia-smi command, one card is error, one card is lost, and when nvida-smi command is executed, carton is slow. Linux	3	879	January 8, 2019
Nvidia-smi is giving GPU is lost on ubuntu-16.04 with AMD Ryzen 7 2700X CPU, X470 Motherboard and two 1080 Ti GPUs Linux	1	543	July 22, 2020
NVIDIA L40S keeps dropping from nvidia-smi GPU - Hardware cuda , kernel , ubuntu	1	225	July 11, 2025
Nvidia-smi reads only 3 GPUs from 4 Linux	7	2980	June 3, 2017
Cannot nvidia-smi Geforce 1070 anymore suddenly. Linux	9	1833	October 12, 2021
Blank screen on boot and one of four GPUs missing in nvidia-smi Linux boot	3	3572	June 20, 2020

nvidia-smi suddenly loss one of three cards

Related topics