nvidia-smi suddenly loss one of three cards

zhejiangyyf · February 1, 2020, 5:19am

I have three cards with cuda on my ubuntu 14.04. The three cards run well most of the time.
But sometimes, the nvidia-smi loss one card, showing only two.
This happens many times on the history, everytime restart the ubuntu will solve the problem.
Everyone of the three has met this problem os that it is no relation with slot or power connector.
I want to know how to solve the problem forever because restart always interrupt my work.

lspci show three cards:
lspci | grep NVIDIA
03:00.0 VGA compatible controller: NVIDIA Corporation Device 1b81 (rev a1)
03:00.1 Audio device: NVIDIA Corporation Device 10f0 (rev a1)
05:00.0 VGA compatible controller: NVIDIA Corporation Device 1b06 (rev a1)
05:00.1 Audio device: NVIDIA Corporation Device 10ef (rev a1)
06:00.0 VGA compatible controller: NVIDIA Corporation Device 1b81 (rev a1)
06:00.1 Audio device: NVIDIA Corporation Device 10f0 (rev a1)

±----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 3627 C python 81MiB |
±----------------------------------------------------------------------------+

generix · February 1, 2020, 5:19am

If this happens under load, maybe a problem with air flow so the card overheats and shuts down. Monitor the temperatures and power draw while it’s working until it shuts down.

zhejiangyyf · February 1, 2020, 5:19am

This always hanppens at the free time, so no relation with overheats.
When on working，the card never shutdown .

generix · February 1, 2020, 5:19am

Please run nvidia-bug-report.sh when this happens and attach output file to your post.

zhejiangyyf · February 1, 2020, 5:19am

How to download nvidia-bug-report.sh?

generix · February 1, 2020, 5:19am

It comes bundled with the driver so should be installed. Icon for attaching files is next to editing your post.
Do you have the persistence daemon started? If not, the gpus would go offline when idle.

zhejiangyyf · February 1, 2020, 5:19am

Just now it happens again !!!
This is the log content.

gist.github.com

https://gist.github.com/zhejiangyyf/fff63ba11c65432dffeaf4bc33efc795

nvdia-smi_log

____________________________________________

Start of NVIDIA bug report log file.  Please include this file, along
with a detailed description of your problem, when reporting a graphics
driver bug via the NVIDIA Linux forum (see devtalk.nvidia.com)
or by sending email to 'linux-bugs@nvidia.com'.

nvidia-bug-report.sh Version: 21455275

Date: 2017年 06月 08日 星期四 10:32:39 CST

This file has been truncated. show original

sandipt · February 1, 2020, 5:19am

In log I see :

[184611.868266] NVRM: RmInitAdapter failed! (0x26:0x65:1097)
[184611.868293] NVRM: rm_init_adapter failed for device bearing minor number 0
[184669.238992] NVRM: RmInitAdapter failed! (0x24:0x65:1060)
[184669.239024] NVRM: rm_init_adapter failed for device bearing minor number 0
[184725.003731] NVRM: RmInitAdapter failed! (0x24:0x65:1060)
[184725.003821] NVRM: rm_init_adapter failed for device bearing minor number 0

If same gpu is hitting this issue again and again that mean it can be faulty gpu. Please replace remove gpu and check.

Are you running any task on these gpus? And that task hit this issue? Is the earlier driver worked on you system without this issue? Any recent change on system? Also please test with latest nvidia driver from http://www.nvidia.com/object/unix.html

Also check why you are getting below errors in dmesg :

[ 0.290379] pci 0000:09:01.0: BAR 14: no space for [mem size 0x00200000]
[ 0.290380] pci 0000:09:01.0: BAR 14: failed to assign [mem size 0x00200000]
[ 0.290381] pci 0000:09:01.0: BAR 15: assigned [mem 0x5d000000-0x5d1fffff 64bit pref]
[ 0.290381] pci 0000:09:04.0: BAR 14: no space for [mem size 0x00200000]
[ 0.290382] pci 0000:09:04.0: BAR 14: failed to assign [mem size 0x00200000]
[ 0.290383] pci 0000:09:04.0: BAR 15: assigned [mem 0x5d200000-0x5d3fffff 64bit pref]
[ 0.290384] pci 0000:09:01.0: BAR 13: assigned [io 0x2000-0x2fff]
[ 0.290385] pci 0000:09:04.0: BAR 13: assigned [io 0x3000-0x3fff]
[ 0.290386] pci 0000:09:04.0: BAR 14: no space for [mem size 0x00200000]
[ 0.290387] pci 0000:09:04.0: BAR 14: failed to assign [mem size 0x00200000]
[ 0.290388] pci 0000:09:01.0: BAR 14: no space for [mem size 0x00200000]
[ 0.290388] pci 0000:09:01.0: BAR 14: failed to assign [mem size 0x00200000]

zhejiangyyf · February 1, 2020, 5:19am

This issue happens on all of the cards not only special one, so there is no faulty gpu.
I always run training task on this card for longtime. The card never hit this issue when it is running.
The issue always happens on the other free card.

I have no change about the system recently. I can try the newest driver.
In my memory this problem is not about special driver version.

I have no idea about errors in dmesg now. I will try search reason about this.

zhejiangyyf · February 1, 2020, 5:19am

Again! The issue hit me Again!
One of my 1070 card is missing. I have to restart the system again.
The log content:

gist.github.com

https://gist.github.com/zhejiangyyf/ad3c3842e6895a26b17047118ab5800e

nvdia_log

____________________________________________

Start of NVIDIA bug report log file.  Please include this file, along
with a detailed description of your problem, when reporting a graphics
driver bug via the NVIDIA Linux forum (see devtalk.nvidia.com)
or by sending email to 'linux-bugs@nvidia.com'.

nvidia-bug-report.sh Version: 21455275

Date: 2017年 06月 09日 星期五 11:20:05 CST

This file has been truncated. show original

±----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 10137 C python 10733MiB |
±----------------------------------------------------------------------------+

sandipt · May 6, 2020, 5:25pm

Please connected one and then two cards in system. I then check if this issue still reproduce? Is the issue reproduce with only one cards? Is the issue reproduce with only two cards?

>>I always run training task on this card for longtime.
May I know how to run this task so we can reproduce this issue here internally to investigate. Please share code or sample application that can trigger this issue. Let me know what are the different apps or packages needed on os.

zhejiangyyf · February 1, 2020, 5:20am

I am very sure that this happened with two cards, because I had two cards for sometimes.
But I am not sure the situation with one card.
In my memory, it never appeared as in the beginning months I only had one card.

Sorry, I have no time to test the two cases as I have important task for the cards right now.

sandipt · May 6, 2020, 5:25pm

>>I always run training task on this card for longtime.
May I know how to run this task so we can reproduce this issue here internally to investigate. Please share code or sample application that can trigger this issue. Let me know what are the different apps or packages needed on os.

zhejiangyyf · February 1, 2020, 5:20am

Sorry, although I have met the issue many times, but I have no idea about the trigger condition.
It is just like a magic comming without any signs at all.

In the following time, I will try to be more careful about the trigger condition.
I will update the topic when I find useful information.

sL1pKn07 · February 1, 2020, 5:20am

the PSU is ok?

marcopardo · February 1, 2020, 5:56am

Hi,

I have the same problem. I have 4x Zotac Nvidia GTX 1060 3GB. I can run multiple heavy tasks (mining, videoencoding/recoding) on the cards for months without any problem. But if I stop the tasks for reorganising/updating I ‘loose’ 1 or 2 cards.
lspci shows all of them but nvidia-smi dont.

The temps and the PSU are okay.
All cards together under load only uses 280W. The system has 550W on its 12V rail and they never ‘get lost’ when they work with their 70W each. Only if the load disappears.

After some minutes the cards are back online again.

here is my log. It is little after this problem (here are all cars good again), but eventually you can see something.
ftp://nvidia:nvidia20171214@78.94.151.94/nvidia-bug-report.log.gz

generix · February 1, 2020, 5:56am

Just a wild guess, see if kernel parameter
pcie_port_pm=off
helps.