Ubuntu 18.04 freeze GPU fallen off the bus

corentin.barbu · November 19, 2019, 9:16am

±----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 1244 G /usr/lib/xorg/Xorg 144MiB |
| 0 2069 G /usr/bin/gnome-shell 124MiB |
| 0 3300 G /usr/lib/firefox/firefox 1MiB |
±----------------------------------------------------------------------------+

It has been several months that my computer is doing complete freeze (no way to open a terminal), the only thing answering is the shortcut to keyboard light (similar shortcut to screen luminosity does nothing). Everytime, the last lines of syslog before the freeze are like:

Nov 19 09:33:59 atlas kernel: [ 2925.348909] NVRM: GPU at PCI:0000:01:00: GPU-a729e359-5a4d-43ef-e1a3-07447144413f
Nov 19 09:33:59 atlas kernel: [ 2925.348926] NVRM: Xid (PCI:0000:01:00): 79, pid=1473, GPU has fallen off the bus.
Nov 19 09:33:59 atlas kernel: [ 2925.348927] NVRM: GPU 0000:01:00.0: GPU has fallen off the bus.
Nov 19 09:33:59 atlas kernel: [ 2925.348933] NVRM: A GPU crash dump has been created. If possible, please run
Nov 19 09:33:59 atlas kernel: [ 2925.348933] NVRM: nvidia-bug-report.sh as root to collect this data before
Nov 19 09:33:59 atlas kernel: [ 2925.348933] NVRM: the NVIDIA kernel module is unloaded.

Nothing specific could be linked to these random freezes.

I have seen that other people have similar issues when PowerMizer changes state. How could I check the log of those states to compare it with syslog?

The output of the last run of nvidia-bug-report.sh is attached.
nvidia-bug-report.log.gz (1.03 MB)

generix · November 19, 2019, 9:37am

XID 79 on a notebook rather points to a hw failure. You could monitor temperatures in case of clogged cooling, otherwise try installing Windows and see if you can replicate the issue, then RMA if possible.

corentin.barbu · November 19, 2019, 9:49am

Hi generix,
I doubt it is hardware related as

an other person on my team with the same machine adquired at 2 months interval has the same issue.
the two machines are new (<1 year)
the bug usually happen within an hour of starting or waking up the computer

Anyway, how can I monitor temperatures for the graphic card ?

I can’t install windows, no licence. What is an RMA ?

generix · November 19, 2019, 9:59am

RMA=Return to Manufacturing Assembly, getting it replaced if still under warranty.
There’s one firmware/cpu bug I know of, mainly affecting a specific HP notebook, you could try if that’s the case on your hw, too:
Please set the kernel parameter
intel_idle.max_cstate=1
and check if the gpu still falls off the bus.
You can monitor temperatures either using nvidia-smi or nvidia-settings.

corentin.barbu · November 19, 2019, 2:27pm

Ok, I’m trying it, I’ll come back next time it bugs…

generix · November 19, 2019, 2:42pm

Looking for a bios update is also worthwhile. Also, another approach would be to disable “c-state control” in bios.

corentin.barbu · November 19, 2019, 7:35pm

I had already updated to the latest BIOS and unfortunately I already had a freeze with the intel_idle.max_cstate=1 kernel model.
I’m currently trying to change the parameters of the GPU. Currently testing if enabling Persistence changes something.

corentin.barbu · November 20, 2019, 7:34am

Unfortunately enabling Persistence didn’t prevent freeze.

One more freeze this morning after wake up from suspend with battery low: freeze within 2 minutes of waking up, like Monday on relatively low battery power. Might be a thing.

Trying setting powermizer level to 1 (Prefer maximum performance) by setting
/usr/bin/nvidia-settings -a “[gpu:0]/GpuPowerMizerMode=1”
as a startup application as described in How to Permanently Set NVIDIA PowerMizer Settings in Ubuntu | rastating.github.io

I can’t find any nvidia specific logs (to check possible status changes related to the bug) are they any that can be activated?

generix · November 20, 2019, 8:15am

All nvidia driver messages are dmesg. That won’t help you, though, since XID 79 is a low level HW issue. The gpu either shut down due to overheating/power issues or got detached from the bus due to pcie bus problems. Nothing the driver can do but telling that the gpu is gone. The gpu obviously also can’t tell you why because it can’t communicate with the driver anymore.

corentin.barbu · November 22, 2019, 10:24am

In any case, desactivating completely the nvidia card seems to completely prevent the bug :
nvidia-settings > PRIME Profiles > Select the GPU you would like to use > Intel (Power Saving Mode)

generix · November 22, 2019, 11:31am

If you turn the gpu off, of course this prevents it from turning itself off unexpectedly.
Did you try to disable “c-states control” in bios?

corentin.barbu · December 4, 2019, 2:22pm

I didn’t try to disable “c-states control” in bios but setting the driver to max performance, as described in my post of the Posted 11/20/2019 07:34 AM worked flawlessly … but has been draining the battery at an incredible rate!
I’ll try the “c-states control” trick

corentin.barbu · December 4, 2019, 2:22pm

I didn’t try to disable “c-states control” in bios but setting the driver to max performance, as described in my post of the Posted 11/20/2019 07:34 AM worked flawlessly … but has been draining the battery at an incredible rate!
I’ll try the “c-states control” trick.

corentin.barbu · December 4, 2019, 2:23pm

I didn’t try to disable “c-states control” in bios but setting the driver to max performance, as described in my post of the Posted 11/20/2019 07:34 AM worked flawlessly … but has been draining the battery at an incredible rate!
I’ll try the “c-states control” trick

corentin.barbu · December 4, 2019, 2:23pm

I didn’t try to disable “c-states control” in bios but setting the driver to max performance, as described in my post of the Posted 11/20/2019 07:34 AM worked flawlessly … but has been draining the battery at an incredible rate!
I’ll try the “c-states control” trick