Nvidia Driver Hangs frequently, only recovers after hard poweroff (logs attached)

Hi,

I want to report a bug I’ve seen multiple times over the past 2 days on a fresh Manjaro Gnome Linux install using the proprietary NVIDIA drivers.

Here are my system specs:

ankit@desktop-linux:~/crashes$ cat /etc/lsb-release
DISTRIB_ID=ManjaroLinux
DISTRIB_RELEASE=21.1.6
DISTRIB_CODENAME=Pahvo
DISTRIB_DESCRIPTION="Manjaro Linux"


ankit@desktop-linux:~/crashes$ nvidia-smi
Tue Nov  9 22:21:04 2021       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.63.01    Driver Version: 470.63.01    CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:29:00.0  On |                  N/A |
| 20%   50C    P0    42W / 180W |    923MiB /  8085MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      4518      G   /usr/lib/Xorg                     492MiB |
|    0   N/A  N/A      4584      G   /usr/bin/gnome-shell              131MiB |
|    0   N/A  N/A      5158      G   ...1/usr/lib/firefox/firefox      286MiB |
|    0   N/A  N/A      8526      G   ...1/usr/lib/firefox/firefox        2MiB |
|    0   N/A  N/A      8872      G   ...1/usr/lib/firefox/firefox        2MiB |
|    0   N/A  N/A      9326      G   ...1/usr/lib/firefox/firefox        2MiB |
+-----------------------------------------------------------------------------+

Periodically, seems like once or twice a day, my display completely freezes, and the mouse/keyboard become unresponsive. Pressing Alt+F3 etc. to switch X FBs has no effect.

I was able to SSH into the machine and tried to collect all the info I could, though I am not a graphics driver expert. This was all collected while the display was still hung and I was SSH’ed in:


  • Restarting GDM with systemctl restart gdm has no effect. The Xorg process was in an uninterruptible sleep and cannot be terminated with `kill -9:
[ankit@desktop-linux crashes]$ ps aux | grep X
root       50905  0.8  0.2 25352196 70668 tty3   D+   21:02   0:09 /usr/lib/Xorg vt3 -displayfd 3 -auth /run/user/1000/gdm/Xauthority -nolisten tcp -background none -noreset -keeptty -novtswitch -verbose 3
  • Here is the output of nvidia-bug-report.sh --safe-mode --extra-system-data. It hung but had partial output: nvidia-bug-report.log.nov9-910pm.gz (593 Bytes)

  • Output of `cat /proc/50905/status:

Name:	Xorg
Umask:	0022
State:	D (disk sleep)
Tgid:	50905
Ngid:	0
Pid:	50905
PPid:	1
TracerPid:	0
Uid:	1000	0	0	0
Gid:	1000	1000	1000	1000
FDSize:	256
Groups:	90 98 963 984 987 991 994 996 998 1000 
NStgid:	50905
NSpid:	50905
NSpgid:	50903
NSsid:	50903
VmPeak:	25354644 kB
VmSize:	25352196 kB
VmLck:	       0 kB
VmPin:	       0 kB
VmHWM:	   74424 kB
VmRSS:	   70668 kB
RssAnon:	   27312 kB
RssFile:	   34548 kB
RssShmem:	    8808 kB
VmData:	   28384 kB
VmStk:	     204 kB
VmExe:	    1672 kB
VmLib:	   51276 kB
VmPTE:	     288 kB
VmSwap:	       0 kB
HugetlbPages:	       0 kB
CoreDumping:	0
THP_enabled:	1
Threads:	1
SigQ:	9/128161
SigPnd:	0000000000000100
ShdPnd:	0000000000006301
SigBlk:	0000000000000000
SigIgn:	0000000000001000
SigCgt:	00000001c18066ef
CapInh:	0000000000000000
CapPrm:	000001ffffffffff
CapEff:	000001ffffffffff
CapBnd:	000001ffffffffff
CapAmb:	0000000000000000
NoNewPrivs:	0
Seccomp:	0
Seccomp_filters:	0
Speculation_Store_Bypass:	thread vulnerable
SpeculationIndirectBranch:	conditional enabled
Cpus_allowed:	00000fff
Cpus_allowed_list:	0-11
Mems_allowed:	00000001
Mems_allowed_list:	0
voluntary_ctxt_switches:	478628
nonvoluntary_ctxt_switches:	1178

  • Here is the Xorg.0.log at around 9:11PM: Xorg.0.nov9_910PM.txt (39.5 KB)

  • Here is the output of cat /proc/50905/syscall showing X11 was hung on an ioctl:

[ankit@desktop-linux crashes]$ sudo cat /proc/50905/syscall > x11_stack_trace.txt 
16 0x13 0xc0106d00 0x7fff5dedda10 0x7fff5dee17e0 0x1 0x1 0x7fff5dedda08 0x7f18ed5ad59b
  • Here is the output of cat /proc/50905/stack showing the stack trace at a (somewhere in nvidia driver land?):
[ankit@desktop-linux crashes]$ sudo cat /proc/50905/stack > x11_stack_trace.txt 
[<0>] down+0x3b/0x50    
[<0>] os_acquire_semaphore+0x31/0x40 [nvidia]    
[<0>] _nv035255rm+0xc/0x30 [nvidia]    

  • Here is the dmesg output starting from the first error i saw (showing a segfault in X11 and an error in NVIDIA?):
[ 5173.083790] gnome-shell[47475]: segfault at 0 ip 00007fedd4529f43 sp 00007ffc0ab887e0 error 4 in libX11.so.6.4.0[7fedd4505000+8a000]
[ 5173.083802] Code: 48 89 f5 53 48 89 d3 48 83 ec 58 64 48 8b 04 25 28 00 00 00 48 89 44 24 48 48 8b 87 30 0a 00 00 48 8b 97 f8 00 00 00 83 e2 01 <48> 8b 30 48 89 34 24 0f 85 53 01 00 00 48 8b 97 a8 00 00 00 49 89
[ 5173.083842] audit: type=1701 audit(1636520521.610:1070): auid=1000 uid=1000 gid=1000 ses=4 subj==unconfined pid=47475 comm="gnome-shell" exe="/usr/bin/gnome-shell" sig=11 res=1
[ 5173.117740] audit: type=1334 audit(1636520521.643:1071): prog-id=120 op=LOAD
[ 5173.117854] audit: type=1334 audit(1636520521.643:1072): prog-id=121 op=LOAD
[ 5173.117904] audit: type=1334 audit(1636520521.643:1073): prog-id=122 op=LOAD
[ 5173.119025] audit: type=1130 audit(1636520521.643:1074): pid=1 uid=0 auid=4294967295 ses=4294967295 subj==unconfined msg='unit=systemd-coredump@2-50488-0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 5173.625374] audit: type=1131 audit(1636520522.150:1075): pid=1 uid=0 auid=4294967295 ses=4294967295 subj==unconfined msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 5173.823393] audit: type=1334 audit(1636520522.350:1076): prog-id=119 op=UNLOAD
[ 5173.823406] audit: type=1334 audit(1636520522.350:1077): prog-id=118 op=UNLOAD
[ 5173.823410] audit: type=1334 audit(1636520522.350:1078): prog-id=117 op=UNLOAD
[ 5178.517411] kauditd_printk_skb: 28 callbacks suppressed
[ 5178.517413] audit: type=1334 audit(1636520527.043:1101): prog-id=124 op=LOAD
[ 5178.517535] audit: type=1334 audit(1636520527.043:1102): prog-id=125 op=LOAD
[ 5178.517599] audit: type=1334 audit(1636520527.043:1103): prog-id=126 op=LOAD
[ 5178.571499] audit: type=1130 audit(1636520527.096:1104): pid=1 uid=0 auid=4294967295 ses=4294967295 subj==unconfined msg='unit=systemd-localed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 5178.621397] audit: type=1130 audit(1636520527.146:1105): pid=1 uid=0 auid=4294967295 ses=4294967295 subj==unconfined msg='unit=geoclue comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 5178.758896] audit: type=1334 audit(1636520527.283:1106): prog-id=127 op=LOAD
[ 5178.758965] audit: type=1334 audit(1636520527.283:1107): prog-id=128 op=LOAD
[ 5178.758991] audit: type=1334 audit(1636520527.283:1108): prog-id=129 op=LOAD
[ 5178.814890] audit: type=1130 audit(1636520527.340:1109): pid=1 uid=0 auid=4294967295 ses=4294967295 subj==unconfined msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 5178.831390] rfkill: input handler disabled
[ 5179.294558] audit: type=1334 audit(1636520527.820:1110): prog-id=130 op=LOAD
[ 5186.446469] kauditd_printk_skb: 3 callbacks suppressed
[ 5186.446474] audit: type=1100 audit(1636520534.970:1114): pid=50868 uid=0 auid=4294967295 ses=4294967295 subj==unconfined msg='op=PAM:authentication grantors=pam_shells,pam_faillock,pam_permit,pam_faillock,pam_gnome_keyring acct="ankit" exe="/usr/lib/gdm-session-worker" hostname=desktop-linux addr=? terminal=/dev/tty1 res=success'
[ 5186.447861] audit: type=1101 audit(1636520534.973:1115): pid=50868 uid=0 auid=4294967295 ses=4294967295 subj==unconfined msg='op=PAM:accounting grantors=pam_access,pam_unix,pam_permit,pam_time acct="ankit" exe="/usr/lib/gdm-session-worker" hostname=desktop-linux addr=? terminal=/dev/tty1 res=success'
[ 5186.448698] audit: type=1103 audit(1636520534.973:1116): pid=50868 uid=0 auid=4294967295 ses=4294967295 subj==unconfined msg='op=PAM:setcred grantors=pam_shells,pam_faillock,pam_permit,pam_faillock,pam_gnome_keyring acct="ankit" exe="/usr/lib/gdm-session-worker" hostname=desktop-linux addr=? terminal=/dev/tty1 res=success'
[ 5186.451341] audit: type=1006 audit(1636520534.976:1117): pid=50868 uid=0 subj==unconfined old-auid=4294967295 auid=1000 tty=(none) old-ses=4294967295 ses=27 res=1
[ 5186.451347] audit: type=1300 audit(1636520534.976:1117): arch=c000003e syscall=1 success=yes exit=4 a0=a a1=7fff25e30600 a2=4 a3=3e8 items=0 ppid=629 pid=50868 auid=1000 uid=0 gid=1000 euid=0 suid=0 fsuid=0 egid=1000 sgid=1000 fsgid=1000 tty=(none) ses=27 comm="gdm-session-wor" exe="/usr/lib/gdm-session-worker" subj==unconfined key=(null)
[ 5186.451350] audit: type=1327 audit(1636520534.976:1117): proctitle=67646D2D73657373696F6E2D776F726B6572205B70616D2F67646D2D70617373776F72645D
[ 5186.503043] audit: type=1105 audit(1636520535.026:1118): pid=50868 uid=0 auid=1000 ses=27 subj==unconfined msg='op=PAM:session_open grantors=pam_loginuid,pam_keyinit,pam_limits,pam_unix,pam_permit,pam_mail,pam_systemd,pam_env,pam_gnome_keyring acct="ankit" exe="/usr/lib/gdm-session-worker" hostname=desktop-linux addr=? terminal=/dev/tty3 res=success'
[ 5186.973288] rfkill: input handler enabled
[ 5188.908607] rfkill: input handler disabled
[ 5188.957034] audit: type=1701 audit(1636520537.483:1119): auid=1000 uid=1000 gid=1000 ses=4 subj==unconfined pid=51268 comm="touchegg" exe="/usr/bin/touchegg" sig=6 res=1
[ 5189.000572] audit: type=1334 audit(1636520537.526:1120): prog-id=131 op=LOAD
[ 5189.000640] audit: type=1334 audit(1636520537.526:1121): prog-id=132 op=LOAD
[ 5200.434336] kauditd_printk_skb: 8 callbacks suppressed
[ 5200.434340] audit: type=1131 audit(1636520548.960:1130): pid=1 uid=0 auid=4294967295 ses=4294967295 subj==unconfined msg='unit=user@120 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 5200.501087] audit: type=1131 audit(1636520549.026:1131): pid=1 uid=0 auid=4294967295 ses=4294967295 subj==unconfined msg='unit=user-runtime-dir@120 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 5203.937551] audit: type=1334 audit(1636520552.463:1132): prog-id=134 op=LOAD
[ 5203.937657] audit: type=1334 audit(1636520552.463:1133): prog-id=135 op=LOAD
[ 5203.937713] audit: type=1334 audit(1636520552.463:1134): prog-id=136 op=LOAD
[ 5203.993682] audit: type=1130 audit(1636520552.520:1135): pid=1 uid=0 auid=4294967295 ses=4294967295 subj==unconfined msg='unit=systemd-timedated comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 5210.005228] audit: type=1131 audit(1636520558.530:1136): pid=1 uid=0 auid=4294967295 ses=4294967295 subj==unconfined msg='unit=fprintd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 5210.226874] audit: type=1334 audit(1636520558.753:1137): prog-id=130 op=UNLOAD
[ 5218.978492] audit: type=1131 audit(1636520567.503:1138): pid=1 uid=0 auid=4294967295 ses=4294967295 subj==unconfined msg='unit=systemd-localed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 5219.200218] audit: type=1334 audit(1636520567.726:1139): prog-id=126 op=UNLOAD
[ 5219.200223] audit: type=1334 audit(1636520567.726:1140): prog-id=125 op=UNLOAD
[ 5219.200225] audit: type=1334 audit(1636520567.726:1141): prog-id=124 op=UNLOAD
[ 5231.951388] audit: type=1326 audit(1636520580.476:1142): auid=1000 uid=1000 gid=1000 ses=4 subj==snap.firefox.firefox (enforce) pid=52553 comm="GeckoMain" exe="/snap/firefox/701/usr/lib/firefox/firefox" sig=0 arch=c000003e syscall=314 compat=0 ip=0x7f556cdeb89d code=0x50000
[ 5234.028508] audit: type=1131 audit(1636520582.553:1143): pid=1 uid=0 auid=4294967295 ses=4294967295 subj==unconfined msg='unit=systemd-timedated comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 5234.088732] audit: type=1334 audit(1636520582.613:1144): prog-id=136 op=UNLOAD
[ 5234.088738] audit: type=1334 audit(1636520582.613:1145): prog-id=135 op=UNLOAD
[ 5234.088741] audit: type=1334 audit(1636520582.613:1146): prog-id=134 op=UNLOAD
[ 5235.045422] audit: type=1131 audit(1636520583.570:1147): pid=1 uid=0 auid=4294967295 ses=4294967295 subj==unconfined msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 5235.200066] audit: type=1334 audit(1636520583.726:1148): prog-id=129 op=UNLOAD
[ 5235.200076] audit: type=1334 audit(1636520583.726:1149): prog-id=128 op=UNLOAD
[ 5235.200082] audit: type=1334 audit(1636520583.726:1150): prog-id=127 op=UNLOAD
[ 5238.982866] audit: type=1131 audit(1636520587.506:1151): pid=1 uid=0 auid=4294967295 ses=4294967295 subj==unconfined msg='unit=geoclue comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 5281.024185] audit: type=1400 audit(1636520629.550:1152): apparmor="DENIED" operation="open" profile="snap.firefox.firefox" name="/proc/54033/environ" pid=52553 comm=427265616B70616420536572766572 requested_mask="r" denied_mask="r" fsuid=1000 ouid=1000
[ 5281.024195] audit: type=1300 audit(1636520629.550:1152): arch=c000003e syscall=257 success=no exit=-13 a0=ffffff9c a1=7f551f35a7a0 a2=80000 a3=0 items=0 ppid=50971 pid=52553 auid=1000 uid=1000 gid=1000 euid=1000 suid=1000 fsuid=1000 egid=1000 sgid=1000 fsgid=1000 tty=(none) ses=4 comm=427265616B70616420536572766572 exe="/snap/firefox/701/usr/lib/firefox/firefox" subj==snap.firefox.firefox (enforce) key=(null)
[ 5281.024199] audit: type=1327 audit(1636520629.550:1152): proctitle="/snap/firefox/701/usr/lib/firefox/firefox"
[ 5281.027500] MediaSu~isor #2[54033]: segfault at 0 ip 00007f5fca679268 sp 00007f5fc1f5a5f0 error 4 in libxul.so[7f5fc9f7a000+53d9000]
[ 5281.027515] Code: 00 00 55 41 57 41 56 41 55 41 54 53 48 83 ec 18 89 d1 41 89 f7 49 89 fc 64 48 8b 04 25 28 00 00 00 48 89 44 24 10 48 8b 47 48 <8b> 30 8d 7e 03 83 e7 fc 41 8b 44 24 50 01 f8 31 d2 f7 f1 41 83 c7
[ 5281.027547] audit: type=1701 audit(1636520629.553:1153): auid=1000 uid=1000 gid=1000 ses=4 subj==snap.firefox.firefox (enforce) pid=53704 comm=4D6564696153757E69736F72202332 exe="/snap/firefox/701/usr/lib/firefox/firefox" sig=11 res=1
[ 5281.080798] audit: type=1334 audit(1636520629.606:1154): prog-id=137 op=LOAD
[ 5281.080940] audit: type=1334 audit(1636520629.606:1155): prog-id=138 op=LOAD
[ 5281.081013] audit: type=1334 audit(1636520629.606:1156): prog-id=139 op=LOAD
[ 5281.082061] audit: type=1130 audit(1636520629.606:1157): pid=1 uid=0 auid=4294967295 ses=4294967295 subj==unconfined msg='unit=systemd-coredump@4-54218-0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 5281.740699] audit: type=1131 audit(1636520630.266:1158): pid=1 uid=0 auid=4294967295 ses=4294967295 subj==unconfined msg='unit=systemd-coredump@4-54218-0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 5395.508240] kauditd_printk_skb: 3 callbacks suppressed
[ 5395.508243] audit: type=1101 audit(1636520744.033:1162): pid=54948 uid=0 auid=4294967295 ses=4294967295 subj==docker-default (enforce) msg='op=PAM:accounting grantors=pam_permit acct="ankit" exe="/usr/sbin/sshd" hostname=172.18.0.1 addr=172.18.0.1 terminal=ssh res=success'
[ 5395.509154] audit: type=1103 audit(1636520744.033:1163): pid=54948 uid=0 auid=4294967295 ses=4294967295 subj==docker-default (enforce) msg='op=PAM:setcred grantors=pam_permit,pam_cap acct="ankit" exe="/usr/sbin/sshd" hostname=172.18.0.1 addr=172.18.0.1 terminal=ssh res=success'
[ 5395.509203] audit: type=1006 audit(1636520744.033:1164): pid=54948 uid=0 subj==docker-default (enforce) old-auid=4294967295 auid=1000 tty=(none) old-ses=4294967295 ses=28 res=1
[ 5395.509206] audit: type=1300 audit(1636520744.033:1164): arch=c000003e syscall=1 success=yes exit=4 a0=3 a1=7ffddd5bd130 a2=4 a3=7f2d19e1e371 items=0 ppid=39525 pid=54948 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=28 comm="sshd" exe="/usr/sbin/sshd" subj==docker-default (enforce) key=(null)
[ 5395.509214] audit: type=1327 audit(1636520744.033:1164): proctitle=737368643A20616E6B6974205B707269765D
[ 5395.513649] audit: type=1105 audit(1636520744.040:1165): pid=54948 uid=0 auid=1000 ses=28 subj==docker-default (enforce) msg='op=PAM:session_open grantors=pam_selinux,pam_loginuid,pam_permit,pam_umask,pam_unix,pam_mail,pam_limits,pam_env,pam_env,pam_selinux acct="ankit" exe="/usr/sbin/sshd" hostname=172.18.0.1 addr=172.18.0.1 terminal=ssh res=success'
[ 5395.514505] audit: type=1103 audit(1636520744.040:1166): pid=54964 uid=0 auid=1000 ses=28 subj==docker-default (enforce) msg='op=PAM:setcred grantors=pam_permit,pam_cap acct="ankit" exe="/usr/sbin/sshd" hostname=172.18.0.1 addr=172.18.0.1 terminal=ssh res=success'
[ 5545.334128] audit: type=1334 audit(1636520893.860:1167): prog-id=140 op=LOAD
[ 5545.334264] audit: type=1334 audit(1636520893.860:1168): prog-id=141 op=LOAD
[ 5545.334321] audit: type=1334 audit(1636520893.860:1169): prog-id=142 op=LOAD
[ 5545.388622] audit: type=1130 audit(1636520893.913:1170): pid=1 uid=0 auid=4294967295 ses=4294967295 subj==unconfined msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 5566.987738] BUG: kernel NULL pointer dereference, address: 00000000000001e9
[ 5566.987744] #PF: supervisor read access in kernel mode
[ 5566.987745] #PF: error_code(0x0000) - not-present page
[ 5566.987747] PGD 0 P4D 0 
[ 5566.987749] Oops: 0000 [#1] PREEMPT SMP NOPTI
[ 5566.987751] CPU: 1 PID: 50601 Comm: irq/114-nvidia Tainted: P           OE     5.13.19-2-MANJARO #1
[ 5566.987753] Hardware name: Micro-Star International Co., Ltd. MS-7B87/B450M GAMING PLUS (MS-7B87), BIOS 1.C4 06/10/2020
[ 5566.987754] RIP: 0010:_nv034124rm+0x34/0x1d0 [nvidia]
[ 5566.988067] Code: 89 f5 41 54 53 48 83 ed 10 48 8b b7 10 23 00 00 48 89 fb 89 4d 0c 45 89 c4 e8 e8 b4 e1 ff 4d 85 ed 49 89 c6 0f 84 ec 00 00 00 <41> f7 85 e8 01 00 00 fd ff ff ff 0f 85 0b 01 00 00 49 8b 75 68 48
[ 5566.988068] RSP: 0018:ffff9e82ceff7d18 EFLAGS: 00010202
[ 5566.988070] RAX: ffff90a714a7f008 RBX: ffff90a5ebe5c008 RCX: 0000000000000001
[ 5566.988071] RDX: 0000000000400100 RSI: ffff90a714a7f008 RDI: ffff90a5ebe5c008
[ 5566.988072] RBP: ffff90a5c4b4ab80 R08: 0000000000000020 R09: 0000000000000000
[ 5566.988073] R10: ffffffffc110c850 R11: ffff90a5c4b4aa08 R12: 0000000000000020
[ 5566.988074] R13: 0000000000000001 R14: ffff90a714a7f008 R15: 0000000000400100
[ 5566.988075] FS:  0000000000000000(0000) GS:ffff90acbea40000(0000) knlGS:0000000000000000
[ 5566.988077] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 5566.988078] CR2: 00000000000001e9 CR3: 000000014e0b2000 CR4: 0000000000350ee0
[ 5566.988079] Call Trace:
[ 5566.988082]  ? _nv009834rm+0xdb/0x1a0 [nvidia]
[ 5566.988406]  ? _nv009822rm+0x176/0x370 [nvidia]
[ 5566.988725]  ? _nv024541rm+0xb3d/0x12b0 [nvidia]
[ 5566.989048]  ? _nv029374rm+0x138/0xd90 [nvidia]
[ 5566.989371]  ? _nv029382rm+0x161/0x440 [nvidia]
[ 5566.989684]  ? _nv000725rm+0xb1/0x250 [nvidia]
[ 5566.989906]  ? disable_irq_nosync+0x10/0x10
[ 5566.989910]  ? rm_isr_bh+0x1c/0x60 [nvidia]
[ 5566.990128]  ? nvidia_isr_kthread_bh+0x1b/0x40 [nvidia]
[ 5566.990314]  ? irq_thread_fn+0x20/0x60
[ 5566.990316]  ? irq_thread+0xf2/0x1a0
[ 5566.990317]  ? irq_finalize_oneshot.part.0+0xe0/0xe0
[ 5566.990319]  ? irq_thread_check_affinity+0xd0/0xd0
[ 5566.990321]  ? kthread+0x133/0x160
[ 5566.990323]  ? set_kthread_struct+0x40/0x40
[ 5566.990324]  ? ret_from_fork+0x22/0x30
[ 5566.990327] Modules linked in: veth xt_nat xt_tcpudp xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xt_addrtype iptable_filter iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c br_netfilter bridge stp llc overlay rfcomm cmac algif_hash algif_skcipher af_alg mousedev joydev uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 snd_usb_audio videobuf2_common mt76x2u mt76x2_common videodev mt76x02_usb mt76_usb snd_usbmidi_lib mt76x02_lib snd_rawmidi snd_seq_device mc mt76 usbhid bnep btusb btrtl btbcm btintel bluetooth uas usb_storage ecdh_generic ecc qrtr ns iwlmvm mac80211 squashfs nvidia_drm(POE) snd_hda_codec_realtek nvidia_modeset(POE) intel_rapl_msr snd_hda_codec_generic intel_rapl_common ledtrig_audio snd_hda_codec_hdmi libarc4 snd_hda_intel edac_mce_amd snd_intel_dspcfg snd_intel_sdw_acpi snd_hda_codec kvm_amd ccp snd_hda_core iwlwifi rng_core wmi_bmof snd_hwdep kvm vfat nvidia(POE) cfg80211 fat drm_kms_helper irqbypass crct10dif_pclmul
[ 5566.990366]  crc32_pclmul ghash_clmulni_intel snd_pcm aesni_intel r8169 snd_timer cec crypto_simd snd realtek syscopyarea cryptd sysfillrect sysimgblt mdio_devres sp5100_tco rapl pcspkr zenpower(OE) libphy i2c_piix4 rfkill soundcore fb_sys_fops ryzen_smu(OE) wmi gpio_amdpt pinctrl_amd gpio_generic mac_hid acpi_cpufreq loop drm agpgart fuse crypto_user ip_tables x_tables ext4 crc32c_generic crc16 mbcache jbd2 crc32c_intel xhci_pci
[ 5566.990388] CR2: 00000000000001e9
[ 5566.990389] ---[ end trace 7bef7c91870e73dd ]---
[ 5566.990390] RIP: 0010:_nv034124rm+0x34/0x1d0 [nvidia]
[ 5566.990683] Code: 89 f5 41 54 53 48 83 ed 10 48 8b b7 10 23 00 00 48 89 fb 89 4d 0c 45 89 c4 e8 e8 b4 e1 ff 4d 85 ed 49 89 c6 0f 84 ec 00 00 00 <41> f7 85 e8 01 00 00 fd ff ff ff 0f 85 0b 01 00 00 49 8b 75 68 48
[ 5566.990685] RSP: 0018:ffff9e82ceff7d18 EFLAGS: 00010202
[ 5566.990686] RAX: ffff90a714a7f008 RBX: ffff90a5ebe5c008 RCX: 0000000000000001
[ 5566.990687] RDX: 0000000000400100 RSI: ffff90a714a7f008 RDI: ffff90a5ebe5c008
[ 5566.990688] RBP: ffff90a5c4b4ab80 R08: 0000000000000020 R09: 0000000000000000
[ 5566.990689] R10: ffffffffc110c850 R11: ffff90a5c4b4aa08 R12: 0000000000000020
[ 5566.990689] R13: 0000000000000001 R14: ffff90a714a7f008 R15: 0000000000400100
[ 5566.990690] FS:  0000000000000000(0000) GS:ffff90acbea40000(0000) knlGS:0000000000000000
[ 5566.990692] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 5566.990693] CR2: 00000000000001e9 CR3: 000000014e0b2000 CR4: 0000000000350ee0
[ 5566.990706] BUG: unable to handle page fault for address: 00000000a0d08190
[ 5566.990707] #PF: supervisor instruction fetch in kernel mode
[ 5566.990708] #PF: error_code(0x0010) - not-present page
[ 5566.990709] PGD 0 P4D 0 
[ 5566.990710] Oops: 0010 [#2] PREEMPT SMP NOPTI
[ 5566.990712] CPU: 1 PID: 50601 Comm: irq/114-nvidia Tainted: P      D    OE     5.13.19-2-MANJARO #1
[ 5566.990713] Hardware name: Micro-Star International Co., Ltd. MS-7B87/B450M GAMING PLUS (MS-7B87), BIOS 1.C4 06/10/2020
[ 5566.990714] RIP: 0010:0xa0d08190
[ 5566.990715] Code: Unable to access opcode bytes at RIP 0xa0d08166.
[ 5566.990716] RSP: 0018:ffff9e82ceff7ea8 EFLAGS: 00010286
[ 5566.990717] RAX: 00000000a0d08190 RBX: 0000000000000000 RCX: 0000000000000000
[ 5566.990718] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff9e82ceff7ed0
[ 5566.990719] RBP: ffff90a704a49f40 R08: 0000000000000001 R09: 0000000000000000
[ 5566.990720] R10: 0000000000000001 R11: ffff90acdf326c00 R12: ffff90a704a4a7b4
[ 5566.990721] R13: ffff9e82ceff7c68 R14: ffff90a704a49f40 R15: 0000000000000000
[ 5566.990721] FS:  0000000000000000(0000) GS:ffff90acbea40000(0000) knlGS:0000000000000000
[ 5566.990723] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 5566.990723] CR2: 00000000a0d08190 CR3: 000000014e0b2000 CR4: 0000000000350ee0
[ 5566.990724] Call Trace:
[ 5566.990725]  ? task_work_run+0x5c/0x90
[ 5566.990728]  ? do_exit+0x375/0xae0
[ 5566.990730]  ? irq_thread_check_affinity+0xd0/0xd0
[ 5566.990732]  ? rewind_stack_do_exit+0x17/0x17
[ 5566.990735] Modules linked in: veth xt_nat xt_tcpudp xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xt_addrtype iptable_filter iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c br_netfilter bridge stp llc overlay rfcomm cmac algif_hash algif_skcipher af_alg mousedev joydev uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 snd_usb_audio videobuf2_common mt76x2u mt76x2_common videodev mt76x02_usb mt76_usb snd_usbmidi_lib mt76x02_lib snd_rawmidi snd_seq_device mc mt76 usbhid bnep btusb btrtl btbcm btintel bluetooth uas usb_storage ecdh_generic ecc qrtr ns iwlmvm mac80211 squashfs nvidia_drm(POE) snd_hda_codec_realtek nvidia_modeset(POE) intel_rapl_msr snd_hda_codec_generic intel_rapl_common ledtrig_audio snd_hda_codec_hdmi libarc4 snd_hda_intel edac_mce_amd snd_intel_dspcfg snd_intel_sdw_acpi snd_hda_codec kvm_amd ccp snd_hda_core iwlwifi rng_core wmi_bmof snd_hwdep kvm vfat nvidia(POE) cfg80211 fat drm_kms_helper irqbypass crct10dif_pclmul
[ 5566.990758]  crc32_pclmul ghash_clmulni_intel snd_pcm aesni_intel r8169 snd_timer cec crypto_simd snd realtek syscopyarea cryptd sysfillrect sysimgblt mdio_devres sp5100_tco rapl pcspkr zenpower(OE) libphy i2c_piix4 rfkill soundcore fb_sys_fops ryzen_smu(OE) wmi gpio_amdpt pinctrl_amd gpio_generic mac_hid acpi_cpufreq loop drm agpgart fuse crypto_user ip_tables x_tables ext4 crc32c_generic crc16 mbcache jbd2 crc32c_intel xhci_pci
[ 5566.990772] CR2: 00000000a0d08190
[ 5566.990772] ---[ end trace 7bef7c91870e73de ]---
[ 5566.990773] RIP: 0010:_nv034124rm+0x34/0x1d0 [nvidia]
[ 5566.991067] Code: 89 f5 41 54 53 48 83 ed 10 48 8b b7 10 23 00 00 48 89 fb 89 4d 0c 45 89 c4 e8 e8 b4 e1 ff 4d 85 ed 49 89 c6 0f 84 ec 00 00 00 <41> f7 85 e8 01 00 00 fd ff ff ff 0f 85 0b 01 00 00 49 8b 75 68 48
[ 5566.991069] RSP: 0018:ffff9e82ceff7d18 EFLAGS: 00010202
[ 5566.991070] RAX: ffff90a714a7f008 RBX: ffff90a5ebe5c008 RCX: 0000000000000001
[ 5566.991071] RDX: 0000000000400100 RSI: ffff90a714a7f008 RDI: ffff90a5ebe5c008
[ 5566.991072] RBP: ffff90a5c4b4ab80 R08: 0000000000000020 R09: 0000000000000000
[ 5566.991072] R10: ffffffffc110c850 R11: ffff90a5c4b4aa08 R12: 0000000000000020
[ 5566.991073] R13: 0000000000000001 R14: ffff90a714a7f008 R15: 0000000000400100
[ 5566.991074] FS:  0000000000000000(0000) GS:ffff90acbea40000(0000) knlGS:0000000000000000
[ 5566.991075] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 5566.991076] CR2: 00000000a0d08190 CR3: 000000014e0b2000 CR4: 0000000000350ee0
[ 5566.991077] Fixing recursive fault but reboot is needed!
[ 5575.438581] audit: type=1131 audit(1636520923.963:1171): pid=1 uid=0 auid=4294967295 ses=4294967295 subj==unconfined msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 5575.649936] audit: type=1334 audit(1636520924.176:1172): prog-id=142 op=UNLOAD
[ 5575.649944] audit: type=1334 audit(1636520924.176:1173): prog-id=141 op=UNLOAD
[ 5575.649947] audit: type=1334 audit(1636520924.176:1174): prog-id=140 op=UNLOAD
[ 5612.582161] audit: type=1101 audit(1636520961.106:1175): pid=57671 uid=0 auid=4294967295 ses=4294967295 subj==unconfined msg='op=PAM:accounting grantors=pam_access,pam_unix,pam_permit,pam_time acct="ankit" exe="/usr/bin/sshd" hostname=192.168.1.52 addr=192.168.1.52 terminal=ssh res=success'
[ 5612.583231] audit: type=1103 audit(1636520961.106:1176): pid=57671 uid=0 auid=4294967295 ses=4294967295 subj==unconfined msg='op=PAM:setcred grantors=pam_shells,pam_faillock,pam_permit,pam_env,pam_faillock acct="ankit" exe="/usr/bin/sshd" hostname=192.168.1.52 addr=192.168.1.52 terminal=ssh res=success'
[ 5612.583311] audit: type=1006 audit(1636520961.110:1177): pid=57671 uid=0 subj==unconfined old-auid=4294967295 auid=1000 tty=(none) old-ses=4294967295 ses=29 res=1
[ 5612.583323] audit: type=1300 audit(1636520961.110:1177): arch=c000003e syscall=1 success=yes exit=4 a0=3 a1=7fffebea20c0 a2=4 a3=3e8 items=0 ppid=619 pid=57671 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=29 comm="sshd" exe="/usr/bin/sshd" subj==unconfined key=(null)
[ 5612.583327] audit: type=1327 audit(1636520961.110:1177): proctitle=737368643A20616E6B6974205B707269765D
[ 5612.630425] audit: type=1105 audit(1636520961.156:1178): pid=57671 uid=0 auid=1000 ses=29 subj==unconfined msg='op=PAM:session_open grantors=pam_loginuid,pam_keyinit,pam_limits,pam_unix,pam_permit,pam_mail,pam_systemd,pam_env acct="ankit" exe="/usr/bin/sshd" hostname=192.168.1.52 addr=192.168.1.52 terminal=ssh res=success'
[ 5612.631442] audit: type=1103 audit(1636520961.156:1179): pid=57697 uid=0 auid=1000 ses=29 subj==unconfined msg='op=PAM:setcred grantors=pam_shells,pam_faillock,pam_permit,pam_env,pam_faillock acct="ankit" exe="/usr/bin/sshd" hostname=192.168.1.52 addr=192.168.1.52 terminal=ssh res=success'
[ 5687.279639] audit: type=1100 audit(1636521035.803:1180): pid=58142 uid=1000 auid=1000 ses=29 subj==unconfined msg='op=PAM:authentication grantors=pam_unix acct="ankit" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/0 res=success'
[ 5687.280646] audit: type=1101 audit(1636521035.806:1181): pid=58142 uid=1000 auid=1000 ses=29 subj==unconfined msg='op=PAM:accounting grantors=pam_unix,pam_permit,pam_time acct="ankit" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/0 res=success'
[ 5687.280783] audit: type=1110 audit(1636521035.806:1182): pid=58142 uid=1000 auid=1000 ses=29 subj==unconfined msg='op=PAM:setcred grantors=pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/0 res=success'
[ 5687.282219] audit: type=1105 audit(1636521035.806:1183): pid=58142 uid=1000 auid=1000 ses=29 subj==unconfined msg='op=PAM:session_open grantors=pam_limits,pam_unix,pam_permit acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/0 res=success'

Even after resetting my system, I got to a blank login screen on Manjaro. Nothing loads after that. Only powering off the system works.

  • Here is the output from nvidia-bug-report.sh --safe-mode --extra-system-data after reset - it still hangs so its partial output: nvidia-bug-report.log.gz.nov9-950pm.post_reset (88.7 KB)

  • Here is partial dmesg output after the reset showing more errors:

[  134.164852] audit: type=1103 audit(1636523617.417:1326): pid=8349 uid=0 auid=1000 ses=5 subj==unconfined msg='op=PAM:setcred grantors=pam_shells,pam_faillock,pam_permit,pam_env,pam_faillock acct="ankit" exe="/usr/bin/sshd" hostname=192.168.1.52 addr=192.168.1.52 terminal=ssh res=success'
[  245.360007] INFO: task systemd-udevd:369 blocked for more than 122 seconds.
[  245.360013]       Tainted: P           OE     5.13.19-2-MANJARO #1
[  245.360015] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  245.360016] task:systemd-udevd   state:D stack:    0 pid:  369 ppid:   353 flags:0x00004224
[  245.360021] Call Trace:
[  245.360026]  __schedule+0x310/0x930
[  245.360033]  schedule+0x5b/0xc0
[  245.360036]  schedule_timeout+0x11c/0x160
[  245.360040]  wait_for_completion+0xa3/0x100
[  245.360044]  __flush_work.isra.0+0x15b/0x210
[  245.360048]  ? flush_workqueue_prep_pwqs+0x130/0x130
[  245.360051]  __cancel_work_timer+0x12e/0x1c0
[  245.360056]  mt76u_stop_tx+0xb1/0x1d0 [mt76_usb]
[  245.360063]  mt76u_queues_deinit+0x23/0x1f0 [mt76_usb]
[  245.360067]  mt76x2u_probe+0x7b/0x130 [mt76x2u]
[  245.360073]  usb_probe_interface+0xe2/0x2a0
[  245.360077]  really_probe+0x384/0x4a0
[  245.360082]  driver_probe_device+0xe8/0x160
[  245.360086]  device_driver_attach+0xa1/0xb0
[  245.360090]  __driver_attach+0x8f/0x150
[  245.360093]  ? device_driver_attach+0xb0/0xb0
[  245.360096]  ? device_driver_attach+0xb0/0xb0
[  245.360099]  bus_for_each_dev+0x89/0xd0
[  245.360102]  bus_add_driver+0x12b/0x1e0
[  245.360106]  driver_register+0x8f/0xe0
[  245.360108]  usb_register_driver+0x84/0x120
[  245.360111]  ? 0xffffffffc04d4000
[  245.360113]  do_one_initcall+0x57/0x220
[  245.360118]  do_init_module+0x5c/0x270
[  245.360121]  load_module+0x2545/0x2740
[  245.360127]  ? __do_sys_init_module+0x12e/0x1b0
[  245.360129]  __do_sys_init_module+0x12e/0x1b0
[  245.360133]  do_syscall_64+0x40/0x80
[  245.360136]  entry_SYSCALL_64_after_hwframe+0x44/0xae
[  245.360140] RIP: 0033:0x7f3c88b5132e
[  245.360142] RSP: 002b:00007fff1272a468 EFLAGS: 00000246 ORIG_RAX: 00000000000000af
[  245.360145] RAX: ffffffffffffffda RBX: 000055afa9014950 RCX: 00007f3c88b5132e
[  245.360147] RDX: 00007f3c88cc3a9d RSI: 0000000000015f19 RDI: 000055afa92b5c20
[  245.360149] RBP: 000055afa92b5c20 R08: 000055afa9011230 R09: 00007fff1272a229
[  245.360150] R10: 000055aaf3fbe371 R11: 0000000000000246 R12: 00007f3c88cc3a9d
[  245.360151] R13: 0000000000000001 R14: 000055afa90138a0 R15: 000055afa9014950
[  298.367097] audit: type=1106 audit(1636523781.621:1327): pid=7824 uid=1000 auid=1000 ses=3 subj==unconfined msg='op=PAM:session_close grantors=pam_limits,pam_unix,pam_permit acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/0 res=success'
[  298.367141] audit: type=1104 audit(1636523781.621:1328): pid=7824 uid=1000 auid=1000 ses=3 subj==unconfined msg='op=PAM:setcred grantors=pam_env,pam_fprintd acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/0 res=success'
[  368.239989] INFO: task systemd-udevd:369 blocked for more than 245 seconds.
[  368.239995]       Tainted: P           OE     5.13.19-2-MANJARO #1
[  368.239997] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  368.239998] task:systemd-udevd   state:D stack:    0 pid:  369 ppid:   353 flags:0x00004224
[  368.240003] Call Trace:
[  368.240007]  __schedule+0x310/0x930
[  368.240015]  schedule+0x5b/0xc0
[  368.240018]  schedule_timeout+0x11c/0x160
[  368.240022]  wait_for_completion+0xa3/0x100
[  368.240026]  __flush_work.isra.0+0x15b/0x210
[  368.240030]  ? flush_workqueue_prep_pwqs+0x130/0x130
[  368.240033]  __cancel_work_timer+0x12e/0x1c0
[  368.240037]  mt76u_stop_tx+0xb1/0x1d0 [mt76_usb]
[  368.240044]  mt76u_queues_deinit+0x23/0x1f0 [mt76_usb]
[  368.240050]  mt76x2u_probe+0x7b/0x130 [mt76x2u]
[  368.240055]  usb_probe_interface+0xe2/0x2a0
[  368.240060]  really_probe+0x384/0x4a0
[  368.240065]  driver_probe_device+0xe8/0x160
[  368.240068]  device_driver_attach+0xa1/0xb0
[  368.240072]  __driver_attach+0x8f/0x150
[  368.240076]  ? device_driver_attach+0xb0/0xb0
[  368.240079]  ? device_driver_attach+0xb0/0xb0
[  368.240082]  bus_for_each_dev+0x89/0xd0
[  368.240085]  bus_add_driver+0x12b/0x1e0
[  368.240089]  driver_register+0x8f/0xe0
[  368.240091]  usb_register_driver+0x84/0x120
[  368.240094]  ? 0xffffffffc04d4000
[  368.240096]  do_one_initcall+0x57/0x220
[  368.240101]  do_init_module+0x5c/0x270
[  368.240104]  load_module+0x2545/0x2740
[  368.240110]  ? __do_sys_init_module+0x12e/0x1b0
[  368.240112]  __do_sys_init_module+0x12e/0x1b0
[  368.240116]  do_syscall_64+0x40/0x80
[  368.240119]  entry_SYSCALL_64_after_hwframe+0x44/0xae
[  368.240122] RIP: 0033:0x7f3c88b5132e
[  368.240124] RSP: 002b:00007fff1272a468 EFLAGS: 00000246 ORIG_RAX: 00000000000000af
[  368.240128] RAX: ffffffffffffffda RBX: 000055afa9014950 RCX: 00007f3c88b5132e
[  368.240130] RDX: 00007f3c88cc3a9d RSI: 0000000000015f19 RDI: 000055afa92b5c20
[  368.240131] RBP: 000055afa92b5c20 R08: 000055afa9011230 R09: 00007fff1272a229
[  368.240132] R10: 000055aaf3fbe371 R11: 0000000000000246 R12: 00007f3c88cc3a9d
[  368.240134] R13: 0000000000000001 R14: 000055afa90138a0 R15: 000055afa9014950
[  368.592568] audit: type=1100 audit(1636523851.844:1329): pid=8380 uid=1000 auid=1000 ses=5 subj==unconfined msg='op=PAM:authentication grantors=pam_unix acct="ankit" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/1 res=success'
[  368.593540] audit: type=1101 audit(1636523851.847:1330): pid=8380 uid=1000 auid=1000 ses=5 subj==unconfined msg='op=PAM:accounting grantors=pam_unix,pam_permit,pam_time acct="ankit" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/1 res=success'
[  368.593632] audit: type=1110 audit(1636523851.847:1331): pid=8380 uid=1000 auid=1000 ses=5 subj==unconfined msg='op=PAM:setcred grantors=pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/1 res=success'
[  368.595955] audit: type=1105 audit(1636523851.847:1332): pid=8380 uid=1000 auid=1000 ses=5 subj==unconfined msg='op=PAM:session_open grantors=pam_limits,pam_unix,pam_permit acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/1 res=success'
[  373.867681] audit: type=1334 audit(1636523857.121:1333): prog-id=28 op=LOAD
[  384.241501] audit: type=1106 audit(1636523867.494:1334): pid=8380 uid=1000 auid=1000 ses=5 subj==unconfined msg='op=PAM:session_close grantors=pam_limits,pam_unix,pam_permit acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/1 res=success'
[  384.241550] audit: type=1104 audit(1636523867.494:1335): pid=8380 uid=1000 auid=1000 ses=5 subj==unconfined msg='op=PAM:setcred grantors=pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/1 res=success'
[  394.531336] audit: type=1101 audit(1636523877.784:1336): pid=8391 uid=1000 auid=1000 ses=5 subj==unconfined msg='op=PAM:accounting grantors=pam_unix,pam_permit,pam_time acct="ankit" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/1 res=success'
[  394.531448] audit: type=1110 audit(1636523877.784:1337): pid=8391 uid=1000 auid=1000 ses=5 subj==unconfined msg='op=PAM:setcred grantors=pam_env,pam_fprintd acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/1 res=success'
[  394.533474] audit: type=1105 audit(1636523877.787:1338): pid=8391 uid=1000 auid=1000 ses=5 subj==unconfined msg='op=PAM:session_open grantors=pam_limits,pam_unix,pam_permit acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/1 res=success'
[  433.145844] audit: type=1106 audit(1636523916.397:1339): pid=8391 uid=1000 auid=1000 ses=5 subj==unconfined msg='op=PAM:session_close grantors=pam_limits,pam_unix,pam_permit acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/1 res=success'
[  433.145859] audit: type=1104 audit(1636523916.397:1340): pid=8391 uid=1000 auid=1000 ses=5 subj==unconfined msg='op=PAM:setcred grantors=pam_env,pam_fprintd acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/1 res=success'
[  463.892642] audit: type=1130 audit(1636523947.144:1341): pid=1 uid=0 auid=4294967295 ses=4294967295 subj==unconfined msg='unit=fprintd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
[  464.089796] audit: type=1334 audit(1636523947.341:1342): prog-id=28 op=UNLOAD
[  472.297326] audit: type=1101 audit(1636523955.551:1343): pid=8409 uid=1000 auid=1000 ses=5 subj==unconfined msg='op=PAM:accounting grantors=pam_unix,pam_permit,pam_time acct="ankit" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/1 res=success'
[  472.297476] audit: type=1110 audit(1636523955.551:1344): pid=8409 uid=1000 auid=1000 ses=5 subj==unconfined msg='op=PAM:setcred grantors=pam_env,pam_fprintd acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/1 res=success'
[  472.299583] audit: type=1105 audit(1636523955.551:1345): pid=8409 uid=1000 auid=1000 ses=5 subj==unconfined msg='op=PAM:session_open grantors=pam_limits,pam_unix,pam_permit acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/1 res=success'
[  472.300933] audit: type=1106 audit(1636523955.554:1346): pid=8409 uid=1000 auid=1000 ses=5 subj==unconfined msg='op=PAM:session_close grantors=pam_limits,pam_unix,pam_permit acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/1 res=success'
[  472.300974] audit: type=1104 audit(1636523955.554:1347): pid=8409 uid=1000 auid=1000 ses=5 subj==unconfined msg='op=PAM:setcred grantors=pam_env,pam_fprintd acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/1 res=success'
[  491.119987] INFO: task systemd-udevd:369 blocked for more than 368 seconds.
[  491.119992]       Tainted: P           OE     5.13.19-2-MANJARO #1
[  491.119994] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  491.119996] task:systemd-udevd   state:D stack:    0 pid:  369 ppid:   353 flags:0x00004224
[  491.120001] Call Trace:
[  491.120005]  __schedule+0x310/0x930
[  491.120013]  schedule+0x5b/0xc0
[  491.120016]  schedule_timeout+0x11c/0x160
[  491.120020]  wait_for_completion+0xa3/0x100
[  491.120024]  __flush_work.isra.0+0x15b/0x210
[  491.120028]  ? flush_workqueue_prep_pwqs+0x130/0x130
[  491.120030]  __cancel_work_timer+0x12e/0x1c0
[  491.120035]  mt76u_stop_tx+0xb1/0x1d0 [mt76_usb]
[  491.120042]  mt76u_queues_deinit+0x23/0x1f0 [mt76_usb]
[  491.120047]  mt76x2u_probe+0x7b/0x130 [mt76x2u]
[  491.120053]  usb_probe_interface+0xe2/0x2a0
[  491.120057]  really_probe+0x384/0x4a0
[  491.120062]  driver_probe_device+0xe8/0x160
[  491.120065]  device_driver_attach+0xa1/0xb0
[  491.120069]  __driver_attach+0x8f/0x150
[  491.120072]  ? device_driver_attach+0xb0/0xb0
[  491.120075]  ? device_driver_attach+0xb0/0xb0
[  491.120078]  bus_for_each_dev+0x89/0xd0
[  491.120081]  bus_add_driver+0x12b/0x1e0
[  491.120084]  driver_register+0x8f/0xe0
[  491.120086]  usb_register_driver+0x84/0x120
[  491.120090]  ? 0xffffffffc04d4000
[  491.120092]  do_one_initcall+0x57/0x220
[  491.120097]  do_init_module+0x5c/0x270
[  491.120100]  load_module+0x2545/0x2740
[  491.120106]  ? __do_sys_init_module+0x12e/0x1b0
[  491.120108]  __do_sys_init_module+0x12e/0x1b0
[  491.120112]  do_syscall_64+0x40/0x80
[  491.120116]  entry_SYSCALL_64_after_hwframe+0x44/0xae
[  491.120119] RIP: 0033:0x7f3c88b5132e
[  491.120121] RSP: 002b:00007fff1272a468 EFLAGS: 00000246 ORIG_RAX: 00000000000000af
[  491.120124] RAX: ffffffffffffffda RBX: 000055afa9014950 RCX: 00007f3c88b5132e
[  491.120126] RDX: 00007f3c88cc3a9d RSI: 0000000000015f19 RDI: 000055afa92b5c20
[  491.120127] RBP: 000055afa92b5c20 R08: 000055afa9011230 R09: 00007fff1272a229
[  491.120128] R10: 000055aaf3fbe371 R11: 0000000000000246 R12: 00007f3c88cc3a9d
[  491.120130] R13: 0000000000000001 R14: 000055afa90138a0 R15: 000055afa9014950
[  542.246448] audit: type=1101 audit(1636524025.497:1348): pid=8417 uid=1000 auid=1000 ses=5 subj==unconfined msg='op=PAM:accounting grantors=pam_unix,pam_permit,pam_time acct="ankit" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/1 res=success'
[  542.246549] audit: type=1110 audit(1636524025.497:1349): pid=8417 uid=1000 auid=1000 ses=5 subj==unconfined msg='op=PAM:setcred grantors=pam_env,pam_fprintd acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/1 res=success'
[  542.248643] audit: type=1105 audit(1636524025.501:1350): pid=8417 uid=1000 auid=1000 ses=5 subj==unconfined msg='op=PAM:session_open grantors=pam_limits,pam_unix,pam_permit acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/1 res=success'
[  546.131275] audit: type=1106 audit(1636524029.384:1351): pid=8417 uid=1000 auid=1000 ses=5 subj==unconfined msg='op=PAM:session_close grantors=pam_limits,pam_unix,pam_permit acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/1 res=success'
[  546.131304] audit: type=1104 audit(1636524029.384:1352): pid=8417 uid=1000 auid=1000 ses=5 subj==unconfined msg='op=PAM:setcred grantors=pam_env,pam_fprintd acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/1 res=success'
[  553.691218] audit: type=1101 audit(1636524036.944:1353): pid=8421 uid=1000 auid=1000 ses=5 subj==unconfined msg='op=PAM:accounting grantors=pam_unix,pam_permit,pam_time acct="ankit" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/1 res=success'
[  553.691310] audit: type=1110 audit(1636524036.944:1354): pid=8421 uid=1000 auid=1000 ses=5 subj==unconfined msg='op=PAM:setcred grantors=pam_env,pam_fprintd acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/1 res=success'
[  553.692842] audit: type=1105 audit(1636524036.944:1355): pid=8421 uid=1000 auid=1000 ses=5 subj==unconfined msg='op=PAM:session_open grantors=pam_limits,pam_unix,pam_permit acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/1 res=success'
[  556.476797] audit: type=1106 audit(1636524039.731:1356): pid=8421 uid=1000 auid=1000 ses=5 subj==unconfined msg='op=PAM:session_close grantors=pam_limits,pam_unix,pam_permit acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/1 res=success'
[  556.476846] audit: type=1104 audit(1636524039.731:1357): pid=8421 uid=1000 auid=1000 ses=5 subj==unconfined msg='op=PAM:setcred grantors=pam_env,pam_fprintd acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/1 res=success'
[  571.142922] audit: type=1101 audit(1636524054.394:1358): pid=8428 uid=1000 auid=1000 ses=5 subj==unconfined msg='op=PAM:accounting grantors=pam_unix,pam_permit,pam_time acct="ankit" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/1 res=success'
[  571.142996] audit: type=1110 audit(1636524054.394:1359): pid=8428 uid=1000 auid=1000 ses=5 subj==unconfined msg='op=PAM:setcred grantors=pam_env,pam_fprintd acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/1 res=success'
[  571.144626] audit: type=1105 audit(1636524054.397:1360): pid=8428 uid=1000 auid=1000 ses=5 subj==unconfined msg='op=PAM:session_open grantors=pam_limits,pam_unix,pam_permit acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/1 res=success'
[  571.146919] audit: type=1106 audit(1636524054.401:1361): pid=8428 uid=1000 auid=1000 ses=5 subj==unconfined msg='op=PAM:session_close grantors=pam_limits,pam_unix,pam_permit acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/1 res=success'
[  571.146939] audit: type=1104 audit(1636524054.401:1362): pid=8428 uid=1000 auid=1000 ses=5 subj==unconfined msg='op=PAM:setcred grantors=pam_env,pam_fprintd acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/1 res=success'
[  600.951777] audit: type=1101 audit(1636524084.204:1363): pid=8436 uid=1000 auid=1000 ses=5 subj==unconfined msg='op=PAM:accounting grantors=pam_unix,pam_permit,pam_time acct="ankit" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/1 res=success'
[  600.951861] audit: type=1110 audit(1636524084.204:1364): pid=8436 uid=1000 auid=1000 ses=5 subj==unconfined msg='op=PAM:setcred grantors=pam_env,pam_fprintd acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/1 res=success'
[  600.953326] audit: type=1105 audit(1636524084.207:1365): pid=8436 uid=1000 auid=1000 ses=5 subj==unconfined msg='op=PAM:session_open grantors=pam_limits,pam_unix,pam_permit acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/1 res=success'
[  606.902098] audit: type=1106 audit(1636524090.154:1366): pid=8436 uid=1000 auid=1000 ses=5 subj==unconfined msg='op=PAM:session_close grantors=pam_limits,pam_unix,pam_permit acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/1 res=success'
[  606.902139] audit: type=1104 audit(1636524090.154:1367): pid=8436 uid=1000 auid=1000 ses=5 subj==unconfined msg='op=PAM:setcred grantors=pam_env,pam_fprintd acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/1 res=success'
[  613.999977] INFO: task systemd-udevd:369 blocked for more than 491 seconds.
[  613.999983]       Tainted: P           OE     5.13.19-2-MANJARO #1
[  613.999985] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  613.999986] task:systemd-udevd   state:D stack:    0 pid:  369 ppid:   353 flags:0x00004224
[  613.999991] Call Trace:
[  613.999995]  __schedule+0x310/0x930
[  614.000003]  schedule+0x5b/0xc0
[  614.000006]  schedule_timeout+0x11c/0x160
[  614.000010]  wait_for_completion+0xa3/0x100
[  614.000014]  __flush_work.isra.0+0x15b/0x210
[  614.000018]  ? flush_workqueue_prep_pwqs+0x130/0x130
[  614.000021]  __cancel_work_timer+0x12e/0x1c0
[  614.000026]  mt76u_stop_tx+0xb1/0x1d0 [mt76_usb]
[  614.000033]  mt76u_queues_deinit+0x23/0x1f0 [mt76_usb]
[  614.000038]  mt76x2u_probe+0x7b/0x130 [mt76x2u]
[  614.000044]  usb_probe_interface+0xe2/0x2a0
[  614.000048]  really_probe+0x384/0x4a0
[  614.000053]  driver_probe_device+0xe8/0x160
[  614.000056]  device_driver_attach+0xa1/0xb0
[  614.000060]  __driver_attach+0x8f/0x150
[  614.000063]  ? device_driver_attach+0xb0/0xb0
[  614.000066]  ? device_driver_attach+0xb0/0xb0
[  614.000069]  bus_for_each_dev+0x89/0xd0
[  614.000073]  bus_add_driver+0x12b/0x1e0
[  614.000076]  driver_register+0x8f/0xe0
[  614.000078]  usb_register_driver+0x84/0x120
[  614.000081]  ? 0xffffffffc04d4000
[  614.000083]  do_one_initcall+0x57/0x220
[  614.000089]  do_init_module+0x5c/0x270
[  614.000092]  load_module+0x2545/0x2740
[  614.000098]  ? __do_sys_init_module+0x12e/0x1b0
[  614.000100]  __do_sys_init_module+0x12e/0x1b0
[  614.000104]  do_syscall_64+0x40/0x80
[  614.000107]  entry_SYSCALL_64_after_hwframe+0x44/0xae
[  614.000111] RIP: 0033:0x7f3c88b5132e
[  614.000113] RSP: 002b:00007fff1272a468 EFLAGS: 00000246 ORIG_RAX: 00000000000000af
[  614.000116] RAX: ffffffffffffffda RBX: 000055afa9014950 RCX: 00007f3c88b5132e
[  614.000118] RDX: 00007f3c88cc3a9d RSI: 0000000000015f19 RDI: 000055afa92b5c20
[  614.000119] RBP: 000055afa92b5c20 R08: 000055afa9011230 R09: 00007fff1272a229
[  614.000120] R10: 000055aaf3fbe371 R11: 0000000000000246 R12: 00007f3c88cc3a9d
[  614.000122] R13: 0000000000000001 R14: 000055afa90138a0 R15: 000055afa9014950
[  621.801269] audit: type=1101 audit(1636524105.054:1368): pid=8444 uid=1000 auid=1000 ses=5 subj==unconfined msg='op=PAM:accounting grantors=pam_unix,pam_permit,pam_time acct="ankit" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/1 res=success'
[  621.801354] audit: type=1110 audit(1636524105.054:1369): pid=8444 uid=1000 auid=1000 ses=5 subj==unconfined msg='op=PAM:setcred grantors=pam_env,pam_fprintd acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/1 res=success'
[  621.802743] audit: type=1105 audit(1636524105.054:1370): pid=8444 uid=1000 auid=1000 ses=5 subj==unconfined msg='op=PAM:session_open grantors=pam_limits,pam_unix,pam_permit acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/1 res=success'
[  692.252636] audit: type=1100 audit(1636524175.504:1371): pid=8455 uid=1000 auid=1000 ses=3 subj==unconfined msg='op=PAM:authentication grantors=pam_unix acct="ankit" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/0 res=success'
[  692.253164] audit: type=1101 audit(1636524175.504:1372): pid=8455 uid=1000 auid=1000 ses=3 subj==unconfined msg='op=PAM:accounting grantors=pam_unix,pam_permit,pam_time acct="ankit" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/0 res=success'
[  692.253263] audit: type=1110 audit(1636524175.504:1373): pid=8455 uid=1000 auid=1000 ses=3 subj==unconfined msg='op=PAM:setcred grantors=pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/0 res=success'
[  692.254661] audit: type=1105 audit(1636524175.507:1374): pid=8455 uid=1000 auid=1000 ses=3 subj==unconfined msg='op=PAM:session_open grantors=pam_limits,pam_unix,pam_permit acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/0 res=success'

I have also noticed firefox on my system often crashes a tab or entirely crashes. It is running in GPU-accelerated mode.

I also recall having these kinds of problems about a year ago on a different Linux distro. I ran Windows for a year+ on the same system and had no crashes like this.

Since this is happening quite frequently, i can provide more info if needed the next time this happens - I would really like some help to get to the bottom of this.These kinds of bugs have been reported all over the web by Nvidia users on linux.

The last crashes came from mt76u, the wifi driver. Please create a new nvidia-bug-report.log in the running state.

Here is the log in a stable state: nvidia-bug-report.log.nov10-525am.stable.gz (311.8 KB)

In case it matters, I am using an onboard Realtek ethernet card for internet access. The Intel Wifi card is active but not connected to a network. I have an Xbox Wireless Receiver on USB- that is the one using the mt76u driver. All the previous logs from the actual time of the crash show stuff related to Nvidia so maybe the issue post reset is unrelated?

Looks like post-reset your mainboard might be confused so all kind of unrelated errors happen.
Also nothing unusual in the logs from working state. Please create an additional nvidia-bug-report.log right after freeze+poweroff.

After updating my cards VBIOS and moving to kernel 5.13.0, it seems to be stable. Thank you, I will update here if I see a freeze again.