Nvidia fails hard and PC (Manjaro) unresponsive

Graphics / Linux Linux
,
1

Hey.
During some of sessions something fails hard, presumably Nvidia driver. PC become unresponsive. At first the picture froze but the sound keep going. After a couple seconds even sound stops and the black screen.
After hard reboot (press power button) everything is fine and the session can be without any issues.

inxi -Gxx                                                                                                                                                ✔ 
Graphics:
  Device-1: NVIDIA GB203 [GeForce RTX 5080] vendor: Gigabyte driver: nvidia
    v: 590.48.01 arch: Lovelace pcie: speed: 32 GT/s lanes: 16 ports:
    active: DP-3 empty: DP-4,DP-5,HDMI-A-3 bus-ID: 01:00.0 chip-ID: 10de:2c02
  Device-2: Advanced Micro Devices [AMD/ATI] Granite Ridge [Radeon Graphics]
    vendor: Gigabyte driver: amdgpu v: kernel arch: RDNA-2 pcie: speed: 16 GT/s
    lanes: 16 ports: active: none empty: DP-1, DP-2, HDMI-A-1, HDMI-A-2,
    Writeback-1 bus-ID: 79:00.0 chip-ID: 1002:13c0 temp: 40.0 C
  Display: wayland server: X.org v: 1.21.1.21 with: Xwayland v: 24.1.9
    compositor: kwin_wayland driver: X: loaded: amdgpu,nvidia
    unloaded: modesetting,nouveau alternate: fbdev,nv,vesa dri: radeonsi
    gpu: nv_platform,nvidia,nvidia-nvswitch display-ID: 0
  Monitor-1: DP-3 model: Samsung Odyssey G61SD res: 2560x1440 hz: 240
    dpi: 109 diag: 686mm (27")
  API: EGL v: 1.5 platforms: device: 0 drv: nvidia device: 2 drv: radeonsi
    device: 3 drv: swrast gbm: drv: nvidia surfaceless: drv: nvidia wayland:
    drv: nvidia x11: drv: nvidia inactive: device-1
  API: OpenGL v: 4.6.0 compat-v: 4.5 vendor: nvidia mesa v: 25.3.4-arch1.1
    glx-v: 1.4 direct-render: yes renderer: NVIDIA GeForce RTX 5080/PCIe/SSE2
    display-ID: :0.0
  API: Vulkan v: 1.4.335 surfaces: N/A device: 0 type: discrete-gpu
    driver: nvidia device-ID: 10de:2c02 device: 1 type: integrated-gpu
    driver: mesa radv device-ID: 1002:13c0
  Info: Tools: api: clinfo, eglinfo, glxinfo, vulkaninfo
    de: kscreen-console,kscreen-doctor gpu: nvidia-settings,nvidia-smi
    wl: wayland-info x11: xdpyinfo,xprop

What I found in the KSystemLog at the time of fail and up to the next loading:

14.02.26 10:42	kernel	NVRM: GPU at PCI:0000:01:00: GPU-d446bb52-9b80-2b25-d6dd-1d3613b91022
14.02.26 10:42	kernel	NVRM: GPU Board Serial Number: 0
14.02.26 10:42	kernel	NVRM: Xid (PCI:0000:01:00): 62, 32344000 0000b670 00000000 206a7a8a 206a6c4a 206a6db8 206a52ae 206a5aca
14.02.26 10:42	kernel	NVRM: Xid (PCI:0000:01:00): 154, GPU recovery action changed from 0x0 (None) to 0x1 (GPU Reset Required)
14.02.26 10:43	kernel	NVRM: Xid (PCI:0000:01:00): 109, pid=1646, name=chrome, channel 0x00000004, errorString CTX SWITCH TIMEOUT, Info 0x54005

14.02.26 10:43	kernel	NVRM: GPU0 _kgspLogXid119: ********************************* GSP Timeout **********************************
14.02.26 10:43	kernel	NVRM: GPU0 _kgspLogXid119: Note: Please also check logs above.
14.02.26 10:43	kernel	NVRM: Xid (PCI:0000:01:00): 119, pid=1646, name=chrome, Timeout after 6s of waiting for RPC response from GPU0 GSP! Expected function 10 (FREE) sequence 6501 (0xbeef0403 0x0).
14.02.26 10:43	kernel	NVRM: GPU0 kgspPrintGspBinBuildId_IMPL: GSP bin buildId: cf812cb3f2f1e8c8209dc2e446fdf536ba9ec88f
14.02.26 10:43	kernel	NVRM:     task watchdog timeout @ pc:0x1a9aa88, partition:4#0, task:3
14.02.26 10:43	kernel	NVRM:     Reported by libos partition:4#5 kernel v3.1 [0] @ ts:4488
14.02.26 10:43	kernel	NVRM:     RISC-V CSR State:
14.02.26 10:43	kernel	NVRM:         sstatus:0x0000000200000020  sscratch:0xffffffffa3013960     sie:0x0000000000000220  sip:0x0000000000000020
14.02.26 10:43	kernel	NVRM:         sepc:0x0000000001a9aa88     stval:0x0000000000000000  scause:0x8000000000000005
14.02.26 10:43	kernel	NVRM:     RISC-V GPR State:
14.02.26 10:43	kernel	NVRM:         ra:0x0000000001a9aa88   sp:0x00000007f680f1c0   gp:0x0000000000000000   tp:0x00000007f6c00000
14.02.26 10:43	kernel	NVRM:         a0:0x0000000000000000   a1:0x00000007ef97a730   a2:0x0000000000000002   a3:0x0000000000b60000
14.02.26 10:43	kernel	NVRM:         a4:0x0000000007a00000   a5:0x0000000008562008   a6:0x000fffffffffffff   a7:0x0000000000000000
14.02.26 10:43	kernel	NVRM:         s0:0x00000007f680f2a0   s1:0x0000000000000002   s2:0x00000007ef97a730   s3:0x0000000000000002
14.02.26 10:43	kernel	NVRM:         s4:0x00000007ef78b5f0   s5:0x0000000000000003   s6:0x00000007f680f2b0   s7:0x000000000000003e
14.02.26 10:43	kernel	NVRM:         s8:0x0000000001526c96   s9:0x0000000004164180  s10:0x000000000000001f  s11:0x00000007efa03d30
14.02.26 10:43	kernel	NVRM:         t0:0x0000000000000005   t1:0x0000000000000003   t2:0x0000000000000000   t3:0x000fffffffffffff
14.02.26 10:43	kernel	NVRM:         t4:0x0000000000000000   t5:0x00000007f680f401   t6:0x0000000000000020
14.02.26 10:43	kernel	NVRM:     Stack Trace:
14.02.26 10:43	kernel	NVRM:         0x0000000001a9aa88
14.02.26 10:43	kernel	NVRM:         0x0000000001526c96
14.02.26 10:43	kernel	NVRM:         0x0000000001aa1678
14.02.26 10:43	kernel	NVRM:         0x0000000001acc430
14.02.26 10:43	kernel	NVRM:         0x000000000169f602
14.02.26 10:43	kernel	NVRM:         0x0000000001684dec
14.02.26 10:43	kernel	NVRM:         0x0000000001b120ee
14.02.26 10:43	kernel	NVRM:         0x000000000143a254
14.02.26 10:43	kernel	NVRM:         0x000000000143a3d2
14.02.26 10:43	kernel	NVRM:         0x000000000143a6de
14.02.26 10:43	kernel	NVRM:         0x000000000179e67c
14.02.26 10:43	kernel	NVRM:         0x000000000179ffae
14.02.26 10:43	kernel	NVRM:         0x00000000017a0e0c
14.02.26 10:43	kernel	NVRM:         0x0000000001aabf30
14.02.26 10:43	kernel	NVRM:         0x0000000001393af6
14.02.26 10:43	kernel	NVRM:         0x000000000139131a
14.02.26 10:43	kernel	NVRM:         0x00000000015264b4
14.02.26 10:43	kernel	NVRM:         0x0000000001ad71dc
14.02.26 10:43	kernel	NVRM:         0x0000000001ad8ed2
14.02.26 10:43	kernel	NVRM:         0x00000000019eb3f0
14.02.26 10:43	kernel	NVRM:         0x0000000001b36596
14.02.26 10:43	kernel	NVRM:         0x00000000019dab2a
14.02.26 10:43	kernel	NVRM:     PC Trace:
14.02.26 10:43	kernel	NVRM:         0x0000000001a9aa88  0x000000000010013e  0x0000000001a9aa88  0x0000000001527e58  0x0000000001a9aa86
14.02.26 10:43	kernel	NVRM:         0x0000000001529316  0x000000000152739a  0x0000000001529320  0x000000000152739a  0x00000000015292f8
14.02.26 10:43	kernel	NVRM:         0x000000000152739a  0x00000000015292e8  0x000000000152739a  0x00000000015292d8  0x000000000152739a
14.02.26 10:43	kernel	NVRM:         0x00000000015292ca  0x0000000001a9a922  0x0000000001b1b594  0x0000000001a9a902  0x0000000001b1b594
14.02.26 10:43	kernel	NVRM:         0x0000000001a9a8f4  0x0000000001529316  0x000000000152739a  0x0000000001529320
14.02.26 10:43	kernel	NVRM:     Local I/O Register State:
14.02.26 10:43	kernel	NVRM:         0x01450800:0x00000000   0x01450900:0xbadf5100   0x01450a00:0x00000000   0x01450c00:0x00000000
14.02.26 10:43	kernel	NVRM:         0x01454a00:0x810490d2   0x01454b00:0x010800d0   0x01454c00:0x00080000   0x01400200:0x00000040
14.02.26 10:43	kernel	NVRM: GPU0 GSP RPC buffer contains function 4100 (RC_TRIGGERED) sequence 0 and data 0x0000000000000001 0x000000000000006d.
14.02.26 10:43	kernel	NVRM: GPU0 RPC history (CPU -> GSP):
14.02.26 10:43	kernel	NVRM:     entry function                     sequence data0              data1              ts_start           ts_end             duration actively_polling
14.02.26 10:43	kernel	NVRM:      0    10   FREE                        6501 0x00000000beef0403 0x0000000000000000 0x00064ac4b59770cd 0x0000000000000000          y
14.02.26 10:43	kernel	NVRM:     -1    76   GSP_RM_CONTROL              6500 0x0000000020800a56 0x000000000000005c 0x00064ac4b5117ac2 0x00064ac4b511b49a  14808us  
14.02.26 10:43	kernel	NVRM:     -2    76   GSP_RM_CONTROL              6499 0x0000000020802802 0x0000000000000004 0x00064ac4b507bc0d 0x00064ac4b507ca8b   3710us  
14.02.26 10:43	kernel	NVRM:     -3    76   GSP_RM_CONTROL              6498 0x0000000020802802 0x0000000000000004 0x00064ac4b4f85b1a 0x00064ac4b4f86997   3709us  
14.02.26 10:43	kernel	NVRM:     -4    76   GSP_RM_CONTROL              6497 0x0000000020802802 0x0000000000000004 0x00064ac4b4e8faba 0x00064ac4b4e908bb   3585us  
14.02.26 10:43	kernel	NVRM:     -5    76   GSP_RM_CONTROL              6496 0x0000000020802802 0x0000000000000004 0x00064ac4b4d99c09 0x00064ac4b4d9a9bb   3506us  
14.02.26 10:43	kernel	NVRM:     -6    76   GSP_RM_CONTROL              6495 0x0000000020802802 0x0000000000000004 0x00064ac4b4ca3ca3 0x00064ac4b4ca4a0a   3431us  
14.02.26 10:43	kernel	NVRM:     -7    76   GSP_RM_CONTROL              6494 0x0000000020802801 0x0000000000000004 0x00064ac4b4a3b4ff 0x00064ac4b4a3bc90   1937us  
14.02.26 10:43	kernel	NVRM: GPU0 RPC event history (CPU <- GSP):
14.02.26 10:43	kernel	NVRM:     entry function                     sequence data0              data1              ts_start           ts_end             duration during_incomplete_rpc
14.02.26 10:43	kernel	NVRM:      0    4100 RC_TRIGGERED                   0 0x0000000000000001 0x000000000000006d 0x00064ac4b59773a7 0x00064ac4b59773bb     20us y
14.02.26 10:43	kernel	NVRM:     -1    4102 OS_ERROR_LOG                   0 0x0000000000000000 0x0000000000000000 0x00064ac4b59766ca 0x00064ac4b59766d8     14us  
14.02.26 10:43	kernel	NVRM:     -2    4130 RECOVERY_ACTION                0 0x0000000000000000 0x0000000000000000 0x00064ac4b5117a9d 0x00064ac4b5117a9f      2us  
14.02.26 10:43	kernel	NVRM:     -3    4102 OS_ERROR_LOG                   0 0x0000000000000000 0x0000000000000000 0x00064ac4b5117a37 0x00064ac4b5117a46     15us  
14.02.26 10:43	kernel	NVRM:     -4    4108 UCODE_LIBOS_PRINT              0 0x0000000000000000 0x0000000000000000 0x00064ac4b5116490 0x00064ac4b5116491      1us  
14.02.26 10:43	kernel	NVRM:     -5    4108 UCODE_LIBOS_PRINT              0 0x0000000000000000 0x0000000000000000 0x00064ac4b5116490 0x00064ac4b5116490           
14.02.26 10:43	kernel	NVRM:     -6    4108 UCODE_LIBOS_PRINT              0 0x0000000000000000 0x0000000000000000 0x00064ac4b511645e 0x00064ac4b511645e           
14.02.26 10:43	kernel	NVRM:     -7    4108 UCODE_LIBOS_PRINT              0 0x0000000000000000 0x0000000000000000 0x00064ac4b511645e 0x00064ac4b511645e           
14.02.26 10:43	kernel	CPU: 5 UID: 1000 PID: 1646 Comm: chrome Tainted: G           OE      6.12.68-1-MANJARO #1 3bf75541c7360dc7269e9d7c5a8d128d5012edfb
14.02.26 10:43	kernel	Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
14.02.26 10:43	kernel	Hardware name: Gigabyte Technology Co., Ltd. X870E AORUS PRO ICE/X870E AORUS PRO ICE, BIOS F9 12/12/2025
14.02.26 10:43	kernel	Call Trace:
14.02.26 10:43	kernel	 <TASK>
14.02.26 10:43	kernel	 dump_stack_lvl+0x5d/0x80
14.02.26 10:43	kernel	 _kgspRpcRecvPoll+0xa62/0xc30 [nvidia ec055136828bcf2c9927aec995f1da756be21750]
14.02.26 10:43	kernel	 _issueRpcAndWait+0xd2/0x930 [nvidia ec055136828bcf2c9927aec995f1da756be21750]
14.02.26 10:43	kernel	 ? osGetCurrentThread+0x26/0x60 [nvidia ec055136828bcf2c9927aec995f1da756be21750]
14.02.26 10:43	kernel	 ? _rmGpuLockIsOwner+0x24/0x90 [nvidia ec055136828bcf2c9927aec995f1da756be21750]
14.02.26 10:43	kernel	 rpcRmApiFree_GSP+0x113/0x1f0 [nvidia ec055136828bcf2c9927aec995f1da756be21750]
14.02.26 10:43	kernel	 ? clientGetResourceRef_IMPL+0x1b/0x40 [nvidia ec055136828bcf2c9927aec995f1da756be21750]
14.02.26 10:43	kernel	 notifyUnregisterEvent_IMPL+0x257/0x350 [nvidia ec055136828bcf2c9927aec995f1da756be21750]
14.02.26 10:43	kernel	 eventDestruct_IMPL+0x70/0xab [nvidia ec055136828bcf2c9927aec995f1da756be21750]
14.02.26 10:43	kernel	 __nvoc_dtor_Event+0x15/0x30 [nvidia ec055136828bcf2c9927aec995f1da756be21750]
14.02.26 10:43	kernel	 __nvoc_objDelete+0x34/0x120 [nvidia ec055136828bcf2c9927aec995f1da756be21750]
14.02.26 10:43	kernel	 clientFreeResource_IMPL+0x665/0x890 [nvidia ec055136828bcf2c9927aec995f1da756be21750]
14.02.26 10:43	kernel	 ? osDevWriteReg032+0x2b/0x80 [nvidia ec055136828bcf2c9927aec995f1da756be21750]
14.02.26 10:43	kernel	 ? _rmGpuLocksAcquire.isra.0+0x3e5/0xbe0 [nvidia ec055136828bcf2c9927aec995f1da756be21750]
14.02.26 10:43	kernel	 ? os_acquire_spinlock+0x12/0x30 [nvidia ec055136828bcf2c9927aec995f1da756be21750]
14.02.26 10:43	kernel	 ? portSyncSpinlockAcquire+0x18/0x30 [nvidia ec055136828bcf2c9927aec995f1da756be21750]
14.02.26 10:43	kernel	 ? serverResLock_Prologue+0x16c/0x310 [nvidia ec055136828bcf2c9927aec995f1da756be21750]
14.02.26 10:43	kernel	 ? _tlsThreadEntryGet+0x82/0x90 [nvidia ec055136828bcf2c9927aec995f1da756be21750]
14.02.26 10:43	kernel	 rmclientFreeResource_IMPL+0xba/0xd3 [nvidia ec055136828bcf2c9927aec995f1da756be21750]
14.02.26 10:43	kernel	 serverFreeResourceTreeUnderLock+0xbb/0x1b0 [nvidia ec055136828bcf2c9927aec995f1da756be21750]
14.02.26 10:43	kernel	 serverFreeResourceTree+0x3c4/0x980 [nvidia ec055136828bcf2c9927aec995f1da756be21750]
14.02.26 10:43	kernel	 ? netif_skb_features+0xcc/0x490
14.02.26 10:43	kernel	 ? netif_skb_features+0xcc/0x490
14.02.26 10:43	kernel	 ? os_acquire_spinlock+0x12/0x30 [nvidia ec055136828bcf2c9927aec995f1da756be21750]
14.02.26 10:43	kernel	 ? portSyncSpinlockAcquire+0x18/0x30 [nvidia ec055136828bcf2c9927aec995f1da756be21750]
14.02.26 10:43	kernel	 ? rmapiInitLockInfo+0x26/0x200 [nvidia ec055136828bcf2c9927aec995f1da756be21750]
14.02.26 10:43	kernel	 ? os_acquire_rwlock_write+0x2b/0x40 [nvidia ec055136828bcf2c9927aec995f1da756be21750]
14.02.26 10:43	kernel	 rmapiFreeWithSecInfo+0x124/0x2a0 [nvidia ec055136828bcf2c9927aec995f1da756be21750]
14.02.26 10:43	kernel	 rmapiFreeWithSecInfoTls+0x57/0x80 [nvidia ec055136828bcf2c9927aec995f1da756be21750]
14.02.26 10:43	kernel	 Nv01FreeWithSecInfo+0x4c/0x60 [nvidia ec055136828bcf2c9927aec995f1da756be21750]
14.02.26 10:43	kernel	 ? security_capable+0x5d/0x1a0
14.02.26 10:43	kernel	 RmIoctl+0x41b/0xdc0 [nvidia ec055136828bcf2c9927aec995f1da756be21750]
14.02.26 10:43	kernel	 ? os_get_monotonic_time_ns+0x3b/0xa0 [nvidia ec055136828bcf2c9927aec995f1da756be21750]
14.02.26 10:43	kernel	 ? os_acquire_spinlock+0x12/0x30 [nvidia ec055136828bcf2c9927aec995f1da756be21750]
14.02.26 10:43	kernel	 ? portSyncSpinlockAcquire+0x18/0x30 [nvidia ec055136828bcf2c9927aec995f1da756be21750]
14.02.26 10:43	kernel	 rm_ioctl+0x78/0x5f0 [nvidia ec055136828bcf2c9927aec995f1da756be21750]
14.02.26 10:43	kernel	 ? sched_balance_rq+0x24c/0x1010
14.02.26 10:43	kernel	 ? __check_object_size+0x1ff/0x220
14.02.26 10:43	kernel	 nvidia_unlocked_ioctl+0x434/0x930 [nvidia ec055136828bcf2c9927aec995f1da756be21750]
14.02.26 10:43	kernel	 __x64_sys_ioctl+0x91/0xd0
14.02.26 10:43	kernel	 do_syscall_64+0x7b/0x190
14.02.26 10:43	kernel	 ? irqtime_account_irq+0x3c/0xc0
14.02.26 10:43	kernel	 ? handle_softirqs+0x18b/0x290
14.02.26 10:43	kernel	 ? irqentry_exit_to_user_mode+0x2c/0x1b0
14.02.26 10:43	kernel	 entry_SYSCALL_64_after_hwframe+0x76/0x7e
14.02.26 10:43	kernel	RIP: 0033:0x7f62b88ac74d
14.02.26 10:43	kernel	Code: 04 25 28 00 00 00 48 89 45 c8 31 c0 48 8d 45 10 c7 45 b0 10 00 00 00 48 89 45 b8 48 8d 45 d0 48 89 45 c0 b8 10 00 00 00 0f 05 <89> c2 3d 00 f0 ff ff 77 1a 48 8b 45 c8 64 48 2b 04 25 28 00 00 00
14.02.26 10:43	kernel	RSP: 002b:00007ffcac182e20 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
14.02.26 10:43	kernel	RAX: ffffffffffffffda RBX: 0000000000000010 RCX: 00007f62b88ac74d
14.02.26 10:43	kernel	RDX: 00007ffcac182f30 RSI: 00000000c0104629 RDI: 000000000000001a
14.02.26 10:43	kernel	RBP: 00007ffcac182e70 R08: 00007ffcac182f30 R09: 00007ffcac182f3c
14.02.26 10:43	kernel	R10: 0000000000000000 R11: 0000000000000246 R12: 000000000000001a
14.02.26 10:43	kernel	R13: 00000000c0104629 R14: 0000000000000029 R15: 00007ffcac182e80
14.02.26 10:43	kernel	 </TASK>
14.02.26 10:43	kernel	NVRM: GPU0 kgspDumpMailbox_TU102: GSP: MAILBOX(0) = 0x00000000
14.02.26 10:43	kernel	NVRM: GPU0 kgspDumpMailbox_TU102: GSP: MAILBOX(1) = 0x00000000
14.02.26 10:43	kernel	NVRM: GPU0 kgspDumpMailbox_TU102: GSP: MAILBOX(2) = 0x00000000
14.02.26 10:43	kernel	NVRM: GPU0 kgspDumpMailbox_TU102: GSP: MAILBOX(3) = 0x00000000
14.02.26 10:43	kernel	NVRM: GPU0 gpuHandleSanityCheckRegReadError_GH100: Possible bad register read: addr: 0x110624,  regvalue: 0xbadf5720,  error code: Unknown SYS_PRI_ERROR_CODE
14.02.26 10:43	kernel	NVRM: GPU0 gpuHandleSanityCheckRegReadError_GH100: Possible bad register read: addr: 0x11062c,  regvalue: 0xbadf5720,  error code: Unknown SYS_PRI_ERROR_CODE
14.02.26 10:43	kernel	NVRM: GPU0 gpuHandleSanityCheckRegReadError_GH100: Possible bad register read: addr: 0x111404,  regvalue: 0xbadf5720,  error code: Unknown SYS_PRI_ERROR_CODE
14.02.26 10:43	kernel	NVRM: GPU0 gpuHandleSanityCheckRegReadError_GH100: Possible bad register read: addr: 0x111408,  regvalue: 0xbadf5720,  error code: Unknown SYS_PRI_ERROR_CODE
14.02.26 10:43	kernel	NVRM: GPU0 kflcnDumpTracepc_GA102: Trace buffer blocked, skipping.
14.02.26 10:43	kernel	NVRM: GPU0 kflcnCoreDumpNondestructive_IMPL: PRI: riscvPc               : 010a4e1e
14.02.26 10:43	kernel	NVRM: GPU0 kflcnCoreDumpNondestructive_IMPL: PRI: riscvCpuctl           : 00000180
14.02.26 10:43	kernel	NVRM: GPU0 kflcnCoreDumpNondestructive_IMPL: PRI: riscvIrqmask          : 810490d2
14.02.26 10:43	kernel	NVRM: GPU0 kflcnCoreDumpNondestructive_IMPL: PRI: riscvIrqdest          : 010800d0
14.02.26 10:43	kernel	NVRM: GPU0 kflcnCoreDumpNondestructive_IMPL: PRI: riscvPrivErrStat      : 00000000
14.02.26 10:43	kernel	NVRM: GPU0 kflcnCoreDumpNondestructive_IMPL: PRI: riscvPrivErrInfo      : badf5100
14.02.26 10:43	kernel	NVRM: GPU0 kflcnCoreDumpNondestructive_IMPL: PRI: riscvPrivErrAddr      : 0000000000000000
14.02.26 10:43	kernel	NVRM: GPU0 kflcnCoreDumpNondestructive_IMPL: PRI: riscvHubErrStat       : 00000000
14.02.26 10:43	kernel	NVRM: GPU0 kflcnCoreDumpNondestructive_IMPL: PRI: falconMailbox         : 0:00000348 1:00000348
14.02.26 10:43	kernel	NVRM: GPU0 kflcnCoreDumpNondestructive_IMPL: PRI: falconIrqstat         : 00400050
14.02.26 10:43	kernel	NVRM: GPU0 kflcnCoreDumpNondestructive_IMPL: PRI: falconIrqmode         : ffacfc24
14.02.26 10:43	kernel	NVRM: GPU0 kflcnCoreDumpNondestructive_IMPL: PRI: fbifInstblk           : 00000000
14.02.26 10:43	kernel	NVRM: GPU0 kflcnCoreDumpNondestructive_IMPL: PRI: fbifCtl               : badf5720
14.02.26 10:43	kernel	NVRM: GPU0 kflcnCoreDumpNondestructive_IMPL: PRI: fbifThrottle          : badf5720
14.02.26 10:43	kernel	NVRM: GPU0 kflcnCoreDumpNondestructive_IMPL: PRI: fbifAchkBlk           : 0:00000000 1:00000000
14.02.26 10:43	kernel	NVRM: GPU0 kflcnCoreDumpNondestructive_IMPL: PRI: fbifAchkCtl           : 0:00000000 1:00000000
14.02.26 10:43	kernel	NVRM: GPU0 kflcnCoreDumpNondestructive_IMPL: PRI: fbifCg1               : 0000000f
14.02.26 10:43	kernel	NVRM: GPU0 _kgspLogXid119: ********************************************************************************
14.02.26 10:43	kernel	NVRM: GPU0 _issueRpcAndWait: rpcRecvPoll timedout for fn 10 sequence 6501!
14.02.26 10:43	kernel	NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d00076; hObject=0xbeef0403; paramsStatus=0x00000002; status=0x00000065
14.02.26 10:43	kernel	NVRM: GPU0 _kgspProcessRpcEvent: Unexpected RPC event from GPU0: 0xa (FREE), sequence: 6501
14.02.26 10:43	kernel	NVRM: GPU0 kgspHealthCheck_TU102: ****************************** GSP-CrashCat Report *******************************
14.02.26 10:43	kernel	NVRM: GPU0 kgspPrintGspBinBuildId_IMPL: GSP bin buildId: cf812cb3f2f1e8c8209dc2e446fdf536ba9ec88f
14.02.26 10:43	kernel	NVRM: Xid (PCI:0000:01:00): 120, pid=1646, name=chrome, GSP task exception: supervisor timer interrupt (cause:0x8000000000000005) @ pc:0x1a98e70, partition:4#0, task:3
14.02.26 10:43	kernel	NVRM:     Reported by libos partition:4#5 kernel v3.1 [0] @ ts:4500
14.02.26 10:43	kernel	NVRM:     RISC-V CSR State:
14.02.26 10:43	kernel	NVRM:         sstatus:0x0000000200000020  sscratch:0xffffffffa3013030     sie:0x0000000000000220  sip:0x0000000000000000
14.02.26 10:43	kernel	NVRM:         sepc:0x0000000001a98e70     stval:0x0000000000000000  scause:0x8000000000000005
14.02.26 10:43	kernel	NVRM:     RISC-V GPR State:
14.02.26 10:43	kernel	NVRM:         ra:0x00000000013ee144   sp:0x00000007f680eee0   gp:0x0000000000000000   tp:0x00000007f6c00000
14.02.26 10:43	kernel	NVRM:         a0:0x00000007ef78b5f0   a1:0x0000000000000000   a2:0x0000000000000049   a3:0x0000000004165280
14.02.26 10:43	kernel	NVRM:         a4:0x000000000000000a   a5:0x0000000000000020   a6:0x000fffffffffffff   a7:0x0000000000000000
14.02.26 10:43	kernel	NVRM:         s0:0x00000007f680ef30   s1:0x00000007f680ef30   s2:0x0000000001acc4bc   s3:0x0000000000000049
14.02.26 10:43	kernel	NVRM:         s4:0x00000007ef78b5f0   s5:0x0000000004164180   s6:0x00000007f680ef30   s7:0x00000007ef78d2b8
14.02.26 10:43	kernel	NVRM:         s8:0x0000000000000049   s9:0x0000000004164180  s10:0x00000417b74c3e40  s11:0x00000007efa03d30
14.02.26 10:43	kernel	NVRM:         t0:0x0000000000000005   t1:0x0000000000000003   t2:0x0000000000000000   t3:0x000fffffffffffff
14.02.26 10:43	kernel	NVRM:         t4:0x0000000000000000   t5:0x00000007f680efc1   t6:0x0000000000000020
14.02.26 10:43	kernel	NVRM:     Stack Trace:
14.02.26 10:43	kernel	NVRM:         0x0000000001a98e70
14.02.26 10:43	kernel	NVRM:         0x0000000001acc4bc
14.02.26 10:43	kernel	NVRM:         0x000000000169f602
14.02.26 10:43	kernel	NVRM:         0x0000000001684dec
14.02.26 10:43	kernel	NVRM:         0x0000000001b120ee
14.02.26 10:43	kernel	NVRM:         0x000000000143a254
14.02.26 10:43	kernel	NVRM:         0x000000000143a3d2
14.02.26 10:43	kernel	NVRM:         0x000000000143a6de
14.02.26 10:43	kernel	NVRM:         0x000000000179e67c
14.02.26 10:43	kernel	NVRM:         0x000000000179f8a6
14.02.26 10:43	kernel	NVRM:         0x00000000019f9fe2
14.02.26 10:43	kernel	NVRM:         0x000000000135d884
14.02.26 10:43	kernel	NVRM:         0x000000000136884c
14.02.26 10:43	kernel	NVRM:         0x0000000001369436
14.02.26 10:43	kernel	NVRM:         0x0000000001827e1e
14.02.26 10:43	kernel	NVRM:         0x0000000001a2719c
14.02.26 10:43	kernel	NVRM:         0x0000000001a373de
14.02.26 10:43	kernel	NVRM:         0x00000000010365cc
14.02.26 10:43	kernel	NVRM:         0x00000000019bd0ac
14.02.26 10:43	kernel	NVRM:         0x00000000019cf62a
14.02.26 10:43	kernel	NVRM:         0x0000000001ad9036
14.02.26 10:43	kernel	NVRM:         0x00000000019eb3f0
14.02.26 10:43	kernel	NVRM:         0x0000000001b36596
14.02.26 10:43	kernel	NVRM:         0x00000000019dab2a
14.02.26 10:43	kernel	NVRM:     PC Trace:
14.02.26 10:43	kernel	NVRM:         0x0000000001005a8a  0x0000000001002a4a
14.02.26 10:43	kernel	NVRM:     Local I/O Register State:
14.02.26 10:43	kernel	NVRM:         0x01450800:0x00000000   0x01450900:0xbadf5045   0x01450a00:0x00000000   0x01450c00:0x00000000
14.02.26 10:43	kernel	NVRM:         0x01454a00:0x810490d2   0x01454b00:0x010800d0   0x01454c00:0x00080000   0x01400200:0x00400050
14.02.26 10:43	kernel	NVRM:     ------------[ end crash report ]------------
14.02.26 10:43	kernel	NVRM: GPU0 GSP RPC buffer contains function 10 (FREE) sequence 6513 and data 0x00000000beef0100 0x0000000000000000.
14.02.26 10:43	kernel	NVRM: GPU0 RPC history (CPU -> GSP):
14.02.26 10:43	kernel	NVRM:     entry function                     sequence data0              data1              ts_start           ts_end             duration actively_polling
14.02.26 10:43	kernel	NVRM:      0    10   FREE                        6513 0x00000000beef0100 0x0000000000000000 0x00064ac4b6123b87 0x0000000000000000          y
14.02.26 10:43	kernel	NVRM:     -1    10   FREE                        6512 0x00000000beef3901 0x0000000000000000 0x00064ac4b6123912 0x00064ac4b6123b81    623us  
14.02.26 10:43	kernel	NVRM:     -2    10   FREE                        6511 0x00000000beef502d 0x0000000000000000 0x00064ac4b6123741 0x00064ac4b612390e    461us  
14.02.26 10:43	kernel	NVRM:     -3    10   FREE                        6510 0x00000000beef9097 0x0000000000000000 0x00064ac4b612343a 0x00064ac4b6123739    767us  
14.02.26 10:43	kernel	NVRM:     -4    10   FREE                        6509 0x00000000beefa0b5 0x0000000000000000 0x00064ac4b61232cc 0x00064ac4b6123435    361us  
14.02.26 10:43	kernel	NVRM:     -5    10   FREE                        6508 0x00000000beef4900 0x0000000000000000 0x00064ac4b6123130 0x00064ac4b61232c7    407us  
14.02.26 10:43	kernel	NVRM:     -6    10   FREE                        6507 0x00000000beef0400 0x0000000000000000 0x00064ac4b612306c 0x00064ac4b612312c    192us  
14.02.26 10:43	kernel	NVRM:     -7    10   FREE                        6506 0x00000000beef4901 0x0000000000000000 0x00064ac4b6122ec9 0x00064ac4b6123067    414us  
14.02.26 10:43	kernel	NVRM: GPU0 RPC event history (CPU <- GSP):
14.02.26 10:43	kernel	NVRM:     entry function                     sequence data0              data1              ts_start           ts_end             duration during_incomplete_rpc
14.02.26 10:43	kernel	NVRM:      0    10   FREE                        6501 0x00000000beef0403 0x0000000000000000 0x00064ac4b6122794 0x00064ac4b6122799      5us  
14.02.26 10:43	kernel	NVRM:     -1    4100 RC_TRIGGERED                   0 0x0000000000000001 0x000000000000006d 0x00064ac4b59773a7 0x00064ac4b59773bb     20us  
14.02.26 10:43	kernel	NVRM:     -2    4102 OS_ERROR_LOG                   0 0x0000000000000000 0x0000000000000000 0x00064ac4b59766ca 0x00064ac4b59766d8     14us  
14.02.26 10:43	kernel	NVRM:     -3    4130 RECOVERY_ACTION                0 0x0000000000000000 0x0000000000000000 0x00064ac4b5117a9d 0x00064ac4b5117a9f      2us  
14.02.26 10:43	kernel	NVRM:     -4    4102 OS_ERROR_LOG                   0 0x0000000000000000 0x0000000000000000 0x00064ac4b5117a37 0x00064ac4b5117a46     15us  
14.02.26 10:43	kernel	NVRM:     -5    4108 UCODE_LIBOS_PRINT              0 0x0000000000000000 0x0000000000000000 0x00064ac4b5116490 0x00064ac4b5116491      1us  
14.02.26 10:43	kernel	NVRM:     -6    4108 UCODE_LIBOS_PRINT              0 0x0000000000000000 0x0000000000000000 0x00064ac4b5116490 0x00064ac4b5116490           
14.02.26 10:43	kernel	NVRM:     -7    4108 UCODE_LIBOS_PRINT              0 0x0000000000000000 0x0000000000000000 0x00064ac4b511645e 0x00064ac4b511645e           
14.02.26 10:43	kernel	NVRM: GPU0 krcRcAndNotifyAllChannels_IMPL: RC all user channels for critical error 120.
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: currentTime: 32e0a208eccbda >= 32e09fedb3a348
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: Timeout was set to: 4000 msecs!
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: currentTime: 32e0a208eccbda >= 32e09fedb3a348
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: Timeout was set to: 4000 msecs!
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: currentTime: 32e0a208eccbda >= 32e09fedb3a348
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: Timeout was set to: 4000 msecs!
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: currentTime: 32e0a208eccbda >= 32e09fedb3a348
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: Timeout was set to: 4000 msecs!
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: currentTime: 32e0a208eccbda >= 32e09fedb3a348
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: Timeout was set to: 4000 msecs!
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: currentTime: 32e0a208eccbda >= 32e09fedb3a348
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: Timeout was set to: 4000 msecs!
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: currentTime: 32e0a208eccbda >= 32e09fedb3a348
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: Timeout was set to: 4000 msecs!
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: currentTime: 32e0a208eccbda >= 32e09fedb3a348
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: Timeout was set to: 4000 msecs!
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: currentTime: 32e0a208eccbda >= 32e09fedb3a348
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: Timeout was set to: 4000 msecs!
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: currentTime: 32e0a208eccbda >= 32e09fedb3a348
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: Timeout was set to: 4000 msecs!
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: currentTime: 32e0a208eccbda >= 32e09fedb3a348
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: Timeout was set to: 4000 msecs!
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: currentTime: 32e0a208eccbda >= 32e09fedb3a348
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: Timeout was set to: 4000 msecs!
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: currentTime: 32e0a208eccbda >= 32e09fedb3a348
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: Timeout was set to: 4000 msecs!
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: currentTime: 32e0a208eccbda >= 32e09fedb3a348
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: Timeout was set to: 4000 msecs!
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: currentTime: 32e0a208eccbda >= 32e09fedb3a348
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: Timeout was set to: 4000 msecs!
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: currentTime: 32e0a208eccbda >= 32e09fedb3a348
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: Timeout was set to: 4000 msecs!
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: currentTime: 32e0a208eccbda >= 32e09fedb3a348
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: Timeout was set to: 4000 msecs!
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: currentTime: 32e0a208eccbda >= 32e09fedb3a348
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: Timeout was set to: 4000 msecs!
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: currentTime: 32e0a208eccbda >= 32e09fedb3a348
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: Timeout was set to: 4000 msecs!
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: currentTime: 32e0a208eccbda >= 32e09fedb3a348
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: Timeout was set to: 4000 msecs!
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: currentTime: 32e0a208eccbda >= 32e09fedb3a348
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: Timeout was set to: 4000 msecs!
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: currentTime: 32e0a208eccbda >= 32e09fedb3a348
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: Timeout was set to: 4000 msecs!
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: currentTime: 32e0a208eccbda >= 32e09fedb3a348
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: Timeout was set to: 4000 msecs!
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: currentTime: 32e0a208eccbda >= 32e09fedb3a348
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: Timeout was set to: 4000 msecs!
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: currentTime: 32e0a208eccbda >= 32e09fedb3a348
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: Timeout was set to: 4000 msecs!
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: currentTime: 32e0a208eccbda >= 32e09fedb3a348
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: Timeout was set to: 4000 msecs!
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: currentTime: 32e0a208eccbda >= 32e09fedb3a348
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: Timeout was set to: 4000 msecs!
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: currentTime: 32e0a208eccbda >= 32e09fedb3a348
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: Timeout was set to: 4000 msecs!
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: currentTime: 32e0a208eccbda >= 32e09fedb3a348
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: Timeout was set to: 4000 msecs!
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: currentTime: 32e0a208eccbda >= 32e09fedb3a348
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: Timeout was set to: 4000 msecs!
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: currentTime: 32e0a208eccbda >= 32e09fedb3a348
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: Timeout was set to: 4000 msecs!
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: currentTime: 32e0a208eccbda >= 32e09fedb3a348
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: Timeout was set to: 4000 msecs!
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: currentTime: 32e0a208eccbda >= 32e09fedb3a348
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: Timeout was set to: 4000 msecs!
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: currentTime: 32e0a208eccbda >= 32e09fedb3a348
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: Timeout was set to: 4000 msecs!
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: currentTime: 32e0a208eccbda >= 32e09fedb3a348
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: Timeout was set to: 4000 msecs!
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: currentTime: 32e0a208eccbda >= 32e09fedb3a348
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: Timeout was set to: 4000 msecs!
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: currentTime: 32e0a208eccbda >= 32e09fedb3a348
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: Timeout was set to: 4000 msecs!
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: currentTime: 32e0a208eccbda >= 32e09fedb3a348
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: Timeout was set to: 4000 msecs!
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: currentTime: 32e0a208eccbda >= 32e09fedb3a348
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: Timeout was set to: 4000 msecs!
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: currentTime: 32e0a208eccbda >= 32e09fedb3a348
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: Timeout was set to: 4000 msecs!
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: currentTime: 32e0a208eccbda >= 32e09fedb3a348
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: Timeout was set to: 4000 msecs!
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: currentTime: 32e0a208eccbda >= 32e09fedb3a348
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: Timeout was set to: 4000 msecs!
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: currentTime: 32e0a208eccbda >= 32e09fedb3a348
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: Timeout was set to: 4000 msecs!
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: currentTime: 32e0a208eccbda >= 32e09fedb3a348
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: Timeout was set to: 4000 msecs!
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: currentTime: 32e0a208eccbda >= 32e09fedb3a348
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: Timeout was set to: 4000 msecs!
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: currentTime: 32e0a208eccbda >= 32e09fedb3a348
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: Timeout was set to: 4000 msecs!
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: currentTime: 32e0a208eccbda >= 32e09fedb3a348
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: Timeout was set to: 4000 msecs!
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: currentTime: 32e0a208eccbda >= 32e09fedb3a348
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: Timeout was set to: 4000 msecs!
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: currentTime: 32e0a208eccbda >= 32e09fedb3a348
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: Timeout was set to: 4000 msecs!
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: currentTime: 32e0a208eccbda >= 32e09fedb3a348
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: Timeout was set to: 4000 msecs!
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: currentTime: 32e0a208eccbda >= 32e09fedb3a348
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: Timeout was set to: 4000 msecs!
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: currentTime: 32e0a208eccbda >= 32e09fedb3a348
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: Timeout was set to: 4000 msecs!
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: currentTime: 32e0a208eccbda >= 32e09fedb3a348
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: Timeout was set to: 4000 msecs!
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: currentTime: 32e0a208eccbda >= 32e09fedb3a348
14.02.26 10:43	kernel	NVRM: GPU0 _threadNodeCheckTimeout: _threadNodeCheckTimeout: Timeout was set to: 4000 msecs!
14.02.26 10:43	kernel	NVRM: GPU0 kgspHealthCheck_TU102: **********************************************************************************
14.02.26 10:43	kernel	NVRM: GPU0 _issueRpcAndWait: rpcRecvPoll failed with status 0x00000062 for fn 10 sequence 6513!
14.02.26 10:43	kernel	NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d00076; hObject=0xbeef0100; paramsStatus=0x00000000; status=0x00000062
14.02.26 10:43	kernel	NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ mem.c:178
14.02.26 10:43	kernel	NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ vaspace_api.c:573
14.02.26 10:43	kernel	NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ mem.c:178
14.02.26 10:43	kernel	NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ vaspace_api.c:573
14.02.26 10:43	kernel	NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
14.02.26 10:43	kernel	NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
14.02.26 10:43	kernel	NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:1375
14.02.26 10:43	kernel	NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
14.02.26 10:43	kernel	NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
14.02.26 10:43	kernel	NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:1375
14.02.26 10:43	kernel	NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
14.02.26 10:43	kernel	NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
14.02.26 10:43	kernel	NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:1375
14.02.26 10:43	kernel	NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
14.02.26 10:43	kernel	NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
14.02.26 10:43	kernel	NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:1375
14.02.26 10:43	kernel	NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
14.02.26 10:43	kernel	NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
14.02.26 10:43	kernel	NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:1375
14.02.26 10:43	kernel	NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
14.02.26 10:43	kernel	NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
14.02.26 10:43	kernel	NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:1375
14.02.26 10:43	kernel	NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ mem.c:178
14.02.26 10:43	kernel	NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ vaspace_api.c:573
14.02.26 10:43	kernel	NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ mem.c:178
14.02.26 10:43	kernel	NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
14.02.26 10:43	kernel	NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
14.02.26 10:43	kernel	NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:1375
14.02.26 10:43	kernel	NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
14.02.26 10:43	kernel	NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
14.02.26 10:43	kernel	NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:1375
14.02.26 10:43	kwin_wayland	A graphics reset attributable to the current GL context occurred.
14.02.26 10:43	kernel	NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
14.02.26 10:43	kernel	NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
14.02.26 10:43	kernel	NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:1375
14.02.26 10:43	kernel	NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
14.02.26 10:43	kernel	NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
14.02.26 10:43	kernel	NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:1375
14.02.26 10:43	kernel	NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
14.02.26 10:43	kernel	NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
14.02.26 10:43	kernel	NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:1375
14.02.26 10:43	kernel	NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
14.02.26 10:43	kernel	NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
14.02.26 10:43	kernel	NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:1375
14.02.26 10:43	kernel	NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
14.02.26 10:43	kernel	NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
14.02.26 10:43	kernel	NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:1375
14.02.26 10:43	kernel	NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
14.02.26 10:43	kernel	NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
14.02.26 10:43	kernel	NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:1375
14.02.26 10:43	kernel	NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
14.02.26 10:43	kernel	NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
14.02.26 10:43	kernel	NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:1375
14.02.26 10:43	kernel	NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
14.02.26 10:43	kernel	NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
14.02.26 10:43	kernel	NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:1375
14.02.26 10:43	kernel	NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
14.02.26 10:43	kernel	NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
14.02.26 10:43	kernel	NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:1375
14.02.26 10:43	kernel	NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
14.02.26 10:43	kernel	NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
14.02.26 10:43	kernel	NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:1375
14.02.26 10:43	kernel	NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
14.02.26 10:43	kernel	NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
14.02.26 10:43	kernel	NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
14.02.26 10:43	kernel	NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
14.02.26 10:43	kernel	NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
14.02.26 10:43	kernel	NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
14.02.26 10:43	kernel	NVRM: GPU0 nvAssertOkFailedNoLog: Assertion failed: Reset required [NV_ERR_RESET_REQUIRED] (0x00000062) returned from pRmApi->Control(pRmApi, pGpu->hInternalClient, pGpu->hInternalSubdevice, NV2080_CTRL_CMD_CE_GET_FAULT_METHOD_BUFFER_SIZE, &params, sizeof(params)) @ kernel_ce.c:857
14.02.26 10:43	kernel	NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (bufSizeInBytes > 0) @ kernel_channel_group_gv100.c:78
14.02.26 10:43	kernel	NVRM: GPU0 kchangrpInit_IMPL:  Fault method buffer allocation failed for group ID 0x5 with status 0x1f
14.02.26 10:43	kernel	NVRM: GPU0 nvAssertOkFailedNoLog: Assertion failed: Invalid argument to call [NV_ERR_INVALID_ARGUMENT] (0x0000001F) returned from kchangrpInit(pGpu, pKernelChannelGroup, pVAS, gfid) @ kernel_channel_group_api.c:273
14.02.26 10:43	kernel	NVRM: GPU0 nvAssertOkFailedNoLog: Assertion failed: Reset required [NV_ERR_RESET_REQUIRED] (0x00000062) returned from pRmApi->Control(pRmApi, pGpu->hInternalClient, pGpu->hInternalSubdevice, NV2080_CTRL_CMD_CE_GET_PHYSICAL_CAPS, &physicalCaps, sizeof(physicalCaps)) @ kernel_ce.c:564
14.02.26 10:43	kernel	NVRM: GPU0 nvAssertOkFailedNoLog: Assertion failed: Reset required [NV_ERR_RESET_REQUIRED] (0x00000062) returned from pRmApi->Control(pRmApi, pGpu->hInternalClient, pGpu->hInternalSubdevice, NV2080_CTRL_CMD_CE_GET_FAULT_METHOD_BUFFER_SIZE, &params, sizeof(params)) @ kernel_ce.c:857
14.02.26 10:43	kernel	NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (bufSizeInBytes > 0) @ kernel_channel_group_gv100.c:78
14.02.26 10:43	kernel	NVRM: GPU0 kchangrpInit_IMPL:  Fault method buffer allocation failed for group ID 0x5 with status 0x1f
14.02.26 10:43	kernel	NVRM: GPU0 nvAssertOkFailedNoLog: Assertion failed: Invalid argument to call [NV_ERR_INVALID_ARGUMENT] (0x0000001F) returned from kchangrpInit(pGpu, pKernelChannelGroup, pVAS, gfid) @ kernel_channel_group_api.c:273
14.02.26 10:43	kernel	NVRM: GPU0 nvAssertOkFailedNoLog: Assertion failed: Invalid argument to call [NV_ERR_INVALID_ARGUMENT] (0x0000001F) returned from pRmApi->AllocWithSecInfo(pRmApi, hClient, hParent, &pChannelGpfifoParams->hPhysChannelGroup, KEPLER_CHANNEL_GROUP_A, NV_PTR_TO_NvP64(&tsgParams), sizeof(tsgParams), RMAPI_ALLOC_FLAGS_SKIP_RPC, NvP64_NULL, &pRmApi->defaultSecInfo) @ kernel_channel.c:383
14.02.26 10:43	kernel	NVRM: GPU0 nvAssertOkFailedNoLog: Assertion failed: Reset required [NV_ERR_RESET_REQUIRED] (0x00000062) returned from pRmApi->Control(pRmApi, pGpu->hInternalClient, pGpu->hInternalSubdevice, NV2080_CTRL_CMD_CE_GET_FAULT_METHOD_BUFFER_SIZE, &params, sizeof(params)) @ kernel_ce.c:857
14.02.26 10:43	kernel	NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (bufSizeInBytes > 0) @ kernel_channel_group_gv100.c:78
14.02.26 10:43	kernel	NVRM: GPU0 kchangrpInit_IMPL:  Fault method buffer allocation failed for group ID 0x5 with status 0x1f
14.02.26 10:43	kernel	NVRM: GPU0 nvAssertOkFailedNoLog: Assertion failed: Invalid argument to call [NV_ERR_INVALID_ARGUMENT] (0x0000001F) returned from kchangrpInit(pGpu, pKernelChannelGroup, pVAS, gfid) @ kernel_channel_group_api.c:273
14.02.26 10:43	kernel	NVRM: GPU0 nvAssertOkFailedNoLog: Assertion failed: Reset required [NV_ERR_RESET_REQUIRED] (0x00000062) returned from pRmApi->Control(pRmApi, pGpu->hInternalClient, pGpu->hInternalSubdevice, NV2080_CTRL_CMD_CE_GET_PHYSICAL_CAPS, &physicalCaps, sizeof(physicalCaps)) @ kernel_ce.c:564
14.02.26 10:43	kernel	NVRM: GPU0 nvAssertOkFailedNoLog: Assertion failed: Reset required [NV_ERR_RESET_REQUIRED] (0x00000062) returned from pRmApi->Control(pRmApi, pGpu->hInternalClient, pGpu->hInternalSubdevice, NV2080_CTRL_CMD_CE_GET_FAULT_METHOD_BUFFER_SIZE, &params, sizeof(params)) @ kernel_ce.c:857
14.02.26 10:43	kernel	NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (bufSizeInBytes > 0) @ kernel_channel_group_gv100.c:78
14.02.26 10:43	kernel	NVRM: GPU0 kchangrpInit_IMPL:  Fault method buffer allocation failed for group ID 0x5 with status 0x1f
14.02.26 10:43	kernel	NVRM: GPU0 nvAssertOkFailedNoLog: Assertion failed: Invalid argument to call [NV_ERR_INVALID_ARGUMENT] (0x0000001F) returned from kchangrpInit(pGpu, pKernelChannelGroup, pVAS, gfid) @ kernel_channel_group_api.c:273
14.02.26 10:43	kernel	NVRM: GPU0 nvAssertOkFailedNoLog: Assertion failed: Invalid argument to call [NV_ERR_INVALID_ARGUMENT] (0x0000001F) returned from pRmApi->AllocWithSecInfo(pRmApi, hClient, hParent, &pChannelGpfifoParams->hPhysChannelGroup, KEPLER_CHANNEL_GROUP_A, NV_PTR_TO_NvP64(&tsgParams), sizeof(tsgParams), RMAPI_ALLOC_FLAGS_SKIP_RPC, NvP64_NULL, &pRmApi->defaultSecInfo) @ kernel_channel.c:383
14.02.26 10:43	kwin_wayland	0x505: GL_OUT_OF_MEMORY error generated. Failed to allocate memory for buffer object.
14.02.26 10:43	kernel	NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: NV_OK == status @ gpu_vaspace.c:4963
14.02.26 10:43	kernel	NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: NV_OK == status @ gpu_vaspace.c:3912
14.02.26 10:43	kernel	NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: NV_OK == status @ gpu_vaspace.c:603
14.02.26 10:43	kernel	NVRM: GPU0 vaspaceapiConstruct_IMPL: Could not construct VA space. Status 62
14.02.26 10:43	kernel	NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ mem.c:178
14.02.26 10:43	kernel	NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
14.02.26 10:43	kernel	NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
14.02.26 10:43	kernel	NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:1375
14.02.26 10:43	kernel	NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ mem.c:178
14.02.26 10:43	kernel	NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
14.02.26 10:43	kernel	NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
14.02.26 10:43	kernel	NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:1375
14.02.26 10:43	kernel	NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: NV_OK == status @ gpu_vaspace.c:4963
14.02.26 10:43	kernel	NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: NV_OK == status @ gpu_vaspace.c:3912
14.02.26 10:43	kernel	NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: NV_OK == status @ gpu_vaspace.c:603
14.02.26 10:43	kernel	NVRM: GPU0 vaspaceapiConstruct_IMPL: Could not construct VA space. Status 62
14.02.26 10:43	kernel	NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: NV_OK == status @ gpu_vaspace.c:4963
14.02.26 10:43	kernel	NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: NV_OK == status @ gpu_vaspace.c:3912
14.02.26 10:43	kernel	NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: NV_OK == status @ gpu_vaspace.c:603
14.02.26 10:43	kernel	NVRM: GPU0 vaspaceapiConstruct_IMPL: Could not construct VA space. Status 62
14.02.26 10:43	kernel	NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ mem.c:178
14.02.26 10:43	kernel	NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
14.02.26 10:43	kernel	NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
14.02.26 10:43	kernel	NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:1375
14.02.26 10:43	kernel	NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ mem.c:178
14.02.26 10:43	kernel	NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
14.02.26 10:43	kernel	NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
14.02.26 10:43	kernel	NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:1375
14.02.26 10:43	kernel	NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: NV_OK == status @ gpu_vaspace.c:4963
14.02.26 10:43	kernel	NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: NV_OK == status @ gpu_vaspace.c:3912
14.02.26 10:43	kernel	NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: NV_OK == status @ gpu_vaspace.c:603
14.02.26 10:43	kernel	NVRM: GPU0 vaspaceapiConstruct_IMPL: Could not construct VA space. Status 62
14.02.26 10:43	systemd-coredump	Process 1092 (kwin_wayland) of user 1000 dumped core.

Stack trace of thread 1092:
#0  0x00007f796a937a82 n/a (libnvidia-eglcore.so.590.48.01 + 0xb37a82)
#1  0x00007f796a94cf1d n/a (libnvidia-eglcore.so.590.48.01 + 0xb4cf1d)
#2  0x00007f796a9492ff n/a (libnvidia-eglcore.so.590.48.01 + 0xb492ff)
#3  0x00007f796a56147e n/a (libnvidia-eglcore.so.590.48.01 + 0x76147e)
#4  0x00007f796a56b1a7 n/a (libnvidia-eglcore.so.590.48.01 + 0x76b1a7)
#5  0x00007f7990ebe2d4 _ZN4KWin13GLFramebuffer19blitFromFramebufferERK5QRectS3_jbb (libkwin.so.6 + 0x2be2d4)
#6  0x00007f7990ebe61e _ZN4KWin13GLFramebuffer20blitFromRenderTargetERKNS_12RenderTargetERKNS_14RenderViewportERK5QRectS9_ (libkwin.so.6 + 0x2be61e)
#7  0x00005600f5c57052 n/a (/usr/bin/kwin_wayland + 0x7f052)
#8  0x00005600f5c5868b n/a (/usr/bin/kwin_wayland + 0x8068b)
#9  0x00007f7990e0f3ec _ZN4KWin14EffectsHandler10drawWindowERKNS_12RenderTargetERKNS_14RenderViewportEPNS_12EffectWindowEiRK7QRegionRNS_15WindowPaintDataE (libkwin.so.6 + 0x20f3ec)
#10 0x00007f7990e0f300 _ZN4KWin14EffectsHandler11paintWindowERKNS_12RenderTargetERKNS_14RenderViewportEPNS_12EffectWindowEiRK7QRegionRNS_15WindowPaintDataE (libkwin.so.6 + 0x20f300)
#11 0x00007f7990f59863 _ZN4KWin14WorkspaceScene11paintWindowERKNS_12RenderTargetERKNS_14RenderViewportEPNS_10WindowItemEiRK7QRegion (libkwin.so.6 + 0x359863)
#12 0x00007f7990f59b5f _ZN4KWin14WorkspaceScene17paintSimpleScreenERKNS_12RenderTargetERKNS_14RenderViewportEiRK7QRegion (libkwin.so.6 + 0x359b5f)
#13 0x00007f7990e0f215 _ZN4KWin14EffectsHandler11paintScreenERKNS_12RenderTargetERKNS_14RenderViewportEiRK7QRegionPNS_6OutputE (libkwin.so.6 + 0x20f215)
#14 0x00007f7990f59ca7 _ZN4KWin14WorkspaceScene5paintERKNS_12RenderTargetERK7QRegion (libkwin.so.6 + 0x359ca7)
#15 0x00007f7990f333a6 _ZN4KWin9SceneView5paintERKNS_12RenderTargetERK7QRegion (libkwin.so.6 + 0x3333a6)
#16 0x00007f7990d8af17 n/a (libkwin.so.6 + 0x18af17)
#17 0x00007f7990d8f621 _ZN4KWin10Compositor9compositeEPNS_10RenderLoopE (libkwin.so.6 + 0x18f621)
#18 0x00007f798e3d734f n/a (libQt6Core.so.6 + 0x1d734f)
#19 0x00007f7990db4c0b _ZN4KWin10RenderLoop14frameRequestedEPS0_ (libkwin.so.6 + 0x1b4c0b)
#20 0x00007f798e3c4936 _ZN7QObject5eventEP6QEvent (libQt6Core.so.6 + 0x1c4936)
#21 0x00007f798f5021c0 _ZN19QApplicationPrivate13notify_helperEP7QObjectP6QEvent (libQt6Widgets.so.6 + 0x1021c0)
#22 0x00007f798e36a958 _ZN16QCoreApplication15notifyInternal2EP7QObjectP6QEvent (libQt6Core.so.6 + 0x16a958)
#23 0x00007f798e52c9af _ZN14QTimerInfoList14activateTimersEv (libQt6Core.so.6 + 0x32c9af)
#24 0x00007f798e52e549 _ZN20QEventDispatcherUNIX13processEventsE6QFlagsIN10QEventLoop17ProcessEventsFlagEE (libQt6Core.so.6 + 0x32e549)
#25 0x00007f798f134b73 _ZN23QUnixEventDispatcherQPA13processEventsE6QFlagsIN10QEventLoop17ProcessEventsFlagEE (libQt6Gui.so.6 + 0x734b73)
#26 0x00007f798e375786 _ZN10QEventLoop4execE6QFlagsINS_17ProcessEventsFlagEE (libQt6Core.so.6 + 0x175786)
#27 0x00007f798e36f3f1 _ZN16QCoreApplication4execEv (libQt6Core.so.6 + 0x16f3f1)
#28 0x00005600f5c14a94 n/a (/usr/bin/kwin_wayland + 0x3ca94)
#29 0x00007f798dc36635 n/a (libc.so.6 + 0x27635)
#30 0x00007f798dc366e9 __libc_start_main (libc.so.6 + 0x276e9)
#31 0x00005600f5c1af55 n/a (/usr/bin/kwin_wayland + 0x42f55)

Stack trace of thread 1170:
#0  0x00007f798dcadfe2 n/a (libc.so.6 + 0x9efe2)
#1  0x00007f798dca216c n/a (libc.so.6 + 0x9316c)
#2  0x00007f798dca21b4 n/a (libc.so.6 + 0x931b4)
#3  0x00007f798dd1ca36 ppoll (libc.so.6 + 0x10da36)
#4  0x00007f798cedd784 n/a (libglib-2.0.so.0 + 0x60784)
#5  0x00007f798cedd865 g_main_context_iteration (libglib-2.0.so.0 + 0x60865)
#6  0x00007f798e648152 _ZN20QEventDispatcherGlib13processEventsE6QFlagsIN10QEventLoop17ProcessEventsFlagEE (libQt6Core.so.6 + 0x448152)
#7  0x00007f798e375786 _ZN10QEventLoop4execE6QFlagsINS_17ProcessEventsFlagEE (libQt6Core.so.6 + 0x175786)
#8  0x00007f798e49041e _ZN7QThread4execEv (libQt6Core.so.6 + 0x29041e)
#9  0x00007f798e52f899 n/a (libQt6Core.so.6 + 0x32f899)
#10 0x00007f798dca598b n/a (libc.so.6 + 0x9698b)
#11 0x00007f798dd29a0c n/a (libc.so.6 + 0x11aa0c)

Stack trace of thread 6692:
#0  0x00007f798dcadfe2 n/a (libc.so.6 + 0x9efe2)
#1  0x00007f798dca216c n/a (libc.so.6 + 0x9316c)
#2  0x00007f798dca27dc n/a (libc.so.6 + 0x937dc)
#3  0x00007f798dca50a8 pthread_cond_timedwait (libc.so.6 + 0x960a8)
#4  0x00007f798e533dc5 _ZN14QWaitCondition4waitEP6QMutex14QDeadlineTimer (libQt6Core.so.6 + 0x333dc5)
#5  0x00007f798e48ee69 n/a (libQt6Core.so.6 + 0x28ee69)
#6  0x00007f798e52f899 n/a (libQt6Core.so.6 + 0x32f899)
#7  0x00007f798dca598b n/a (libc.so.6 + 0x9698b)
#8  0x00007f798dd29a0c n/a (libc.so.6 + 0x11aa0c)

Stack trace of thread 1141:
#0  0x00007f798dcadfe2 n/a (libc.so.6 + 0x9efe2)
#1  0x00007f798dca216c n/a (libc.so.6 + 0x9316c)
#2  0x00007f798dca21b4 n/a (libc.so.6 + 0x931b4)
#3  0x00007f798dd1ca36 ppoll (libc.so.6 + 0x10da36)
#4  0x00007f798cedd784 n/a (libglib-2.0.so.0 + 0x60784)
#5  0x00007f798cedd865 g_main_context_iteration (libglib-2.0.so.0 + 0x60865)
#6  0x00007f798e648152 _ZN20QEventDispatcherGlib13processEventsE6QFlagsIN10QEventLoop17ProcessEventsFlagEE (libQt6Core.so.6 + 0x448152)
#7  0x00007f798e375786 _ZN10QEventLoop4execE6QFlagsINS_17ProcessEventsFlagEE (libQt6Core.so.6 + 0x175786)
#8  0x00007f798e49041e _ZN7QThread4execEv (libQt6Core.so.6 + 0x29041e)
#9  0x00007f798e52f899 n/a (libQt6Core.so.6 + 0x32f899)
#10 0x00007f798dca598b n/a (libc.so.6 + 0x9698b)
#11 0x00007f798dd29a0c n/a (libc.so.6 + 0x11aa0c)

Stack trace of thread 1139:
#0  0x00007f798dcadfe2 n/a (libc.so.6 + 0x9efe2)
#1  0x00007f798dca216c n/a (libc.so.6 + 0x9316c)
#2  0x00007f798dca27dc n/a (libc.so.6 + 0x937dc)
#3  0x00007f798dca52d2 pthread_cond_clockwait (libc.so.6 + 0x962d2)
#4  0x00007f799108eedc n/a (libkwin.so.6 + 0x48eedc)
#5  0x00007f799108fc0a n/a (libkwin.so.6 + 0x48fc0a)
#6  0x00007f7991089c66 n/a (libkwin.so.6 + 0x489c66)
#7  0x00007f798dcaae64 n/a (libc.so.6 + 0x9be64)
#8  0x00007f798dcaaed9 pthread_once (libc.so.6 + 0x9bed9)
#9  0x00007f799108a6ff n/a (libkwin.so.6 + 0x48a6ff)
#10 0x00007f798e48a585 n/a (libQt6Core.so.6 + 0x28a585)
#11 0x00007f798e52f899 n/a (libQt6Core.so.6 + 0x32f899)
#12 0x00007f798dca598b n/a (libc.so.6 + 0x9698b)
#13 0x00007f798dd29a0c n/a (libc.so.6 + 0x11aa0c)

Stack trace of thread 1118:
#0  0x00007f798dcadfe2 n/a (libc.so.6 + 0x9efe2)
#1  0x00007f798dca216c n/a (libc.so.6 + 0x9316c)
#2  0x00007f798dca21b4 n/a (libc.so.6 + 0x931b4)
#3  0x00007f798dd1ca36 ppoll (libc.so.6 + 0x10da36)
#4  0x00007f798cedd784 n/a (libglib-2.0.so.0 + 0x60784)
#5  0x00007f798cedd865 g_main_context_iteration (libglib-2.0.so.0 + 0x60865)
#6  0x00007f798e648152 _ZN20QEventDispatcherGlib13processEventsE6QFlagsIN10QEventLoop17ProcessEventsFlagEE (libQt6Core.so.6 + 0x448152)
#7  0x00007f798e375786 _ZN10QEventLoop4execE6QFlagsINS_17ProcessEventsFlagEE (libQt6Core.so.6 + 0x175786)
#8  0x00007f798e49041e _ZN7QThread4execEv (libQt6Core.so.6 + 0x29041e)
#9  0x00007f798e97603e n/a (libQt6DBus.so.6 + 0x3703e)
#10 0x00007f798e52f899 n/a (libQt6Core.so.6 + 0x32f899)
#11 0x00007f798dca598b n/a (libc.so.6 + 0x9698b)
#12 0x00007f798dd29a0c n/a (libc.so.6 + 0x11aa0c)
ELF object binary architecture: AMD x86-64

Not able to insert full log, post size limits, so I’ve attached the file
NVIDIA fail logs.txt (128.5 KB)

2

Some update.
Tried to use 570 version of a driver, video-hybrid-amd-nvidia-570xx-prime to be more precise, as a lot of post all around the Internet suggest but the issue keeps happening. Only the error in the KSystemLog a bit different:

20 19:56:47 PC-NAME kwin_wayland[1135]: Please report this at https://forums.developer.nvidia.com/c/gpu-graphics/linux
20 19:56:47 PC-NAME kwin_wayland[1135]: With the output of 'sudo dmesg' and 'journalctl --user-unit plasma-kwin_wayland --boot 0'

As the error requested I’m adding outputs of commands with additional journalctl --user-unit plasma-kwin_wayland --boot -1
journal-boot-1.txt (31.0 KB)
journal-boot0.txt (2.0 KB)
journal-dmesg.txt (120.7 KB)

3

What you are describing is typical of wayland+nvidia, they don’t work well together, and weird bugs/instability will appear.

In fact, the baseline for determining whether or not nvidia+everything else is working properly, is using xorg. So if you switch display servers, (if xorg is installed) by logging out → clicking the top right icon → selecting to use the xorg option → logging back in, and then test your system, you can quickly find out whether or not it’s a wayland(experimental) problem or perhaps something else.

I see you are reporting you have an “NVIDIA GB203 [GeForce RTX 5080” and many people have reported problems with that card, and I believe you in particular need the nvidia driver + open kernel modules to use it. I’m not sure what the exact package on arch linux is, it might be something similar to nvidia-open

or actually in manjaro you have a GUI window that displays drivers for you to download, and you should be able to find an “nvidia open” driver there.

Other than a hardware configuration, that may be an issue, I tihnk those are the primary concerns.

4

Ok, then I’ll just wait until this combination will be more stable. At the mean time I get to know about the REISUB to reboot the system in described cases more gracefully.

5

I would rename the topic to something like “GSP crashes on 590.48.01”

I doubt this is wayland related. More likely is some weird combination of kernel and driver version, or less likely something Manjaro/config-specific.

I’m using wayland on Fedora 43 with drivers from RPM Fusion and the experience is great with no tweaking required.

6

Today I experienced what seems to be a very similar crash, on a brand new laptop running Archlinux, Xorg and I3.

I was just using the laptop (Brave, Emacs, terminal, etc), I left it for a minute, when I was back the whole screen froze, mouse and keyboard inputs did nothing, not even ctrl + alt + Fkeys.

Then I ssh into the laptop, looked at the journal and saw the GPU driver crash.

Here are some details about my hardware setup:

Laptop: Lenovo 21RS003NUS

Driver version: 590.48.01

Kernel and driver packages:

  • linux 6.19.6.arch1-1
  • linux-firmware-nvidia 20260309-1
  • nvidia-open 590.48.01-15
  • nvidia-settings 590.48.01-1
  • nvidia-utils 590.48.01-4

Trigger (according to Gemini): Brave/Chromium hardware acceleration triggered a Context Switch Timeout (Xid 109) which then escalated to a GSP Firmware Crash (Xid 119/120)

Current workaround: creating /etc/modprobe.d/nvidia-gsp.conf with the following content:

options nvidia NVreg_EnableGpuFirmware=0

Unfortunately I forgot to run nvidia-bug-report.sh right after the crash, but I did collect the journal logs containing the crash and later run the script.

gpu-crash.log (86.7 KB)

nvidia-bug-report.log.gz (511.7 KB)