Xid 119 GSP Timeout on RTX 6000 Pro Blackwell (575.64.3) under Load – Reproducible Crash

Graphics / Linux Linux
1

Hardware:

  • GPU: RTX 6000 Pro Blackwell (sm120)
  • Board Serial Number: 1791625015739
  • Motherboard: System76 Thelio Mega / TRX50 AI TOP
  • BIOS: F4 Z5 03/06/2025

Software Environment:

  • OS: Ubuntu 24.04 (64-bit)
  • Kernel: 6.14.0-22-generic
  • Driver: 575.64.3 (released 2025-07-01)
  • CUDA: 12.9
  • Driver mode: nvidia-open (open kernel modules)
  • GSP: Enabled (required for open driver path)

Problem Description:
Whenever the GPU is placed under sustained load (e.g., running Ollama for LLM inference), the system experiences a full GPU failure. The error is always a GSP timeout at GSP_RM_CONTROL (function 76), resulting in Xid 119, and requires a full system reboot to recover.

This is 100% reproducible — occurs consistently within 1–2 minutes of initiating load on the GPU.

Log Excerpt (from dmesg):

php

CopyEdit

NVRM: Xid (PCI:0000:81:00): 119, pid=97581, name=ollama, Timeout after 45s of waiting for RPC response from GPU0 GSP! Expected function 76 (GSP_RM_CONTROL)
...
NVRM: _issueRpcAndWait: rpcRecvPoll timedout for fn 76!

Steps to Reproduce:

  1. Boot into Ubuntu 24.04 with kernel 6.14 and driver 575.64.3 (nvidia-open)
  2. Run any sustained GPU compute task using Ollama or similar (loads LLM weights and runs inference)
  3. Wait for 30–90 seconds under load
  4. Observe Xid 119 GSP Timeout
  5. System enters unrecoverable GPU failure state — requires full reboot

What I’ve Tried:

  • Tried with multiple kernel versions (6.14, 6.5, 6.1 LTS) — issue persists
  • Driver versions 570.x and 575.x — all affected
  • GSP cannot be disabled due to open driver requirement
  • Verified full hardware compatibility and PCIe stability
  • Confirmed all system firmware and BIOS are fully updated

Impact:
The GPU cannot be used for any production workload under Linux due to this bug. It appears to be a low-level GSP firmware or RPC handler issue under the nvidia-open driver path, possibly triggered by specific workloads issuing rapid RPCs to GSP.

Bug Report Attachment:
nvidia-bug-report.log.gz (439.0 KB)

Attached

Additional Info:
If more debug steps are needed (e.g., enabling more verbose RPC traces or firmware dumps), I’m happy to test them.

2

Same problem with my Nvidia RTX 5090 FE, Open Kernel Module 575.64.05 on an MS-7B09/X399 SLI PLUS motherboard (the incident happend ~20 days after the last reboot; excerpt from kern.log) running Proxmox 8:

...At first, some startup messages:
2025-07-26T15:16:22.070112+02:00 ThreadripperIntern kernel: [    0.000000] Linux version 6.8.12-12-pve (build@proxmox) (gcc (Debian 12.2.0-14+deb12u1) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #1 SMP PREEMPT_DYNAMIC PMX 6.8.12-12 (2025-07-14T13:20Z) ()
2025-07-26T15:16:22.070120+02:00 ThreadripperIntern kernel: [    0.000000] Command line: initrd=\EFI\proxmox\6.8.12-12-pve\initrd.img-6.8.12-12-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs
...
2025-07-26T15:16:22.070395+02:00 ThreadripperIntern kernel: [    0.000000] DMI: Micro-Star International Co., Ltd. MS-7B09/X399 SLI PLUS (MS-7B09), BIOS A.89 08/11/2022
...
2025-07-26T15:16:22.073345+02:00 ThreadripperIntern kernel: [   15.455895] nvidia: module verification failed: signature and/or required key missing - tainting kernel
2025-07-26T15:16:22.073345+02:00 ThreadripperIntern kernel: [   15.563081] nvidia-nvlink: Nvlink Core is being initialized, major device number 511
2025-07-26T15:16:22.073345+02:00 ThreadripperIntern kernel: [   15.566073] 
2025-07-26T15:16:22.073346+02:00 ThreadripperIntern kernel: [   15.570203] nvidia 0000:0a:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=none:owns=none
2025-07-26T15:16:22.073350+02:00 ThreadripperIntern kernel: [   15.586093] NVRM: loading NVIDIA UNIX Open Kernel Module for x86_64  575.64.05  Release Build  (dvs-builder@U22-A23-13-1)  Fri Jul 18 16:00:10 UTC 2025
...
2025-07-26T15:16:22.073477+02:00 ThreadripperIntern kernel: [   15.648101] nvidia-modeset: Loading NVIDIA UNIX Open Kernel Mode Setting Driver for x86_64  575.64.05  Release Build  (dvs-builder@U22-A23-13-1)  Fri Jul 18 15:48:34 UTC 2025
...
2025-07-26T15:16:22.073573+02:00 ThreadripperIntern kernel: [   16.522542] [drm] [nvidia-drm] [GPU ID 0x00000a00] Loading driver
2025-07-26T15:16:22.073574+02:00 ThreadripperIntern kernel: [   16.522553] [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:0a:00.0 on minor 1
...

2025-07-26T15:16:22.073621+02:00 ThreadripperIntern kernel: [   21.996827] audit: type=1400 audit(1753535781.053:6): apparmor="STATUS" operation="profile_load" profile="unconfined" name="nvidia_modprobe" pid=1897 comm="apparmor_parser"
2025-07-26T15:16:22.073621+02:00 ThreadripperIntern kernel: [   21.996866] audit: type=1400 audit(1753535781.053:7): apparmor="STATUS" operation="profile_load" profile="unconfined" name="nvidia_modprobe//kmod" pid=1897 comm="apparmor_parser"
...

2025-07-26T15:16:23.143774+02:00 ThreadripperIntern kernel: [   24.085042] NVRM: kbifInitLtr_GB202: LTR is disabled in the hierarchy
2025-07-26T15:16:23.160780+02:00 ThreadripperIntern kernel: [   24.101707] Loading iSCSI transport class v2.0-870.
2025-07-26T15:16:23.408884+02:00 ThreadripperIntern kernel: [   24.350188] NVRM: Persistence mode is deprecated and will be removed in a future release. Please use nvidia-persistenced instead.
...here comes the incident:
2025-08-15T23:12:29.437778+02:00 ThreadripperIntern kernel: [1756572.638323] [UFW BLOCK] IN=vmbr0 OUT= MAC=33:33:00:00:00:01:f0:9f:c2:2c:xx:xx:xx:xx SRC=fe80:0000:0000:0000:xxxx:xxxx:xxxx:xxxx DST=xxxx:0000:0000:0000:0000:0000:0000:xxxx LEN=319 TC=0 HOPLIMIT=1 FLOWLBL=297128 PROTO=UDP SPT=54299 DPT=10001 LEN=279 
2025-08-15T23:12:36.496777+02:00 ThreadripperIntern kernel: [1756579.696915] [UFW BLOCK] IN=vmbr0 OUT= MAC=33:33:00:00:00:01:f0:9f:c2:cb:xx:xx:xx:xx SRC=fe80:0000:0000:0000:xxxx:xxxx:xxxx:xxxx DST=xxxx:0000:0000:0000:0000:0000:0000:xxxx LEN=320 TC=0 HOPLIMIT=1 FLOWLBL=524291 PROTO=UDP SPT=35385 DPT=10001 LEN=280 
2025-08-15T23:12:48.792185+02:00 ThreadripperIntern kernel: [1756591.995476] NVRM: _kgspLogXid119: ********************************* GSP Timeout **********************************
2025-08-15T23:12:48.792198+02:00 ThreadripperIntern kernel: [1756591.997880] NVRM: _kgspLogXid119: Note: Please also check logs above.
2025-08-15T23:12:48.797071+02:00 ThreadripperIntern kernel: [1756592.000338] NVRM: GPU at PCI:0000:0a:00: GPU-701b524f-c979-7284-9737-0fecd8ca5f9b
2025-08-15T23:12:48.797077+02:00 ThreadripperIntern kernel: [1756592.002781] NVRM: GPU Board Serial Number: 1791625003310
2025-08-15T23:12:48.798832+02:00 ThreadripperIntern kernel: [1756592.004541] NVRM: Xid (PCI:0000:0a:00): 119, pid=305083, name=llama-server, Timeout after 45s of waiting for RPC response from GPU0 GSP! Expected function 76 (GSP_RM_CONTROL) (0x20800a9a 0x8).
2025-08-15T23:12:48.802060+02:00 ThreadripperIntern kernel: [1756592.006173] NVRM: GPU0 GSP RPC buffer contains function 76 (GSP_RM_CONTROL) and data 0x0000000020800a9a 0x0000000000000008.
2025-08-15T23:12:48.802065+02:00 ThreadripperIntern kernel: [1756592.007769] NVRM: GPU0 RPC history (CPU -> GSP):
2025-08-15T23:12:48.803632+02:00 ThreadripperIntern kernel: [1756592.009342] NVRM:     entry function                   data0              data1              ts_start           ts_end             duration actively_polling
2025-08-15T23:12:48.806777+02:00 ThreadripperIntern kernel: [1756592.010920] NVRM:      0    76   GSP_RM_CONTROL        0x0000000020800a9a 0x0000000000000008 0x00063c6dd76fb99f 0x0000000000000000          y
2025-08-15T23:12:48.806781+02:00 ThreadripperIntern kernel: [1756592.012485] NVRM:     -1    76   GSP_RM_CONTROL        0x0000000020800a9a 0x0000000000000008 0x00063c6dd691da54 0x00063c6dd691db44    240us  
2025-08-15T23:12:48.809862+02:00 ThreadripperIntern kernel: [1756592.014035] NVRM:     -2    76   GSP_RM_CONTROL        0x0000000020800a9a 0x0000000000000008 0x00063c6dd5a52f98 0x00063c6dd5a54f24   8076us  
2025-08-15T23:12:48.809867+02:00 ThreadripperIntern kernel: [1756592.015568] NVRM:     -3    76   GSP_RM_CONTROL        0x0000000020800a9a 0x0000000000000008 0x00063c6dd4c769bb 0x00063c6dd4c76ad2    279us  
2025-08-15T23:12:48.811372+02:00 ThreadripperIntern kernel: [1756592.017082] NVRM:     -4    76   GSP_RM_CONTROL        0x0000000020800a9a 0x0000000000000008 0x00063c6dd3dabf4c 0x00063c6dd3dade37   7915us  
2025-08-15T23:12:48.814353+02:00 ThreadripperIntern kernel: [1756592.018587] NVRM:     -5    76   GSP_RM_CONTROL        0x0000000020800a9a 0x0000000000000008 0x00063c6dd2fcfa6e 0x00063c6dd2fcfb8c    286us  
2025-08-15T23:12:48.814357+02:00 ThreadripperIntern kernel: [1756592.020063] NVRM:     -6    76   GSP_RM_CONTROL        0x0000000020800a9a 0x0000000000000008 0x00063c6dd2104ea0 0x00063c6dd2106e03   8035us  
2025-08-15T23:12:48.815819+02:00 ThreadripperIntern kernel: [1756592.021529] NVRM:     -7    76   GSP_RM_CONTROL        0x0000000020800a9a 0x0000000000000008 0x00063c6dd1328635 0x00063c6dd1328753    286us  
2025-08-15T23:12:48.818693+02:00 ThreadripperIntern kernel: [1756592.022977] NVRM: GPU0 RPC event history (CPU <- GSP):
2025-08-15T23:12:48.818698+02:00 ThreadripperIntern kernel: [1756592.024400] NVRM:     entry function                   data0              data1              ts_start           ts_end             duration during_incomplete_rpc
2025-08-15T23:12:48.821540+02:00 ThreadripperIntern kernel: [1756592.025830] NVRM:      0    4124 GSP_LOCKDOWN_NOTICE   0x0000000000000000 0x0000000000000000 0x00063ad4dd7389a2 0x00063ad4dd7389a3      1us  
2025-08-15T23:12:48.821545+02:00 ThreadripperIntern kernel: [1756592.027249] NVRM:     -1    4124 GSP_LOCKDOWN_NOTICE   0x0000000000000001 0x0000000000000000 0x00063ad4dd737586 0x00063ad4dd737586           
2025-08-15T23:12:48.822944+02:00 ThreadripperIntern kernel: [1756592.028653] NVRM:     -2    4124 GSP_LOCKDOWN_NOTICE   0x0000000000000000 0x0000000000000000 0x00063ad4dd73748e 0x00063ad4dd73748e           
2025-08-15T23:12:48.825709+02:00 ThreadripperIntern kernel: [1756592.030043] NVRM:     -3    4124 GSP_LOCKDOWN_NOTICE   0x0000000000000001 0x0000000000000000 0x00063ad4dd7370e5 0x00063ad4dd7370e6      1us  
2025-08-15T23:12:48.825715+02:00 ThreadripperIntern kernel: [1756592.031418] NVRM:     -4    4124 GSP_LOCKDOWN_NOTICE   0x0000000000000000 0x0000000000000000 0x00063ad4dd720dfb 0x00063ad4dd720dfb           
2025-08-15T23:12:48.827066+02:00 ThreadripperIntern kernel: [1756592.032775] NVRM:     -5    4124 GSP_LOCKDOWN_NOTICE   0x0000000000000001 0x0000000000000000 0x00063ad4dd720d72 0x00063ad4dd720d72           
2025-08-15T23:12:48.828405+02:00 ThreadripperIntern kernel: [1756592.034115] NVRM:     -6    4128 GSP_POST_NOCAT_RECORD 0x0000000000000002 0x0000000000000027 0x00063ad4dd72091f 0x00063ad4dd720922      3us  
2025-08-15T23:12:48.829729+02:00 ThreadripperIntern kernel: [1756592.035439] NVRM:     -7    4124 GSP_LOCKDOWN_NOTICE   0x0000000000000000 0x0000000000000000 0x00063ad4dd71fb47 0x00063ad4dd71fb47           
2025-08-15T23:12:48.831040+02:00 ThreadripperIntern kernel: [1756592.036751] CPU: 2 PID: 305083 Comm: llama-server Tainted: P           OE      6.8.12-12-pve #1
2025-08-15T23:12:48.891925+02:00 ThreadripperIntern kernel: [1756592.038038] Hardware name: Micro-Star International Co., Ltd. MS-7B09/X399 SLI PLUS (MS-7B09), BIOS A.89 08/11/2022
2025-08-15T23:12:48.891931+02:00 ThreadripperIntern kernel: [1756592.039311] Call Trace:
2025-08-15T23:12:48.891931+02:00 ThreadripperIntern kernel: [1756592.040563]  <TASK>
2025-08-15T23:12:48.891932+02:00 ThreadripperIntern kernel: [1756592.041797]  dump_stack_lvl+0x76/0xa0
2025-08-15T23:12:48.891932+02:00 ThreadripperIntern kernel: [1756592.043021]  dump_stack+0x10/0x20
2025-08-15T23:12:48.891933+02:00 ThreadripperIntern kernel: [1756592.044222]  os_dump_stack+0xe/0x20 [nvidia]
2025-08-15T23:12:48.891933+02:00 ThreadripperIntern kernel: [1756592.045560]  _kgspRpcRecvPoll+0x593/0x760 [nvidia]
2025-08-15T23:12:48.891933+02:00 ThreadripperIntern kernel: [1756592.046937]  _issueRpcAndWait+0xd2/0x900 [nvidia]
2025-08-15T23:12:48.891934+02:00 ThreadripperIntern kernel: [1756592.048272]  ? srso_return_thunk+0x5/0x5f
2025-08-15T23:12:48.891934+02:00 ThreadripperIntern kernel: [1756592.049413]  ? osGetCurrentThread+0x26/0x60 [nvidia]
2025-08-15T23:12:48.891935+02:00 ThreadripperIntern kernel: [1756592.050751]  ? srso_return_thunk+0x5/0x5f
2025-08-15T23:12:48.891935+02:00 ThreadripperIntern kernel: [1756592.051857]  ? srso_return_thunk+0x5/0x5f
2025-08-15T23:12:48.891936+02:00 ThreadripperIntern kernel: [1756592.052946]  ? os_mem_set+0x14/0x20 [nvidia]
2025-08-15T23:12:48.891936+02:00 ThreadripperIntern kernel: [1756592.054159]  ? srso_return_thunk+0x5/0x5f
2025-08-15T23:12:48.891936+02:00 ThreadripperIntern kernel: [1756592.055216]  rpcRmApiControl_GSP+0x76f/0x940 [nvidia]
2025-08-15T23:12:48.891937+02:00 ThreadripperIntern kernel: [1756592.056439]  kperfBoostSet_IMPL+0x69/0x70 [nvidia]
2025-08-15T23:12:48.891937+02:00 ThreadripperIntern kernel: [1756592.057658]  resControl_IMPL+0x1a9/0x1b0 [nvidia]
2025-08-15T23:12:48.891938+02:00 ThreadripperIntern kernel: [1756592.058837]  serverControl+0x47e/0x590 [nvidia]
2025-08-15T23:12:48.891938+02:00 ThreadripperIntern kernel: [1756592.060000]  _rmapiRmControl+0x4f2/0x820 [nvidia]
2025-08-15T23:12:48.891950+02:00 ThreadripperIntern kernel: [1756592.061165]  rmapiControlWithSecInfo+0x79/0x140 [nvidia]
2025-08-15T23:12:48.891951+02:00 ThreadripperIntern kernel: [1756592.062306]  ? os_acquire_spinlock+0x12/0x30 [nvidia]
2025-08-15T23:12:48.891952+02:00 ThreadripperIntern kernel: [1756592.063395]  ? srso_return_thunk+0x5/0x5f
2025-08-15T23:12:48.891953+02:00 ThreadripperIntern kernel: [1756592.064326]  rmapiControlWithSecInfoTls+0x8f/0xf0 [nvidia]
2025-08-15T23:12:48.891954+02:00 ThreadripperIntern kernel: [1756592.065429]  _nv04ControlWithSecInfo+0x8d/0xa0 [nvidia]
2025-08-15T23:12:48.891955+02:00 ThreadripperIntern kernel: [1756592.066509]  ? srso_return_thunk+0x5/0x5f
2025-08-15T23:12:48.891956+02:00 ThreadripperIntern kernel: [1756592.067392]  ? native_send_call_func_single_ipi+0x13/0x20
2025-08-15T23:12:48.891957+02:00 ThreadripperIntern kernel: [1756592.068262]  ? srso_return_thunk+0x5/0x5f
2025-08-15T23:12:48.891958+02:00 ThreadripperIntern kernel: [1756592.069114]  ? __smp_call_single_queue+0xe0/0x140
2025-08-15T23:12:48.891958+02:00 ThreadripperIntern kernel: [1756592.069952]  ? srso_return_thunk+0x5/0x5f
2025-08-15T23:12:48.891959+02:00 ThreadripperIntern kernel: [1756592.070771]  ? srso_return_thunk+0x5/0x5f
2025-08-15T23:12:48.891959+02:00 ThreadripperIntern kernel: [1756592.071571]  ? security_capable+0x47/0x80
2025-08-15T23:12:48.891960+02:00 ThreadripperIntern kernel: [1756592.072355]  RmIoctl+0x90b/0xda0 [nvidia]
2025-08-15T23:12:48.891961+02:00 ThreadripperIntern kernel: [1756592.073333]  ? os_get_current_tick+0x3c/0xb0 [nvidia]
2025-08-15T23:12:48.891961+02:00 ThreadripperIntern kernel: [1756592.074223]  ? srso_return_thunk+0x5/0x5f
2025-08-15T23:12:48.891962+02:00 ThreadripperIntern kernel: [1756592.074959]  ? os_acquire_spinlock+0x12/0x30 [nvidia]
2025-08-15T23:12:48.891964+02:00 ThreadripperIntern kernel: [1756592.075824]  ? srso_return_thunk+0x5/0x5f
2025-08-15T23:12:48.891964+02:00 ThreadripperIntern kernel: [1756592.076528]  ? portSyncSpinlockAcquire+0x18/0x30 [nvidia]
2025-08-15T23:12:48.891964+02:00 ThreadripperIntern kernel: [1756592.077386]  ? srso_return_thunk+0x5/0x5f
2025-08-15T23:12:48.891965+02:00 ThreadripperIntern kernel: [1756592.078059]  rm_ioctl+0x66/0x4f0 [nvidia]
2025-08-15T23:12:48.891965+02:00 ThreadripperIntern kernel: [1756592.078930]  ? srso_return_thunk+0x5/0x5f
2025-08-15T23:12:48.891965+02:00 ThreadripperIntern kernel: [1756592.079573]  nvidia_unlocked_ioctl+0x69c/0x920 [nvidia]
2025-08-15T23:12:48.891966+02:00 ThreadripperIntern kernel: [1756592.080336]  __x64_sys_ioctl+0xa3/0xf0
2025-08-15T23:12:48.891966+02:00 ThreadripperIntern kernel: [1756592.080947]  x64_sys_call+0xa71/0x2480
2025-08-15T23:12:48.891966+02:00 ThreadripperIntern kernel: [1756592.081540]  do_syscall_64+0x81/0x170
2025-08-15T23:12:48.891967+02:00 ThreadripperIntern kernel: [1756592.082114]  ? srso_return_thunk+0x5/0x5f
2025-08-15T23:12:48.891968+02:00 ThreadripperIntern kernel: [1756592.082670]  ? update_process_times+0x8e/0xb0
2025-08-15T23:12:48.891968+02:00 ThreadripperIntern kernel: [1756592.083212]  ? srso_return_thunk+0x5/0x5f
2025-08-15T23:12:48.891969+02:00 ThreadripperIntern kernel: [1756592.083735]  ? srso_return_thunk+0x5/0x5f
2025-08-15T23:12:48.891969+02:00 ThreadripperIntern kernel: [1756592.084250]  ? timerqueue_add+0xa6/0xd0
2025-08-15T23:12:48.891970+02:00 ThreadripperIntern kernel: [1756592.084758]  ? srso_return_thunk+0x5/0x5f
2025-08-15T23:12:48.891971+02:00 ThreadripperIntern kernel: [1756592.085263]  ? srso_return_thunk+0x5/0x5f
2025-08-15T23:12:48.891971+02:00 ThreadripperIntern kernel: [1756592.085762]  ? ktime_get+0x48/0xc0
2025-08-15T23:12:48.891972+02:00 ThreadripperIntern kernel: [1756592.086249]  ? __pfx_tick_nohz_highres_handler+0x10/0x10
2025-08-15T23:12:48.891973+02:00 ThreadripperIntern kernel: [1756592.086728]  ? srso_return_thunk+0x5/0x5f
2025-08-15T23:12:48.891973+02:00 ThreadripperIntern kernel: [1756592.087199]  ? lapic_next_event+0x15/0x30
2025-08-15T23:12:48.891974+02:00 ThreadripperIntern kernel: [1756592.087665]  ? srso_return_thunk+0x5/0x5f
2025-08-15T23:12:48.891975+02:00 ThreadripperIntern kernel: [1756592.088122]  ? clockevents_program_event+0xb6/0x140
2025-08-15T23:12:48.891976+02:00 ThreadripperIntern kernel: [1756592.088586]  ? srso_return_thunk+0x5/0x5f
2025-08-15T23:12:48.891976+02:00 ThreadripperIntern kernel: [1756592.089047]  ? tick_program_event+0x43/0xa0
2025-08-15T23:12:48.891977+02:00 ThreadripperIntern kernel: [1756592.089510]  ? srso_return_thunk+0x5/0x5f
2025-08-15T23:12:48.891978+02:00 ThreadripperIntern kernel: [1756592.089972]  ? hrtimer_interrupt+0x11f/0x250
2025-08-15T23:12:48.891979+02:00 ThreadripperIntern kernel: [1756592.090437]  ? srso_return_thunk+0x5/0x5f
2025-08-15T23:12:48.891980+02:00 ThreadripperIntern kernel: [1756592.090900]  ? irqentry_exit_to_user_mode+0x7b/0x260
2025-08-15T23:12:48.891980+02:00 ThreadripperIntern kernel: [1756592.091366]  ? srso_return_thunk+0x5/0x5f
2025-08-15T23:12:48.891980+02:00 ThreadripperIntern kernel: [1756592.091822]  ? irqentry_exit+0x43/0x50
2025-08-15T23:12:48.891981+02:00 ThreadripperIntern kernel: [1756592.092271]  ? srso_return_thunk+0x5/0x5f
2025-08-15T23:12:48.891982+02:00 ThreadripperIntern kernel: [1756592.092714]  entry_SYSCALL_64_after_hwframe+0x78/0x80
2025-08-15T23:12:48.891983+02:00 ThreadripperIntern kernel: [1756592.093160] RIP: 0033:0x78fa953d5ded
2025-08-15T23:12:48.891983+02:00 ThreadripperIntern kernel: [1756592.093629] Code: 04 25 28 00 00 00 48 89 45 c8 31 c0 48 8d 45 10 c7 45 b0 10 00 00 00 48 89 45 b8 48 8d 45 d0 48 89 45 c0 b8 10 00 00 00 0f 05 <89> c2 3d 00 f0 ff ff 77 1a 48 8b 45 c8 64 48 2b 04 25 28 00 00 00
2025-08-15T23:12:48.891984+02:00 ThreadripperIntern kernel: [1756592.094117] RSP: 002b:00007ffe4add1910 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
2025-08-15T23:12:48.891984+02:00 ThreadripperIntern kernel: [1756592.094605] RAX: ffffffffffffffda RBX: 00007ffe4add1a30 RCX: 000078fa953d5ded
2025-08-15T23:12:48.891984+02:00 ThreadripperIntern kernel: [1756592.095095] RDX: 00007ffe4add1a30 RSI: 00000000c020462a RDI: 0000000000000008
2025-08-15T23:12:48.891985+02:00 ThreadripperIntern kernel: [1756592.095589] RBP: 00007ffe4add1960 R08: 00007ffe4add1a30 R09: 00007ffe4add1a4c
2025-08-15T23:12:48.891985+02:00 ThreadripperIntern kernel: [1756592.096082] R10: 00007ffe4add1a30 R11: 0000000000000246 R12: 00000000c020462a
2025-08-15T23:12:48.891986+02:00 ThreadripperIntern kernel: [1756592.096568] R13: 0000000000000008 R14: 00007ffe4add1a4c R15: 00007ffe4add1980
2025-08-15T23:12:48.892001+02:00 ThreadripperIntern kernel: [1756592.097054]  </TASK>
2025-08-15T23:12:48.892002+02:00 ThreadripperIntern kernel: [1756592.097538] NVRM: _kgspLogXid119: ********************************************************************************
2025-08-15T23:12:48.892828+02:00 ThreadripperIntern kernel: [1756592.098037] NVRM: _issueRpcAndWait: rpcRecvPoll timedout for fn 76!
2025-08-15T23:13:04.581686+02:00 ThreadripperIntern kernel: [1756607.786029] [UFW BLOCK] IN=vmbr0 OUT= MAC=33:33:00:00:00:01:f0:9f:c2:2c:0c:9f:86:dd SRC=fe80:0000:0000:0000:xxxx:xxxx:xxxx:xxxx DST=xxxx:0000:0000:0000:0000:0000:0000:xxxx LEN=319 TC=0 HOPLIMIT=1 FLOWLBL=223890 PROTO=UDP SPT=54983 DPT=10001 LEN=279 
2025-08-15T23:13:16.631778+02:00 ThreadripperIntern kernel: [1756619.835708] [UFW BLOCK] IN=vmbr0 OUT= MAC=33:33:00:00:00:01:f0:9f:c2:cb:e5:20:86:dd SRC=fe80:0000:0000:0000:xxxx:xxxx:xxxx:xxxx DST=xxxx:0000:0000:0000:0000:0000:0000:xxxx LEN=320 TC=0 HOPLIMIT=1 FLOWLBL=2877 PROTO=UDP SPT=39978 DPT=10001 LEN=280 
2025-08-15T23:13:34.904203+02:00 ThreadripperIntern kernel: [1756638.107996] NVRM: Xid (PCI:0000:0a:00): 119, pid=305083, name=llama-server, Timeout after 45s of waiting for RPC response from GPU0 GSP! Expected function 76 (GSP_RM_CONTROL) (0x20801702 0x4).
2025-08-15T23:13:34.904216+02:00 ThreadripperIntern kernel: [1756638.108739] NVRM: _issueRpcAndWait: rpcRecvPoll timedout for fn 76!
2025-08-15T23:13:34.904216+02:00 ThreadripperIntern kernel: [1756638.109398] NVRM: subdeviceCtrlCmdMcServiceInterrupts_IMPL: NVRM_RPC: NV2080_CTRL_CMD_MC_SERVICE_INTERRUPTS failed with error 0x65
2025-08-15T23:13:39.736774+02:00 ThreadripperIntern kernel: [1756642.939886] [UFW BLOCK] IN=vmbr0 OUT= MAC=33:33:00:00:00:01:f0:9f:c2:2c:0c:9f:86:dd SRC=fe80:0000:0000:0000:xxxx:xxxx:xxxx:xxxx DST=xxxx:0000:0000:0000:0000:0000:0000:xxxx LEN=319 TC=0 HOPLIMIT=1 FLOWLBL=568490 PROTO=UDP SPT=57161 DPT=10001 LEN=279 
2025-08-15T23:14:19.907378+02:00 ThreadripperIntern kernel: [1756683.110499] NVRM: Xid (PCI:0000:0a:00): 119, pid=305083, name=llama-server, Timeout after 45s of waiting for RPC response from GPU0 GSP! Expected function 76 (GSP_RM_CONTROL) (0x20800a9a 0x8).
2025-08-15T23:14:19.907389+02:00 ThreadripperIntern kernel: [1756683.111308] NVRM: nvAssertFailedNoLog: Assertion failed: Back to back GSP RPC timeout detected! GPU marked for reset @ kernel_gsp.c:2314
2025-08-15T23:14:19.907395+02:00 ThreadripperIntern kernel: [1756683.112091] NVRM: _issueRpcAndWait: rpcRecvPoll timedout for fn 76!

This had already happened in the past with an older driver 575.57.08 which was even more unstable and had uncorrectable PCIe errors on startup (the incident happened ~7 days after the last reboot, excerpt from syslog):

...At first some startup messages:
2025-07-16T17:52:45.419076+02:00 ThreadripperIntern kernel: [    8.727853] nvidiafb: Device ID: 10de2b85
2025-07-16T17:52:45.419077+02:00 ThreadripperIntern kernel: [    8.728640] nvidiafb: unknown NV_ARCH
2025-07-16T17:52:45.419088+02:00 ThreadripperIntern kernel: [    8.832170] nvidia: module verification failed: signature and/or required key missing - tainting kernel
2025-07-16T17:52:45.419093+02:00 ThreadripperIntern kernel: [    9.021128] nvidia-nvlink: Nvlink Core is being initialized, major device number 234
2025-07-16T17:52:45.419098+02:00 ThreadripperIntern kernel: [    9.026555] nvidia 0000:0a:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=none:owns=none
2025-07-16T17:52:45.419110+02:00 ThreadripperIntern kernel: [    9.040084] NVRM: loading NVIDIA UNIX Open Kernel Module for x86_64  575.57.08  Release Build  (dvs-builder@U22-I3-H04-01-5)  Sat May 24 07:03:13 UTC 2025
2025-07-16T17:52:45.419146+02:00 ThreadripperIntern kernel: [    9.091238] nvidia-modeset: Loading NVIDIA UNIX Open Kernel Mode Setting Driver for x86_64  575.57.08  Release Build  (dvs-builder@U22-I3-H04-01-5)  Sat May 24 06:53:21 UTC 2025
2025-07-16T17:52:45.419226+02:00 ThreadripperIntern kernel: [    9.914362] [drm] [nvidia-drm] [GPU ID 0x00000a00] Loading driver
2025-07-16T17:52:45.419235+02:00 ThreadripperIntern kernel: [   10.949855] NVRM: kbifInitLtr_GB202: LTR is disabled in the hierarchy
2025-07-16T17:52:45.419236+02:00 ThreadripperIntern kernel: [   11.109314] [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:0a:00.0 on minor 1
2025-07-16T17:52:45.419236+02:00 ThreadripperIntern kernel: [   11.110617] nvidia 0000:0a:00.0: [drm] Cannot find any crtc or sizes
2025-07-16T17:52:45.419245+02:00 ThreadripperIntern kernel: [   14.338391] audit: type=1400 audit(1752681164.394:6): apparmor="STATUS" operation="profile_load" profile="unconfined" name="nvidia_modprobe" pid=1830 comm="apparmor_parser"
2025-07-16T17:52:45.419246+02:00 ThreadripperIntern kernel: [   14.338431] audit: type=1400 audit(1752681164.394:7): apparmor="STATUS" operation="profile_load" profile="unconfined" name="nvidia_modprobe//kmod" pid=1830 comm="apparmor_parser"
2025-07-16T17:52:45.430597+02:00 ThreadripperIntern systemd[1]: Started nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-07-16T17:52:54.454575+02:00 ThreadripperIntern kernel: [   24.395088] nvidia 0000:0a:00.0: PCIe Bus Error: severity=Uncorrectable (Non-Fatal), type=Data Link Layer, (Completer ID)
2025-07-16T17:52:54.454588+02:00 ThreadripperIntern kernel: [   24.395125] nvidia 0000:0a:00.0:   device [10de:2b85] error status/mask=ffffffff/00004000
2025-07-16T17:52:54.454589+02:00 ThreadripperIntern kernel: [   24.395146] nvidia 0000:0a:00.0:    [ 0] Undefined
2025-07-16T17:52:54.454589+02:00 ThreadripperIntern kernel: [   24.395161] nvidia 0000:0a:00.0:    [ 1] Unknown Error Bit
2025-07-16T17:52:54.454590+02:00 ThreadripperIntern kernel: [   24.395175] nvidia 0000:0a:00.0:    [ 2] Unknown Error Bit
2025-07-16T17:52:54.454590+02:00 ThreadripperIntern kernel: [   24.395190] nvidia 0000:0a:00.0:    [ 3] Unknown Error Bit
2025-07-16T17:52:54.454591+02:00 ThreadripperIntern kernel: [   24.395204] nvidia 0000:0a:00.0:    [ 4] DLP
2025-07-16T17:52:54.454592+02:00 ThreadripperIntern kernel: [   24.395218] nvidia 0000:0a:00.0:    [ 5] SDES
2025-07-16T17:52:54.454592+02:00 ThreadripperIntern kernel: [   24.395233] nvidia 0000:0a:00.0:    [ 6] Unknown Error Bit
2025-07-16T17:52:54.454593+02:00 ThreadripperIntern kernel: [   24.395247] nvidia 0000:0a:00.0:    [ 7] Unknown Error Bit
2025-07-16T17:52:54.454593+02:00 ThreadripperIntern kernel: [   24.395261] nvidia 0000:0a:00.0:    [ 8] Unknown Error Bit
2025-07-16T17:52:54.454594+02:00 ThreadripperIntern kernel: [   24.395275] nvidia 0000:0a:00.0:    [ 9] Unknown Error Bit
2025-07-16T17:52:54.454595+02:00 ThreadripperIntern kernel: [   24.395289] nvidia 0000:0a:00.0:    [10] Unknown Error Bit
2025-07-16T17:52:54.454595+02:00 ThreadripperIntern kernel: [   24.395303] nvidia 0000:0a:00.0:    [11] Unknown Error Bit
2025-07-16T17:52:54.454596+02:00 ThreadripperIntern kernel: [   24.395316] nvidia 0000:0a:00.0:    [12] TLP
2025-07-16T17:52:54.454596+02:00 ThreadripperIntern kernel: [   24.395330] nvidia 0000:0a:00.0:    [13] FCP
2025-07-16T17:52:54.454597+02:00 ThreadripperIntern kernel: [   24.395344] nvidia 0000:0a:00.0:    [15] CmpltAbrt
2025-07-16T17:52:54.454597+02:00 ThreadripperIntern kernel: [   24.395358] nvidia 0000:0a:00.0:    [16] UnxCmplt
2025-07-16T17:52:54.454598+02:00 ThreadripperIntern kernel: [   24.395372] nvidia 0000:0a:00.0:    [17] RxOF
2025-07-16T17:52:54.454598+02:00 ThreadripperIntern kernel: [   24.395386] nvidia 0000:0a:00.0:    [18] MalfTLP
2025-07-16T17:52:54.454599+02:00 ThreadripperIntern kernel: [   24.395399] nvidia 0000:0a:00.0:    [19] ECRC
2025-07-16T17:52:54.454599+02:00 ThreadripperIntern kernel: [   24.395413] nvidia 0000:0a:00.0:    [20] UnsupReq
2025-07-16T17:52:54.454600+02:00 ThreadripperIntern kernel: [   24.395427] nvidia 0000:0a:00.0:    [21] ACSViol
2025-07-16T17:52:54.454600+02:00 ThreadripperIntern kernel: [   24.395441] nvidia 0000:0a:00.0:    [22] UncorrIntErr
2025-07-16T17:52:54.454601+02:00 ThreadripperIntern kernel: [   24.395456] nvidia 0000:0a:00.0:    [23] BlockedTLP
2025-07-16T17:52:54.454601+02:00 ThreadripperIntern kernel: [   24.395470] nvidia 0000:0a:00.0:    [24] AtomicOpBlocked
2025-07-16T17:52:54.454602+02:00 ThreadripperIntern kernel: [   24.395486] nvidia 0000:0a:00.0:    [25] TLPBlockedErr
2025-07-16T17:52:54.454602+02:00 ThreadripperIntern kernel: [   24.395502] nvidia 0000:0a:00.0:    [26] PoisonTLPBlocked
2025-07-16T17:52:54.454604+02:00 ThreadripperIntern kernel: [   24.395518] nvidia 0000:0a:00.0:    [27] Unknown Error Bit
2025-07-16T17:52:54.454605+02:00 ThreadripperIntern kernel: [   24.395534] nvidia 0000:0a:00.0:    [28] Unknown Error Bit
2025-07-16T17:52:54.454605+02:00 ThreadripperIntern kernel: [   24.395550] nvidia 0000:0a:00.0:    [29] Unknown Error Bit
2025-07-16T17:52:54.454606+02:00 ThreadripperIntern kernel: [   24.395567] nvidia 0000:0a:00.0:    [30] Unknown Error Bit
2025-07-16T17:52:54.454606+02:00 ThreadripperIntern kernel: [   24.395583] nvidia 0000:0a:00.0:    [31] Unknown Error Bit
2025-07-16T17:52:54.454607+02:00 ThreadripperIntern kernel: [   24.395600] nvidia 0000:0a:00.0: AER:   TLP Header: ffffffff ffffffff ffffffff ffffffff
2025-07-16T17:52:54.454607+02:00 ThreadripperIntern kernel: [   24.395636] nvidia 0000:0a:00.0: AER: can't recover (no error_detected callback)
2025-07-16T17:52:54.454610+02:00 ThreadripperIntern kernel: [   24.395765] nvidia 0000:0a:00.0: PCIe Bus Error: severity=Uncorrectable (Non-Fatal), type=Transaction Layer, (Requester ID)
2025-07-16T17:52:54.454610+02:00 ThreadripperIntern kernel: [   24.395795] nvidia 0000:0a:00.0:   device [10de:2b85] error status/mask=00004000/00400000
2025-07-16T17:52:54.454629+02:00 ThreadripperIntern kernel: [   24.395818] nvidia 0000:0a:00.0:    [14] CmpltTO                (First)
2025-07-16T17:52:54.454630+02:00 ThreadripperIntern kernel: [   24.395842] nvidia 0000:0a:00.0: AER: can't recover (no error_detected callback)
2025-07-16T17:52:54.454632+02:00 ThreadripperIntern kernel: [   24.395997] nvidia 0000:0a:00.0: PCIe Bus Error: severity=Uncorrectable (Non-Fatal), type=Transaction Layer, (Requester ID)
2025-07-16T17:52:54.454633+02:00 ThreadripperIntern kernel: [   24.396024] nvidia 0000:0a:00.0:   device [10de:2b85] error status/mask=00004000/00400000
2025-07-16T17:52:54.454633+02:00 ThreadripperIntern kernel: [   24.396045] nvidia 0000:0a:00.0:    [14] CmpltTO                (First)
2025-07-16T17:52:54.454634+02:00 ThreadripperIntern kernel: [   24.396068] nvidia 0000:0a:00.0: AER: can't recover (no error_detected callback)
...here comes the incident:
2025-07-23T20:05:35.988071+02:00 ThreadripperIntern kernel: [612779.889199] [UFW BLOCK] IN=vmbr1 OUT= MAC=33:33:00:00:00:01:f0:9f:c2:xx:xx:xx:xx:xx SRC=xxxx:0000:0000:0000:xxxx:xxxx:xxxx:xxxx DST=xxxx:0000:0000:0000:0000:0000:xxxx:xxxx LEN=319 TC=0 HOPLIMIT=1 FLOWLBL=892222 PROTO=UDP SPT=57627 DPT=10001 LEN=279 
2025-07-23T20:05:37.578696+02:00 ThreadripperIntern docker[1227594]: srv  params_from_: Chat format: Content-only
2025-07-23T20:05:37.579278+02:00 ThreadripperIntern docker[1227594]: slot launch_slot_: id  0 | task 88917 | processing task
2025-07-23T20:05:37.579347+02:00 ThreadripperIntern docker[1227594]: slot update_slots: id  0 | task 88917 | new prompt, n_ctx_slot = 16384, n_keep = 0, n_prompt_tokens = 12
2025-07-23T20:05:37.579380+02:00 ThreadripperIntern docker[1227594]: slot update_slots: id  0 | task 88917 | need to evaluate at least 1 token for each active slot, n_past = 12, n_prompt_tokens = 12
2025-07-23T20:05:37.579435+02:00 ThreadripperIntern docker[1227594]: slot update_slots: id  0 | task 88917 | kv cache rm [11, end)
2025-07-23T20:05:37.579466+02:00 ThreadripperIntern docker[1227594]: slot update_slots: id  0 | task 88917 | prompt processing progress, n_past = 12, n_tokens = 1, progress = 0.083333
2025-07-23T20:05:37.579495+02:00 ThreadripperIntern docker[1227594]: slot update_slots: id  0 | task 88917 | prompt done, n_past = 12, n_tokens = 1
2025-07-23T20:05:37.633910+02:00 ThreadripperIntern docker[1227594]: slot      release: id  0 | task 88917 | stop processing: n_past = 14, truncated = 0
2025-07-23T20:05:37.634006+02:00 ThreadripperIntern docker[1227594]: slot print_timing: id  0 | task 88917 |
2025-07-23T20:05:37.634037+02:00 ThreadripperIntern docker[1227594]: prompt eval time =      24.20 ms /     1 tokens (   24.20 ms per token,    41.33 tokens per second)
2025-07-23T20:05:37.634070+02:00 ThreadripperIntern docker[1227594]:        eval time =      31.25 ms /     3 tokens (   10.42 ms per token,    95.98 tokens per second)
2025-07-23T20:05:37.634107+02:00 ThreadripperIntern docker[1227594]:       total time =      55.45 ms /     4 tokens
2025-07-23T20:05:37.634140+02:00 ThreadripperIntern docker[1227594]: srv  update_slots: all slots are idle
2025-07-23T20:05:37.634182+02:00 ThreadripperIntern docker[1227594]: srv  log_server_r: request: POST /v1/chat/completions xxx.xxx.xxx.xxx 200
2025-07-23T20:05:57.621441+02:00 ThreadripperIntern kernel: [612801.522335] [UFW BLOCK] IN=vmbr0 OUT= MAC=33:33:00:00:00:01:f0:9f:c2:xx:xx:xx:xx:xx SRC=xxxx:0000:0000:0000:xxxx:xxxx:xxxx:xxxx DST=xxxx:0000:0000:0000:0000:0000:xxxx:xxxx LEN=320 TC=0 HOPLIMIT=1 FLOWLBL=613511 PROTO=UDP SPT=39637 DPT=10001 LEN=280 
2025-07-23T20:06:07.618429+02:00 ThreadripperIntern docker[1227594]: srv  params_from_: Chat format: Content-only
2025-07-23T20:06:07.618935+02:00 ThreadripperIntern docker[1227594]: slot launch_slot_: id  0 | task 88921 | processing task
2025-07-23T20:06:07.618960+02:00 ThreadripperIntern docker[1227594]: slot update_slots: id  0 | task 88921 | new prompt, n_ctx_slot = 16384, n_keep = 0, n_prompt_tokens = 12
2025-07-23T20:06:07.619020+02:00 ThreadripperIntern docker[1227594]: slot update_slots: id  0 | task 88921 | need to evaluate at least 1 token for each active slot, n_past = 12, n_prompt_tokens = 12
2025-07-23T20:06:07.619040+02:00 ThreadripperIntern docker[1227594]: slot update_slots: id  0 | task 88921 | kv cache rm [11, end)
2025-07-23T20:06:07.619058+02:00 ThreadripperIntern docker[1227594]: slot update_slots: id  0 | task 88921 | prompt processing progress, n_past = 12, n_tokens = 1, progress = 0.083333
2025-07-23T20:06:07.619078+02:00 ThreadripperIntern docker[1227594]: slot update_slots: id  0 | task 88921 | prompt done, n_past = 12, n_tokens = 1
2025-07-23T20:06:13.618886+02:00 ThreadripperIntern docker[1227594]: srv  cancel_tasks: cancel task, id_task = 88921
2025-07-23T20:06:13.619041+02:00 ThreadripperIntern docker[1227594]: srv  log_server_r: request: POST /v1/chat/completions xxx.xxx.xxx.xxx 200
2025-07-23T20:06:16.167298+02:00 ThreadripperIntern kernel: [612820.067893] [UFW BLOCK] IN=vmbr0 OUT= MAC=33:33:00:00:00:01:f0:9f:c2:xx:xx:xx:xx:xx SRC=xxxx:0000:0000:0000:xxxx:xxxx:xxxx:xxxx DST=xxxx:0000:0000:0000:0000:0000:xxxx:xxxx LEN=319 TC=0 HOPLIMIT=1 FLOWLBL=988521 PROTO=UDP SPT=39581 DPT=10001 LEN=279 
2025-07-23T20:06:37.652800+02:00 ThreadripperIntern docker[1227594]: srv  params_from_: Chat format: Content-only
2025-07-23T20:06:42.774287+02:00 ThreadripperIntern kernel: [612846.673998] [UFW BLOCK] IN=vmbr0 OUT= MAC=33:33:00:00:00:01:f0:9f:c2:xx:xx:xx:xx:xx SRC=xxxx:0000:0000:0000:xxxx:xxxx:xxxx:xxxx DST=xxxx:0000:0000:0000:0000:0000:xxxx:xxxx LEN=320 TC=0 HOPLIMIT=1 FLOWLBL=551691 PROTO=UDP SPT=40919 DPT=10001 LEN=280 
2025-07-23T20:06:43.653221+02:00 ThreadripperIntern docker[1227594]: srv  cancel_tasks: cancel task, id_task = 88924
2025-07-23T20:06:43.653424+02:00 ThreadripperIntern docker[1227594]: srv  log_server_r: request: POST /v1/chat/completions xxx.xxx.xxx.xxx 200
2025-07-23T20:06:52.621437+02:00 ThreadripperIntern kernel: [612856.523956] NVRM: _kgspLogXid119: ********************************* GSP Timeout **********************************
2025-07-23T20:06:52.621455+02:00 ThreadripperIntern kernel: [612856.526094] NVRM: _kgspLogXid119: Note: Please also check logs above.
2025-07-23T20:06:52.625742+02:00 ThreadripperIntern kernel: [612856.528273] NVRM: GPU at PCI:0000:0a:00: GPU-701b524f-c979-7284-9737-0fecd8ca5f9b
2025-07-23T20:06:52.625751+02:00 ThreadripperIntern kernel: [612856.530413] NVRM: GPU Board Serial Number: 1791625003310
2025-07-23T20:06:52.628827+02:00 ThreadripperIntern kernel: [612856.531941] NVRM: Xid (PCI:0000:0a:00): 119, pid=1227848, name=llama-server, Timeout after 45s of waiting for RPC response from GPU0 GSP! Expected function 76 (GSP_RM_CONTROL) (0x20800a9a 0x8).
2025-07-23T20:06:52.630401+02:00 ThreadripperIntern kernel: [612856.533524] NVRM: GPU0 GSP RPC buffer contains function 76 (GSP_RM_CONTROL) and data 0x0000000020800a9a 0x0000000000000008.
2025-07-23T20:06:52.630405+02:00 ThreadripperIntern kernel: [612856.535071] NVRM: GPU0 RPC history (CPU -> GSP):
2025-07-23T20:06:52.633413+02:00 ThreadripperIntern kernel: [612856.536589] NVRM:     entry function                   data0              data1              ts_start           ts_end             duration actively_polling
2025-07-23T20:06:52.633420+02:00 ThreadripperIntern kernel: [612856.538086] NVRM:      0    76   GSP_RM_CONTROL        0x0000000020800a9a 0x0000000000000008 0x00063a9c902072c7 0x0000000000000000          y
2025-07-23T20:06:52.636405+02:00 ThreadripperIntern kernel: [612856.539595] NVRM:     -1    76   GSP_RM_CONTROL        0x0000000020800a9a 0x0000000000000008 0x00063a9c8f42c04c 0x00063a9c8f42c166    282us  
2025-07-23T20:06:52.636411+02:00 ThreadripperIntern kernel: [612856.541078] NVRM:     -2    76   GSP_RM_CONTROL        0x0000000020800a9a 0x0000000000000008 0x00063a9c8e5615de 0x00063a9c8e5634db   7933us  
2025-07-23T20:06:52.639324+02:00 ThreadripperIntern kernel: [612856.542562] NVRM:     -3    76   GSP_RM_CONTROL        0x0000000020800a9a 0x0000000000000008 0x00063a9c8d78404a 0x00063a9c8d78416c    290us  
2025-07-23T20:06:52.639331+02:00 ThreadripperIntern kernel: [612856.543998] NVRM:     -4    76   GSP_RM_CONTROL        0x0000000020800a9a 0x0000000000000008 0x00063a9c8c8b96de 0x00063a9c8c8bb538   7770us  
2025-07-23T20:06:52.642166+02:00 ThreadripperIntern kernel: [612856.545441] NVRM:     -5    76   GSP_RM_CONTROL        0x0000000020800a9a 0x0000000000000008 0x00063a9c8bade16c 0x00063a9c8bade286    282us  
2025-07-23T20:06:52.642171+02:00 ThreadripperIntern kernel: [612856.546840] NVRM:     -6    76   GSP_RM_CONTROL        0x0000000020800a9a 0x0000000000000008 0x00063a9c8ac1361f 0x00063a9c8ac15645   8230us  
2025-07-23T20:06:52.644903+02:00 ThreadripperIntern kernel: [612856.548221] NVRM:     -7    76   GSP_RM_CONTROL        0x0000000020800a9a 0x0000000000000008 0x00063a9c89e3859b 0x00063a9c89e386bd    290us  
2025-07-23T20:06:52.644908+02:00 ThreadripperIntern kernel: [612856.549574] NVRM: GPU0 RPC event history (CPU <- GSP):
2025-07-23T20:06:52.647634+02:00 ThreadripperIntern kernel: [612856.550962] NVRM:     entry function                   data0              data1              ts_start           ts_end             duration during_incomplete_rpc
2025-07-23T20:06:52.647642+02:00 ThreadripperIntern kernel: [612856.552307] NVRM:      0    4128 GSP_POST_NOCAT_RECORD 0x0000000000000005 0x000000000155d57a 0x00063a0de2cec85a 0x00063a0de2cec85a           
2025-07-23T20:06:52.650282+02:00 ThreadripperIntern kernel: [612856.553645] NVRM:     -1    4128 GSP_POST_NOCAT_RECORD 0x0000000000000005 0x000000000155d568 0x00063a0de2cec858 0x00063a0de2cec859      1us  
2025-07-23T20:06:52.650289+02:00 ThreadripperIntern kernel: [612856.554955] NVRM:     -2    4128 GSP_POST_NOCAT_RECORD 0x0000000000000005 0x000000000155d57a 0x00063a0de2cec856 0x00063a0de2cec857      1us  
2025-07-23T20:06:52.652861+02:00 ThreadripperIntern kernel: [612856.556259] NVRM:     -3    4128 GSP_POST_NOCAT_RECORD 0x0000000000000005 0x000000000155d568 0x00063a0de2cec850 0x00063a0de2cec855      5us  
2025-07-23T20:06:52.652868+02:00 ThreadripperIntern kernel: [612856.557534] NVRM:     -4    4124 GSP_LOCKDOWN_NOTICE   0x0000000000000000 0x0000000000000000 0x00063a0de200ad51 0x00063a0de200ad51           
2025-07-23T20:06:52.655376+02:00 ThreadripperIntern kernel: [612856.558808] NVRM:     -5    4124 GSP_LOCKDOWN_NOTICE   0x0000000000000001 0x0000000000000000 0x00063a0de200993d 0x00063a0de200993d           
2025-07-23T20:06:52.655383+02:00 ThreadripperIntern kernel: [612856.560049] NVRM:     -6    4124 GSP_LOCKDOWN_NOTICE   0x0000000000000000 0x0000000000000000 0x00063a0de200984b 0x00063a0de200984c      1us  
2025-07-23T20:06:52.657808+02:00 ThreadripperIntern kernel: [612856.561277] NVRM:     -7    4124 GSP_LOCKDOWN_NOTICE   0x0000000000000001 0x0000000000000000 0x00063a0de2009492 0x00063a0de2009493      1us  
2025-07-23T20:06:52.657814+02:00 ThreadripperIntern kernel: [612856.562481] CPU: 2 PID: 1227848 Comm: llama-server Tainted: P           OE      6.8.12-11-pve #1
2025-07-23T20:06:52.712534+02:00 ThreadripperIntern kernel: [612856.563673] Hardware name: Micro-Star International Co., Ltd. MS-7B09/X399 SLI PLUS (MS-7B09), BIOS A.89 08/11/2022
2025-07-23T20:06:52.712544+02:00 ThreadripperIntern kernel: [612856.564845] Call Trace:
2025-07-23T20:06:52.712545+02:00 ThreadripperIntern kernel: [612856.565994]  <TASK>
2025-07-23T20:06:52.712546+02:00 ThreadripperIntern kernel: [612856.567127]  dump_stack_lvl+0x76/0xa0
2025-07-23T20:06:52.712546+02:00 ThreadripperIntern kernel: [612856.568244]  dump_stack+0x10/0x20
2025-07-23T20:06:52.712547+02:00 ThreadripperIntern kernel: [612856.569339]  os_dump_stack+0xe/0x20 [nvidia]
2025-07-23T20:06:52.712548+02:00 ThreadripperIntern kernel: [612856.570583]  _kgspRpcRecvPoll+0x593/0x760 [nvidia]
2025-07-23T20:06:52.712548+02:00 ThreadripperIntern kernel: [612856.571875]  _issueRpcAndWait+0xd2/0x900 [nvidia]
2025-07-23T20:06:52.712549+02:00 ThreadripperIntern kernel: [612856.573107]  ? srso_return_thunk+0x5/0x5f
2025-07-23T20:06:52.712549+02:00 ThreadripperIntern kernel: [612856.574137]  ? osGetCurrentThread+0x26/0x60 [nvidia]
2025-07-23T20:06:52.712550+02:00 ThreadripperIntern kernel: [612856.575380]  ? srso_return_thunk+0x5/0x5f
2025-07-23T20:06:52.712563+02:00 ThreadripperIntern kernel: [612856.576371]  ? srso_return_thunk+0x5/0x5f
2025-07-23T20:06:52.712564+02:00 ThreadripperIntern kernel: [612856.577351]  ? os_mem_set+0x14/0x20 [nvidia]
2025-07-23T20:06:52.712565+02:00 ThreadripperIntern kernel: [612856.578454]  ? srso_return_thunk+0x5/0x5f
2025-07-23T20:06:52.712565+02:00 ThreadripperIntern kernel: [612856.579396]  rpcRmApiControl_GSP+0x76f/0x940 [nvidia]
2025-07-23T20:06:52.712566+02:00 ThreadripperIntern kernel: [612856.580516]  kperfBoostSet_IMPL+0x69/0x70 [nvidia]
2025-07-23T20:06:52.712566+02:00 ThreadripperIntern kernel: [612856.581630]  resControl_IMPL+0x1a9/0x1b0 [nvidia]
2025-07-23T20:06:52.712567+02:00 ThreadripperIntern kernel: [612856.582701]  serverControl+0x47e/0x590 [nvidia]
2025-07-23T20:06:52.712570+02:00 ThreadripperIntern kernel: [612856.583757]  _rmapiRmControl+0x4f2/0x820 [nvidia]
2025-07-23T20:06:52.712570+02:00 ThreadripperIntern kernel: [612856.584819]  ? srso_return_thunk+0x5/0x5f
2025-07-23T20:06:52.712571+02:00 ThreadripperIntern kernel: [612856.585660]  rmapiControlWithSecInfo+0x79/0x140 [nvidia]
2025-07-23T20:06:52.712571+02:00 ThreadripperIntern kernel: [612856.586685]  ? os_acquire_spinlock+0x12/0x30 [nvidia]
2025-07-23T20:06:52.712571+02:00 ThreadripperIntern kernel: [612856.587642]  ? srso_return_thunk+0x5/0x5f
2025-07-23T20:06:52.712572+02:00 ThreadripperIntern kernel: [612856.588438]  rmapiControlWithSecInfoTls+0x8f/0xf0 [nvidia]
2025-07-23T20:06:52.712572+02:00 ThreadripperIntern kernel: [612856.589410]  _nv04ControlWithSecInfo+0x8d/0xa0 [nvidia]
2025-07-23T20:06:52.712573+02:00 ThreadripperIntern kernel: [612856.590365]  ? srso_return_thunk+0x5/0x5f
2025-07-23T20:06:52.712573+02:00 ThreadripperIntern kernel: [612856.591104]  ? srso_return_thunk+0x5/0x5f
2025-07-23T20:06:52.712574+02:00 ThreadripperIntern kernel: [612856.591825]  ? security_capable+0x47/0x80
2025-07-23T20:06:52.712574+02:00 ThreadripperIntern kernel: [612856.592535]  RmIoctl+0x90b/0xda0 [nvidia]
2025-07-23T20:06:52.712575+02:00 ThreadripperIntern kernel: [612856.593456]  ? ttwu_queue_wakelist+0x101/0x110
2025-07-23T20:06:52.712575+02:00 ThreadripperIntern kernel: [612856.594121]  ? os_get_current_tick+0x3c/0xb0 [nvidia]
2025-07-23T20:06:52.712576+02:00 ThreadripperIntern kernel: [612856.594920]  ? srso_return_thunk+0x5/0x5f
2025-07-23T20:06:52.712576+02:00 ThreadripperIntern kernel: [612856.595560]  ? os_acquire_spinlock+0x12/0x30 [nvidia]
2025-07-23T20:06:52.712577+02:00 ThreadripperIntern kernel: [612856.596326]  ? srso_return_thunk+0x5/0x5f
2025-07-23T20:06:52.712577+02:00 ThreadripperIntern kernel: [612856.596925]  ? portSyncSpinlockAcquire+0x18/0x30 [nvidia]
2025-07-23T20:06:52.712578+02:00 ThreadripperIntern kernel: [612856.597692]  ? srso_return_thunk+0x5/0x5f
2025-07-23T20:06:52.712578+02:00 ThreadripperIntern kernel: [612856.598260]  rm_ioctl+0x66/0x4f0 [nvidia]
2025-07-23T20:06:52.712579+02:00 ThreadripperIntern kernel: [612856.599048]  nvidia_unlocked_ioctl+0x69c/0x920 [nvidia]
2025-07-23T20:06:52.712580+02:00 ThreadripperIntern kernel: [612856.599730]  __x64_sys_ioctl+0xa3/0xf0
2025-07-23T20:06:52.712580+02:00 ThreadripperIntern kernel: [612856.600248]  x64_sys_call+0xa71/0x2480
2025-07-23T20:06:52.712581+02:00 ThreadripperIntern kernel: [612856.600762]  do_syscall_64+0x81/0x170
2025-07-23T20:06:52.712581+02:00 ThreadripperIntern kernel: [612856.601269]  ? srso_return_thunk+0x5/0x5f
2025-07-23T20:06:52.712582+02:00 ThreadripperIntern kernel: [612856.601774]  ? srso_return_thunk+0x5/0x5f
2025-07-23T20:06:52.712582+02:00 ThreadripperIntern kernel: [612856.602272]  ? srso_return_thunk+0x5/0x5f
2025-07-23T20:06:52.712583+02:00 ThreadripperIntern kernel: [612856.602755]  ? ktime_get+0x48/0xc0
2025-07-23T20:06:52.712583+02:00 ThreadripperIntern kernel: [612856.603223]  ? srso_return_thunk+0x5/0x5f
2025-07-23T20:06:52.712584+02:00 ThreadripperIntern kernel: [612856.603688]  ? advance_periodic_target_expiration+0x4a/0xc0 [kvm]
2025-07-23T20:06:52.712584+02:00 ThreadripperIntern kernel: [612856.604222]  ? srso_return_thunk+0x5/0x5f
2025-07-23T20:06:52.712585+02:00 ThreadripperIntern kernel: [612856.604678]  ? timerqueue_add+0x69/0xd0
2025-07-23T20:06:52.712585+02:00 ThreadripperIntern kernel: [612856.605128]  ? srso_return_thunk+0x5/0x5f
2025-07-23T20:06:52.712586+02:00 ThreadripperIntern kernel: [612856.605579]  ? srso_return_thunk+0x5/0x5f
2025-07-23T20:06:52.712586+02:00 ThreadripperIntern kernel: [612856.606027]  ? ktime_get+0x48/0xc0
2025-07-23T20:06:52.712587+02:00 ThreadripperIntern kernel: [612856.606477]  ? apic_timer_fn+0x40/0x60 [kvm]
2025-07-23T20:06:52.712587+02:00 ThreadripperIntern kernel: [612856.606993]  ? srso_return_thunk+0x5/0x5f
2025-07-23T20:06:52.712587+02:00 ThreadripperIntern kernel: [612856.607443]  ? lapic_next_event+0x15/0x30
2025-07-23T20:06:52.712588+02:00 ThreadripperIntern kernel: [612856.607887]  ? srso_return_thunk+0x5/0x5f
2025-07-23T20:06:52.712588+02:00 ThreadripperIntern kernel: [612856.608328]  ? clockevents_program_event+0xb6/0x140
2025-07-23T20:06:52.712589+02:00 ThreadripperIntern kernel: [612856.608762]  ? srso_return_thunk+0x5/0x5f
2025-07-23T20:06:52.712589+02:00 ThreadripperIntern kernel: [612856.609189]  ? tick_program_event+0x43/0xa0
2025-07-23T20:06:52.712589+02:00 ThreadripperIntern kernel: [612856.609617]  ? srso_return_thunk+0x5/0x5f
2025-07-23T20:06:52.712590+02:00 ThreadripperIntern kernel: [612856.610042]  ? hrtimer_interrupt+0x11f/0x250
2025-07-23T20:06:52.712590+02:00 ThreadripperIntern kernel: [612856.610484]  ? srso_return_thunk+0x5/0x5f
2025-07-23T20:06:52.712591+02:00 ThreadripperIntern kernel: [612856.610909]  ? irqentry_exit_to_user_mode+0x7b/0x260
2025-07-23T20:06:52.712591+02:00 ThreadripperIntern kernel: [612856.611344]  ? srso_return_thunk+0x5/0x5f
2025-07-23T20:06:52.712591+02:00 ThreadripperIntern kernel: [612856.611767]  ? irqentry_exit+0x43/0x50
2025-07-23T20:06:52.712592+02:00 ThreadripperIntern kernel: [612856.612195]  ? srso_return_thunk+0x5/0x5f
2025-07-23T20:06:52.712592+02:00 ThreadripperIntern kernel: [612856.612618]  entry_SYSCALL_64_after_hwframe+0x78/0x80
2025-07-23T20:06:52.712593+02:00 ThreadripperIntern kernel: [612856.613033] RIP: 0033:0x776d1af92ded
2025-07-23T20:06:52.712593+02:00 ThreadripperIntern kernel: [612856.613463] Code: 04 25 28 00 00 00 48 89 45 c8 31 c0 48 8d 45 10 c7 45 b0 10 00 00 00 48 89 45 b8 48 8d 45 d0 48 89 45 c0 b8 10 00 00 00 0f 05 <89> c2 3d 00 f0 ff ff 77 1a 48 8b 45 c8 64 48 2b 04 25 28 00 00 00
2025-07-23T20:06:52.712594+02:00 ThreadripperIntern kernel: [612856.613914] RSP: 002b:00007fff2a00e3e0 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
2025-07-23T20:06:52.712595+02:00 ThreadripperIntern kernel: [612856.614374] RAX: ffffffffffffffda RBX: 00007fff2a00e500 RCX: 0000776d1af92ded
2025-07-23T20:06:52.712595+02:00 ThreadripperIntern kernel: [612856.614833] RDX: 00007fff2a00e500 RSI: 00000000c020462a RDI: 0000000000000008
2025-07-23T20:06:52.712596+02:00 ThreadripperIntern kernel: [612856.615287] RBP: 00007fff2a00e430 R08: 00007fff2a00e500 R09: 00007fff2a00e51c
2025-07-23T20:06:52.712596+02:00 ThreadripperIntern kernel: [612856.615745] R10: 00007fff2a00e500 R11: 0000000000000246 R12: 00000000c020462a
2025-07-23T20:06:52.712596+02:00 ThreadripperIntern kernel: [612856.616205] R13: 0000000000000008 R14: 00007fff2a00e51c R15: 00007fff2a00e450
2025-07-23T20:06:52.712597+02:00 ThreadripperIntern kernel: [612856.616671]  </TASK>
2025-07-23T20:06:52.712598+02:00 ThreadripperIntern kernel: [612856.617147] NVRM: _kgspLogXid119: ********************************************************************************
2025-07-23T20:06:52.713508+02:00 ThreadripperIntern kernel: [612856.617667] NVRM: _issueRpcAndWait: rpcRecvPoll timedout for fn 76!

I am running llama.cpp every ~30 seconds with a very basic request (“Hauptstadt Frankreichs? Antworte kurz!”) for health checks.

edit: I’ve sent the full nvidia-bug-report.log.gz to linux-bugs@nvidia.com already

3

Same error again. This time the system ran stable with driver 580.76.05 from 2025-08-20 until 2025-09-03 (~13.5 days):

2025-09-03T08:11:54.836308+02:00 ThreadripperIntern kernel: [1176840.277457] NVRM: _kgspLogXid119: ********************************* GSP Timeout **********************************
2025-09-03T08:11:54.836333+02:00 ThreadripperIntern kernel: [1176840.279874] NVRM: _kgspLogXid119: Note: Please also check logs above.
2025-09-03T08:11:54.841205+02:00 ThreadripperIntern kernel: [1176840.282325] NVRM: GPU at PCI:0000:0a:00: GPU-701b524f-c979-7284-9737-0fecd8ca5f9b
2025-09-03T08:11:54.841218+02:00 ThreadripperIntern kernel: [1176840.284773] NVRM: GPU Board Serial Number: 1791625003310
2025-09-03T08:11:54.846048+02:00 ThreadripperIntern kernel: [1176840.287217] NVRM: Xid (PCI:0000:0a:00): 119, pid=9920, name=llama-server, Timeout after 45s of waiting for RPC response from GPU0 GSP! Expected function 76 (GSP_RM_CONTROL) (0x20800a9a 0x8).
2025-09-03T08:11:54.847723+02:00 ThreadripperIntern kernel: [1176840.289639] NVRM: GPU0 GSP RPC buffer contains function 76 (GSP_RM_CONTROL) and data 0x0000000020800a9a 0x0000000000000008.
2025-09-03T08:11:54.847728+02:00 ThreadripperIntern kernel: [1176840.291290] NVRM: GPU0 RPC history (CPU -> GSP):
2025-09-03T08:11:54.849337+02:00 ThreadripperIntern kernel: [1176840.292903] NVRM:     entry function                   data0              data1              ts_start           ts_end             duration actively_polling
2025-09-03T08:11:54.852616+02:00 ThreadripperIntern kernel: [1176840.294556] NVRM:      0    76   GSP_RM_CONTROL        0x0000000020800a9a 0x0000000000000008 0x00063ddf788df212 0x0000000000000000          y
2025-09-03T08:11:54.854232+02:00 ThreadripperIntern kernel: [1176840.296202] NVRM:     -1    76   GSP_RM_CONTROL        0x0000000020800a9a 0x0000000000000008 0x00063ddf77b038cb 0x00063ddf77b039f4    297us  
2025-09-03T08:11:54.855826+02:00 ThreadripperIntern kernel: [1176840.297807] NVRM:     -2    76   GSP_RM_CONTROL        0x0000000020800a9a 0x0000000000000008 0x00063ddf76c38d04 0x00063ddf76c3ad02   8190us  
2025-09-03T08:11:54.855834+02:00 ThreadripperIntern kernel: [1176840.299395] NVRM:     -3    76   GSP_RM_CONTROL        0x0000000020800a9a 0x0000000000000008 0x00063ddf75e5d318 0x00063ddf75e5d448    304us  
2025-09-03T08:11:54.858951+02:00 ThreadripperIntern kernel: [1176840.300962] NVRM:     -4    76   GSP_RM_CONTROL        0x0000000020800a9a 0x0000000000000008 0x00063ddf74f921d0 0x00063ddf74f945b6   9190us  
2025-09-03T08:11:54.858960+02:00 ThreadripperIntern kernel: [1176840.302517] NVRM:     -5    76   GSP_RM_CONTROL        0x0000000020800a9a 0x0000000000000008 0x00063ddf741b6f1d 0x00063ddf741b704b    302us  
2025-09-03T08:11:54.862001+02:00 ThreadripperIntern kernel: [1176840.304045] NVRM:     -6    76   GSP_RM_CONTROL        0x0000000020800a9a 0x0000000000000008 0x00063ddf732ec40a 0x00063ddf732ee38c   8066us  
2025-09-03T08:11:54.862007+02:00 ThreadripperIntern kernel: [1176840.305566] NVRM:     -7    76   GSP_RM_CONTROL        0x0000000020800a9a 0x0000000000000008 0x00063ddf72510f24 0x00063ddf72511055    305us  
2025-09-03T08:11:54.863479+02:00 ThreadripperIntern kernel: [1176840.307045] NVRM: GPU0 RPC event history (CPU <- GSP):
2025-09-03T08:11:54.864959+02:00 ThreadripperIntern kernel: [1176840.308525] NVRM:     entry function                   data0              data1              ts_start           ts_end             duration during_incomplete_rpc
2025-09-03T08:11:54.867965+02:00 ThreadripperIntern kernel: [1176840.310031] NVRM:      0    4124 GSP_LOCKDOWN_NOTICE   0x0000000000000000 0x0000000000000000 0x00063ccd7a9f7133 0x00063ccd7a9f7133           
2025-09-03T08:11:54.867970+02:00 ThreadripperIntern kernel: [1176840.311530] NVRM:     -1    4124 GSP_LOCKDOWN_NOTICE   0x0000000000000001 0x0000000000000000 0x00063ccd7a9f5cea 0x00063ccd7a9f5cea           
2025-09-03T08:11:54.870905+02:00 ThreadripperIntern kernel: [1176840.313000] NVRM:     -2    4124 GSP_LOCKDOWN_NOTICE   0x0000000000000000 0x0000000000000000 0x00063ccd7a9f5bcf 0x00063ccd7a9f5bcf           
2025-09-03T08:11:54.872363+02:00 ThreadripperIntern kernel: [1176840.314498] NVRM:     -3    4124 GSP_LOCKDOWN_NOTICE   0x0000000000000001 0x0000000000000000 0x00063ccd7a9f5816 0x00063ccd7a9f5816           
2025-09-03T08:11:54.872368+02:00 ThreadripperIntern kernel: [1176840.315929] NVRM:     -4    4124 GSP_LOCKDOWN_NOTICE   0x0000000000000000 0x0000000000000000 0x00063ccd7a9de9b6 0x00063ccd7a9de9b6           
2025-09-03T08:11:54.875182+02:00 ThreadripperIntern kernel: [1176840.317358] NVRM:     -5    4124 GSP_LOCKDOWN_NOTICE   0x0000000000000001 0x0000000000000000 0x00063ccd7a9de925 0x00063ccd7a9de925           
2025-09-03T08:11:54.875187+02:00 ThreadripperIntern kernel: [1176840.318746] NVRM:     -6    4124 GSP_LOCKDOWN_NOTICE   0x0000000000000000 0x0000000000000000 0x00063ccd7a9dd61d 0x00063ccd7a9dd61e      1us  
2025-09-03T08:11:54.877932+02:00 ThreadripperIntern kernel: [1176840.320136] NVRM:     -7    4124 GSP_LOCKDOWN_NOTICE   0x0000000000000001 0x0000000000000000 0x00063ccd7a9db2ab 0x00063ccd7a9db2ac      1us  
2025-09-03T08:11:54.877937+02:00 ThreadripperIntern kernel: [1176840.321497] CPU: 18 PID: 9920 Comm: llama-server Tainted: P           OE      6.8.12-12-pve #1
2025-09-03T08:11:54.941693+02:00 ThreadripperIntern kernel: [1176840.322834] Hardware name: Micro-Star International Co., Ltd. MS-7B09/X399 SLI PLUS (MS-7B09), BIOS A.89 08/11/2022
2025-09-03T08:11:54.941705+02:00 ThreadripperIntern kernel: [1176840.324178] Call Trace:
2025-09-03T08:11:54.941707+02:00 ThreadripperIntern kernel: [1176840.325500]  <TASK>
2025-09-03T08:11:54.941707+02:00 ThreadripperIntern kernel: [1176840.326787]  dump_stack_lvl+0x76/0xa0
2025-09-03T08:11:54.941708+02:00 ThreadripperIntern kernel: [1176840.328068]  dump_stack+0x10/0x20
2025-09-03T08:11:54.941708+02:00 ThreadripperIntern kernel: [1176840.329324]  os_dump_stack+0xe/0x20 [nvidia]
2025-09-03T08:11:54.941709+02:00 ThreadripperIntern kernel: [1176840.330737]  _kgspRpcRecvPoll+0x62b/0x800 [nvidia]
2025-09-03T08:11:54.941709+02:00 ThreadripperIntern kernel: [1176840.332193]  _issueRpcAndWait+0xd2/0x900 [nvidia]
2025-09-03T08:11:54.941709+02:00 ThreadripperIntern kernel: [1176840.333578]  ? srso_return_thunk+0x5/0x5f
2025-09-03T08:11:54.941710+02:00 ThreadripperIntern kernel: [1176840.334764]  ? osGetCurrentThread+0x26/0x60 [nvidia]
2025-09-03T08:11:54.941710+02:00 ThreadripperIntern kernel: [1176840.336186]  ? srso_return_thunk+0x5/0x5f
2025-09-03T08:11:54.941710+02:00 ThreadripperIntern kernel: [1176840.337339]  ? srso_return_thunk+0x5/0x5f
2025-09-03T08:11:54.941711+02:00 ThreadripperIntern kernel: [1176840.338472]  ? os_mem_set+0x14/0x20 [nvidia]
2025-09-03T08:11:54.941711+02:00 ThreadripperIntern kernel: [1176840.339735]  ? srso_return_thunk+0x5/0x5f
2025-09-03T08:11:54.941711+02:00 ThreadripperIntern kernel: [1176840.340928]  rpcRmApiControl_GSP+0x76f/0x940 [nvidia]
2025-09-03T08:11:54.941712+02:00 ThreadripperIntern kernel: [1176840.342226]  kperfBoostSet_IMPL+0x69/0x70 [nvidia]
2025-09-03T08:11:54.941712+02:00 ThreadripperIntern kernel: [1176840.343504]  resControl_IMPL+0x1a9/0x1b0 [nvidia]
2025-09-03T08:11:54.941713+02:00 ThreadripperIntern kernel: [1176840.344734]  serverControl+0x47e/0x590 [nvidia]
2025-09-03T08:11:54.941713+02:00 ThreadripperIntern kernel: [1176840.345950]  _rmapiRmControl+0x4f2/0x820 [nvidia]
2025-09-03T08:11:54.941713+02:00 ThreadripperIntern kernel: [1176840.347203]  ? sched_clock_noinstr+0x9/0x10
2025-09-03T08:11:54.941714+02:00 ThreadripperIntern kernel: [1176840.348211]  rmapiControlWithSecInfo+0x79/0x140 [nvidia]
2025-09-03T08:11:54.941714+02:00 ThreadripperIntern kernel: [1176840.349403]  ? os_acquire_spinlock+0x12/0x30 [nvidia]
2025-09-03T08:11:54.941714+02:00 ThreadripperIntern kernel: [1176840.350538]  ? srso_return_thunk+0x5/0x5f
2025-09-03T08:11:54.941715+02:00 ThreadripperIntern kernel: [1176840.351502]  rmapiControlWithSecInfoTls+0x8f/0xf0 [nvidia]
2025-09-03T08:11:54.941715+02:00 ThreadripperIntern kernel: [1176840.352656]  _nv04ControlWithSecInfo+0x8d/0xa0 [nvidia]
2025-09-03T08:11:54.941715+02:00 ThreadripperIntern kernel: [1176840.353789]  ? srso_return_thunk+0x5/0x5f
2025-09-03T08:11:54.941716+02:00 ThreadripperIntern kernel: [1176840.354704]  ? security_capable+0x47/0x80
2025-09-03T08:11:54.941716+02:00 ThreadripperIntern kernel: [1176840.355601]  RmIoctl+0x90b/0xda0 [nvidia]
2025-09-03T08:11:54.941716+02:00 ThreadripperIntern kernel: [1176840.356718]  ? os_get_monotonic_time_ns+0x3c/0xb0 [nvidia]
2025-09-03T08:11:54.941717+02:00 ThreadripperIntern kernel: [1176840.357734]  ? srso_return_thunk+0x5/0x5f
2025-09-03T08:11:54.941735+02:00 ThreadripperIntern kernel: [1176840.358587]  ? os_acquire_spinlock+0x12/0x30 [nvidia]
2025-09-03T08:11:54.941735+02:00 ThreadripperIntern kernel: [1176840.359572]  ? srso_return_thunk+0x5/0x5f
2025-09-03T08:11:54.941736+02:00 ThreadripperIntern kernel: [1176840.360384]  ? portSyncSpinlockAcquire+0x18/0x30 [nvidia]
2025-09-03T08:11:54.941736+02:00 ThreadripperIntern kernel: [1176840.361366]  ? srso_return_thunk+0x5/0x5f
2025-09-03T08:11:54.941737+02:00 ThreadripperIntern kernel: [1176840.362151]  rm_ioctl+0x66/0x4f0 [nvidia]
2025-09-03T08:11:54.941737+02:00 ThreadripperIntern kernel: [1176840.363155]  nvidia_unlocked_ioctl+0x69c/0x920 [nvidia]
2025-09-03T08:11:54.941737+02:00 ThreadripperIntern kernel: [1176840.364047]  __x64_sys_ioctl+0xa3/0xf0
2025-09-03T08:11:54.941738+02:00 ThreadripperIntern kernel: [1176840.364777]  x64_sys_call+0xa71/0x2480
2025-09-03T08:11:54.941738+02:00 ThreadripperIntern kernel: [1176840.365500]  do_syscall_64+0x81/0x170
2025-09-03T08:11:54.941739+02:00 ThreadripperIntern kernel: [1176840.366199]  ? srso_return_thunk+0x5/0x5f
2025-09-03T08:11:54.941739+02:00 ThreadripperIntern kernel: [1176840.366867]  ? srso_return_thunk+0x5/0x5f
2025-09-03T08:11:54.941740+02:00 ThreadripperIntern kernel: [1176840.367537]  ? srso_return_thunk+0x5/0x5f
2025-09-03T08:11:54.941740+02:00 ThreadripperIntern kernel: [1176840.368178]  ? ktime_get+0x48/0xc0
2025-09-03T08:11:54.941740+02:00 ThreadripperIntern kernel: [1176840.368789]  ? srso_return_thunk+0x5/0x5f
2025-09-03T08:11:54.941741+02:00 ThreadripperIntern kernel: [1176840.369394]  ? advance_periodic_target_expiration+0x4a/0xc0 [kvm]
2025-09-03T08:11:54.941741+02:00 ThreadripperIntern kernel: [1176840.370053]  ? srso_return_thunk+0x5/0x5f
2025-09-03T08:11:54.941742+02:00 ThreadripperIntern kernel: [1176840.370632]  ? timerqueue_add+0x69/0xd0
2025-09-03T08:11:54.941742+02:00 ThreadripperIntern kernel: [1176840.371202]  ? srso_return_thunk+0x5/0x5f
2025-09-03T08:11:54.941743+02:00 ThreadripperIntern kernel: [1176840.371760]  ? srso_return_thunk+0x5/0x5f
2025-09-03T08:11:54.941743+02:00 ThreadripperIntern kernel: [1176840.372310]  ? ktime_get+0x48/0xc0
2025-09-03T08:11:54.941743+02:00 ThreadripperIntern kernel: [1176840.372845]  ? apic_timer_fn+0x40/0x60 [kvm]
2025-09-03T08:11:54.941744+02:00 ThreadripperIntern kernel: [1176840.373462]  ? srso_return_thunk+0x5/0x5f
2025-09-03T08:11:54.941744+02:00 ThreadripperIntern kernel: [1176840.373981]  ? lapic_next_event+0x15/0x30
2025-09-03T08:11:54.941745+02:00 ThreadripperIntern kernel: [1176840.374505]  ? srso_return_thunk+0x5/0x5f
2025-09-03T08:11:54.941745+02:00 ThreadripperIntern kernel: [1176840.375004]  ? clockevents_program_event+0xb6/0x140
2025-09-03T08:11:54.941745+02:00 ThreadripperIntern kernel: [1176840.375518]  ? srso_return_thunk+0x5/0x5f
2025-09-03T08:11:54.941746+02:00 ThreadripperIntern kernel: [1176840.376004]  ? tick_program_event+0x43/0xa0
2025-09-03T08:11:54.941746+02:00 ThreadripperIntern kernel: [1176840.376508]  ? srso_return_thunk+0x5/0x5f
2025-09-03T08:11:54.941747+02:00 ThreadripperIntern kernel: [1176840.376994]  ? hrtimer_interrupt+0x11f/0x250
2025-09-03T08:11:54.941747+02:00 ThreadripperIntern kernel: [1176840.377503]  ? srso_return_thunk+0x5/0x5f
2025-09-03T08:11:54.941748+02:00 ThreadripperIntern kernel: [1176840.377990]  ? irqentry_exit_to_user_mode+0x7b/0x260
2025-09-03T08:11:54.941749+02:00 ThreadripperIntern kernel: [1176840.378500]  ? srso_return_thunk+0x5/0x5f
2025-09-03T08:11:54.941750+02:00 ThreadripperIntern kernel: [1176840.378989]  ? irqentry_exit+0x43/0x50
2025-09-03T08:11:54.941750+02:00 ThreadripperIntern kernel: [1176840.379486]  ? srso_return_thunk+0x5/0x5f
2025-09-03T08:11:54.941751+02:00 ThreadripperIntern kernel: [1176840.379968]  entry_SYSCALL_64_after_hwframe+0x78/0x80
2025-09-03T08:11:54.941752+02:00 ThreadripperIntern kernel: [1176840.380466] RIP: 0033:0x7c4a257c1ded
2025-09-03T08:11:54.941752+02:00 ThreadripperIntern kernel: [1176840.380956] Code: 04 25 28 00 00 00 48 89 45 c8 31 c0 48 8d 45 10 c7 45 b0 10 00 00 00 48 89 45 b8 48 8d 45 d0 48 89 45 c0 b8 10 00 00 00 0f 05 <89> c2 3d 00 f0 ff ff 77 1a 48 8b 45 c8 64 48 2b 04 25 28 00 00 00
2025-09-03T08:11:54.941753+02:00 ThreadripperIntern kernel: [1176840.381481] RSP: 002b:00007ffee72871d0 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
2025-09-03T08:11:54.941754+02:00 ThreadripperIntern kernel: [1176840.381995] RAX: ffffffffffffffda RBX: 00007ffee72872f0 RCX: 00007c4a257c1ded
2025-09-03T08:11:54.941755+02:00 ThreadripperIntern kernel: [1176840.382527] RDX: 00007ffee72872f0 RSI: 00000000c020462a RDI: 0000000000000008
2025-09-03T08:11:54.941756+02:00 ThreadripperIntern kernel: [1176840.383054] RBP: 00007ffee7287220 R08: 00007ffee72872f0 R09: 00007ffee728730c
2025-09-03T08:11:54.941756+02:00 ThreadripperIntern kernel: [1176840.383585] R10: 00007ffee72872f0 R11: 0000000000000246 R12: 00000000c020462a
2025-09-03T08:11:54.941757+02:00 ThreadripperIntern kernel: [1176840.384123] R13: 0000000000000008 R14: 00007ffee728730c R15: 00007ffee7287240
2025-09-03T08:11:54.941758+02:00 ThreadripperIntern kernel: [1176840.384656]  </TASK>
2025-09-03T08:11:54.941759+02:00 ThreadripperIntern kernel: [1176840.385248] NVRM: _kgspLogXid119: ********************************************************************************
2025-09-03T08:11:54.942910+02:00 ThreadripperIntern kernel: [1176840.385788] NVRM: _issueRpcAndWait: rpcRecvPoll timedout for fn 76!

Driver in use:

root@ThreadripperIntern:~# nvidia-smi
Wed Aug 20 17:20:09 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.76.05              Driver Version: 580.76.05      CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 5090        On  |   00000000:0A:00.0 Off |                  N/A |
| 30%   46C    P1             76W /  400W |   22621MiB /  32607MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A            4631      C   ./llama-server                        22612MiB |
+-----------------------------------------------------------------------------------------+

Now I’ve tried to deactivate GSP according to Chapter 43. GSP Firmware

Disabling GSP Mode
The driver can be forced to disable use of GSP firmware by setting the kernel module parameter NVreg_EnableGpuFirmware=0.

Unfortunately I can’t confirm this was successful, because according to the docs…

Query Mode
The nvidia-smi utility can be used to query the current use of GSP firmware. It will display a valid version if GSP firmware is enabled, or “N/A” if disabled

But for me it still shows the version number:

root@ThreadripperIntern:~# nvidia-smi -q | grep -i "GSP Firmware"
    GSP Firmware Version                  : 580.76.05
root@ThreadripperIntern:~#

This is how I tried to deactivate GSP:

# first I put the following text into a new file /etc/modprobe.d/nvidia-gsp.conf:
root@ThreadripperIntern:~# cat /etc/modprobe.d/nvidia-gsp.conf
# disable the GSP firmware, forcing the driver to handle all tasks

options nvidia NVreg_EnableGpuFirmware=0
root@ThreadripperIntern:~# 

# then I did 
root@ThreadripperIntern:~# update-initramfs -u -k all

# finally
root@ThreadripperIntern:~# reboot

Directly after bootup llama-cpp didn’t recognize the GPU (ggml_cuda_init: failed to initialize CUDA: no CUDA-capable device is detected) but a few minutes later when I restarted llama-cpp, initialization was successful and now my RTX 5090 is running again.

Let’s see for how long until the next crash.

4

I’m not sure this is an option with Blackwell, which requires the open module driver.

According to the open module section:

“…because the open kernel modules depend on the GPU System Processor (GSP)…”.

5

@robert248 thanks for the info, yes apparently it can’t run without GSP. GSP Firmware is still loaded:

root@ThreadripperIntern:~# nvidia-smi -q | grep -i "GSP Firmware"
    GSP Firmware Version                  : 580.76.05
root@ThreadripperIntern:~#

and even with the options nvidia NVreg_EnableGpuFirmware=0 in place, I got the next crash (edit: this time it ran without problems from 2025-09-12 15:57:00 until 2025-10-25 04:02:05 i.e. 42 days. But the system was last rebooted 2025-10-16 17:49:35 i.e. crash-free since last reboot for 8 days 10 hours):

Oct 25 04:02:05 ThreadripperIntern kernel: NVRM: _kgspLogXid119: ********************************* GSP Timeout **********************************
Oct 25 04:02:05 ThreadripperIntern kernel: NVRM: _kgspLogXid119: Note: Please also check logs above.
Oct 25 04:02:05 ThreadripperIntern kernel: NVRM: GPU at PCI:0000:0a:00: GPU-701b524f-c979-7284-9737-0fecd8ca5f9b
Oct 25 04:02:05 ThreadripperIntern kernel: NVRM: GPU Board Serial Number: 1791625003310
Oct 25 04:02:05 ThreadripperIntern kernel: NVRM: Xid (PCI:0000:0a:00): 119, pid=4614, name=llama-server, Timeout after 45s of waiting for RPC response from GPU0 GSP! Expected function 76 (GSP_RM_CONTROL) (0x20800a9a 0x8).
Oct 25 04:02:05 ThreadripperIntern kernel: NVRM: GPU0 GSP RPC buffer contains function 76 (GSP_RM_CONTROL) and data 0x0000000020800a9a 0x0000000000000008.
Oct 25 04:02:05 ThreadripperIntern kernel: NVRM: GPU0 RPC history (CPU -> GSP):
Oct 25 04:02:05 ThreadripperIntern kernel: NVRM:     entry function                   data0              data1              ts_start           ts_end             duration actively_polling
Oct 25 04:02:05 ThreadripperIntern kernel: NVRM:      0    76   GSP_RM_CONTROL        0x0000000020800a9a 0x0000000000000008 0x000641f20adffbfb 0x0000000000000000          y
Oct 25 04:02:05 ThreadripperIntern kernel: NVRM:     -1    76   GSP_RM_CONTROL        0x0000000020800a9a 0x0000000000000008 0x000641f20a02446b 0x000641f20a02459b    304us  
Oct 25 04:02:05 ThreadripperIntern kernel: NVRM:     -2    76   GSP_RM_CONTROL        0x0000000020800a9a 0x0000000000000008 0x000641f209159883 0x000641f20915b9ae   8491us  
Oct 25 04:02:05 ThreadripperIntern kernel: NVRM:     -3    76   GSP_RM_CONTROL        0x0000000020800a9a 0x0000000000000008 0x000641f208d81921 0x000641f208d81a21    256us  
Oct 25 04:02:05 ThreadripperIntern kernel: NVRM:     -4    76   GSP_RM_CONTROL        0x0000000020800a9a 0x0000000000000008 0x000641f2074b3159 0x000641f2074b5200   8359us  
Oct 25 04:02:05 ThreadripperIntern kernel: NVRM:     -5    76   GSP_RM_CONTROL        0x0000000020800a9a 0x0000000000000008 0x000641f2066d7399 0x000641f2066d74c9    304us  
Oct 25 04:02:05 ThreadripperIntern kernel: NVRM:     -6    76   GSP_RM_CONTROL        0x0000000020800a9a 0x0000000000000008 0x000641f20580c699 0x000641f20580e850   8631us  
Oct 25 04:02:05 ThreadripperIntern kernel: NVRM:     -7    76   GSP_RM_CONTROL        0x0000000020800a9a 0x0000000000000008 0x000641f204a31071 0x000641f204a31170    255us  
Oct 25 04:02:05 ThreadripperIntern kernel: NVRM: GPU0 RPC event history (CPU <- GSP):
Oct 25 04:02:05 ThreadripperIntern kernel: NVRM:     entry function                   data0              data1              ts_start           ts_end             duration during_incomplete_rpc
Oct 25 04:02:05 ThreadripperIntern kernel: NVRM:      0    4124 GSP_LOCKDOWN_NOTICE   0x0000000000000000 0x0000000000000000 0x0006414890e21da6 0x0006414890e21da7      1us  
Oct 25 04:02:05 ThreadripperIntern kernel: NVRM:     -1    4124 GSP_LOCKDOWN_NOTICE   0x0000000000000001 0x0000000000000000 0x0006414890e20984 0x0006414890e20984           
Oct 25 04:02:05 ThreadripperIntern kernel: NVRM:     -2    4124 GSP_LOCKDOWN_NOTICE   0x0000000000000000 0x0000000000000000 0x0006414890e2086b 0x0006414890e2086c      1us  
Oct 25 04:02:05 ThreadripperIntern kernel: NVRM:     -3    4124 GSP_LOCKDOWN_NOTICE   0x0000000000000001 0x0000000000000000 0x0006414890e204bf 0x0006414890e204c0      1us  
Oct 25 04:02:05 ThreadripperIntern kernel: NVRM:     -4    4124 GSP_LOCKDOWN_NOTICE   0x0000000000000000 0x0000000000000000 0x0006414890e09459 0x0006414890e09459           
Oct 25 04:02:05 ThreadripperIntern kernel: NVRM:     -5    4124 GSP_LOCKDOWN_NOTICE   0x0000000000000001 0x0000000000000000 0x0006414890e093d0 0x0006414890e093d1      1us  
Oct 25 04:02:05 ThreadripperIntern kernel: NVRM:     -6    4124 GSP_LOCKDOWN_NOTICE   0x0000000000000000 0x0000000000000000 0x0006414890e080e4 0x0006414890e080e4           
Oct 25 04:02:05 ThreadripperIntern kernel: NVRM:     -7    4124 GSP_LOCKDOWN_NOTICE   0x0000000000000001 0x0000000000000000 0x0006414890e05d71 0x0006414890e05d71           
Oct 25 04:02:05 ThreadripperIntern kernel: NVRM: _kgspLogXid119: ********************************************************************************
Oct 25 04:02:05 ThreadripperIntern kernel: NVRM: _issueRpcAndWait: rpcRecvPoll timedout for fn 76!
Oct 25 04:02:51 ThreadripperIntern kernel: NVRM: Xid (PCI:0000:0a:00): 119, pid=4614, name=llama-server, Timeout after 45s of waiting for RPC response from GPU0 GSP! Expected function 76 (GSP_RM_CONTROL) (0x20801702 0x4).
Oct 25 04:02:51 ThreadripperIntern kernel: NVRM: _issueRpcAndWait: rpcRecvPoll timedout for fn 76!
Oct 25 04:02:51 ThreadripperIntern kernel: NVRM: subdeviceCtrlCmdMcServiceInterrupts_IMPL: NVRM_RPC: NV2080_CTRL_CMD_MC_SERVICE_INTERRUPTS failed with error 0x65
Oct 25 04:03:36 ThreadripperIntern kernel: NVRM: Xid (PCI:0000:0a:00): 119, pid=4614, name=llama-server, Timeout after 45s of waiting for RPC response from GPU0 GSP! Expected function 76 (GSP_RM_CONTROL) (0x20800a9a 0x8).
Oct 25 04:03:36 ThreadripperIntern kernel: NVRM: nvAssertFailedNoLog: Assertion failed: Back to back GSP RPC timeout detected! GPU marked for reset @ kernel_gsp.c:2344
Oct 25 04:03:36 ThreadripperIntern kernel: NVRM: _issueRpcAndWait: rpcRecvPoll timedout for fn 76!
Oct 25 04:03:36 ThreadripperIntern kernel: NVRM: Xid (PCI:0000:0a:00): 154, GPU recovery action changed from 0x0 (None) to 0x1 (GPU Reset Required)
Oct 25 04:03:37 ThreadripperIntern kernel: NVRM: subdeviceCtrlCmdMcServiceInterrupts_IMPL: NVRM_RPC: NV2080_CTRL_CMD_MC_SERVICE_INTERRUPTS failed with error 0x62
Oct 25 04:03:38 ThreadripperIntern kernel: NVRM: subdeviceCtrlCmdMcServiceInterrupts_IMPL: NVRM_RPC: NV2080_CTRL_CMD_MC_SERVICE_INTERRUPTS failed with error 0x62
Oct 25 04:03:39 ThreadripperIntern kernel: NVRM: subdeviceCtrlCmdMcServiceInterrupts_IMPL: NVRM_RPC: NV2080_CTRL_CMD_MC_SERVICE_INTERRUPTS failed with error 0x62

I’ve contacted @robert248 about it and he told me:

For me the solution was replacing (RMA process) my defective card with a working card. I think there was some bad NVRAM.

I’ll do an RMA process now too, and post back here once my new card arrives and has been running for a while!

edit: current driver is:

root@ThreadripperIntern:~# nvidia-smi
Fri Nov 14 19:42:21 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.76.05              Driver Version: 580.76.05      CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 5090        On  |   00000000:0A:00.0 N/A |                  N/A |
|ERR!  ERR! ERR!             N/A  /  N/A  |   22623MiB /  32607MiB |     N/A      Default |
|                                         |                        |                 ERR! |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+
root@ThreadripperIntern:~# nvidia-smi -q | grep -i "GSP Firmware"
    GSP Firmware Version                  : 580.76.05
root@ThreadripperIntern:~#

edit: RMA request is 904421