580.x open module: softlockup panics on B300

Graphics / Linux Linux
, ,
1

Environment

  • GPU: 8x NVIDIA B300 SXM6 AC
  • Driver: 580.159.04 open kernel module (GSP); behavior also verified against 580.178.04 sources
  • Kernel: 6.6.y, Ubuntu 24.04 userland, CONFIG_PREEMPT_VOLUNTARY
  • Workload: ~150 CUDA client processes per node (ray + vLLM inference, 21 processes per GPU, MPS dual instance)

Evidence comes from three hosts — two crashed, one live

  • Host A and Host B: crashed (kdump captured). Both panicked with
    softlockup: hung tasks (softlockup_panic=1), ~4 hours apart, same workload.
  • Host C: identical symptom, has NOT crashed. Same driver/workload, 6 GPU-vector
    CPUs saturated at 70-97%. All bpftrace/perf numbers below were captured live on
    Host C while degraded     — they are measurements of the failure mode in progress,
    not post-mortem reconstruction.

Crashed hosts: NMI backtraces (kdump)

At panic time, multiple CPUs (one per busy GPU) were spinning inside the GPU
interrupt handler on the event spinlock:

__raw_spin_lock_irqsave
_raw_spin_lock_irqsave
os_acquire_spinlock              [nvidia]   <- spin_lock_irqsave, IRQs off
portSyncSpinlockAcquire          [nvidia]
osDereferenceObjectCount         [nvidia]
osNotifyEvent                    [nvidia]
_gpuEngineEventNotificationListNotify [nvidia]
gkflcnServiceNotificationInterrupt_IMPL [nvidia]
intrServiceNotificationRecords_IMPL [nvidia]
_intrServiceNonStallLeaf_TU102   [nvidia]
intrServiceNonStall_TU102        [nvidia]
osInterruptPending               [nvidia]
osIsr                            [nvidia]
rm_isr                           [nvidia]
nvidia_isr                       [nvidia]
nvidia_isr_msix                  [nvidia]
handle_edge_irq / common_interrupt

On Host B, the thread that exceeded the 40s softlockup threshold was itself
executing an NVIDIA ioctl — i.e., both the CPU consumer (IRQ path above) and the
stalled victim are driver code paths:

watchdog: BUG: soft lockup - CPU#144 stuck for 44s! [ray::DashboardA:57169]
...
RIP: 0010:refHasAncestor+0x8/0x40 [nvidia]
 ? clientRefIterNext+0xd0/0x130          [nvidia]
 _gpushareddataRequestDataPoll+0x8d/0x100 [nvidia]   <- RUSD data poll,
 resControl_IMPL+0x1a9/0x1b0             [nvidia]       iterates ALL RM clients
 serverControl+0x47e/0x590               [nvidia]
 _rmapiRmControl+0x4f2/0x820             [nvidia]
 rmapiControlWithSecInfo+0x79/0x140      [nvidia]
 rmapiControlWithSecInfoTls+0x8f/0xf0    [nvidia]
 _nv04ControlWithSecInfo+0x8d/0xa0       [nvidia]
 RmIoctl+0x90b/0xda0                     [nvidia]
 rm_ioctl+0x66/0x4f0                     [nvidia]
 nvidia_unlocked_ioctl+0x4d3/0x9b0       [nvidia]
 __x64_sys_ioctl / do_syscall_64

With the CPU ~94% consumed by interrupt servicing, this client-iteration loop
(no scheduling point) could not finish within 40s → watchdog → host panic.
(On Host A the starved victim was an xfs log-completion kworker instead; any
kernel path without a scheduling point on a saturated CPU triggers the same.)

kdump vmcores/dmesg of both crashed hosts available on request.

Live measurements on Host C (same symptom, pre-crash)

  • nvidia interrupt rate: ~8.6k/s (healthy-node baseline ~10.7k/s — NOT an interrupt storm)
  • Per-interrupt handler duration (tracepoint irq_handler_entry/exit): bimodal,
    heavy population 2-4 ms per interrupt (healthy baseline: tens of us)
  • os_acquire_spinlock acquisitions per interrupt: bimodal, heavy population 512-2048
  • Lock hold time: <1 us; lock wait: 4-16 us; ~540k lock acquisitions/sec machine-wide
  • perf (99Hz, saturated CPUs): ~30% of samples in native_queued_spin_lock_slowpath,
    call chains dominated by the nvidia_isr_msix → … → osNotifyEvent path above;
    contending contexts include the hard-IRQ path, ksoftirqd, and user threads inside
    nvidia ioctls (cuda-EvtHandlr, application threads).

Arithmetic cross-check: ~1000-2000 events notified per interrupt x ~2 global-lock
round-trips each (reference + dereference in the osNotifyEvent path) x 4-16 us
contention = 2-4 ms/interrupt, matching the duration histogram; summed ISR time
matches the 6 saturated CPUs.

Code references (public sources)

  • src/nvidia/src/kernel/rmapi/event_notification.c, _gpuEngineEventNotificationListNotify
    (L338-438 @ tag 580.95.05): per-entry notification loop
  • src/nvidia/arch/nvalloc/unix/src/os.c: referenceEventForPost / dereferenceEventForPost
    acquire nv->event_spinlock per event
  • Note: 595.104.02 / 615.78.08 added a bDoRefCount parameter to osNotifyEvent that skips
    the second acquisition — this halves the cost, but the O(N)-events-per-interrupt x
    shared-spinlock structure remains, and the mitigation is NOT in the 580 production branch.