# Real time patch set linux driver BUG()

**URL:** <https://forums.developer.nvidia.com/t/real-time-patch-set-linux-driver-bug/147865>\
**Category:** Linux\
**Tags:** linux, cuda, kernel\
**Created:** [August 21, 2020, 11:46pm UTC](https://forums.developer.nvidia.com/t/real-time-patch-set-linux-driver-bug/147865 "2020-08-21T23:46:07Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![nvsnek](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@nvsnek](https://forums.developer.nvidia.com/u/nvsnek)\
**Post date:** [August 21, 2020, 11:46pm UTC](https://forums.developer.nvidia.com/t/real-time-patch-set-linux-driver-bug/147865/1 "2020-08-21T23:46:07Z")

</div>

I’m running into an issue while attempting to use the 418.67 display driver with a custom kernel running the real time patch set. I’m intermittently hitting a “BUG: scheduling while atomic” from one of the nvidia isr threads.

I’m aware that this is not exactly an officially supported configuration. Nonetheless, it’s still important to me, and I’d appreciate help resolving the issue, as I’m likely not the only person running the real time patch set with cuda.

The kernel being used is essentially taken stock from rt stable releases here: git://git.kernel.org/pub/scm/linux/kernel/git/rt/linux-stable-rt.git.

The release branch used is rather old, v4.19.90-rt35. CONFIG\_PREEMPT\_RT\_FULL=y is being used.

The driver is fairly old as well, it’s the driver release that was packaged with a prior cuda 10.1.168 release.

$ md5sum ./NVIDIA-Linux-x86\_64-418.67.run  
662865d9a7b5ef1ac3402e098a5fb91f ./NVIDIA-Linux-x86\_64-418.67.run

I get the backtrace while running a cuda app that runs kernels and uses the video codec sdk.

[5493.745489] BUG: scheduling while atomic: irq/216-s-nvidi/9203/0x00000002  
[5493.745490] Modules linked in: nvidia\_uvm(O) sch\_fq ipt\_MASQUERADE iptable\_nat nf\_nat\_ipv4 nf\_nat bpfilter intel\_rapl x86\_pkg\_temp\_thermal intel\_powerclamp crct10dif\_pclmul crct10dif\_common crc32\_pclmul crc32c\_intel ghash\_clmulni\_intel pcbc aesni\_intel nvidia\_drm(PO) aes\_x86\_64 crypto\_simd nvidia\_modeset(PO) cryptd glue\_helper nvidia(PO) ast ttm lpc\_ich mfd\_core mei\_me mei ipmi\_ssif ipmi\_si ipmi\_devintf ipmi\_msghandler ftdi\_sio usbserial cdc\_acm  
[5493.745507] Preemption disabled at:  
[5493.745508] [\<0000000000000000\>] (null)  
[5493.745511] CPU: 37 PID: 9203 Comm: irq/216-s-nvidi Tainted: P W O 4.19.90-rt35-app #1  
[5493.745512] Hardware name: Intel Corporation S2600BPB/S2600BPB, BIOS SE5C620.86B.02.01.0008.031920191559 03/19/2019  
[5493.745512] Call Trace:  
[5493.745524] dump\_stack+0x68/0x9b  
[5493.745528] \_\_schedule\_bug+0x71/0xc0  
[5493.745530] \_\_schedule+0x56d/0x680  
[5493.745531] schedule+0x4c/0xf0  
[5493.745534] rt\_spin\_lock\_slowlock\_locked+0x108/0x2d0  
[5493.745536] rt\_spin\_lock\_slowlock+0x50/0x80  
[5493.745541] \_\_wake\_up\_common\_lock+0x61/0xb0  
[5493.745758] \_nv029653rm+0x147/0x180 [nvidia]  
[5493.745966] ? \_nv015202rm+0x62/0x140 [nvidia]  
[5493.746121] ? \_nv018379rm+0x70/0x190 [nvidia]  
[5493.746312] ? \_nv024595rm+0x1cf/0x5e0 [nvidia]  
[5493.746424] ? \_nv000938rm+0xe8/0x150 [nvidia]  
[5493.746429] ? irq\_forced\_thread\_fn+0x70/0x70  
[5493.746540] ? rm\_isr\_bh+0x1c/0x60 [nvidia]  
[5493.746606] ? nvidia\_isr\_common\_bh+0x62/0x70 [nvidia]  
[5493.746608] ? irq\_thread\_fn+0x1b/0x60  
[5493.746609] ? irq\_thread+0x12e/0x1b0  
[5493.746610] ? preempt\_count\_sub+0x94/0xe0  
[5493.746611] ? wake\_threads\_waitq+0x30/0x30  
[5493.746614] ? kthread+0xf5/0x130  
[5493.746615] ? irq\_thread\_check\_affinity+0x80/0x80  
[5493.746616] ? kthread\_bind+0x30/0x30  
[5493.746619] ? ret\_from\_fork+0x24/0x30

The last listed call site from the nvidia driver before it lands in kernel code is [\_nv029653rm+0x147]

This symbol is in the binary component of the nvidia driver.

$ objdump -d ./kernel/nvidia/nv-kernel.o\_binary | grep " \<\_nv029653rm\>:"  
0000000000704750 \<\_nv029653rm\>:

Closely looking at the last call site reveals a little more:

$ objdump -d ./kernel/nvidia/nv-kernel.o\_binary | grep -C 1 704897  
70488f: 4c 89 f7 mov %r14,%rdi  
704892: e8 00 00 00 00 callq 704897 \<\_nv029653rm+0x147\>. # 4 byte reloc here.  
704897: 31 c0 xor %eax,%eax  
704899: e9 d5 fe ff ff jmpq 704773 \<\_nv029653rm+0x23\>

The actual final callsite is likely nv\_post\_event:

objdump -r ./kernel/nvidia/nv-kernel.o\_binary | grep 704893  
0000000000704893 R\_X86\_64\_PC32 nv\_post\_event-0x0000000000000004 # call to nv\_post\_event

nv\_post\_event from the open source component of the driver is as follows:

void NV\_API\_CALL nv\_post\_event(  
nv\_state\_t \*nv,  
nv\_event\_t \*event,  
NvHandle handle,  
NvU32 index,  
NvBool data\_valid  
)  
{  
nv\_file\_private\_t \*nvfp = event-\>file;  
unsigned long eflags;  
nvidia\_event\_t \*nvet;

```
NV_SPIN_LOCK_IRQSAVE(&nvfp->fp_lock, eflags);

if (data_valid)
{
    NV_KMALLOC_ATOMIC(nvet, sizeof(nvidia_event_t));
    if (nvet == NULL)
    {
        NV_SPIN_UNLOCK_IRQRESTORE(&nvfp->fp_lock, eflags);
        return;
    }

    if (nvfp->event_tail != NULL)
        nvfp->event_tail->next = nvet;
    if (nvfp->event_head == NULL)
        nvfp->event_head = nvet;
    nvfp->event_tail = nvet;
    nvet->next = NULL;

    nvet->event = *event;
    nvet->event.hObject = handle;
    nvet->event.index = index;
}

nvfp->event_pending = TRUE;

NV_SPIN_UNLOCK_IRQRESTORE(&nvfp->fp_lock, eflags);

wake_up_interruptible(&nvfp->waitqueue);

```

}

On PREEMPT\_RT\_FULL, NV\_SPIN\_LOCK\_IRQSAVE cannot block (as it is a raw\_spin\_lock), so it’s very likely the call to wake\_up\_interruptible is to blame.

The core of wake\_up\_interruptible immediately contains a call to spin\_lock\_irq. On PREEMPT\_RT\_FULL kernels, spin locks are replaced  
with blocking priority inheriting mutexes. The last part of the stack trace shows blocking on just that:

[5493.745534] rt\_spin\_lock\_slowlock\_locked+0x108/0x2d0  
[5493.745536] rt\_spin\_lock\_slowlock+0x50/0x80  
[5493.745541] \_\_wake\_up\_common\_lock+0x61/0xb0

Are there any modifications that can be made to the open source layer to avoid the call to wake\_up\_interruptible? Can the call be deferred to a preemptable context with rcu? Can nvidia change binary component of the kernel driver to avoid calls to nv\_post\_event in non-preemptable contexts? I’ve repeated the same tests with a recent R450 release, and find the same issue, albeit with a different set of mangled binary symbol names on the backtrace.

Am I correct in my analysis of the bug? Any help is appreciated.

---

<div class="post-metadata">

**Author:** ![jamcpca](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@jamcpca](https://forums.developer.nvidia.com/u/jamcpca)\
**Post date:** [December 3, 2020, 10:36pm UTC](https://forums.developer.nvidia.com/t/real-time-patch-set-linux-driver-bug/147865/2 "2020-12-03T22:36:09Z")

</div>

Hi,

Have you received any information to solve it?  
We have the same problem.

Regards,

---

<div class="post-metadata">

**Author:** ![chaoqun.fan](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@chaoqun.fan](https://forums.developer.nvidia.com/u/chaoqun.fan)\
**Post date:** [February 17, 2022, 8:39am UTC](https://forums.developer.nvidia.com/t/real-time-patch-set-linux-driver-bug/147865/3 "2022-02-17T08:39:00Z")

</div>

Hi,

Have you received any information to solve it?  
We have the same problem.

Regards,