Nvidia Driver BUG: scheduling while atomic for PREEMPT_RT kernel on RTX3060 GPU

liuyue · November 10, 2022, 9:54am

hi，
I want to use RT patch on the ubuntu 20.04 PC. The hardware and software environments I use are as follows:
Hardware environment: ThinkStation k 1710700/32G/256G+1T/500W power supply/high-performance video card RTX3060 (12G)
System version: Ubuntu 20.04.1
Linux Kernel: linux 5.13.0
Rt-patch：patch-5.13-rt1.patch.gz
nvidia-driver：nvidia-driver-510, nvidia-driver-520

Linux kernel config：
make menuconfig
General Setup ->Preemption Model is set to Fully Preemptable Kernel (RT)
General Setup ->Timers subsystem ->Timer tick handling is set to Full dynamics system
General Setup ->Timers subsystem Enable High Resolution Timer Support
Processor type and features ->Timer frequency is set to 1000 HZ

sudo gedit .config
CONFIG_ SYSTEM_ TRUSTED_ KEY=““
CONFIG_ SYSTEM_ REVOCATION_ KEYS=”"

Build Nvidia driver with IGNORE_PREEMPT_RT_PRESENCE=1

sudo env NV_VERBOSE=1 \ make -j8 NV_EXCLUDE_BUILD_MODULES=‘’ \ KERNEL_UNAME=$(uname -r) \ IGNORE_XEN_PRESENCE=1 \ IGNORE_CC_MISMATCH=1 \ IGNORE_PREEMPT_RT_PRESENCE=1 \ SYSSRC=/lib/modules/$(uname -r)/build \ LD=/usr/bin/ld.bfd \ modules

I have successfully compiled and installed the RT kernel and nvidia-driver-510 driver, and run glmark2 and RT test at the same time. An error is reported in the kernel log, and there is a probability that the kernel is stuck

Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699076] BUG: scheduling while atomic: irq/140-s-nvidi/1223/0x00000002
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699078] Modules linked in: rfcomm xt_multiport ip6table_filter ip6_tables xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype iptable_filter iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c bpfilter br_netfilter bridge stp llc cmac algif_hash algif_skcipher af_alg overlay bnep nls_iso8859_1 nvidia_uvm(POE) mei_hdcp intel_rapl_msr intel_tcc_cooling x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul input_leds ghash_clmulni_intel joydev binfmt_misc aesni_intel crypto_simd cryptd rapl intel_cstate wmi_bmof rtw88_8822ce rtw88_8822c rtw88_pci snd_sof_pci_intel_cnl rtw88_core snd_sof_intel_hda_common btusb ee1004 btrtl soundwire_intel mac80211 btbcm soundwire_generic_allocation btintel soundwire_cadence snd_sof_intel_hda bluetooth cfg80211 snd_sof_pci ecdh_generic snd_sof_xtensa_dsp libarc4 ecc snd_sof snd_soc_hdac_hda snd_hda_ext_core snd_hda_codec_realtek snd_soc_acpi_intel_match snd_soc_acpi
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699102] snd_hda_codec_generic soundwire_bus ledtrig_audio snd_soc_core snd_hda_codec_hdmi snd_compress ac97_bus snd_pcm_dmaengine nvidia_drm(POE) snd_hda_intel nvidia_modeset(POE) snd_intel_dspcfg snd_intel_sdw_acpi snd_hda_codec snd_hda_core snd_hwdep nvidia(POE) snd_pcm snd_seq_midi snd_seq_midi_event snd_rawmidi drm_kms_helper snd_seq cec mac_hid rc_core snd_seq_device fb_sys_fops syscopyarea snd_timer sysfillrect int3400_thermal processor_thermal_device int3403_thermal sysimgblt acpi_pad acpi_tad acpi_thermal_rel processor_thermal_rfim snd processor_thermal_mbox processor_thermal_rapl soundcore intel_rapl_common mei_me int340x_thermal_zone intel_soc_dts_iosf mei intel_pch_thermal sch_fq_codel msr parport_pc ppdev lp parport drm ramoops reed_solomon efi_pstore ip_tables x_tables autofs4 hid_generic usbhid hid crc32_pclmul nvme r8169 i2c_i801 ahci xhci_pci i2c_smbus nvme_core realtek libahci xhci_pci_renesas wmi video pinctrl_cannonlake
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699126] CPU: 2 PID: 1223 Comm: irq/140-s-nvidi Tainted: P OE 5.13.0-rt1 #1
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699127] Hardware name: LENOVO 12B4A01MCD/3313, BIOS M4PKT0EA 12/15/2021
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699128] Call Trace:
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699130] dump_stack+0x7d/0x9c
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699133] __schedule_bug.cold+0x4c/0x5d
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699135] __schedule+0x766/0x990
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699138] ? rt_mutex_setprio+0x148/0x4f0
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699139] schedule_rtlock+0x1f/0x40
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699141] rtlock_slowlock_locked+0x3f8/0xf90
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699142] ? default_wake_function+0x1a/0x30
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699144] ? autoremove_wake_function+0x12/0x40
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699147] rt_read_lock+0x68/0xa0
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699148] ep_poll_callback+0x34/0x240
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699149] __wake_up_common+0x7b/0x140
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699151] __wake_up_common_lock+0x77/0xb0
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699152] __wake_up+0x13/0x20
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699153] nv_post_event+0x6e/0x130 [nvidia]
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699308] _nv033648rm+0x59/0x80 [nvidia]
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699454] WARNING: kernel stack frame pointer at 00000000b4a331b1 in irq/140-s-nvidi:1223 has bad value 000000004184be0c
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699455] unwind stack type:0 next_sp:0000000000000000 mask:0x2 graph_idx:0
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699456] 00000000937b2db2: ffffba21c1df79a8 (0xffffba21c1df79a8)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699458] 000000002d79566e: ffffffff9d3a7404 (show_trace_log_lvl+0x20d/0x2f4)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699460] 000000009c2a8f91: ffffffffc1120629 (_nv033648rm+0x59/0x80 [nvidia])
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699608] 00000000831ac702: ffffba21c1df7cb8 (0xffffba21c1df7cb8)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699609] 0000000062c5b820: 0000000000000000 …
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699609] 00000000513fe63a: 000000009ca03ffe (0x9ca03ffe)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699610] 00000000ac7acb7f: 0000000000000002 (0x2)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699610] 00000000fc8e4cab: 0000000000000001 (0x1)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699611] 00000000aa10c8ec: ffffba21c1df4000 (0xffffba21c1df4000)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699611] 00000000f1acefa1: ffffba21c1df8000 (0xffffba21c1df8000)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699612] 00000000c8fe049f: 0000000000000000 …
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699612] 000000000d325cf5: ffffba21c1df4000 (0xffffba21c1df4000)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699613] 000000003e9e869b: ffffba21c1df8000 (0xffffba21c1df8000)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699613] 000000009f84d4ad: 0000000000000000 …
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699614] 000000009a336926: 0000000000000002 (0x2)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699614] 0000000048c1b318: ffff99e4b6570000 (0xffff99e4b6570000)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699615] 00000000e1f6367c: 0000000100000000 (0x100000000)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699615] 00000000b70a72b3: ffffba21c1df7cb0 (0xffffba21c1df7cb0)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699616] 00000000678a6dcf: ffffba21c1df78c0 (0xffffba21c1df78c0)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699616] 00000000baec0de0: ffffffffc1120629 (_nv033648rm+0x59/0x80 [nvidia])
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699760] 000000006c04c7d7: 0000000000000000 …
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699761] 0000000054c6bd28: 2ca20746673cdf00 (0x2ca20746673cdf00)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699761] 00000000f74ae855: 0000000000000286 (0x286)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699762] 00000000cac07d14: 0000000000000000 …
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699762] 0000000051eb183c: ffff99e4b6570000 (0xffff99e4b6570000)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699763] 000000001bdce17b: 0000000000000001 (0x1)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699763] 0000000061487d58: 0000000000000002 (0x2)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699764] 0000000059e1b790: ffffba21c1df79b8 (0xffffba21c1df79b8)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699764] 00000000e50cf3e9: ffffffff9d3a7528 (show_stack+0x3d/0x3f)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699765] 000000006b126ad4: ffffba21c1df79d8 (0xffffba21c1df79d8)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699766] 000000001584c574: ffffffff9d3c781c (dump_stack+0x7d/0x9c)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699768] 0000000032a80a6b: 0000000000029740 (0x29740)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699769] 00000000ac1c6e12: ffff99ebbdaa9740 (0xffff99ebbdaa9740)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699769] 0000000087a1c4f4: ffffba21c1df79e8 (0xffffba21c1df79e8)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699770] 00000000ae8eff22: ffffffff9d3aed2d (__schedule_bug.cold+0x4c/0x5d)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699772] 00000000844d7648: ffffba21c1df7a40 (0xffffba21c1df7a40)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699772] 0000000073364397: ffffffff9d4148f6 (__schedule+0x766/0x990)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699774] 000000001a928bee: ffffffff9c8eb828 (rt_mutex_setprio+0x148/0x4f0)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699776] 00000000b3cebb67: ffffffff9deb17b0 (idle_sched_class+0xd0/0xd0)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699778] 00000000ed109af1: ffffffff9deb17b0 (idle_sched_class+0xd0/0xd0)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699779] 00000000f51cc06a: 2ca20746673cdf00 (0x2ca20746673cdf00)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699780] 000000008886f708: ffff99e4b6570000 (0xffff99e4b6570000)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699780] 00000000e25746ed: ffff99e4b6570000 (0xffff99e4b6570000)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699781] 00000000945d4a6e: ffff99e4b6570000 (0xffff99e4b6570000)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699781] 0000000089cb8129: ffff99e4c4fc28c0 (0xffff99e4c4fc28c0)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699782] 000000006fbcb38a: ffffba21c1df7ab8 (0xffffba21c1df7ab8)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699782] 000000008cb0463f: ffffba21c1df7a58 (0xffffba21c1df7a58)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699783] 00000000bb795c00: ffffffff9d414c6f (schedule_rtlock+0x1f/0x40)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699784] 00000000b820b118: ffff99e48ce21b98 (0xffff99e48ce21b98)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699785] 00000000b62a1093: ffffba21c1df7b38 (0xffffba21c1df7b38)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699785] 00000000b96ed557: ffffffff9d41a2d8 (rtlock_slowlock_locked+0x3f8/0xf90)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699786] 00000000269a3ded: ffffba21c1df7a78 (0xffffba21c1df7a78)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699787] 00000000c61855a7: ffffffff9c8e99fa (default_wake_function+0x1a/0x30)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699789] 00000000d39c1b42: ffffba21c1df7a90 (0xffffba21c1df7a90)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699790] 00000000763cfd00: ffffffff9c909a82 (autoremove_wake_function+0x12/0x40)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699791] 00000000c6c10109: ffff99e48ce21b28 (0xffff99e48ce21b28)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699792] 000000001a34045f: ffffba21c1df7ae0 (0xffffba21c1df7ae0)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699792] 00000000764aad33: ffffba21c1df7ab8 (0xffffba21c1df7ab8)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699793] 00000000d597bb72: ffff99e4c4fc28c0 (0xffff99e4c4fc28c0)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699793] 00000000c8e82d25: 0000000300000000 (0x300000000)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699794] 000000009335cc68: ffff99e48ce21b40 (0xffff99e48ce21b40)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699794] 0000000077f4b31e: 0000000000000001 (0x1)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699795] 00000000b34a230c: ffffba21c540fc60 (0xffffba21c540fc60)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699795] 00000000c1c96ce7: 0000000000000000 …
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699796] 000000008b481687: ffffba21c1df7ad0 (0xffffba21c1df7ad0)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699796] 0000000090e52d1f: 0000000000000000 …
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699796] 00000000c1161f1e: ffff99e4b6570000 (0xffff99e4b6570000)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699797] 0000000089fa6abc: ffff99e48ce21b98 (0xffff99e48ce21b98)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699797] 00000000b3105f41: 0000003100001000 (0x3100001000)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699798] 0000000069aea143: 0000000000000000 …
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699798] 00000000b6f29168: 2ca20746673cdf00 (0x2ca20746673cdf00)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699799] 000000006716d377: ffff99e48ce21b90 (0xffff99e48ce21b90)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699799] 00000000a57f5d71: ffff99e48ce21b98 (0xffff99e48ce21b98)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699800] 00000000618cf608: ffff99e480c48200 (0xffff99e480c48200)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699800] 00000000d0e999ba: ffff99e48ce21b90 (0xffff99e48ce21b90)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699801] 000000001655a568: 0000000000000000 …
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699801] 0000000079a27a5b: ffffba21c1df7b58 (0xffffba21c1df7b58)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699802] 00000000241d47b6: ffffffff9d41af68 (rt_read_lock+0x68/0xa0)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699803] 000000004733dd6d: ffff99e4cba0d610 (0xffff99e4cba0d610)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699803] 000000000c01c490: ffff99e48ce21b00 (0xffff99e48ce21b00)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699804] 00000000a261790a: ffffba21c1df7b98 (0xffffba21c1df7b98)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699804] 000000004afa098e: ffffffff9cb7fa34 (ep_poll_callback+0x34/0x240)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699806] 00000000cfd7c950: ffffba21c1df7b78 (0xffffba21c1df7b78)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699807] 0000000086d4671d: ffff99e4e2b32cd8 (0xffff99e4e2b32cd8)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699807] 00000000827722e6: 0000000000000000 …
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699807] 00000000ac19a2fb: ffffba21c1df7be8 (0xffffba21c1df7be8)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699808] 000000003714af0c: ffffffff9c9091cb (__wake_up_common+0x7b/0x140)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699809] 00000000d33ed338: ffffba21c1df7bf8 (0xffffba21c1df7bf8)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699810] 00000000b2d2bd27: 0000000100000001 (0x100000001)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699810] 00000000460fce76: ffff99e4e2b32cf0 (0xffff99e4e2b32cf0)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699811] 0000000099d2c0f2: ffff99e4e2b32cd0 (0xffff99e4e2b32cd0)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699811] 000000007de58f3d: 0000000000000001 (0x1)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699812] 000000004d4bd692: 0000000000000000 …
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699812] 0000000065466c63: 0000000000000001 (0x1)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699812] 0000000015a4fab7: ffffba21c1df7c50 (0xffffba21c1df7c50)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699813] 0000000073a55cde: ffffffff9c909587 (__wake_up_common_lock+0x77/0xb0)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699814] 00000000cd63e3f2: 0000000000000000 …
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699814] 00000000df389070: ffffba21c1df7c10 (0xffffba21c1df7c10)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699815] 00000000c5803703: ffffba21c1df7c10 (0xffffba21c1df7c10)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699815] 00000000c049f818: 2ca20746673cdf00 (0x2ca20746673cdf00)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699816] 000000004aa08621: ffff99e4e2b32c00 (0xffff99e4e2b32c00)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699816] 000000006e9767b2: ffff99e4e2b32ccc (0xffff99e4e2b32ccc)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699817] 00000000f18aff5a: ffff99e4e90abf08 (0xffff99e4e90abf08)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699817] 000000005290d95c: 0000000000000001 (0x1)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699818] 00000000db1ac800: 00000000b00f0003 (0xb00f0003)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699818] 0000000029a91b79: ffffba21c1df7c60 (0xffffba21c1df7c60)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699819] 000000002cf4ef4b: ffffffff9c9095d3 (__wake_up+0x13/0x20)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699820] 00000000b27c9b3a: ffffba21c1df7cb0 (0xffffba21c1df7cb0)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699821] 000000009e22c014: ffffffffc07ef1ae (nv_post_event+0x6e/0x130 [nvidia])
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699917] 0000000081ed64d8: 0000000000000082 (0x82)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699918] 000000002cca9c0e: 0000000000000286 (0x286)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699918] 000000007465cf35: 0000000000000000 …
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.699918] 0000000087633885: ffffffffc2b5ba60 (nv_kthread_q+0x60/0xfffffffffe68f600 [nvidia])
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.700027] 000000006a66a251: ffff99e4e90abf08 (0xffff99e4e90abf08)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.700028] 00000000c72bce97: 0000000000000000 …
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.700028] 00000000b4a331b1: ffff99e4b6565af0 (0xffff99e4b6565af0)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.700028] 00000000212b686d: ffffffffc1120629 (_nv033648rm+0x59/0x80 [nvidia])
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.700176] 00000000e80f19ff: 0000000000000000 …
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.700176] 0000000092bf40de: ffff99e4b65b0008 (0xffff99e4b65b0008)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.700177] 00000000679a454e: 0000000000000000 …
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.700177] 000000008ec49ace: ffff99e4880e3988 (0xffff99e4880e3988)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.700178] 00000000178dd63c: 0000000000000000 …
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.700178] 000000001954f8ba: ffffffffc1121b8b (_nv031682rm+0xab/0xc0 [nvidia])
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.700320] 00000000ee253c66: ffff99e4b6565c30 (0xffff99e4b6565c30)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.700321] 0000000050a65d25: ffffffffc0880234 (_nv029915rm+0xf4/0x120 [nvidia])
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.700447] 000000005a903a26: ffff99e4b65b0008 (0xffff99e4b65b0008)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.700447] 00000000b045d6df: 0000000000000100 (0x100)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.700448] 00000000ac7c3689: 0000000000000000 …
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.700448] 000000002f376d79: 0000000100000001 (0x100000001)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.700449] 0000000053c3a1eb: 0000000000000100 (0x100)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.700449] 00000000791860da: ffffffffc0c9ab63 (_nv022968rm+0x663/0x12d0 [nvidia])
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.700688] 00000000ed7c42ef: 0000000000000001 (0x1)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.700688] 0000000066a6e046: ffffffffc0c9ab71 (_nv022968rm+0x671/0x12d0 [nvidia])
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.700927] 00000000c00f276a: ffff99e4b65b0008 (0xffff99e4b65b0008)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.700928] 0000000013aa5598: ffff99e4b66c2008 (0xffff99e4b66c2008)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.700928] 0000000095c4cfd0: ffff99e4850c5008 (0xffff99e4850c5008)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.700929] 00000000fbd68c85: 0000000000000000 …
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.700929] 00000000f20748fe: ffff99e4b678a1a8 (0xffff99e4b678a1a8)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.700930] 00000000024899bf: ffffffffc0c5afaf (_nv023439rm+0x9f/0x1b0 [nvidia])
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.701167] 0000000068b3ca29: ffff99e4b678a1a8 (0xffff99e4b678a1a8)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.701168] 000000004aa03bf0: ffff99e4b66c21f8 (0xffff99e4b66c21f8)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.701169] 00000000ba5ceeae: ffff99e4b65b0008 (0xffff99e4b65b0008)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.701169] 000000005d61f382: ffff99e4b6565d83 (0xffff99e4b6565d83)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.701170] 00000000e53efe62: ffff99e4b65b0008 (0xffff99e4b65b0008)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.701170] 00000000db649b9b: ffffffffc0d44830 (_nv026154rm+0x60/0xc0 [nvidia])
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.701408] 00000000aef8a2c8: 000000000000001f (0x1f)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.701408] 0000000038570e8e: 000000000000004a (0x4a)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.701409] 00000000e64d44c2: 0000000000000000 …
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.701409] 00000000ae32394c: ffffffffc0d46353 (_nv010190rm+0x173/0x2c0 [nvidia])
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.701645] 0000000053fb2a4b: ffff99e4b65b0008 (0xffff99e4b65b0008)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.701645] 000000002c9b8990: ffff99e4b678a008 (0xffff99e4b678a008)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.701646] 00000000b66125ac: ffff99e4b678a1a8 (0xffff99e4b678a1a8)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.701646] 000000009639b2ed: ffff99e4b678b8a0 (0xffff99e4b678b8a0)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.701647] 00000000f424d6e2: 0000000000000000 …
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.701647] 0000000086ba471b: ffffffffc0d46722 (_nv026168rm+0xc2/0x1f0 [nvidia])
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.701885] 00000000037b7806: ffff99e4b65b0008 (0xffff99e4b65b0008)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.701885] 000000009f95f9ea: ffff99e4b678a008 (0xffff99e4b678a008)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.701886] 0000000044839dd8: 0000000000000001 (0x1)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.701886] 0000000069ff794e: ffff99e4b65b0008 (0xffff99e4b65b0008)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.701887] 00000000013523f1: ffff99e4b6565e70 (0xffff99e4b6565e70)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.701887] 000000003d283abb: ffffffffc114b6f8 (_nv000652rm+0x118/0x150 [nvidia])
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.702024] 00000000fd05f2c7: ffff99e4b6563000 (0xffff99e4b6563000)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.702024] 000000006f679e66: ffffba21c1df7e70 (0xffffba21c1df7e70)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.702025] 0000000091845514: ffff99e48a20c800 (0xffff99e48a20c800)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.702025] 00000000f956f423: ffff99e48a20c800 (0xffff99e48a20c800)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.702026] 00000000682444cf: ffffffff9c928180 (irq_forced_thread_fn+0x90/0x90)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.702028] 0000000012dd9f56: ffffffffc114bffc (rm_isr_bh+0x1c/0x60 [nvidia])
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.702163] 00000000c4e4489c: ffff99e4b6570000 (0xffff99e4b6570000)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.702163] 0000000031e1910a: ffff99e4888eac40 (0xffff99e4888eac40)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.702164] 0000000001c84717: ffff99e4888eac00 (0xffff99e4888eac00)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.702164] 00000000ade66f08: ffffffffc07ec88f (nvidia_isr_kthread_bh+0x1f/0x40 [nvidia])
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.702261] 000000001ab193a9: ffffba21c1df7e98 (0xffffba21c1df7e98)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.702261] 000000007d500bb2: ffffffff9c9281a8 (irq_thread_fn+0x28/0x60)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.702263] 000000002b7b9e53: ffff99e4b6570000 (0xffff99e4b6570000)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.702264] 000000002f360754: ffff99e4888eac00 (0xffff99e4888eac00)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.702264] 00000000322fd332: ffff99e48a20c800 (0xffff99e48a20c800)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.702265] 0000000021bcada1: ffffba21c1df7f00 (0xffffba21c1df7f00)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.702265] 000000006fdbddcd: ffffffff9c92892d (irq_thread+0xdd/0x190)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.702267] 00000000d08c3f3a: ffff99e4b6570000 (0xffff99e4b6570000)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.702267] 000000003087053e: ffff99e4b6570000 (0xffff99e4b6570000)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.702268] 00000000b99795a9: 0000000000000000 …
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.702268] 0000000019c2f364: ffffffff9c928210 (wake_threads_waitq+0x30/0x30)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.702269] 00000000efa62152: 2ca20746673cdf00 (0x2ca20746673cdf00)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.702270] 000000000b1094c6: ffff99e4b6570000 (0xffff99e4b6570000)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.702270] 00000000dfb75bf3: ffff99e48a2bd500 (0xffff99e48a2bd500)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.702271] 00000000a022119a: ffff99e4a0f18100 (0xffff99e4a0f18100)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.702271] 00000000d03e3bd5: ffffffff9c928850 (irq_thread_check_affinity+0xe0/0xe0)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.702273] 00000000b26fb339: ffff99e4888eac00 (0xffff99e4888eac00)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.702274] 00000000e9114939: ffffba21c1df7f48 (0xffffba21c1df7f48)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.702274] 00000000c22258bb: ffffffff9c8d0d4b (kthread+0x15b/0x1a0)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.702276] 000000001bb457dc: ffffba21c1ba3900 (0xffffba21c1ba3900)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.702277] 000000005f2162e0: ffff99e48a2bd540 (0xffff99e48a2bd540)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.702277] 000000001a40ff48: ffffffff9c8d0bf0 (kthread_park+0x90/0x90)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.702279] 000000003fa90dc9: ffff99e4a0f18100 (0xffff99e4a0f18100)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.702279] 000000002d56c473: 0000000000000000 …
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.702280] 000000003682cd9d: ffffba21c1df7f59 (0xffffba21c1df7f59)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.702280] 0000000031b82430: ffffffff9c80462f (ret_from_fork+0x1f/0x30)
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.702282] 000000001ab42698: 0000000000000000 …
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.702283] ? _nv031682rm+0xab/0xc0 [nvidia]
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.702425] ? _nv029915rm+0xf4/0x120 [nvidia]
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.702551] ? _nv022968rm+0x663/0x12d0 [nvidia]
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.702791] ? _nv022968rm+0x671/0x12d0 [nvidia]
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.703028] ? _nv023439rm+0x9f/0x1b0 [nvidia]
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.703269] ? _nv026154rm+0x60/0xc0 [nvidia]
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.703507] ? _nv010190rm+0x173/0x2c0 [nvidia]
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.703742] ? _nv026168rm+0xc2/0x1f0 [nvidia]
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.703979] ? _nv000652rm+0x118/0x150 [nvidia]
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.704114] ? irq_forced_thread_fn+0x90/0x90
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.704116] ? rm_isr_bh+0x1c/0x60 [nvidia]
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.704250] ? nvidia_isr_kthread_bh+0x1f/0x40 [nvidia]
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.704346] ? irq_thread_fn+0x28/0x60
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.704348] ? irq_thread+0xdd/0x190
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.704350] ? wake_threads_waitq+0x30/0x30
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.704351] ? irq_thread_check_affinity+0xe0/0xe0
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.704353] ? kthread+0x15b/0x1a0
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.704354] ? kthread_park+0x90/0x90
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.704356] ? ret_from_fork+0x1f/0x30
Oct 27 15:42:37 qj00346-ThinkStation-K-C1 kernel: [ 1336.704375] BUG: scheduling while atomic: irq/140-s-nvidi/1223/0x00000000

doubt:

Whether Nvidia driver has corresponding patch support for RT kernel;
In the RT kernel, how to solve the above reported bug by modifying the Nvidia driver driver or kernel code?

Hope to get your reply, thank you