[Fri Apr 30 01:15:28 2021] nvidia-uvm: Loaded the UVM driver, major device number 508. [Fri Apr 30 01:23:13 2021] iwlwifi 0000:45:00.0: Error sending SCAN_REQ_UMAC: time out after 2000ms. [Fri Apr 30 01:23:13 2021] iwlwifi 0000:45:00.0: Current CMD queue read_ptr 244 write_ptr 245 [Fri Apr 30 01:23:13 2021] iwlwifi 0000:45:00.0: Start IWL Error Log Dump: [Fri Apr 30 01:23:13 2021] iwlwifi 0000:45:00.0: Status: 0x00000040, count: 6 [Fri Apr 30 01:23:13 2021] iwlwifi 0000:45:00.0: Loaded firmware version: 48.4fa0041f.0 [Fri Apr 30 01:23:13 2021] iwlwifi 0000:45:00.0: 0x00000084 | NMI_INTERRUPT_UNKNOWN [Fri Apr 30 01:23:13 2021] iwlwifi 0000:45:00.0: 0x000022F0 | trm_hw_status0 [Fri Apr 30 01:23:13 2021] iwlwifi 0000:45:00.0: 0x00000000 | trm_hw_status1 [Fri Apr 30 01:23:13 2021] iwlwifi 0000:45:00.0: 0x004F8E3C | branchlink2 [Fri Apr 30 01:23:13 2021] iwlwifi 0000:45:00.0: 0x004E4FF4 | interruptlink1 [Fri Apr 30 01:23:13 2021] iwlwifi 0000:45:00.0: 0x004E4FF4 | interruptlink2 [Fri Apr 30 01:23:13 2021] iwlwifi 0000:45:00.0: 0x004F4AD2 | data1 [Fri Apr 30 01:23:13 2021] iwlwifi 0000:45:00.0: 0xFF000000 | data2 [Fri Apr 30 01:23:13 2021] iwlwifi 0000:45:00.0: 0xF0000008 | data3 [Fri Apr 30 01:23:13 2021] iwlwifi 0000:45:00.0: 0x00000000 | beacon time [Fri Apr 30 01:23:13 2021] iwlwifi 0000:45:00.0: 0x75A5C565 | tsf low [Fri Apr 30 01:23:13 2021] iwlwifi 0000:45:00.0: 0x00000001 | tsf hi [Fri Apr 30 01:23:13 2021] iwlwifi 0000:45:00.0: 0x00000000 | time gp1 [Fri Apr 30 01:23:13 2021] iwlwifi 0000:45:00.0: 0x75A62645 | time gp2 [Fri Apr 30 01:23:13 2021] iwlwifi 0000:45:00.0: 0x00000001 | uCode revision type [Fri Apr 30 01:23:13 2021] iwlwifi 0000:45:00.0: 0x00000030 | uCode version major [Fri Apr 30 01:23:13 2021] iwlwifi 0000:45:00.0: 0x4FA0041F | uCode version minor [Fri Apr 30 01:23:13 2021] iwlwifi 0000:45:00.0: 0x00000340 | hw version [Fri Apr 30 01:23:13 2021] iwlwifi 0000:45:00.0: 0x00C89000 | board version [Fri Apr 30 01:23:13 2021] iwlwifi 0000:45:00.0: 0x80B5FD04 | hcmd [Fri Apr 30 01:23:13 2021] iwlwifi 0000:45:00.0: 0x00020000 | isr0 [Fri Apr 30 01:23:13 2021] iwlwifi 0000:45:00.0: 0x00000000 | isr1 [Fri Apr 30 01:23:13 2021] iwlwifi 0000:45:00.0: 0x08F84802 | isr2 [Fri Apr 30 01:23:13 2021] iwlwifi 0000:45:00.0: 0x04C1FFCC | isr3 [Fri Apr 30 01:23:13 2021] iwlwifi 0000:45:00.0: 0x00000000 | isr4 [Fri Apr 30 01:23:13 2021] iwlwifi 0000:45:00.0: 0x00F3019C | last cmd Id [Fri Apr 30 01:23:13 2021] iwlwifi 0000:45:00.0: 0x004F4AD2 | wait_event [Fri Apr 30 01:23:13 2021] iwlwifi 0000:45:00.0: 0x00000080 | l2p_control [Fri Apr 30 01:23:13 2021] iwlwifi 0000:45:00.0: 0x00000000 | l2p_duration [Fri Apr 30 01:23:13 2021] iwlwifi 0000:45:00.0: 0x0000003F | l2p_mhvalid [Fri Apr 30 01:23:13 2021] iwlwifi 0000:45:00.0: 0x00000000 | l2p_addr_match [Fri Apr 30 01:23:13 2021] iwlwifi 0000:45:00.0: 0x00000008 | lmpm_pmg_sel [Fri Apr 30 01:23:13 2021] iwlwifi 0000:45:00.0: 0x00000000 | timestamp [Fri Apr 30 01:23:13 2021] iwlwifi 0000:45:00.0: 0x0000187C | flow_handler [Fri Apr 30 01:23:13 2021] iwlwifi 0000:45:00.0: Start IWL Error Log Dump: [Fri Apr 30 01:23:13 2021] iwlwifi 0000:45:00.0: Status: 0x00000040, count: 7 [Fri Apr 30 01:23:13 2021] iwlwifi 0000:45:00.0: 0x20000066 | NMI_INTERRUPT_HOST [Fri Apr 30 01:23:13 2021] iwlwifi 0000:45:00.0: 0x00000000 | umac branchlink1 [Fri Apr 30 01:23:13 2021] iwlwifi 0000:45:00.0: 0xC008CC3C | umac branchlink2 [Fri Apr 30 01:23:13 2021] iwlwifi 0000:45:00.0: 0xC008609C | umac interruptlink1 [Fri Apr 30 01:23:13 2021] iwlwifi 0000:45:00.0: 0x8048D2C2 | umac interruptlink2 [Fri Apr 30 01:23:13 2021] iwlwifi 0000:45:00.0: 0x01000000 | umac data1 [Fri Apr 30 01:23:13 2021] iwlwifi 0000:45:00.0: 0x8048D2C2 | umac data2 [Fri Apr 30 01:23:13 2021] iwlwifi 0000:45:00.0: 0x00000000 | umac data3 [Fri Apr 30 01:23:13 2021] iwlwifi 0000:45:00.0: 0x00000030 | umac major [Fri Apr 30 01:23:13 2021] iwlwifi 0000:45:00.0: 0x4FA0041F | umac minor [Fri Apr 30 01:23:13 2021] iwlwifi 0000:45:00.0: 0x75A6263B | frame pointer [Fri Apr 30 01:23:13 2021] iwlwifi 0000:45:00.0: 0xC0887F1C | stack pointer [Fri Apr 30 01:23:13 2021] iwlwifi 0000:45:00.0: 0x00F4010D | last host cmd [Fri Apr 30 01:23:13 2021] iwlwifi 0000:45:00.0: 0x00010009 | isr status reg [Fri Apr 30 01:23:13 2021] iwlwifi 0000:45:00.0: Fseq Registers: [Fri Apr 30 01:23:13 2021] iwlwifi 0000:45:00.0: 0xE0000101 | FSEQ_ERROR_CODE [Fri Apr 30 01:23:13 2021] iwlwifi 0000:45:00.0: 0x00290006 | FSEQ_TOP_INIT_VERSION [Fri Apr 30 01:23:13 2021] iwlwifi 0000:45:00.0: 0x80050008 | FSEQ_CNVIO_INIT_VERSION [Fri Apr 30 01:23:13 2021] iwlwifi 0000:45:00.0: 0x0000A503 | FSEQ_OTP_VERSION [Fri Apr 30 01:23:13 2021] iwlwifi 0000:45:00.0: 0x80000003 | FSEQ_TOP_CONTENT_VERSION [Fri Apr 30 01:23:13 2021] iwlwifi 0000:45:00.0: 0x4552414E | FSEQ_ALIVE_TOKEN [Fri Apr 30 01:23:13 2021] iwlwifi 0000:45:00.0: 0x00100530 | FSEQ_CNVI_ID [Fri Apr 30 01:23:13 2021] iwlwifi 0000:45:00.0: 0x00000532 | FSEQ_CNVR_ID [Fri Apr 30 01:23:13 2021] iwlwifi 0000:45:00.0: 0x00100530 | CNVI_AUX_MISC_CHIP [Fri Apr 30 01:23:13 2021] iwlwifi 0000:45:00.0: 0x00000532 | CNVR_AUX_MISC_CHIP [Fri Apr 30 01:23:13 2021] iwlwifi 0000:45:00.0: 0x05B0905B | CNVR_SCU_SD_REGS_SD_REG_DIG_DCDC_VTRIM [Fri Apr 30 01:23:13 2021] iwlwifi 0000:45:00.0: 0x0000025B | CNVR_SCU_SD_REGS_SD_REG_ACTIVE_VDIG_MIRROR [Fri Apr 30 01:23:13 2021] iwlwifi 0000:45:00.0: Collecting data: trigger 2 fired. [Fri Apr 30 01:23:13 2021] ieee80211 phy0: Hardware restart was requested [Fri Apr 30 01:23:13 2021] iwlwifi 0000:45:00.0: Scan failed! ret -110 [Fri Apr 30 01:23:13 2021] ------------[ cut here ]------------ [Fri Apr 30 01:23:13 2021] WARNING: CPU: 25 PID: 4562 at /build/linux-hwe-5.4-PtAV0J/linux-hwe-5.4-5.4.0/net/mac80211/scan.c:395 __ieee80211_scan_completed+0x2e6/0x3b0 [mac80211] [Fri Apr 30 01:23:13 2021] Modules linked in: nvidia_uvm(OE) veth xt_nat xt_tcpudp xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype iptable_filter iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 bpfilter br_netfilter bridge stp llc aufs cmac overlay bnep snd_hda_codec_hdmi nvidia_drm(POE) nvidia_modeset(POE) edac_mce_amd snd_usb_audio snd_usbmidi_lib mc kvm_amd kvm iwlmvm mac80211 crct10dif_pclmul crc32_pclmul btusb nls_iso8859_1 btrtl nvidia(POE) libarc4 btbcm ghash_clmulni_intel btintel bluetooth iwlwifi ecdh_generic aesni_intel snd_seq_midi input_leds ecc crypto_simd snd_seq_midi_event cryptd glue_helper cfg80211 wmi_bmof snd_hda_intel snd_rawmidi snd_intel_dspcfg joydev snd_hda_codec nouveau snd_hda_core snd_hwdep snd_seq snd_pcm mxm_wmi video snd_seq_device ccp ttm snd_timer drm_kms_helper snd drm i2c_algo_bit fb_sys_fops syscopyarea sysfillrect sysimgblt k10temp soundcore mac_hid sch_fq_codel parport_pc ppdev lp parport ip_tables x_tables [Fri Apr 30 01:23:13 2021] autofs4 btrfs xor zstd_compress raid6_pq libcrc32c dm_mirror dm_region_hash dm_log hid_generic usbhid hid nvme nvme_core ahci r8169 libahci atlantic realtek i2c_piix4 wmi [Fri Apr 30 01:23:13 2021] CPU: 25 PID: 4562 Comm: kworker/u256:3 Tainted: P OE 5.4.0-72-generic #80~18.04.1-Ubuntu [Fri Apr 30 01:23:13 2021] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./TRX40 Creator, BIOS P1.70 05/29/2020 [Fri Apr 30 01:23:13 2021] Workqueue: phy0 ieee80211_scan_work [mac80211] [Fri Apr 30 01:23:13 2021] RIP: 0010:__ieee80211_scan_completed+0x2e6/0x3b0 [mac80211] [Fri Apr 30 01:23:13 2021] Code: 1d b7 84 09 00 48 85 db 74 1b 48 8b 03 48 8b 7b 08 48 83 c3 18 4c 89 e6 e8 97 c3 72 cc 48 8b 03 48 85 c0 75 e8 e9 dc fe ff ff <0f> 0b 48 83 c4 08 5b 41 5c 41 5d 41 5e 41 5f 5d c3 65 8b 05 c2 b9 [Fri Apr 30 01:23:13 2021] RSP: 0018:fffface9478b3df0 EFLAGS: 00010246 [Fri Apr 30 01:23:13 2021] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000 [Fri Apr 30 01:23:13 2021] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9dcbbbe987a0 [Fri Apr 30 01:23:13 2021] RBP: fffface9478b3e20 R08: ffff9dcbbbe99908 R09: 8080808080808080 [Fri Apr 30 01:23:13 2021] R10: fffface9478b3e08 R11: fefefefefefefeff R12: ffff9dcbbbe987a0 [Fri Apr 30 01:23:13 2021] R13: 0000000000000001 R14: 0000000000000000 R15: 0ffff9dcbcf33410 [Fri Apr 30 01:23:13 2021] FS: 0000000000000000(0000) GS:ffff9dcbfd440000(0000) knlGS:0000000000000000 [Fri Apr 30 01:23:13 2021] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fri Apr 30 01:23:13 2021] CR2: 00007fb1c506a4d0 CR3: 0000001fc56ce000 CR4: 0000000000340ee0 [Fri Apr 30 01:23:13 2021] Call Trace: [Fri Apr 30 01:23:13 2021] ieee80211_scan_work+0x6c/0x4d0 [mac80211] [Fri Apr 30 01:23:13 2021] process_one_work+0x20f/0x400 [Fri Apr 30 01:23:13 2021] worker_thread+0x34/0x410 [Fri Apr 30 01:23:13 2021] kthread+0x121/0x140 [Fri Apr 30 01:23:13 2021] ? process_one_work+0x400/0x400 [Fri Apr 30 01:23:13 2021] ? kthread_park+0x90/0x90 [Fri Apr 30 01:23:13 2021] ret_from_fork+0x22/0x40 [Fri Apr 30 01:23:13 2021] ---[ end trace c3ee37f01c9a84e9 ]--- [Fri Apr 30 01:23:31 2021] watchdog: BUG: soft lockup - CPU#16 stuck for 22s! [python3:5432] [Fri Apr 30 01:23:31 2021] Modules linked in: nvidia_uvm(OE) veth xt_nat xt_tcpudp xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype iptable_filter iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 bpfilter br_netfilter bridge stp llc aufs cmac overlay bnep snd_hda_codec_hdmi nvidia_drm(POE) nvidia_modeset(POE) edac_mce_amd snd_usb_audio snd_usbmidi_lib mc kvm_amd kvm iwlmvm mac80211 crct10dif_pclmul crc32_pclmul btusb nls_iso8859_1 btrtl nvidia(POE) libarc4 btbcm ghash_clmulni_intel btintel bluetooth iwlwifi ecdh_generic aesni_intel snd_seq_midi input_leds ecc crypto_simd snd_seq_midi_event cryptd glue_helper cfg80211 wmi_bmof snd_hda_intel snd_rawmidi snd_intel_dspcfg joydev snd_hda_codec nouveau snd_hda_core snd_hwdep snd_seq snd_pcm mxm_wmi video snd_seq_device ccp ttm snd_timer drm_kms_helper snd drm i2c_algo_bit fb_sys_fops syscopyarea sysfillrect sysimgblt k10temp soundcore mac_hid sch_fq_codel parport_pc ppdev lp parport ip_tables x_tables [Fri Apr 30 01:23:31 2021] autofs4 btrfs xor zstd_compress raid6_pq libcrc32c dm_mirror dm_region_hash dm_log hid_generic usbhid hid nvme nvme_core ahci r8169 libahci atlantic realtek i2c_piix4 wmi [Fri Apr 30 01:23:31 2021] CPU: 16 PID: 5432 Comm: python3 Tainted: P W OE 5.4.0-72-generic #80~18.04.1-Ubuntu [Fri Apr 30 01:23:31 2021] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./TRX40 Creator, BIOS P1.70 05/29/2020 [Fri Apr 30 01:23:31 2021] RIP: 0010:iommu_unmap_page+0x56/0x100 [Fri Apr 30 01:23:31 2021] Code: 00 00 48 8d 42 ff 48 89 d3 48 85 d0 0f 85 a9 00 00 00 4c 8d af 94 00 00 00 4c 8d a7 98 00 00 00 49 89 f6 45 31 ff 48 8d 4d c8 <4c> 89 e6 4c 89 f2 4c 89 ef e8 fc fc ff ff 48 85 c0 48 89 c6 74 3c [Fri Apr 30 01:23:31 2021] RSP: 0018:fffface947c0f930 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13 [Fri Apr 30 01:23:31 2021] RAX: 0000000000000fff RBX: 0000000000001000 RCX: fffface947c0f930 [Fri Apr 30 01:23:31 2021] RDX: 0000000000001000 RSI: 00008bf20fb51000 RDI: ffff9dcbe418b000 [Fri Apr 30 01:23:31 2021] RBP: fffface947c0f968 R08: fffface947c0f930 R09: 0000000000000004 [Fri Apr 30 01:23:31 2021] R10: 0000000000000000 R11: 000000000000001b R12: ffff9dcbe418b098 [Fri Apr 30 01:23:31 2021] R13: ffff9dcbe418b094 R14: 00008bf20fb51000 R15: 0000000000000000 [Fri Apr 30 01:23:31 2021] FS: 00007f9e29fff700(0000) GS:ffff9dcbfd200000(0000) knlGS:0000000000000000 [Fri Apr 30 01:23:31 2021] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fri Apr 30 01:23:31 2021] CR2: 00007fb3cf8144d0 CR3: 000000108660a000 CR4: 0000000000340ee0 [Fri Apr 30 01:23:31 2021] Call Trace: [Fri Apr 30 01:23:31 2021] __unmap_single.isra.27+0x66/0x130 [Fri Apr 30 01:23:31 2021] unmap_sg+0x5f/0x70 [Fri Apr 30 01:23:31 2021] nv_unmap_dma_map_scatterlist+0x57/0xa0 [nvidia] [Fri Apr 30 01:23:31 2021] nv_dma_unmap_pages+0x56/0x100 [nvidia] [Fri Apr 30 01:23:31 2021] nv_dma_unmap_alloc+0x16/0x30 [nvidia] [Fri Apr 30 01:23:31 2021] _nv032962rm+0xc1/0x1b0 [nvidia] [Fri Apr 30 01:23:31 2021] WARNING: kernel stack frame pointer at 00000000b7038048 in python3:5432 has bad value 00000000dd821cf2 [Fri Apr 30 01:23:31 2021] unwind stack type:0 next_sp:0000000000000000 mask:0x2 graph_idx:0 [Fri Apr 30 01:23:31 2021] 0000000069bf7ccc: fffface947c0f9a8 (0xfffface947c0f9a8) [Fri Apr 30 01:23:31 2021] 0000000043edac86: ffffffff8ccde856 (__unmap_single.isra.27+0x66/0x130) [Fri Apr 30 01:23:31 2021] 00000000ac9c71ee: fffface947c0f9a8 (0xfffface947c0f9a8) [Fri Apr 30 01:23:31 2021] 0000000080055af4: ffff9dcbe418b000 (0xffff9dcbe418b000) [Fri Apr 30 01:23:31 2021] 00000000c84967a1: 00007ffb00000000 (0x7ffb00000000) [Fri Apr 30 01:23:31 2021] 000000000444a76b: ffff9dcbf42ff0b0 (0xffff9dcbf42ff0b0) [Fri Apr 30 01:23:31 2021] 00000000ed1e8db4: ffff9dc7ec217000 (0xffff9dc7ec217000) [Fri Apr 30 01:23:31 2021] 000000000067cdca: 000000000000040c (0x40c) [Fri Apr 30 01:23:31 2021] 000000001e2ea068: fffface947c0f9e0 (0xfffface947c0f9e0) [Fri Apr 30 01:23:31 2021] 0000000078e25588: ffffffff8ccded5f (unmap_sg+0x5f/0x70) [Fri Apr 30 01:23:31 2021] 000000007748936a: 0000000000000000 ... [Fri Apr 30 01:23:31 2021] 000000004a5c60d7: ffff9dbd6fc96648 (0xffff9dbd6fc96648) [Fri Apr 30 01:23:31 2021] 00000000b262069e: ffff9dcbf5ab86c8 (0xffff9dcbf5ab86c8) [Fri Apr 30 01:23:31 2021] 000000001f9e2e06: ffff9dbd6fc96648 (0xffff9dbd6fc96648) [Fri Apr 30 01:23:31 2021] 00000000c7bedb37: fffface947c91010 (0xfffface947c91010) [Fri Apr 30 01:23:31 2021] 00000000c6caa441: fffface947c0fa00 (0xfffface947c0fa00) [Fri Apr 30 01:23:31 2021] 00000000a1556e26: ffffffffc850ca97 (nv_unmap_dma_map_scatterlist+0x57/0xa0 [nvidia]) [Fri Apr 30 01:23:31 2021] 0000000066a13f91: ffff9dc4858fdf38 (0xffff9dc4858fdf38) [Fri Apr 30 01:23:31 2021] 00000000ca5c413d: 0000000000400000 (0x400000) [Fri Apr 30 01:23:31 2021] 00000000f5a80299: fffface947c0fa30 (0xfffface947c0fa30) [Fri Apr 30 01:23:31 2021] 00000000e67694a6: ffffffffc850d286 (nv_dma_unmap_pages+0x56/0x100 [nvidia]) [Fri Apr 30 01:23:31 2021] 000000001cb0d9ba: ffff9dc4858fdf38 (0xffff9dc4858fdf38) [Fri Apr 30 01:23:31 2021] 000000002f14b4d2: ffff9dcbf5ab8000 (0xffff9dcbf5ab8000) [Fri Apr 30 01:23:31 2021] 0000000069424864: ffffffffca3b1980 (_nv020292rm+0x878/0xfffffffffead8ef8 [nvidia]) [Fri Apr 30 01:23:31 2021] 00000000fe21e051: fffface976849008 (0xfffface976849008) [Fri Apr 30 01:23:31 2021] 00000000573ff102: fffface947c0fa50 (0xfffface947c0fa50) [Fri Apr 30 01:23:31 2021] 000000008a749e3d: ffffffffc850d5d6 (nv_dma_unmap_alloc+0x16/0x30 [nvidia]) [Fri Apr 30 01:23:31 2021] 00000000fbf9aa29: ffff9dcbb8834008 (0xffff9dcbb8834008) [Fri Apr 30 01:23:31 2021] 00000000df957855: fffface9869cd008 (0xfffface9869cd008) [Fri Apr 30 01:23:31 2021] 00000000b7038048: ffff9dc4858fdf30 (0xffff9dc4858fdf30) [Fri Apr 30 01:23:31 2021] 00000000d1c2481f: ffffffffc8dd1d51 (_nv032962rm+0xc1/0x1b0 [nvidia]) [Fri Apr 30 01:23:31 2021] 00000000288bfe86: fffface9869cd008 (0xfffface9869cd008) [Fri Apr 30 01:23:31 2021] 00000000ff7abb8b: ffff9dcbb72a1c08 (0xffff9dcbb72a1c08) [Fri Apr 30 01:23:31 2021] 000000003e261b27: fffface9869cd008 (0xfffface9869cd008) [Fri Apr 30 01:23:31 2021] 00000000d29621e2: ffffffffc858dfdb (_nv027787rm+0x9b/0xc0 [nvidia]) [Fri Apr 30 01:23:31 2021] 000000009b2756bb: 0000000000000000 ... [Fri Apr 30 01:23:31 2021] 000000004f8eea6d: ffff9dcb854c3c08 (0xffff9dcb854c3c08) [Fri Apr 30 01:23:31 2021] 00000000fba9f0ea: ffffffffca3b1980 (_nv020292rm+0x878/0xfffffffffead8ef8 [nvidia]) [Fri Apr 30 01:23:31 2021] 00000000488f09e7: 00000000caf00e16 (0xcaf00e16) [Fri Apr 30 01:23:31 2021] 00000000c9ffc1da: fffface947c91010 (0xfffface947c91010) [Fri Apr 30 01:23:31 2021] 000000004b46812b: ffffffffc85ca614 (_nv030989rm+0x124/0x130 [nvidia]) [Fri Apr 30 01:23:31 2021] 0000000042e51a53: ffff9dcafce20cc0 (0xffff9dcafce20cc0) [Fri Apr 30 01:23:31 2021] 00000000c8c10315: ffff9dc4858fb000 (0xffff9dc4858fb000) [Fri Apr 30 01:23:31 2021] 000000004fa4c108: ffff9dcb854c3c08 (0xffff9dcb854c3c08) [Fri Apr 30 01:23:31 2021] 000000005cbbed36: fffface947c0fb18 (0xfffface947c0fb18) [Fri Apr 30 01:23:31 2021] 00000000e0482500: ffff9dcafce20e20 (0xffff9dcafce20e20) [Fri Apr 30 01:23:31 2021] 00000000b3a6789c: ffffffffc8dda36c (rm_gpu_ops_free_duped_handle+0x1c/0x60 [nvidia]) [Fri Apr 30 01:23:31 2021] 00000000e3293010: ffff9dc4858fb000 (0xffff9dc4858fb000) [Fri Apr 30 01:23:31 2021] 00000000d79c0e07: ffff9dcb854c3c08 (0xffff9dcb854c3c08) [Fri Apr 30 01:23:31 2021] 000000009f6d8c4a: 00000000caf00e16 (0xcaf00e16) [Fri Apr 30 01:23:31 2021] 0000000070eff960: ffffffffc8518cfa (nvUvmInterfaceFreeDupedHandle+0x2a/0x40 [nvidia]) [Fri Apr 30 01:23:31 2021] 00000000f9fa4dc8: ffff9dcafce20cc0 (0xffff9dcafce20cc0) [Fri Apr 30 01:23:31 2021] 00000000d13fd17e: ffff9dcbd7748a80 (0xffff9dcbd7748a80) [Fri Apr 30 01:23:31 2021] 000000007b420adc: dead000000000100 (0xdead000000000100) [Fri Apr 30 01:23:31 2021] 00000000020425c6: fffface947c0fb38 (0xfffface947c0fb38) [Fri Apr 30 01:23:31 2021] 00000000e873d43c: ffffffffc111f5ec (uvm_ext_gpu_map_free_internal+0x5c/0x70 [nvidia_uvm]) [Fri Apr 30 01:23:31 2021] 00000000991d9594: fffface947c0fbc0 (0xfffface947c0fbc0) [Fri Apr 30 01:23:31 2021] 00000000a3ac1c58: dead000000000122 (0xdead000000000122) [Fri Apr 30 01:23:31 2021] 00000000bf721194: fffface947c0fb48 (0xfffface947c0fb48) [Fri Apr 30 01:23:31 2021] 00000000b1098fb9: ffffffffc112018e (uvm_ext_gpu_map_free+0xe/0x20 [nvidia_uvm]) [Fri Apr 30 01:23:31 2021] 000000001792e323: fffface947c0fb98 (0xfffface947c0fb98) [Fri Apr 30 01:23:31 2021] 00000000d704b266: ffffffffc10e6d60 (uvm_deferred_free_object_list+0x70/0x130 [nvidia_uvm]) [Fri Apr 30 01:23:31 2021] 0000000091bd88c9: ffff9dcadc8e9400 (0xffff9dcadc8e9400) [Fri Apr 30 01:23:31 2021] 00000000e32a2b61: 000000000000001e (0x1e) [Fri Apr 30 01:23:31 2021] 000000005a15f847: ea5409fb16e14200 (0xea5409fb16e14200) [Fri Apr 30 01:23:31 2021] 00000000f237af18: ffff9dcb863a22b0 (0xffff9dcb863a22b0) [Fri Apr 30 01:23:31 2021] 00000000d3d612a3: ffff9dcb863a2000 (0xffff9dcb863a2000) [Fri Apr 30 01:23:31 2021] 00000000ca703d3a: ffffffffc11845a0 (g_uvm_tools_cdev+0x80/0xfffffffffffd7ae0 [nvidia_uvm]) [Fri Apr 30 01:23:31 2021] 000000005486e411: fffface947c91010 (0xfffface947c91010) [Fri Apr 30 01:23:31 2021] 00000000016f2b36: ffff9dcb863a2150 (0xffff9dcb863a2150) [Fri Apr 30 01:23:31 2021] 000000003cb62248: fffface947c0fc28 (0xfffface947c0fc28) [Fri Apr 30 01:23:31 2021] 0000000011588eea: ffffffffc10e7169 (uvm_va_space_destroy+0x349/0x480 [nvidia_uvm]) [Fri Apr 30 01:23:31 2021] 00000000d6b1daad: ffff9dcb863a2158 (0xffff9dcb863a2158) [Fri Apr 30 01:23:31 2021] 00000000c0a8512e: ffff9dcb863a2110 (0xffff9dcb863a2110) [Fri Apr 30 01:23:31 2021] 000000008898f5d1: ffff9dcb863a2138 (0xffff9dcb863a2138) [Fri Apr 30 01:23:31 2021] 000000001bd1a177: ffff9dcafce20e28 (0xffff9dcafce20e28) [Fri Apr 30 01:23:31 2021] 000000009f4f4842: ffff9dcac349b6f8 (0xffff9dcac349b6f8) [Fri Apr 30 01:23:31 2021] 00000000840041a8: 0000000002020202 (0x2020202) [Fri Apr 30 01:23:31 2021] 00000000730dc739: 0000000000000000 ... [Fri Apr 30 01:23:31 2021] 000000000adcf998: ea5409fb16e14200 (0xea5409fb16e14200) [Fri Apr 30 01:23:31 2021] 00000000adc1a9a4: ffff9dcb863a2000 (0xffff9dcb863a2000) [Fri Apr 30 01:23:31 2021] 000000009b15b3b8: 0000000000000001 (0x1) [Fri Apr 30 01:23:31 2021] 000000003928aa0f: ffff9dcadf06f960 (0xffff9dcadf06f960) [Fri Apr 30 01:23:31 2021] 00000000d32e496b: ffff9dcbd595c520 (0xffff9dcbd595c520) [Fri Apr 30 01:23:31 2021] 000000005b8790ff: ffff9dcb0156acc0 (0xffff9dcb0156acc0) [Fri Apr 30 01:23:31 2021] 000000002c38f399: fffface947c0fc48 (0xfffface947c0fc48) [Fri Apr 30 01:23:31 2021] 0000000062eebcbd: ffffffffc10d4f9c (uvm_release.isra.10+0x7c/0x90 [nvidia_uvm]) [Fri Apr 30 01:23:31 2021] 0000000082899705: ffff9dcafdfbd700 (0xffff9dcafdfbd700) [Fri Apr 30 01:23:31 2021] 00000000d63ff977: 0000000000000001 (0x1) [Fri Apr 30 01:23:31 2021] 000000009bfc72c7: fffface947c0fca0 (0xfffface947c0fca0) [Fri Apr 30 01:23:31 2021] 000000005245bc5b: ffffffffc10d508d (uvm_release_entry+0x6d/0xa0 [nvidia_uvm]) [Fri Apr 30 01:23:31 2021] 0000000046668a4f: 0000000000000000 ... [Fri Apr 30 01:23:31 2021] 0000000068b551dc: ffff9dcade5d0000 (0xffff9dcade5d0000) [Fri Apr 30 01:23:31 2021] 000000006965fc3a: 0000000000000001 (0x1) [Fri Apr 30 01:23:31 2021] 0000000013d81e02: fffface947c0fc70 (0xfffface947c0fc70) [Fri Apr 30 01:23:31 2021] 000000000624579a: ffff9dcafdfbd700 (0xffff9dcafdfbd700) [Fri Apr 30 01:23:31 2021] 000000003eccf655: 000000000008001f (0x8001f) [Fri Apr 30 01:23:31 2021] 000000007dccf950: ea5409fb16e14200 (0xea5409fb16e14200) [Fri Apr 30 01:23:31 2021] 00000000067546bd: ffff9dcafdfbd700 (0xffff9dcafdfbd700) [Fri Apr 30 01:23:31 2021] 00000000da03ef60: 000000000008001f (0x8001f) [Fri Apr 30 01:23:31 2021] 00000000b1de7a83: fffface947c0fcf0 (0xfffface947c0fcf0) [Fri Apr 30 01:23:31 2021] 00000000c3fa48eb: ffffffff8c8ddb16 (__fput+0xc6/0x260) [Fri Apr 30 01:23:31 2021] 000000009ef448d1: ffff9dcafdfbd710 (0xffff9dcafdfbd710) [Fri Apr 30 01:23:31 2021] 00000000c5d4434e: ffff9dcadf06f960 (0xffff9dcadf06f960) [Fri Apr 30 01:23:31 2021] 0000000065715619: 00000008179fc644 (0x8179fc644) [Fri Apr 30 01:23:31 2021] 000000007d0a1c3f: ffff9dcafdfbdc00 (0xffff9dcafdfbdc00) [Fri Apr 30 01:23:31 2021] 00000000504406d9: ffff9dcade5d0000 (0xffff9dcade5d0000) [Fri Apr 30 01:23:31 2021] 00000000f7321380: ffff9dcade5d0b28 (0xffff9dcade5d0b28) [Fri Apr 30 01:23:31 2021] 00000000e1d97a02: ffffffff8e188f60 (__key.52317+0x8/0x8) [Fri Apr 30 01:23:31 2021] 00000000c106c615: 0000000000000000 ... [Fri Apr 30 01:23:31 2021] 0000000045a16b01: fffface947c0fd00 (0xfffface947c0fd00) [Fri Apr 30 01:23:31 2021] 000000000049d54f: ffffffff8c8ddd1e (____fput+0xe/0x10) [Fri Apr 30 01:23:31 2021] 00000000490dc17d: fffface947c0fd40 (0xfffface947c0fd40) [Fri Apr 30 01:23:31 2021] 000000002b778fcc: ffffffff8c6c482d (task_work_run+0x9d/0xc0) [Fri Apr 30 01:23:31 2021] 00000000b0660563: ffff9dcade5d0b64 (0xffff9dcade5d0b64) [Fri Apr 30 01:23:31 2021] 000000002817c9a9: ffff9dcade5d0000 (0xffff9dcade5d0000) [Fri Apr 30 01:23:31 2021] 00000000c90741b1: ffff9dcbbc476e80 (0xffff9dcbbc476e80) [Fri Apr 30 01:23:31 2021] 00000000dcbe47a9: ffff9dcade5d0001 (0xffff9dcade5d0001) [Fri Apr 30 01:23:31 2021] 000000009ba6b484: ffff9dcade5d0000 (0xffff9dcade5d0000) [Fri Apr 30 01:23:31 2021] 0000000038340c9d: ffff9dcbbc476ef8 (0xffff9dcbbc476ef8) [Fri Apr 30 01:23:31 2021] 00000000f663cdef: fffface947c0fdb8 (0xfffface947c0fdb8) [Fri Apr 30 01:23:31 2021] 000000001d40dda0: ffffffff8c6a4ed9 (do_exit+0x389/0xb90) [Fri Apr 30 01:23:31 2021] 00000000aacc6b32: 0000000000000000 ... [Fri Apr 30 01:23:31 2021] 00000000a2336631: ea5409fb16e14200 (0xea5409fb16e14200) [Fri Apr 30 01:23:31 2021] 00000000328707d2: ffff9dcade5d0b60 (0xffff9dcade5d0b60) [Fri Apr 30 01:23:31 2021] 00000000274d4b21: ffff9dcade70c5c0 (0xffff9dcade70c5c0) [Fri Apr 30 01:23:31 2021] 0000000069b0a234: 0000000000000001 (0x1) [Fri Apr 30 01:23:31 2021] 00000000745b7242: ffff9dcbf4bbbf44 (0xffff9dcbf4bbbf44) [Fri Apr 30 01:23:31 2021] 00000000c4887071: 0000000000000001 (0x1) [Fri Apr 30 01:23:31 2021] 000000006b46a596: ea5409fb16e14200 (0xea5409fb16e14200) [Fri Apr 30 01:23:31 2021] 000000000959775c: 0000000000000100 (0x100) [Fri Apr 30 01:23:31 2021] 0000000034c17fca: ffff9dcb07184800 (0xffff9dcb07184800) [Fri Apr 30 01:23:31 2021] 00000000a7e4b9b6: ffff9dcade5d0000 (0xffff9dcade5d0000) [Fri Apr 30 01:23:31 2021] 000000008386626a: 00000000400004d8 (0x400004d8) [Fri Apr 30 01:23:31 2021] 0000000050667310: fffface947c0fe78 (0xfffface947c0fe78) [Fri Apr 30 01:23:31 2021] 000000003f5692fd: fffface947c0fde8 (0xfffface947c0fde8) [Fri Apr 30 01:23:31 2021] 000000001f9495ce: ffffffff8c6a5783 (do_group_exit+0x43/0xa0) [Fri Apr 30 01:23:31 2021] 00000000d6223a0b: 0000000000000008 (0x8) [Fri Apr 30 01:23:31 2021] 0000000099981326: 0000000000000009 (0x9) [Fri Apr 30 01:23:31 2021] 000000000eaf4fb4: fffface947c0fe98 (0xfffface947c0fe98) [Fri Apr 30 01:23:31 2021] 00000000857589ee: 00000000400004d8 (0x400004d8) [Fri Apr 30 01:23:31 2021] 000000000eee3212: fffface947c0fe48 (0xfffface947c0fe48) [Fri Apr 30 01:23:31 2021] 00000000c9910e62: ffffffff8c6b34bf (get_signal+0x14f/0x860) [Fri Apr 30 01:23:31 2021] 0000000034e32bf7: ea5409fb16e14200 (0xea5409fb16e14200) [Fri Apr 30 01:23:31 2021] 00000000273a74b6: 0000000039d7c1d8 (0x39d7c1d8) [Fri Apr 30 01:23:31 2021] 0000000024c1fe7e: 0000000000000000 ... [Fri Apr 30 01:23:31 2021] 000000009a9f28a5: ffff9dcb07184800 (0xffff9dcb07184800) [Fri Apr 30 01:23:31 2021] 000000006c4e6145: ffff9dcb062b7380 (0xffff9dcb062b7380) [Fri Apr 30 01:23:31 2021] 0000000004ca56ed: fffface947c0ff58 (0xfffface947c0ff58) [Fri Apr 30 01:23:31 2021] 00000000bc0a1a7a: fffface947c0ff58 (0xfffface947c0ff58) [Fri Apr 30 01:23:31 2021] 0000000056e1af7c: 0000000000000000 ... [Fri Apr 30 01:23:31 2021] 00000000d0fd3461: fffface947c0ff00 (0xfffface947c0ff00) [Fri Apr 30 01:23:31 2021] 00000000ce3b6b5b: ffffffff8c630854 (do_signal+0x34/0x6d0) [Fri Apr 30 01:23:31 2021] 000000001713b625: ffffffff8d2001a4 (__switch_to_asm+0x34/0x70) [Fri Apr 30 01:23:31 2021] 000000006ca4de16: ffff9dcbfd82adc0 (0xffff9dcbfd82adc0) [Fri Apr 30 01:23:31 2021] 00000000c480fee0: ffff9dcade5d0000 (0xffff9dcade5d0000) [Fri Apr 30 01:23:31 2021] 0000000076c758ec: ffff9dcbd5961740 (0xffff9dcbd5961740) [Fri Apr 30 01:23:31 2021] 000000005fb00b8a: ea5409fb16e14200 (0xea5409fb16e14200) [Fri Apr 30 01:23:31 2021] 000000009d41a831: 0000000000000000 ... [Fri Apr 30 01:23:31 2021] 0000000060023ec0: 0000000000000009 (0x9) [Fri Apr 30 01:23:31 2021] 000000009d65b262: 0000000039d7c1d8 (0x39d7c1d8) [Fri Apr 30 01:23:31 2021] 000000008da3dc0b: fffface947c0ff28 (0xfffface947c0ff28) [Fri Apr 30 01:23:31 2021] 000000006586e5ec: ffffffff8c74bbcf (__x64_sys_futex+0x13f/0x190) [Fri Apr 30 01:23:31 2021] 00000000aa9a63ce: 00000000ffffffff (0xffffffff) [Fri Apr 30 01:23:31 2021] 00000000bd192324: 0000000000000000 ... [Fri Apr 30 01:23:31 2021] 0000000000fa59fd: ea5409fb16e14200 (0xea5409fb16e14200) [Fri Apr 30 01:23:31 2021] 0000000032007783: 0000000000004004 (0x4004) [Fri Apr 30 01:23:31 2021] 00000000f88470e8: fffface947c0ff58 (0xfffface947c0ff58) [Fri Apr 30 01:23:31 2021] 000000002fbe8473: 0000000000000000 ... [Fri Apr 30 01:23:31 2021] 0000000009b5ee6a: fffface947c0ff28 (0xfffface947c0ff28) [Fri Apr 30 01:23:31 2021] 000000002bfe40b5: ffffffff8c603e50 (exit_to_usermode_loop+0x90/0x130) [Fri Apr 30 01:23:31 2021] 000000008a4562b6: ffff9dcade5d0000 (0xffff9dcade5d0000) [Fri Apr 30 01:23:31 2021] 0000000092c324c2: fffface947c0ff58 (0xfffface947c0ff58) [Fri Apr 30 01:23:31 2021] 00000000685b38f7: 0000000000000000 ... [Fri Apr 30 01:23:31 2021] 00000000b81a58e2: fffface947c0ff48 (0xfffface947c0ff48) [Fri Apr 30 01:23:31 2021] 00000000f7383774: ffffffff8c604320 (do_syscall_64+0x170/0x190) [Fri Apr 30 01:23:31 2021] 00000000b67b7537: 0000000000000000 ... [Fri Apr 30 01:23:31 2021] 00000000cbba2b77: ffffffff8d20008c (entry_SYSCALL_64_after_hwframe+0x44/0xa9) [Fri Apr 30 01:23:31 2021] 000000008be18d02: 0000000000083c1c (0x83c1c) [Fri Apr 30 01:23:31 2021] 00000000b580a6fc: 0000000039d7c188 (0x39d7c188) [Fri Apr 30 01:23:31 2021] 000000009f66702d: 0000000000000000 ... [Fri Apr 30 01:23:31 2021] 000000001c60eb12: 0000000039d7c1d8 (0x39d7c1d8) [Fri Apr 30 01:23:31 2021] 00000000775275e5: 0000000039d7c1d4 (0x39d7c1d4) [Fri Apr 30 01:23:31 2021] 00000000cb058112: 0000000039d7c1b0 (0x39d7c1b0) [Fri Apr 30 01:23:31 2021] 00000000a3f73e2c: 0000000000000246 (0x246) [Fri Apr 30 01:23:31 2021] 00000000878beed6: 0000000000000000 ... [Fri Apr 30 01:23:31 2021] 0000000065051273: fffffffffffffe00 (0xfffffffffffffe00) [Fri Apr 30 01:23:31 2021] 0000000055e3c5a4: 00007fa19f40cad3 (0x7fa19f40cad3) [Fri Apr 30 01:23:31 2021] 00000000e685e452: 0000000000000000 ... [Fri Apr 30 01:23:31 2021] 000000001788153d: 0000000000000080 (0x80) [Fri Apr 30 01:23:31 2021] 0000000039d7c84e: 0000000039d7c1d8 (0x39d7c1d8) [Fri Apr 30 01:23:31 2021] 00000000b645b8e9: 00000000000000ca (0xca) [Fri Apr 30 01:23:31 2021] 0000000017aa7be4: 00007fa19f40cad3 (0x7fa19f40cad3) [Fri Apr 30 01:23:31 2021] 000000006a4b3aa2: 0000000000000033 (0x33) [Fri Apr 30 01:23:31 2021] 0000000024bcdfaa: 0000000000000246 (0x246) [Fri Apr 30 01:23:31 2021] 00000000d7c5f90f: 00007f9e29ffed00 (0x7f9e29ffed00) [Fri Apr 30 01:23:31 2021] 000000000f018d22: 000000000000002b (0x2b) [Fri Apr 30 01:23:31 2021] ? _nv027787rm+0x9b/0xc0 [nvidia] [Fri Apr 30 01:23:31 2021] ? _nv030989rm+0x124/0x130 [nvidia] [Fri Apr 30 01:23:31 2021] ? rm_gpu_ops_free_duped_handle+0x1c/0x60 [nvidia] [Fri Apr 30 01:23:31 2021] ? nvUvmInterfaceFreeDupedHandle+0x2a/0x40 [nvidia] [Fri Apr 30 01:23:31 2021] ? uvm_ext_gpu_map_free_internal+0x5c/0x70 [nvidia_uvm] [Fri Apr 30 01:23:31 2021] ? uvm_ext_gpu_map_free+0xe/0x20 [nvidia_uvm] [Fri Apr 30 01:23:31 2021] ? uvm_deferred_free_object_list+0x70/0x130 [nvidia_uvm] [Fri Apr 30 01:23:31 2021] ? uvm_va_space_destroy+0x349/0x480 [nvidia_uvm] [Fri Apr 30 01:23:31 2021] ? uvm_release.isra.10+0x7c/0x90 [nvidia_uvm] [Fri Apr 30 01:23:31 2021] ? uvm_release_entry+0x6d/0xa0 [nvidia_uvm] [Fri Apr 30 01:23:31 2021] ? __fput+0xc6/0x260 [Fri Apr 30 01:23:31 2021] ? ____fput+0xe/0x10 [Fri Apr 30 01:23:31 2021] ? task_work_run+0x9d/0xc0 [Fri Apr 30 01:23:31 2021] ? do_exit+0x389/0xb90 [Fri Apr 30 01:23:31 2021] ? do_group_exit+0x43/0xa0 [Fri Apr 30 01:23:31 2021] ? get_signal+0x14f/0x860 [Fri Apr 30 01:23:31 2021] ? do_signal+0x34/0x6d0 [Fri Apr 30 01:23:31 2021] ? __switch_to_asm+0x34/0x70 [Fri Apr 30 01:23:31 2021] ? __x64_sys_futex+0x13f/0x190 [Fri Apr 30 01:23:31 2021] ? exit_to_usermode_loop+0x90/0x130 [Fri Apr 30 01:23:31 2021] ? do_syscall_64+0x170/0x190 [Fri Apr 30 01:23:31 2021] ? entry_SYSCALL_64_after_hwframe+0x44/0xa9 [Fri Apr 30 01:23:59 2021] watchdog: BUG: soft lockup - CPU#16 stuck for 22s! [python3:5432] [Fri Apr 30 01:23:59 2021] Modules linked in: nvidia_uvm(OE) veth xt_nat xt_tcpudp xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype iptable_filter iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 bpfilter br_netfilter bridge stp llc aufs cmac overlay bnep snd_hda_codec_hdmi nvidia_drm(POE) nvidia_modeset(POE) edac_mce_amd snd_usb_audio snd_usbmidi_lib mc kvm_amd kvm iwlmvm mac80211 crct10dif_pclmul crc32_pclmul btusb nls_iso8859_1 btrtl nvidia(POE) libarc4 btbcm ghash_clmulni_intel btintel bluetooth iwlwifi ecdh_generic aesni_intel snd_seq_midi input_leds ecc crypto_simd snd_seq_midi_event cryptd glue_helper cfg80211 wmi_bmof snd_hda_intel snd_rawmidi snd_intel_dspcfg joydev snd_hda_codec nouveau snd_hda_core snd_hwdep snd_seq snd_pcm mxm_wmi video snd_seq_device ccp ttm snd_timer drm_kms_helper snd drm i2c_algo_bit fb_sys_fops syscopyarea sysfillrect sysimgblt k10temp soundcore mac_hid sch_fq_codel parport_pc ppdev lp parport ip_tables x_tables [Fri Apr 30 01:23:59 2021] autofs4 btrfs xor zstd_compress raid6_pq libcrc32c dm_mirror dm_region_hash dm_log hid_generic usbhid hid nvme nvme_core ahci r8169 libahci atlantic realtek i2c_piix4 wmi [Fri Apr 30 01:23:59 2021] CPU: 16 PID: 5432 Comm: python3 Tainted: P W OEL 5.4.0-72-generic #80~18.04.1-Ubuntu [Fri Apr 30 01:23:59 2021] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./TRX40 Creator, BIOS P1.70 05/29/2020 [Fri Apr 30 01:23:59 2021] RIP: 0010:iommu_unmap_page+0xb8/0x100 [Fri Apr 30 01:23:59 2021] Code: 00 00 48 29 c1 31 c0 48 d3 e2 48 c7 04 c6 00 00 00 00 48 83 c0 01 39 c2 7f f0 48 8b 45 c8 48 89 c2 49 01 c7 48 f7 da 4c 21 f2 <4c> 39 fb 4c 8d 34 10 77 91 4d 85 ff 75 23 48 8b 7d d0 65 48 33 3c [Fri Apr 30 01:23:59 2021] RSP: 0018:fffface947c0f930 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [Fri Apr 30 01:23:59 2021] RAX: 0000008000000000 RBX: 0000000000001000 RCX: 0000000000000027 [Fri Apr 30 01:23:59 2021] RDX: 00008a8000000000 RSI: 0000000000000000 RDI: 0000000000000000 [Fri Apr 30 01:23:59 2021] RBP: fffface947c0f968 R08: fffface947c0f930 R09: 0000000000000004 [Fri Apr 30 01:23:59 2021] R10: 0000000000000000 R11: 000000000000001b R12: ffff9dcbe418b098 [Fri Apr 30 01:23:59 2021] R13: ffff9dcbe418b094 R14: 00008a8cb99db000 R15: 0000008000000000 [Fri Apr 30 01:23:59 2021] FS: 00007f9e29fff700(0000) GS:ffff9dcbfd200000(0000) knlGS:0000000000000000 [Fri Apr 30 01:23:59 2021] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fri Apr 30 01:23:59 2021] CR2: 00007fb3cf8144d0 CR3: 000000108660a000 CR4: 0000000000340ee0 [Fri Apr 30 01:23:59 2021] Call Trace: [Fri Apr 30 01:23:59 2021] __unmap_single.isra.27+0x66/0x130 [Fri Apr 30 01:23:59 2021] unmap_sg+0x5f/0x70 [Fri Apr 30 01:23:59 2021] nv_unmap_dma_map_scatterlist+0x57/0xa0 [nvidia] [Fri Apr 30 01:23:59 2021] nv_dma_unmap_pages+0x56/0x100 [nvidia] [Fri Apr 30 01:23:59 2021] nv_dma_unmap_alloc+0x16/0x30 [nvidia] [Fri Apr 30 01:23:59 2021] _nv032962rm+0xc1/0x1b0 [nvidia] [Fri Apr 30 01:23:59 2021] ? _nv027787rm+0x9b/0xc0 [nvidia] [Fri Apr 30 01:23:59 2021] ? _nv030989rm+0x124/0x130 [nvidia] [Fri Apr 30 01:23:59 2021] ? rm_gpu_ops_free_duped_handle+0x1c/0x60 [nvidia] [Fri Apr 30 01:23:59 2021] ? nvUvmInterfaceFreeDupedHandle+0x2a/0x40 [nvidia] [Fri Apr 30 01:23:59 2021] ? uvm_ext_gpu_map_free_internal+0x5c/0x70 [nvidia_uvm] [Fri Apr 30 01:23:59 2021] ? uvm_ext_gpu_map_free+0xe/0x20 [nvidia_uvm] [Fri Apr 30 01:23:59 2021] ? uvm_deferred_free_object_list+0x70/0x130 [nvidia_uvm] [Fri Apr 30 01:23:59 2021] ? uvm_va_space_destroy+0x349/0x480 [nvidia_uvm] [Fri Apr 30 01:23:59 2021] ? uvm_release.isra.10+0x7c/0x90 [nvidia_uvm] [Fri Apr 30 01:23:59 2021] ? uvm_release_entry+0x6d/0xa0 [nvidia_uvm] [Fri Apr 30 01:23:59 2021] ? __fput+0xc6/0x260 [Fri Apr 30 01:23:59 2021] ? ____fput+0xe/0x10 [Fri Apr 30 01:23:59 2021] ? task_work_run+0x9d/0xc0 [Fri Apr 30 01:23:59 2021] ? do_exit+0x389/0xb90 [Fri Apr 30 01:23:59 2021] ? do_group_exit+0x43/0xa0 [Fri Apr 30 01:23:59 2021] ? get_signal+0x14f/0x860 [Fri Apr 30 01:23:59 2021] ? do_signal+0x34/0x6d0 [Fri Apr 30 01:23:59 2021] ? __switch_to_asm+0x34/0x70 [Fri Apr 30 01:23:59 2021] ? __x64_sys_futex+0x13f/0x190 [Fri Apr 30 01:23:59 2021] ? exit_to_usermode_loop+0x90/0x130 [Fri Apr 30 01:23:59 2021] ? do_syscall_64+0x170/0x190 [Fri Apr 30 01:23:59 2021] ? entry_SYSCALL_64_after_hwframe+0x44/0xa9 [Fri Apr 30 01:24:08 2021] rcu: INFO: rcu_sched self-detected stall on CPU [Fri Apr 30 01:24:08 2021] rcu: 16-....: (14998 ticks this GP) idle=6ea/1/0x4000000000000002 softirq=8618/8618 fqs=7499 [Fri Apr 30 01:24:08 2021] (t=15000 jiffies g=47413 q=99087) [Fri Apr 30 01:24:08 2021] NMI backtrace for cpu 16 [Fri Apr 30 01:24:08 2021] CPU: 16 PID: 5432 Comm: python3 Tainted: P W OEL 5.4.0-72-generic #80~18.04.1-Ubuntu [Fri Apr 30 01:24:08 2021] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./TRX40 Creator, BIOS P1.70 05/29/2020 [Fri Apr 30 01:24:08 2021] Call Trace: [Fri Apr 30 01:24:08 2021] [Fri Apr 30 01:24:08 2021] dump_stack+0x6d/0x8b [Fri Apr 30 01:24:08 2021] nmi_cpu_backtrace+0x94/0xa0 [Fri Apr 30 01:24:08 2021] ? lapic_can_unplug_cpu+0xb0/0xb0 [Fri Apr 30 01:24:08 2021] nmi_trigger_cpumask_backtrace+0xe7/0x130 [Fri Apr 30 01:24:08 2021] arch_trigger_cpumask_backtrace+0x19/0x20 [Fri Apr 30 01:24:08 2021] rcu_dump_cpu_stacks+0x9e/0xce [Fri Apr 30 01:24:08 2021] rcu_sched_clock_irq+0x508/0x770 [Fri Apr 30 01:24:08 2021] ? tick_sched_do_timer+0x60/0x60 [Fri Apr 30 01:24:08 2021] update_process_times+0x28/0x50 [Fri Apr 30 01:24:08 2021] tick_sched_handle+0x25/0x70 [Fri Apr 30 01:24:08 2021] ? tick_sched_do_timer+0x58/0x60 [Fri Apr 30 01:24:08 2021] tick_sched_timer+0x3c/0x80 [Fri Apr 30 01:24:08 2021] __hrtimer_run_queues+0x10f/0x280 [Fri Apr 30 01:24:08 2021] hrtimer_interrupt+0xe7/0x240 [Fri Apr 30 01:24:08 2021] smp_apic_timer_interrupt+0x6f/0x130 [Fri Apr 30 01:24:08 2021] apic_timer_interrupt+0xf/0x20 [Fri Apr 30 01:24:08 2021] [Fri Apr 30 01:24:08 2021] RIP: 0010:iommu_unmap_page+0xe5/0x100 [Fri Apr 30 01:24:08 2021] Code: 4c 8d 34 10 77 91 4d 85 ff 75 23 48 8b 7d d0 65 48 33 3c 25 28 00 00 00 4c 89 f8 75 1c 48 83 c4 10 5b 41 5c 41 5d 41 5e 41 5f <5d> c3 0f 0b 49 8d 47 ff 4c 85 f8 74 d4 0f 0b e8 87 44 9c ff 0f 1f [Fri Apr 30 01:24:08 2021] RSP: 0018:fffface947c0f968 EFLAGS: 00000282 ORIG_RAX: ffffffffffffff13 [Fri Apr 30 01:24:08 2021] RAX: 0000008000000000 RBX: 00008f4a65450000 RCX: 0000000000000027 [Fri Apr 30 01:24:08 2021] RDX: 00008f0000000000 RSI: 0000000000000000 RDI: 0000000000000000 [Fri Apr 30 01:24:08 2021] RBP: fffface947c0f968 R08: fffface947c0f930 R09: 0000000000000004 [Fri Apr 30 01:24:08 2021] R10: 0000000000000000 R11: 000000000000001b R12: ffff9dcbe418b000 [Fri Apr 30 01:24:08 2021] R13: 00008ff9fffff000 R14: 00000000ffffffff R15: 00007ffa00000000 [Fri Apr 30 01:24:08 2021] __unmap_single.isra.27+0x66/0x130 [Fri Apr 30 01:24:08 2021] unmap_sg+0x5f/0x70 [Fri Apr 30 01:24:08 2021] nv_unmap_dma_map_scatterlist+0x57/0xa0 [nvidia] [Fri Apr 30 01:24:08 2021] nv_dma_unmap_pages+0x56/0x100 [nvidia] [Fri Apr 30 01:24:08 2021] nv_dma_unmap_alloc+0x16/0x30 [nvidia] [Fri Apr 30 01:24:08 2021] _nv032962rm+0xc1/0x1b0 [nvidia] [Fri Apr 30 01:24:08 2021] ? _nv027787rm+0x9b/0xc0 [nvidia] [Fri Apr 30 01:24:08 2021] ? _nv030989rm+0x124/0x130 [nvidia] [Fri Apr 30 01:24:08 2021] ? rm_gpu_ops_free_duped_handle+0x1c/0x60 [nvidia] [Fri Apr 30 01:24:08 2021] ? nvUvmInterfaceFreeDupedHandle+0x2a/0x40 [nvidia] [Fri Apr 30 01:24:08 2021] ? uvm_ext_gpu_map_free_internal+0x5c/0x70 [nvidia_uvm] [Fri Apr 30 01:24:08 2021] ? uvm_ext_gpu_map_free+0xe/0x20 [nvidia_uvm] [Fri Apr 30 01:24:08 2021] ? uvm_deferred_free_object_list+0x70/0x130 [nvidia_uvm] [Fri Apr 30 01:24:08 2021] ? uvm_va_space_destroy+0x349/0x480 [nvidia_uvm] [Fri Apr 30 01:24:08 2021] ? uvm_release.isra.10+0x7c/0x90 [nvidia_uvm] [Fri Apr 30 01:24:08 2021] ? uvm_release_entry+0x6d/0xa0 [nvidia_uvm] [Fri Apr 30 01:24:08 2021] ? __fput+0xc6/0x260 [Fri Apr 30 01:24:08 2021] ? ____fput+0xe/0x10 [Fri Apr 30 01:24:08 2021] ? task_work_run+0x9d/0xc0 [Fri Apr 30 01:24:08 2021] ? do_exit+0x389/0xb90 [Fri Apr 30 01:24:08 2021] ? do_group_exit+0x43/0xa0 [Fri Apr 30 01:24:08 2021] ? get_signal+0x14f/0x860 [Fri Apr 30 01:24:08 2021] ? do_signal+0x34/0x6d0 [Fri Apr 30 01:24:08 2021] ? __switch_to_asm+0x34/0x70 [Fri Apr 30 01:24:08 2021] ? __x64_sys_futex+0x13f/0x190 [Fri Apr 30 01:24:08 2021] ? exit_to_usermode_loop+0x90/0x130 [Fri Apr 30 01:24:08 2021] ? do_syscall_64+0x170/0x190 [Fri Apr 30 01:24:08 2021] ? entry_SYSCALL_64_after_hwframe+0x44/0xa9 [Fri Apr 30 01:24:14 2021] rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 16-... } 15141 jiffies s: 901 root: 0x2/. [Fri Apr 30 01:24:14 2021] rcu: blocking rcu_node structures: l=1:16-31:0x1/. [Fri Apr 30 01:24:14 2021] Task dump for CPU 16: [Fri Apr 30 01:24:14 2021] python3 R running task 0 5432 4271 0x8000400a [Fri Apr 30 01:24:14 2021] Call Trace: [Fri Apr 30 01:24:14 2021] ? unmap_sg+0x5f/0x70 [Fri Apr 30 01:24:14 2021] ? nv_unmap_dma_map_scatterlist+0x57/0xa0 [nvidia] [Fri Apr 30 01:24:14 2021] ? nv_dma_unmap_pages+0x56/0x100 [nvidia] [Fri Apr 30 01:24:14 2021] ? nv_dma_unmap_alloc+0x16/0x30 [nvidia] [Fri Apr 30 01:24:14 2021] ? _nv032962rm+0xc1/0x1b0 [nvidia] [Fri Apr 30 01:24:14 2021] ? _nv027787rm+0x9b/0xc0 [nvidia] [Fri Apr 30 01:24:14 2021] ? _nv030989rm+0x124/0x130 [nvidia] [Fri Apr 30 01:24:14 2021] ? rm_gpu_ops_free_duped_handle+0x1c/0x60 [nvidia] [Fri Apr 30 01:24:14 2021] ? nvUvmInterfaceFreeDupedHandle+0x2a/0x40 [nvidia] [Fri Apr 30 01:24:14 2021] ? uvm_ext_gpu_map_free_internal+0x5c/0x70 [nvidia_uvm] [Fri Apr 30 01:24:14 2021] ? uvm_ext_gpu_map_free+0xe/0x20 [nvidia_uvm] [Fri Apr 30 01:24:14 2021] ? uvm_deferred_free_object_list+0x70/0x130 [nvidia_uvm] [Fri Apr 30 01:24:14 2021] ? uvm_va_space_destroy+0x349/0x480 [nvidia_uvm] [Fri Apr 30 01:24:14 2021] ? uvm_release.isra.10+0x7c/0x90 [nvidia_uvm] [Fri Apr 30 01:24:14 2021] ? uvm_release_entry+0x6d/0xa0 [nvidia_uvm] [Fri Apr 30 01:24:14 2021] ? __fput+0xc6/0x260 [Fri Apr 30 01:24:14 2021] ? ____fput+0xe/0x10 [Fri Apr 30 01:24:14 2021] ? task_work_run+0x9d/0xc0 [Fri Apr 30 01:24:14 2021] ? do_exit+0x389/0xb90 [Fri Apr 30 01:24:14 2021] ? do_group_exit+0x43/0xa0 [Fri Apr 30 01:24:14 2021] ? get_signal+0x14f/0x860 [Fri Apr 30 01:24:14 2021] ? do_signal+0x34/0x6d0 [Fri Apr 30 01:24:14 2021] ? __switch_to_asm+0x34/0x70 [Fri Apr 30 01:24:14 2021] ? __x64_sys_futex+0x13f/0x190 [Fri Apr 30 01:24:14 2021] ? exit_to_usermode_loop+0x90/0x130 [Fri Apr 30 01:24:14 2021] ? do_syscall_64+0x170/0x190 [Fri Apr 30 01:24:14 2021] ? entry_SYSCALL_64_after_hwframe+0x44/0xa9 [Fri Apr 30 01:24:35 2021] watchdog: BUG: soft lockup - CPU#16 stuck for 23s! [python3:5432] [Fri Apr 30 01:24:35 2021] Modules linked in: nvidia_uvm(OE) veth xt_nat xt_tcpudp xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype iptable_filter iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 bpfilter br_netfilter bridge stp llc aufs cmac overlay bnep snd_hda_codec_hdmi nvidia_drm(POE) nvidia_modeset(POE) edac_mce_amd snd_usb_audio snd_usbmidi_lib mc kvm_amd kvm iwlmvm mac80211 crct10dif_pclmul crc32_pclmul btusb nls_iso8859_1 btrtl nvidia(POE) libarc4 btbcm ghash_clmulni_intel btintel bluetooth iwlwifi ecdh_generic aesni_intel snd_seq_midi input_leds ecc crypto_simd snd_seq_midi_event cryptd glue_helper cfg80211 wmi_bmof snd_hda_intel snd_rawmidi snd_intel_dspcfg joydev snd_hda_codec nouveau snd_hda_core snd_hwdep snd_seq snd_pcm mxm_wmi video snd_seq_device ccp ttm snd_timer drm_kms_helper snd drm i2c_algo_bit fb_sys_fops syscopyarea sysfillrect sysimgblt k10temp soundcore mac_hid sch_fq_codel parport_pc ppdev lp parport ip_tables x_tables [Fri Apr 30 01:24:35 2021] autofs4 btrfs xor zstd_compress raid6_pq libcrc32c dm_mirror dm_region_hash dm_log hid_generic usbhid hid nvme nvme_core ahci r8169 libahci atlantic realtek i2c_piix4 wmi [Fri Apr 30 01:24:35 2021] CPU: 16 PID: 5432 Comm: python3 Tainted: P W OEL 5.4.0-72-generic #80~18.04.1-Ubuntu [Fri Apr 30 01:24:35 2021] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./TRX40 Creator, BIOS P1.70 05/29/2020 [Fri Apr 30 01:24:35 2021] RIP: 0010:iommu_unmap_page+0x56/0x100 [Fri Apr 30 01:24:35 2021] Code: 00 00 48 8d 42 ff 48 89 d3 48 85 d0 0f 85 a9 00 00 00 4c 8d af 94 00 00 00 4c 8d a7 98 00 00 00 49 89 f6 45 31 ff 48 8d 4d c8 <4c> 89 e6 4c 89 f2 4c 89 ef e8 fc fc ff ff 48 85 c0 48 89 c6 74 3c [Fri Apr 30 01:24:35 2021] RSP: 0018:fffface947c0f930 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13 [Fri Apr 30 01:24:35 2021] RAX: 0000000000000fff RBX: 0000000000001000 RCX: fffface947c0f930 [Fri Apr 30 01:24:35 2021] RDX: 0000000000001000 RSI: 00008d511a75c000 RDI: ffff9dcbe418b000 [Fri Apr 30 01:24:35 2021] RBP: fffface947c0f968 R08: fffface947c0f930 R09: 0000000000000004 [Fri Apr 30 01:24:35 2021] R10: 0000000000000000 R11: 000000000000001b R12: ffff9dcbe418b098 [Fri Apr 30 01:24:35 2021] R13: ffff9dcbe418b094 R14: 00008d511a75c000 R15: 0000000000000000 [Fri Apr 30 01:24:35 2021] FS: 00007f9e29fff700(0000) GS:ffff9dcbfd200000(0000) knlGS:0000000000000000 [Fri Apr 30 01:24:35 2021] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fri Apr 30 01:24:35 2021] CR2: 00007fb3cf8144d0 CR3: 000000108660a000 CR4: 0000000000340ee0 [Fri Apr 30 01:24:35 2021] Call Trace: [Fri Apr 30 01:24:35 2021] __unmap_single.isra.27+0x66/0x130 [Fri Apr 30 01:24:35 2021] unmap_sg+0x5f/0x70 [Fri Apr 30 01:24:35 2021] nv_unmap_dma_map_scatterlist+0x57/0xa0 [nvidia] [Fri Apr 30 01:24:35 2021] nv_dma_unmap_pages+0x56/0x100 [nvidia] [Fri Apr 30 01:24:35 2021] nv_dma_unmap_alloc+0x16/0x30 [nvidia] [Fri Apr 30 01:24:35 2021] _nv032962rm+0xc1/0x1b0 [nvidia] [Fri Apr 30 01:24:35 2021] ? _nv027787rm+0x9b/0xc0 [nvidia] [Fri Apr 30 01:24:35 2021] ? _nv030989rm+0x124/0x130 [nvidia] [Fri Apr 30 01:24:35 2021] ? rm_gpu_ops_free_duped_handle+0x1c/0x60 [nvidia] [Fri Apr 30 01:24:35 2021] ? nvUvmInterfaceFreeDupedHandle+0x2a/0x40 [nvidia] [Fri Apr 30 01:24:35 2021] ? uvm_ext_gpu_map_free_internal+0x5c/0x70 [nvidia_uvm] [Fri Apr 30 01:24:35 2021] ? uvm_ext_gpu_map_free+0xe/0x20 [nvidia_uvm] [Fri Apr 30 01:24:35 2021] ? uvm_deferred_free_object_list+0x70/0x130 [nvidia_uvm] [Fri Apr 30 01:24:35 2021] ? uvm_va_space_destroy+0x349/0x480 [nvidia_uvm] [Fri Apr 30 01:24:35 2021] ? uvm_release.isra.10+0x7c/0x90 [nvidia_uvm] [Fri Apr 30 01:24:35 2021] ? uvm_release_entry+0x6d/0xa0 [nvidia_uvm] [Fri Apr 30 01:24:35 2021] ? __fput+0xc6/0x260 [Fri Apr 30 01:24:35 2021] ? ____fput+0xe/0x10 [Fri Apr 30 01:24:35 2021] ? task_work_run+0x9d/0xc0 [Fri Apr 30 01:24:35 2021] ? do_exit+0x389/0xb90 [Fri Apr 30 01:24:35 2021] ? do_group_exit+0x43/0xa0 [Fri Apr 30 01:24:35 2021] ? get_signal+0x14f/0x860 [Fri Apr 30 01:24:35 2021] ? do_signal+0x34/0x6d0 [Fri Apr 30 01:24:35 2021] ? __switch_to_asm+0x34/0x70 [Fri Apr 30 01:24:35 2021] ? __x64_sys_futex+0x13f/0x190 [Fri Apr 30 01:24:35 2021] ? exit_to_usermode_loop+0x90/0x130 [Fri Apr 30 01:24:35 2021] ? do_syscall_64+0x170/0x190 [Fri Apr 30 01:24:35 2021] ? entry_SYSCALL_64_after_hwframe+0x44/0xa9 [Fri Apr 30 01:25:03 2021] watchdog: BUG: soft lockup - CPU#16 stuck for 23s! [python3:5432] [Fri Apr 30 01:25:03 2021] Modules linked in: nvidia_uvm(OE) veth xt_nat xt_tcpudp xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype iptable_filter iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 bpfilter br_netfilter bridge stp llc aufs cmac overlay bnep snd_hda_codec_hdmi nvidia_drm(POE) nvidia_modeset(POE) edac_mce_amd snd_usb_audio snd_usbmidi_lib mc kvm_amd kvm iwlmvm mac80211 crct10dif_pclmul crc32_pclmul btusb nls_iso8859_1 btrtl nvidia(POE) libarc4 btbcm ghash_clmulni_intel btintel bluetooth iwlwifi ecdh_generic aesni_intel snd_seq_midi input_leds ecc crypto_simd snd_seq_midi_event cryptd glue_helper cfg80211 wmi_bmof snd_hda_intel snd_rawmidi snd_intel_dspcfg joydev snd_hda_codec nouveau snd_hda_core snd_hwdep snd_seq snd_pcm mxm_wmi video snd_seq_device ccp ttm snd_timer drm_kms_helper snd drm i2c_algo_bit fb_sys_fops syscopyarea sysfillrect sysimgblt k10temp soundcore mac_hid sch_fq_codel parport_pc ppdev lp parport ip_tables x_tables [Fri Apr 30 01:25:03 2021] autofs4 btrfs xor zstd_compress raid6_pq libcrc32c dm_mirror dm_region_hash dm_log hid_generic usbhid hid nvme nvme_core ahci r8169 libahci atlantic realtek i2c_piix4 wmi [Fri Apr 30 01:25:03 2021] CPU: 16 PID: 5432 Comm: python3 Tainted: P W OEL 5.4.0-72-generic #80~18.04.1-Ubuntu [Fri Apr 30 01:25:03 2021] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./TRX40 Creator, BIOS P1.70 05/29/2020 [Fri Apr 30 01:25:03 2021] RIP: 0010:fetch_pte.isra.7+0x49/0x1a0 [Fri Apr 30 01:25:03 2021] Code: 00 00 00 48 d3 e0 48 83 e8 01 48 39 d0 0f 82 5c 01 00 00 41 8d 41 ff 55 48 89 d7 44 8d 1c c0 48 89 e5 53 48 8b 36 41 8d 4b 0c <48> d3 ef 81 e7 ff 01 00 00 4c 8d 14 fe be 01 00 00 00 48 d3 e6 85 [Fri Apr 30 01:25:03 2021] RSP: 0018:fffface947c0f918 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [Fri Apr 30 01:25:03 2021] RAX: 0000000000000003 RBX: 0000000000001000 RCX: 0000000000000027 [Fri Apr 30 01:25:03 2021] RDX: 00008be556a17000 RSI: ffff9dc8e85cd000 RDI: 00008be556a17000 [Fri Apr 30 01:25:03 2021] RBP: fffface947c0f920 R08: fffface947c0f930 R09: 0000000000000004 [Fri Apr 30 01:25:03 2021] R10: 0000000000000000 R11: 000000000000001b R12: ffff9dcbe418b098 [Fri Apr 30 01:25:03 2021] R13: ffff9dcbe418b094 R14: 00008be556a17000 R15: 0000000000000000 [Fri Apr 30 01:25:03 2021] FS: 00007f9e29fff700(0000) GS:ffff9dcbfd200000(0000) knlGS:0000000000000000 [Fri Apr 30 01:25:03 2021] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fri Apr 30 01:25:03 2021] CR2: 00007fb3cf8144d0 CR3: 000000108660a000 CR4: 0000000000340ee0 [Fri Apr 30 01:25:03 2021] Call Trace: [Fri Apr 30 01:25:03 2021] iommu_unmap_page+0x64/0x100 [Fri Apr 30 01:25:03 2021] __unmap_single.isra.27+0x66/0x130 [Fri Apr 30 01:25:03 2021] unmap_sg+0x5f/0x70 [Fri Apr 30 01:25:03 2021] nv_unmap_dma_map_scatterlist+0x57/0xa0 [nvidia] [Fri Apr 30 01:25:03 2021] nv_dma_unmap_pages+0x56/0x100 [nvidia] [Fri Apr 30 01:25:03 2021] nv_dma_unmap_alloc+0x16/0x30 [nvidia] [Fri Apr 30 01:25:03 2021] _nv032962rm+0xc1/0x1b0 [nvidia] [Fri Apr 30 01:25:03 2021] ? _nv027787rm+0x9b/0xc0 [nvidia] [Fri Apr 30 01:25:03 2021] ? _nv030989rm+0x124/0x130 [nvidia] [Fri Apr 30 01:25:03 2021] ? rm_gpu_ops_free_duped_handle+0x1c/0x60 [nvidia] [Fri Apr 30 01:25:03 2021] ? nvUvmInterfaceFreeDupedHandle+0x2a/0x40 [nvidia] [Fri Apr 30 01:25:03 2021] ? uvm_ext_gpu_map_free_internal+0x5c/0x70 [nvidia_uvm] [Fri Apr 30 01:25:03 2021] ? uvm_ext_gpu_map_free+0xe/0x20 [nvidia_uvm] [Fri Apr 30 01:25:03 2021] ? uvm_deferred_free_object_list+0x70/0x130 [nvidia_uvm] [Fri Apr 30 01:25:03 2021] ? uvm_va_space_destroy+0x349/0x480 [nvidia_uvm] [Fri Apr 30 01:25:03 2021] ? uvm_release.isra.10+0x7c/0x90 [nvidia_uvm] [Fri Apr 30 01:25:03 2021] ? uvm_release_entry+0x6d/0xa0 [nvidia_uvm] [Fri Apr 30 01:25:03 2021] ? __fput+0xc6/0x260 [Fri Apr 30 01:25:03 2021] ? ____fput+0xe/0x10 [Fri Apr 30 01:25:03 2021] ? task_work_run+0x9d/0xc0 [Fri Apr 30 01:25:03 2021] ? do_exit+0x389/0xb90 [Fri Apr 30 01:25:03 2021] ? do_group_exit+0x43/0xa0 [Fri Apr 30 01:25:03 2021] ? get_signal+0x14f/0x860 [Fri Apr 30 01:25:03 2021] ? do_signal+0x34/0x6d0 [Fri Apr 30 01:25:03 2021] ? __switch_to_asm+0x34/0x70 [Fri Apr 30 01:25:03 2021] ? __x64_sys_futex+0x13f/0x190 [Fri Apr 30 01:25:03 2021] ? exit_to_usermode_loop+0x90/0x130 [Fri Apr 30 01:25:03 2021] ? do_syscall_64+0x170/0x190 [Fri Apr 30 01:25:03 2021] ? entry_SYSCALL_64_after_hwframe+0x44/0xa9 [Fri Apr 30 01:25:11 2021] iwlwifi 0000:45:00.0: HCMD_ACTIVE already clear for command SCAN_REQ_UMAC [Fri Apr 30 01:25:11 2021] iwlwifi 0000:45:00.0: Applying debug destination EXTERNAL_DRAM [Fri Apr 30 01:25:11 2021] iwlwifi 0000:45:00.0: FW already configured (0) - re-configuring [Fri Apr 30 01:25:11 2021] ------------[ cut here ]------------ [Fri Apr 30 01:25:11 2021] WARNING: CPU: 16 PID: 412 at /build/linux-hwe-5.4-PtAV0J/linux-hwe-5.4-5.4.0/drivers/net/wireless/intel/iwlwifi/mvm/scan.c:1929 iwl_mvm_rx_umac_scan_complete_notif+0x1f9/0x210 [iwlmvm] [Fri Apr 30 01:25:11 2021] Modules linked in: nvidia_uvm(OE) veth xt_nat xt_tcpudp xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype iptable_filter iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 bpfilter br_netfilter bridge stp llc aufs cmac overlay bnep snd_hda_codec_hdmi nvidia_drm(POE) nvidia_modeset(POE) edac_mce_amd snd_usb_audio snd_usbmidi_lib mc kvm_amd kvm iwlmvm mac80211 crct10dif_pclmul crc32_pclmul btusb nls_iso8859_1 btrtl nvidia(POE) libarc4 btbcm ghash_clmulni_intel btintel bluetooth iwlwifi ecdh_generic aesni_intel snd_seq_midi input_leds ecc crypto_simd snd_seq_midi_event cryptd glue_helper cfg80211 wmi_bmof snd_hda_intel snd_rawmidi snd_intel_dspcfg joydev snd_hda_codec nouveau snd_hda_core snd_hwdep snd_seq snd_pcm mxm_wmi video snd_seq_device ccp ttm snd_timer drm_kms_helper snd drm i2c_algo_bit fb_sys_fops syscopyarea sysfillrect sysimgblt k10temp soundcore mac_hid sch_fq_codel parport_pc ppdev lp parport ip_tables x_tables [Fri Apr 30 01:25:11 2021] autofs4 btrfs xor zstd_compress raid6_pq libcrc32c dm_mirror dm_region_hash dm_log hid_generic usbhid hid nvme nvme_core ahci r8169 libahci atlantic realtek i2c_piix4 wmi [Fri Apr 30 01:25:11 2021] CPU: 16 PID: 412 Comm: kworker/16:1 Tainted: P W OEL 5.4.0-72-generic #80~18.04.1-Ubuntu [Fri Apr 30 01:25:11 2021] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./TRX40 Creator, BIOS P1.70 05/29/2020 [Fri Apr 30 01:25:11 2021] Workqueue: events iwl_mvm_async_handlers_wk [iwlmvm] [Fri Apr 30 01:25:11 2021] RIP: 0010:iwl_mvm_rx_umac_scan_complete_notif+0x1f9/0x210 [iwlmvm] [Fri Apr 30 01:25:11 2021] Code: cc e8 db a1 aa ff 41 c7 84 24 60 1d 00 00 00 00 00 00 41 8b 8c 24 40 1d 00 00 41 8b 85 c4 1d 00 00 44 8b 4d cc e9 7c fe ff ff <0f> 0b e9 2e ff ff ff e8 fb 61 47 cb 90 66 2e 0f 1f 84 00 00 00 00 [Fri Apr 30 01:25:11 2021] RSP: 0018:fffface941337df0 EFLAGS: 00010246 [Fri Apr 30 01:25:11 2021] RAX: 0000000000000000 RBX: ffff9dcbbef63000 RCX: 0000000000000000 [Fri Apr 30 01:25:11 2021] RDX: ffffffffc1229b10 RSI: ffff9dbbd2806e90 RDI: ffff9dcbbbe99e88 [Fri Apr 30 01:25:11 2021] RBP: fffface941337e28 R08: 0000000000000000 R09: 0000000000000000 [Fri Apr 30 01:25:11 2021] R10: fffface941337e10 R11: 0000000000000042 R12: fffface941337e40 [Fri Apr 30 01:25:11 2021] R13: ffff9dcbbbe99e88 R14: ffff9dbbd2806e80 R15: dead000000000100 [Fri Apr 30 01:25:11 2021] FS: 0000000000000000(0000) GS:ffff9dcbfd200000(0000) knlGS:0000000000000000 [Fri Apr 30 01:25:11 2021] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fri Apr 30 01:25:11 2021] CR2: 00007fb3cf8144d0 CR3: 000000108660a000 CR4: 0000000000340ee0 [Fri Apr 30 01:25:11 2021] Call Trace: [Fri Apr 30 01:25:11 2021] ? __mutex_lock_slowpath+0x13/0x20 [Fri Apr 30 01:25:11 2021] iwl_mvm_async_handlers_wk+0xb0/0x140 [iwlmvm] [Fri Apr 30 01:25:11 2021] process_one_work+0x20f/0x400 [Fri Apr 30 01:25:11 2021] worker_thread+0x34/0x410 [Fri Apr 30 01:25:11 2021] kthread+0x121/0x140 [Fri Apr 30 01:25:11 2021] ? process_one_work+0x400/0x400 [Fri Apr 30 01:25:11 2021] ? kthread_park+0x90/0x90 [Fri Apr 30 01:25:11 2021] ret_from_fork+0x22/0x40 [Fri Apr 30 01:25:11 2021] ---[ end trace c3ee37f01c9a84ea ]--- [Fri Apr 30 01:25:31 2021] watchdog: BUG: soft lockup - CPU#40 stuck for 22s! [python3:5432] [Fri Apr 30 01:25:31 2021] Modules linked in: nvidia_uvm(OE) veth xt_nat xt_tcpudp xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype iptable_filter iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 bpfilter br_netfilter bridge stp llc aufs cmac overlay bnep snd_hda_codec_hdmi nvidia_drm(POE) nvidia_modeset(POE) edac_mce_amd snd_usb_audio snd_usbmidi_lib mc kvm_amd kvm iwlmvm mac80211 crct10dif_pclmul crc32_pclmul btusb nls_iso8859_1 btrtl nvidia(POE) libarc4 btbcm ghash_clmulni_intel btintel bluetooth iwlwifi ecdh_generic aesni_intel snd_seq_midi input_leds ecc crypto_simd snd_seq_midi_event cryptd glue_helper cfg80211 wmi_bmof snd_hda_intel snd_rawmidi snd_intel_dspcfg joydev snd_hda_codec nouveau snd_hda_core snd_hwdep snd_seq snd_pcm mxm_wmi video snd_seq_device ccp ttm snd_timer drm_kms_helper snd drm i2c_algo_bit fb_sys_fops syscopyarea sysfillrect sysimgblt k10temp soundcore mac_hid sch_fq_codel parport_pc ppdev lp parport ip_tables x_tables [Fri Apr 30 01:25:31 2021] autofs4 btrfs xor zstd_compress raid6_pq libcrc32c dm_mirror dm_region_hash dm_log hid_generic usbhid hid nvme nvme_core ahci r8169 libahci atlantic realtek i2c_piix4 wmi [Fri Apr 30 01:25:31 2021] CPU: 40 PID: 5432 Comm: python3 Tainted: P W OEL 5.4.0-72-generic #80~18.04.1-Ubuntu [Fri Apr 30 01:25:31 2021] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./TRX40 Creator, BIOS P1.70 05/29/2020 [Fri Apr 30 01:25:31 2021] RIP: 0010:fetch_pte.isra.7+0x32/0x1a0 [Fri Apr 30 01:25:31 2021] Code: 00 00 00 00 44 8b 0f 49 89 c8 41 83 f9 05 7f 1a 43 8d 4c c9 0c b8 01 00 00 00 48 d3 e0 48 83 e8 01 48 39 d0 0f 82 5c 01 00 00 <41> 8d 41 ff 55 48 89 d7 44 8d 1c c0 48 89 e5 53 48 8b 36 41 8d 4b [Fri Apr 30 01:25:31 2021] RSP: 0018:fffface947c0f928 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [Fri Apr 30 01:25:31 2021] RAX: 0000ffffffffffff RBX: 0000000000001000 RCX: 0000000000000030 [Fri Apr 30 01:25:31 2021] RDX: 00008a827e53f000 RSI: ffff9dcbdc371098 RDI: ffff9dcbdc371094 [Fri Apr 30 01:25:31 2021] RBP: fffface947c0f968 R08: fffface947c0f930 R09: 0000000000000004 [Fri Apr 30 01:25:31 2021] R10: 0000000000000000 R11: 000000000000001b R12: ffff9dcbdc371098 [Fri Apr 30 01:25:31 2021] R13: ffff9dcbdc371094 R14: 00008a827e53f000 R15: 0000000000000000 [Fri Apr 30 01:25:31 2021] FS: 00007f9e29fff700(0000) GS:ffff9dcbfd800000(0000) knlGS:0000000000000000 [Fri Apr 30 01:25:31 2021] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fri Apr 30 01:25:31 2021] CR2: 00007f00cbc90150 CR3: 0000001fbb9b0000 CR4: 0000000000340ee0 [Fri Apr 30 01:25:31 2021] Call Trace: [Fri Apr 30 01:25:31 2021] ? iommu_unmap_page+0x64/0x100 [Fri Apr 30 01:25:31 2021] __unmap_single.isra.27+0x66/0x130 [Fri Apr 30 01:25:31 2021] unmap_sg+0x5f/0x70 [Fri Apr 30 01:25:31 2021] nv_unmap_dma_map_scatterlist+0x57/0xa0 [nvidia] [Fri Apr 30 01:25:31 2021] nv_dma_unmap_pages+0x56/0x100 [nvidia] [Fri Apr 30 01:25:31 2021] nv_dma_unmap_alloc+0x16/0x30 [nvidia] [Fri Apr 30 01:25:31 2021] _nv032962rm+0xc1/0x1b0 [nvidia] [Fri Apr 30 01:25:31 2021] ? _nv027787rm+0x9b/0xc0 [nvidia] [Fri Apr 30 01:25:31 2021] ? _nv030989rm+0x124/0x130 [nvidia] [Fri Apr 30 01:25:31 2021] ? rm_gpu_ops_free_duped_handle+0x1c/0x60 [nvidia] [Fri Apr 30 01:25:31 2021] ? nvUvmInterfaceFreeDupedHandle+0x2a/0x40 [nvidia] [Fri Apr 30 01:25:31 2021] ? uvm_ext_gpu_map_free_internal+0x5c/0x70 [nvidia_uvm] [Fri Apr 30 01:25:31 2021] ? uvm_ext_gpu_map_free+0xe/0x20 [nvidia_uvm] [Fri Apr 30 01:25:31 2021] ? uvm_deferred_free_object_list+0x70/0x130 [nvidia_uvm] [Fri Apr 30 01:25:31 2021] ? uvm_va_space_destroy+0x349/0x480 [nvidia_uvm] [Fri Apr 30 01:25:31 2021] ? uvm_release.isra.10+0x7c/0x90 [nvidia_uvm] [Fri Apr 30 01:25:31 2021] ? uvm_release_entry+0x6d/0xa0 [nvidia_uvm] [Fri Apr 30 01:25:31 2021] ? __fput+0xc6/0x260 [Fri Apr 30 01:25:31 2021] ? ____fput+0xe/0x10 [Fri Apr 30 01:25:31 2021] ? task_work_run+0x9d/0xc0 [Fri Apr 30 01:25:31 2021] ? do_exit+0x389/0xb90 [Fri Apr 30 01:25:31 2021] ? do_group_exit+0x43/0xa0 [Fri Apr 30 01:25:31 2021] ? get_signal+0x14f/0x860 [Fri Apr 30 01:25:31 2021] ? do_signal+0x34/0x6d0 [Fri Apr 30 01:25:31 2021] ? __switch_to_asm+0x34/0x70 [Fri Apr 30 01:25:31 2021] ? __x64_sys_futex+0x13f/0x190 [Fri Apr 30 01:25:31 2021] ? exit_to_usermode_loop+0x90/0x130 [Fri Apr 30 01:25:31 2021] ? do_syscall_64+0x170/0x190 [Fri Apr 30 01:25:31 2021] ? entry_SYSCALL_64_after_hwframe+0x44/0xa9 [Fri Apr 30 01:25:59 2021] watchdog: BUG: soft lockup - CPU#40 stuck for 22s! [python3:5432] [Fri Apr 30 01:25:59 2021] Modules linked in: nvidia_uvm(OE) veth xt_nat xt_tcpudp xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype iptable_filter iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 bpfilter br_netfilter bridge stp llc aufs cmac overlay bnep snd_hda_codec_hdmi nvidia_drm(POE) nvidia_modeset(POE) edac_mce_amd snd_usb_audio snd_usbmidi_lib mc kvm_amd kvm iwlmvm mac80211 crct10dif_pclmul crc32_pclmul btusb nls_iso8859_1 btrtl nvidia(POE) libarc4 btbcm ghash_clmulni_intel btintel bluetooth iwlwifi ecdh_generic aesni_intel snd_seq_midi input_leds ecc crypto_simd snd_seq_midi_event cryptd glue_helper cfg80211 wmi_bmof snd_hda_intel snd_rawmidi snd_intel_dspcfg joydev snd_hda_codec nouveau snd_hda_core snd_hwdep snd_seq snd_pcm mxm_wmi video snd_seq_device ccp ttm snd_timer drm_kms_helper snd drm i2c_algo_bit fb_sys_fops syscopyarea sysfillrect sysimgblt k10temp soundcore mac_hid sch_fq_codel parport_pc ppdev lp parport ip_tables x_tables [Fri Apr 30 01:25:59 2021] autofs4 btrfs xor zstd_compress raid6_pq libcrc32c dm_mirror dm_region_hash dm_log hid_generic usbhid hid nvme nvme_core ahci r8169 libahci atlantic realtek i2c_piix4 wmi [Fri Apr 30 01:25:59 2021] CPU: 40 PID: 5432 Comm: python3 Tainted: P W OEL 5.4.0-72-generic #80~18.04.1-Ubuntu [Fri Apr 30 01:25:59 2021] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./TRX40 Creator, BIOS P1.70 05/29/2020 [Fri Apr 30 01:25:59 2021] RIP: 0010:iommu_unmap_page+0xac/0x100 [Fri Apr 30 01:25:59 2021] Code: e1 48 c1 ea 03 48 8d 04 d2 ba 01 00 00 00 48 29 c1 31 c0 48 d3 e2 48 c7 04 c6 00 00 00 00 48 83 c0 01 39 c2 7f f0 48 8b 45 c8 <48> 89 c2 49 01 c7 48 f7 da 4c 21 f2 4c 39 fb 4c 8d 34 10 77 91 4d [Fri Apr 30 01:25:59 2021] RSP: 0018:fffface947c0f930 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13 [Fri Apr 30 01:25:59 2021] RAX: 0000008000000000 RBX: 0000000000001000 RCX: 0000000000000027 [Fri Apr 30 01:25:59 2021] RDX: 0000891472939000 RSI: 0000000000000000 RDI: 0000000000000000 [Fri Apr 30 01:25:59 2021] RBP: fffface947c0f968 R08: fffface947c0f930 R09: 0000000000000004 [Fri Apr 30 01:25:59 2021] R10: 0000000000000000 R11: 000000000000001b R12: ffff9dcbdc371098 [Fri Apr 30 01:25:59 2021] R13: ffff9dcbdc371094 R14: 0000891472939000 R15: 0000000000000000 [Fri Apr 30 01:25:59 2021] FS: 00007f9e29fff700(0000) GS:ffff9dcbfd800000(0000) knlGS:0000000000000000 [Fri Apr 30 01:25:59 2021] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fri Apr 30 01:25:59 2021] CR2: 00007f00cbc90150 CR3: 0000001fbb9b0000 CR4: 0000000000340ee0 [Fri Apr 30 01:25:59 2021] Call Trace: [Fri Apr 30 01:25:59 2021] __unmap_single.isra.27+0x66/0x130 [Fri Apr 30 01:25:59 2021] unmap_sg+0x5f/0x70 [Fri Apr 30 01:25:59 2021] nv_unmap_dma_map_scatterlist+0x57/0xa0 [nvidia] [Fri Apr 30 01:25:59 2021] nv_dma_unmap_pages+0x56/0x100 [nvidia] [Fri Apr 30 01:25:59 2021] nv_dma_unmap_alloc+0x16/0x30 [nvidia] [Fri Apr 30 01:25:59 2021] _nv032962rm+0xc1/0x1b0 [nvidia] [Fri Apr 30 01:25:59 2021] ? _nv027787rm+0x9b/0xc0 [nvidia] [Fri Apr 30 01:25:59 2021] ? _nv030989rm+0x124/0x130 [nvidia] [Fri Apr 30 01:25:59 2021] ? rm_gpu_ops_free_duped_handle+0x1c/0x60 [nvidia] [Fri Apr 30 01:25:59 2021] ? nvUvmInterfaceFreeDupedHandle+0x2a/0x40 [nvidia] [Fri Apr 30 01:25:59 2021] ? uvm_ext_gpu_map_free_internal+0x5c/0x70 [nvidia_uvm] [Fri Apr 30 01:25:59 2021] ? uvm_ext_gpu_map_free+0xe/0x20 [nvidia_uvm] [Fri Apr 30 01:25:59 2021] ? uvm_deferred_free_object_list+0x70/0x130 [nvidia_uvm] [Fri Apr 30 01:25:59 2021] ? uvm_va_space_destroy+0x349/0x480 [nvidia_uvm] [Fri Apr 30 01:25:59 2021] ? uvm_release.isra.10+0x7c/0x90 [nvidia_uvm] [Fri Apr 30 01:25:59 2021] ? uvm_release_entry+0x6d/0xa0 [nvidia_uvm] [Fri Apr 30 01:25:59 2021] ? __fput+0xc6/0x260 [Fri Apr 30 01:25:59 2021] ? ____fput+0xe/0x10 [Fri Apr 30 01:25:59 2021] ? task_work_run+0x9d/0xc0 [Fri Apr 30 01:25:59 2021] ? do_exit+0x389/0xb90 [Fri Apr 30 01:25:59 2021] ? do_group_exit+0x43/0xa0 [Fri Apr 30 01:25:59 2021] ? get_signal+0x14f/0x860 [Fri Apr 30 01:25:59 2021] ? do_signal+0x34/0x6d0 [Fri Apr 30 01:25:59 2021] ? __switch_to_asm+0x34/0x70 [Fri Apr 30 01:25:59 2021] ? __x64_sys_futex+0x13f/0x190 [Fri Apr 30 01:25:59 2021] ? exit_to_usermode_loop+0x90/0x130 [Fri Apr 30 01:25:59 2021] ? do_syscall_64+0x170/0x190 [Fri Apr 30 01:25:59 2021] ? entry_SYSCALL_64_after_hwframe+0x44/0xa9 [Fri Apr 30 01:26:11 2021] rcu: INFO: rcu_sched self-detected stall on CPU [Fri Apr 30 01:26:11 2021] rcu: 40-....: (15000 ticks this GP) idle=f7a/1/0x4000000000000002 softirq=7610/7610 fqs=7490 [Fri Apr 30 01:26:11 2021] (t=15001 jiffies g=47417 q=298047) [Fri Apr 30 01:26:11 2021] NMI backtrace for cpu 40 [Fri Apr 30 01:26:11 2021] CPU: 40 PID: 5432 Comm: python3 Tainted: P W OEL 5.4.0-72-generic #80~18.04.1-Ubuntu [Fri Apr 30 01:26:11 2021] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./TRX40 Creator, BIOS P1.70 05/29/2020 [Fri Apr 30 01:26:11 2021] Call Trace: [Fri Apr 30 01:26:11 2021] [Fri Apr 30 01:26:11 2021] dump_stack+0x6d/0x8b [Fri Apr 30 01:26:11 2021] nmi_cpu_backtrace+0x94/0xa0 [Fri Apr 30 01:26:11 2021] ? lapic_can_unplug_cpu+0xb0/0xb0 [Fri Apr 30 01:26:11 2021] nmi_trigger_cpumask_backtrace+0xe7/0x130 [Fri Apr 30 01:26:11 2021] arch_trigger_cpumask_backtrace+0x19/0x20 [Fri Apr 30 01:26:11 2021] rcu_dump_cpu_stacks+0x9e/0xce [Fri Apr 30 01:26:11 2021] rcu_sched_clock_irq+0x508/0x770 [Fri Apr 30 01:26:11 2021] ? tick_sched_do_timer+0x60/0x60 [Fri Apr 30 01:26:11 2021] update_process_times+0x28/0x50 [Fri Apr 30 01:26:11 2021] tick_sched_handle+0x25/0x70 [Fri Apr 30 01:26:11 2021] tick_sched_timer+0x3c/0x80 [Fri Apr 30 01:26:11 2021] __hrtimer_run_queues+0x10f/0x280 [Fri Apr 30 01:26:11 2021] hrtimer_interrupt+0xe7/0x240 [Fri Apr 30 01:26:11 2021] smp_apic_timer_interrupt+0x6f/0x130 [Fri Apr 30 01:26:11 2021] apic_timer_interrupt+0xf/0x20 [Fri Apr 30 01:26:11 2021] [Fri Apr 30 01:26:11 2021] RIP: 0010:iommu_unmap_page+0xdf/0x100 [Fri Apr 30 01:26:11 2021] Code: 4c 21 f2 4c 39 fb 4c 8d 34 10 77 91 4d 85 ff 75 23 48 8b 7d d0 65 48 33 3c 25 28 00 00 00 4c 89 f8 75 1c 48 83 c4 10 5b 41 5c <41> 5d 41 5e 41 5f 5d c3 0f 0b 49 8d 47 ff 4c 85 f8 74 d4 0f 0b e8 [Fri Apr 30 01:26:11 2021] RSP: 0018:fffface947c0f950 EFLAGS: 00000282 ORIG_RAX: ffffffffffffff13 [Fri Apr 30 01:26:11 2021] RAX: 0000008000000000 RBX: 00008f36df023000 RCX: 0000000000000027 [Fri Apr 30 01:26:11 2021] RDX: 00008f0000000000 RSI: 0000000000000000 RDI: 0000000000000000 [Fri Apr 30 01:26:11 2021] RBP: fffface947c0f968 R08: fffface947c0f930 R09: 0000000000000004 [Fri Apr 30 01:26:11 2021] R10: 0000000000000000 R11: 000000000000001b R12: ffff9dcbdc371000 [Fri Apr 30 01:26:11 2021] R13: ffff9dcbdc371094 R14: 00008f8000000000 R15: 0000008000000000 [Fri Apr 30 01:26:11 2021] __unmap_single.isra.27+0x66/0x130 [Fri Apr 30 01:26:11 2021] unmap_sg+0x5f/0x70 [Fri Apr 30 01:26:11 2021] nv_unmap_dma_map_scatterlist+0x57/0xa0 [nvidia] [Fri Apr 30 01:26:11 2021] nv_dma_unmap_pages+0x56/0x100 [nvidia] [Fri Apr 30 01:26:11 2021] nv_dma_unmap_alloc+0x16/0x30 [nvidia] [Fri Apr 30 01:26:11 2021] _nv032962rm+0xc1/0x1b0 [nvidia] [Fri Apr 30 01:26:11 2021] ? _nv027787rm+0x9b/0xc0 [nvidia] [Fri Apr 30 01:26:11 2021] ? _nv030989rm+0x124/0x130 [nvidia] [Fri Apr 30 01:26:11 2021] ? rm_gpu_ops_free_duped_handle+0x1c/0x60 [nvidia] [Fri Apr 30 01:26:11 2021] ? nvUvmInterfaceFreeDupedHandle+0x2a/0x40 [nvidia] [Fri Apr 30 01:26:11 2021] ? uvm_ext_gpu_map_free_internal+0x5c/0x70 [nvidia_uvm] [Fri Apr 30 01:26:11 2021] ? uvm_ext_gpu_map_free+0xe/0x20 [nvidia_uvm] [Fri Apr 30 01:26:11 2021] ? uvm_deferred_free_object_list+0x70/0x130 [nvidia_uvm] [Fri Apr 30 01:26:11 2021] ? uvm_va_space_destroy+0x349/0x480 [nvidia_uvm] [Fri Apr 30 01:26:11 2021] ? uvm_release.isra.10+0x7c/0x90 [nvidia_uvm] [Fri Apr 30 01:26:11 2021] ? uvm_release_entry+0x6d/0xa0 [nvidia_uvm] [Fri Apr 30 01:26:11 2021] ? __fput+0xc6/0x260 [Fri Apr 30 01:26:11 2021] ? ____fput+0xe/0x10 [Fri Apr 30 01:26:11 2021] ? task_work_run+0x9d/0xc0 [Fri Apr 30 01:26:11 2021] ? do_exit+0x389/0xb90 [Fri Apr 30 01:26:11 2021] ? do_group_exit+0x43/0xa0 [Fri Apr 30 01:26:11 2021] ? get_signal+0x14f/0x860 [Fri Apr 30 01:26:11 2021] ? do_signal+0x34/0x6d0 [Fri Apr 30 01:26:11 2021] ? __switch_to_asm+0x34/0x70 [Fri Apr 30 01:26:11 2021] ? __x64_sys_futex+0x13f/0x190 [Fri Apr 30 01:26:11 2021] ? exit_to_usermode_loop+0x90/0x130 [Fri Apr 30 01:26:11 2021] ? do_syscall_64+0x170/0x190 [Fri Apr 30 01:26:11 2021] ? entry_SYSCALL_64_after_hwframe+0x44/0xa9 [Fri Apr 30 01:26:35 2021] watchdog: BUG: soft lockup - CPU#40 stuck for 23s! [python3:5432] [Fri Apr 30 01:26:35 2021] Modules linked in: nvidia_uvm(OE) veth xt_nat xt_tcpudp xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype iptable_filter iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 bpfilter br_netfilter bridge stp llc aufs cmac overlay bnep snd_hda_codec_hdmi nvidia_drm(POE) nvidia_modeset(POE) edac_mce_amd snd_usb_audio snd_usbmidi_lib mc kvm_amd kvm iwlmvm mac80211 crct10dif_pclmul crc32_pclmul btusb nls_iso8859_1 btrtl nvidia(POE) libarc4 btbcm ghash_clmulni_intel btintel bluetooth iwlwifi ecdh_generic aesni_intel snd_seq_midi input_leds ecc crypto_simd snd_seq_midi_event cryptd glue_helper cfg80211 wmi_bmof snd_hda_intel snd_rawmidi snd_intel_dspcfg joydev snd_hda_codec nouveau snd_hda_core snd_hwdep snd_seq snd_pcm mxm_wmi video snd_seq_device ccp ttm snd_timer drm_kms_helper snd drm i2c_algo_bit fb_sys_fops syscopyarea sysfillrect sysimgblt k10temp soundcore mac_hid sch_fq_codel parport_pc ppdev lp parport ip_tables x_tables [Fri Apr 30 01:26:35 2021] autofs4 btrfs xor zstd_compress raid6_pq libcrc32c dm_mirror dm_region_hash dm_log hid_generic usbhid hid nvme nvme_core ahci r8169 libahci atlantic realtek i2c_piix4 wmi [Fri Apr 30 01:26:35 2021] CPU: 40 PID: 5432 Comm: python3 Tainted: P W OEL 5.4.0-72-generic #80~18.04.1-Ubuntu [Fri Apr 30 01:26:35 2021] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./TRX40 Creator, BIOS P1.70 05/29/2020 [Fri Apr 30 01:26:35 2021] RIP: 0010:iommu_unmap_page+0x64/0x100 [Fri Apr 30 01:26:35 2021] Code: a9 00 00 00 4c 8d af 94 00 00 00 4c 8d a7 98 00 00 00 49 89 f6 45 31 ff 48 8d 4d c8 4c 89 e6 4c 89 f2 4c 89 ef e8 fc fc ff ff <48> 85 c0 48 89 c6 74 3c 48 b8 8f e3 38 8e e3 38 8e e3 f3 48 0f bc [Fri Apr 30 01:26:35 2021] RSP: 0018:fffface947c0f930 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13 [Fri Apr 30 01:26:35 2021] RAX: 0000000000000000 RBX: 0000000000001000 RCX: 0000000000000027 [Fri Apr 30 01:26:35 2021] RDX: 00008bd1969f0000 RSI: 0000008000000000 RDI: 0000000000000000 [Fri Apr 30 01:26:35 2021] RBP: fffface947c0f968 R08: fffface947c0f930 R09: 0000000000000004 [Fri Apr 30 01:26:35 2021] R10: 0000000000000000 R11: 000000000000001b R12: ffff9dcbdc371098 [Fri Apr 30 01:26:35 2021] R13: ffff9dcbdc371094 R14: 00008bd1969f0000 R15: 0000000000000000 [Fri Apr 30 01:26:35 2021] FS: 00007f9e29fff700(0000) GS:ffff9dcbfd800000(0000) knlGS:0000000000000000 [Fri Apr 30 01:26:35 2021] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fri Apr 30 01:26:35 2021] CR2: 00007f00cbc90150 CR3: 0000001fbb9b0000 CR4: 0000000000340ee0 [Fri Apr 30 01:26:35 2021] Call Trace: [Fri Apr 30 01:26:35 2021] __unmap_single.isra.27+0x66/0x130 [Fri Apr 30 01:26:35 2021] unmap_sg+0x5f/0x70 [Fri Apr 30 01:26:35 2021] nv_unmap_dma_map_scatterlist+0x57/0xa0 [nvidia] [Fri Apr 30 01:26:35 2021] nv_dma_unmap_pages+0x56/0x100 [nvidia] [Fri Apr 30 01:26:35 2021] nv_dma_unmap_alloc+0x16/0x30 [nvidia] [Fri Apr 30 01:26:35 2021] _nv032962rm+0xc1/0x1b0 [nvidia] [Fri Apr 30 01:26:35 2021] ? _nv027787rm+0x9b/0xc0 [nvidia] [Fri Apr 30 01:26:35 2021] ? _nv030989rm+0x124/0x130 [nvidia] [Fri Apr 30 01:26:35 2021] ? rm_gpu_ops_free_duped_handle+0x1c/0x60 [nvidia] [Fri Apr 30 01:26:35 2021] ? nvUvmInterfaceFreeDupedHandle+0x2a/0x40 [nvidia] [Fri Apr 30 01:26:35 2021] ? uvm_ext_gpu_map_free_internal+0x5c/0x70 [nvidia_uvm] [Fri Apr 30 01:26:35 2021] ? uvm_ext_gpu_map_free+0xe/0x20 [nvidia_uvm] [Fri Apr 30 01:26:35 2021] ? uvm_deferred_free_object_list+0x70/0x130 [nvidia_uvm] [Fri Apr 30 01:26:35 2021] ? uvm_va_space_destroy+0x349/0x480 [nvidia_uvm] [Fri Apr 30 01:26:35 2021] ? uvm_release.isra.10+0x7c/0x90 [nvidia_uvm] [Fri Apr 30 01:26:35 2021] ? uvm_release_entry+0x6d/0xa0 [nvidia_uvm] [Fri Apr 30 01:26:35 2021] ? __fput+0xc6/0x260 [Fri Apr 30 01:26:35 2021] ? ____fput+0xe/0x10 [Fri Apr 30 01:26:35 2021] ? task_work_run+0x9d/0xc0 [Fri Apr 30 01:26:35 2021] ? do_exit+0x389/0xb90 [Fri Apr 30 01:26:35 2021] ? do_group_exit+0x43/0xa0 [Fri Apr 30 01:26:35 2021] ? get_signal+0x14f/0x860 [Fri Apr 30 01:26:35 2021] ? do_signal+0x34/0x6d0 [Fri Apr 30 01:26:35 2021] ? __switch_to_asm+0x34/0x70 [Fri Apr 30 01:26:35 2021] ? __x64_sys_futex+0x13f/0x190 [Fri Apr 30 01:26:35 2021] ? exit_to_usermode_loop+0x90/0x130 [Fri Apr 30 01:26:35 2021] ? do_syscall_64+0x170/0x190 [Fri Apr 30 01:26:35 2021] ? entry_SYSCALL_64_after_hwframe+0x44/0xa9 [Fri Apr 30 01:27:03 2021] watchdog: BUG: soft lockup - CPU#40 stuck for 23s! [python3:5432] [Fri Apr 30 01:27:03 2021] Modules linked in: nvidia_uvm(OE) veth xt_nat xt_tcpudp xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype iptable_filter iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 bpfilter br_netfilter bridge stp llc aufs cmac overlay bnep snd_hda_codec_hdmi nvidia_drm(POE) nvidia_modeset(POE) edac_mce_amd snd_usb_audio snd_usbmidi_lib mc kvm_amd kvm iwlmvm mac80211 crct10dif_pclmul crc32_pclmul btusb nls_iso8859_1 btrtl nvidia(POE) libarc4 btbcm ghash_clmulni_intel btintel bluetooth iwlwifi ecdh_generic aesni_intel snd_seq_midi input_leds ecc crypto_simd snd_seq_midi_event cryptd glue_helper cfg80211 wmi_bmof snd_hda_intel snd_rawmidi snd_intel_dspcfg joydev snd_hda_codec nouveau snd_hda_core snd_hwdep snd_seq snd_pcm mxm_wmi video snd_seq_device ccp ttm snd_timer drm_kms_helper snd drm i2c_algo_bit fb_sys_fops syscopyarea sysfillrect sysimgblt k10temp soundcore mac_hid sch_fq_codel parport_pc ppdev lp parport ip_tables x_tables [Fri Apr 30 01:27:03 2021] autofs4 btrfs xor zstd_compress raid6_pq libcrc32c dm_mirror dm_region_hash dm_log hid_generic usbhid hid nvme nvme_core ahci r8169 libahci atlantic realtek i2c_piix4 wmi [Fri Apr 30 01:27:03 2021] CPU: 40 PID: 5432 Comm: python3 Tainted: P W OEL 5.4.0-72-generic #80~18.04.1-Ubuntu [Fri Apr 30 01:27:03 2021] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./TRX40 Creator, BIOS P1.70 05/29/2020 [Fri Apr 30 01:27:03 2021] RIP: 0010:iommu_unmap_page+0x64/0x100 [Fri Apr 30 01:27:03 2021] Code: a9 00 00 00 4c 8d af 94 00 00 00 4c 8d a7 98 00 00 00 49 89 f6 45 31 ff 48 8d 4d c8 4c 89 e6 4c 89 f2 4c 89 ef e8 fc fc ff ff <48> 85 c0 48 89 c6 74 3c 48 b8 8f e3 38 8e e3 38 8e e3 f3 48 0f bc [Fri Apr 30 01:27:03 2021] RSP: 0018:fffface947c0f930 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13 [Fri Apr 30 01:27:03 2021] RAX: 0000000000000000 RBX: 0000000000001000 RCX: 0000000000000027 [Fri Apr 30 01:27:03 2021] RDX: 00008a5fe36e2000 RSI: 0000008000000000 RDI: 0000000000000000 [Fri Apr 30 01:27:03 2021] RBP: fffface947c0f968 R08: fffface947c0f930 R09: 0000000000000004 [Fri Apr 30 01:27:03 2021] R10: 0000000000000000 R11: 000000000000001b R12: ffff9dcbdc371098 [Fri Apr 30 01:27:03 2021] R13: ffff9dcbdc371094 R14: 00008a5fe36e2000 R15: 0000000000000000 [Fri Apr 30 01:27:03 2021] FS: 00007f9e29fff700(0000) GS:ffff9dcbfd800000(0000) knlGS:0000000000000000 [Fri Apr 30 01:27:03 2021] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fri Apr 30 01:27:03 2021] CR2: 00007f00cbc90150 CR3: 0000001fbb9b0000 CR4: 0000000000340ee0 [Fri Apr 30 01:27:03 2021] Call Trace: [Fri Apr 30 01:27:03 2021] __unmap_single.isra.27+0x66/0x130 [Fri Apr 30 01:27:03 2021] unmap_sg+0x5f/0x70 [Fri Apr 30 01:27:03 2021] nv_unmap_dma_map_scatterlist+0x57/0xa0 [nvidia] [Fri Apr 30 01:27:03 2021] nv_dma_unmap_pages+0x56/0x100 [nvidia] [Fri Apr 30 01:27:03 2021] nv_dma_unmap_alloc+0x16/0x30 [nvidia] [Fri Apr 30 01:27:03 2021] _nv032962rm+0xc1/0x1b0 [nvidia] [Fri Apr 30 01:27:03 2021] ? _nv027787rm+0x9b/0xc0 [nvidia] [Fri Apr 30 01:27:03 2021] ? _nv030989rm+0x124/0x130 [nvidia] [Fri Apr 30 01:27:03 2021] ? rm_gpu_ops_free_duped_handle+0x1c/0x60 [nvidia] [Fri Apr 30 01:27:03 2021] ? nvUvmInterfaceFreeDupedHandle+0x2a/0x40 [nvidia] [Fri Apr 30 01:27:03 2021] ? uvm_ext_gpu_map_free_internal+0x5c/0x70 [nvidia_uvm] [Fri Apr 30 01:27:03 2021] ? uvm_ext_gpu_map_free+0xe/0x20 [nvidia_uvm] [Fri Apr 30 01:27:03 2021] ? uvm_deferred_free_object_list+0x70/0x130 [nvidia_uvm] [Fri Apr 30 01:27:03 2021] ? uvm_va_space_destroy+0x349/0x480 [nvidia_uvm] [Fri Apr 30 01:27:03 2021] ? uvm_release.isra.10+0x7c/0x90 [nvidia_uvm] [Fri Apr 30 01:27:03 2021] ? uvm_release_entry+0x6d/0xa0 [nvidia_uvm] [Fri Apr 30 01:27:03 2021] ? __fput+0xc6/0x260 [Fri Apr 30 01:27:03 2021] ? ____fput+0xe/0x10 [Fri Apr 30 01:27:03 2021] ? task_work_run+0x9d/0xc0 [Fri Apr 30 01:27:03 2021] ? do_exit+0x389/0xb90 [Fri Apr 30 01:27:03 2021] ? do_group_exit+0x43/0xa0 [Fri Apr 30 01:27:03 2021] ? get_signal+0x14f/0x860 [Fri Apr 30 01:27:03 2021] ? do_signal+0x34/0x6d0 [Fri Apr 30 01:27:03 2021] ? __switch_to_asm+0x34/0x70 [Fri Apr 30 01:27:03 2021] ? __x64_sys_futex+0x13f/0x190 [Fri Apr 30 01:27:03 2021] ? exit_to_usermode_loop+0x90/0x130 [Fri Apr 30 01:27:03 2021] ? do_syscall_64+0x170/0x190 [Fri Apr 30 01:27:03 2021] ? entry_SYSCALL_64_after_hwframe+0x44/0xa9 [Fri Apr 30 01:27:25 2021] iwlwifi 0000:45:00.0: Error sending SCAN_REQ_UMAC: time out after 2000ms. [Fri Apr 30 01:27:25 2021] iwlwifi 0000:45:00.0: Current CMD queue read_ptr 50 write_ptr 51 [Fri Apr 30 01:27:25 2021] iwlwifi 0000:45:00.0: Start IWL Error Log Dump: [Fri Apr 30 01:27:25 2021] iwlwifi 0000:45:00.0: Status: 0x00000040, count: 6 [Fri Apr 30 01:27:25 2021] iwlwifi 0000:45:00.0: Loaded firmware version: 48.4fa0041f.0 [Fri Apr 30 01:27:25 2021] iwlwifi 0000:45:00.0: 0x00000084 | NMI_INTERRUPT_UNKNOWN [Fri Apr 30 01:27:25 2021] iwlwifi 0000:45:00.0: 0x0080A210 | trm_hw_status0 [Fri Apr 30 01:27:25 2021] iwlwifi 0000:45:00.0: 0x00000000 | trm_hw_status1 [Fri Apr 30 01:27:25 2021] iwlwifi 0000:45:00.0: 0x004F8E3C | branchlink2 [Fri Apr 30 01:27:25 2021] iwlwifi 0000:45:00.0: 0x004E4FF4 | interruptlink1 [Fri Apr 30 01:27:25 2021] iwlwifi 0000:45:00.0: 0x004E4FF4 | interruptlink2 [Fri Apr 30 01:27:25 2021] iwlwifi 0000:45:00.0: 0x004F4AD2 | data1 [Fri Apr 30 01:27:25 2021] iwlwifi 0000:45:00.0: 0xFF000000 | data2 [Fri Apr 30 01:27:25 2021] iwlwifi 0000:45:00.0: 0xF0000008 | data3 [Fri Apr 30 01:27:25 2021] iwlwifi 0000:45:00.0: 0x00000000 | beacon time [Fri Apr 30 01:27:25 2021] iwlwifi 0000:45:00.0: 0x07FCF03F | tsf low [Fri Apr 30 01:27:25 2021] iwlwifi 0000:45:00.0: 0x00000000 | tsf hi [Fri Apr 30 01:27:25 2021] iwlwifi 0000:45:00.0: 0x00000000 | time gp1 [Fri Apr 30 01:27:25 2021] iwlwifi 0000:45:00.0: 0x07FD5121 | time gp2 [Fri Apr 30 01:27:25 2021] iwlwifi 0000:45:00.0: 0x00000001 | uCode revision type [Fri Apr 30 01:27:25 2021] iwlwifi 0000:45:00.0: 0x00000030 | uCode version major [Fri Apr 30 01:27:25 2021] iwlwifi 0000:45:00.0: 0x4FA0041F | uCode version minor [Fri Apr 30 01:27:25 2021] iwlwifi 0000:45:00.0: 0x00000340 | hw version [Fri Apr 30 01:27:25 2021] iwlwifi 0000:45:00.0: 0x00C89000 | board version [Fri Apr 30 01:27:25 2021] iwlwifi 0000:45:00.0: 0x80C7FD04 | hcmd [Fri Apr 30 01:27:25 2021] iwlwifi 0000:45:00.0: 0x24020000 | isr0 [Fri Apr 30 01:27:25 2021] iwlwifi 0000:45:00.0: 0x01000000 | isr1 [Fri Apr 30 01:27:25 2021] iwlwifi 0000:45:00.0: 0x08F84802 | isr2 [Fri Apr 30 01:27:25 2021] iwlwifi 0000:45:00.0: 0x04C1FFCC | isr3 [Fri Apr 30 01:27:25 2021] iwlwifi 0000:45:00.0: 0x00000000 | isr4 [Fri Apr 30 01:27:25 2021] iwlwifi 0000:45:00.0: 0x0031019C | last cmd Id [Fri Apr 30 01:27:25 2021] iwlwifi 0000:45:00.0: 0x004F4AD2 | wait_event [Fri Apr 30 01:27:25 2021] iwlwifi 0000:45:00.0: 0x00000080 | l2p_control [Fri Apr 30 01:27:25 2021] iwlwifi 0000:45:00.0: 0x00010014 | l2p_duration [Fri Apr 30 01:27:25 2021] iwlwifi 0000:45:00.0: 0x0000003F | l2p_mhvalid [Fri Apr 30 01:27:25 2021] iwlwifi 0000:45:00.0: 0x00000000 | l2p_addr_match [Fri Apr 30 01:27:25 2021] iwlwifi 0000:45:00.0: 0x00000008 | lmpm_pmg_sel [Fri Apr 30 01:27:25 2021] iwlwifi 0000:45:00.0: 0x00000000 | timestamp [Fri Apr 30 01:27:25 2021] iwlwifi 0000:45:00.0: 0x000048B8 | flow_handler [Fri Apr 30 01:27:25 2021] iwlwifi 0000:45:00.0: Start IWL Error Log Dump: [Fri Apr 30 01:27:25 2021] iwlwifi 0000:45:00.0: Status: 0x00000040, count: 7 [Fri Apr 30 01:27:25 2021] iwlwifi 0000:45:00.0: 0x20000066 | NMI_INTERRUPT_HOST [Fri Apr 30 01:27:25 2021] iwlwifi 0000:45:00.0: 0x00000000 | umac branchlink1 [Fri Apr 30 01:27:25 2021] iwlwifi 0000:45:00.0: 0xC008CC3C | umac branchlink2 [Fri Apr 30 01:27:25 2021] iwlwifi 0000:45:00.0: 0xC008609C | umac interruptlink1 [Fri Apr 30 01:27:25 2021] iwlwifi 0000:45:00.0: 0xC00875EC | umac interruptlink2 [Fri Apr 30 01:27:25 2021] iwlwifi 0000:45:00.0: 0x01000000 | umac data1 [Fri Apr 30 01:27:25 2021] iwlwifi 0000:45:00.0: 0xC00875EC | umac data2 [Fri Apr 30 01:27:25 2021] iwlwifi 0000:45:00.0: 0x00000000 | umac data3 [Fri Apr 30 01:27:25 2021] iwlwifi 0000:45:00.0: 0x00000030 | umac major [Fri Apr 30 01:27:25 2021] iwlwifi 0000:45:00.0: 0x4FA0041F | umac minor [Fri Apr 30 01:27:25 2021] iwlwifi 0000:45:00.0: 0x07FD511A | frame pointer [Fri Apr 30 01:27:25 2021] iwlwifi 0000:45:00.0: 0xC0887F0C | stack pointer [Fri Apr 30 01:27:25 2021] iwlwifi 0000:45:00.0: 0x0032010D | last host cmd [Fri Apr 30 01:27:25 2021] iwlwifi 0000:45:00.0: 0x00000009 | isr status reg [Fri Apr 30 01:27:25 2021] iwlwifi 0000:45:00.0: Fseq Registers: [Fri Apr 30 01:27:25 2021] iwlwifi 0000:45:00.0: 0xE0000101 | FSEQ_ERROR_CODE [Fri Apr 30 01:27:25 2021] iwlwifi 0000:45:00.0: 0x00290006 | FSEQ_TOP_INIT_VERSION [Fri Apr 30 01:27:25 2021] iwlwifi 0000:45:00.0: 0x80050008 | FSEQ_CNVIO_INIT_VERSION [Fri Apr 30 01:27:25 2021] iwlwifi 0000:45:00.0: 0x0000A503 | FSEQ_OTP_VERSION [Fri Apr 30 01:27:25 2021] iwlwifi 0000:45:00.0: 0x80000003 | FSEQ_TOP_CONTENT_VERSION [Fri Apr 30 01:27:25 2021] iwlwifi 0000:45:00.0: 0x4552414E | FSEQ_ALIVE_TOKEN [Fri Apr 30 01:27:25 2021] iwlwifi 0000:45:00.0: 0x00100530 | FSEQ_CNVI_ID [Fri Apr 30 01:27:25 2021] iwlwifi 0000:45:00.0: 0x00000532 | FSEQ_CNVR_ID [Fri Apr 30 01:27:25 2021] iwlwifi 0000:45:00.0: 0x00100530 | CNVI_AUX_MISC_CHIP [Fri Apr 30 01:27:25 2021] iwlwifi 0000:45:00.0: 0x00000532 | CNVR_AUX_MISC_CHIP [Fri Apr 30 01:27:25 2021] iwlwifi 0000:45:00.0: 0x05B0905B | CNVR_SCU_SD_REGS_SD_REG_DIG_DCDC_VTRIM [Fri Apr 30 01:27:25 2021] iwlwifi 0000:45:00.0: 0x0000025B | CNVR_SCU_SD_REGS_SD_REG_ACTIVE_VDIG_MIRROR [Fri Apr 30 01:27:25 2021] iwlwifi 0000:45:00.0: Collecting data: trigger 2 fired. [Fri Apr 30 01:27:25 2021] ieee80211 phy0: Hardware restart was requested [Fri Apr 30 01:27:25 2021] iwlwifi 0000:45:00.0: Scan failed! ret -110 [Fri Apr 30 01:27:25 2021] ------------[ cut here ]------------ [Fri Apr 30 01:27:25 2021] WARNING: CPU: 0 PID: 4036 at /build/linux-hwe-5.4-PtAV0J/linux-hwe-5.4-5.4.0/net/mac80211/scan.c:395 __ieee80211_scan_completed+0x2e6/0x3b0 [mac80211] [Fri Apr 30 01:27:25 2021] Modules linked in: nvidia_uvm(OE) veth xt_nat xt_tcpudp xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype iptable_filter iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 bpfilter br_netfilter bridge stp llc aufs cmac overlay bnep snd_hda_codec_hdmi nvidia_drm(POE) nvidia_modeset(POE) edac_mce_amd snd_usb_audio snd_usbmidi_lib mc kvm_amd kvm iwlmvm mac80211 crct10dif_pclmul crc32_pclmul btusb nls_iso8859_1 btrtl nvidia(POE) libarc4 btbcm ghash_clmulni_intel btintel bluetooth iwlwifi ecdh_generic aesni_intel snd_seq_midi input_leds ecc crypto_simd snd_seq_midi_event cryptd glue_helper cfg80211 wmi_bmof snd_hda_intel snd_rawmidi snd_intel_dspcfg joydev snd_hda_codec nouveau snd_hda_core snd_hwdep snd_seq snd_pcm mxm_wmi video snd_seq_device ccp ttm snd_timer drm_kms_helper snd drm i2c_algo_bit fb_sys_fops syscopyarea sysfillrect sysimgblt k10temp soundcore mac_hid sch_fq_codel parport_pc ppdev lp parport ip_tables x_tables [Fri Apr 30 01:27:25 2021] autofs4 btrfs xor zstd_compress raid6_pq libcrc32c dm_mirror dm_region_hash dm_log hid_generic usbhid hid nvme nvme_core ahci r8169 libahci atlantic realtek i2c_piix4 wmi [Fri Apr 30 01:27:25 2021] CPU: 0 PID: 4036 Comm: kworker/u256:0 Tainted: P W OEL 5.4.0-72-generic #80~18.04.1-Ubuntu [Fri Apr 30 01:27:25 2021] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./TRX40 Creator, BIOS P1.70 05/29/2020 [Fri Apr 30 01:27:25 2021] Workqueue: phy0 ieee80211_scan_work [mac80211] [Fri Apr 30 01:27:25 2021] RIP: 0010:__ieee80211_scan_completed+0x2e6/0x3b0 [mac80211] [Fri Apr 30 01:27:25 2021] Code: 1d b7 84 09 00 48 85 db 74 1b 48 8b 03 48 8b 7b 08 48 83 c3 18 4c 89 e6 e8 97 c3 72 cc 48 8b 03 48 85 c0 75 e8 e9 dc fe ff ff <0f> 0b 48 83 c4 08 5b 41 5c 41 5d 41 5e 41 5f 5d c3 65 8b 05 c2 b9 [Fri Apr 30 01:27:25 2021] RSP: 0018:fffface947427df0 EFLAGS: 00010246 [Fri Apr 30 01:27:25 2021] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000 [Fri Apr 30 01:27:25 2021] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9dcbbbe987a0 [Fri Apr 30 01:27:25 2021] RBP: fffface947427e20 R08: ffff9dcbbbe99908 R09: 8080808080808080 [Fri Apr 30 01:27:25 2021] R10: fffface947427e08 R11: fefefefefefefeff R12: ffff9dcbbbe987a0 [Fri Apr 30 01:27:25 2021] R13: 0000000000000001 R14: 0000000000000000 R15: 0ffff9dcbcf33411 [Fri Apr 30 01:27:25 2021] FS: 0000000000000000(0000) GS:ffff9dcbfce00000(0000) knlGS:0000000000000000 [Fri Apr 30 01:27:25 2021] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fri Apr 30 01:27:25 2021] CR2: 000055557d315d80 CR3: 0000001f9bf1c000 CR4: 0000000000340ef0 [Fri Apr 30 01:27:25 2021] Call Trace: [Fri Apr 30 01:27:25 2021] ieee80211_scan_work+0x6c/0x4d0 [mac80211] [Fri Apr 30 01:27:25 2021] process_one_work+0x20f/0x400 [Fri Apr 30 01:27:25 2021] worker_thread+0x34/0x410 [Fri Apr 30 01:27:25 2021] kthread+0x121/0x140 [Fri Apr 30 01:27:25 2021] ? process_one_work+0x400/0x400 [Fri Apr 30 01:27:25 2021] ? kthread_park+0x90/0x90 [Fri Apr 30 01:27:25 2021] ret_from_fork+0x22/0x40 [Fri Apr 30 01:27:25 2021] ---[ end trace c3ee37f01c9a84eb ]--- [Fri Apr 30 01:27:35 2021] watchdog: BUG: soft lockup - CPU#16 stuck for 22s! [python3:5432] [Fri Apr 30 01:27:35 2021] Modules linked in: nvidia_uvm(OE) veth xt_nat xt_tcpudp xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype iptable_filter iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 bpfilter br_netfilter bridge stp llc aufs cmac overlay bnep snd_hda_codec_hdmi nvidia_drm(POE) nvidia_modeset(POE) edac_mce_amd snd_usb_audio snd_usbmidi_lib mc kvm_amd kvm iwlmvm mac80211 crct10dif_pclmul crc32_pclmul btusb nls_iso8859_1 btrtl nvidia(POE) libarc4 btbcm ghash_clmulni_intel btintel bluetooth iwlwifi ecdh_generic aesni_intel snd_seq_midi input_leds ecc crypto_simd snd_seq_midi_event cryptd glue_helper cfg80211 wmi_bmof snd_hda_intel snd_rawmidi snd_intel_dspcfg joydev snd_hda_codec nouveau snd_hda_core snd_hwdep snd_seq snd_pcm mxm_wmi video snd_seq_device ccp ttm snd_timer drm_kms_helper snd drm i2c_algo_bit fb_sys_fops syscopyarea sysfillrect sysimgblt k10temp soundcore mac_hid sch_fq_codel parport_pc ppdev lp parport ip_tables x_tables [Fri Apr 30 01:27:35 2021] autofs4 btrfs xor zstd_compress raid6_pq libcrc32c dm_mirror dm_region_hash dm_log hid_generic usbhid hid nvme nvme_core ahci r8169 libahci atlantic realtek i2c_piix4 wmi [Fri Apr 30 01:27:35 2021] CPU: 16 PID: 5432 Comm: python3 Tainted: P W OEL 5.4.0-72-generic #80~18.04.1-Ubuntu [Fri Apr 30 01:27:35 2021] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./TRX40 Creator, BIOS P1.70 05/29/2020 [Fri Apr 30 01:27:35 2021] RIP: 0010:iommu_unmap_page+0xbf/0x100 [Fri Apr 30 01:27:35 2021] Code: 48 d3 e2 48 c7 04 c6 00 00 00 00 48 83 c0 01 39 c2 7f f0 48 8b 45 c8 48 89 c2 49 01 c7 48 f7 da 4c 21 f2 4c 39 fb 4c 8d 34 10 <77> 91 4d 85 ff 75 23 48 8b 7d d0 65 48 33 3c 25 28 00 00 00 4c 89 [Fri Apr 30 01:27:35 2021] RSP: 0018:fffface947c0f930 EFLAGS: 00000287 ORIG_RAX: ffffffffffffff13 [Fri Apr 30 01:27:35 2021] RAX: 0000008000000000 RBX: 0000000000001000 RCX: 0000000000000027 [Fri Apr 30 01:27:35 2021] RDX: 00008b0000000000 RSI: 0000000000000000 RDI: 0000000000000000 [Fri Apr 30 01:27:35 2021] RBP: fffface947c0f968 R08: fffface947c0f930 R09: 0000000000000004 [Fri Apr 30 01:27:35 2021] R10: 0000000000000000 R11: 000000000000001b R12: ffff9dcbdc370098 [Fri Apr 30 01:27:35 2021] R13: ffff9dcbdc370094 R14: 00008b8000000000 R15: 0000008000000000 [Fri Apr 30 01:27:35 2021] FS: 00007f9e29fff700(0000) GS:ffff9dcbfd200000(0000) knlGS:0000000000000000 [Fri Apr 30 01:27:35 2021] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fri Apr 30 01:27:35 2021] CR2: 00007fb3cf8144d0 CR3: 000000200f28c000 CR4: 0000000000340ee0 [Fri Apr 30 01:27:35 2021] Call Trace: [Fri Apr 30 01:27:35 2021] __unmap_single.isra.27+0x66/0x130 [Fri Apr 30 01:27:35 2021] unmap_sg+0x5f/0x70 [Fri Apr 30 01:27:35 2021] nv_unmap_dma_map_scatterlist+0x57/0xa0 [nvidia] [Fri Apr 30 01:27:35 2021] nv_dma_unmap_pages+0x56/0x100 [nvidia] [Fri Apr 30 01:27:35 2021] nv_dma_unmap_alloc+0x16/0x30 [nvidia] [Fri Apr 30 01:27:35 2021] _nv032962rm+0xc1/0x1b0 [nvidia] [Fri Apr 30 01:27:35 2021] ? _nv027787rm+0x9b/0xc0 [nvidia] [Fri Apr 30 01:27:35 2021] ? _nv030989rm+0x124/0x130 [nvidia] [Fri Apr 30 01:27:35 2021] ? rm_gpu_ops_free_duped_handle+0x1c/0x60 [nvidia] [Fri Apr 30 01:27:35 2021] ? nvUvmInterfaceFreeDupedHandle+0x2a/0x40 [nvidia] [Fri Apr 30 01:27:35 2021] ? uvm_ext_gpu_map_free_internal+0x5c/0x70 [nvidia_uvm] [Fri Apr 30 01:27:35 2021] ? uvm_ext_gpu_map_free+0xe/0x20 [nvidia_uvm] [Fri Apr 30 01:27:35 2021] ? uvm_deferred_free_object_list+0x70/0x130 [nvidia_uvm] [Fri Apr 30 01:27:35 2021] ? uvm_va_space_destroy+0x349/0x480 [nvidia_uvm] [Fri Apr 30 01:27:35 2021] ? uvm_release.isra.10+0x7c/0x90 [nvidia_uvm] [Fri Apr 30 01:27:35 2021] ? uvm_release_entry+0x6d/0xa0 [nvidia_uvm] [Fri Apr 30 01:27:35 2021] ? __fput+0xc6/0x260 [Fri Apr 30 01:27:35 2021] ? ____fput+0xe/0x10 [Fri Apr 30 01:27:35 2021] ? task_work_run+0x9d/0xc0 [Fri Apr 30 01:27:35 2021] ? do_exit+0x389/0xb90 [Fri Apr 30 01:27:35 2021] ? do_group_exit+0x43/0xa0 [Fri Apr 30 01:27:35 2021] ? get_signal+0x14f/0x860 [Fri Apr 30 01:27:35 2021] ? do_signal+0x34/0x6d0 [Fri Apr 30 01:27:35 2021] ? __switch_to_asm+0x34/0x70 [Fri Apr 30 01:27:35 2021] ? __x64_sys_futex+0x13f/0x190 [Fri Apr 30 01:27:35 2021] ? exit_to_usermode_loop+0x90/0x130 [Fri Apr 30 01:27:35 2021] ? do_syscall_64+0x170/0x190 [Fri Apr 30 01:27:35 2021] ? entry_SYSCALL_64_after_hwframe+0x44/0xa9 [Fri Apr 30 01:28:03 2021] watchdog: BUG: soft lockup - CPU#16 stuck for 22s! [python3:5432] [Fri Apr 30 01:28:03 2021] Modules linked in: nvidia_uvm(OE) veth xt_nat xt_tcpudp xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype iptable_filter iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 bpfilter br_netfilter bridge stp llc aufs cmac overlay bnep snd_hda_codec_hdmi nvidia_drm(POE) nvidia_modeset(POE) edac_mce_amd snd_usb_audio snd_usbmidi_lib mc kvm_amd kvm iwlmvm mac80211 crct10dif_pclmul crc32_pclmul btusb nls_iso8859_1 btrtl nvidia(POE) libarc4 btbcm ghash_clmulni_intel btintel bluetooth iwlwifi ecdh_generic aesni_intel snd_seq_midi input_leds ecc crypto_simd snd_seq_midi_event cryptd glue_helper cfg80211 wmi_bmof snd_hda_intel snd_rawmidi snd_intel_dspcfg joydev snd_hda_codec nouveau snd_hda_core snd_hwdep snd_seq snd_pcm mxm_wmi video snd_seq_device ccp ttm snd_timer drm_kms_helper snd drm i2c_algo_bit fb_sys_fops syscopyarea sysfillrect sysimgblt k10temp soundcore mac_hid sch_fq_codel parport_pc ppdev lp parport ip_tables x_tables [Fri Apr 30 01:28:03 2021] autofs4 btrfs xor zstd_compress raid6_pq libcrc32c dm_mirror dm_region_hash dm_log hid_generic usbhid hid nvme nvme_core ahci r8169 libahci atlantic realtek i2c_piix4 wmi [Fri Apr 30 01:28:03 2021] CPU: 16 PID: 5432 Comm: python3 Tainted: P W OEL 5.4.0-72-generic #80~18.04.1-Ubuntu [Fri Apr 30 01:28:03 2021] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./TRX40 Creator, BIOS P1.70 05/29/2020 [Fri Apr 30 01:28:03 2021] RIP: 0010:iommu_unmap_page+0xd/0x100 [Fri Apr 30 01:28:03 2021] Code: 48 c1 ea 0c e8 a4 da ff ff 48 85 c0 74 02 5d c3 0f 0b 5d c3 e8 a4 ff ff ff 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 41 57 41 56 <41> 55 41 54 53 48 83 ec 10 65 48 8b 04 25 28 00 00 00 48 89 45 d0 [Fri Apr 30 01:28:03 2021] RSP: 0018:fffface947c0f958 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [Fri Apr 30 01:28:03 2021] RAX: 0000008000000000 RBX: 000089a56d4a0000 RCX: 0000000000000027 [Fri Apr 30 01:28:03 2021] RDX: 0000000000001000 RSI: 000089a56d49f000 RDI: ffff9dcbdc370000 [Fri Apr 30 01:28:03 2021] RBP: fffface947c0f968 R08: fffface947c0f930 R09: 0000000000000004 [Fri Apr 30 01:28:03 2021] R10: 0000000000000000 R11: 000000000000001b R12: ffff9dcbdc370000 [Fri Apr 30 01:28:03 2021] R13: 00008ff9fffff000 R14: 00000000ffffffff R15: 00007ffa00000000 [Fri Apr 30 01:28:03 2021] FS: 00007f9e29fff700(0000) GS:ffff9dcbfd200000(0000) knlGS:0000000000000000 [Fri Apr 30 01:28:03 2021] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fri Apr 30 01:28:03 2021] CR2: 00007fb3cf8144d0 CR3: 000000200f28c000 CR4: 0000000000340ee0 [Fri Apr 30 01:28:03 2021] Call Trace: [Fri Apr 30 01:28:03 2021] __unmap_single.isra.27+0x66/0x130 [Fri Apr 30 01:28:03 2021] unmap_sg+0x5f/0x70 [Fri Apr 30 01:28:03 2021] nv_unmap_dma_map_scatterlist+0x57/0xa0 [nvidia] [Fri Apr 30 01:28:03 2021] nv_dma_unmap_pages+0x56/0x100 [nvidia] [Fri Apr 30 01:28:03 2021] nv_dma_unmap_alloc+0x16/0x30 [nvidia] [Fri Apr 30 01:28:03 2021] _nv032962rm+0xc1/0x1b0 [nvidia] [Fri Apr 30 01:28:03 2021] ? _nv027787rm+0x9b/0xc0 [nvidia] [Fri Apr 30 01:28:03 2021] ? _nv030989rm+0x124/0x130 [nvidia] [Fri Apr 30 01:28:03 2021] ? rm_gpu_ops_free_duped_handle+0x1c/0x60 [nvidia] [Fri Apr 30 01:28:03 2021] ? nvUvmInterfaceFreeDupedHandle+0x2a/0x40 [nvidia] [Fri Apr 30 01:28:03 2021] ? uvm_ext_gpu_map_free_internal+0x5c/0x70 [nvidia_uvm] [Fri Apr 30 01:28:03 2021] ? uvm_ext_gpu_map_free+0xe/0x20 [nvidia_uvm] [Fri Apr 30 01:28:03 2021] ? uvm_deferred_free_object_list+0x70/0x130 [nvidia_uvm] [Fri Apr 30 01:28:03 2021] ? uvm_va_space_destroy+0x349/0x480 [nvidia_uvm] [Fri Apr 30 01:28:03 2021] ? uvm_release.isra.10+0x7c/0x90 [nvidia_uvm] [Fri Apr 30 01:28:03 2021] ? uvm_release_entry+0x6d/0xa0 [nvidia_uvm] [Fri Apr 30 01:28:03 2021] ? __fput+0xc6/0x260 [Fri Apr 30 01:28:03 2021] ? ____fput+0xe/0x10 [Fri Apr 30 01:28:03 2021] ? task_work_run+0x9d/0xc0 [Fri Apr 30 01:28:03 2021] ? do_exit+0x389/0xb90 [Fri Apr 30 01:28:03 2021] ? do_group_exit+0x43/0xa0 [Fri Apr 30 01:28:03 2021] ? get_signal+0x14f/0x860 [Fri Apr 30 01:28:03 2021] ? do_signal+0x34/0x6d0 [Fri Apr 30 01:28:03 2021] ? __switch_to_asm+0x34/0x70 [Fri Apr 30 01:28:03 2021] ? __x64_sys_futex+0x13f/0x190 [Fri Apr 30 01:28:03 2021] ? exit_to_usermode_loop+0x90/0x130 [Fri Apr 30 01:28:03 2021] ? do_syscall_64+0x170/0x190 [Fri Apr 30 01:28:03 2021] ? entry_SYSCALL_64_after_hwframe+0x44/0xa9 [Fri Apr 30 01:28:14 2021] rcu: INFO: rcu_sched self-detected stall on CPU [Fri Apr 30 01:28:14 2021] rcu: 16-....: (14999 ticks this GP) idle=502/1/0x4000000000000002 softirq=8622/8622 fqs=7499 [Fri Apr 30 01:28:14 2021] (t=15000 jiffies g=47421 q=492876) [Fri Apr 30 01:28:14 2021] NMI backtrace for cpu 16 [Fri Apr 30 01:28:14 2021] CPU: 16 PID: 5432 Comm: python3 Tainted: P W OEL 5.4.0-72-generic #80~18.04.1-Ubuntu [Fri Apr 30 01:28:14 2021] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./TRX40 Creator, BIOS P1.70 05/29/2020 [Fri Apr 30 01:28:14 2021] Call Trace: [Fri Apr 30 01:28:14 2021] [Fri Apr 30 01:28:14 2021] dump_stack+0x6d/0x8b [Fri Apr 30 01:28:14 2021] nmi_cpu_backtrace+0x94/0xa0 [Fri Apr 30 01:28:14 2021] ? lapic_can_unplug_cpu+0xb0/0xb0 [Fri Apr 30 01:28:14 2021] nmi_trigger_cpumask_backtrace+0xe7/0x130 [Fri Apr 30 01:28:14 2021] arch_trigger_cpumask_backtrace+0x19/0x20 [Fri Apr 30 01:28:14 2021] rcu_dump_cpu_stacks+0x9e/0xce [Fri Apr 30 01:28:14 2021] rcu_sched_clock_irq+0x508/0x770 [Fri Apr 30 01:28:14 2021] ? tick_sched_do_timer+0x60/0x60 [Fri Apr 30 01:28:14 2021] update_process_times+0x28/0x50 [Fri Apr 30 01:28:14 2021] tick_sched_handle+0x25/0x70 [Fri Apr 30 01:28:14 2021] ? tick_sched_do_timer+0x58/0x60 [Fri Apr 30 01:28:14 2021] tick_sched_timer+0x3c/0x80 [Fri Apr 30 01:28:14 2021] __hrtimer_run_queues+0x10f/0x280 [Fri Apr 30 01:28:14 2021] hrtimer_interrupt+0xe7/0x240 [Fri Apr 30 01:28:14 2021] smp_apic_timer_interrupt+0x6f/0x130 [Fri Apr 30 01:28:14 2021] apic_timer_interrupt+0xf/0x20 [Fri Apr 30 01:28:14 2021] [Fri Apr 30 01:28:14 2021] RIP: 0010:fetch_pte.isra.7+0x3e/0x1a0 [Fri Apr 30 01:28:14 2021] Code: f9 05 7f 1a 43 8d 4c c9 0c b8 01 00 00 00 48 d3 e0 48 83 e8 01 48 39 d0 0f 82 5c 01 00 00 41 8d 41 ff 55 48 89 d7 44 8d 1c c0 <48> 89 e5 53 48 8b 36 41 8d 4b 0c 48 d3 ef 81 e7 ff 01 00 00 4c 8d [Fri Apr 30 01:28:14 2021] RSP: 0018:fffface947c0f920 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [Fri Apr 30 01:28:14 2021] RAX: 0000000000000003 RBX: 0000000000001000 RCX: 0000000000000030 [Fri Apr 30 01:28:14 2021] RDX: 00008f484ac4f000 RSI: ffff9dcbdc370098 RDI: 00008f484ac4f000 [Fri Apr 30 01:28:14 2021] RBP: fffface947c0f968 R08: fffface947c0f930 R09: 0000000000000004 [Fri Apr 30 01:28:14 2021] R10: 0000000000000000 R11: 000000000000001b R12: ffff9dcbdc370098 [Fri Apr 30 01:28:14 2021] R13: ffff9dcbdc370094 R14: 00008f484ac4f000 R15: 0000000000000000 [Fri Apr 30 01:28:14 2021] ? iommu_unmap_page+0x64/0x100 [Fri Apr 30 01:28:14 2021] __unmap_single.isra.27+0x66/0x130 [Fri Apr 30 01:28:14 2021] unmap_sg+0x5f/0x70 [Fri Apr 30 01:28:14 2021] nv_unmap_dma_map_scatterlist+0x57/0xa0 [nvidia] [Fri Apr 30 01:28:14 2021] nv_dma_unmap_pages+0x56/0x100 [nvidia] [Fri Apr 30 01:28:14 2021] nv_dma_unmap_alloc+0x16/0x30 [nvidia] [Fri Apr 30 01:28:14 2021] _nv032962rm+0xc1/0x1b0 [nvidia] [Fri Apr 30 01:28:14 2021] ? _nv027787rm+0x9b/0xc0 [nvidia] [Fri Apr 30 01:28:14 2021] ? _nv030989rm+0x124/0x130 [nvidia] [Fri Apr 30 01:28:14 2021] ? rm_gpu_ops_free_duped_handle+0x1c/0x60 [nvidia] [Fri Apr 30 01:28:14 2021] ? nvUvmInterfaceFreeDupedHandle+0x2a/0x40 [nvidia] [Fri Apr 30 01:28:14 2021] ? uvm_ext_gpu_map_free_internal+0x5c/0x70 [nvidia_uvm] [Fri Apr 30 01:28:14 2021] ? uvm_ext_gpu_map_free+0xe/0x20 [nvidia_uvm] [Fri Apr 30 01:28:14 2021] ? uvm_deferred_free_object_list+0x70/0x130 [nvidia_uvm] [Fri Apr 30 01:28:14 2021] ? uvm_va_space_destroy+0x349/0x480 [nvidia_uvm] [Fri Apr 30 01:28:14 2021] ? uvm_release.isra.10+0x7c/0x90 [nvidia_uvm] [Fri Apr 30 01:28:14 2021] ? uvm_release_entry+0x6d/0xa0 [nvidia_uvm] [Fri Apr 30 01:28:14 2021] ? __fput+0xc6/0x260 [Fri Apr 30 01:28:14 2021] ? ____fput+0xe/0x10 [Fri Apr 30 01:28:14 2021] ? task_work_run+0x9d/0xc0 [Fri Apr 30 01:28:14 2021] ? do_exit+0x389/0xb90 [Fri Apr 30 01:28:14 2021] ? do_group_exit+0x43/0xa0 [Fri Apr 30 01:28:14 2021] ? get_signal+0x14f/0x860 [Fri Apr 30 01:28:14 2021] ? do_signal+0x34/0x6d0 [Fri Apr 30 01:28:14 2021] ? __switch_to_asm+0x34/0x70 [Fri Apr 30 01:28:14 2021] ? __x64_sys_futex+0x13f/0x190 [Fri Apr 30 01:28:14 2021] ? exit_to_usermode_loop+0x90/0x130 [Fri Apr 30 01:28:14 2021] ? do_syscall_64+0x170/0x190 [Fri Apr 30 01:28:14 2021] ? entry_SYSCALL_64_after_hwframe+0x44/0xa9 [Fri Apr 30 01:28:26 2021] rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 16-... } 15117 jiffies s: 905 root: 0x2/. [Fri Apr 30 01:28:26 2021] rcu: blocking rcu_node structures: l=1:16-31:0x1/. [Fri Apr 30 01:28:26 2021] Task dump for CPU 16: [Fri Apr 30 01:28:26 2021] python3 R running task 0 5432 4271 0x8000400a [Fri Apr 30 01:28:26 2021] Call Trace: [Fri Apr 30 01:28:26 2021] ? apic_timer_interrupt+0xa/0x20 [Fri Apr 30 01:28:26 2021] ? fetch_pte.isra.7+0x5b/0x1a0 [Fri Apr 30 01:28:26 2021] ? iommu_unmap_page+0x64/0x100 [Fri Apr 30 01:28:26 2021] ? __unmap_single.isra.27+0x66/0x130 [Fri Apr 30 01:28:26 2021] ? unmap_sg+0x5f/0x70 [Fri Apr 30 01:28:26 2021] ? nv_unmap_dma_map_scatterlist+0x57/0xa0 [nvidia] [Fri Apr 30 01:28:26 2021] ? nv_dma_unmap_pages+0x56/0x100 [nvidia] [Fri Apr 30 01:28:26 2021] ? nv_dma_unmap_alloc+0x16/0x30 [nvidia] [Fri Apr 30 01:28:26 2021] ? _nv032962rm+0xc1/0x1b0 [nvidia] [Fri Apr 30 01:28:26 2021] ? _nv027787rm+0x9b/0xc0 [nvidia] [Fri Apr 30 01:28:26 2021] ? _nv030989rm+0x124/0x130 [nvidia] [Fri Apr 30 01:28:26 2021] ? rm_gpu_ops_free_duped_handle+0x1c/0x60 [nvidia] [Fri Apr 30 01:28:26 2021] ? nvUvmInterfaceFreeDupedHandle+0x2a/0x40 [nvidia] [Fri Apr 30 01:28:26 2021] ? uvm_ext_gpu_map_free_internal+0x5c/0x70 [nvidia_uvm] [Fri Apr 30 01:28:26 2021] ? uvm_ext_gpu_map_free+0xe/0x20 [nvidia_uvm] [Fri Apr 30 01:28:26 2021] ? uvm_deferred_free_object_list+0x70/0x130 [nvidia_uvm] [Fri Apr 30 01:28:26 2021] ? uvm_va_space_destroy+0x349/0x480 [nvidia_uvm] [Fri Apr 30 01:28:26 2021] ? uvm_release.isra.10+0x7c/0x90 [nvidia_uvm] [Fri Apr 30 01:28:26 2021] ? uvm_release_entry+0x6d/0xa0 [nvidia_uvm] [Fri Apr 30 01:28:26 2021] ? __fput+0xc6/0x260 [Fri Apr 30 01:28:26 2021] ? ____fput+0xe/0x10 [Fri Apr 30 01:28:26 2021] ? task_work_run+0x9d/0xc0 [Fri Apr 30 01:28:26 2021] ? do_exit+0x389/0xb90 [Fri Apr 30 01:28:26 2021] ? do_group_exit+0x43/0xa0 [Fri Apr 30 01:28:26 2021] ? get_signal+0x14f/0x860 [Fri Apr 30 01:28:26 2021] ? do_signal+0x34/0x6d0 [Fri Apr 30 01:28:26 2021] ? __switch_to_asm+0x34/0x70 [Fri Apr 30 01:28:26 2021] ? __x64_sys_futex+0x13f/0x190 [Fri Apr 30 01:28:26 2021] ? exit_to_usermode_loop+0x90/0x130 [Fri Apr 30 01:28:26 2021] ? do_syscall_64+0x170/0x190 [Fri Apr 30 01:28:26 2021] ? entry_SYSCALL_64_after_hwframe+0x44/0xa9 [Fri Apr 30 01:28:39 2021] watchdog: BUG: soft lockup - CPU#16 stuck for 22s! [python3:5432] [Fri Apr 30 01:28:39 2021] Modules linked in: nvidia_uvm(OE) veth xt_nat xt_tcpudp xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype iptable_filter iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 bpfilter br_netfilter bridge stp llc aufs cmac overlay bnep snd_hda_codec_hdmi nvidia_drm(POE) nvidia_modeset(POE) edac_mce_amd snd_usb_audio snd_usbmidi_lib mc kvm_amd kvm iwlmvm mac80211 crct10dif_pclmul crc32_pclmul btusb nls_iso8859_1 btrtl nvidia(POE) libarc4 btbcm ghash_clmulni_intel btintel bluetooth iwlwifi ecdh_generic aesni_intel snd_seq_midi input_leds ecc crypto_simd snd_seq_midi_event cryptd glue_helper cfg80211 wmi_bmof snd_hda_intel snd_rawmidi snd_intel_dspcfg joydev snd_hda_codec nouveau snd_hda_core snd_hwdep snd_seq snd_pcm mxm_wmi video snd_seq_device ccp ttm snd_timer drm_kms_helper snd drm i2c_algo_bit fb_sys_fops syscopyarea sysfillrect sysimgblt k10temp soundcore mac_hid sch_fq_codel parport_pc ppdev lp parport ip_tables x_tables [Fri Apr 30 01:28:39 2021] autofs4 btrfs xor zstd_compress raid6_pq libcrc32c dm_mirror dm_region_hash dm_log hid_generic usbhid hid nvme nvme_core ahci r8169 libahci atlantic realtek i2c_piix4 wmi [Fri Apr 30 01:28:39 2021] CPU: 16 PID: 5432 Comm: python3 Tainted: P W OEL 5.4.0-72-generic #80~18.04.1-Ubuntu [Fri Apr 30 01:28:39 2021] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./TRX40 Creator, BIOS P1.70 05/29/2020 [Fri Apr 30 01:28:39 2021] RIP: 0010:fetch_pte.isra.7+0x25/0x1a0 [Fri Apr 30 01:28:39 2021] Code: 00 00 00 00 00 0f 1f 44 00 00 48 c7 01 00 00 00 00 44 8b 0f 49 89 c8 41 83 f9 05 7f 1a 43 8d 4c c9 0c b8 01 00 00 00 48 d3 e0 <48> 83 e8 01 48 39 d0 0f 82 5c 01 00 00 41 8d 41 ff 55 48 89 d7 44 [Fri Apr 30 01:28:39 2021] RSP: 0018:fffface947c0f928 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [Fri Apr 30 01:28:39 2021] RAX: 0001000000000000 RBX: 0000000000001000 RCX: 0000000000000030 [Fri Apr 30 01:28:39 2021] RDX: 00008c70196a4000 RSI: ffff9dcbdc370098 RDI: ffff9dcbdc370094 [Fri Apr 30 01:28:39 2021] RBP: fffface947c0f968 R08: fffface947c0f930 R09: 0000000000000004 [Fri Apr 30 01:28:39 2021] R10: 0000000000000000 R11: 000000000000001b R12: ffff9dcbdc370098 [Fri Apr 30 01:28:39 2021] R13: ffff9dcbdc370094 R14: 00008c70196a4000 R15: 0000000000000000 [Fri Apr 30 01:28:39 2021] FS: 00007f9e29fff700(0000) GS:ffff9dcbfd200000(0000) knlGS:0000000000000000 [Fri Apr 30 01:28:39 2021] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fri Apr 30 01:28:39 2021] CR2: 00007fb3cf8144d0 CR3: 000000200f28c000 CR4: 0000000000340ee0 [Fri Apr 30 01:28:39 2021] Call Trace: [Fri Apr 30 01:28:39 2021] ? iommu_unmap_page+0x64/0x100 [Fri Apr 30 01:28:39 2021] __unmap_single.isra.27+0x66/0x130 [Fri Apr 30 01:28:39 2021] unmap_sg+0x5f/0x70 [Fri Apr 30 01:28:39 2021] nv_unmap_dma_map_scatterlist+0x57/0xa0 [nvidia] [Fri Apr 30 01:28:39 2021] nv_dma_unmap_pages+0x56/0x100 [nvidia] [Fri Apr 30 01:28:39 2021] nv_dma_unmap_alloc+0x16/0x30 [nvidia] [Fri Apr 30 01:28:39 2021] _nv032962rm+0xc1/0x1b0 [nvidia] [Fri Apr 30 01:28:39 2021] ? _nv027787rm+0x9b/0xc0 [nvidia] [Fri Apr 30 01:28:39 2021] ? _nv030989rm+0x124/0x130 [nvidia] [Fri Apr 30 01:28:39 2021] ? rm_gpu_ops_free_duped_handle+0x1c/0x60 [nvidia] [Fri Apr 30 01:28:39 2021] ? nvUvmInterfaceFreeDupedHandle+0x2a/0x40 [nvidia] [Fri Apr 30 01:28:39 2021] ? uvm_ext_gpu_map_free_internal+0x5c/0x70 [nvidia_uvm] [Fri Apr 30 01:28:39 2021] ? uvm_ext_gpu_map_free+0xe/0x20 [nvidia_uvm] [Fri Apr 30 01:28:39 2021] ? uvm_deferred_free_object_list+0x70/0x130 [nvidia_uvm] [Fri Apr 30 01:28:39 2021] ? uvm_va_space_destroy+0x349/0x480 [nvidia_uvm] [Fri Apr 30 01:28:39 2021] ? uvm_release.isra.10+0x7c/0x90 [nvidia_uvm] [Fri Apr 30 01:28:39 2021] ? uvm_release_entry+0x6d/0xa0 [nvidia_uvm] [Fri Apr 30 01:28:39 2021] ? __fput+0xc6/0x260 [Fri Apr 30 01:28:39 2021] ? ____fput+0xe/0x10 [Fri Apr 30 01:28:39 2021] ? task_work_run+0x9d/0xc0 [Fri Apr 30 01:28:39 2021] ? do_exit+0x389/0xb90 [Fri Apr 30 01:28:39 2021] ? do_group_exit+0x43/0xa0 [Fri Apr 30 01:28:39 2021] ? get_signal+0x14f/0x860 [Fri Apr 30 01:28:39 2021] ? do_signal+0x34/0x6d0 [Fri Apr 30 01:28:39 2021] ? __switch_to_asm+0x34/0x70 [Fri Apr 30 01:28:39 2021] ? __x64_sys_futex+0x13f/0x190 [Fri Apr 30 01:28:39 2021] ? exit_to_usermode_loop+0x90/0x130 [Fri Apr 30 01:28:39 2021] ? do_syscall_64+0x170/0x190 [Fri Apr 30 01:28:39 2021] ? entry_SYSCALL_64_after_hwframe+0x44/0xa9 [Fri Apr 30 01:29:07 2021] watchdog: BUG: soft lockup - CPU#16 stuck for 22s! [python3:5432] [Fri Apr 30 01:29:07 2021] Modules linked in: nvidia_uvm(OE) veth xt_nat xt_tcpudp xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype iptable_filter iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 bpfilter br_netfilter bridge stp llc aufs cmac overlay bnep snd_hda_codec_hdmi nvidia_drm(POE) nvidia_modeset(POE) edac_mce_amd snd_usb_audio snd_usbmidi_lib mc kvm_amd kvm iwlmvm mac80211 crct10dif_pclmul crc32_pclmul btusb nls_iso8859_1 btrtl nvidia(POE) libarc4 btbcm ghash_clmulni_intel btintel bluetooth iwlwifi ecdh_generic aesni_intel snd_seq_midi input_leds ecc crypto_simd snd_seq_midi_event cryptd glue_helper cfg80211 wmi_bmof snd_hda_intel snd_rawmidi snd_intel_dspcfg joydev snd_hda_codec nouveau snd_hda_core snd_hwdep snd_seq snd_pcm mxm_wmi video snd_seq_device ccp ttm snd_timer drm_kms_helper snd drm i2c_algo_bit fb_sys_fops syscopyarea sysfillrect sysimgblt k10temp soundcore mac_hid sch_fq_codel parport_pc ppdev lp parport ip_tables x_tables [Fri Apr 30 01:29:07 2021] autofs4 btrfs xor zstd_compress raid6_pq libcrc32c dm_mirror dm_region_hash dm_log hid_generic usbhid hid nvme nvme_core ahci r8169 libahci atlantic realtek i2c_piix4 wmi [Fri Apr 30 01:29:07 2021] CPU: 16 PID: 5432 Comm: python3 Tainted: P W OEL 5.4.0-72-generic #80~18.04.1-Ubuntu [Fri Apr 30 01:29:07 2021] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./TRX40 Creator, BIOS P1.70 05/29/2020 [Fri Apr 30 01:29:07 2021] RIP: 0010:fetch_pte.isra.7+0x66/0x1a0 [Fri Apr 30 01:29:07 2021] Code: 1c c0 48 89 e5 53 48 8b 36 41 8d 4b 0c 48 d3 ef 81 e7 ff 01 00 00 4c 8d 14 fe be 01 00 00 00 48 d3 e6 85 c0 49 89 30 49 8b 3a <0f> 8e b9 00 00 00 40 f6 c7 01 0f 84 1f 01 00 00 48 89 fe 48 c1 ee [Fri Apr 30 01:29:07 2021] RSP: 0018:fffface947c0f918 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [Fri Apr 30 01:29:07 2021] RAX: 0000000000000003 RBX: 0000000000001000 RCX: 0000000000000027 [Fri Apr 30 01:29:07 2021] RDX: 00008b11015ef000 RSI: 0000008000000000 RDI: 0000000000000000 [Fri Apr 30 01:29:07 2021] RBP: fffface947c0f920 R08: fffface947c0f930 R09: 0000000000000004 [Fri Apr 30 01:29:07 2021] R10: ffff9dc8ebfae8b0 R11: 000000000000001b R12: ffff9dcbdc370098 [Fri Apr 30 01:29:07 2021] R13: ffff9dcbdc370094 R14: 00008b11015ef000 R15: 0000000000000000 [Fri Apr 30 01:29:07 2021] FS: 00007f9e29fff700(0000) GS:ffff9dcbfd200000(0000) knlGS:0000000000000000 [Fri Apr 30 01:29:07 2021] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fri Apr 30 01:29:07 2021] CR2: 00007fb3cf8144d0 CR3: 000000200f28c000 CR4: 0000000000340ee0 [Fri Apr 30 01:29:07 2021] Call Trace: [Fri Apr 30 01:29:07 2021] iommu_unmap_page+0x64/0x100 [Fri Apr 30 01:29:07 2021] __unmap_single.isra.27+0x66/0x130 [Fri Apr 30 01:29:07 2021] unmap_sg+0x5f/0x70 [Fri Apr 30 01:29:07 2021] nv_unmap_dma_map_scatterlist+0x57/0xa0 [nvidia] [Fri Apr 30 01:29:07 2021] nv_dma_unmap_pages+0x56/0x100 [nvidia] [Fri Apr 30 01:29:07 2021] nv_dma_unmap_alloc+0x16/0x30 [nvidia] [Fri Apr 30 01:29:07 2021] _nv032962rm+0xc1/0x1b0 [nvidia] [Fri Apr 30 01:29:07 2021] ? _nv027787rm+0x9b/0xc0 [nvidia] [Fri Apr 30 01:29:07 2021] ? _nv030989rm+0x124/0x130 [nvidia] [Fri Apr 30 01:29:07 2021] ? rm_gpu_ops_free_duped_handle+0x1c/0x60 [nvidia] [Fri Apr 30 01:29:07 2021] ? nvUvmInterfaceFreeDupedHandle+0x2a/0x40 [nvidia] [Fri Apr 30 01:29:07 2021] ? uvm_ext_gpu_map_free_internal+0x5c/0x70 [nvidia_uvm] [Fri Apr 30 01:29:07 2021] ? uvm_ext_gpu_map_free+0xe/0x20 [nvidia_uvm] [Fri Apr 30 01:29:07 2021] ? uvm_deferred_free_object_list+0x70/0x130 [nvidia_uvm] [Fri Apr 30 01:29:07 2021] ? uvm_va_space_destroy+0x349/0x480 [nvidia_uvm] [Fri Apr 30 01:29:07 2021] ? uvm_release.isra.10+0x7c/0x90 [nvidia_uvm] [Fri Apr 30 01:29:07 2021] ? uvm_release_entry+0x6d/0xa0 [nvidia_uvm] [Fri Apr 30 01:29:07 2021] ? __fput+0xc6/0x260 [Fri Apr 30 01:29:07 2021] ? ____fput+0xe/0x10 [Fri Apr 30 01:29:07 2021] ? task_work_run+0x9d/0xc0 [Fri Apr 30 01:29:07 2021] ? do_exit+0x389/0xb90 [Fri Apr 30 01:29:07 2021] ? do_group_exit+0x43/0xa0 [Fri Apr 30 01:29:07 2021] ? get_signal+0x14f/0x860 [Fri Apr 30 01:29:07 2021] ? do_signal+0x34/0x6d0 [Fri Apr 30 01:29:07 2021] ? __switch_to_asm+0x34/0x70 [Fri Apr 30 01:29:07 2021] ? __x64_sys_futex+0x13f/0x190 [Fri Apr 30 01:29:07 2021] ? exit_to_usermode_loop+0x90/0x130 [Fri Apr 30 01:29:07 2021] ? do_syscall_64+0x170/0x190 [Fri Apr 30 01:29:07 2021] ? entry_SYSCALL_64_after_hwframe+0x44/0xa9 [Fri Apr 30 01:29:17 2021] iwlwifi 0000:45:00.0: HCMD_ACTIVE already clear for command SCAN_REQ_UMAC [Fri Apr 30 01:29:17 2021] iwlwifi 0000:45:00.0: Applying debug destination EXTERNAL_DRAM [Fri Apr 30 01:29:17 2021] iwlwifi 0000:45:00.0: FW already configured (0) - re-configuring [Fri Apr 30 01:29:17 2021] ------------[ cut here ]------------ [Fri Apr 30 01:29:17 2021] WARNING: CPU: 16 PID: 412 at /build/linux-hwe-5.4-PtAV0J/linux-hwe-5.4-5.4.0/drivers/net/wireless/intel/iwlwifi/mvm/scan.c:1929 iwl_mvm_rx_umac_scan_complete_notif+0x1f9/0x210 [iwlmvm] [Fri Apr 30 01:29:17 2021] Modules linked in: nvidia_uvm(OE) veth xt_nat xt_tcpudp xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype iptable_filter iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 bpfilter br_netfilter bridge stp llc aufs cmac overlay bnep snd_hda_codec_hdmi nvidia_drm(POE) nvidia_modeset(POE) edac_mce_amd snd_usb_audio snd_usbmidi_lib mc kvm_amd kvm iwlmvm mac80211 crct10dif_pclmul crc32_pclmul btusb nls_iso8859_1 btrtl nvidia(POE) libarc4 btbcm ghash_clmulni_intel btintel bluetooth iwlwifi ecdh_generic aesni_intel snd_seq_midi input_leds ecc crypto_simd snd_seq_midi_event cryptd glue_helper cfg80211 wmi_bmof snd_hda_intel snd_rawmidi snd_intel_dspcfg joydev snd_hda_codec nouveau snd_hda_core snd_hwdep snd_seq snd_pcm mxm_wmi video snd_seq_device ccp ttm snd_timer drm_kms_helper snd drm i2c_algo_bit fb_sys_fops syscopyarea sysfillrect sysimgblt k10temp soundcore mac_hid sch_fq_codel parport_pc ppdev lp parport ip_tables x_tables [Fri Apr 30 01:29:17 2021] autofs4 btrfs xor zstd_compress raid6_pq libcrc32c dm_mirror dm_region_hash dm_log hid_generic usbhid hid nvme nvme_core ahci r8169 libahci atlantic realtek i2c_piix4 wmi [Fri Apr 30 01:29:17 2021] CPU: 16 PID: 412 Comm: kworker/16:1 Tainted: P W OEL 5.4.0-72-generic #80~18.04.1-Ubuntu [Fri Apr 30 01:29:17 2021] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./TRX40 Creator, BIOS P1.70 05/29/2020 [Fri Apr 30 01:29:17 2021] Workqueue: events iwl_mvm_async_handlers_wk [iwlmvm] [Fri Apr 30 01:29:17 2021] RIP: 0010:iwl_mvm_rx_umac_scan_complete_notif+0x1f9/0x210 [iwlmvm] [Fri Apr 30 01:29:17 2021] Code: cc e8 db a1 aa ff 41 c7 84 24 60 1d 00 00 00 00 00 00 41 8b 8c 24 40 1d 00 00 41 8b 85 c4 1d 00 00 44 8b 4d cc e9 7c fe ff ff <0f> 0b e9 2e ff ff ff e8 fb 61 47 cb 90 66 2e 0f 1f 84 00 00 00 00 [Fri Apr 30 01:29:17 2021] RSP: 0018:fffface941337df0 EFLAGS: 00010246 [Fri Apr 30 01:29:17 2021] RAX: 0000000000000000 RBX: ffff9dcbee5e3000 RCX: 0000000000000000 [Fri Apr 30 01:29:17 2021] RDX: ffffffffc1229b10 RSI: ffff9dbbd2806e90 RDI: ffff9dcbbbe99e88 [Fri Apr 30 01:29:17 2021] RBP: fffface941337e28 R08: 0000000000000000 R09: 0000000000000000 [Fri Apr 30 01:29:17 2021] R10: fffface941337e10 R11: 0000000000000043 R12: fffface941337e40 [Fri Apr 30 01:29:17 2021] R13: ffff9dcbbbe99e88 R14: ffff9dbbd2806e80 R15: dead000000000100 [Fri Apr 30 01:29:17 2021] FS: 0000000000000000(0000) GS:ffff9dcbfd200000(0000) knlGS:0000000000000000 [Fri Apr 30 01:29:17 2021] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fri Apr 30 01:29:17 2021] CR2: 00007fb3cf8144d0 CR3: 000000200f28c000 CR4: 0000000000340ee0 [Fri Apr 30 01:29:17 2021] Call Trace: [Fri Apr 30 01:29:17 2021] ? __mutex_lock_slowpath+0x13/0x20 [Fri Apr 30 01:29:17 2021] iwl_mvm_async_handlers_wk+0xb0/0x140 [iwlmvm] [Fri Apr 30 01:29:17 2021] process_one_work+0x20f/0x400 [Fri Apr 30 01:29:17 2021] worker_thread+0x34/0x410 [Fri Apr 30 01:29:17 2021] kthread+0x121/0x140 [Fri Apr 30 01:29:17 2021] ? process_one_work+0x400/0x400 [Fri Apr 30 01:29:17 2021] ? kthread_park+0x90/0x90 [Fri Apr 30 01:29:17 2021] ret_from_fork+0x22/0x40 [Fri Apr 30 01:29:17 2021] ---[ end trace c3ee37f01c9a84ec ]--- [Fri Apr 30 01:29:39 2021] watchdog: BUG: soft lockup - CPU#40 stuck for 22s! [python3:5432] [Fri Apr 30 01:29:39 2021] Modules linked in: nvidia_uvm(OE) veth xt_nat xt_tcpudp xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype iptable_filter iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 bpfilter br_netfilter bridge stp llc aufs cmac overlay bnep snd_hda_codec_hdmi nvidia_drm(POE) nvidia_modeset(POE) edac_mce_amd snd_usb_audio snd_usbmidi_lib mc kvm_amd kvm iwlmvm mac80211 crct10dif_pclmul crc32_pclmul btusb nls_iso8859_1 btrtl nvidia(POE) libarc4 btbcm ghash_clmulni_intel btintel bluetooth iwlwifi ecdh_generic aesni_intel snd_seq_midi input_leds ecc crypto_simd snd_seq_midi_event cryptd glue_helper cfg80211 wmi_bmof snd_hda_intel snd_rawmidi snd_intel_dspcfg joydev snd_hda_codec nouveau snd_hda_core snd_hwdep snd_seq snd_pcm mxm_wmi video snd_seq_device ccp ttm snd_timer drm_kms_helper snd drm i2c_algo_bit fb_sys_fops syscopyarea sysfillrect sysimgblt k10temp soundcore mac_hid sch_fq_codel parport_pc ppdev lp parport ip_tables x_tables [Fri Apr 30 01:29:39 2021] autofs4 btrfs xor zstd_compress raid6_pq libcrc32c dm_mirror dm_region_hash dm_log hid_generic usbhid hid nvme nvme_core ahci r8169 libahci atlantic realtek i2c_piix4 wmi [Fri Apr 30 01:29:39 2021] CPU: 40 PID: 5432 Comm: python3 Tainted: P W OEL 5.4.0-72-generic #80~18.04.1-Ubuntu [Fri Apr 30 01:29:39 2021] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./TRX40 Creator, BIOS P1.70 05/29/2020 [Fri Apr 30 01:29:39 2021] RIP: 0010:fetch_pte.isra.7+0x49/0x1a0 [Fri Apr 30 01:29:39 2021] Code: 00 00 00 48 d3 e0 48 83 e8 01 48 39 d0 0f 82 5c 01 00 00 41 8d 41 ff 55 48 89 d7 44 8d 1c c0 48 89 e5 53 48 8b 36 41 8d 4b 0c <48> d3 ef 81 e7 ff 01 00 00 4c 8d 14 fe be 01 00 00 00 48 d3 e6 85 [Fri Apr 30 01:29:39 2021] RSP: 0018:fffface947c0f918 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [Fri Apr 30 01:29:39 2021] RAX: 0000000000000003 RBX: 0000000000001000 RCX: 0000000000000027 [Fri Apr 30 01:29:39 2021] RDX: 00008bd9d15b4000 RSI: ffff9dc8e85cd000 RDI: 00008bd9d15b4000 [Fri Apr 30 01:29:39 2021] RBP: fffface947c0f920 R08: fffface947c0f930 R09: 0000000000000004 [Fri Apr 30 01:29:39 2021] R10: 0000000000000000 R11: 000000000000001b R12: ffff9dcbe418b098 [Fri Apr 30 01:29:39 2021] R13: ffff9dcbe418b094 R14: 00008bd9d15b4000 R15: 0000000000000000 [Fri Apr 30 01:29:39 2021] FS: 00007f9e29fff700(0000) GS:ffff9dcbfd800000(0000) knlGS:0000000000000000 [Fri Apr 30 01:29:39 2021] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fri Apr 30 01:29:39 2021] CR2: 00007f00cbc90150 CR3: 0000001f54eb2000 CR4: 0000000000340ee0 [Fri Apr 30 01:29:39 2021] Call Trace: [Fri Apr 30 01:29:39 2021] iommu_unmap_page+0x64/0x100 [Fri Apr 30 01:29:39 2021] __unmap_single.isra.27+0x66/0x130 [Fri Apr 30 01:29:39 2021] unmap_sg+0x5f/0x70 [Fri Apr 30 01:29:39 2021] nv_unmap_dma_map_scatterlist+0x57/0xa0 [nvidia] [Fri Apr 30 01:29:39 2021] nv_dma_unmap_pages+0x56/0x100 [nvidia] [Fri Apr 30 01:29:39 2021] nv_dma_unmap_alloc+0x16/0x30 [nvidia] [Fri Apr 30 01:29:39 2021] _nv032962rm+0xc1/0x1b0 [nvidia] [Fri Apr 30 01:29:39 2021] ? _nv027787rm+0x9b/0xc0 [nvidia] [Fri Apr 30 01:29:39 2021] ? _nv030989rm+0x124/0x130 [nvidia] [Fri Apr 30 01:29:39 2021] ? rm_gpu_ops_free_duped_handle+0x1c/0x60 [nvidia] [Fri Apr 30 01:29:39 2021] ? nvUvmInterfaceFreeDupedHandle+0x2a/0x40 [nvidia] [Fri Apr 30 01:29:39 2021] ? uvm_ext_gpu_map_free_internal+0x5c/0x70 [nvidia_uvm] [Fri Apr 30 01:29:39 2021] ? uvm_ext_gpu_map_free+0xe/0x20 [nvidia_uvm] [Fri Apr 30 01:29:39 2021] ? uvm_deferred_free_object_list+0x70/0x130 [nvidia_uvm] [Fri Apr 30 01:29:39 2021] ? uvm_va_space_destroy+0x349/0x480 [nvidia_uvm] [Fri Apr 30 01:29:39 2021] ? uvm_release.isra.10+0x7c/0x90 [nvidia_uvm] [Fri Apr 30 01:29:39 2021] ? uvm_release_entry+0x6d/0xa0 [nvidia_uvm] [Fri Apr 30 01:29:39 2021] ? __fput+0xc6/0x260 [Fri Apr 30 01:29:39 2021] ? ____fput+0xe/0x10 [Fri Apr 30 01:29:39 2021] ? task_work_run+0x9d/0xc0 [Fri Apr 30 01:29:39 2021] ? do_exit+0x389/0xb90 [Fri Apr 30 01:29:39 2021] ? do_group_exit+0x43/0xa0 [Fri Apr 30 01:29:39 2021] ? get_signal+0x14f/0x860 [Fri Apr 30 01:29:39 2021] ? do_signal+0x34/0x6d0 [Fri Apr 30 01:29:39 2021] ? __switch_to_asm+0x34/0x70 [Fri Apr 30 01:29:39 2021] ? __x64_sys_futex+0x13f/0x190 [Fri Apr 30 01:29:39 2021] ? exit_to_usermode_loop+0x90/0x130 [Fri Apr 30 01:29:39 2021] ? do_syscall_64+0x170/0x190 [Fri Apr 30 01:29:39 2021] ? entry_SYSCALL_64_after_hwframe+0x44/0xa9 [Fri Apr 30 01:30:07 2021] watchdog: BUG: soft lockup - CPU#40 stuck for 22s! [python3:5432] [Fri Apr 30 01:30:07 2021] Modules linked in: nvidia_uvm(OE) veth xt_nat xt_tcpudp xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype iptable_filter iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 bpfilter br_netfilter bridge stp llc aufs cmac overlay bnep snd_hda_codec_hdmi nvidia_drm(POE) nvidia_modeset(POE) edac_mce_amd snd_usb_audio snd_usbmidi_lib mc kvm_amd kvm iwlmvm mac80211 crct10dif_pclmul crc32_pclmul btusb nls_iso8859_1 btrtl nvidia(POE) libarc4 btbcm ghash_clmulni_intel btintel bluetooth iwlwifi ecdh_generic aesni_intel snd_seq_midi input_leds ecc crypto_simd snd_seq_midi_event cryptd glue_helper cfg80211 wmi_bmof snd_hda_intel snd_rawmidi snd_intel_dspcfg joydev snd_hda_codec nouveau snd_hda_core snd_hwdep snd_seq snd_pcm mxm_wmi video snd_seq_device ccp ttm snd_timer drm_kms_helper snd drm i2c_algo_bit fb_sys_fops syscopyarea sysfillrect sysimgblt k10temp soundcore mac_hid sch_fq_codel parport_pc ppdev lp parport ip_tables x_tables [Fri Apr 30 01:30:07 2021] autofs4 btrfs xor zstd_compress raid6_pq libcrc32c dm_mirror dm_region_hash dm_log hid_generic usbhid hid nvme nvme_core ahci r8169 libahci atlantic realtek i2c_piix4 wmi [Fri Apr 30 01:30:07 2021] CPU: 40 PID: 5432 Comm: python3 Tainted: P W OEL 5.4.0-72-generic #80~18.04.1-Ubuntu [Fri Apr 30 01:30:07 2021] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./TRX40 Creator, BIOS P1.70 05/29/2020 [Fri Apr 30 01:30:07 2021] RIP: 0010:iommu_unmap_page+0xd3/0x100 [Fri Apr 30 01:30:07 2021] Code: 8b 45 c8 48 89 c2 49 01 c7 48 f7 da 4c 21 f2 4c 39 fb 4c 8d 34 10 77 91 4d 85 ff 75 23 48 8b 7d d0 65 48 33 3c 25 28 00 00 00 <4c> 89 f8 75 1c 48 83 c4 10 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 0b [Fri Apr 30 01:30:07 2021] RSP: 0018:fffface947c0f930 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13 [Fri Apr 30 01:30:07 2021] RAX: 0000007fffffffff RBX: 0000000000001000 RCX: 0000000000000027 [Fri Apr 30 01:30:07 2021] RDX: 00008a0000000000 RSI: 0000000000000000 RDI: 0000000000000000 [Fri Apr 30 01:30:07 2021] RBP: fffface947c0f968 R08: fffface947c0f930 R09: 0000000000000004 [Fri Apr 30 01:30:07 2021] R10: 0000000000000000 R11: 000000000000001b R12: ffff9dcbe418b098 [Fri Apr 30 01:30:07 2021] R13: ffff9dcbe418b094 R14: 00008a8000000000 R15: 0000008000000000 [Fri Apr 30 01:30:07 2021] FS: 00007f9e29fff700(0000) GS:ffff9dcbfd800000(0000) knlGS:0000000000000000 [Fri Apr 30 01:30:07 2021] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fri Apr 30 01:30:07 2021] CR2: 00007f00cbc90150 CR3: 0000001f54eb2000 CR4: 0000000000340ee0 [Fri Apr 30 01:30:07 2021] Call Trace: [Fri Apr 30 01:30:07 2021] __unmap_single.isra.27+0x66/0x130 [Fri Apr 30 01:30:07 2021] unmap_sg+0x5f/0x70 [Fri Apr 30 01:30:07 2021] nv_unmap_dma_map_scatterlist+0x57/0xa0 [nvidia] [Fri Apr 30 01:30:07 2021] nv_dma_unmap_pages+0x56/0x100 [nvidia] [Fri Apr 30 01:30:07 2021] nv_dma_unmap_alloc+0x16/0x30 [nvidia] [Fri Apr 30 01:30:07 2021] _nv032962rm+0xc1/0x1b0 [nvidia] [Fri Apr 30 01:30:07 2021] ? _nv027787rm+0x9b/0xc0 [nvidia] [Fri Apr 30 01:30:07 2021] ? _nv030989rm+0x124/0x130 [nvidia] [Fri Apr 30 01:30:07 2021] ? rm_gpu_ops_free_duped_handle+0x1c/0x60 [nvidia] [Fri Apr 30 01:30:07 2021] ? nvUvmInterfaceFreeDupedHandle+0x2a/0x40 [nvidia] [Fri Apr 30 01:30:07 2021] ? uvm_ext_gpu_map_free_internal+0x5c/0x70 [nvidia_uvm] [Fri Apr 30 01:30:07 2021] ? uvm_ext_gpu_map_free+0xe/0x20 [nvidia_uvm] [Fri Apr 30 01:30:07 2021] ? uvm_deferred_free_object_list+0x70/0x130 [nvidia_uvm] [Fri Apr 30 01:30:07 2021] ? uvm_va_space_destroy+0x349/0x480 [nvidia_uvm] [Fri Apr 30 01:30:07 2021] ? uvm_release.isra.10+0x7c/0x90 [nvidia_uvm] [Fri Apr 30 01:30:07 2021] ? uvm_release_entry+0x6d/0xa0 [nvidia_uvm] [Fri Apr 30 01:30:07 2021] ? __fput+0xc6/0x260 [Fri Apr 30 01:30:07 2021] ? ____fput+0xe/0x10 [Fri Apr 30 01:30:07 2021] ? task_work_run+0x9d/0xc0 [Fri Apr 30 01:30:07 2021] ? do_exit+0x389/0xb90 [Fri Apr 30 01:30:07 2021] ? do_group_exit+0x43/0xa0 [Fri Apr 30 01:30:07 2021] ? get_signal+0x14f/0x860 [Fri Apr 30 01:30:07 2021] ? do_signal+0x34/0x6d0 [Fri Apr 30 01:30:07 2021] ? __switch_to_asm+0x34/0x70 [Fri Apr 30 01:30:07 2021] ? __x64_sys_futex+0x13f/0x190 [Fri Apr 30 01:30:07 2021] ? exit_to_usermode_loop+0x90/0x130 [Fri Apr 30 01:30:07 2021] ? do_syscall_64+0x170/0x190 [Fri Apr 30 01:30:07 2021] ? entry_SYSCALL_64_after_hwframe+0x44/0xa9 [Fri Apr 30 01:30:17 2021] rcu: INFO: rcu_sched self-detected stall on CPU [Fri Apr 30 01:30:17 2021] rcu: 40-....: (15000 ticks this GP) idle=12a/1/0x4000000000000002 softirq=7614/7614 fqs=7492 [Fri Apr 30 01:30:17 2021] (t=15001 jiffies g=47425 q=496981) [Fri Apr 30 01:30:17 2021] NMI backtrace for cpu 40 [Fri Apr 30 01:30:17 2021] CPU: 40 PID: 5432 Comm: python3 Tainted: P W OEL 5.4.0-72-generic #80~18.04.1-Ubuntu [Fri Apr 30 01:30:17 2021] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./TRX40 Creator, BIOS P1.70 05/29/2020 [Fri Apr 30 01:30:17 2021] Call Trace: [Fri Apr 30 01:30:17 2021] [Fri Apr 30 01:30:17 2021] dump_stack+0x6d/0x8b [Fri Apr 30 01:30:17 2021] nmi_cpu_backtrace+0x94/0xa0 [Fri Apr 30 01:30:17 2021] ? lapic_can_unplug_cpu+0xb0/0xb0 [Fri Apr 30 01:30:17 2021] nmi_trigger_cpumask_backtrace+0xe7/0x130 [Fri Apr 30 01:30:17 2021] arch_trigger_cpumask_backtrace+0x19/0x20 [Fri Apr 30 01:30:17 2021] rcu_dump_cpu_stacks+0x9e/0xce [Fri Apr 30 01:30:17 2021] rcu_sched_clock_irq+0x508/0x770 [Fri Apr 30 01:30:17 2021] ? tick_sched_do_timer+0x60/0x60 [Fri Apr 30 01:30:17 2021] update_process_times+0x28/0x50 [Fri Apr 30 01:30:17 2021] tick_sched_handle+0x25/0x70 [Fri Apr 30 01:30:17 2021] tick_sched_timer+0x3c/0x80 [Fri Apr 30 01:30:17 2021] __hrtimer_run_queues+0x10f/0x280 [Fri Apr 30 01:30:17 2021] hrtimer_interrupt+0xe7/0x240 [Fri Apr 30 01:30:17 2021] smp_apic_timer_interrupt+0x6f/0x130 [Fri Apr 30 01:30:17 2021] apic_timer_interrupt+0xf/0x20 [Fri Apr 30 01:30:17 2021] [Fri Apr 30 01:30:17 2021] RIP: 0010:iommu_unmap_page+0x6/0x100 [Fri Apr 30 01:30:17 2021] Code: 48 89 c2 48 83 ea 01 48 c1 ea 0c e8 a4 da ff ff 48 85 c0 74 02 5d c3 0f 0b 5d c3 e8 a4 ff ff ff 0f 1f 40 00 0f 1f 44 00 00 55 <48> 89 e5 41 57 41 56 41 55 41 54 53 48 83 ec 10 65 48 8b 04 25 28 [Fri Apr 30 01:30:17 2021] RSP: 0018:fffface947c0f968 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [Fri Apr 30 01:30:17 2021] RAX: 0000008000000000 RBX: 00008f5e4555c000 RCX: 0000000000000027 [Fri Apr 30 01:30:17 2021] RDX: 0000000000001000 RSI: 00008f5e4555b000 RDI: ffff9dcbe418b000 [Fri Apr 30 01:30:17 2021] RBP: fffface947c0f9a8 R08: fffface947c0f930 R09: 0000000000000004 [Fri Apr 30 01:30:17 2021] R10: 0000000000000000 R11: 000000000000001b R12: ffff9dcbe418b000 [Fri Apr 30 01:30:17 2021] R13: 00008ffbfffff000 R14: 00000000ffffffff R15: 00007ffc00000000 [Fri Apr 30 01:30:17 2021] ? __unmap_single.isra.27+0x66/0x130 [Fri Apr 30 01:30:17 2021] unmap_sg+0x5f/0x70 [Fri Apr 30 01:30:17 2021] nv_unmap_dma_map_scatterlist+0x57/0xa0 [nvidia] [Fri Apr 30 01:30:17 2021] nv_dma_unmap_pages+0x56/0x100 [nvidia] [Fri Apr 30 01:30:17 2021] nv_dma_unmap_alloc+0x16/0x30 [nvidia] [Fri Apr 30 01:30:17 2021] _nv032962rm+0xc1/0x1b0 [nvidia] [Fri Apr 30 01:30:17 2021] ? _nv027787rm+0x9b/0xc0 [nvidia] [Fri Apr 30 01:30:17 2021] ? _nv030989rm+0x124/0x130 [nvidia] [Fri Apr 30 01:30:17 2021] ? rm_gpu_ops_free_duped_handle+0x1c/0x60 [nvidia] [Fri Apr 30 01:30:17 2021] ? nvUvmInterfaceFreeDupedHandle+0x2a/0x40 [nvidia] [Fri Apr 30 01:30:17 2021] ? uvm_ext_gpu_map_free_internal+0x5c/0x70 [nvidia_uvm] [Fri Apr 30 01:30:17 2021] ? uvm_ext_gpu_map_free+0xe/0x20 [nvidia_uvm] [Fri Apr 30 01:30:17 2021] ? uvm_deferred_free_object_list+0x70/0x130 [nvidia_uvm] [Fri Apr 30 01:30:17 2021] ? uvm_va_space_destroy+0x349/0x480 [nvidia_uvm] [Fri Apr 30 01:30:17 2021] ? uvm_release.isra.10+0x7c/0x90 [nvidia_uvm] [Fri Apr 30 01:30:17 2021] ? uvm_release_entry+0x6d/0xa0 [nvidia_uvm] [Fri Apr 30 01:30:17 2021] ? __fput+0xc6/0x260 [Fri Apr 30 01:30:17 2021] ? ____fput+0xe/0x10 [Fri Apr 30 01:30:17 2021] ? task_work_run+0x9d/0xc0 [Fri Apr 30 01:30:17 2021] ? do_exit+0x389/0xb90 [Fri Apr 30 01:30:17 2021] ? do_group_exit+0x43/0xa0 [Fri Apr 30 01:30:17 2021] ? get_signal+0x14f/0x860 [Fri Apr 30 01:30:17 2021] ? do_signal+0x34/0x6d0 [Fri Apr 30 01:30:17 2021] ? __switch_to_asm+0x34/0x70 [Fri Apr 30 01:30:17 2021] ? __x64_sys_futex+0x13f/0x190 [Fri Apr 30 01:30:17 2021] ? exit_to_usermode_loop+0x90/0x130 [Fri Apr 30 01:30:17 2021] ? do_syscall_64+0x170/0x190 [Fri Apr 30 01:30:17 2021] ? entry_SYSCALL_64_after_hwframe+0x44/0xa9 [Fri Apr 30 01:30:39 2021] watchdog: BUG: soft lockup - CPU#17 stuck for 23s! [python3:5432] [Fri Apr 30 01:30:39 2021] Modules linked in: nvidia_uvm(OE) veth xt_nat xt_tcpudp xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype iptable_filter iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 bpfilter br_netfilter bridge stp llc aufs cmac overlay bnep snd_hda_codec_hdmi nvidia_drm(POE) nvidia_modeset(POE) edac_mce_amd snd_usb_audio snd_usbmidi_lib mc kvm_amd kvm iwlmvm mac80211 crct10dif_pclmul crc32_pclmul btusb nls_iso8859_1 btrtl nvidia(POE) libarc4 btbcm ghash_clmulni_intel btintel bluetooth iwlwifi ecdh_generic aesni_intel snd_seq_midi input_leds ecc crypto_simd snd_seq_midi_event cryptd glue_helper cfg80211 wmi_bmof snd_hda_intel snd_rawmidi snd_intel_dspcfg joydev snd_hda_codec nouveau snd_hda_core snd_hwdep snd_seq snd_pcm mxm_wmi video snd_seq_device ccp ttm snd_timer drm_kms_helper snd drm i2c_algo_bit fb_sys_fops syscopyarea sysfillrect sysimgblt k10temp soundcore mac_hid sch_fq_codel parport_pc ppdev lp parport ip_tables x_tables [Fri Apr 30 01:30:39 2021] autofs4 btrfs xor zstd_compress raid6_pq libcrc32c dm_mirror dm_region_hash dm_log hid_generic usbhid hid nvme nvme_core ahci r8169 libahci atlantic realtek i2c_piix4 wmi [Fri Apr 30 01:30:39 2021] CPU: 17 PID: 5432 Comm: python3 Tainted: P W OEL 5.4.0-72-generic #80~18.04.1-Ubuntu [Fri Apr 30 01:30:39 2021] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./TRX40 Creator, BIOS P1.70 05/29/2020 [Fri Apr 30 01:30:39 2021] RIP: 0010:fetch_pte.isra.7+0x49/0x1a0 [Fri Apr 30 01:30:39 2021] Code: 00 00 00 48 d3 e0 48 83 e8 01 48 39 d0 0f 82 5c 01 00 00 41 8d 41 ff 55 48 89 d7 44 8d 1c c0 48 89 e5 53 48 8b 36 41 8d 4b 0c <48> d3 ef 81 e7 ff 01 00 00 4c 8d 14 fe be 01 00 00 00 48 d3 e6 85 [Fri Apr 30 01:30:39 2021] RSP: 0018:fffface947c0f918 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [Fri Apr 30 01:30:39 2021] RAX: 0000000000000003 RBX: 0000000000001000 RCX: 0000000000000027 [Fri Apr 30 01:30:39 2021] RDX: 00008b2aba2ff000 RSI: ffff9dc8e9ec4000 RDI: 00008b2aba2ff000 [Fri Apr 30 01:30:39 2021] RBP: fffface947c0f920 R08: fffface947c0f930 R09: 0000000000000004 [Fri Apr 30 01:30:39 2021] R10: 0000000000000000 R11: 000000000000001b R12: ffff9dcbdc371098 [Fri Apr 30 01:30:39 2021] R13: ffff9dcbdc371094 R14: 00008b2aba2ff000 R15: 0000000000000000 [Fri Apr 30 01:30:39 2021] FS: 00007f9e29fff700(0000) GS:ffff9dcbfd240000(0000) knlGS:0000000000000000 [Fri Apr 30 01:30:39 2021] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fri Apr 30 01:30:39 2021] CR2: 00007fbfe811b058 CR3: 0000001fbb9b0000 CR4: 0000000000340ee0 [Fri Apr 30 01:30:39 2021] Call Trace: [Fri Apr 30 01:30:39 2021] iommu_unmap_page+0x64/0x100 [Fri Apr 30 01:30:39 2021] __unmap_single.isra.27+0x66/0x130 [Fri Apr 30 01:30:39 2021] unmap_sg+0x5f/0x70 [Fri Apr 30 01:30:39 2021] nv_unmap_dma_map_scatterlist+0x57/0xa0 [nvidia] [Fri Apr 30 01:30:39 2021] nv_dma_unmap_pages+0x56/0x100 [nvidia] [Fri Apr 30 01:30:39 2021] nv_dma_unmap_alloc+0x16/0x30 [nvidia] [Fri Apr 30 01:30:39 2021] _nv032962rm+0xc1/0x1b0 [nvidia] [Fri Apr 30 01:30:39 2021] ? _nv027787rm+0x9b/0xc0 [nvidia] [Fri Apr 30 01:30:39 2021] ? _nv030989rm+0x124/0x130 [nvidia] [Fri Apr 30 01:30:39 2021] ? rm_gpu_ops_free_duped_handle+0x1c/0x60 [nvidia] [Fri Apr 30 01:30:39 2021] ? nvUvmInterfaceFreeDupedHandle+0x2a/0x40 [nvidia] [Fri Apr 30 01:30:39 2021] ? uvm_ext_gpu_map_free_internal+0x5c/0x70 [nvidia_uvm] [Fri Apr 30 01:30:39 2021] ? uvm_ext_gpu_map_free+0xe/0x20 [nvidia_uvm] [Fri Apr 30 01:30:39 2021] ? uvm_deferred_free_object_list+0x70/0x130 [nvidia_uvm] [Fri Apr 30 01:30:39 2021] ? uvm_va_space_destroy+0x349/0x480 [nvidia_uvm] [Fri Apr 30 01:30:39 2021] ? uvm_release.isra.10+0x7c/0x90 [nvidia_uvm] [Fri Apr 30 01:30:39 2021] ? uvm_release_entry+0x6d/0xa0 [nvidia_uvm] [Fri Apr 30 01:30:39 2021] ? __fput+0xc6/0x260 [Fri Apr 30 01:30:39 2021] ? ____fput+0xe/0x10 [Fri Apr 30 01:30:39 2021] ? task_work_run+0x9d/0xc0 [Fri Apr 30 01:30:39 2021] ? do_exit+0x389/0xb90 [Fri Apr 30 01:30:39 2021] ? do_group_exit+0x43/0xa0 [Fri Apr 30 01:30:39 2021] ? get_signal+0x14f/0x860 [Fri Apr 30 01:30:39 2021] ? do_signal+0x34/0x6d0 [Fri Apr 30 01:30:39 2021] ? __switch_to_asm+0x34/0x70 [Fri Apr 30 01:30:39 2021] ? __x64_sys_futex+0x13f/0x190 [Fri Apr 30 01:30:39 2021] ? exit_to_usermode_loop+0x90/0x130 [Fri Apr 30 01:30:39 2021] ? do_syscall_64+0x170/0x190 [Fri Apr 30 01:30:39 2021] ? entry_SYSCALL_64_after_hwframe+0x44/0xa9 [Fri Apr 30 01:31:07 2021] watchdog: BUG: soft lockup - CPU#17 stuck for 23s! [python3:5432] [Fri Apr 30 01:31:07 2021] Modules linked in: nvidia_uvm(OE) veth xt_nat xt_tcpudp xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype iptable_filter iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 bpfilter br_netfilter bridge stp llc aufs cmac overlay bnep snd_hda_codec_hdmi nvidia_drm(POE) nvidia_modeset(POE) edac_mce_amd snd_usb_audio snd_usbmidi_lib mc kvm_amd kvm iwlmvm mac80211 crct10dif_pclmul crc32_pclmul btusb nls_iso8859_1 btrtl nvidia(POE) libarc4 btbcm ghash_clmulni_intel btintel bluetooth iwlwifi ecdh_generic aesni_intel snd_seq_midi input_leds ecc crypto_simd snd_seq_midi_event cryptd glue_helper cfg80211 wmi_bmof snd_hda_intel snd_rawmidi snd_intel_dspcfg joydev snd_hda_codec nouveau snd_hda_core snd_hwdep snd_seq snd_pcm mxm_wmi video snd_seq_device ccp ttm snd_timer drm_kms_helper snd drm i2c_algo_bit fb_sys_fops syscopyarea sysfillrect sysimgblt k10temp soundcore mac_hid sch_fq_codel parport_pc ppdev lp parport ip_tables x_tables [Fri Apr 30 01:31:07 2021] autofs4 btrfs xor zstd_compress raid6_pq libcrc32c dm_mirror dm_region_hash dm_log hid_generic usbhid hid nvme nvme_core ahci r8169 libahci atlantic realtek i2c_piix4 wmi [Fri Apr 30 01:31:07 2021] CPU: 17 PID: 5432 Comm: python3 Tainted: P W OEL 5.4.0-72-generic #80~18.04.1-Ubuntu [Fri Apr 30 01:31:07 2021] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./TRX40 Creator, BIOS P1.70 05/29/2020 [Fri Apr 30 01:31:07 2021] RIP: 0010:fetch_pte.isra.7+0x29/0x1a0 [Fri Apr 30 01:31:07 2021] Code: 00 0f 1f 44 00 00 48 c7 01 00 00 00 00 44 8b 0f 49 89 c8 41 83 f9 05 7f 1a 43 8d 4c c9 0c b8 01 00 00 00 48 d3 e0 48 83 e8 01 <48> 39 d0 0f 82 5c 01 00 00 41 8d 41 ff 55 48 89 d7 44 8d 1c c0 48 [Fri Apr 30 01:31:07 2021] RSP: 0018:fffface947c0f928 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [Fri Apr 30 01:31:07 2021] RAX: 0000ffffffffffff RBX: 0000000000001000 RCX: 0000000000000030 [Fri Apr 30 01:31:07 2021] RDX: 000089c3f93e3000 RSI: ffff9dcbdc371098 RDI: ffff9dcbdc371094 [Fri Apr 30 01:31:07 2021] RBP: fffface947c0f968 R08: fffface947c0f930 R09: 0000000000000004 [Fri Apr 30 01:31:07 2021] R10: 0000000000000000 R11: 000000000000001b R12: ffff9dcbdc371098 [Fri Apr 30 01:31:07 2021] R13: ffff9dcbdc371094 R14: 000089c3f93e3000 R15: 0000000000000000 [Fri Apr 30 01:31:07 2021] FS: 00007f9e29fff700(0000) GS:ffff9dcbfd240000(0000) knlGS:0000000000000000 [Fri Apr 30 01:31:07 2021] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fri Apr 30 01:31:07 2021] CR2: 00007fbfe811b058 CR3: 0000001fbb9b0000 CR4: 0000000000340ee0 [Fri Apr 30 01:31:07 2021] Call Trace: [Fri Apr 30 01:31:07 2021] ? iommu_unmap_page+0x64/0x100 [Fri Apr 30 01:31:07 2021] __unmap_single.isra.27+0x66/0x130 [Fri Apr 30 01:31:07 2021] unmap_sg+0x5f/0x70 [Fri Apr 30 01:31:07 2021] nv_unmap_dma_map_scatterlist+0x57/0xa0 [nvidia] [Fri Apr 30 01:31:07 2021] nv_dma_unmap_pages+0x56/0x100 [nvidia] [Fri Apr 30 01:31:07 2021] nv_dma_unmap_alloc+0x16/0x30 [nvidia] [Fri Apr 30 01:31:07 2021] _nv032962rm+0xc1/0x1b0 [nvidia] [Fri Apr 30 01:31:07 2021] ? _nv027787rm+0x9b/0xc0 [nvidia] [Fri Apr 30 01:31:07 2021] ? _nv030989rm+0x124/0x130 [nvidia] [Fri Apr 30 01:31:07 2021] ? rm_gpu_ops_free_duped_handle+0x1c/0x60 [nvidia] [Fri Apr 30 01:31:07 2021] ? nvUvmInterfaceFreeDupedHandle+0x2a/0x40 [nvidia] [Fri Apr 30 01:31:07 2021] ? uvm_ext_gpu_map_free_internal+0x5c/0x70 [nvidia_uvm] [Fri Apr 30 01:31:07 2021] ? uvm_ext_gpu_map_free+0xe/0x20 [nvidia_uvm] [Fri Apr 30 01:31:07 2021] ? uvm_deferred_free_object_list+0x70/0x130 [nvidia_uvm] [Fri Apr 30 01:31:07 2021] ? uvm_va_space_destroy+0x349/0x480 [nvidia_uvm] [Fri Apr 30 01:31:07 2021] ? uvm_release.isra.10+0x7c/0x90 [nvidia_uvm] [Fri Apr 30 01:31:07 2021] ? uvm_release_entry+0x6d/0xa0 [nvidia_uvm] [Fri Apr 30 01:31:07 2021] ? __fput+0xc6/0x260 [Fri Apr 30 01:31:07 2021] ? ____fput+0xe/0x10 [Fri Apr 30 01:31:07 2021] ? task_work_run+0x9d/0xc0 [Fri Apr 30 01:31:07 2021] ? do_exit+0x389/0xb90 [Fri Apr 30 01:31:07 2021] ? do_group_exit+0x43/0xa0 [Fri Apr 30 01:31:07 2021] ? get_signal+0x14f/0x860 [Fri Apr 30 01:31:07 2021] ? do_signal+0x34/0x6d0 [Fri Apr 30 01:31:07 2021] ? __switch_to_asm+0x34/0x70 [Fri Apr 30 01:31:07 2021] ? __x64_sys_futex+0x13f/0x190 [Fri Apr 30 01:31:07 2021] ? exit_to_usermode_loop+0x90/0x130 [Fri Apr 30 01:31:07 2021] ? do_syscall_64+0x170/0x190 [Fri Apr 30 01:31:07 2021] ? entry_SYSCALL_64_after_hwframe+0x44/0xa9 [Fri Apr 30 01:31:18 2021] rcu: INFO: rcu_sched self-detected stall on CPU [Fri Apr 30 01:31:18 2021] rcu: 17-....: (15000 ticks this GP) idle=1ea/1/0x4000000000000002 softirq=7538/7538 fqs=7297 [Fri Apr 30 01:31:18 2021] (t=15000 jiffies g=47429 q=398564) [Fri Apr 30 01:31:18 2021] NMI backtrace for cpu 17 [Fri Apr 30 01:31:18 2021] CPU: 17 PID: 5432 Comm: python3 Tainted: P W OEL 5.4.0-72-generic #80~18.04.1-Ubuntu [Fri Apr 30 01:31:18 2021] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./TRX40 Creator, BIOS P1.70 05/29/2020 [Fri Apr 30 01:31:18 2021] Call Trace: [Fri Apr 30 01:31:18 2021] [Fri Apr 30 01:31:18 2021] dump_stack+0x6d/0x8b [Fri Apr 30 01:31:18 2021] nmi_cpu_backtrace+0x94/0xa0 [Fri Apr 30 01:31:18 2021] ? lapic_can_unplug_cpu+0xb0/0xb0 [Fri Apr 30 01:31:18 2021] nmi_trigger_cpumask_backtrace+0xe7/0x130 [Fri Apr 30 01:31:18 2021] arch_trigger_cpumask_backtrace+0x19/0x20 [Fri Apr 30 01:31:18 2021] rcu_dump_cpu_stacks+0x9e/0xce [Fri Apr 30 01:31:18 2021] rcu_sched_clock_irq+0x508/0x770 [Fri Apr 30 01:31:18 2021] ? tick_sched_do_timer+0x60/0x60 [Fri Apr 30 01:31:18 2021] update_process_times+0x28/0x50 [Fri Apr 30 01:31:18 2021] tick_sched_handle+0x25/0x70 [Fri Apr 30 01:31:18 2021] tick_sched_timer+0x3c/0x80 [Fri Apr 30 01:31:18 2021] __hrtimer_run_queues+0x10f/0x280 [Fri Apr 30 01:31:18 2021] hrtimer_interrupt+0xe7/0x240 [Fri Apr 30 01:31:18 2021] smp_apic_timer_interrupt+0x6f/0x130 [Fri Apr 30 01:31:18 2021] apic_timer_interrupt+0xf/0x20 [Fri Apr 30 01:31:18 2021] [Fri Apr 30 01:31:18 2021] RIP: 0010:fetch_pte.isra.7+0x2c/0x1a0 [Fri Apr 30 01:31:18 2021] Code: 44 00 00 48 c7 01 00 00 00 00 44 8b 0f 49 89 c8 41 83 f9 05 7f 1a 43 8d 4c c9 0c b8 01 00 00 00 48 d3 e0 48 83 e8 01 48 39 d0 <0f> 82 5c 01 00 00 41 8d 41 ff 55 48 89 d7 44 8d 1c c0 48 89 e5 53 [Fri Apr 30 01:31:18 2021] RSP: 0018:fffface947c0f928 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [Fri Apr 30 01:31:18 2021] RAX: 0000ffffffffffff RBX: 0000000000001000 RCX: 0000000000000030 [Fri Apr 30 01:31:18 2021] RDX: 00008f484a214000 RSI: ffff9dcbdc371098 RDI: ffff9dcbdc371094 [Fri Apr 30 01:31:18 2021] RBP: fffface947c0f968 R08: fffface947c0f930 R09: 0000000000000004 [Fri Apr 30 01:31:18 2021] R10: 0000000000000000 R11: 000000000000001b R12: ffff9dcbdc371098 [Fri Apr 30 01:31:18 2021] R13: ffff9dcbdc371094 R14: 00008f484a214000 R15: 0000000000000000 [Fri Apr 30 01:31:18 2021] ? iommu_unmap_page+0x64/0x100 [Fri Apr 30 01:31:18 2021] __unmap_single.isra.27+0x66/0x130 [Fri Apr 30 01:31:18 2021] unmap_sg+0x5f/0x70 [Fri Apr 30 01:31:18 2021] nv_unmap_dma_map_scatterlist+0x57/0xa0 [nvidia] [Fri Apr 30 01:31:18 2021] nv_dma_unmap_pages+0x56/0x100 [nvidia] [Fri Apr 30 01:31:18 2021] nv_dma_unmap_alloc+0x16/0x30 [nvidia] [Fri Apr 30 01:31:18 2021] _nv032962rm+0xc1/0x1b0 [nvidia] [Fri Apr 30 01:31:18 2021] ? _nv027787rm+0x9b/0xc0 [nvidia] [Fri Apr 30 01:31:18 2021] ? _nv030989rm+0x124/0x130 [nvidia] [Fri Apr 30 01:31:18 2021] ? rm_gpu_ops_free_duped_handle+0x1c/0x60 [nvidia] [Fri Apr 30 01:31:18 2021] ? nvUvmInterfaceFreeDupedHandle+0x2a/0x40 [nvidia] [Fri Apr 30 01:31:18 2021] ? uvm_ext_gpu_map_free_internal+0x5c/0x70 [nvidia_uvm] [Fri Apr 30 01:31:18 2021] ? uvm_ext_gpu_map_free+0xe/0x20 [nvidia_uvm] [Fri Apr 30 01:31:18 2021] ? uvm_deferred_free_object_list+0x70/0x130 [nvidia_uvm] [Fri Apr 30 01:31:18 2021] ? uvm_va_space_destroy+0x349/0x480 [nvidia_uvm] [Fri Apr 30 01:31:18 2021] ? uvm_release.isra.10+0x7c/0x90 [nvidia_uvm] [Fri Apr 30 01:31:18 2021] ? uvm_release_entry+0x6d/0xa0 [nvidia_uvm] [Fri Apr 30 01:31:18 2021] ? __fput+0xc6/0x260 [Fri Apr 30 01:31:18 2021] ? ____fput+0xe/0x10 [Fri Apr 30 01:31:18 2021] ? task_work_run+0x9d/0xc0 [Fri Apr 30 01:31:18 2021] ? do_exit+0x389/0xb90 [Fri Apr 30 01:31:18 2021] ? do_group_exit+0x43/0xa0 [Fri Apr 30 01:31:18 2021] ? get_signal+0x14f/0x860 [Fri Apr 30 01:31:18 2021] ? do_signal+0x34/0x6d0 [Fri Apr 30 01:31:18 2021] ? __switch_to_asm+0x34/0x70 [Fri Apr 30 01:31:18 2021] ? __x64_sys_futex+0x13f/0x190 [Fri Apr 30 01:31:18 2021] ? exit_to_usermode_loop+0x90/0x130 [Fri Apr 30 01:31:18 2021] ? do_syscall_64+0x170/0x190 [Fri Apr 30 01:31:18 2021] ? entry_SYSCALL_64_after_hwframe+0x44/0xa9 [Fri Apr 30 01:31:39 2021] watchdog: BUG: soft lockup - CPU#41 stuck for 23s! [python3:5432] [Fri Apr 30 01:31:39 2021] Modules linked in: nvidia_uvm(OE) veth xt_nat xt_tcpudp xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype iptable_filter iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 bpfilter br_netfilter bridge stp llc aufs cmac overlay bnep snd_hda_codec_hdmi nvidia_drm(POE) nvidia_modeset(POE) edac_mce_amd snd_usb_audio snd_usbmidi_lib mc kvm_amd kvm iwlmvm mac80211 crct10dif_pclmul crc32_pclmul btusb nls_iso8859_1 btrtl nvidia(POE) libarc4 btbcm ghash_clmulni_intel btintel bluetooth iwlwifi ecdh_generic aesni_intel snd_seq_midi input_leds ecc crypto_simd snd_seq_midi_event cryptd glue_helper cfg80211 wmi_bmof snd_hda_intel snd_rawmidi snd_intel_dspcfg joydev snd_hda_codec nouveau snd_hda_core snd_hwdep snd_seq snd_pcm mxm_wmi video snd_seq_device ccp ttm snd_timer drm_kms_helper snd drm i2c_algo_bit fb_sys_fops syscopyarea sysfillrect sysimgblt k10temp soundcore mac_hid sch_fq_codel parport_pc ppdev lp parport ip_tables x_tables [Fri Apr 30 01:31:39 2021] autofs4 btrfs xor zstd_compress raid6_pq libcrc32c dm_mirror dm_region_hash dm_log hid_generic usbhid hid nvme nvme_core ahci r8169 libahci atlantic realtek i2c_piix4 wmi [Fri Apr 30 01:31:39 2021] CPU: 41 PID: 5432 Comm: python3 Tainted: P W OEL 5.4.0-72-generic #80~18.04.1-Ubuntu [Fri Apr 30 01:31:39 2021] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./TRX40 Creator, BIOS P1.70 05/29/2020 [Fri Apr 30 01:31:39 2021] RIP: 0010:fetch_pte.isra.7+0x29/0x1a0 [Fri Apr 30 01:31:39 2021] Code: 00 0f 1f 44 00 00 48 c7 01 00 00 00 00 44 8b 0f 49 89 c8 41 83 f9 05 7f 1a 43 8d 4c c9 0c b8 01 00 00 00 48 d3 e0 48 83 e8 01 <48> 39 d0 0f 82 5c 01 00 00 41 8d 41 ff 55 48 89 d7 44 8d 1c c0 48 [Fri Apr 30 01:31:39 2021] RSP: 0018:fffface947c0f928 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [Fri Apr 30 01:31:39 2021] RAX: 0000ffffffffffff RBX: 0000000000001000 RCX: 0000000000000030 [Fri Apr 30 01:31:39 2021] RDX: 00008a7d86563000 RSI: ffff9dcbdc370098 RDI: ffff9dcbdc370094 [Fri Apr 30 01:31:39 2021] RBP: fffface947c0f968 R08: fffface947c0f930 R09: 0000000000000004 [Fri Apr 30 01:31:39 2021] R10: 0000000000000000 R11: 000000000000001b R12: ffff9dcbdc370098 [Fri Apr 30 01:31:39 2021] R13: ffff9dcbdc370094 R14: 00008a7d86563000 R15: 0000000000000000 [Fri Apr 30 01:31:39 2021] FS: 00007f9e29fff700(0000) GS:ffff9dcbfd840000(0000) knlGS:0000000000000000 [Fri Apr 30 01:31:39 2021] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fri Apr 30 01:31:39 2021] CR2: 000056488693401c CR3: 0000001fbb9b0000 CR4: 0000000000340ee0 [Fri Apr 30 01:31:39 2021] Call Trace: [Fri Apr 30 01:31:39 2021] ? iommu_unmap_page+0x64/0x100 [Fri Apr 30 01:31:39 2021] __unmap_single.isra.27+0x66/0x130 [Fri Apr 30 01:31:39 2021] unmap_sg+0x5f/0x70 [Fri Apr 30 01:31:39 2021] nv_unmap_dma_map_scatterlist+0x57/0xa0 [nvidia] [Fri Apr 30 01:31:39 2021] nv_dma_unmap_pages+0x56/0x100 [nvidia] [Fri Apr 30 01:31:39 2021] nv_dma_unmap_alloc+0x16/0x30 [nvidia] [Fri Apr 30 01:31:39 2021] _nv032962rm+0xc1/0x1b0 [nvidia] [Fri Apr 30 01:31:39 2021] ? _nv027787rm+0x9b/0xc0 [nvidia] [Fri Apr 30 01:31:39 2021] ? _nv030989rm+0x124/0x130 [nvidia] [Fri Apr 30 01:31:39 2021] ? rm_gpu_ops_free_duped_handle+0x1c/0x60 [nvidia] [Fri Apr 30 01:31:39 2021] ? nvUvmInterfaceFreeDupedHandle+0x2a/0x40 [nvidia] [Fri Apr 30 01:31:39 2021] ? uvm_ext_gpu_map_free_internal+0x5c/0x70 [nvidia_uvm] [Fri Apr 30 01:31:39 2021] ? uvm_ext_gpu_map_free+0xe/0x20 [nvidia_uvm] [Fri Apr 30 01:31:39 2021] ? uvm_deferred_free_object_list+0x70/0x130 [nvidia_uvm] [Fri Apr 30 01:31:39 2021] ? uvm_va_space_destroy+0x349/0x480 [nvidia_uvm] [Fri Apr 30 01:31:39 2021] ? uvm_release.isra.10+0x7c/0x90 [nvidia_uvm] [Fri Apr 30 01:31:39 2021] ? uvm_release_entry+0x6d/0xa0 [nvidia_uvm] [Fri Apr 30 01:31:39 2021] ? __fput+0xc6/0x260 [Fri Apr 30 01:31:39 2021] ? ____fput+0xe/0x10 [Fri Apr 30 01:31:39 2021] ? task_work_run+0x9d/0xc0 [Fri Apr 30 01:31:39 2021] ? do_exit+0x389/0xb90 [Fri Apr 30 01:31:39 2021] ? do_group_exit+0x43/0xa0 [Fri Apr 30 01:31:39 2021] ? get_signal+0x14f/0x860 [Fri Apr 30 01:31:39 2021] ? do_signal+0x34/0x6d0 [Fri Apr 30 01:31:39 2021] ? __switch_to_asm+0x34/0x70 [Fri Apr 30 01:31:39 2021] ? __x64_sys_futex+0x13f/0x190 [Fri Apr 30 01:31:39 2021] ? exit_to_usermode_loop+0x90/0x130 [Fri Apr 30 01:31:39 2021] ? do_syscall_64+0x170/0x190 [Fri Apr 30 01:31:39 2021] ? entry_SYSCALL_64_after_hwframe+0x44/0xa9 [Fri Apr 30 01:32:07 2021] watchdog: BUG: soft lockup - CPU#41 stuck for 23s! [python3:5432] [Fri Apr 30 01:32:07 2021] Modules linked in: nvidia_uvm(OE) veth xt_nat xt_tcpudp xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype iptable_filter iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 bpfilter br_netfilter bridge stp llc aufs cmac overlay bnep snd_hda_codec_hdmi nvidia_drm(POE) nvidia_modeset(POE) edac_mce_amd snd_usb_audio snd_usbmidi_lib mc kvm_amd kvm iwlmvm mac80211 crct10dif_pclmul crc32_pclmul btusb nls_iso8859_1 btrtl nvidia(POE) libarc4 btbcm ghash_clmulni_intel btintel bluetooth iwlwifi ecdh_generic aesni_intel snd_seq_midi input_leds ecc crypto_simd snd_seq_midi_event cryptd glue_helper cfg80211 wmi_bmof snd_hda_intel snd_rawmidi snd_intel_dspcfg joydev snd_hda_codec nouveau snd_hda_core snd_hwdep snd_seq snd_pcm mxm_wmi video snd_seq_device ccp ttm snd_timer drm_kms_helper snd drm i2c_algo_bit fb_sys_fops syscopyarea sysfillrect sysimgblt k10temp soundcore mac_hid sch_fq_codel parport_pc ppdev lp parport ip_tables x_tables [Fri Apr 30 01:32:07 2021] autofs4 btrfs xor zstd_compress raid6_pq libcrc32c dm_mirror dm_region_hash dm_log hid_generic usbhid hid nvme nvme_core ahci r8169 libahci atlantic realtek i2c_piix4 wmi [Fri Apr 30 01:32:07 2021] CPU: 41 PID: 5432 Comm: python3 Tainted: P W OEL 5.4.0-72-generic #80~18.04.1-Ubuntu [Fri Apr 30 01:32:07 2021] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./TRX40 Creator, BIOS P1.70 05/29/2020 [Fri Apr 30 01:32:07 2021] RIP: 0010:iommu_unmap_page+0xb/0x100 [Fri Apr 30 01:32:07 2021] Code: ea 01 48 c1 ea 0c e8 a4 da ff ff 48 85 c0 74 02 5d c3 0f 0b 5d c3 e8 a4 ff ff ff 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 41 57 <41> 56 41 55 41 54 53 48 83 ec 10 65 48 8b 04 25 28 00 00 00 48 89 [Fri Apr 30 01:32:07 2021] RSP: 0018:fffface947c0f960 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [Fri Apr 30 01:32:07 2021] RAX: 0000008000000000 RBX: 0000891093c5f000 RCX: 0000000000000027 [Fri Apr 30 01:32:07 2021] RDX: 0000000000001000 RSI: 0000891093c5e000 RDI: ffff9dcbdc370000 [Fri Apr 30 01:32:07 2021] RBP: fffface947c0f968 R08: fffface947c0f930 R09: 0000000000000004 [Fri Apr 30 01:32:07 2021] R10: 0000000000000000 R11: 000000000000001b R12: ffff9dcbdc370000 [Fri Apr 30 01:32:07 2021] R13: 00008ffbfffff000 R14: 00000000ffffffff R15: 00007ffc00000000 [Fri Apr 30 01:32:07 2021] FS: 00007f9e29fff700(0000) GS:ffff9dcbfd840000(0000) knlGS:0000000000000000 [Fri Apr 30 01:32:07 2021] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fri Apr 30 01:32:07 2021] CR2: 000056488693401c CR3: 0000001fbb9b0000 CR4: 0000000000340ee0 [Fri Apr 30 01:32:07 2021] Call Trace: [Fri Apr 30 01:32:07 2021] __unmap_single.isra.27+0x66/0x130 [Fri Apr 30 01:32:07 2021] unmap_sg+0x5f/0x70 [Fri Apr 30 01:32:07 2021] nv_unmap_dma_map_scatterlist+0x57/0xa0 [nvidia] [Fri Apr 30 01:32:07 2021] nv_dma_unmap_pages+0x56/0x100 [nvidia] [Fri Apr 30 01:32:07 2021] nv_dma_unmap_alloc+0x16/0x30 [nvidia] [Fri Apr 30 01:32:07 2021] _nv032962rm+0xc1/0x1b0 [nvidia] [Fri Apr 30 01:32:07 2021] ? _nv027787rm+0x9b/0xc0 [nvidia] [Fri Apr 30 01:32:07 2021] ? _nv030989rm+0x124/0x130 [nvidia] [Fri Apr 30 01:32:07 2021] ? rm_gpu_ops_free_duped_handle+0x1c/0x60 [nvidia] [Fri Apr 30 01:32:07 2021] ? nvUvmInterfaceFreeDupedHandle+0x2a/0x40 [nvidia] [Fri Apr 30 01:32:07 2021] ? uvm_ext_gpu_map_free_internal+0x5c/0x70 [nvidia_uvm] [Fri Apr 30 01:32:07 2021] ? uvm_ext_gpu_map_free+0xe/0x20 [nvidia_uvm] [Fri Apr 30 01:32:07 2021] ? uvm_deferred_free_object_list+0x70/0x130 [nvidia_uvm] [Fri Apr 30 01:32:07 2021] ? uvm_va_space_destroy+0x349/0x480 [nvidia_uvm] [Fri Apr 30 01:32:07 2021] ? uvm_release.isra.10+0x7c/0x90 [nvidia_uvm] [Fri Apr 30 01:32:07 2021] ? uvm_release_entry+0x6d/0xa0 [nvidia_uvm] [Fri Apr 30 01:32:07 2021] ? __fput+0xc6/0x260 [Fri Apr 30 01:32:07 2021] ? ____fput+0xe/0x10 [Fri Apr 30 01:32:07 2021] ? task_work_run+0x9d/0xc0 [Fri Apr 30 01:32:07 2021] ? do_exit+0x389/0xb90 [Fri Apr 30 01:32:07 2021] ? do_group_exit+0x43/0xa0 [Fri Apr 30 01:32:07 2021] ? get_signal+0x14f/0x860 [Fri Apr 30 01:32:07 2021] ? do_signal+0x34/0x6d0 [Fri Apr 30 01:32:07 2021] ? __switch_to_asm+0x34/0x70 [Fri Apr 30 01:32:07 2021] ? __x64_sys_futex+0x13f/0x190 [Fri Apr 30 01:32:07 2021] ? exit_to_usermode_loop+0x90/0x130 [Fri Apr 30 01:32:07 2021] ? do_syscall_64+0x170/0x190 [Fri Apr 30 01:32:07 2021] ? entry_SYSCALL_64_after_hwframe+0x44/0xa9 [Fri Apr 30 01:32:19 2021] rcu: INFO: rcu_sched self-detected stall on CPU [Fri Apr 30 01:32:19 2021] rcu: 41-....: (15000 ticks this GP) idle=afe/1/0x4000000000000002 softirq=8425/8426 fqs=7378 [Fri Apr 30 01:32:19 2021] (t=15000 jiffies g=47433 q=297038) [Fri Apr 30 01:32:19 2021] NMI backtrace for cpu 41 [Fri Apr 30 01:32:19 2021] CPU: 41 PID: 5432 Comm: python3 Tainted: P W OEL 5.4.0-72-generic #80~18.04.1-Ubuntu [Fri Apr 30 01:32:19 2021] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./TRX40 Creator, BIOS P1.70 05/29/2020 [Fri Apr 30 01:32:19 2021] Call Trace: [Fri Apr 30 01:32:19 2021] [Fri Apr 30 01:32:19 2021] dump_stack+0x6d/0x8b [Fri Apr 30 01:32:19 2021] nmi_cpu_backtrace+0x94/0xa0 [Fri Apr 30 01:32:19 2021] ? lapic_can_unplug_cpu+0xb0/0xb0 [Fri Apr 30 01:32:19 2021] nmi_trigger_cpumask_backtrace+0xe7/0x130 [Fri Apr 30 01:32:19 2021] arch_trigger_cpumask_backtrace+0x19/0x20 [Fri Apr 30 01:32:19 2021] rcu_dump_cpu_stacks+0x9e/0xce [Fri Apr 30 01:32:19 2021] rcu_sched_clock_irq+0x508/0x770 [Fri Apr 30 01:32:19 2021] ? tick_sched_do_timer+0x60/0x60 [Fri Apr 30 01:32:19 2021] update_process_times+0x28/0x50 [Fri Apr 30 01:32:19 2021] tick_sched_handle+0x25/0x70 [Fri Apr 30 01:32:19 2021] tick_sched_timer+0x3c/0x80 [Fri Apr 30 01:32:19 2021] __hrtimer_run_queues+0x10f/0x280 [Fri Apr 30 01:32:19 2021] hrtimer_interrupt+0xe7/0x240 [Fri Apr 30 01:32:19 2021] smp_apic_timer_interrupt+0x6f/0x130 [Fri Apr 30 01:32:19 2021] apic_timer_interrupt+0xf/0x20 [Fri Apr 30 01:32:19 2021] [Fri Apr 30 01:32:19 2021] RIP: 0010:iommu_unmap_page+0x3e/0x100 [Fri Apr 30 01:32:19 2021] Code: ec 10 65 48 8b 04 25 28 00 00 00 48 89 45 d0 31 c0 48 85 d2 0f 84 b9 00 00 00 48 8d 42 ff 48 89 d3 48 85 d0 0f 85 a9 00 00 00 <4c> 8d af 94 00 00 00 4c 8d a7 98 00 00 00 49 89 f6 45 31 ff 48 8d [Fri Apr 30 01:32:19 2021] RSP: 0018:fffface947c0f930 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13 [Fri Apr 30 01:32:19 2021] RAX: 0000000000000fff RBX: 0000000000001000 RCX: 0000000000000027 [Fri Apr 30 01:32:19 2021] RDX: 0000000000001000 RSI: 00008f3ead7f2000 RDI: ffff9dcbdc370000 [Fri Apr 30 01:32:19 2021] RBP: fffface947c0f968 R08: fffface947c0f930 R09: 0000000000000004 [Fri Apr 30 01:32:19 2021] R10: 0000000000000000 R11: 000000000000001b R12: ffff9dcbdc370000 [Fri Apr 30 01:32:19 2021] R13: 00008ffbfffff000 R14: 00000000ffffffff R15: 00007ffc00000000 [Fri Apr 30 01:32:19 2021] __unmap_single.isra.27+0x66/0x130 [Fri Apr 30 01:32:19 2021] unmap_sg+0x5f/0x70 [Fri Apr 30 01:32:19 2021] nv_unmap_dma_map_scatterlist+0x57/0xa0 [nvidia] [Fri Apr 30 01:32:19 2021] nv_dma_unmap_pages+0x56/0x100 [nvidia] [Fri Apr 30 01:32:19 2021] nv_dma_unmap_alloc+0x16/0x30 [nvidia] [Fri Apr 30 01:32:19 2021] _nv032962rm+0xc1/0x1b0 [nvidia] [Fri Apr 30 01:32:19 2021] ? _nv027787rm+0x9b/0xc0 [nvidia] [Fri Apr 30 01:32:19 2021] ? _nv030989rm+0x124/0x130 [nvidia] [Fri Apr 30 01:32:19 2021] ? rm_gpu_ops_free_duped_handle+0x1c/0x60 [nvidia] [Fri Apr 30 01:32:19 2021] ? nvUvmInterfaceFreeDupedHandle+0x2a/0x40 [nvidia] [Fri Apr 30 01:32:19 2021] ? uvm_ext_gpu_map_free_internal+0x5c/0x70 [nvidia_uvm] [Fri Apr 30 01:32:19 2021] ? uvm_ext_gpu_map_free+0xe/0x20 [nvidia_uvm] [Fri Apr 30 01:32:19 2021] ? uvm_deferred_free_object_list+0x70/0x130 [nvidia_uvm] [Fri Apr 30 01:32:19 2021] ? uvm_va_space_destroy+0x349/0x480 [nvidia_uvm] [Fri Apr 30 01:32:19 2021] ? uvm_release.isra.10+0x7c/0x90 [nvidia_uvm] [Fri Apr 30 01:32:19 2021] ? uvm_release_entry+0x6d/0xa0 [nvidia_uvm] [Fri Apr 30 01:32:19 2021] ? __fput+0xc6/0x260 [Fri Apr 30 01:32:19 2021] ? ____fput+0xe/0x10 [Fri Apr 30 01:32:19 2021] ? task_work_run+0x9d/0xc0 [Fri Apr 30 01:32:19 2021] ? do_exit+0x389/0xb90 [Fri Apr 30 01:32:19 2021] ? do_group_exit+0x43/0xa0 [Fri Apr 30 01:32:19 2021] ? get_signal+0x14f/0x860 [Fri Apr 30 01:32:19 2021] ? do_signal+0x34/0x6d0 [Fri Apr 30 01:32:19 2021] ? __switch_to_asm+0x34/0x70 [Fri Apr 30 01:32:19 2021] ? __x64_sys_futex+0x13f/0x190 [Fri Apr 30 01:32:19 2021] ? exit_to_usermode_loop+0x90/0x130 [Fri Apr 30 01:32:19 2021] ? do_syscall_64+0x170/0x190 [Fri Apr 30 01:32:19 2021] ? entry_SYSCALL_64_after_hwframe+0x44/0xa9 [Fri Apr 30 01:32:43 2021] watchdog: BUG: soft lockup - CPU#41 stuck for 21s! [python3:5432] [Fri Apr 30 01:32:43 2021] Modules linked in: nvidia_uvm(OE) veth xt_nat xt_tcpudp xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype iptable_filter iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 bpfilter br_netfilter bridge stp llc aufs cmac overlay bnep snd_hda_codec_hdmi nvidia_drm(POE) nvidia_modeset(POE) edac_mce_amd snd_usb_audio snd_usbmidi_lib mc kvm_amd kvm iwlmvm mac80211 crct10dif_pclmul crc32_pclmul btusb nls_iso8859_1 btrtl nvidia(POE) libarc4 btbcm ghash_clmulni_intel btintel bluetooth iwlwifi ecdh_generic aesni_intel snd_seq_midi input_leds ecc crypto_simd snd_seq_midi_event cryptd glue_helper cfg80211 wmi_bmof snd_hda_intel snd_rawmidi snd_intel_dspcfg joydev snd_hda_codec nouveau snd_hda_core snd_hwdep snd_seq snd_pcm mxm_wmi video snd_seq_device ccp ttm snd_timer drm_kms_helper snd drm i2c_algo_bit fb_sys_fops syscopyarea sysfillrect sysimgblt k10temp soundcore mac_hid sch_fq_codel parport_pc ppdev lp parport ip_tables x_tables [Fri Apr 30 01:32:43 2021] autofs4 btrfs xor zstd_compress raid6_pq libcrc32c dm_mirror dm_region_hash dm_log hid_generic usbhid hid nvme nvme_core ahci r8169 libahci atlantic realtek i2c_piix4 wmi [Fri Apr 30 01:32:43 2021] CPU: 41 PID: 5432 Comm: python3 Tainted: P W OEL 5.4.0-72-generic #80~18.04.1-Ubuntu [Fri Apr 30 01:32:43 2021] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./TRX40 Creator, BIOS P1.70 05/29/2020 [Fri Apr 30 01:32:43 2021] RIP: 0010:iommu_unmap_page+0xdd/0x100 [Fri Apr 30 01:32:43 2021] Code: f7 da 4c 21 f2 4c 39 fb 4c 8d 34 10 77 91 4d 85 ff 75 23 48 8b 7d d0 65 48 33 3c 25 28 00 00 00 4c 89 f8 75 1c 48 83 c4 10 5b <41> 5c 41 5d 41 5e 41 5f 5d c3 0f 0b 49 8d 47 ff 4c 85 f8 74 d4 0f [Fri Apr 30 01:32:43 2021] RSP: 0018:fffface947c0f948 EFLAGS: 00000282 ORIG_RAX: ffffffffffffff13 [Fri Apr 30 01:32:43 2021] RAX: 0000008000000000 RBX: 00008bd794bb7000 RCX: 0000000000000027 [Fri Apr 30 01:32:43 2021] RDX: 00008b8000000000 RSI: 0000000000000000 RDI: 0000000000000000 [Fri Apr 30 01:32:43 2021] RBP: fffface947c0f968 R08: fffface947c0f930 R09: 0000000000000004 [Fri Apr 30 01:32:43 2021] R10: 0000000000000000 R11: 000000000000001b R12: ffff9dcbe418b098 [Fri Apr 30 01:32:43 2021] R13: ffff9dcbe418b094 R14: 00008c0000000000 R15: 0000008000000000 [Fri Apr 30 01:32:43 2021] FS: 00007f9e29fff700(0000) GS:ffff9dcbfd840000(0000) knlGS:0000000000000000 [Fri Apr 30 01:32:43 2021] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fri Apr 30 01:32:43 2021] CR2: 000056488693401c CR3: 0000001fbb9b0000 CR4: 0000000000340ee0 [Fri Apr 30 01:32:43 2021] Call Trace: [Fri Apr 30 01:32:43 2021] __unmap_single.isra.27+0x66/0x130 [Fri Apr 30 01:32:43 2021] unmap_sg+0x5f/0x70 [Fri Apr 30 01:32:43 2021] nv_unmap_dma_map_scatterlist+0x57/0xa0 [nvidia] [Fri Apr 30 01:32:43 2021] nv_dma_unmap_pages+0x56/0x100 [nvidia] [Fri Apr 30 01:32:43 2021] nv_dma_unmap_alloc+0x16/0x30 [nvidia] [Fri Apr 30 01:32:43 2021] _nv032962rm+0xc1/0x1b0 [nvidia] [Fri Apr 30 01:32:43 2021] ? _nv027787rm+0x9b/0xc0 [nvidia] [Fri Apr 30 01:32:43 2021] ? _nv030989rm+0x124/0x130 [nvidia] [Fri Apr 30 01:32:43 2021] ? rm_gpu_ops_free_duped_handle+0x1c/0x60 [nvidia] [Fri Apr 30 01:32:43 2021] ? nvUvmInterfaceFreeDupedHandle+0x2a/0x40 [nvidia] [Fri Apr 30 01:32:43 2021] ? uvm_ext_gpu_map_free_internal+0x5c/0x70 [nvidia_uvm] [Fri Apr 30 01:32:43 2021] ? uvm_ext_gpu_map_free+0xe/0x20 [nvidia_uvm] [Fri Apr 30 01:32:43 2021] ? uvm_deferred_free_object_list+0x70/0x130 [nvidia_uvm] [Fri Apr 30 01:32:43 2021] ? uvm_va_space_destroy+0x349/0x480 [nvidia_uvm] [Fri Apr 30 01:32:43 2021] ? uvm_release.isra.10+0x7c/0x90 [nvidia_uvm] [Fri Apr 30 01:32:43 2021] ? uvm_release_entry+0x6d/0xa0 [nvidia_uvm] [Fri Apr 30 01:32:43 2021] ? __fput+0xc6/0x260 [Fri Apr 30 01:32:43 2021] ? ____fput+0xe/0x10 [Fri Apr 30 01:32:43 2021] ? task_work_run+0x9d/0xc0 [Fri Apr 30 01:32:43 2021] ? do_exit+0x389/0xb90 [Fri Apr 30 01:32:43 2021] ? do_group_exit+0x43/0xa0 [Fri Apr 30 01:32:43 2021] ? get_signal+0x14f/0x860 [Fri Apr 30 01:32:43 2021] ? do_signal+0x34/0x6d0 [Fri Apr 30 01:32:43 2021] ? __switch_to_asm+0x34/0x70 [Fri Apr 30 01:32:43 2021] ? __x64_sys_futex+0x13f/0x190 [Fri Apr 30 01:32:43 2021] ? exit_to_usermode_loop+0x90/0x130 [Fri Apr 30 01:32:43 2021] ? do_syscall_64+0x170/0x190 [Fri Apr 30 01:32:43 2021] ? entry_SYSCALL_64_after_hwframe+0x44/0xa9 [Fri Apr 30 01:33:15 2021] watchdog: BUG: soft lockup - CPU#16 stuck for 22s! [python3:5432] [Fri Apr 30 01:33:15 2021] Modules linked in: nvidia_uvm(OE) veth xt_nat xt_tcpudp xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype iptable_filter iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 bpfilter br_netfilter bridge stp llc aufs cmac overlay bnep snd_hda_codec_hdmi nvidia_drm(POE) nvidia_modeset(POE) edac_mce_amd snd_usb_audio snd_usbmidi_lib mc kvm_amd kvm iwlmvm mac80211 crct10dif_pclmul crc32_pclmul btusb nls_iso8859_1 btrtl nvidia(POE) libarc4 btbcm ghash_clmulni_intel btintel bluetooth iwlwifi ecdh_generic aesni_intel snd_seq_midi input_leds ecc crypto_simd snd_seq_midi_event cryptd glue_helper cfg80211 wmi_bmof snd_hda_intel snd_rawmidi snd_intel_dspcfg joydev snd_hda_codec nouveau snd_hda_core snd_hwdep snd_seq snd_pcm mxm_wmi video snd_seq_device ccp ttm snd_timer drm_kms_helper snd drm i2c_algo_bit fb_sys_fops syscopyarea sysfillrect sysimgblt k10temp soundcore mac_hid sch_fq_codel parport_pc ppdev lp parport ip_tables x_tables [Fri Apr 30 01:33:15 2021] autofs4 btrfs xor zstd_compress raid6_pq libcrc32c dm_mirror dm_region_hash dm_log hid_generic usbhid hid nvme nvme_core ahci r8169 libahci atlantic realtek i2c_piix4 wmi [Fri Apr 30 01:33:15 2021] CPU: 16 PID: 5432 Comm: python3 Tainted: P W OEL 5.4.0-72-generic #80~18.04.1-Ubuntu [Fri Apr 30 01:33:15 2021] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./TRX40 Creator, BIOS P1.70 05/29/2020 [Fri Apr 30 01:33:15 2021] RIP: 0010:iommu_unmap_page+0xd3/0x100 [Fri Apr 30 01:33:15 2021] Code: 8b 45 c8 48 89 c2 49 01 c7 48 f7 da 4c 21 f2 4c 39 fb 4c 8d 34 10 77 91 4d 85 ff 75 23 48 8b 7d d0 65 48 33 3c 25 28 00 00 00 <4c> 89 f8 75 1c 48 83 c4 10 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 0b [Fri Apr 30 01:33:15 2021] RSP: 0018:fffface947c0f930 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13 [Fri Apr 30 01:33:15 2021] RAX: 0000007fffffffff RBX: 0000000000001000 RCX: 0000000000000027 [Fri Apr 30 01:33:15 2021] RDX: 00008c8000000000 RSI: 0000000000000000 RDI: 0000000000000000 [Fri Apr 30 01:33:15 2021] RBP: fffface947c0f968 R08: fffface947c0f930 R09: 0000000000000004 [Fri Apr 30 01:33:15 2021] R10: 0000000000000000 R11: 000000000000001b R12: ffff9dcbdc371098 [Fri Apr 30 01:33:15 2021] R13: ffff9dcbdc371094 R14: 00008d0000000000 R15: 0000008000000000 [Fri Apr 30 01:33:15 2021] FS: 00007f9e29fff700(0000) GS:ffff9dcbfd200000(0000) knlGS:0000000000000000 [Fri Apr 30 01:33:15 2021] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fri Apr 30 01:33:15 2021] CR2: 00005648868d483c CR3: 0000001fe6be2000 CR4: 0000000000340ee0 [Fri Apr 30 01:33:15 2021] Call Trace: [Fri Apr 30 01:33:15 2021] __unmap_single.isra.27+0x66/0x130 [Fri Apr 30 01:33:15 2021] unmap_sg+0x5f/0x70 [Fri Apr 30 01:33:15 2021] nv_unmap_dma_map_scatterlist+0x57/0xa0 [nvidia] [Fri Apr 30 01:33:15 2021] nv_dma_unmap_pages+0x56/0x100 [nvidia] [Fri Apr 30 01:33:15 2021] nv_dma_unmap_alloc+0x16/0x30 [nvidia] [Fri Apr 30 01:33:15 2021] _nv032962rm+0xc1/0x1b0 [nvidia] [Fri Apr 30 01:33:15 2021] ? _nv027787rm+0x9b/0xc0 [nvidia] [Fri Apr 30 01:33:15 2021] ? _nv030989rm+0x124/0x130 [nvidia] [Fri Apr 30 01:33:15 2021] ? rm_gpu_ops_free_duped_handle+0x1c/0x60 [nvidia] [Fri Apr 30 01:33:15 2021] ? nvUvmInterfaceFreeDupedHandle+0x2a/0x40 [nvidia] [Fri Apr 30 01:33:15 2021] ? uvm_ext_gpu_map_free_internal+0x5c/0x70 [nvidia_uvm] [Fri Apr 30 01:33:15 2021] ? uvm_ext_gpu_map_free+0xe/0x20 [nvidia_uvm] [Fri Apr 30 01:33:15 2021] ? uvm_deferred_free_object_list+0x70/0x130 [nvidia_uvm] [Fri Apr 30 01:33:15 2021] ? uvm_va_space_destroy+0x349/0x480 [nvidia_uvm] [Fri Apr 30 01:33:15 2021] ? uvm_release.isra.10+0x7c/0x90 [nvidia_uvm] [Fri Apr 30 01:33:15 2021] ? uvm_release_entry+0x6d/0xa0 [nvidia_uvm] [Fri Apr 30 01:33:15 2021] ? __fput+0xc6/0x260 [Fri Apr 30 01:33:15 2021] ? ____fput+0xe/0x10 [Fri Apr 30 01:33:15 2021] ? task_work_run+0x9d/0xc0 [Fri Apr 30 01:33:15 2021] ? do_exit+0x389/0xb90 [Fri Apr 30 01:33:15 2021] ? do_group_exit+0x43/0xa0 [Fri Apr 30 01:33:15 2021] ? get_signal+0x14f/0x860 [Fri Apr 30 01:33:15 2021] ? do_signal+0x34/0x6d0 [Fri Apr 30 01:33:15 2021] ? __switch_to_asm+0x34/0x70 [Fri Apr 30 01:33:15 2021] ? __x64_sys_futex+0x13f/0x190 [Fri Apr 30 01:33:15 2021] ? exit_to_usermode_loop+0x90/0x130 [Fri Apr 30 01:33:15 2021] ? do_syscall_64+0x170/0x190 [Fri Apr 30 01:33:15 2021] ? entry_SYSCALL_64_after_hwframe+0x44/0xa9 [Fri Apr 30 01:33:43 2021] watchdog: BUG: soft lockup - CPU#40 stuck for 22s! [python3:5432] [Fri Apr 30 01:33:43 2021] Modules linked in: nvidia_uvm(OE) veth xt_nat xt_tcpudp xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype iptable_filter iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 bpfilter br_netfilter bridge stp llc aufs cmac overlay bnep snd_hda_codec_hdmi nvidia_drm(POE) nvidia_modeset(POE) edac_mce_amd snd_usb_audio snd_usbmidi_lib mc kvm_amd kvm iwlmvm mac80211 crct10dif_pclmul crc32_pclmul btusb nls_iso8859_1 btrtl nvidia(POE) libarc4 btbcm ghash_clmulni_intel btintel bluetooth iwlwifi ecdh_generic aesni_intel snd_seq_midi input_leds ecc crypto_simd snd_seq_midi_event cryptd glue_helper cfg80211 wmi_bmof snd_hda_intel snd_rawmidi snd_intel_dspcfg joydev snd_hda_codec nouveau snd_hda_core snd_hwdep snd_seq snd_pcm mxm_wmi video snd_seq_device ccp ttm snd_timer drm_kms_helper snd drm i2c_algo_bit fb_sys_fops syscopyarea sysfillrect sysimgblt k10temp soundcore mac_hid sch_fq_codel parport_pc ppdev lp parport ip_tables x_tables [Fri Apr 30 01:33:43 2021] autofs4 btrfs xor zstd_compress raid6_pq libcrc32c dm_mirror dm_region_hash dm_log hid_generic usbhid hid nvme nvme_core ahci r8169 libahci atlantic realtek i2c_piix4 wmi [Fri Apr 30 01:33:43 2021] CPU: 40 PID: 5432 Comm: python3 Tainted: P W OEL 5.4.0-72-generic #80~18.04.1-Ubuntu [Fri Apr 30 01:33:43 2021] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./TRX40 Creator, BIOS P1.70 05/29/2020 [Fri Apr 30 01:33:43 2021] RIP: 0010:fetch_pte.isra.7+0x63/0x1a0 [Fri Apr 30 01:33:43 2021] Code: d7 44 8d 1c c0 48 89 e5 53 48 8b 36 41 8d 4b 0c 48 d3 ef 81 e7 ff 01 00 00 4c 8d 14 fe be 01 00 00 00 48 d3 e6 85 c0 49 89 30 <49> 8b 3a 0f 8e b9 00 00 00 40 f6 c7 01 0f 84 1f 01 00 00 48 89 fe [Fri Apr 30 01:33:43 2021] RSP: 0018:fffface947c0f918 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [Fri Apr 30 01:33:43 2021] RAX: 0000000000000003 RBX: 0000000000001000 RCX: 0000000000000027 [Fri Apr 30 01:33:43 2021] RDX: 00008b431880b000 RSI: 0000008000000000 RDI: 0000000000000116 [Fri Apr 30 01:33:43 2021] RBP: fffface947c0f920 R08: fffface947c0f930 R09: 0000000000000004 [Fri Apr 30 01:33:43 2021] R10: ffff9dc8ebfae8b0 R11: 000000000000001b R12: ffff9dcbdc370098 [Fri Apr 30 01:33:43 2021] R13: ffff9dcbdc370094 R14: 00008b431880b000 R15: 0000000000000000 [Fri Apr 30 01:33:43 2021] FS: 00007f9e29fff700(0000) GS:ffff9dcbfd800000(0000) knlGS:0000000000000000 [Fri Apr 30 01:33:43 2021] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fri Apr 30 01:33:43 2021] CR2: 00007f00cbc90150 CR3: 0000001fbb9b0000 CR4: 0000000000340ee0 [Fri Apr 30 01:33:43 2021] Call Trace: [Fri Apr 30 01:33:43 2021] iommu_unmap_page+0x64/0x100 [Fri Apr 30 01:33:43 2021] __unmap_single.isra.27+0x66/0x130 [Fri Apr 30 01:33:43 2021] unmap_sg+0x5f/0x70 [Fri Apr 30 01:33:43 2021] nv_unmap_dma_map_scatterlist+0x57/0xa0 [nvidia] [Fri Apr 30 01:33:43 2021] nv_dma_unmap_pages+0x56/0x100 [nvidia] [Fri Apr 30 01:33:43 2021] nv_dma_unmap_alloc+0x16/0x30 [nvidia] [Fri Apr 30 01:33:43 2021] _nv032962rm+0xc1/0x1b0 [nvidia] [Fri Apr 30 01:33:43 2021] ? _nv027787rm+0x9b/0xc0 [nvidia] [Fri Apr 30 01:33:43 2021] ? _nv030989rm+0x124/0x130 [nvidia] [Fri Apr 30 01:33:43 2021] ? rm_gpu_ops_free_duped_handle+0x1c/0x60 [nvidia] [Fri Apr 30 01:33:43 2021] ? nvUvmInterfaceFreeDupedHandle+0x2a/0x40 [nvidia] [Fri Apr 30 01:33:43 2021] ? uvm_ext_gpu_map_free_internal+0x5c/0x70 [nvidia_uvm] [Fri Apr 30 01:33:43 2021] ? uvm_ext_gpu_map_free+0xe/0x20 [nvidia_uvm] [Fri Apr 30 01:33:43 2021] ? uvm_deferred_free_object_list+0x70/0x130 [nvidia_uvm] [Fri Apr 30 01:33:43 2021] ? uvm_va_space_destroy+0x349/0x480 [nvidia_uvm] [Fri Apr 30 01:33:43 2021] ? uvm_release.isra.10+0x7c/0x90 [nvidia_uvm] [Fri Apr 30 01:33:43 2021] ? uvm_release_entry+0x6d/0xa0 [nvidia_uvm] [Fri Apr 30 01:33:43 2021] ? __fput+0xc6/0x260 [Fri Apr 30 01:33:43 2021] ? ____fput+0xe/0x10 [Fri Apr 30 01:33:43 2021] ? task_work_run+0x9d/0xc0 [Fri Apr 30 01:33:43 2021] ? do_exit+0x389/0xb90 [Fri Apr 30 01:33:43 2021] ? do_group_exit+0x43/0xa0 [Fri Apr 30 01:33:43 2021] ? get_signal+0x14f/0x860 [Fri Apr 30 01:33:43 2021] ? do_signal+0x34/0x6d0 [Fri Apr 30 01:33:43 2021] ? __switch_to_asm+0x34/0x70 [Fri Apr 30 01:33:43 2021] ? __x64_sys_futex+0x13f/0x190 [Fri Apr 30 01:33:43 2021] ? exit_to_usermode_loop+0x90/0x130 [Fri Apr 30 01:33:43 2021] ? do_syscall_64+0x170/0x190 [Fri Apr 30 01:33:43 2021] ? entry_SYSCALL_64_after_hwframe+0x44/0xa9 [Fri Apr 30 01:34:15 2021] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [python3:5432] [Fri Apr 30 01:34:15 2021] Modules linked in: nvidia_uvm(OE) veth xt_nat xt_tcpudp xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype iptable_filter iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 bpfilter br_netfilter bridge stp llc aufs cmac overlay bnep snd_hda_codec_hdmi nvidia_drm(POE) nvidia_modeset(POE) edac_mce_amd snd_usb_audio snd_usbmidi_lib mc kvm_amd kvm iwlmvm mac80211 crct10dif_pclmul crc32_pclmul btusb nls_iso8859_1 btrtl nvidia(POE) libarc4 btbcm ghash_clmulni_intel btintel bluetooth iwlwifi ecdh_generic aesni_intel snd_seq_midi input_leds ecc crypto_simd snd_seq_midi_event cryptd glue_helper cfg80211 wmi_bmof snd_hda_intel snd_rawmidi snd_intel_dspcfg joydev snd_hda_codec nouveau snd_hda_core snd_hwdep snd_seq snd_pcm mxm_wmi video snd_seq_device ccp ttm snd_timer drm_kms_helper snd drm i2c_algo_bit fb_sys_fops syscopyarea sysfillrect sysimgblt k10temp soundcore mac_hid sch_fq_codel parport_pc ppdev lp parport ip_tables x_tables [Fri Apr 30 01:34:15 2021] autofs4 btrfs xor zstd_compress raid6_pq libcrc32c dm_mirror dm_region_hash dm_log hid_generic usbhid hid nvme nvme_core ahci r8169 libahci atlantic realtek i2c_piix4 wmi [Fri Apr 30 01:34:15 2021] CPU: 3 PID: 5432 Comm: python3 Tainted: P W OEL 5.4.0-72-generic #80~18.04.1-Ubuntu [Fri Apr 30 01:34:15 2021] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./TRX40 Creator, BIOS P1.70 05/29/2020 [Fri Apr 30 01:34:15 2021] RIP: 0010:iommu_unmap_page+0xc4/0x100 [Fri Apr 30 01:34:15 2021] Code: 04 c6 00 00 00 00 48 83 c0 01 39 c2 7f f0 48 8b 45 c8 48 89 c2 49 01 c7 48 f7 da 4c 21 f2 4c 39 fb 4c 8d 34 10 77 91 4d 85 ff <75> 23 48 8b 7d d0 65 48 33 3c 25 28 00 00 00 4c 89 f8 75 1c 48 83 [Fri Apr 30 01:34:15 2021] RSP: 0018:fffface947c0f930 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [Fri Apr 30 01:34:15 2021] RAX: 0000008000000000 RBX: 0000000000001000 RCX: 0000000000000027 [Fri Apr 30 01:34:15 2021] RDX: 00008c0000000000 RSI: 0000000000000000 RDI: 0000000000000000 [Fri Apr 30 01:34:15 2021] RBP: fffface947c0f968 R08: fffface947c0f930 R09: 0000000000000004 [Fri Apr 30 01:34:15 2021] R10: 0000000000000000 R11: 000000000000001b R12: ffff9dcbe418b098 [Fri Apr 30 01:34:15 2021] R13: ffff9dcbe418b094 R14: 00008c8000000000 R15: 0000008000000000 [Fri Apr 30 01:34:15 2021] FS: 00007f9e29fff700(0000) GS:ffff9dcbfcec0000(0000) knlGS:0000000000000000 [Fri Apr 30 01:34:15 2021] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fri Apr 30 01:34:15 2021] CR2: 00007f34d2eec8a8 CR3: 000000108660a000 CR4: 0000000000340ee0 [Fri Apr 30 01:34:15 2021] Call Trace: [Fri Apr 30 01:34:15 2021] __unmap_single.isra.27+0x66/0x130 [Fri Apr 30 01:34:15 2021] unmap_sg+0x5f/0x70 [Fri Apr 30 01:34:15 2021] nv_unmap_dma_map_scatterlist+0x57/0xa0 [nvidia] [Fri Apr 30 01:34:15 2021] nv_dma_unmap_pages+0x56/0x100 [nvidia] [Fri Apr 30 01:34:15 2021] nv_dma_unmap_alloc+0x16/0x30 [nvidia] [Fri Apr 30 01:34:15 2021] _nv032962rm+0xc1/0x1b0 [nvidia] [Fri Apr 30 01:34:15 2021] ? _nv027787rm+0x9b/0xc0 [nvidia] [Fri Apr 30 01:34:15 2021] ? _nv030989rm+0x124/0x130 [nvidia] [Fri Apr 30 01:34:15 2021] ? rm_gpu_ops_free_duped_handle+0x1c/0x60 [nvidia] [Fri Apr 30 01:34:15 2021] ? nvUvmInterfaceFreeDupedHandle+0x2a/0x40 [nvidia] [Fri Apr 30 01:34:15 2021] ? uvm_ext_gpu_map_free_internal+0x5c/0x70 [nvidia_uvm] [Fri Apr 30 01:34:15 2021] ? uvm_ext_gpu_map_free+0xe/0x20 [nvidia_uvm] [Fri Apr 30 01:34:15 2021] ? uvm_deferred_free_object_list+0x70/0x130 [nvidia_uvm] [Fri Apr 30 01:34:15 2021] ? uvm_va_space_destroy+0x349/0x480 [nvidia_uvm] [Fri Apr 30 01:34:15 2021] ? uvm_release.isra.10+0x7c/0x90 [nvidia_uvm] [Fri Apr 30 01:34:15 2021] ? uvm_release_entry+0x6d/0xa0 [nvidia_uvm] [Fri Apr 30 01:34:15 2021] ? __fput+0xc6/0x260 [Fri Apr 30 01:34:15 2021] ? ____fput+0xe/0x10 [Fri Apr 30 01:34:15 2021] ? task_work_run+0x9d/0xc0 [Fri Apr 30 01:34:15 2021] ? do_exit+0x389/0xb90 [Fri Apr 30 01:34:15 2021] ? do_group_exit+0x43/0xa0 [Fri Apr 30 01:34:15 2021] ? get_signal+0x14f/0x860 [Fri Apr 30 01:34:15 2021] ? do_signal+0x34/0x6d0 [Fri Apr 30 01:34:15 2021] ? __switch_to_asm+0x34/0x70 [Fri Apr 30 01:34:15 2021] ? __x64_sys_futex+0x13f/0x190 [Fri Apr 30 01:34:15 2021] ? exit_to_usermode_loop+0x90/0x130 [Fri Apr 30 01:34:15 2021] ? do_syscall_64+0x170/0x190 [Fri Apr 30 01:34:15 2021] ? entry_SYSCALL_64_after_hwframe+0x44/0xa9 [Fri Apr 30 01:34:43 2021] watchdog: BUG: soft lockup - CPU#27 stuck for 22s! [python3:5432] [Fri Apr 30 01:34:43 2021] Modules linked in: nvidia_uvm(OE) veth xt_nat xt_tcpudp xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype iptable_filter iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 bpfilter br_netfilter bridge stp llc aufs cmac overlay bnep snd_hda_codec_hdmi nvidia_drm(POE) nvidia_modeset(POE) edac_mce_amd snd_usb_audio snd_usbmidi_lib mc kvm_amd kvm iwlmvm mac80211 crct10dif_pclmul crc32_pclmul btusb nls_iso8859_1 btrtl nvidia(POE) libarc4 btbcm ghash_clmulni_intel btintel bluetooth iwlwifi ecdh_generic aesni_intel snd_seq_midi input_leds ecc crypto_simd snd_seq_midi_event cryptd glue_helper cfg80211 wmi_bmof snd_hda_intel snd_rawmidi snd_intel_dspcfg joydev snd_hda_codec nouveau snd_hda_core snd_hwdep snd_seq snd_pcm mxm_wmi video snd_seq_device ccp ttm snd_timer drm_kms_helper snd drm i2c_algo_bit fb_sys_fops syscopyarea sysfillrect sysimgblt k10temp soundcore mac_hid sch_fq_codel parport_pc ppdev lp parport ip_tables x_tables [Fri Apr 30 01:34:43 2021] autofs4 btrfs xor zstd_compress raid6_pq libcrc32c dm_mirror dm_region_hash dm_log hid_generic usbhid hid nvme nvme_core ahci r8169 libahci atlantic realtek i2c_piix4 wmi [Fri Apr 30 01:34:43 2021] CPU: 27 PID: 5432 Comm: python3 Tainted: P W OEL 5.4.0-72-generic #80~18.04.1-Ubuntu [Fri Apr 30 01:34:43 2021] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./TRX40 Creator, BIOS P1.70 05/29/2020 [Fri Apr 30 01:34:43 2021] RIP: 0010:iommu_unmap_page+0xed/0x100 [Fri Apr 30 01:34:43 2021] Code: ff 75 23 48 8b 7d d0 65 48 33 3c 25 28 00 00 00 4c 89 f8 75 1c 48 83 c4 10 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 0b 49 8d 47 ff <4c> 85 f8 74 d4 0f 0b e8 87 44 9c ff 0f 1f 80 00 00 00 00 0f 1f 44 [Fri Apr 30 01:34:43 2021] RSP: 0018:fffface947c0f930 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [Fri Apr 30 01:34:43 2021] RAX: 0000007fffffffff RBX: 0000000000001000 RCX: 0000000000000027 [Fri Apr 30 01:34:43 2021] RDX: 00008a8000000000 RSI: 0000000000000000 RDI: 0000000000000000 [Fri Apr 30 01:34:43 2021] RBP: fffface947c0f968 R08: fffface947c0f930 R09: 0000000000000004 [Fri Apr 30 01:34:43 2021] R10: 0000000000000000 R11: 000000000000001b R12: ffff9dcbdc371098 [Fri Apr 30 01:34:43 2021] R13: ffff9dcbdc371094 R14: 00008b0000000000 R15: 0000008000000000 [Fri Apr 30 01:34:43 2021] FS: 00007f9e29fff700(0000) GS:ffff9dcbfd4c0000(0000) knlGS:0000000000000000 [Fri Apr 30 01:34:43 2021] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fri Apr 30 01:34:43 2021] CR2: 000000c42038f000 CR3: 0000001ff5844000 CR4: 0000000000340ee0 [Fri Apr 30 01:34:43 2021] Call Trace: [Fri Apr 30 01:34:43 2021] __unmap_single.isra.27+0x66/0x130 [Fri Apr 30 01:34:43 2021] unmap_sg+0x5f/0x70 [Fri Apr 30 01:34:43 2021] nv_unmap_dma_map_scatterlist+0x57/0xa0 [nvidia] [Fri Apr 30 01:34:43 2021] nv_dma_unmap_pages+0x56/0x100 [nvidia] [Fri Apr 30 01:34:43 2021] nv_dma_unmap_alloc+0x16/0x30 [nvidia] [Fri Apr 30 01:34:43 2021] _nv032962rm+0xc1/0x1b0 [nvidia] [Fri Apr 30 01:34:43 2021] ? _nv027787rm+0x9b/0xc0 [nvidia] [Fri Apr 30 01:34:43 2021] ? _nv030989rm+0x124/0x130 [nvidia] [Fri Apr 30 01:34:43 2021] ? rm_gpu_ops_free_duped_handle+0x1c/0x60 [nvidia] [Fri Apr 30 01:34:43 2021] ? nvUvmInterfaceFreeDupedHandle+0x2a/0x40 [nvidia] [Fri Apr 30 01:34:43 2021] ? uvm_ext_gpu_map_free_internal+0x5c/0x70 [nvidia_uvm] [Fri Apr 30 01:34:43 2021] ? uvm_ext_gpu_map_free+0xe/0x20 [nvidia_uvm] [Fri Apr 30 01:34:43 2021] ? uvm_deferred_free_object_list+0x70/0x130 [nvidia_uvm] [Fri Apr 30 01:34:43 2021] ? uvm_va_space_destroy+0x349/0x480 [nvidia_uvm] [Fri Apr 30 01:34:43 2021] ? uvm_release.isra.10+0x7c/0x90 [nvidia_uvm] [Fri Apr 30 01:34:43 2021] ? uvm_release_entry+0x6d/0xa0 [nvidia_uvm] [Fri Apr 30 01:34:43 2021] ? __fput+0xc6/0x260 [Fri Apr 30 01:34:43 2021] ? ____fput+0xe/0x10 [Fri Apr 30 01:34:43 2021] ? task_work_run+0x9d/0xc0 [Fri Apr 30 01:34:43 2021] ? do_exit+0x389/0xb90 [Fri Apr 30 01:34:43 2021] ? do_group_exit+0x43/0xa0 [Fri Apr 30 01:34:43 2021] ? get_signal+0x14f/0x860 [Fri Apr 30 01:34:43 2021] ? do_signal+0x34/0x6d0 [Fri Apr 30 01:34:43 2021] ? __switch_to_asm+0x34/0x70 [Fri Apr 30 01:34:43 2021] ? __x64_sys_futex+0x13f/0x190 [Fri Apr 30 01:34:43 2021] ? exit_to_usermode_loop+0x90/0x130 [Fri Apr 30 01:34:43 2021] ? do_syscall_64+0x170/0x190 [Fri Apr 30 01:34:43 2021] ? entry_SYSCALL_64_after_hwframe+0x44/0xa9 [Fri Apr 30 01:35:15 2021] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [python3:5432] [Fri Apr 30 01:35:15 2021] Modules linked in: nvidia_uvm(OE) veth xt_nat xt_tcpudp xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype iptable_filter iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 bpfilter br_netfilter bridge stp llc aufs cmac overlay bnep snd_hda_codec_hdmi nvidia_drm(POE) nvidia_modeset(POE) edac_mce_amd snd_usb_audio snd_usbmidi_lib mc kvm_amd kvm iwlmvm mac80211 crct10dif_pclmul crc32_pclmul btusb nls_iso8859_1 btrtl nvidia(POE) libarc4 btbcm ghash_clmulni_intel btintel bluetooth iwlwifi ecdh_generic aesni_intel snd_seq_midi input_leds ecc crypto_simd snd_seq_midi_event cryptd glue_helper cfg80211 wmi_bmof snd_hda_intel snd_rawmidi snd_intel_dspcfg joydev snd_hda_codec nouveau snd_hda_core snd_hwdep snd_seq snd_pcm mxm_wmi video snd_seq_device ccp ttm snd_timer drm_kms_helper snd drm i2c_algo_bit fb_sys_fops syscopyarea sysfillrect sysimgblt k10temp soundcore mac_hid sch_fq_codel parport_pc ppdev lp parport ip_tables x_tables [Fri Apr 30 01:35:15 2021] autofs4 btrfs xor zstd_compress raid6_pq libcrc32c dm_mirror dm_region_hash dm_log hid_generic usbhid hid nvme nvme_core ahci r8169 libahci atlantic realtek i2c_piix4 wmi [Fri Apr 30 01:35:15 2021] CPU: 3 PID: 5432 Comm: python3 Tainted: P W OEL 5.4.0-72-generic #80~18.04.1-Ubuntu [Fri Apr 30 01:35:15 2021] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./TRX40 Creator, BIOS P1.70 05/29/2020 [Fri Apr 30 01:35:15 2021] RIP: 0010:iommu_unmap_page+0xe5/0x100 [Fri Apr 30 01:35:15 2021] Code: 4c 8d 34 10 77 91 4d 85 ff 75 23 48 8b 7d d0 65 48 33 3c 25 28 00 00 00 4c 89 f8 75 1c 48 83 c4 10 5b 41 5c 41 5d 41 5e 41 5f <5d> c3 0f 0b 49 8d 47 ff 4c 85 f8 74 d4 0f 0b e8 87 44 9c ff 0f 1f [Fri Apr 30 01:35:15 2021] RSP: 0018:fffface947c0f968 EFLAGS: 00000282 ORIG_RAX: ffffffffffffff13 [Fri Apr 30 01:35:15 2021] RAX: 0000008000000000 RBX: 00008b89b7f07000 RCX: 0000000000000027 [Fri Apr 30 01:35:15 2021] RDX: 00008b8000000000 RSI: 0000000000000000 RDI: 0000000000000000 [Fri Apr 30 01:35:15 2021] RBP: fffface947c0f968 R08: fffface947c0f930 R09: 0000000000000004 [Fri Apr 30 01:35:15 2021] R10: 0000000000000000 R11: 000000000000001b R12: ffff9dcbdc370000 [Fri Apr 30 01:35:15 2021] R13: 00008ffe80000000 R14: 00000000fff80000 R15: 00007fff00000000 [Fri Apr 30 01:35:15 2021] FS: 00007f9e29fff700(0000) GS:ffff9dcbfcec0000(0000) knlGS:0000000000000000 [Fri Apr 30 01:35:15 2021] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fri Apr 30 01:35:15 2021] CR2: 00007f34d2eec8a8 CR3: 000000202bfd0000 CR4: 0000000000340ee0 [Fri Apr 30 01:35:15 2021] Call Trace: [Fri Apr 30 01:35:15 2021] __unmap_single.isra.27+0x66/0x130 [Fri Apr 30 01:35:15 2021] unmap_sg+0x5f/0x70 [Fri Apr 30 01:35:15 2021] nv_unmap_dma_map_scatterlist+0x57/0xa0 [nvidia] [Fri Apr 30 01:35:15 2021] nv_dma_unmap_pages+0x56/0x100 [nvidia] [Fri Apr 30 01:35:15 2021] nv_dma_unmap_alloc+0x16/0x30 [nvidia] [Fri Apr 30 01:35:15 2021] _nv032962rm+0xc1/0x1b0 [nvidia] [Fri Apr 30 01:35:15 2021] ? _nv027787rm+0x9b/0xc0 [nvidia] [Fri Apr 30 01:35:15 2021] ? _nv030989rm+0x124/0x130 [nvidia] [Fri Apr 30 01:35:15 2021] ? rm_gpu_ops_free_duped_handle+0x1c/0x60 [nvidia] [Fri Apr 30 01:35:15 2021] ? nvUvmInterfaceFreeDupedHandle+0x2a/0x40 [nvidia] [Fri Apr 30 01:35:15 2021] ? uvm_ext_gpu_map_free_internal+0x5c/0x70 [nvidia_uvm] [Fri Apr 30 01:35:15 2021] ? uvm_ext_gpu_map_free+0xe/0x20 [nvidia_uvm] [Fri Apr 30 01:35:15 2021] ? uvm_deferred_free_object_list+0x70/0x130 [nvidia_uvm] [Fri Apr 30 01:35:15 2021] ? uvm_va_space_destroy+0x349/0x480 [nvidia_uvm] [Fri Apr 30 01:35:15 2021] ? uvm_release.isra.10+0x7c/0x90 [nvidia_uvm] [Fri Apr 30 01:35:15 2021] ? uvm_release_entry+0x6d/0xa0 [nvidia_uvm] [Fri Apr 30 01:35:15 2021] ? __fput+0xc6/0x260 [Fri Apr 30 01:35:15 2021] ? ____fput+0xe/0x10 [Fri Apr 30 01:35:15 2021] ? task_work_run+0x9d/0xc0 [Fri Apr 30 01:35:15 2021] ? do_exit+0x389/0xb90 [Fri Apr 30 01:35:15 2021] ? do_group_exit+0x43/0xa0 [Fri Apr 30 01:35:15 2021] ? get_signal+0x14f/0x860 [Fri Apr 30 01:35:15 2021] ? do_signal+0x34/0x6d0 [Fri Apr 30 01:35:15 2021] ? __switch_to_asm+0x34/0x70 [Fri Apr 30 01:35:15 2021] ? __x64_sys_futex+0x13f/0x190 [Fri Apr 30 01:35:15 2021] ? exit_to_usermode_loop+0x90/0x130 [Fri Apr 30 01:35:15 2021] ? do_syscall_64+0x170/0x190 [Fri Apr 30 01:35:15 2021] ? entry_SYSCALL_64_after_hwframe+0x44/0xa9 [Fri Apr 30 01:35:51 2021] watchdog: BUG: soft lockup - CPU#27 stuck for 22s! [python3:5432] [Fri Apr 30 01:35:51 2021] Modules linked in: nvidia_uvm(OE) veth xt_nat xt_tcpudp xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype iptable_filter iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 bpfilter br_netfilter bridge stp llc aufs cmac overlay bnep snd_hda_codec_hdmi nvidia_drm(POE) nvidia_modeset(POE) edac_mce_amd snd_usb_audio snd_usbmidi_lib mc kvm_amd kvm iwlmvm mac80211 crct10dif_pclmul crc32_pclmul btusb nls_iso8859_1 btrtl nvidia(POE) libarc4 btbcm ghash_clmulni_intel btintel bluetooth iwlwifi ecdh_generic aesni_intel snd_seq_midi input_leds ecc crypto_simd snd_seq_midi_event cryptd glue_helper cfg80211 wmi_bmof snd_hda_intel snd_rawmidi snd_intel_dspcfg joydev snd_hda_codec nouveau snd_hda_core snd_hwdep snd_seq snd_pcm mxm_wmi video snd_seq_device ccp ttm snd_timer drm_kms_helper snd drm i2c_algo_bit fb_sys_fops syscopyarea sysfillrect sysimgblt k10temp soundcore mac_hid sch_fq_codel parport_pc ppdev lp parport ip_tables x_tables [Fri Apr 30 01:35:51 2021] autofs4 btrfs xor zstd_compress raid6_pq libcrc32c dm_mirror dm_region_hash dm_log hid_generic usbhid hid nvme nvme_core ahci r8169 libahci atlantic realtek i2c_piix4 wmi [Fri Apr 30 01:35:51 2021] CPU: 27 PID: 5432 Comm: python3 Tainted: P W OEL 5.4.0-72-generic #80~18.04.1-Ubuntu [Fri Apr 30 01:35:51 2021] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./TRX40 Creator, BIOS P1.70 05/29/2020 [Fri Apr 30 01:35:51 2021] RIP: 0010:fetch_pte.isra.7+0x0/0x1a0 [Fri Apr 30 01:35:51 2021] Code: 66 cb 98 ff b8 02 00 00 00 5b 41 5c 41 5d 41 5e 41 5f 5d c3 b8 ed ff ff ff eb ee 0f 1f 44 00 00 66 2e 0f 1f 84 00 00 00 00 00 <0f> 1f 44 00 00 48 c7 01 00 00 00 00 44 8b 0f 49 89 c8 41 83 f9 05 [Fri Apr 30 01:35:51 2021] RSP: 0018:fffface947c0f840 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13 [Fri Apr 30 01:35:51 2021] RAX: 0000000000000fff RBX: 0000000000001000 RCX: fffface947c0f848 [Fri Apr 30 01:35:51 2021] RDX: 00008e36362d3000 RSI: ffff9dcbe81fb098 RDI: ffff9dcbe81fb094 [Fri Apr 30 01:35:51 2021] RBP: fffface947c0f880 R08: fffface947c0f848 R09: 0000000000000004 [Fri Apr 30 01:35:51 2021] R10: 0000000000000000 R11: 000000000000001b R12: ffff9dcbe81fb098 [Fri Apr 30 01:35:51 2021] R13: ffff9dcbe81fb094 R14: 00008e36362d3000 R15: 0000000000000000 [Fri Apr 30 01:35:51 2021] FS: 00007f9e29fff700(0000) GS:ffff9dcbfd4c0000(0000) knlGS:0000000000000000 [Fri Apr 30 01:35:51 2021] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fri Apr 30 01:35:51 2021] CR2: 000000c42038f000 CR3: 0000001fe61a0000 CR4: 0000000000340ee0 [Fri Apr 30 01:35:51 2021] Call Trace: [Fri Apr 30 01:35:51 2021] ? iommu_unmap_page+0x64/0x100 [Fri Apr 30 01:35:51 2021] __unmap_single.isra.27+0x66/0x130 [Fri Apr 30 01:35:51 2021] unmap_sg+0x5f/0x70 [Fri Apr 30 01:35:51 2021] nv_unmap_dma_map_scatterlist+0x57/0xa0 [nvidia] [Fri Apr 30 01:35:51 2021] nv_dma_unmap_pages+0x56/0x100 [nvidia] [Fri Apr 30 01:35:51 2021] nv_dma_unmap_alloc+0x16/0x30 [nvidia] [Fri Apr 30 01:35:51 2021] _nv032962rm+0xc1/0x1b0 [nvidia] [Fri Apr 30 01:35:51 2021] ? _nv027787rm+0x9b/0xc0 [nvidia] [Fri Apr 30 01:35:51 2021] ? _nv008192rm+0x29/0x40 [nvidia] [Fri Apr 30 01:35:51 2021] ? _nv028912rm+0x7f/0x2a0 [nvidia] [Fri Apr 30 01:35:51 2021] ? _nv028890rm+0x263/0x900 [nvidia] [Fri Apr 30 01:35:51 2021] ? _nv002524rm+0x9/0x20 [nvidia] [Fri Apr 30 01:35:51 2021] ? _nv003516rm+0x1b/0x80 [nvidia] [Fri Apr 30 01:35:51 2021] ? _nv013017rm+0x6fe/0x770 [nvidia] [Fri Apr 30 01:35:51 2021] ? _nv037997rm+0xb3/0x150 [nvidia] [Fri Apr 30 01:35:51 2021] ? _nv037996rm+0x297/0x4e0 [nvidia] [Fri Apr 30 01:35:51 2021] ? _nv037991rm+0x60/0x70 [nvidia] [Fri Apr 30 01:35:51 2021] ? _nv037992rm+0x7b/0xb0 [nvidia] [Fri Apr 30 01:35:51 2021] ? _nv036285rm+0x40/0xe0 [nvidia] [Fri Apr 30 01:35:51 2021] ? _nv000686rm+0x67/0xa0 [nvidia] [Fri Apr 30 01:35:51 2021] ? rm_cleanup_file_private+0xea/0x140 [nvidia] [Fri Apr 30 01:35:51 2021] ? uvm_kvfree+0x30/0x70 [nvidia_uvm] [Fri Apr 30 01:35:51 2021] ? nvidia_close+0x14e/0x2d0 [nvidia] [Fri Apr 30 01:35:51 2021] ? nvidia_frontend_close+0x2f/0x50 [nvidia] [Fri Apr 30 01:35:51 2021] ? __fput+0xc6/0x260 [Fri Apr 30 01:35:51 2021] ? ____fput+0xe/0x10 [Fri Apr 30 01:35:51 2021] ? task_work_run+0x9d/0xc0 [Fri Apr 30 01:35:51 2021] ? do_exit+0x389/0xb90 [Fri Apr 30 01:35:51 2021] ? do_group_exit+0x43/0xa0 [Fri Apr 30 01:35:51 2021] ? get_signal+0x14f/0x860 [Fri Apr 30 01:35:51 2021] ? do_signal+0x34/0x6d0 [Fri Apr 30 01:35:51 2021] ? __switch_to_asm+0x34/0x70 [Fri Apr 30 01:35:51 2021] ? __x64_sys_futex+0x13f/0x190 [Fri Apr 30 01:35:51 2021] ? exit_to_usermode_loop+0x90/0x130 [Fri Apr 30 01:35:51 2021] ? do_syscall_64+0x170/0x190 [Fri Apr 30 01:35:51 2021] ? entry_SYSCALL_64_after_hwframe+0x44/0xa9 [Fri Apr 30 01:36:19 2021] watchdog: BUG: soft lockup - CPU#27 stuck for 22s! [python3:5432] [Fri Apr 30 01:36:19 2021] Modules linked in: nvidia_uvm(OE) veth xt_nat xt_tcpudp xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype iptable_filter iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 bpfilter br_netfilter bridge stp llc aufs cmac overlay bnep snd_hda_codec_hdmi nvidia_drm(POE) nvidia_modeset(POE) edac_mce_amd snd_usb_audio snd_usbmidi_lib mc kvm_amd kvm iwlmvm mac80211 crct10dif_pclmul crc32_pclmul btusb nls_iso8859_1 btrtl nvidia(POE) libarc4 btbcm ghash_clmulni_intel btintel bluetooth iwlwifi ecdh_generic aesni_intel snd_seq_midi input_leds ecc crypto_simd snd_seq_midi_event cryptd glue_helper cfg80211 wmi_bmof snd_hda_intel snd_rawmidi snd_intel_dspcfg joydev snd_hda_codec nouveau snd_hda_core snd_hwdep snd_seq snd_pcm mxm_wmi video snd_seq_device ccp ttm snd_timer drm_kms_helper snd drm i2c_algo_bit fb_sys_fops syscopyarea sysfillrect sysimgblt k10temp soundcore mac_hid sch_fq_codel parport_pc ppdev lp parport ip_tables x_tables [Fri Apr 30 01:36:19 2021] autofs4 btrfs xor zstd_compress raid6_pq libcrc32c dm_mirror dm_region_hash dm_log hid_generic usbhid hid nvme nvme_core ahci r8169 libahci atlantic realtek i2c_piix4 wmi [Fri Apr 30 01:36:19 2021] CPU: 27 PID: 5432 Comm: python3 Tainted: P W OEL 5.4.0-72-generic #80~18.04.1-Ubuntu [Fri Apr 30 01:36:19 2021] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./TRX40 Creator, BIOS P1.70 05/29/2020 [Fri Apr 30 01:36:19 2021] RIP: 0010:iommu_unmap_page+0xf0/0x100 [Fri Apr 30 01:36:19 2021] Code: 48 8b 7d d0 65 48 33 3c 25 28 00 00 00 4c 89 f8 75 1c 48 83 c4 10 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 0b 49 8d 47 ff 4c 85 f8 <74> d4 0f 0b e8 87 44 9c ff 0f 1f 80 00 00 00 00 0f 1f 44 00 00 8b [Fri Apr 30 01:36:19 2021] RSP: 0018:fffface947c0f848 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13 [Fri Apr 30 01:36:19 2021] RAX: 0000007fffffffff RBX: 0000000000001000 RCX: 0000000000000027 [Fri Apr 30 01:36:19 2021] RDX: 00008c8000000000 RSI: 0000000000000000 RDI: 0000000000000000 [Fri Apr 30 01:36:19 2021] RBP: fffface947c0f880 R08: fffface947c0f848 R09: 0000000000000004 [Fri Apr 30 01:36:19 2021] R10: 0000000000000000 R11: 000000000000001b R12: ffff9dcbe81fb098 [Fri Apr 30 01:36:19 2021] R13: ffff9dcbe81fb094 R14: 00008d0000000000 R15: 0000008000000000 [Fri Apr 30 01:36:19 2021] FS: 00007f9e29fff700(0000) GS:ffff9dcbfd4c0000(0000) knlGS:0000000000000000 [Fri Apr 30 01:36:19 2021] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fri Apr 30 01:36:19 2021] CR2: 000000c42038f000 CR3: 0000001fe61a0000 CR4: 0000000000340ee0 [Fri Apr 30 01:36:19 2021] Call Trace: [Fri Apr 30 01:36:19 2021] __unmap_single.isra.27+0x66/0x130 [Fri Apr 30 01:36:19 2021] unmap_sg+0x5f/0x70 [Fri Apr 30 01:36:19 2021] nv_unmap_dma_map_scatterlist+0x57/0xa0 [nvidia] [Fri Apr 30 01:36:19 2021] nv_dma_unmap_pages+0x56/0x100 [nvidia] [Fri Apr 30 01:36:19 2021] nv_dma_unmap_alloc+0x16/0x30 [nvidia] [Fri Apr 30 01:36:19 2021] _nv032962rm+0xc1/0x1b0 [nvidia] [Fri Apr 30 01:36:19 2021] ? _nv027787rm+0x9b/0xc0 [nvidia] [Fri Apr 30 01:36:19 2021] ? _nv008192rm+0x29/0x40 [nvidia] [Fri Apr 30 01:36:19 2021] ? _nv028912rm+0x7f/0x2a0 [nvidia] [Fri Apr 30 01:36:19 2021] ? _nv028890rm+0x263/0x900 [nvidia] [Fri Apr 30 01:36:19 2021] ? _nv002524rm+0x9/0x20 [nvidia] [Fri Apr 30 01:36:19 2021] ? _nv003516rm+0x1b/0x80 [nvidia] [Fri Apr 30 01:36:19 2021] ? _nv013017rm+0x6fe/0x770 [nvidia] [Fri Apr 30 01:36:19 2021] ? _nv037997rm+0xb3/0x150 [nvidia] [Fri Apr 30 01:36:19 2021] ? _nv037996rm+0x297/0x4e0 [nvidia] [Fri Apr 30 01:36:19 2021] ? _nv037991rm+0x60/0x70 [nvidia] [Fri Apr 30 01:36:19 2021] ? _nv037992rm+0x7b/0xb0 [nvidia] [Fri Apr 30 01:36:19 2021] ? _nv036285rm+0x40/0xe0 [nvidia] [Fri Apr 30 01:36:19 2021] ? _nv000686rm+0x67/0xa0 [nvidia] [Fri Apr 30 01:36:19 2021] ? rm_cleanup_file_private+0xea/0x140 [nvidia] [Fri Apr 30 01:36:19 2021] ? uvm_kvfree+0x30/0x70 [nvidia_uvm] [Fri Apr 30 01:36:19 2021] ? nvidia_close+0x14e/0x2d0 [nvidia] [Fri Apr 30 01:36:19 2021] ? nvidia_frontend_close+0x2f/0x50 [nvidia] [Fri Apr 30 01:36:19 2021] ? __fput+0xc6/0x260 [Fri Apr 30 01:36:19 2021] ? ____fput+0xe/0x10 [Fri Apr 30 01:36:19 2021] ? task_work_run+0x9d/0xc0 [Fri Apr 30 01:36:19 2021] ? do_exit+0x389/0xb90 [Fri Apr 30 01:36:19 2021] ? do_group_exit+0x43/0xa0 [Fri Apr 30 01:36:19 2021] ? get_signal+0x14f/0x860 [Fri Apr 30 01:36:19 2021] ? do_signal+0x34/0x6d0 [Fri Apr 30 01:36:19 2021] ? __switch_to_asm+0x34/0x70 [Fri Apr 30 01:36:19 2021] ? __x64_sys_futex+0x13f/0x190 [Fri Apr 30 01:36:19 2021] ? exit_to_usermode_loop+0x90/0x130 [Fri Apr 30 01:36:19 2021] ? do_syscall_64+0x170/0x190 [Fri Apr 30 01:36:19 2021] ? entry_SYSCALL_64_after_hwframe+0x44/0xa9 [Fri Apr 30 01:36:24 2021] rcu: INFO: rcu_sched self-detected stall on CPU [Fri Apr 30 01:36:24 2021] rcu: 27-....: (14999 ticks this GP) idle=eaa/1/0x4000000000000002 softirq=6179/6179 fqs=7499 [Fri Apr 30 01:36:24 2021] (t=15000 jiffies g=47517 q=93395) [Fri Apr 30 01:36:24 2021] NMI backtrace for cpu 27 [Fri Apr 30 01:36:24 2021] CPU: 27 PID: 5432 Comm: python3 Tainted: P W OEL 5.4.0-72-generic #80~18.04.1-Ubuntu [Fri Apr 30 01:36:24 2021] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./TRX40 Creator, BIOS P1.70 05/29/2020 [Fri Apr 30 01:36:24 2021] Call Trace: [Fri Apr 30 01:36:24 2021] [Fri Apr 30 01:36:24 2021] dump_stack+0x6d/0x8b [Fri Apr 30 01:36:24 2021] nmi_cpu_backtrace+0x94/0xa0 [Fri Apr 30 01:36:24 2021] ? lapic_can_unplug_cpu+0xb0/0xb0 [Fri Apr 30 01:36:24 2021] nmi_trigger_cpumask_backtrace+0xe7/0x130 [Fri Apr 30 01:36:24 2021] arch_trigger_cpumask_backtrace+0x19/0x20 [Fri Apr 30 01:36:24 2021] rcu_dump_cpu_stacks+0x9e/0xce [Fri Apr 30 01:36:24 2021] rcu_sched_clock_irq+0x508/0x770 [Fri Apr 30 01:36:24 2021] ? tick_sched_do_timer+0x60/0x60 [Fri Apr 30 01:36:24 2021] update_process_times+0x28/0x50 [Fri Apr 30 01:36:24 2021] tick_sched_handle+0x25/0x70 [Fri Apr 30 01:36:24 2021] ? tick_sched_do_timer+0x58/0x60 [Fri Apr 30 01:36:24 2021] tick_sched_timer+0x3c/0x80 [Fri Apr 30 01:36:24 2021] __hrtimer_run_queues+0x10f/0x280 [Fri Apr 30 01:36:24 2021] hrtimer_interrupt+0xe7/0x240 [Fri Apr 30 01:36:24 2021] smp_apic_timer_interrupt+0x6f/0x130 [Fri Apr 30 01:36:24 2021] apic_timer_interrupt+0xf/0x20 [Fri Apr 30 01:36:24 2021] [Fri Apr 30 01:36:24 2021] RIP: 0010:fetch_pte.isra.7+0x66/0x1a0 [Fri Apr 30 01:36:24 2021] Code: 1c c0 48 89 e5 53 48 8b 36 41 8d 4b 0c 48 d3 ef 81 e7 ff 01 00 00 4c 8d 14 fe be 01 00 00 00 48 d3 e6 85 c0 49 89 30 49 8b 3a <0f> 8e b9 00 00 00 40 f6 c7 01 0f 84 1f 01 00 00 48 89 fe 48 c1 ee [Fri Apr 30 01:36:24 2021] RSP: 0018:fffface947c0f830 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [Fri Apr 30 01:36:24 2021] RAX: 0000000000000003 RBX: 0000000000001000 RCX: 0000000000000027 [Fri Apr 30 01:36:24 2021] RDX: 00008f69de517000 RSI: 0000008000000000 RDI: 0000000000000000 [Fri Apr 30 01:36:24 2021] RBP: fffface947c0f838 R08: fffface947c0f848 R09: 0000000000000004 [Fri Apr 30 01:36:24 2021] R10: ffff9dc8ebc988f0 R11: 000000000000001b R12: ffff9dcbe81fb098 [Fri Apr 30 01:36:24 2021] R13: ffff9dcbe81fb094 R14: 00008f69de517000 R15: 0000000000000000 [Fri Apr 30 01:36:24 2021] iommu_unmap_page+0x64/0x100 [Fri Apr 30 01:36:24 2021] __unmap_single.isra.27+0x66/0x130 [Fri Apr 30 01:36:24 2021] unmap_sg+0x5f/0x70 [Fri Apr 30 01:36:24 2021] nv_unmap_dma_map_scatterlist+0x57/0xa0 [nvidia] [Fri Apr 30 01:36:24 2021] nv_dma_unmap_pages+0x56/0x100 [nvidia] [Fri Apr 30 01:36:24 2021] nv_dma_unmap_alloc+0x16/0x30 [nvidia] [Fri Apr 30 01:36:24 2021] _nv032962rm+0xc1/0x1b0 [nvidia] [Fri Apr 30 01:36:24 2021] ? _nv027787rm+0x9b/0xc0 [nvidia] [Fri Apr 30 01:36:24 2021] ? _nv008192rm+0x29/0x40 [nvidia] [Fri Apr 30 01:36:24 2021] ? _nv028912rm+0x7f/0x2a0 [nvidia] [Fri Apr 30 01:36:24 2021] ? _nv028890rm+0x263/0x900 [nvidia] [Fri Apr 30 01:36:24 2021] ? _nv002524rm+0x9/0x20 [nvidia] [Fri Apr 30 01:36:24 2021] ? _nv003516rm+0x1b/0x80 [nvidia] [Fri Apr 30 01:36:24 2021] ? _nv013017rm+0x6fe/0x770 [nvidia] [Fri Apr 30 01:36:24 2021] ? _nv037997rm+0xb3/0x150 [nvidia] [Fri Apr 30 01:36:24 2021] ? _nv037996rm+0x297/0x4e0 [nvidia] [Fri Apr 30 01:36:24 2021] ? _nv037991rm+0x60/0x70 [nvidia] [Fri Apr 30 01:36:24 2021] ? _nv037992rm+0x7b/0xb0 [nvidia] [Fri Apr 30 01:36:24 2021] ? _nv036285rm+0x40/0xe0 [nvidia] [Fri Apr 30 01:36:24 2021] ? _nv000686rm+0x67/0xa0 [nvidia] [Fri Apr 30 01:36:24 2021] ? rm_cleanup_file_private+0xea/0x140 [nvidia] [Fri Apr 30 01:36:24 2021] ? uvm_kvfree+0x30/0x70 [nvidia_uvm] [Fri Apr 30 01:36:24 2021] ? nvidia_close+0x14e/0x2d0 [nvidia] [Fri Apr 30 01:36:24 2021] ? nvidia_frontend_close+0x2f/0x50 [nvidia] [Fri Apr 30 01:36:24 2021] ? __fput+0xc6/0x260 [Fri Apr 30 01:36:24 2021] ? ____fput+0xe/0x10 [Fri Apr 30 01:36:24 2021] ? task_work_run+0x9d/0xc0 [Fri Apr 30 01:36:24 2021] ? do_exit+0x389/0xb90 [Fri Apr 30 01:36:24 2021] ? do_group_exit+0x43/0xa0 [Fri Apr 30 01:36:24 2021] ? get_signal+0x14f/0x860 [Fri Apr 30 01:36:24 2021] ? do_signal+0x34/0x6d0 [Fri Apr 30 01:36:24 2021] ? __switch_to_asm+0x34/0x70 [Fri Apr 30 01:36:24 2021] ? __x64_sys_futex+0x13f/0x190 [Fri Apr 30 01:36:24 2021] ? exit_to_usermode_loop+0x90/0x130 [Fri Apr 30 01:36:24 2021] ? do_syscall_64+0x170/0x190 [Fri Apr 30 01:36:24 2021] ? entry_SYSCALL_64_after_hwframe+0x44/0xa9 [Fri Apr 30 01:36:51 2021] watchdog: BUG: soft lockup - CPU#27 stuck for 22s! [python3:5432] [Fri Apr 30 01:36:51 2021] Modules linked in: nvidia_uvm(OE) veth xt_nat xt_tcpudp xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype iptable_filter iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 bpfilter br_netfilter bridge stp llc aufs cmac overlay bnep snd_hda_codec_hdmi nvidia_drm(POE) nvidia_modeset(POE) edac_mce_amd snd_usb_audio snd_usbmidi_lib mc kvm_amd kvm iwlmvm mac80211 crct10dif_pclmul crc32_pclmul btusb nls_iso8859_1 btrtl nvidia(POE) libarc4 btbcm ghash_clmulni_intel btintel bluetooth iwlwifi ecdh_generic aesni_intel snd_seq_midi input_leds ecc crypto_simd snd_seq_midi_event cryptd glue_helper cfg80211 wmi_bmof snd_hda_intel snd_rawmidi snd_intel_dspcfg joydev snd_hda_codec nouveau snd_hda_core snd_hwdep snd_seq snd_pcm mxm_wmi video snd_seq_device ccp ttm snd_timer drm_kms_helper snd drm i2c_algo_bit fb_sys_fops syscopyarea sysfillrect sysimgblt k10temp soundcore mac_hid sch_fq_codel parport_pc ppdev lp parport ip_tables x_tables [Fri Apr 30 01:36:51 2021] autofs4 btrfs xor zstd_compress raid6_pq libcrc32c dm_mirror dm_region_hash dm_log hid_generic usbhid hid nvme nvme_core ahci r8169 libahci atlantic realtek i2c_piix4 wmi [Fri Apr 30 01:36:51 2021] CPU: 27 PID: 5432 Comm: python3 Tainted: P W OEL 5.4.0-72-generic #80~18.04.1-Ubuntu [Fri Apr 30 01:36:51 2021] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./TRX40 Creator, BIOS P1.70 05/29/2020 [Fri Apr 30 01:36:51 2021] RIP: 0010:iommu_unmap_page+0x56/0x100 [Fri Apr 30 01:36:51 2021] Code: 00 00 48 8d 42 ff 48 89 d3 48 85 d0 0f 85 a9 00 00 00 4c 8d af 94 00 00 00 4c 8d a7 98 00 00 00 49 89 f6 45 31 ff 48 8d 4d c8 <4c> 89 e6 4c 89 f2 4c 89 ef e8 fc fc ff ff 48 85 c0 48 89 c6 74 3c [Fri Apr 30 01:36:51 2021] RSP: 0018:fffface947c0f848 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13 [Fri Apr 30 01:36:51 2021] RAX: 0000000000000fff RBX: 0000000000001000 RCX: fffface947c0f848 [Fri Apr 30 01:36:51 2021] RDX: 0000000000001000 RSI: 00008dab8173e000 RDI: ffff9dcbe81fb000 [Fri Apr 30 01:36:51 2021] RBP: fffface947c0f880 R08: fffface947c0f848 R09: 0000000000000004 [Fri Apr 30 01:36:51 2021] R10: 0000000000000000 R11: 000000000000001b R12: ffff9dcbe81fb098 [Fri Apr 30 01:36:51 2021] R13: ffff9dcbe81fb094 R14: 00008dab8173e000 R15: 0000000000000000 [Fri Apr 30 01:36:51 2021] FS: 00007f9e29fff700(0000) GS:ffff9dcbfd4c0000(0000) knlGS:0000000000000000 [Fri Apr 30 01:36:51 2021] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fri Apr 30 01:36:51 2021] CR2: 000000c42038f000 CR3: 0000001fe61a0000 CR4: 0000000000340ee0 [Fri Apr 30 01:36:51 2021] Call Trace: [Fri Apr 30 01:36:51 2021] __unmap_single.isra.27+0x66/0x130 [Fri Apr 30 01:36:51 2021] unmap_sg+0x5f/0x70 [Fri Apr 30 01:36:51 2021] nv_unmap_dma_map_scatterlist+0x57/0xa0 [nvidia] [Fri Apr 30 01:36:51 2021] nv_dma_unmap_pages+0x56/0x100 [nvidia] [Fri Apr 30 01:36:51 2021] nv_dma_unmap_alloc+0x16/0x30 [nvidia] [Fri Apr 30 01:36:51 2021] _nv032962rm+0xc1/0x1b0 [nvidia] [Fri Apr 30 01:36:51 2021] ? _nv027787rm+0x9b/0xc0 [nvidia] [Fri Apr 30 01:36:51 2021] ? _nv008192rm+0x29/0x40 [nvidia] [Fri Apr 30 01:36:51 2021] ? _nv028912rm+0x7f/0x2a0 [nvidia] [Fri Apr 30 01:36:51 2021] ? _nv028890rm+0x263/0x900 [nvidia] [Fri Apr 30 01:36:51 2021] ? _nv002524rm+0x9/0x20 [nvidia] [Fri Apr 30 01:36:51 2021] ? _nv003516rm+0x1b/0x80 [nvidia] [Fri Apr 30 01:36:51 2021] ? _nv013017rm+0x6fe/0x770 [nvidia] [Fri Apr 30 01:36:51 2021] ? _nv037997rm+0xb3/0x150 [nvidia] [Fri Apr 30 01:36:51 2021] ? _nv037996rm+0x297/0x4e0 [nvidia] [Fri Apr 30 01:36:51 2021] ? _nv037991rm+0x60/0x70 [nvidia] [Fri Apr 30 01:36:51 2021] ? _nv037992rm+0x7b/0xb0 [nvidia] [Fri Apr 30 01:36:51 2021] ? _nv036285rm+0x40/0xe0 [nvidia] [Fri Apr 30 01:36:51 2021] ? _nv000686rm+0x67/0xa0 [nvidia] [Fri Apr 30 01:36:51 2021] ? rm_cleanup_file_private+0xea/0x140 [nvidia] [Fri Apr 30 01:36:51 2021] ? uvm_kvfree+0x30/0x70 [nvidia_uvm] [Fri Apr 30 01:36:51 2021] ? nvidia_close+0x14e/0x2d0 [nvidia] [Fri Apr 30 01:36:51 2021] ? nvidia_frontend_close+0x2f/0x50 [nvidia] [Fri Apr 30 01:36:51 2021] ? __fput+0xc6/0x260 [Fri Apr 30 01:36:51 2021] ? ____fput+0xe/0x10 [Fri Apr 30 01:36:51 2021] ? task_work_run+0x9d/0xc0 [Fri Apr 30 01:36:51 2021] ? do_exit+0x389/0xb90 [Fri Apr 30 01:36:51 2021] ? do_group_exit+0x43/0xa0 [Fri Apr 30 01:36:51 2021] ? get_signal+0x14f/0x860 [Fri Apr 30 01:36:51 2021] ? do_signal+0x34/0x6d0 [Fri Apr 30 01:36:51 2021] ? __switch_to_asm+0x34/0x70 [Fri Apr 30 01:36:51 2021] ? __x64_sys_futex+0x13f/0x190 [Fri Apr 30 01:36:51 2021] ? exit_to_usermode_loop+0x90/0x130 [Fri Apr 30 01:36:51 2021] ? do_syscall_64+0x170/0x190 [Fri Apr 30 01:36:51 2021] ? entry_SYSCALL_64_after_hwframe+0x44/0xa9 [Fri Apr 30 01:37:19 2021] watchdog: BUG: soft lockup - CPU#27 stuck for 22s! [python3:5432] [Fri Apr 30 01:37:19 2021] Modules linked in: nvidia_uvm(OE) veth xt_nat xt_tcpudp xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype iptable_filter iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 bpfilter br_netfilter bridge stp llc aufs cmac overlay bnep snd_hda_codec_hdmi nvidia_drm(POE) nvidia_modeset(POE) edac_mce_amd snd_usb_audio snd_usbmidi_lib mc kvm_amd kvm iwlmvm mac80211 crct10dif_pclmul crc32_pclmul btusb nls_iso8859_1 btrtl nvidia(POE) libarc4 btbcm ghash_clmulni_intel btintel bluetooth iwlwifi ecdh_generic aesni_intel snd_seq_midi input_leds ecc crypto_simd snd_seq_midi_event cryptd glue_helper cfg80211 wmi_bmof snd_hda_intel snd_rawmidi snd_intel_dspcfg joydev snd_hda_codec nouveau snd_hda_core snd_hwdep snd_seq snd_pcm mxm_wmi video snd_seq_device ccp ttm snd_timer drm_kms_helper snd drm i2c_algo_bit fb_sys_fops syscopyarea sysfillrect sysimgblt k10temp soundcore mac_hid sch_fq_codel parport_pc ppdev lp parport ip_tables x_tables [Fri Apr 30 01:37:19 2021] autofs4 btrfs xor zstd_compress raid6_pq libcrc32c dm_mirror dm_region_hash dm_log hid_generic usbhid hid nvme nvme_core ahci r8169 libahci atlantic realtek i2c_piix4 wmi [Fri Apr 30 01:37:19 2021] CPU: 27 PID: 5432 Comm: python3 Tainted: P W OEL 5.4.0-72-generic #80~18.04.1-Ubuntu [Fri Apr 30 01:37:19 2021] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./TRX40 Creator, BIOS P1.70 05/29/2020 [Fri Apr 30 01:37:19 2021] RIP: 0010:iommu_unmap_page+0xac/0x100 [Fri Apr 30 01:37:19 2021] Code: e1 48 c1 ea 03 48 8d 04 d2 ba 01 00 00 00 48 29 c1 31 c0 48 d3 e2 48 c7 04 c6 00 00 00 00 48 83 c0 01 39 c2 7f f0 48 8b 45 c8 <48> 89 c2 49 01 c7 48 f7 da 4c 21 f2 4c 39 fb 4c 8d 34 10 77 91 4d [Fri Apr 30 01:37:19 2021] RSP: 0018:fffface947c0f848 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13 [Fri Apr 30 01:37:19 2021] RAX: 0000008000000000 RBX: 0000000000001000 RCX: 0000000000000027 [Fri Apr 30 01:37:19 2021] RDX: 00008c596e967000 RSI: 0000000000000000 RDI: 0000000000000000 [Fri Apr 30 01:37:19 2021] RBP: fffface947c0f880 R08: fffface947c0f848 R09: 0000000000000004 [Fri Apr 30 01:37:19 2021] R10: 0000000000000000 R11: 000000000000001b R12: ffff9dcbe81fb098 [Fri Apr 30 01:37:19 2021] R13: ffff9dcbe81fb094 R14: 00008c596e967000 R15: 0000000000000000 [Fri Apr 30 01:37:19 2021] FS: 00007f9e29fff700(0000) GS:ffff9dcbfd4c0000(0000) knlGS:0000000000000000 [Fri Apr 30 01:37:19 2021] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fri Apr 30 01:37:19 2021] CR2: 000000c42038f000 CR3: 0000001fe61a0000 CR4: 0000000000340ee0 [Fri Apr 30 01:37:19 2021] Call Trace: [Fri Apr 30 01:37:19 2021] __unmap_single.isra.27+0x66/0x130 [Fri Apr 30 01:37:19 2021] unmap_sg+0x5f/0x70 [Fri Apr 30 01:37:19 2021] nv_unmap_dma_map_scatterlist+0x57/0xa0 [nvidia] [Fri Apr 30 01:37:19 2021] nv_dma_unmap_pages+0x56/0x100 [nvidia] [Fri Apr 30 01:37:19 2021] nv_dma_unmap_alloc+0x16/0x30 [nvidia] [Fri Apr 30 01:37:19 2021] _nv032962rm+0xc1/0x1b0 [nvidia] [Fri Apr 30 01:37:19 2021] ? _nv027787rm+0x9b/0xc0 [nvidia] [Fri Apr 30 01:37:19 2021] ? _nv008192rm+0x29/0x40 [nvidia] [Fri Apr 30 01:37:19 2021] ? _nv028912rm+0x7f/0x2a0 [nvidia] [Fri Apr 30 01:37:19 2021] ? _nv028890rm+0x263/0x900 [nvidia] [Fri Apr 30 01:37:19 2021] ? _nv002524rm+0x9/0x20 [nvidia] [Fri Apr 30 01:37:19 2021] ? _nv003516rm+0x1b/0x80 [nvidia] [Fri Apr 30 01:37:19 2021] ? _nv013017rm+0x6fe/0x770 [nvidia] [Fri Apr 30 01:37:19 2021] ? _nv037997rm+0xb3/0x150 [nvidia] [Fri Apr 30 01:37:19 2021] ? _nv037996rm+0x297/0x4e0 [nvidia] [Fri Apr 30 01:37:19 2021] ? _nv037991rm+0x60/0x70 [nvidia] [Fri Apr 30 01:37:19 2021] ? _nv037992rm+0x7b/0xb0 [nvidia] [Fri Apr 30 01:37:19 2021] ? _nv036285rm+0x40/0xe0 [nvidia] [Fri Apr 30 01:37:19 2021] ? _nv000686rm+0x67/0xa0 [nvidia] [Fri Apr 30 01:37:19 2021] ? rm_cleanup_file_private+0xea/0x140 [nvidia] [Fri Apr 30 01:37:19 2021] ? uvm_kvfree+0x30/0x70 [nvidia_uvm] [Fri Apr 30 01:37:19 2021] ? nvidia_close+0x14e/0x2d0 [nvidia] [Fri Apr 30 01:37:19 2021] ? nvidia_frontend_close+0x2f/0x50 [nvidia] [Fri Apr 30 01:37:19 2021] ? __fput+0xc6/0x260 [Fri Apr 30 01:37:19 2021] ? ____fput+0xe/0x10 [Fri Apr 30 01:37:19 2021] ? task_work_run+0x9d/0xc0 [Fri Apr 30 01:37:19 2021] ? do_exit+0x389/0xb90 [Fri Apr 30 01:37:19 2021] ? do_group_exit+0x43/0xa0 [Fri Apr 30 01:37:19 2021] ? get_signal+0x14f/0x860 [Fri Apr 30 01:37:19 2021] ? do_signal+0x34/0x6d0 [Fri Apr 30 01:37:19 2021] ? __switch_to_asm+0x34/0x70 [Fri Apr 30 01:37:19 2021] ? __x64_sys_futex+0x13f/0x190 [Fri Apr 30 01:37:19 2021] ? exit_to_usermode_loop+0x90/0x130 [Fri Apr 30 01:37:19 2021] ? do_syscall_64+0x170/0x190 [Fri Apr 30 01:37:19 2021] ? entry_SYSCALL_64_after_hwframe+0x44/0xa9 [Fri Apr 30 01:37:29 2021] INFO: task nv_queue:1674 blocked for more than 120 seconds. [Fri Apr 30 01:37:29 2021] Tainted: P W OEL 5.4.0-72-generic #80~18.04.1-Ubuntu [Fri Apr 30 01:37:29 2021] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [Fri Apr 30 01:37:29 2021] nv_queue D 0 1674 2 0x80004000 [Fri Apr 30 01:37:29 2021] Call Trace: [Fri Apr 30 01:37:29 2021] __schedule+0x292/0x710 [Fri Apr 30 01:37:29 2021] ? update_load_avg+0x17d/0x680 [Fri Apr 30 01:37:29 2021] schedule+0x33/0xa0 [Fri Apr 30 01:37:29 2021] schedule_timeout+0x1d3/0x320 [Fri Apr 30 01:37:29 2021] ? __switch_to_asm+0x34/0x70 [Fri Apr 30 01:37:29 2021] ? __switch_to_asm+0x40/0x70 [Fri Apr 30 01:37:29 2021] ? __switch_to_asm+0x34/0x70 [Fri Apr 30 01:37:29 2021] ? __switch_to_asm+0x40/0x70 [Fri Apr 30 01:37:29 2021] ? __switch_to_asm+0x34/0x70 [Fri Apr 30 01:37:29 2021] ? __switch_to_asm+0x40/0x70 [Fri Apr 30 01:37:29 2021] ? __switch_to_asm+0x34/0x70 [Fri Apr 30 01:37:29 2021] __down+0x91/0xe0 [Fri Apr 30 01:37:29 2021] ? x86_gsbase_write_task+0x20/0x30 [Fri Apr 30 01:37:29 2021] ? __switch_to_asm+0x40/0x70 [Fri Apr 30 01:37:29 2021] down+0x41/0x50 [Fri Apr 30 01:37:29 2021] ? down+0x41/0x50 [Fri Apr 30 01:37:29 2021] os_acquire_mutex+0x34/0x40 [nvidia] [Fri Apr 30 01:37:29 2021] os_acquire_semaphore+0xe/0x10 [nvidia] [Fri Apr 30 01:37:29 2021] _nv035446rm+0xc/0x30 [nvidia] [Fri Apr 30 01:37:29 2021] ? _nv035438rm+0xc/0x20 [nvidia] [Fri Apr 30 01:37:29 2021] ? _nv036298rm+0x155/0x1a0 [nvidia] [Fri Apr 30 01:37:29 2021] ? _nv041724rm+0x60/0x270 [nvidia] [Fri Apr 30 01:37:29 2021] ? rm_execute_work_item+0x56/0x120 [nvidia] [Fri Apr 30 01:37:29 2021] ? os_execute_work_item+0x4a/0x60 [nvidia] [Fri Apr 30 01:37:29 2021] ? _main_loop+0x76/0x140 [nvidia] [Fri Apr 30 01:37:29 2021] ? kthread+0x121/0x140 [Fri Apr 30 01:37:29 2021] ? _raw_q_schedule+0x80/0x80 [nvidia] [Fri Apr 30 01:37:29 2021] ? kthread_park+0x90/0x90 [Fri Apr 30 01:37:29 2021] ? ret_from_fork+0x22/0x40 [Fri Apr 30 01:37:29 2021] INFO: task nv_queue:1773 blocked for more than 120 seconds. [Fri Apr 30 01:37:29 2021] Tainted: P W OEL 5.4.0-72-generic #80~18.04.1-Ubuntu [Fri Apr 30 01:37:29 2021] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [Fri Apr 30 01:37:29 2021] nv_queue D 0 1773 2 0x80004000 [Fri Apr 30 01:37:29 2021] Call Trace: [Fri Apr 30 01:37:29 2021] __schedule+0x292/0x710 [Fri Apr 30 01:37:29 2021] ? update_load_avg+0x17d/0x680 [Fri Apr 30 01:37:29 2021] schedule+0x33/0xa0 [Fri Apr 30 01:37:29 2021] schedule_timeout+0x1d3/0x320 [Fri Apr 30 01:37:29 2021] ? __switch_to_asm+0x34/0x70 [Fri Apr 30 01:37:29 2021] ? __switch_to_asm+0x40/0x70 [Fri Apr 30 01:37:29 2021] ? __switch_to_asm+0x34/0x70 [Fri Apr 30 01:37:29 2021] ? __switch_to_asm+0x40/0x70 [Fri Apr 30 01:37:29 2021] ? __switch_to_asm+0x34/0x70 [Fri Apr 30 01:37:29 2021] ? __switch_to_asm+0x40/0x70 [Fri Apr 30 01:37:29 2021] ? __switch_to_asm+0x34/0x70 [Fri Apr 30 01:37:29 2021] __down+0x91/0xe0 [Fri Apr 30 01:37:29 2021] ? x86_gsbase_write_task+0x20/0x30 [Fri Apr 30 01:37:29 2021] ? __switch_to_asm+0x40/0x70 [Fri Apr 30 01:37:29 2021] down+0x41/0x50 [Fri Apr 30 01:37:29 2021] ? down+0x41/0x50 [Fri Apr 30 01:37:29 2021] os_acquire_mutex+0x34/0x40 [nvidia] [Fri Apr 30 01:37:29 2021] os_acquire_semaphore+0xe/0x10 [nvidia] [Fri Apr 30 01:37:29 2021] _nv035446rm+0xc/0x30 [nvidia] [Fri Apr 30 01:37:29 2021] ? _nv035438rm+0xc/0x20 [nvidia] [Fri Apr 30 01:37:29 2021] ? _nv036298rm+0x155/0x1a0 [nvidia] [Fri Apr 30 01:37:29 2021] ? _nv041724rm+0x60/0x270 [nvidia] [Fri Apr 30 01:37:29 2021] ? rm_execute_work_item+0x56/0x120 [nvidia] [Fri Apr 30 01:37:29 2021] ? os_execute_work_item+0x4a/0x60 [nvidia] [Fri Apr 30 01:37:29 2021] ? _main_loop+0x76/0x140 [nvidia] [Fri Apr 30 01:37:29 2021] ? kthread+0x121/0x140 [Fri Apr 30 01:37:29 2021] ? _raw_q_schedule+0x80/0x80 [nvidia] [Fri Apr 30 01:37:29 2021] ? kthread_park+0x90/0x90 [Fri Apr 30 01:37:29 2021] ? ret_from_fork+0x22/0x40 [Fri Apr 30 01:37:29 2021] INFO: task nv_queue:1836 blocked for more than 120 seconds. [Fri Apr 30 01:37:29 2021] Tainted: P W OEL 5.4.0-72-generic #80~18.04.1-Ubuntu [Fri Apr 30 01:37:29 2021] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [Fri Apr 30 01:37:29 2021] nv_queue D 0 1836 2 0x80004000 [Fri Apr 30 01:37:29 2021] Call Trace: [Fri Apr 30 01:37:29 2021] __schedule+0x292/0x710 [Fri Apr 30 01:37:29 2021] ? update_load_avg+0x17d/0x680 [Fri Apr 30 01:37:29 2021] schedule+0x33/0xa0 [Fri Apr 30 01:37:29 2021] schedule_timeout+0x1d3/0x320 [Fri Apr 30 01:37:29 2021] ? __switch_to_asm+0x34/0x70 [Fri Apr 30 01:37:29 2021] ? __switch_to_asm+0x40/0x70 [Fri Apr 30 01:37:29 2021] ? __switch_to_asm+0x34/0x70 [Fri Apr 30 01:37:29 2021] ? __switch_to_asm+0x40/0x70 [Fri Apr 30 01:37:29 2021] ? __switch_to_asm+0x34/0x70 [Fri Apr 30 01:37:29 2021] ? __switch_to_asm+0x40/0x70 [Fri Apr 30 01:37:29 2021] ? __switch_to_asm+0x34/0x70 [Fri Apr 30 01:37:29 2021] __down+0x91/0xe0 [Fri Apr 30 01:37:29 2021] ? x86_gsbase_write_task+0x20/0x30 [Fri Apr 30 01:37:29 2021] ? __switch_to_asm+0x40/0x70 [Fri Apr 30 01:37:29 2021] down+0x41/0x50 [Fri Apr 30 01:37:29 2021] ? down+0x41/0x50 [Fri Apr 30 01:37:29 2021] os_acquire_mutex+0x34/0x40 [nvidia] [Fri Apr 30 01:37:29 2021] os_acquire_semaphore+0xe/0x10 [nvidia] [Fri Apr 30 01:37:29 2021] _nv035446rm+0xc/0x30 [nvidia] [Fri Apr 30 01:37:29 2021] ? _nv035438rm+0x15/0x20 [nvidia] [Fri Apr 30 01:37:29 2021] ? _nv036298rm+0x155/0x1a0 [nvidia] [Fri Apr 30 01:37:29 2021] ? _nv041724rm+0x60/0x270 [nvidia] [Fri Apr 30 01:37:29 2021] ? rm_execute_work_item+0x56/0x120 [nvidia] [Fri Apr 30 01:37:29 2021] ? os_execute_work_item+0x4a/0x60 [nvidia] [Fri Apr 30 01:37:29 2021] ? _main_loop+0x76/0x140 [nvidia] [Fri Apr 30 01:37:29 2021] ? kthread+0x121/0x140 [Fri Apr 30 01:37:29 2021] ? _raw_q_schedule+0x80/0x80 [nvidia] [Fri Apr 30 01:37:29 2021] ? kthread_park+0x90/0x90 [Fri Apr 30 01:37:29 2021] ? ret_from_fork+0x22/0x40 [Fri Apr 30 01:37:47 2021] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [python3:5432] [Fri Apr 30 01:37:47 2021] Modules linked in: nvidia_uvm(OE) veth xt_nat xt_tcpudp xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype iptable_filter iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 bpfilter br_netfilter bridge stp llc aufs cmac overlay bnep snd_hda_codec_hdmi nvidia_drm(POE) nvidia_modeset(POE) edac_mce_amd snd_usb_audio snd_usbmidi_lib mc kvm_amd kvm iwlmvm mac80211 crct10dif_pclmul crc32_pclmul btusb nls_iso8859_1 btrtl nvidia(POE) libarc4 btbcm ghash_clmulni_intel btintel bluetooth iwlwifi ecdh_generic aesni_intel snd_seq_midi input_leds ecc crypto_simd snd_seq_midi_event cryptd glue_helper cfg80211 wmi_bmof snd_hda_intel snd_rawmidi snd_intel_dspcfg joydev snd_hda_codec nouveau snd_hda_core snd_hwdep snd_seq snd_pcm mxm_wmi video snd_seq_device ccp ttm snd_timer drm_kms_helper snd drm i2c_algo_bit fb_sys_fops syscopyarea sysfillrect sysimgblt k10temp soundcore mac_hid sch_fq_codel parport_pc ppdev lp parport ip_tables x_tables [Fri Apr 30 01:37:47 2021] autofs4 btrfs xor zstd_compress raid6_pq libcrc32c dm_mirror dm_region_hash dm_log hid_generic usbhid hid nvme nvme_core ahci r8169 libahci atlantic realtek i2c_piix4 wmi [Fri Apr 30 01:37:47 2021] CPU: 3 PID: 5432 Comm: python3 Tainted: P W OEL 5.4.0-72-generic #80~18.04.1-Ubuntu [Fri Apr 30 01:37:47 2021] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./TRX40 Creator, BIOS P1.70 05/29/2020 [Fri Apr 30 01:37:47 2021] RIP: 0010:fetch_pte.isra.7+0x66/0x1a0 [Fri Apr 30 01:37:47 2021] Code: 1c c0 48 89 e5 53 48 8b 36 41 8d 4b 0c 48 d3 ef 81 e7 ff 01 00 00 4c 8d 14 fe be 01 00 00 00 48 d3 e6 85 c0 49 89 30 49 8b 3a <0f> 8e b9 00 00 00 40 f6 c7 01 0f 84 1f 01 00 00 48 89 fe 48 c1 ee [Fri Apr 30 01:37:47 2021] RSP: 0018:fffface947c0f830 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [Fri Apr 30 01:37:47 2021] RAX: 0000000000000003 RBX: 0000000000001000 RCX: 0000000000000027 [Fri Apr 30 01:37:47 2021] RDX: 00008afcf4dd9000 RSI: 0000008000000000 RDI: 0000000000000000 [Fri Apr 30 01:37:47 2021] RBP: fffface947c0f838 R08: fffface947c0f848 R09: 0000000000000004 [Fri Apr 30 01:37:47 2021] R10: ffff9dc8ebc988a8 R11: 000000000000001b R12: ffff9dcbe81fb098 [Fri Apr 30 01:37:47 2021] R13: ffff9dcbe81fb094 R14: 00008afcf4dd9000 R15: 0000000000000000 [Fri Apr 30 01:37:47 2021] FS: 00007f9e29fff700(0000) GS:ffff9dcbfcec0000(0000) knlGS:0000000000000000 [Fri Apr 30 01:37:47 2021] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fri Apr 30 01:37:47 2021] CR2: 00007f34d2eec8a8 CR3: 0000001fe61a0000 CR4: 0000000000340ee0 [Fri Apr 30 01:37:47 2021] Call Trace: [Fri Apr 30 01:37:47 2021] iommu_unmap_page+0x64/0x100 [Fri Apr 30 01:37:47 2021] __unmap_single.isra.27+0x66/0x130 [Fri Apr 30 01:37:47 2021] unmap_sg+0x5f/0x70 [Fri Apr 30 01:37:47 2021] nv_unmap_dma_map_scatterlist+0x57/0xa0 [nvidia] [Fri Apr 30 01:37:47 2021] nv_dma_unmap_pages+0x56/0x100 [nvidia] [Fri Apr 30 01:37:47 2021] nv_dma_unmap_alloc+0x16/0x30 [nvidia] [Fri Apr 30 01:37:47 2021] _nv032962rm+0xc1/0x1b0 [nvidia] [Fri Apr 30 01:37:47 2021] ? _nv027787rm+0x9b/0xc0 [nvidia] [Fri Apr 30 01:37:47 2021] ? _nv008192rm+0x29/0x40 [nvidia] [Fri Apr 30 01:37:47 2021] ? _nv028912rm+0x7f/0x2a0 [nvidia] [Fri Apr 30 01:37:47 2021] ? _nv028890rm+0x263/0x900 [nvidia] [Fri Apr 30 01:37:47 2021] ? _nv002524rm+0x9/0x20 [nvidia] [Fri Apr 30 01:37:47 2021] ? _nv003516rm+0x1b/0x80 [nvidia] [Fri Apr 30 01:37:47 2021] ? _nv013017rm+0x6fe/0x770 [nvidia] [Fri Apr 30 01:37:47 2021] ? _nv037997rm+0xb3/0x150 [nvidia] [Fri Apr 30 01:37:47 2021] ? _nv037996rm+0x297/0x4e0 [nvidia] [Fri Apr 30 01:37:47 2021] ? _nv037991rm+0x60/0x70 [nvidia] [Fri Apr 30 01:37:47 2021] ? _nv037992rm+0x7b/0xb0 [nvidia] [Fri Apr 30 01:37:47 2021] ? _nv036285rm+0x40/0xe0 [nvidia] [Fri Apr 30 01:37:47 2021] ? _nv000686rm+0x67/0xa0 [nvidia] [Fri Apr 30 01:37:47 2021] ? rm_cleanup_file_private+0xea/0x140 [nvidia] [Fri Apr 30 01:37:47 2021] ? uvm_kvfree+0x30/0x70 [nvidia_uvm] [Fri Apr 30 01:37:47 2021] ? nvidia_close+0x14e/0x2d0 [nvidia] [Fri Apr 30 01:37:47 2021] ? nvidia_frontend_close+0x2f/0x50 [nvidia] [Fri Apr 30 01:37:47 2021] ? __fput+0xc6/0x260 [Fri Apr 30 01:37:47 2021] ? ____fput+0xe/0x10 [Fri Apr 30 01:37:47 2021] ? task_work_run+0x9d/0xc0 [Fri Apr 30 01:37:47 2021] ? do_exit+0x389/0xb90 [Fri Apr 30 01:37:47 2021] ? do_group_exit+0x43/0xa0 [Fri Apr 30 01:37:47 2021] ? get_signal+0x14f/0x860 [Fri Apr 30 01:37:47 2021] ? do_signal+0x34/0x6d0 [Fri Apr 30 01:37:47 2021] ? __switch_to_asm+0x34/0x70 [Fri Apr 30 01:37:47 2021] ? __x64_sys_futex+0x13f/0x190 [Fri Apr 30 01:37:47 2021] ? exit_to_usermode_loop+0x90/0x130 [Fri Apr 30 01:37:47 2021] ? do_syscall_64+0x170/0x190 [Fri Apr 30 01:37:47 2021] ? entry_SYSCALL_64_after_hwframe+0x44/0xa9 [Fri Apr 30 01:38:15 2021] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [python3:5432] [Fri Apr 30 01:38:15 2021] Modules linked in: nvidia_uvm(OE) veth xt_nat xt_tcpudp xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype iptable_filter iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 bpfilter br_netfilter bridge stp llc aufs cmac overlay bnep snd_hda_codec_hdmi nvidia_drm(POE) nvidia_modeset(POE) edac_mce_amd snd_usb_audio snd_usbmidi_lib mc kvm_amd kvm iwlmvm mac80211 crct10dif_pclmul crc32_pclmul btusb nls_iso8859_1 btrtl nvidia(POE) libarc4 btbcm ghash_clmulni_intel btintel bluetooth iwlwifi ecdh_generic aesni_intel snd_seq_midi input_leds ecc crypto_simd snd_seq_midi_event cryptd glue_helper cfg80211 wmi_bmof snd_hda_intel snd_rawmidi snd_intel_dspcfg joydev snd_hda_codec nouveau snd_hda_core snd_hwdep snd_seq snd_pcm mxm_wmi video snd_seq_device ccp ttm snd_timer drm_kms_helper snd drm i2c_algo_bit fb_sys_fops syscopyarea sysfillrect sysimgblt k10temp soundcore mac_hid sch_fq_codel parport_pc ppdev lp parport ip_tables x_tables [Fri Apr 30 01:38:15 2021] autofs4 btrfs xor zstd_compress raid6_pq libcrc32c dm_mirror dm_region_hash dm_log hid_generic usbhid hid nvme nvme_core ahci r8169 libahci atlantic realtek i2c_piix4 wmi [Fri Apr 30 01:38:15 2021] CPU: 3 PID: 5432 Comm: python3 Tainted: P W OEL 5.4.0-72-generic #80~18.04.1-Ubuntu [Fri Apr 30 01:38:15 2021] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./TRX40 Creator, BIOS P1.70 05/29/2020 [Fri Apr 30 01:38:15 2021] RIP: 0010:fetch_pte.isra.7+0x22/0x1a0 [Fri Apr 30 01:38:15 2021] Code: 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 c7 01 00 00 00 00 44 8b 0f 49 89 c8 41 83 f9 05 7f 1a 43 8d 4c c9 0c b8 01 00 00 00 <48> d3 e0 48 83 e8 01 48 39 d0 0f 82 5c 01 00 00 41 8d 41 ff 55 48 [Fri Apr 30 01:38:15 2021] RSP: 0018:fffface947c0f840 EFLAGS: 00000297 ORIG_RAX: ffffffffffffff13 [Fri Apr 30 01:38:15 2021] RAX: 0000000000000001 RBX: 0000000000001000 RCX: 0000000000000030 [Fri Apr 30 01:38:15 2021] RDX: 000089af73403000 RSI: ffff9dcbe81fb098 RDI: ffff9dcbe81fb094 [Fri Apr 30 01:38:15 2021] RBP: fffface947c0f880 R08: fffface947c0f848 R09: 0000000000000004 [Fri Apr 30 01:38:15 2021] R10: 0000000000000000 R11: 000000000000001b R12: ffff9dcbe81fb098 [Fri Apr 30 01:38:15 2021] R13: ffff9dcbe81fb094 R14: 000089af73403000 R15: 0000000000000000 [Fri Apr 30 01:38:15 2021] FS: 00007f9e29fff700(0000) GS:ffff9dcbfcec0000(0000) knlGS:0000000000000000 [Fri Apr 30 01:38:15 2021] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fri Apr 30 01:38:15 2021] CR2: 00007f34d2eec8a8 CR3: 0000001fe61a0000 CR4: 0000000000340ee0 [Fri Apr 30 01:38:15 2021] Call Trace: [Fri Apr 30 01:38:15 2021] ? iommu_unmap_page+0x64/0x100 [Fri Apr 30 01:38:15 2021] __unmap_single.isra.27+0x66/0x130 [Fri Apr 30 01:38:15 2021] unmap_sg+0x5f/0x70 [Fri Apr 30 01:38:15 2021] nv_unmap_dma_map_scatterlist+0x57/0xa0 [nvidia] [Fri Apr 30 01:38:15 2021] nv_dma_unmap_pages+0x56/0x100 [nvidia] [Fri Apr 30 01:38:15 2021] nv_dma_unmap_alloc+0x16/0x30 [nvidia] [Fri Apr 30 01:38:15 2021] _nv032962rm+0xc1/0x1b0 [nvidia] [Fri Apr 30 01:38:15 2021] ? _nv027787rm+0x9b/0xc0 [nvidia] [Fri Apr 30 01:38:15 2021] ? _nv008192rm+0x29/0x40 [nvidia] [Fri Apr 30 01:38:15 2021] ? _nv028912rm+0x7f/0x2a0 [nvidia] [Fri Apr 30 01:38:15 2021] ? _nv028890rm+0x263/0x900 [nvidia] [Fri Apr 30 01:38:15 2021] ? _nv002524rm+0x9/0x20 [nvidia] [Fri Apr 30 01:38:15 2021] ? _nv003516rm+0x1b/0x80 [nvidia] [Fri Apr 30 01:38:15 2021] ? _nv013017rm+0x6fe/0x770 [nvidia] [Fri Apr 30 01:38:15 2021] ? _nv037997rm+0xb3/0x150 [nvidia] [Fri Apr 30 01:38:15 2021] ? _nv037996rm+0x297/0x4e0 [nvidia] [Fri Apr 30 01:38:15 2021] ? _nv037991rm+0x60/0x70 [nvidia] [Fri Apr 30 01:38:15 2021] ? _nv037992rm+0x7b/0xb0 [nvidia] [Fri Apr 30 01:38:15 2021] ? _nv036285rm+0x40/0xe0 [nvidia] [Fri Apr 30 01:38:15 2021] ? _nv000686rm+0x67/0xa0 [nvidia] [Fri Apr 30 01:38:15 2021] ? rm_cleanup_file_private+0xea/0x140 [nvidia] [Fri Apr 30 01:38:15 2021] ? uvm_kvfree+0x30/0x70 [nvidia_uvm] [Fri Apr 30 01:38:15 2021] ? nvidia_close+0x14e/0x2d0 [nvidia] [Fri Apr 30 01:38:15 2021] ? nvidia_frontend_close+0x2f/0x50 [nvidia] [Fri Apr 30 01:38:15 2021] ? __fput+0xc6/0x260 [Fri Apr 30 01:38:15 2021] ? ____fput+0xe/0x10 [Fri Apr 30 01:38:15 2021] ? task_work_run+0x9d/0xc0 [Fri Apr 30 01:38:15 2021] ? do_exit+0x389/0xb90 [Fri Apr 30 01:38:15 2021] ? do_group_exit+0x43/0xa0 [Fri Apr 30 01:38:15 2021] ? get_signal+0x14f/0x860 [Fri Apr 30 01:38:15 2021] ? do_signal+0x34/0x6d0 [Fri Apr 30 01:38:15 2021] ? __switch_to_asm+0x34/0x70 [Fri Apr 30 01:38:15 2021] ? __x64_sys_futex+0x13f/0x190 [Fri Apr 30 01:38:15 2021] ? exit_to_usermode_loop+0x90/0x130 [Fri Apr 30 01:38:15 2021] ? do_syscall_64+0x170/0x190 [Fri Apr 30 01:38:15 2021] ? entry_SYSCALL_64_after_hwframe+0x44/0xa9 [Fri Apr 30 01:38:26 2021] rcu: INFO: rcu_sched self-detected stall on CPU [Fri Apr 30 01:38:26 2021] rcu: 3-....: (15000 ticks this GP) idle=0b2/1/0x4000000000000002 softirq=7208/7208 fqs=7488 [Fri Apr 30 01:38:26 2021] (t=15001 jiffies g=47521 q=320008) [Fri Apr 30 01:38:26 2021] NMI backtrace for cpu 3 [Fri Apr 30 01:38:26 2021] CPU: 3 PID: 5432 Comm: python3 Tainted: P W OEL 5.4.0-72-generic #80~18.04.1-Ubuntu [Fri Apr 30 01:38:26 2021] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./TRX40 Creator, BIOS P1.70 05/29/2020 [Fri Apr 30 01:38:26 2021] Call Trace: [Fri Apr 30 01:38:26 2021] [Fri Apr 30 01:38:26 2021] dump_stack+0x6d/0x8b [Fri Apr 30 01:38:26 2021] nmi_cpu_backtrace+0x94/0xa0 [Fri Apr 30 01:38:26 2021] ? lapic_can_unplug_cpu+0xb0/0xb0 [Fri Apr 30 01:38:26 2021] nmi_trigger_cpumask_backtrace+0xe7/0x130 [Fri Apr 30 01:38:26 2021] arch_trigger_cpumask_backtrace+0x19/0x20 [Fri Apr 30 01:38:26 2021] rcu_dump_cpu_stacks+0x9e/0xce [Fri Apr 30 01:38:26 2021] rcu_sched_clock_irq+0x508/0x770 [Fri Apr 30 01:38:26 2021] ? tick_sched_do_timer+0x60/0x60 [Fri Apr 30 01:38:26 2021] update_process_times+0x28/0x50 [Fri Apr 30 01:38:26 2021] tick_sched_handle+0x25/0x70 [Fri Apr 30 01:38:26 2021] tick_sched_timer+0x3c/0x80 [Fri Apr 30 01:38:26 2021] __hrtimer_run_queues+0x10f/0x280 [Fri Apr 30 01:38:26 2021] hrtimer_interrupt+0xe7/0x240 [Fri Apr 30 01:38:26 2021] smp_apic_timer_interrupt+0x6f/0x130 [Fri Apr 30 01:38:26 2021] apic_timer_interrupt+0xf/0x20 [Fri Apr 30 01:38:26 2021] [Fri Apr 30 01:38:26 2021] RIP: 0010:fetch_pte.isra.7+0x29/0x1a0 [Fri Apr 30 01:38:26 2021] Code: 00 0f 1f 44 00 00 48 c7 01 00 00 00 00 44 8b 0f 49 89 c8 41 83 f9 05 7f 1a 43 8d 4c c9 0c b8 01 00 00 00 48 d3 e0 48 83 e8 01 <48> 39 d0 0f 82 5c 01 00 00 41 8d 41 ff 55 48 89 d7 44 8d 1c c0 48 [Fri Apr 30 01:38:26 2021] RSP: 0018:fffface947c0f840 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [Fri Apr 30 01:38:26 2021] RAX: 0000ffffffffffff RBX: 0000000000001000 RCX: 0000000000000030 [Fri Apr 30 01:38:26 2021] RDX: 00008f6ec31fb000 RSI: ffff9dcbe81fb098 RDI: ffff9dcbe81fb094 [Fri Apr 30 01:38:26 2021] RBP: fffface947c0f880 R08: fffface947c0f848 R09: 0000000000000004 [Fri Apr 30 01:38:26 2021] R10: 0000000000000000 R11: 000000000000001b R12: ffff9dcbe81fb098 [Fri Apr 30 01:38:26 2021] R13: ffff9dcbe81fb094 R14: 00008f6ec31fb000 R15: 0000000000000000 [Fri Apr 30 01:38:26 2021] ? iommu_unmap_page+0x64/0x100 [Fri Apr 30 01:38:26 2021] __unmap_single.isra.27+0x66/0x130 [Fri Apr 30 01:38:26 2021] unmap_sg+0x5f/0x70 [Fri Apr 30 01:38:26 2021] nv_unmap_dma_map_scatterlist+0x57/0xa0 [nvidia] [Fri Apr 30 01:38:26 2021] nv_dma_unmap_pages+0x56/0x100 [nvidia] [Fri Apr 30 01:38:26 2021] nv_dma_unmap_alloc+0x16/0x30 [nvidia] [Fri Apr 30 01:38:26 2021] _nv032962rm+0xc1/0x1b0 [nvidia] [Fri Apr 30 01:38:26 2021] ? _nv027787rm+0x9b/0xc0 [nvidia] [Fri Apr 30 01:38:26 2021] ? _nv008192rm+0x29/0x40 [nvidia] [Fri Apr 30 01:38:26 2021] ? _nv028912rm+0x7f/0x2a0 [nvidia] [Fri Apr 30 01:38:26 2021] ? _nv028890rm+0x263/0x900 [nvidia] [Fri Apr 30 01:38:26 2021] ? _nv002524rm+0x9/0x20 [nvidia] [Fri Apr 30 01:38:26 2021] ? _nv003516rm+0x1b/0x80 [nvidia] [Fri Apr 30 01:38:26 2021] ? _nv013017rm+0x6fe/0x770 [nvidia] [Fri Apr 30 01:38:26 2021] ? _nv037997rm+0xb3/0x150 [nvidia] [Fri Apr 30 01:38:26 2021] ? _nv037996rm+0x297/0x4e0 [nvidia] [Fri Apr 30 01:38:26 2021] ? _nv037991rm+0x60/0x70 [nvidia] [Fri Apr 30 01:38:26 2021] ? _nv037992rm+0x7b/0xb0 [nvidia] [Fri Apr 30 01:38:26 2021] ? _nv036285rm+0x40/0xe0 [nvidia] [Fri Apr 30 01:38:26 2021] ? _nv000686rm+0x67/0xa0 [nvidia] [Fri Apr 30 01:38:26 2021] ? rm_cleanup_file_private+0xea/0x140 [nvidia] [Fri Apr 30 01:38:26 2021] ? uvm_kvfree+0x30/0x70 [nvidia_uvm] [Fri Apr 30 01:38:26 2021] ? nvidia_close+0x14e/0x2d0 [nvidia] [Fri Apr 30 01:38:26 2021] ? nvidia_frontend_close+0x2f/0x50 [nvidia] [Fri Apr 30 01:38:26 2021] ? __fput+0xc6/0x260 [Fri Apr 30 01:38:26 2021] ? ____fput+0xe/0x10 [Fri Apr 30 01:38:26 2021] ? task_work_run+0x9d/0xc0 [Fri Apr 30 01:38:26 2021] ? do_exit+0x389/0xb90 [Fri Apr 30 01:38:26 2021] ? do_group_exit+0x43/0xa0 [Fri Apr 30 01:38:26 2021] ? get_signal+0x14f/0x860 [Fri Apr 30 01:38:26 2021] ? do_signal+0x34/0x6d0 [Fri Apr 30 01:38:26 2021] ? __switch_to_asm+0x34/0x70 [Fri Apr 30 01:38:26 2021] ? __x64_sys_futex+0x13f/0x190 [Fri Apr 30 01:38:26 2021] ? exit_to_usermode_loop+0x90/0x130 [Fri Apr 30 01:38:26 2021] ? do_syscall_64+0x170/0x190 [Fri Apr 30 01:38:26 2021] ? entry_SYSCALL_64_after_hwframe+0x44/0xa9 [Fri Apr 30 01:38:51 2021] watchdog: BUG: soft lockup - CPU#27 stuck for 22s! [python3:5432] [Fri Apr 30 01:38:51 2021] Modules linked in: nvidia_uvm(OE) veth xt_nat xt_tcpudp xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype iptable_filter iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 bpfilter br_netfilter bridge stp llc aufs cmac overlay bnep snd_hda_codec_hdmi nvidia_drm(POE) nvidia_modeset(POE) edac_mce_amd snd_usb_audio snd_usbmidi_lib mc kvm_amd kvm iwlmvm mac80211 crct10dif_pclmul crc32_pclmul btusb nls_iso8859_1 btrtl nvidia(POE) libarc4 btbcm ghash_clmulni_intel btintel bluetooth iwlwifi ecdh_generic aesni_intel snd_seq_midi input_leds ecc crypto_simd snd_seq_midi_event cryptd glue_helper cfg80211 wmi_bmof snd_hda_intel snd_rawmidi snd_intel_dspcfg joydev snd_hda_codec nouveau snd_hda_core snd_hwdep snd_seq snd_pcm mxm_wmi video snd_seq_device ccp ttm snd_timer drm_kms_helper snd drm i2c_algo_bit fb_sys_fops syscopyarea sysfillrect sysimgblt k10temp soundcore mac_hid sch_fq_codel parport_pc ppdev lp parport ip_tables x_tables [Fri Apr 30 01:38:51 2021] autofs4 btrfs xor zstd_compress raid6_pq libcrc32c dm_mirror dm_region_hash dm_log hid_generic usbhid hid nvme nvme_core ahci r8169 libahci atlantic realtek i2c_piix4 wmi [Fri Apr 30 01:38:51 2021] CPU: 27 PID: 5432 Comm: python3 Tainted: P W OEL 5.4.0-72-generic #80~18.04.1-Ubuntu [Fri Apr 30 01:38:51 2021] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./TRX40 Creator, BIOS P1.70 05/29/2020 [Fri Apr 30 01:38:51 2021] RIP: 0010:fetch_pte.isra.7+0x63/0x1a0 [Fri Apr 30 01:38:51 2021] Code: d7 44 8d 1c c0 48 89 e5 53 48 8b 36 41 8d 4b 0c 48 d3 ef 81 e7 ff 01 00 00 4c 8d 14 fe be 01 00 00 00 48 d3 e6 85 c0 49 89 30 <49> 8b 3a 0f 8e b9 00 00 00 40 f6 c7 01 0f 84 1f 01 00 00 48 89 fe [Fri Apr 30 01:38:51 2021] RSP: 0018:fffface947c0f830 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [Fri Apr 30 01:38:51 2021] RAX: 0000000000000003 RBX: 0000000000001000 RCX: 0000000000000027 [Fri Apr 30 01:38:51 2021] RDX: 00008c9025016000 RSI: 0000008000000000 RDI: 0000000000000119 [Fri Apr 30 01:38:51 2021] RBP: fffface947c0f838 R08: fffface947c0f848 R09: 0000000000000004 [Fri Apr 30 01:38:51 2021] R10: ffff9dc8ebc988c8 R11: 000000000000001b R12: ffff9dcbe81fb098 [Fri Apr 30 01:38:51 2021] R13: ffff9dcbe81fb094 R14: 00008c9025016000 R15: 0000000000000000 [Fri Apr 30 01:38:51 2021] FS: 00007f9e29fff700(0000) GS:ffff9dcbfd4c0000(0000) knlGS:0000000000000000 [Fri Apr 30 01:38:51 2021] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fri Apr 30 01:38:51 2021] CR2: 000000c42038f000 CR3: 0000001ff5844000 CR4: 0000000000340ee0 [Fri Apr 30 01:38:51 2021] Call Trace: [Fri Apr 30 01:38:51 2021] iommu_unmap_page+0x64/0x100 [Fri Apr 30 01:38:51 2021] __unmap_single.isra.27+0x66/0x130 [Fri Apr 30 01:38:51 2021] unmap_sg+0x5f/0x70 [Fri Apr 30 01:38:51 2021] nv_unmap_dma_map_scatterlist+0x57/0xa0 [nvidia] [Fri Apr 30 01:38:51 2021] nv_dma_unmap_pages+0x56/0x100 [nvidia] [Fri Apr 30 01:38:51 2021] nv_dma_unmap_alloc+0x16/0x30 [nvidia] [Fri Apr 30 01:38:51 2021] _nv032962rm+0xc1/0x1b0 [nvidia] [Fri Apr 30 01:38:51 2021] ? _nv027787rm+0x9b/0xc0 [nvidia] [Fri Apr 30 01:38:51 2021] ? _nv008192rm+0x29/0x40 [nvidia] [Fri Apr 30 01:38:51 2021] ? _nv028912rm+0x7f/0x2a0 [nvidia] [Fri Apr 30 01:38:51 2021] ? _nv028890rm+0x263/0x900 [nvidia] [Fri Apr 30 01:38:51 2021] ? _nv002524rm+0x9/0x20 [nvidia] [Fri Apr 30 01:38:51 2021] ? _nv003516rm+0x1b/0x80 [nvidia] [Fri Apr 30 01:38:51 2021] ? _nv013017rm+0x6fe/0x770 [nvidia] [Fri Apr 30 01:38:51 2021] ? _nv037997rm+0xb3/0x150 [nvidia] [Fri Apr 30 01:38:51 2021] ? _nv037996rm+0x297/0x4e0 [nvidia] [Fri Apr 30 01:38:51 2021] ? _nv037991rm+0x60/0x70 [nvidia] [Fri Apr 30 01:38:51 2021] ? _nv037992rm+0x7b/0xb0 [nvidia] [Fri Apr 30 01:38:51 2021] ? _nv036285rm+0x40/0xe0 [nvidia] [Fri Apr 30 01:38:51 2021] ? _nv000686rm+0x67/0xa0 [nvidia] [Fri Apr 30 01:38:51 2021] ? rm_cleanup_file_private+0xea/0x140 [nvidia] [Fri Apr 30 01:38:51 2021] ? uvm_kvfree+0x30/0x70 [nvidia_uvm] [Fri Apr 30 01:38:51 2021] ? nvidia_close+0x14e/0x2d0 [nvidia] [Fri Apr 30 01:38:51 2021] ? nvidia_frontend_close+0x2f/0x50 [nvidia] [Fri Apr 30 01:38:51 2021] ? __fput+0xc6/0x260 [Fri Apr 30 01:38:51 2021] ? ____fput+0xe/0x10 [Fri Apr 30 01:38:51 2021] ? task_work_run+0x9d/0xc0 [Fri Apr 30 01:38:51 2021] ? do_exit+0x389/0xb90 [Fri Apr 30 01:38:51 2021] ? do_group_exit+0x43/0xa0 [Fri Apr 30 01:38:51 2021] ? get_signal+0x14f/0x860 [Fri Apr 30 01:38:51 2021] ? do_signal+0x34/0x6d0 [Fri Apr 30 01:38:51 2021] ? __switch_to_asm+0x34/0x70 [Fri Apr 30 01:38:51 2021] ? __x64_sys_futex+0x13f/0x190 [Fri Apr 30 01:38:51 2021] ? exit_to_usermode_loop+0x90/0x130 [Fri Apr 30 01:38:51 2021] ? do_syscall_64+0x170/0x190 [Fri Apr 30 01:38:51 2021] ? entry_SYSCALL_64_after_hwframe+0x44/0xa9 [Fri Apr 30 01:39:23 2021] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [python3:5432] [Fri Apr 30 01:39:23 2021] Modules linked in: nvidia_uvm(OE) veth xt_nat xt_tcpudp xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype iptable_filter iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 bpfilter br_netfilter bridge stp llc aufs cmac overlay bnep snd_hda_codec_hdmi nvidia_drm(POE) nvidia_modeset(POE) edac_mce_amd snd_usb_audio snd_usbmidi_lib mc kvm_amd kvm iwlmvm mac80211 crct10dif_pclmul crc32_pclmul btusb nls_iso8859_1 btrtl nvidia(POE) libarc4 btbcm ghash_clmulni_intel btintel bluetooth iwlwifi ecdh_generic aesni_intel snd_seq_midi input_leds ecc crypto_simd snd_seq_midi_event cryptd glue_helper cfg80211 wmi_bmof snd_hda_intel snd_rawmidi snd_intel_dspcfg joydev snd_hda_codec nouveau snd_hda_core snd_hwdep snd_seq snd_pcm mxm_wmi video snd_seq_device ccp ttm snd_timer drm_kms_helper snd drm i2c_algo_bit fb_sys_fops syscopyarea sysfillrect sysimgblt k10temp soundcore mac_hid sch_fq_codel parport_pc ppdev lp parport ip_tables x_tables [Fri Apr 30 01:39:23 2021] autofs4 btrfs xor zstd_compress raid6_pq libcrc32c dm_mirror dm_region_hash dm_log hid_generic usbhid hid nvme nvme_core ahci r8169 libahci atlantic realtek i2c_piix4 wmi [Fri Apr 30 01:39:23 2021] CPU: 3 PID: 5432 Comm: python3 Tainted: P W OEL 5.4.0-72-generic #80~18.04.1-Ubuntu [Fri Apr 30 01:39:23 2021] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./TRX40 Creator, BIOS P1.70 05/29/2020 [Fri Apr 30 01:39:23 2021] RIP: 0010:iommu_unmap_page+0x67/0x100 [Fri Apr 30 01:39:23 2021] Code: 00 4c 8d af 94 00 00 00 4c 8d a7 98 00 00 00 49 89 f6 45 31 ff 48 8d 4d c8 4c 89 e6 4c 89 f2 4c 89 ef e8 fc fc ff ff 48 85 c0 <48> 89 c6 74 3c 48 b8 8f e3 38 8e e3 38 8e e3 f3 48 0f bc 4d c8 48 [Fri Apr 30 01:39:23 2021] RSP: 0018:fffface947c0f848 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13 [Fri Apr 30 01:39:23 2021] RAX: 0000000000000000 RBX: 0000000000001000 RCX: 0000000000000027 [Fri Apr 30 01:39:23 2021] RDX: 00008d53744a8000 RSI: 0000008000000000 RDI: 0000000000000000 [Fri Apr 30 01:39:23 2021] RBP: fffface947c0f880 R08: fffface947c0f848 R09: 0000000000000004 [Fri Apr 30 01:39:23 2021] R10: 0000000000000000 R11: 000000000000001b R12: ffff9dcbe81fb098 [Fri Apr 30 01:39:23 2021] R13: ffff9dcbe81fb094 R14: 00008d53744a8000 R15: 0000000000000000 [Fri Apr 30 01:39:23 2021] FS: 00007f9e29fff700(0000) GS:ffff9dcbfcec0000(0000) knlGS:0000000000000000 [Fri Apr 30 01:39:23 2021] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fri Apr 30 01:39:23 2021] CR2: 00007f64a0000010 CR3: 000000202bfd0000 CR4: 0000000000340ee0 [Fri Apr 30 01:39:23 2021] Call Trace: [Fri Apr 30 01:39:23 2021] __unmap_single.isra.27+0x66/0x130 [Fri Apr 30 01:39:23 2021] unmap_sg+0x5f/0x70 [Fri Apr 30 01:39:23 2021] nv_unmap_dma_map_scatterlist+0x57/0xa0 [nvidia] [Fri Apr 30 01:39:23 2021] nv_dma_unmap_pages+0x56/0x100 [nvidia] [Fri Apr 30 01:39:23 2021] nv_dma_unmap_alloc+0x16/0x30 [nvidia] [Fri Apr 30 01:39:23 2021] _nv032962rm+0xc1/0x1b0 [nvidia] [Fri Apr 30 01:39:23 2021] ? _nv027787rm+0x9b/0xc0 [nvidia] [Fri Apr 30 01:39:23 2021] ? _nv008192rm+0x29/0x40 [nvidia] [Fri Apr 30 01:39:23 2021] ? _nv028912rm+0x7f/0x2a0 [nvidia] [Fri Apr 30 01:39:23 2021] ? _nv028890rm+0x263/0x900 [nvidia] [Fri Apr 30 01:39:23 2021] ? _nv002524rm+0x9/0x20 [nvidia] [Fri Apr 30 01:39:23 2021] ? _nv003516rm+0x1b/0x80 [nvidia] [Fri Apr 30 01:39:23 2021] ? _nv013017rm+0x6fe/0x770 [nvidia] [Fri Apr 30 01:39:23 2021] ? _nv037997rm+0xb3/0x150 [nvidia] [Fri Apr 30 01:39:23 2021] ? _nv037996rm+0x297/0x4e0 [nvidia] [Fri Apr 30 01:39:23 2021] ? _nv037991rm+0x60/0x70 [nvidia] [Fri Apr 30 01:39:23 2021] ? _nv037992rm+0x7b/0xb0 [nvidia] [Fri Apr 30 01:39:23 2021] ? _nv036285rm+0x40/0xe0 [nvidia] [Fri Apr 30 01:39:23 2021] ? _nv000686rm+0x67/0xa0 [nvidia] [Fri Apr 30 01:39:23 2021] ? rm_cleanup_file_private+0xea/0x140 [nvidia] [Fri Apr 30 01:39:23 2021] ? uvm_kvfree+0x30/0x70 [nvidia_uvm] [Fri Apr 30 01:39:23 2021] ? nvidia_close+0x14e/0x2d0 [nvidia] [Fri Apr 30 01:39:23 2021] ? nvidia_frontend_close+0x2f/0x50 [nvidia] [Fri Apr 30 01:39:23 2021] ? __fput+0xc6/0x260 [Fri Apr 30 01:39:23 2021] ? ____fput+0xe/0x10 [Fri Apr 30 01:39:23 2021] ? task_work_run+0x9d/0xc0 [Fri Apr 30 01:39:23 2021] ? do_exit+0x389/0xb90 [Fri Apr 30 01:39:23 2021] ? do_group_exit+0x43/0xa0 [Fri Apr 30 01:39:23 2021] ? get_signal+0x14f/0x860 [Fri Apr 30 01:39:23 2021] ? do_signal+0x34/0x6d0 [Fri Apr 30 01:39:23 2021] ? __switch_to_asm+0x34/0x70 [Fri Apr 30 01:39:23 2021] ? __x64_sys_futex+0x13f/0x190 [Fri Apr 30 01:39:23 2021] ? exit_to_usermode_loop+0x90/0x130 [Fri Apr 30 01:39:23 2021] ? do_syscall_64+0x170/0x190 [Fri Apr 30 01:39:23 2021] ? entry_SYSCALL_64_after_hwframe+0x44/0xa9