[Thu Jan 26 09:00:27 2023] [nvidia-vgpu-vfio] 00000003-0000-0000-0000-000999004007: vGPU destroy failed: 0x38 [Thu Jan 26 09:00:27 2023] [nvidia-vgpu-vfio] 00000003-0000-0000-0000-000999004007: Failed to destroy vGPU device, ret: -1 [Thu Jan 26 09:00:27 2023] [nvidia-vgpu-vfio] No vGPU device found, close failed [Thu Jan 26 09:00:27 2023] ------------[ cut here ]------------ [Thu Jan 26 09:00:27 2023] nvidia-vgpu-vfio 00000003-0000-0000-0000-000999004007: Removing from iommu group 348 [Thu Jan 26 09:00:27 2023] WARNING: CPU: 0 PID: 19805 at drivers/vfio/vfio_iommu_type1.c:2629 vfio_iommu_type1_detach_group+0x6e5/0x6f0 [vfio_iommu_type1] [Thu Jan 26 09:00:27 2023] nvidia-vgpu-vfio 00000003-0000-0000-0000-000999004007: MDEV: detaching iommu [Thu Jan 26 09:00:27 2023] Modules linked in: cmac nls_utf8 cifs cifs_arc4 cifs_md4 fscache netfs ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_tables iptable_filter bpfilter sctp ip6_udp_tunnel udp_tunnel pci_pf_stub nf_tables dummy nfnetlink_cttimeout bonding tls openvswitch nsh nf_conncount nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 softdog nfnetlink_log nfnetlink binfmt_misc nvidia_vgpu_vfio(O) intel_rapl_msr intel_rapl_common i10nm_edac nfit x86_pkg_temp_thermal ipmi_ssif intel_powerclamp coretemp nvidia(PO) kvm_intel kvm crct10dif_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper cec rc_core rndis_host intel_cstate i2c_algo_bit fb_sys_fops cdc_ether syscopyarea isst_if_mbox_pci isst_if_mmio pcspkr efi_pstore usbnet mdev sysfillrect input_leds isst_if_common joydev mii sysimgblt ioatdma mei_me mei intel_pch_thermal dca acpi_ipmi ipmi_si ipmi_devintf ipmi_msghandler acpi_pad acpi_power_meter [Thu Jan 26 09:00:27 2023] mac_hid vhost_net vhost vhost_iotlb tap ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi vfio_pci vfio_pci_core vfio_virqfd irqbypass vfio_iommu_type1 vfio drm sunrpc ip_tables x_tables autofs4 zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) btrfs blake2b_generic xor zstd_compress raid6_pq libcrc32c simplefb ses enclosure scsi_transport_sas hid_generic usbmouse usbhid hid crc32_pclmul xhci_pci xhci_pci_renesas i40e megaraid_sas i2c_i801 ahci i2c_smbus xhci_hcd libahci intel_pmt [Thu Jan 26 09:00:27 2023] CPU: 0 PID: 19805 Comm: CPU 2/KVM Tainted: P O 5.15.83-1-pve #1 [Thu Jan 26 09:00:27 2023] Hardware name: Supermicro SYS-620C-TN12R/X12DDW-A6, BIOS 1.4 08/04/2022 [Thu Jan 26 09:00:27 2023] RIP: 0010:vfio_iommu_type1_detach_group+0x6e5/0x6f0 [vfio_iommu_type1] [Thu Jan 26 09:00:27 2023] Code: 78 00 74 0a eb 24 48 89 df e8 77 e9 ff ff 4c 89 ef e8 9f 64 4c d7 48 89 c6 48 85 c0 75 e8 e9 fd fd ff ff 0f 0b e9 09 ff ff ff <0f> 0b eb e0 e8 a2 47 bb d7 66 90 0f 1f 44 00 00 55 48 89 e5 41 57 [Thu Jan 26 09:00:27 2023] RSP: 0018:ff75db32c0177ad8 EFLAGS: 00010286 [Thu Jan 26 09:00:27 2023] RAX: ff3bb20180447548 RBX: ff3bb201a1e2c240 RCX: ff3bb20180447558 [Thu Jan 26 09:00:27 2023] RDX: 0000000080800056 RSI: 0000000080800055 RDI: 0000000000000000 [Thu Jan 26 09:00:27 2023] RBP: ff75db32c0177b48 R08: 0000000000000001 R09: 0000000000000001 [Thu Jan 26 09:00:27 2023] R10: ff3bb201739ca400 R11: 000000000000000d R12: ff3bb20180447540 [Thu Jan 26 09:00:27 2023] R13: ff3bb201a1e2c288 R14: ff75db32c0177af8 R15: 0000000000000000 [Thu Jan 26 09:00:27 2023] FS: 0000000000000000(0000) GS:ff3bb21fff600000(0000) knlGS:0000000000000000 [Thu Jan 26 09:00:27 2023] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Thu Jan 26 09:00:27 2023] CR2: 00007f9f1e2e3000 CR3: 0000000119410004 CR4: 0000000000773ef0 [Thu Jan 26 09:00:27 2023] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [Thu Jan 26 09:00:27 2023] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [Thu Jan 26 09:00:27 2023] PKRU: 55555554 [Thu Jan 26 09:00:27 2023] Call Trace: [Thu Jan 26 09:00:27 2023] [Thu Jan 26 09:00:27 2023] __vfio_group_unset_container+0x4a/0x190 [vfio] [Thu Jan 26 09:00:27 2023] vfio_group_try_dissolve_container+0x2d/0x40 [vfio] [Thu Jan 26 09:00:27 2023] vfio_group_put_external_user+0x13/0x40 [vfio] [Thu Jan 26 09:00:27 2023] kvm_vfio_destroy+0x72/0xf0 [kvm] [Thu Jan 26 09:00:27 2023] kvm_destroy_vm+0x1c5/0x2f0 [kvm] [Thu Jan 26 09:00:27 2023] kvm_put_kvm+0x2a/0x50 [kvm] [Thu Jan 26 09:00:27 2023] kvm_vm_release+0x22/0x30 [kvm] [Thu Jan 26 09:00:27 2023] __fput+0x9c/0x280 [Thu Jan 26 09:00:27 2023] ____fput+0xe/0x20 [Thu Jan 26 09:00:27 2023] task_work_run+0x6a/0xb0 [Thu Jan 26 09:00:27 2023] do_exit+0x354/0xa20 [Thu Jan 26 09:00:27 2023] do_group_exit+0x3b/0xb0 [Thu Jan 26 09:00:27 2023] get_signal+0x154/0x910 [Thu Jan 26 09:00:27 2023] ? do_futex+0x147/0xb90 [Thu Jan 26 09:00:27 2023] arch_do_signal_or_restart+0xf7/0x850 [Thu Jan 26 09:00:27 2023] ? __fget_files+0x86/0xc0 [Thu Jan 26 09:00:27 2023] exit_to_user_mode_prepare+0x11d/0x1b0 [Thu Jan 26 09:00:27 2023] syscall_exit_to_user_mode+0x27/0x50 [Thu Jan 26 09:00:27 2023] do_syscall_64+0x69/0xc0 [Thu Jan 26 09:00:27 2023] ? syscall_exit_to_user_mode+0x27/0x50 [Thu Jan 26 09:00:27 2023] ? do_syscall_64+0x69/0xc0 [Thu Jan 26 09:00:27 2023] ? do_syscall_64+0x69/0xc0 [Thu Jan 26 09:00:27 2023] ? do_syscall_64+0x69/0xc0 [Thu Jan 26 09:00:27 2023] ? exc_page_fault+0x89/0x170 [Thu Jan 26 09:00:27 2023] entry_SYSCALL_64_after_hwframe+0x61/0xcb [Thu Jan 26 09:00:27 2023] RIP: 0033:0x7f27d2a937b2 [Thu Jan 26 09:00:27 2023] Code: Unable to access opcode bytes at RIP 0x7f27d2a93788. [Thu Jan 26 09:00:27 2023] RSP: 002b:00007f27cd5f9280 EFLAGS: 00000282 ORIG_RAX: 00000000000000ca [Thu Jan 26 09:00:27 2023] RAX: fffffffffffffe00 RBX: 0000000000000014 RCX: 00007f27d2a937b2 [Thu Jan 26 09:00:27 2023] RDX: 0000000000000000 RSI: 0000000000000080 RDI: 00005641f23a4568 [Thu Jan 26 09:00:27 2023] RBP: 00005641f23a4540 R08: 0000000000000000 R09: 00005641f0b69ec0 [Thu Jan 26 09:00:27 2023] R10: 0000000000000000 R11: 0000000000000282 R12: 0000000000000000 [Thu Jan 26 09:00:27 2023] R13: 00005641f0b6b780 R14: 00005641f23a4568 R15: 00007f27cd5f92b0 [Thu Jan 26 09:00:27 2023] [Thu Jan 26 09:00:27 2023] ---[ end trace ec1b46aec749dc79 ]--- [Thu Jan 26 09:00:27 2023] ------------[ cut here ]------------ [Thu Jan 26 09:00:27 2023] WARNING: CPU: 23 PID: 19805 at drivers/vfio/vfio_iommu_type1.c:1178 vfio_remove_dma+0xd0/0xe0 [vfio_iommu_type1] [Thu Jan 26 09:00:27 2023] Modules linked in: cmac nls_utf8 cifs cifs_arc4 cifs_md4 fscache netfs ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_tables iptable_filter bpfilter sctp ip6_udp_tunnel udp_tunnel pci_pf_stub nf_tables dummy nfnetlink_cttimeout bonding tls openvswitch nsh nf_conncount nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 softdog nfnetlink_log nfnetlink binfmt_misc nvidia_vgpu_vfio(O) intel_rapl_msr intel_rapl_common i10nm_edac nfit x86_pkg_temp_thermal ipmi_ssif intel_powerclamp coretemp nvidia(PO) kvm_intel kvm crct10dif_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper cec rc_core rndis_host intel_cstate i2c_algo_bit fb_sys_fops cdc_ether syscopyarea isst_if_mbox_pci isst_if_mmio pcspkr efi_pstore usbnet mdev sysfillrect input_leds isst_if_common joydev mii sysimgblt ioatdma mei_me mei intel_pch_thermal dca acpi_ipmi ipmi_si ipmi_devintf ipmi_msghandler acpi_pad acpi_power_meter [Thu Jan 26 09:00:27 2023] mac_hid vhost_net vhost vhost_iotlb tap ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi vfio_pci vfio_pci_core vfio_virqfd irqbypass vfio_iommu_type1 vfio drm sunrpc ip_tables x_tables autofs4 zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) btrfs blake2b_generic xor zstd_compress raid6_pq libcrc32c simplefb ses enclosure scsi_transport_sas hid_generic usbmouse usbhid hid crc32_pclmul xhci_pci xhci_pci_renesas i40e megaraid_sas i2c_i801 ahci i2c_smbus xhci_hcd libahci intel_pmt [Thu Jan 26 09:00:27 2023] CPU: 23 PID: 19805 Comm: CPU 2/KVM Tainted: P W O 5.15.83-1-pve #1 [Thu Jan 26 09:00:27 2023] Hardware name: Supermicro SYS-620C-TN12R/X12DDW-A6, BIOS 1.4 08/04/2022 [Thu Jan 26 09:00:27 2023] RIP: 0010:vfio_remove_dma+0xd0/0xe0 [vfio_iommu_type1] [Thu Jan 26 09:00:27 2023] Code: 00 00 00 01 31 c9 31 d2 be 03 00 00 00 48 8d bb 98 00 00 00 e8 21 9d f5 d6 eb c1 be 03 00 00 00 4c 89 c7 e8 a2 cf 47 d7 eb 89 <0f> 0b e9 4a ff ff ff 66 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 [Thu Jan 26 09:00:27 2023] RSP: 0018:ff75db32c0177ab0 EFLAGS: 00010286 [Thu Jan 26 09:00:27 2023] RAX: ff3bb20140119fc0 RBX: ff3bb201a1e2c240 RCX: 00000000802a0008 [Thu Jan 26 09:00:27 2023] RDX: 00000000802a0009 RSI: ff3bb2018142b720 RDI: ff3bb201a1e2c240 [Thu Jan 26 09:00:27 2023] RBP: ff75db32c0177ac8 R08: ff3bb2018142b720 R09: 0000000000000001 [Thu Jan 26 09:00:27 2023] R10: ff3bb2018142b700 R11: dead000000000122 R12: ff3bb2018142b720 [Thu Jan 26 09:00:27 2023] R13: ff3bb201a1e2c288 R14: ff75db32c0177af8 R15: 0000000000000000 [Thu Jan 26 09:00:27 2023] FS: 0000000000000000(0000) GS:ff3bb21fffbc0000(0000) knlGS:0000000000000000 [Thu Jan 26 09:00:27 2023] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Thu Jan 26 09:00:27 2023] CR2: 00005601e1441248 CR3: 00000001155c4005 CR4: 0000000000773ee0 [Thu Jan 26 09:00:27 2023] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [Thu Jan 26 09:00:27 2023] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [Thu Jan 26 09:00:27 2023] PKRU: 55555554 [Thu Jan 26 09:00:27 2023] Call Trace: [Thu Jan 26 09:00:27 2023] [Thu Jan 26 09:00:27 2023] vfio_iommu_type1_detach_group+0x6c9/0x6f0 [vfio_iommu_type1] [Thu Jan 26 09:00:27 2023] __vfio_group_unset_container+0x4a/0x190 [vfio] [Thu Jan 26 09:00:27 2023] vfio_group_try_dissolve_container+0x2d/0x40 [vfio] [Thu Jan 26 09:00:27 2023] vfio_group_put_external_user+0x13/0x40 [vfio] [Thu Jan 26 09:00:27 2023] kvm_vfio_destroy+0x72/0xf0 [kvm] [Thu Jan 26 09:00:27 2023] kvm_destroy_vm+0x1c5/0x2f0 [kvm] [Thu Jan 26 09:00:27 2023] kvm_put_kvm+0x2a/0x50 [kvm] [Thu Jan 26 09:00:27 2023] kvm_vm_release+0x22/0x30 [kvm] [Thu Jan 26 09:00:27 2023] __fput+0x9c/0x280 [Thu Jan 26 09:00:27 2023] ____fput+0xe/0x20 [Thu Jan 26 09:00:27 2023] task_work_run+0x6a/0xb0 [Thu Jan 26 09:00:27 2023] do_exit+0x354/0xa20 [Thu Jan 26 09:00:27 2023] do_group_exit+0x3b/0xb0 [Thu Jan 26 09:00:27 2023] get_signal+0x154/0x910 [Thu Jan 26 09:00:27 2023] ? do_futex+0x147/0xb90 [Thu Jan 26 09:00:27 2023] arch_do_signal_or_restart+0xf7/0x850 [Thu Jan 26 09:00:27 2023] ? __fget_files+0x86/0xc0 [Thu Jan 26 09:00:27 2023] exit_to_user_mode_prepare+0x11d/0x1b0 [Thu Jan 26 09:00:27 2023] syscall_exit_to_user_mode+0x27/0x50 [Thu Jan 26 09:00:27 2023] do_syscall_64+0x69/0xc0 [Thu Jan 26 09:00:27 2023] ? syscall_exit_to_user_mode+0x27/0x50 [Thu Jan 26 09:00:27 2023] ? do_syscall_64+0x69/0xc0 [Thu Jan 26 09:00:27 2023] ? do_syscall_64+0x69/0xc0 [Thu Jan 26 09:00:27 2023] ? do_syscall_64+0x69/0xc0 [Thu Jan 26 09:00:27 2023] ? exc_page_fault+0x89/0x170 [Thu Jan 26 09:00:27 2023] entry_SYSCALL_64_after_hwframe+0x61/0xcb [Thu Jan 26 09:00:27 2023] RIP: 0033:0x7f27d2a937b2 [Thu Jan 26 09:00:27 2023] Code: Unable to access opcode bytes at RIP 0x7f27d2a93788. [Thu Jan 26 09:00:27 2023] RSP: 002b:00007f27cd5f9280 EFLAGS: 00000282 ORIG_RAX: 00000000000000ca [Thu Jan 26 09:00:27 2023] RAX: fffffffffffffe00 RBX: 0000000000000014 RCX: 00007f27d2a937b2 [Thu Jan 26 09:00:27 2023] RDX: 0000000000000000 RSI: 0000000000000080 RDI: 00005641f23a4568 [Thu Jan 26 09:00:27 2023] RBP: 00005641f23a4540 R08: 0000000000000000 R09: 00005641f0b69ec0 [Thu Jan 26 09:00:27 2023] R10: 0000000000000000 R11: 0000000000000282 R12: 0000000000000000 [Thu Jan 26 09:00:27 2023] R13: 00005641f0b6b780 R14: 00005641f23a4568 R15: 00007f27cd5f92b0 [Thu Jan 26 09:00:27 2023] [Thu Jan 26 09:00:27 2023] ---[ end trace ec1b46aec749dc7a ]--- [Thu Jan 26 09:00:34 2023] [nvidia-vgpu-vfio] 00000007-0000-0000-0000-000999004007: Failed to terminate vGPU process. [Thu Jan 26 09:00:34 2023] [nvidia-vgpu-vfio] 00000007-0000-0000-0000-000999004007: vGPU destroy failed: 0x17 [Thu Jan 26 09:00:34 2023] [nvidia-vgpu-vfio] 00000007-0000-0000-0000-000999004007: Failed to destroy vGPU device, ret: -16 [Thu Jan 26 09:00:34 2023] nvidia-vgpu-vfio 00000007-0000-0000-0000-000999004007: Removing from iommu group 352 [Thu Jan 26 09:00:34 2023] nvidia-vgpu-vfio 00000007-0000-0000-0000-000999004007: MDEV: detaching iommu [Thu Jan 26 09:00:41 2023] [nvidia-vgpu-vfio] 00000000-0000-0000-0000-000999004007: Failed to terminate vGPU process. [Thu Jan 26 09:00:41 2023] [nvidia-vgpu-vfio] 00000000-0000-0000-0000-000999004007: vGPU destroy failed: 0x17 [Thu Jan 26 09:00:41 2023] [nvidia-vgpu-vfio] 00000000-0000-0000-0000-000999004007: Failed to destroy vGPU device, ret: -16 [Thu Jan 26 09:00:41 2023] nvidia-vgpu-vfio 00000000-0000-0000-0000-000999004007: Removing from iommu group 345 [Thu Jan 26 09:00:41 2023] nvidia-vgpu-vfio 00000000-0000-0000-0000-000999004007: MDEV: detaching iommu [Thu Jan 26 09:00:48 2023] [nvidia-vgpu-vfio] 00000006-0000-0000-0000-000999004007: Failed to terminate vGPU process. [Thu Jan 26 09:00:48 2023] [nvidia-vgpu-vfio] 00000006-0000-0000-0000-000999004007: vGPU destroy failed: 0x17 [Thu Jan 26 09:00:48 2023] [nvidia-vgpu-vfio] 00000006-0000-0000-0000-000999004007: Failed to destroy vGPU device, ret: -16 [Thu Jan 26 09:00:48 2023] nvidia-vgpu-vfio 00000006-0000-0000-0000-000999004007: Removing from iommu group 351 [Thu Jan 26 09:00:48 2023] nvidia-vgpu-vfio 00000006-0000-0000-0000-000999004007: MDEV: detaching iommu [Thu Jan 26 09:00:55 2023] [nvidia-vgpu-vfio] 00000002-0000-0000-0000-000999004007: Failed to terminate vGPU process. [Thu Jan 26 09:00:55 2023] [nvidia-vgpu-vfio] 00000002-0000-0000-0000-000999004007: vGPU destroy failed: 0x17 [Thu Jan 26 09:00:55 2023] [nvidia-vgpu-vfio] 00000002-0000-0000-0000-000999004007: Failed to destroy vGPU device, ret: -16 [Thu Jan 26 09:00:55 2023] nvidia-vgpu-vfio 00000002-0000-0000-0000-000999004007: Removing from iommu group 347 [Thu Jan 26 09:00:55 2023] nvidia-vgpu-vfio 00000002-0000-0000-0000-000999004007: MDEV: detaching iommu [Thu Jan 26 09:01:02 2023] [nvidia-vgpu-vfio] 00000001-0000-0000-0000-000999004007: Failed to terminate vGPU process. [Thu Jan 26 09:01:02 2023] [nvidia-vgpu-vfio] 00000001-0000-0000-0000-000999004007: vGPU destroy failed: 0x17 [Thu Jan 26 09:01:02 2023] [nvidia-vgpu-vfio] 00000001-0000-0000-0000-000999004007: Failed to destroy vGPU device, ret: -16 [Thu Jan 26 09:01:02 2023] nvidia-vgpu-vfio 00000001-0000-0000-0000-000999004007: Removing from iommu group 346 [Thu Jan 26 09:01:02 2023] nvidia-vgpu-vfio 00000001-0000-0000-0000-000999004007: MDEV: detaching iommu [Thu Jan 26 09:01:10 2023] [nvidia-vgpu-vfio] 00000008-0000-0000-0000-000999004007: Failed to terminate vGPU process. [Thu Jan 26 09:01:10 2023] [nvidia-vgpu-vfio] 00000008-0000-0000-0000-000999004007: vGPU destroy failed: 0x17 [Thu Jan 26 09:01:10 2023] [nvidia-vgpu-vfio] 00000008-0000-0000-0000-000999004007: Failed to destroy vGPU device, ret: -16 [Thu Jan 26 09:01:10 2023] nvidia-vgpu-vfio 00000008-0000-0000-0000-000999004007: Removing from iommu group 353 [Thu Jan 26 09:01:10 2023] nvidia-vgpu-vfio 00000008-0000-0000-0000-000999004007: MDEV: detaching iommu [Thu Jan 26 09:01:17 2023] [nvidia-vgpu-vfio] 00000009-0000-0000-0000-000999004007: Failed to terminate vGPU process. [Thu Jan 26 09:01:17 2023] [nvidia-vgpu-vfio] 00000009-0000-0000-0000-000999004007: vGPU destroy failed: 0x17 [Thu Jan 26 09:01:17 2023] [nvidia-vgpu-vfio] 00000009-0000-0000-0000-000999004007: Failed to destroy vGPU device, ret: -16 [Thu Jan 26 09:01:17 2023] nvidia-vgpu-vfio 00000009-0000-0000-0000-000999004007: Removing from iommu group 354 [Thu Jan 26 09:01:17 2023] nvidia-vgpu-vfio 00000009-0000-0000-0000-000999004007: MDEV: detaching iommu [Thu Jan 26 09:01:24 2023] [nvidia-vgpu-vfio] 00000004-0000-0000-0000-000999004007: Failed to terminate vGPU process. [Thu Jan 26 09:01:24 2023] [nvidia-vgpu-vfio] 00000004-0000-0000-0000-000999004007: vGPU destroy failed: 0x17 [Thu Jan 26 09:01:24 2023] [nvidia-vgpu-vfio] 00000004-0000-0000-0000-000999004007: Failed to destroy vGPU device, ret: -16 [Thu Jan 26 09:01:24 2023] nvidia-vgpu-vfio 00000004-0000-0000-0000-000999004007: Removing from iommu group 349 [Thu Jan 26 09:01:24 2023] nvidia-vgpu-vfio 00000004-0000-0000-0000-000999004007: MDEV: detaching iommu [Thu Jan 26 09:01:31 2023] [nvidia-vgpu-vfio] 00000005-0000-0000-0000-000999004007: Failed to terminate vGPU process. [Thu Jan 26 09:01:31 2023] [nvidia-vgpu-vfio] 00000005-0000-0000-0000-000999004007: vGPU destroy failed: 0x17 [Thu Jan 26 09:01:31 2023] [nvidia-vgpu-vfio] 00000005-0000-0000-0000-000999004007: Failed to destroy vGPU device, ret: -16 [Thu Jan 26 09:01:31 2023] nvidia-vgpu-vfio 00000005-0000-0000-0000-000999004007: Removing from iommu group 350 [Thu Jan 26 09:01:31 2023] nvidia-vgpu-vfio 00000005-0000-0000-0000-000999004007: MDEV: detaching iommu