Kernel crash / softlockup with OFED 5.8-4.1.5.0 LTS on Ubuntu 19.04

Hello, my test to reproduce this is quite simple. On an Ubuntu 19.04 system, create an md RAID 0 set with 16 SSDs, then run fio (3.28) against the /dev/mdX device with 4K random write or random read IO with a 128 iodepth and 32 threads. Within seconds, there will be a CPU softlockup, followed by a crash. I have reproduced this on multiple machines. Below is the snippet from the kernel log. Concerned since this is happening with the LTS OFED. I am happy to provide more details. Thank you in advance for any assistance you can provide.

2024-04-05_12:31:02 linuxhost kernel: [ 3688.802913] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [fio:96379]
2024-04-05_12:31:02 linuxhost kernel: [ 3688.809905] Modules linked in: nvmet(O) scst_qla2xxx(O) qla2xxx(O) pci_stub nvme_fc(O) scsi_transport_fc scst(O) dlm aggengine(O) mst_pciconf(O) ib_ipoib(O) sch_fq_codel ib_iser(O) iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi nvme_rdma(O) rdma_cm(O) iw_cm(O) ib_cm(O) nvme_tcp(O) nvme_fabrics(O) amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass nls_iso8859_1 crct10dif_pclmul crc32_pclmul ghash_clmulni_intel ipmi_ssif aesni_intel aes_x86_64 crypto_simd cryptd glue_helper ccp k10temp ipmi_si ipmi_devintf ipmi_msghandler zfs(PO) qat_api(O) spl(O) intel_qat(O) uio bonding 8021q garp mrp stp llc blocklayoutdriver xfs nfsv4 nfsd nfs auth_rpcgss fscache nfs_acl lockd grace sunrpc ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear mlx5_ib(O) ib_uverbs(O) ib_core(O) mlx5_core(O) i40e(O) intel_auxiliary(O) tls psample mlxdevm(O) auxiliary(O) igb devlink nvme(O) i2c_piix4
2024-04-05_12:31:02 linuxhost kernel: [ 3688.809946] i2c_algo_bit nvme_core(O) dca mlx_compat(O) [last unloaded: mst_pci]
2024-04-05_12:31:02 linuxhost kernel: [ 3688.809950] CPU: 0 PID: 96379 Comm: fio Kdump: loaded Tainted: P O 5.0.0-25-7-gcc9 #40-24
2024-04-05_12:31:02 linuxhost kernel: [ 3688.809951] Hardware name: EFCTxx, BIOS 5.14 08/26/2021
2024-04-05_12:31:02 linuxhost kernel: [ 3688.809956] RIP: 0010:__do_softirq+0x76/0x2f4
2024-04-05_12:31:02 linuxhost kernel: [ 3688.809958] Code: 00 00 44 0f b7 f2 c7 45 c0 0a 00 00 00 48 89 45 b0 48 89 45 a0 44 89 75 d4 65 66 c7 05 51 25 e2 6f 00 00 fb 66 0f 1f 44 00 00 ff ff ff ff 48 c7 45 c8 00 51 a0 90 0f bc 45 d4 83 c0 01 89 45
2024-04-05_12:31:02 linuxhost kernel: [ 3688.809958] RSP: 0000:ffff8aba0e803f50 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff13
2024-04-05_12:31:02 linuxhost kernel: [ 3688.809960] RAX: ffff8ab8f442c500 RBX: 0000000000000000 RCX: 0000000000000020
2024-04-05_12:31:02 linuxhost kernel: [ 3688.809960] RDX: 0000000000000002 RSI: 000000000000612f RDI: 0000000000400140
2024-04-05_12:31:02 linuxhost kernel: [ 3688.809961] RBP: ffff8aba0e803fb8 R08: 0000000000000001 R09: 00000354a1e67af0
2024-04-05_12:31:02 linuxhost kernel: [ 3688.809961] R10: 00000354a1a971f0 R11: 0000000000001ce0 R12: ffff8aba0e815f00
2024-04-05_12:31:02 linuxhost kernel: [ 3688.809962] R13: 0000000000000000 R14: 0000000000000002 R15: 0000000000000000
2024-04-05_12:31:02 linuxhost kernel: [ 3688.809963] FS: 00007fd06ddc2680(0000) GS:ffff8aba0e800000(0000) knlGS:0000000000000000
2024-04-05_12:31:02 linuxhost kernel: [ 3688.809963] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
2024-04-05_12:31:02 linuxhost kernel: [ 3688.809964] CR2: 00007fd06dd67000 CR3: 0000001e76f66000 CR4: 0000000000340ef0
2024-04-05_12:31:02 linuxhost kernel: [ 3688.809964] Call Trace:
2024-04-05_12:31:02 linuxhost kernel: [ 3688.809965]
2024-04-05_12:31:02 linuxhost kernel: [ 3688.809970] ? hrtimer_interrupt+0x13b/0x220
2024-04-05_12:31:02 linuxhost kernel: [ 3688.809973] irq_exit+0xc5/0xd0
2024-04-05_12:31:02 linuxhost kernel: [ 3688.809975] smp_apic_timer_interrupt+0x7b/0x150
2024-04-05_12:31:02 linuxhost kernel: [ 3688.809976] apic_timer_interrupt+0xf/0x20
2024-04-05_12:31:02 linuxhost kernel: [ 3688.809976]
2024-04-05_12:31:02 linuxhost kernel: [ 3688.809978] RIP: 0033:0x558b7bfc573f
2024-04-05_12:31:02 linuxhost kernel: [ 3688.809979] Code: 04 00 48 8d 3d 5e cc 04 00 e8 fd da fc ff 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 55 48 89 fd 53 48 83 ec 28 8b 35 45 07 09 00 <64> 48 8b 04 25 28 00 00 00 48 89 44 24 18 31 c0 83 fe 02 74 3c 83
2024-04-05_12:31:02 linuxhost kernel: [ 3688.809979] RSP: 002b:00007ffe59d2a7e0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
2024-04-05_12:31:02 linuxhost kernel: [ 3688.809980] RAX: 0000000000000000 RBX: 0000000000000018 RCX: 00007fd07717d2e9
2024-04-05_12:31:02 linuxhost kernel: [ 3688.809980] RDX: 0000000000000001 RSI: 0000000000000003 RDI: 00007ffe59d2a840
2024-04-05_12:31:02 linuxhost kernel: [ 3688.809981] RBP: 00007ffe59d2a840 R08: 0000558b7caf3000 R09: 0000000000000018
2024-04-05_12:31:02 linuxhost kernel: [ 3688.809981] R10: 0000558b7cabbac0 R11: 0000000000000246 R12: 0000558b7caa5a10
2024-04-05_12:31:02 linuxhost kernel: [ 3688.809982] R13: 0000000000000000 R14: 0000000000000001 R15: 0000558b7caaabc0
2024-04-05_12:31:30 linuxhost kernel: [ 3716.802094] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [fio:96379]
2024-04-05_12:31:30 linuxhost kernel: [ 3716.809083] Modules linked in: nvmet(O) scst_qla2xxx(O) qla2xxx(O) pci_stub nvme_fc(O) scsi_transport_fc scst(O) dlm aggengine(O) mst_pciconf(O) ib_ipoib(O) sch_fq_codel ib_iser(O) iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi nvme_rdma(O) rdma_cm(O) iw_cm(O) ib_cm(O) nvme_tcp(O) nvme_fabrics(O) amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass nls_iso8859_1 crct10dif_pclmul crc32_pclmul ghash_clmulni_intel ipmi_ssif aesni_intel aes_x86_64 crypto_simd cryptd glue_helper ccp k10temp ipmi_si ipmi_devintf ipmi_msghandler zfs(PO) qat_api(O) spl(O) intel_qat(O) uio bonding 8021q garp mrp stp llc blocklayoutdriver xfs nfsv4 nfsd nfs auth_rpcgss fscache nfs_acl lockd grace sunrpc ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear mlx5_ib(O) ib_uverbs(O) ib_core(O) mlx5_core(O) i40e(O) intel_auxiliary(O) tls psample mlxdevm(O) auxiliary(O) igb devlink nvme(O) i2c_piix4
2024-04-05_12:31:30 linuxhost kernel: [ 3716.809121] i2c_algo_bit nvme_core(O) dca mlx_compat(O) [last unloaded: mst_pci]
2024-04-05_12:31:30 linuxhost kernel: [ 3716.809125] CPU: 0 PID: 96379 Comm: fio Kdump: loaded Tainted: P O L 5.0.0-25-7-gcc9 #40-24
2024-04-05_12:31:30 linuxhost kernel: [ 3716.809126] Hardware name: EFCTxx, BIOS 5.14 08/26/2021
2024-04-05_12:31:30 linuxhost kernel: [ 3716.809131] RIP: 0010:__do_softirq+0x76/0x2f4
2024-04-05_12:31:30 linuxhost kernel: [ 3716.809133] Code: 00 00 44 0f b7 f2 c7 45 c0 0a 00 00 00 48 89 45 b0 48 89 45 a0 44 89 75 d4 65 66 c7 05 51 25 e2 6f 00 00 fb 66 0f 1f 44 00 00 ff ff ff ff 48 c7 45 c8 00 51 a0 90 0f bc 45 d4 83 c0 01 89 45
2024-04-05_12:31:30 linuxhost kernel: [ 3716.809133] RSP: 0000:ffff8aba0e803f50 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff13
2024-04-05_12:31:30 linuxhost kernel: [ 3716.809135] RAX: ffff8ab8f442c500 RBX: 0000000000000000 RCX: 0000000000000020
2024-04-05_12:31:30 linuxhost kernel: [ 3716.809135] RDX: 0000000000000002 RSI: 000000000000612f RDI: 0000000000400140
2024-04-05_12:31:30 linuxhost kernel: [ 3716.809136] RBP: ffff8aba0e803fb8 R08: 0000000000000001 R09: 00000354a1e67af0
2024-04-05_12:31:30 linuxhost kernel: [ 3716.809136] R10: 00000354a1a971f0 R11: 0000000000001ce0 R12: ffff8aba0e815f00
2024-04-05_12:31:30 linuxhost kernel: [ 3716.809137] R13: 0000000000000000 R14: 0000000000000002 R15: 0000000000000000
2024-04-05_12:31:30 linuxhost kernel: [ 3716.809138] FS: 00007fd06ddc2680(0000) GS:ffff8aba0e800000(0000) knlGS:0000000000000000
2024-04-05_12:31:30 linuxhost kernel: [ 3716.809138] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
2024-04-05_12:31:30 linuxhost kernel: [ 3716.809139] CR2: 00007fd06dd67000 CR3: 0000001e76f66000 CR4: 0000000000340ef0
2024-04-05_12:31:30 linuxhost kernel: [ 3716.809139] Call Trace:
2024-04-05_12:31:30 linuxhost kernel: [ 3716.809141]
2024-04-05_12:31:30 linuxhost kernel: [ 3716.809145] ? hrtimer_interrupt+0x13b/0x220
2024-04-05_12:31:30 linuxhost kernel: [ 3716.809148] irq_exit+0xc5/0xd0
2024-04-05_12:31:30 linuxhost kernel: [ 3716.809149] smp_apic_timer_interrupt+0x7b/0x150
2024-04-05_12:31:30 linuxhost kernel: [ 3716.809150] apic_timer_interrupt+0xf/0x20
2024-04-05_12:31:30 linuxhost kernel: [ 3716.809151]
2024-04-05_12:31:30 linuxhost kernel: [ 3716.809152] RIP: 0033:0x558b7bfc573f
2024-04-05_12:31:30 linuxhost kernel: [ 3716.809153] Code: 04 00 48 8d 3d 5e cc 04 00 e8 fd da fc ff 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 55 48 89 fd 53 48 83 ec 28 8b 35 45 07 09 00 <64> 48 8b 04 25 28 00 00 00 48 89 44 24 18 31 c0 83 fe 02 74 3c 83
2024-04-05_12:31:30 linuxhost kernel: [ 3716.809153] RSP: 002b:00007ffe59d2a7e0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
2024-04-05_12:31:30 linuxhost kernel: [ 3716.809154] RAX: 0000000000000000 RBX: 0000000000000018 RCX: 00007fd07717d2e9
2024-04-05_12:31:30 linuxhost kernel: [ 3716.809155] RDX: 0000000000000001 RSI: 0000000000000003 RDI: 00007ffe59d2a840
2024-04-05_12:31:30 linuxhost kernel: [ 3716.809155] RBP: 00007ffe59d2a840 R08: 0000558b7caf3000 R09: 0000000000000018
2024-04-05_12:31:30 linuxhost kernel: [ 3716.809156] R10: 0000558b7cabbac0 R11: 0000000000000246 R12: 0000558b7caa5a10
2024-04-05_12:31:30 linuxhost kernel: [ 3716.809156] R13: 0000000000000000 R14: 0000000000000001 R15: 0000558b7caaabc0

2024-04-05_12:32:46 linuxhost kernel: [ 3792.799880] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [fio:99155]
2024-04-05_12:32:46 linuxhost kernel: [ 3792.806868] Modules linked in: nvmet(O) scst_qla2xxx(O) qla2xxx(O) pci_stub nvme_fc(O) scsi_transport_fc scst(O) dlm aggengine(O) mst_pciconf(O) ib_ipoib(O) sch_fq_codel ib_iser(O) iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi nvme_rdma(O) rdma_cm(O) iw_cm(O) ib_cm(O) nvme_tcp(O) nvme_fabrics(O) amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass nls_iso8859_1 crct10dif_pclmul crc32_pclmul ghash_clmulni_intel ipmi_ssif aesni_intel aes_x86_64 crypto_simd cryptd glue_helper ccp k10temp ipmi_si ipmi_devintf ipmi_msghandler zfs(PO) qat_api(O) spl(O) intel_qat(O) uio bonding 8021q garp mrp stp llc blocklayoutdriver xfs nfsv4 nfsd nfs auth_rpcgss fscache nfs_acl lockd grace sunrpc ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear mlx5_ib(O) ib_uverbs(O) ib_core(O) mlx5_core(O) i40e(O) intel_auxiliary(O) tls psample mlxdevm(O) auxiliary(O) igb devlink nvme(O) i2c_piix4
2024-04-05_12:32:46 linuxhost kernel: [ 3792.806909] i2c_algo_bit nvme_core(O) dca mlx_compat(O) [last unloaded: mst_pci]
2024-04-05_12:32:46 linuxhost kernel: [ 3792.806913] CPU: 0 PID: 99155 Comm: fio Kdump: loaded Tainted: P O L 5.0.0-25-7-gcc9 #40-24
2024-04-05_12:32:46 linuxhost kernel: [ 3792.806913] Hardware name: EFCTxx, BIOS 5.14 08/26/2021
2024-04-05_12:32:46 linuxhost kernel: [ 3792.806918] RIP: 0010:__do_softirq+0x76/0x2f4
2024-04-05_12:32:46 linuxhost kernel: [ 3792.806920] Code: 00 00 44 0f b7 f2 c7 45 c0 0a 00 00 00 48 89 45 b0 48 89 45 a0 44 89 75 d4 65 66 c7 05 51 25 e2 6f 00 00 fb 66 0f 1f 44 00 00 ff ff ff ff 48 c7 45 c8 00 51 a0 90 0f bc 45 d4 83 c0 01 89 45
2024-04-05_12:32:46 linuxhost kernel: [ 3792.806920] RSP: 0000:ffff8aba0e803f50 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff13
2024-04-05_12:32:46 linuxhost kernel: [ 3792.806922] RAX: ffff8ab8f47f8000 RBX: 0000000000000000 RCX: 0000000000000020
2024-04-05_12:32:46 linuxhost kernel: [ 3792.806922] RDX: 0000000000000282 RSI: 00000000000060fc RDI: 0000000000400040
2024-04-05_12:32:46 linuxhost kernel: [ 3792.806923] RBP: ffff8aba0e803fb8 R08: 0000000000000003 R09: 0000036cbb7537f0
2024-04-05_12:32:46 linuxhost kernel: [ 3792.806923] R10: 0000036cbb382ef0 R11: 0000000000001146 R12: ffff8aba0e815f00
2024-04-05_12:32:46 linuxhost kernel: [ 3792.806924] R13: 0000000000000000 R14: 0000000000000282 R15: 0000000000000000
2024-04-05_12:32:46 linuxhost kernel: [ 3792.806925] FS: 00007f76a446f680(0000) GS:ffff8aba0e800000(0000) knlGS:0000000000000000
2024-04-05_12:32:46 linuxhost kernel: [ 3792.806925] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
2024-04-05_12:32:46 linuxhost kernel: [ 3792.806926] CR2: 00007f76a333fdd0 CR3: 0000001e8cd24000 CR4: 0000000000340ef0
2024-04-05_12:32:46 linuxhost kernel: [ 3792.806926] Call Trace:
2024-04-05_12:32:46 linuxhost kernel: [ 3792.806928]
2024-04-05_12:32:46 linuxhost kernel: [ 3792.806932] ? hrtimer_interrupt+0x13b/0x220
2024-04-05_12:32:46 linuxhost kernel: [ 3792.806935] irq_exit+0xc5/0xd0
2024-04-05_12:32:46 linuxhost kernel: [ 3792.806937] smp_apic_timer_interrupt+0x7b/0x150
2024-04-05_12:32:46 linuxhost kernel: [ 3792.806938] apic_timer_interrupt+0xf/0x20
2024-04-05_12:32:46 linuxhost kernel: [ 3792.806938]
2024-04-05_12:32:46 linuxhost kernel: [ 3792.806940] RIP: 0033:0x7f76ad89c5aa
2024-04-05_12:32:46 linuxhost kernel: [ 3792.806941] Code: c5 fe 7f 47 60 c5 fe 7f 44 17 80 48 01 fa 48 83 e2 80 48 39 d1 74 ba c5 fd 7f 01 c5 fd 7f 41 20 c5 fd 7f 41 40 c5 fd 7f 41 60 <48> 81 c1 80 00 00 00 48 39 ca 75 e1 c5 f8 77 c3 80 fa 10 73 1c c4
2024-04-05_12:32:46 linuxhost kernel: [ 3792.806941] RSP: 002b:00007ffc2a3ad1f8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
2024-04-05_12:32:46 linuxhost kernel: [ 3792.806942] RAX: 00007f769ab8b010 RBX: 0000000000000001 RCX: 00007f769aed3280
2024-04-05_12:32:46 linuxhost kernel: [ 3792.806942] RDX: 00007f769b4a3000 RSI: 0000000000000000 RDI: 00007f769ab8b010
2024-04-05_12:32:46 linuxhost kernel: [ 3792.806943] RBP: 000055fe9d12cb60 R08: 0000000000000000 R09: 00007f76ae584048
2024-04-05_12:32:46 linuxhost kernel: [ 3792.806943] R10: 0000000000000000 R11: 0000000000000000 R12: 00007f769bcf07a0
2024-04-05_12:32:46 linuxhost kernel: [ 3792.806944] R13: 0000000000000000 R14: 000055fe9d15ec40 R15: 00007f769bcf07a0
2024-04-05_12:33:14 linuxhost kernel: [ 3820.799079] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [fio:99155]
2024-04-05_12:33:14 linuxhost kernel: [ 3820.806071] Modules linked in: nvmet(O) scst_qla2xxx(O) qla2xxx(O) pci_stub nvme_fc(O) scsi_transport_fc scst(O) dlm aggengine(O) mst_pciconf(O) ib_ipoib(O) sch_fq_codel ib_iser(O) iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi nvme_rdma(O) rdma_cm(O) iw_cm(O) ib_cm(O) nvme_tcp(O) nvme_fabrics(O) amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass nls_iso8859_1 crct10dif_pclmul crc32_pclmul ghash_clmulni_intel ipmi_ssif aesni_intel aes_x86_64 crypto_simd cryptd glue_helper ccp k10temp ipmi_si ipmi_devintf ipmi_msghandler zfs(PO) qat_api(O) spl(O) intel_qat(O) uio bonding 8021q garp mrp stp llc blocklayoutdriver xfs nfsv4 nfsd nfs auth_rpcgss fscache nfs_acl lockd grace sunrpc ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear mlx5_ib(O) ib_uverbs(O) ib_core(O) mlx5_core(O) i40e(O) intel_auxiliary(O) tls psample mlxdevm(O) auxiliary(O) igb devlink nvme(O) i2c_piix4
2024-04-05_12:33:14 linuxhost kernel: [ 3820.806112] i2c_algo_bit nvme_core(O) dca mlx_compat(O) [last unloaded: mst_pci]
2024-04-05_12:33:14 linuxhost kernel: [ 3820.806116] CPU: 0 PID: 99155 Comm: fio Kdump: loaded Tainted: P O L 5.0.0-25-7-gcc9 #40-24
2024-04-05_12:33:14 linuxhost kernel: [ 3820.806117] Hardware name: EFCTxx, BIOS 5.14 08/26/2021
2024-04-05_12:33:14 linuxhost kernel: [ 3820.806122] RIP: 0010:__do_softirq+0x76/0x2f4
2024-04-05_12:33:14 linuxhost kernel: [ 3820.806123] Code: 00 00 44 0f b7 f2 c7 45 c0 0a 00 00 00 48 89 45 b0 48 89 45 a0 44 89 75 d4 65 66 c7 05 51 25 e2 6f 00 00 fb 66 0f 1f 44 00 00 ff ff ff ff 48 c7 45 c8 00 51 a0 90 0f bc 45 d4 83 c0 01 89 45
2024-04-05_12:33:14 linuxhost kernel: [ 3820.806124] RSP: 0000:ffff8aba0e803f50 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff13
2024-04-05_12:33:14 linuxhost kernel: [ 3820.806126] RAX: ffff8ab8f47f8000 RBX: 0000000000000000 RCX: 0000000000000020
2024-04-05_12:33:14 linuxhost kernel: [ 3820.806126] RDX: 0000000000000282 RSI: 00000000000060fc RDI: 0000000000400040
2024-04-05_12:33:14 linuxhost kernel: [ 3820.806127] RBP: ffff8aba0e803fb8 R08: 0000000000000003 R09: 0000036cbb7537f0
2024-04-05_12:33:14 linuxhost kernel: [ 3820.806127] R10: 0000036cbb382ef0 R11: 0000000000001146 R12: ffff8aba0e815f00
2024-04-05_12:33:14 linuxhost kernel: [ 3820.806127] R13: 0000000000000000 R14: 0000000000000282 R15: 0000000000000000
2024-04-05_12:33:14 linuxhost kernel: [ 3820.806128] FS: 00007f76a446f680(0000) GS:ffff8aba0e800000(0000) knlGS:0000000000000000
2024-04-05_12:33:14 linuxhost kernel: [ 3820.806129] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
2024-04-05_12:33:14 linuxhost kernel: [ 3820.806129] CR2: 00007f76a333fdd0 CR3: 0000001e8cd24000 CR4: 0000000000340ef0
2024-04-05_12:33:14 linuxhost kernel: [ 3820.806130] Call Trace:
2024-04-05_12:33:14 linuxhost kernel: [ 3820.806131]
2024-04-05_12:33:14 linuxhost kernel: [ 3820.806136] ? hrtimer_interrupt+0x13b/0x220
2024-04-05_12:33:14 linuxhost kernel: [ 3820.806139] irq_exit+0xc5/0xd0
2024-04-05_12:33:14 linuxhost kernel: [ 3820.806140] smp_apic_timer_interrupt+0x7b/0x150
2024-04-05_12:33:14 linuxhost kernel: [ 3820.806142] apic_timer_interrupt+0xf/0x20
2024-04-05_12:33:14 linuxhost kernel: [ 3820.806142]
2024-04-05_12:33:14 linuxhost kernel: [ 3820.806144] RIP: 0033:0x7f76ad89c5aa
2024-04-05_12:33:14 linuxhost kernel: [ 3820.806145] Code: c5 fe 7f 47 60 c5 fe 7f 44 17 80 48 01 fa 48 83 e2 80 48 39 d1 74 ba c5 fd 7f 01 c5 fd 7f 41 20 c5 fd 7f 41 40 c5 fd 7f 41 60 <48> 81 c1 80 00 00 00 48 39 ca 75 e1 c5 f8 77 c3 80 fa 10 73 1c c4
2024-04-05_12:33:14 linuxhost kernel: [ 3820.806145] RSP: 002b:00007ffc2a3ad1f8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
2024-04-05_12:33:14 linuxhost kernel: [ 3820.806146] RAX: 00007f769ab8b010 RBX: 0000000000000001 RCX: 00007f769aed3280
2024-04-05_12:33:14 linuxhost kernel: [ 3820.806147] RDX: 00007f769b4a3000 RSI: 0000000000000000 RDI: 00007f769ab8b010
2024-04-05_12:33:14 linuxhost kernel: [ 3820.806147] RBP: 000055fe9d12cb60 R08: 0000000000000000 R09: 00007f76ae584048
2024-04-05_12:33:14 linuxhost kernel: [ 3820.806147] R10: 0000000000000000 R11: 0000000000000000 R12: 00007f769bcf07a0
2024-04-05_12:33:14 linuxhost kernel: [ 3820.806148] R13: 0000000000000000 R14: 000055fe9d15ec40 R15: 00007f769bcf07a0

Hello,

Unfortunately, Ubuntu 19.04 is not supported with MLNX_OFED 5.8-4.1.5.0, and has not been tested of verified.
To see which Ubuntu versions are supported, please see the OFED User Manual, here: General Support - NVIDIA Docs

Please try using MLNX_OFED with one of the supported OS’s, and try your test again.
If you still encounter issues after that, please open a case at: enterprisesupport@nvidia.com, and it will be handled according to entitlement.

Best Regards,
Jonathan.