Kernel panic on 5-15 with srp_process_rsp

Infrastructure & Networking Software And Drivers Mellanox OFED
1

Hi everyone!

I’d like to report a kernel crash issue with MLNX_OFED_LINUX-23.04-0.5.3.3-ol8.6-x86_64 driver I’ve encountered recently.
I have a storage server with SAS disk shelf and the crash happens if SAS shelf is disconnected unexpectedly from the server. In my case it was faulty cable but the situation can be easely reproduced just manually disconnecting cable from the shelf or the server side. I also looked at the latest MLNX_OFED_SRC-23.10-5.1.4.0 and see that the problem is still there.

Looking through the code it seems like endif does not properly guard scmnd from NULL pointer dereferencing. The patch is pretty simple and attached below, I’ve test on my system and it works just fine.

Here is a crash log form Linux version 5.15.0-206.153.7.1.el8uek.x86_64

[ 0.000000] Linux version 5.15.0-206.153.7.1.el8uek.x86_64 (mockbuild@host-100-100-224-14) (gcc (GCC) 11.4.1 20230605 (Red Hat 11.4.1-2.1.0.1), GNU ld version 2.36.1-4.0.2.el8_6) #2 SMP Wed May 22 20:49:34 PDT 2024
[ 0.000000] Command line: BOOT_IMAGE=(hd107,gpt2)/vmlinuz-5.15.0-206.153.7.1.el8uek.x86_64 root=UUID=d96de1d2-9797-4f75-b37f-445ba306f939 ro pci=noaer log_buf_len=4M vga=791 nomodeset amd_iommu=pt iommu=pt transparent_hugepage=never libata.allow_tpm=1 noibrs noibpb nopti nospectre_v2 nospectre_v1 l1tf=off nospec_store_bypass_disable no_stf_barrier mds=off tsx=on tsx_async_abort=off mitigations=off console=ttyS0,115200 console=tty0 crashkernel=0M-2G:128M,2G-6G:256M,6G-8G:512M,8G-64G:768M,64G-200G:1G,200G-:2G resume=UUID=4bfff0d7-4c3b-4a11-bf32-3f43ebc07db9 rhgb quiet

[ 4627.231127] blk_update_request: I/O error, dev dm-27, sector 46883435504 op 0x0:(READ) flags 0x0 phys_seg 2 prio class 0
[ 4627.231132] Buffer I/O error on dev dm-27, logical block 11720858876, async page read
[ 4627.231135] Buffer I/O error on dev dm-27, logical block 11720858877, async page read
[ 4627.231149] blk_update_request: I/O error, dev dm-26, sector 46883435504 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
[ 4627.231170] blk_update_request: I/O error, dev dm-26, sector 46883435504 op 0x0:(READ) flags 0x0 phys_seg 2 prio class 0
[ 4627.231174] Buffer I/O error on dev dm-26, logical block 11720858876, async page read
[ 4627.231175] Buffer I/O error on dev dm-26, logical block 11720858877, async page read
[ 4627.231177] blk_update_request: I/O error, dev dm-25, sector 39069529584 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
[ 4627.231196] blk_update_request: I/O error, dev dm-25, sector 39069529584 op 0x0:(READ) flags 0x0 phys_seg 2 prio class 0
[ 4627.231199] Buffer I/O error on dev dm-25, logical block 9767382396, async page read
[ 4627.231201] Buffer I/O error on dev dm-25, logical block 9767382397, async page read
[ 4627.233230] blk_update_request: I/O error, dev dm-27, sector 46883435392 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
[ 4627.233231] blk_update_request: I/O error, dev dm-24, sector 39069529472 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
[ 4627.233241] Buffer I/O error on dev dm-26, logical block 11720858848, async page read
[ 4627.233244] Buffer I/O error on dev dm-24, logical block 9767382368, async page read

[ 4664.061527] device-mapper: multipath: 252:25: Failing path 70:224.
[ 4664.061560] device-mapper: multipath: 252:25: Failing path 70:240.
[ 4664.061599] device-mapper: multipath: 252:26: Failing path 71:0.
[ 4664.061632] device-mapper: multipath: 252:26: Failing path 71:16.
[ 4664.061671] device-mapper: multipath: 252:27: Failing path 71:32.
[ 4664.061703] device-mapper: multipath: 252:27: Failing path 71:48.

[ 4672.101770] scsi host18: SRP abort called
[ 4672.101772] scsi host19: SRP abort called
[ 4672.102774] scsi host18: SRP abort called
[ 4672.102775] scsi host19: SRP abort called
[ 4672.102780] scsi host18: SRP abort called
[ 4672.166762] scsi host18: SRP abort called
[ 4672.166766] scsi host19: SRP abort called
[ 4672.166772] scsi host19: SRP abort called
[ 4672.792385] scsi host19: Null scmnd for RSP w/tag 0x000000000e001f received on ch 14 / QP 0x163
[ 4672.792386] scsi host19: Null scmnd for RSP w/tag 0x000000000f002b received on ch 15 / QP 0x165
[ 4672.792386] scsi host18: Null scmnd for RSP w/tag 0x000000000e0022 received on ch 14 / QP 0x63
[ 4672.792387] scsi host18: Null scmnd for RSP w/tag 0x0000000020000c received on ch 32 / QP 0x88
[ 4672.792388] scsi host18: Null scmnd for RSP w/tag 0x000000002e002f received on ch 46 / QP 0xa4
[ 4672.792391] scsi host19: Null scmnd for RSP w/tag 0x000000000e001e received on ch 14 / QP 0x163
[ 4672.792396] scsi host19: Null scmnd for RSP w/tag 0x000000000e001a received on ch 14 / QP 0x163
[ 4672.792400] scsi host18: Null scmnd for RSP w/tag 0x000000000e0023 received on ch 14 / QP 0x63
[ 4672.792400] scsi host18: Null scmnd for RSP w/tag 0x000000002f0028 received on ch 47 / QP 0xa6
[ 4672.792403] scsi host18: Null scmnd for RSP w/tag 0x000000000e0024 received on ch 14 / QP 0x63
[ 4672.792411] scsi host19: Null scmnd for RSP w/tag 0x000000002e001a received on ch 46 / QP 0x1a3
[ 4672.792411] scsi host19: Null scmnd for RSP w/tag 0x000000002f0019 received on ch 47 / QP 0x1a5
[ 4672.792425] scsi host19: Null scmnd for RSP w/tag 0x000000000e0018 received on ch 14 / QP 0x163
[ 4672.792431] scsi host19: Null scmnd for RSP w/tag 0x000000000e0019 received on ch 14 / QP 0x163
[ 4672.792432] BUG: kernel NULL pointer dereference, address: 00000000000001c8
[ 4672.801293] #PF: supervisor write access in kernel mode
[ 4672.801295] #PF: error_code(0x0002) - not-present page
[ 4672.801296] PGD 123e44067 P4D 16b226067 PUD 10a9be067 PMD 0
[ 4672.824656] Oops: 0002 [#1] SMP NOPTI
[ 4672.828383] CPU: 39 PID: 0 Comm: swapper/39 Kdump: loaded Tainted: G OE 5.15.0-206.153.7.1.el8uek.x86_64 #2
…
[ 4672.850274] RIP: 0010:srp_process_rsp+0xd5/0x2f7 [ib_srp]
[ 4672.855775] Code: 39 3c 24 0f 85 cc 00 00 00 4d 85 ff 0f 84 c3 00 00 00 49 c7 84 24 30 03 00 00 00 00 00 00 4c 89 ef e8 0f 26 d6 df 0f b6 43 13 <41> 89 87 c8 01 00 00 0f b6 43 12 a8 02 74 3d 8b 4b 1c be 60 00 00
[ 4672.874816] RSP: 0018:ff46aa7d4cf04e40 EFLAGS: 00010246
[ 4672.880128] RAX: 0000000000000000 RBX: ff2e23120cc6c000 RCX: 0000000000000000
[ 4672.887372] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[ 4672.894625] RBP: ff2e2330d2c49500 R08: 0000000000000000 R09: 0000000000000000
[ 4672.901871] R10: 0000000000000000 R11: 0000000000000000 R12: ff2e2311eb957800
[ 4672.909114] R13: ff2e2330d2c49510 R14: ff2e2311eb957b30 R15: 0000000000000000
[ 4672.916363] FS: 0000000000000000(0000) GS:ff2e23303ffc0000(0000) knlGS:0000000000000000
[ 4672.924576] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 4672.930417] CR2: 00000000000001c8 CR3: 0000000170110005 CR4: 0000000000771ee0
[ 4672.937660] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 4672.944905] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 4672.952152] PKRU: 55555554
[ 4672.954909] Call Trace:
[ 4672.957406]
[ 4672.959460] ? show_trace_log_lvl+0x1d6/0x2f9
[ 4672.963895] ? show_trace_log_lvl+0x1d6/0x2f9
[ 4672.968322] ? srp_recv_done+0x82/0x1c5 [ib_srp]
[ 4672.973019] ? __die_body.cold+0x8/0xa
[ 4672.976832] ? page_fault_oops+0x16d/0x1ac
[ 4672.981010] ? exc_page_fault+0x68/0x13b
[ 4672.985004] ? asm_exc_page_fault+0x22/0x27
[ 4672.989262] ? srp_process_rsp+0xd5/0x2f7 [ib_srp]
[ 4672.994138] srp_recv_done+0x82/0x1c5 [ib_srp]
[ 4672.998666] __ib_process_cq+0x8b/0x190 [ib_core]
[ 4673.003484] ib_poll_handler+0x2b/0xc0 [ib_core]
[ 4673.008203] irq_poll_softirq+0x95/0x12f
[ 4673.012200] __do_softirq+0xcd/0x2a5
[ 4673.015840] ? sched_clock_cpu+0x9/0xb6
[ 4673.023448] __irq_exit_rcu+0xc7/0xf1
[ 4673.030874] common_interrupt+0x80/0x98
[ 4673.038519]
[ 4673.044342]
[ 4673.050115] asm_common_interrupt+0x22/0x27
[ 4673.057896] RIP: 0010:cpuidle_enter_state+0xc7/0x35d
[ 4673.066370] Code: 8b 3d ad a1 6d 5f e8 78 87 7c ff 49 89 c5 0f 1f 44 00 00 31 ff e8 39 a3 7c ff 45 84 ff 0f 85 0c 01 00 00 fb 66 0f 1f 44 00 00 <45> 85 f6 0f 88 18 01 00 00 49 63 d6 4c 2b 2c 24 48 8d 04 52 48 8d
[ 4673.092473] RSP: 0018:ff46aa7d4c77fe98 EFLAGS: 00000246
[ 4673.101305] RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0000000000000000
[ 4673.112068] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[ 4673.122736] RBP: ff78aa5cc13c9900 R08: 0000000000000000 R09: 0000000000000000
[ 4673.133395] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa24e8320
[ 4673.144028] R13: 0000043ff821114d R14: 0000000000000001 R15: 0000000000000000
[ 4673.154646] ? cpuidle_enter_state+0xb7/0x35d
[ 4673.162426] cpuidle_enter+0x29/0x40
[ 4673.169299] cpuidle_idle_call+0x143/0x1de
[ 4673.176579] do_idle+0x81/0xd2
[ 4673.182732] cpu_startup_entry+0x19/0x1b
[ 4673.189630] secondary_startup_64_no_verify+0xc2/0x0
[ 4673.197523]
[ 4673.202584] Modules linked in: iscsi_tcp libiscsi_tcp libiscsi drbd(OE) iscsi_scst(OE) qla2x00tgt(OE) qla2xxx_scst(OE) nvme_fc(OE) scsi_transport_fc ib_srpt(OE) av_calc_nm_c_sse(OE) av_calc_arf8_avx(OE) av_calc_arf8_sse(OE) av_calc_arf8_c_avx(OE) av_calc_arf8_c_sse(OE) av_calc_arf64(OE) ca_bridge(OE) nvme_tcp(OE) nvme_rdma(OE) nvme_fabrics(OE) nvmet_tcp(OE) nvmet_rdma(OE) nvmet(OE) nvme(OE) nvme_core(OE) scst_vdisk(OE) scst_local(OE) scst(OE) dlm local_rvm(OE) rp_ira_range(OE) rp_lsc(OE) xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nft_counter vfio_pci vfio_pci_core vfio_virqfd cuse scsi_transport_iscsi rdma_ucm(OE) ib_srp(OE) scsi_transport_srp(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) sunrpc vfat fat dm_round_robin dm_multipath dm_mod ipmi_ssif intel_rapl_msr intel_rapl_common i10nm_edac nfit libnvdimm x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel iTCO_wdt
[ 4673.202633] iTCO_vendor_support kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel sha1_ssse3 aesni_intel crypto_simd isst_if_mbox_pci isst_if_mmio cryptd acpi_ipmi pcspkr isst_if_common ses enclosure ipmi_si mei_me ipmi_devintf i2c_i801 mei ipmi_msghandler i2c_smbus ioatdma intel_pch_thermal intel_pmt fuse ext4 mbcache jbd2 mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_vram_helper drm_kms_helper sd_mod t10_pi sg syscopyarea sysfillrect sysimgblt fb_sys_fops cec drm_ttm_helper ttm drm ahci mlx5_core(OE) libahci mlxfw(OE) pci_hyperv_intf psample sha256_ssse3 mpt3sas(OE) mlxdevm(OE) libata mlx_compat(OE) raid_class igb(OE) tls scsi_transport_sas vmd wmi dca
[ 4673.401785] CR2: 00000000000001c8

The patch is an attachment (replace .txt with .patch).

— ib_srp.c 2023-12-26 18:30:20.000000000 +0300
+++ ib_srp_patched.c 2025-05-16 15:22:30.130305900 +0300
@@ -2149,9 +2149,9 @@
} else {
#ifndef HAVE_SCSI_HOST_TEMPLATE_INIT_CMD_PRIV
scmnd = NULL;
+#endif
}
if (!scmnd) {
-endif
shost_printk(KERN_ERR, target->scsi_host,
“Null scmnd for RSP w/tag %#016llx received on ch %td / QP %#x\n”,
rsp->tag, ch - target->ch, ch->qp->qp_num);

Best regards
Max Zev

srp_process_rsp.txt (420 Bytes)

2

Hi Max

Thanks for the very detailed report and for pinpointing this to srp_process_rsp() – that’s really helpful.

The small diff you posted:

} else {
#ifndef HAVE_SCSI_HOST_TEMPLATE_INIT_CMD_PRIV
scmnd = NULL;
+#endif
}
if (!scmnd) {
-#endif
shost_printk(KERN_ERR, target->scsi_host,
"Null scmnd for RSP w/tag %#016llx received on ch %td / QP %#x\n",
rsp->tag, ch - target->ch, ch->qp->qp_num);

does make sense: the original #endif placement can cause the if (!scmnd) block to be compiled out in some configurations, which then allows a NULL scmnd to slip through and be dereferenced later.

That said, we can’t treat this as an official fix without going through our normal review/validation process, and we also need your exact FW/PSID/environment.

Could you please open an NVIDIA Enterprise Support ticket and attach a sysinfo snapshot from the affected server (provided by ofed-scripts)? That will give our dev team the full picture to reproduce and integrate a reviewed fix into a future release.

In the meantime, if your local patch solves the crash in your setup, you can keep using it at your own risk, but we’d recommend moving to an official build once a validated fix is available.

Thanks,
Xu

1 Like
3

Indeed,
I’ve just looked through my post and realised that I missed few details:
I have a storage server, that consists of 2 Supermicro servers connected by InfiniBand via IB SRP for SCSI synchronization. Also, SAS disk JBOD is connected to each of them (mpt3sas). The crash happens if SAS shelf is disconnected unexpectedly from one of the servers. At the first glance, SAS has nothing to do with IB SRP, but in practice they affect each other because in my case IB SRP is used as internal channel to transfer SCSI commands between servers and JBOD: one of the servers is a SCSI target that handles data stream from SCSI-initiator, this stream goes simultaneously to the second server via IB SRP and the JBOD. The crash happens because a pointer to SCSI command inside ib_srp.c becomes NULL due to the loss of connection to SAS shelf.

The driver is compiled with HAVE_BLK_TAGS 1, HAVE_SCSI_HOST_TEMPLATE_INIT_CMD_PRIV 1

I’ve opened an NVIDIA Enterprise Support ticket as you said. Hope it helps.

Best regards,
Max Zev