MLNX OFED 24.10-4.1.4.0 Null pointer dereference(mlx5e_xfrm_add_policy)

Infrastructure & Networking Software And Drivers Mellanox OFED
1

Hi Team - I am running vanilla ubuntu 24.04 with kernel 6.8.0-100-generic and with below NICs

# lspci | grep -i Mellanox
01:00.0 Ethernet controller: Mellanox Technologies MT2892 Family [ConnectX-6 Dx]
01:00.1 Ethernet controller: Mellanox Technologies MT2892 Family [ConnectX-6 Dx]
c1:00.0 Ethernet controller: Mellanox Technologies MT2892 Family [ConnectX-6 Dx]
c1:00.1 Ethernet controller: Mellanox Technologies MT2892 Family [ConnectX-6 Dx]

I have downloaded the Linux drivers from Linux InfiniBand Drivers and using the MLNX OFED 24.10-4.1.4.0.

After installing the kernel modules when I am bringing up the tunnels with hw_offload as packet then I am seeing the below kernel panic with Null pointer dereference. Can you help me in debugging this issue further.

Appreciate any help in this regard.

Regards,

Rajiv

root@mag-master-10-106-172-173:~# sudo journalctl -b -1 -e
Apr 08 10:34:28 mag-master-10-106-172-173 kernel: RIP: 0010:mlx5e_xfrm_add_policy+0x1c/0x470 [mlx5_core]
Apr 08 10:34:28 mag-master-10-106-172-173 kernel: IPv4: martian source 169.254.169.254 from 10.106.172.35, on dev internet
Apr 08 10:34:28 mag-master-10-106-172-173 kernel: Code: 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 55 48 89 e5 41 57 41 56 49 89 f6 41 55 41 54 53 4c 8b af 30 03 00 00 <49> 83 bd 90 58 00 00 00 0f 84 b0 03 00 00 80 bf 79 01 00 00 00 4d
Apr 08 10:34:28 mag-master-10-106-172-173 kernel: RSP: 0018:ff6b6aac2f207700 EFLAGS: 00010206
Apr 08 10:34:28 mag-master-10-106-172-173 kernel: ll header: 00000000: ff ff ff ff ff ff 00 0c 29 06 8b b8 08 06
Apr 08 10:34:28 mag-master-10-106-172-173 kernel:
Apr 08 10:34:28 mag-master-10-106-172-173 kernel: RAX: ffffffffc05fdae0 RBX: ff442f8285b49000 RCX: 0000000000000000
Apr 08 10:34:28 mag-master-10-106-172-173 kernel: RDX: 0000000000000001 RSI: ff6b6aac2f2079c8 RDI: ff442f8285b49000
Apr 08 10:34:28 mag-master-10-106-172-173 kernel: RBP: ff6b6aac2f207728 R08: ff6b6aac2f2079c8 R09: 0000000000000000
Apr 08 10:34:28 mag-master-10-106-172-173 kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ff442f82c6870000
Apr 08 10:34:28 mag-master-10-106-172-173 kernel: R13: 0000000000000000 R14: ff6b6aac2f2079c8 R15: ff442f8285b49000
Apr 08 10:34:28 mag-master-10-106-172-173 kernel: FS:  0000784ae6418c40(0000) GS:ff44303ecbb00000(0000) knlGS:0000000000000000
Apr 08 10:34:28 mag-master-10-106-172-173 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Apr 08 10:34:28 mag-master-10-106-172-173 kernel: CR2: 0000000000005890 CR3: 00000001f9f90003 CR4: 0000000000f71ef0
Apr 08 10:34:28 mag-master-10-106-172-173 kernel: PKRU: 55555554
Apr 08 10:34:28 mag-master-10-106-172-173 kernel: Call Trace:
Apr 08 10:34:28 mag-master-10-106-172-173 kernel:  <TASK>
Apr 08 10:34:28 mag-master-10-106-172-173 kernel:  xfrm_dev_policy_add+0xef/0x1d0
Apr 08 10:34:28 mag-master-10-106-172-173 kernel:  xfrm_policy_construct+0x15a/0x1c0 [xfrm_user]
Apr 08 10:34:28 mag-master-10-106-172-173 kernel:  xfrm_add_policy+0xd5/0x250 [xfrm_user]
Apr 08 10:34:28 mag-master-10-106-172-173 kernel:  xfrm_user_rcv_msg+0x2b9/0x390 [xfrm_user]
Apr 08 10:34:28 mag-master-10-106-172-173 kernel:  ? xa_load+0x87/0xf0
Apr 08 10:34:28 mag-master-10-106-172-173 kernel:  ? __pfx_xfrm_user_rcv_msg+0x10/0x10 [xfrm_user]
Apr 08 10:34:28 mag-master-10-106-172-173 kernel:  netlink_rcv_skb+0x5a/0x110
Apr 08 10:34:28 mag-master-10-106-172-173 kernel:  xfrm_netlink_rcv+0x35/0x50 [xfrm_user]
Apr 08 10:34:28 mag-master-10-106-172-173 kernel:  netlink_unicast+0x289/0x3d0
Apr 08 10:34:28 mag-master-10-106-172-173 kernel:  netlink_sendmsg+0x214/0x470
Apr 08 10:34:28 mag-master-10-106-172-173 kernel:  __sys_sendto+0x21b/0x230
Apr 08 10:34:28 mag-master-10-106-172-173 kernel:  __x64_sys_sendto+0x24/0x40
Apr 08 10:34:28 mag-master-10-106-172-173 kernel:  x64_sys_call+0x1b2d/0x25a0
Apr 08 10:34:28 mag-master-10-106-172-173 kernel:  do_syscall_64+0x7f/0x180
Apr 08 10:34:28 mag-master-10-106-172-173 kernel:  ? do_syscall_64+0x8c/0x180
Apr 08 10:34:28 mag-master-10-106-172-173 kernel:  ? do_syscall_64+0x8c/0x180
Apr 08 10:34:28 mag-master-10-106-172-173 kernel:  ? syscall_exit_to_user_mode+0x43/0x1e0
Apr 08 10:34:28 mag-master-10-106-172-173 kernel:  ? audit_reset_context+0x257/0x340
Apr 08 10:34:28 mag-master-10-106-172-173 kernel:  ? __audit_syscall_exit+0xd1/0x140
Apr 08 10:34:28 mag-master-10-106-172-173 kernel:  ? syscall_exit_work+0x12b/0x150
Apr 08 10:34:28 mag-master-10-106-172-173 kernel:  ? arch_exit_to_user_mode_prepare.isra.0+0x1a/0xe0
Apr 08 10:34:28 mag-master-10-106-172-173 kernel:  ? syscall_exit_to_user_mode+0x43/0x1e0
Apr 08 10:34:28 mag-master-10-106-172-173 kernel:  ? do_syscall_64+0x8c/0x180
Apr 08 10:34:28 mag-master-10-106-172-173 kernel:  ? arch_exit_to_user_mode_prepare.isra.0+0x95/0xe0
Apr 08 10:34:28 mag-master-10-106-172-173 kernel:  ? syscall_exit_to_user_mode+0x43/0x1e0
Apr 08 10:34:28 mag-master-10-106-172-173 kernel:  ? do_syscall_64+0x8c/0x180
Apr 08 10:34:28 mag-master-10-106-172-173 kernel:  ? do_syscall_64+0x8c/0x180
Apr 08 10:34:28 mag-master-10-106-172-173 kernel:  ? do_syscall_64+0x8c/0x180
Apr 08 10:34:28 mag-master-10-106-172-173 kernel:  ? irqentry_exit+0x43/0x50
Apr 08 10:34:28 mag-master-10-106-172-173 kernel:  ? exc_page_fault+0x94/0x1b0
Apr 08 10:34:28 mag-master-10-106-172-173 kernel:  entry_SYSCALL_64_after_hwframe+0x78/0x80
Apr 08 10:34:28 mag-master-10-106-172-173 kernel: RIP: 0033:0x784ae632c0d7
Apr 08 10:34:28 mag-master-10-106-172-173 kernel: Code: c7 c0 ff ff ff ff eb be 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 80 3d 75 ef 0d 00 00 41 89 ca 74 10 b8 2c 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 69 c3 55 48 89 e5 53 48 83 ec 38 44 89 4d d0
Apr 08 10:34:28 mag-master-10-106-172-173 kernel: RSP: 002b:00007ffd09fbe1f8 EFLAGS: 00000202 ORIG_RAX: 000000000000002c
Apr 08 10:34:28 mag-master-10-106-172-173 kernel: RAX: ffffffffffffffda RBX: 000056456520e5a0 RCX: 0000784ae632c0d7
Apr 08 10:34:28 mag-master-10-106-172-173 kernel: RDX: 00000000000000c4 RSI: 00007ffd09fbe310 RDI: 000000000000000b
Apr 08 10:34:28 mag-master-10-106-172-173 kernel: RBP: 00007ffd09fbe2a0 R08: 00007ffd09fbe258 R09: 000000000000000c
Apr 08 10:34:28 mag-master-10-106-172-173 kernel: R10: 0000000000000000 R11: 0000000000000202 R12: 00000000000000c4
Apr 08 10:34:28 mag-master-10-106-172-173 kernel: R13: 00007ffd09fbe258 R14: 00007ffd09fbe310 R15: 00000000000000cb
Apr 08 10:34:28 mag-master-10-106-172-173 kernel:  </TASK>
Apr 08 10:34:28 mag-master-10-106-172-173 kernel: Modules linked in: xfrm_interface xfrm6_tunnel tunnel6 btrfs blake2b_generic xor raid6_pq ufs qnx4 minix ntfs msdos jfs nls_ucs2_utils xfrm_user xfrm_algo ipt_REJECT nf_reject_ipv4 ipt_rpfilter xt_CT xt_set xt_multiport ip_set_hash_ip ip_set>
Apr 08 10:34:28 mag-master-10-106-172-173 kernel:  i2c_algo_bit
Apr 08 10:34:28 mag-master-10-106-172-173 kernel: IPv4: martian source 10.127.99.1 from 10.127.99.61, on dev internet
Apr 08 10:34:28 mag-master-10-106-172-173 kernel:  ipmi_ssif ses enclosure ccp
Apr 08 10:34:28 mag-master-10-106-172-173 kernel: ll header: 00000000: ff ff ff ff ff ff 00 50 56 9f eb da 08 06
2

Hello @rginotra ,

Thank you for posting your query on NVIDIA Community.

As per the MLNX_OFED 24.10-4.1.4.0 release notes, the kernel version currently in use 6.8.x on Ubuntu 24.04 is not part of the validated/tested support matrix for this OFED release.

In such configurations, behavior cannot be guaranteed.

We recommend using a kernel version that is listed as supported in the MLNX_OFED release notes to ensure a stable and validated setup.

Please refer to this link for more information : MLNX_OFED v24.10-4.1.4.0 support matrix

If you experience issues aligning the software stack per support matrix, a valid support Entitlement for the HCA in use will be needed to perform additional troubleshooting.
If there an active entitlement/support contract in place, please do not hesitate to open a support ticket by logging into the ESP Portal and submitting a new case.
For contracts, please reach out to Networking-Contracts@nvidia.com

Thanks,
NVEX Networking Technical Support Team