Setting up NVMe-oF Target Offload: Unable to Set `num_p2p_queues` Parameter

Infrastructure & Networking Software And Drivers Mellanox OFED
1

Hello Mellanox community,

I am trying to set up NVMe-oF target offload and ran into an issue with configuring the num_p2p_queues parameter. I followed the tutorial and some related posts but encountered the following problems:

Here’s what I’ve tried so far:

  1. Directly loading the module with: modprobe nvme num_p2p_queues=1
  2. Modifying /etc/modprobe.d/nvme.conf to include the parameter, running dracut -f, and then rebooting.
  3. Disabling IOMMU and repeating the above steps.

Despite these attempts, the following error appears in dmesg when reloading the NVMe kernel module:

[  228.658656] nvme: unknown parameter 'num_p2p_queues' ignored
[  228.659232] nvme nvme0: pci function 0000:5e:00.0
[  228.659786] nvme nvme1: pci function 0000:5f:00.0
[  228.660111] nvme nvme2: pci function 0000:60:00.0
[  228.660450] nvme nvme3: pci function 0000:61:00.0
[  228.670473] nvme nvme0: Shutdown timeout set to 10 seconds
[  228.675947] nvme nvme3: Shutdown timeout set to 10 seconds
[  228.675960] nvme nvme1: Shutdown timeout set to 10 seconds
[  228.675986] nvme nvme2: Shutdown timeout set to 10 seconds
[  228.694798] nvme nvme0: 72/0/0 default/read/poll queues
[  228.703078] nvme nvme1: 72/0/0 default/read/poll queues
[  228.714138] nvme nvme2: 72/0/0 default/read/poll queues
[  228.718780] nvme nvme3: 72/0/0 default/read/poll queues

It seems that the num_p2p_queues parameter is not being recognized or applied. Verifying the parameter confirms this:

$ cat /sys/module/nvme/parameters/num_p2p_queues
cat: /sys/module/nvme/parameters/num_p2p_queues: No such file or directory

Could anyone provide guidance on what might be missing or misconfigured? Is it necessary to reinstall Mellanox OFED, or is there another step I should take? Any help or hints would be greatly appreciated! Thank you!

I did reading on followings, but not so helpful to have nvmeof target offload:

2

NICs:

$ ibv_devinfo
hca_id:	mlx5_0
	transport:			InfiniBand (0)
	fw_ver:				16.35.3006
	node_guid:			b859:9f03:0012:168c
	sys_image_guid:			b859:9f03:0012:168c
	vendor_id:			0x02c9
	vendor_part_id:			4119
	hw_ver:				0x0
	board_id:			MT_0000000010
	phys_port_cnt:			1
		port:	1
			state:			PORT_ACTIVE (4)
			max_mtu:		4096 (5)
			active_mtu:		4096 (5)
			sm_lid:			5
			port_lid:		1
			port_lmc:		0x00
			link_layer:		InfiniBand

hca_id:	mlx5_1
	transport:			InfiniBand (0)
	fw_ver:				16.35.2000
	node_guid:			b859:9f03:0012:0e44
	sys_image_guid:			b859:9f03:0012:0e44
	vendor_id:			0x02c9
	vendor_part_id:			4119
	hw_ver:				0x0
	board_id:			MT_0000000010
	phys_port_cnt:			1
		port:	1
			state:			PORT_ACTIVE (4)
			max_mtu:		4096 (5)
			active_mtu:		4096 (5)
			sm_lid:			5
			port_lid:		11
			port_lmc:		0x00
			link_layer:		InfiniBand

OS:

$ uname -a
Linux 6.6.1-zabbly+ #ubuntu22.04 SMP PREEMPT_DYNAMIC Fri Nov 10 19:28:30 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

MOFED:

$ ofed_info
MLNX_OFED_LINUX-5.9-0.5.6.0 (OFED-5.9-0.5.6):

clusterkit:
/sw/release/mlnx_ofed/IBHPC/MLNX_OFED_LINUX-5.9-0.5.5/SRPMS/clusterkit-1.8.429-1.59055.src.rpm

dapl:
/sw/release/mlnx_ofed/IBHPC/MLNX_OFED_LINUX-5.9-0.5.5/SRPMS/dapl-2.1.10.1.mlnx-OFED.4.9.0.1.5.59055.src.rpm

dpcp:
/sw/release/mlnx_ofed/IBHPC/MLNX_OFED_LINUX-5.9-0.5.5/SRPMS/dpcp-1.1.38-1.59055.src.rpm

dump_pr:
/sw/release/mlnx_ofed/IBHPC/MLNX_OFED_LINUX-5.9-0.5.5/SRPMS/dump_pr-1.0-1.MLNX20230110.gac314ef.59055.src.rpm

hcoll:
/sw/release/mlnx_ofed/IBHPC/MLNX_OFED_LINUX-5.9-0.5.5/SRPMS/hcoll-4.8.3221-1.59055.src.rpm

ibarr:
/sw/release/mlnx_ofed/IBHPC/MLNX_OFED_LINUX-5.9-0.5.5/SRPMS/ibarr-0.1.3-1.59055.src.rpm

ibdump:
/sw/release/mlnx_ofed/IBHPC/MLNX_OFED_LINUX-5.9-0.5.5/SRPMS/ibdump-6.0.0-1.59055.src.rpm

ibsim:
/sw/release/mlnx_ofed/IBHPC/MLNX_OFED_LINUX-5.9-0.5.5/SRPMS/ibsim-0.12-1.59055.src.rpm

ibutils2:
/sw/release/mlnx_ofed/IBHPC/MLNX_OFED_LINUX-5.9-0.5.5/SRPMS/ibutils2-2.1.1-0.159.MLNX20230118.g1214769c.59055.src.rpm

iser:
/sw/release/mlnx_ofed/IBHPC/MLNX_OFED_LINUX-5.9-0.5.5/SRPMS/iser-5.9-OFED.5.9.0.5.4.1.src.rpm

isert:
/sw/release/mlnx_ofed/IBHPC/MLNX_OFED_LINUX-5.9-0.5.5/SRPMS/isert-5.9-OFED.5.9.0.5.4.1.src.rpm

kernel-mft:
mlnx_ofed_mft/kernel-mft-4.23.0-104.src.rpm

knem:
/sw/release/mlnx_ofed/IBHPC/MLNX_OFED_LINUX-5.9-0.5.5/SRPMS/knem-1.1.4.90mlnx1-OFED.4.9.6.0.0.1.src.rpm

libvma:
/sw/release/mlnx_ofed/IBHPC/MLNX_OFED_LINUX-5.9-0.5.5/SRPMS/libvma-9.8.1-1.src.rpm

libxlio:
/sw/release/mlnx_ofed/IBHPC/MLNX_OFED_LINUX-5.9-0.5.5/SRPMS/libxlio-2.1.4-1.src.rpm

mlnx-dpdk:
/sw/release/mlnx_ofed/IBHPC/MLNX_OFED_LINUX-5.9-0.5.5/SRPMS/mlnx-dpdk-22.11.0-0.0.8.59055.src.rpm

mlnx-en:
/sw/release/mlnx_ofed/IBHPC/MLNX_OFED_LINUX-5.9-0.5.5/SRPMS/mlnx-en-5.9-0.5.5.0.g8e3d458.src.rpm

mlnx-ethtool:
/sw/release/mlnx_ofed/IBHPC/MLNX_OFED_LINUX-5.9-0.5.5/SRPMS/mlnx-ethtool-6.0-1.59055.src.rpm

mlnx-iproute2:
/sw/release/mlnx_ofed/IBHPC/MLNX_OFED_LINUX-5.9-0.5.5/SRPMS/mlnx-iproute2-6.0.0-1.59055.src.rpm

mlnx-nfsrdma:
/sw/release/mlnx_ofed/IBHPC/MLNX_OFED_LINUX-5.9-0.5.5/SRPMS/mlnx-nfsrdma-5.9-OFED.5.9.0.5.4.1.src.rpm

mlnx-nvme:
/sw/release/mlnx_ofed/IBHPC/MLNX_OFED_LINUX-5.9-0.5.5/SRPMS/mlnx-nvme-5.9-OFED.5.9.0.5.4.1.src.rpm

mlnx-ofa_kernel:
/sw/release/mlnx_ofed/IBHPC/MLNX_OFED_LINUX-5.9-0.5.5/SRPMS/mlnx-ofa_kernel-5.9-OFED.5.9.0.5.5.1.src.rpm

mlnx-tools:
/sw/release/mlnx_ofed/IBHPC/MLNX_OFED_LINUX-5.9-0.5.5/SRPMS/mlnx-tools-5.2.0-0.59055.src.rpm

mlx-steering-dump:
/sw/release/mlnx_ofed/IBHPC/MLNX_OFED_LINUX-5.9-0.5.5/SRPMS/mlx-steering-dump-1.0.0-0.59055.src.rpm

mpi-selector:
/sw/release/mlnx_ofed/IBHPC/MLNX_OFED_LINUX-5.9-0.5.5/SRPMS/mpi-selector-1.0.3-1.59055.src.rpm

mpitests:
/sw/release/mlnx_ofed/IBHPC/MLNX_OFED_LINUX-5.9-0.5.5/SRPMS/mpitests-3.2.20-de56b6b.59055.src.rpm

mstflint:
/sw/release/mlnx_ofed/IBHPC/MLNX_OFED_LINUX-5.9-0.5.5/SRPMS/mstflint-4.16.1-2.59055.src.rpm

multiperf:
/sw/release/mlnx_ofed/IBHPC/MLNX_OFED_LINUX-5.9-0.5.5/SRPMS/multiperf-3.0-3.0.59055.src.rpm

ofed-docs:
docs.git mlnx_ofed-4.0
commit 3d1b0afb7bc190ae5f362223043f76b2b45971cc

openmpi:
/sw/release/mlnx_ofed/IBHPC/MLNX_OFED_LINUX-5.9-0.5.5/SRPMS/openmpi-4.1.5rc2-1.59055.src.rpm

opensm:
/sw/release/mlnx_ofed/IBHPC/MLNX_OFED_LINUX-5.9-0.5.5/SRPMS/opensm-5.14.0.MLNX20230123.8bab53ab-0.1.59055.src.rpm

openvswitch:
/sw/release/mlnx_ofed/IBHPC/MLNX_OFED_LINUX-5.9-0.5.5/SRPMS/openvswitch-2.17.6-1.59055.src.rpm

perftest:
/sw/release/mlnx_ofed/IBHPC/MLNX_OFED_LINUX-5.9-0.5.5/SRPMS/perftest-4.5-0.20.gac7cca5.59055.src.rpm

rdma-core:
/sw/release/mlnx_ofed/IBHPC/MLNX_OFED_LINUX-5.9-0.5.5/SRPMS/rdma-core-59mlnx44-1.59055.src.rpm

rshim:
/sw/release/mlnx_ofed/IBHPC/MLNX_OFED_LINUX-5.9-0.5.5/SRPMS/rshim-2.0.6-18.g955dbef.src.rpm

sharp:
/sw/release/mlnx_ofed/IBHPC/MLNX_OFED_LINUX-5.9-0.5.5/SRPMS/sharp-3.2.0.MLNX20230122.a97f1d1c-1.59055.src.rpm

sockperf:
/sw/release/mlnx_ofed/IBHPC/MLNX_OFED_LINUX-5.9-0.5.5/SRPMS/sockperf-3.10-0.git5ebd327da983.59055.src.rpm

srp:
/sw/release/mlnx_ofed/IBHPC/MLNX_OFED_LINUX-5.9-0.5.5/SRPMS/srp-5.9-OFED.5.9.0.5.4.1.src.rpm

ucx:
/sw/release/mlnx_ofed/IBHPC/MLNX_OFED_LINUX-5.9-0.5.5/SRPMS/ucx-1.15.0-1.59055.src.rpm

xpmem:
/sw/release/mlnx_ofed/IBHPC/MLNX_OFED_LINUX-5.9-0.5.5/SRPMS/xpmem-2.6.4-1.59055.src.rpm


Installed Packages:
-------------------
ii  dapl2-utils                                   2.1.10.1.mlnx-OFED.4.9.0.1.5.59056                    amd64        Utilities for use with the DAPL libraries
ii  dpcp                                          1.1.38-1.59056                                        amd64        Direct Packet Control Plane (DPCP) is a library to use Devx
ii  dump-pr                                       1.0-1.MLNX20230110.gac314ef.59056                     amd64        Dump PathRecord Plugin
ii  hcoll                                         4.8.3221-1.59056                                      amd64        Hierarchical collectives (HCOLL)
ii  ibacm                                         59mlnx44-1.59056                                      amd64        InfiniBand Communication Manager Assistant (ACM)
ii  ibarr:amd64                                   0.1.3-1.59056                                         amd64        Nvidia address and route userspace resolution services for Infiniband
ii  ibdump                                        6.0.0-1.59056                                         amd64        Mellanox packets sniffer tool
ii  ibsim                                         0.12-1.59056                                          amd64        InfiniBand fabric simulator for management
ii  ibsim-doc                                     0.12-1.59056                                          all          documentation for ibsim
ii  ibutils2                                      2.1.1-0.159.MLNX20230118.g1214769c.59056              amd64        OpenIB Mellanox InfiniBand Diagnostic Tools
ii  ibverbs-providers:amd64                       59mlnx44-1.59056                                      amd64        User space provider drivers for libibverbs
ii  ibverbs-utils                                 59mlnx44-1.59056                                      amd64        Examples for the libibverbs library
ii  infiniband-diags                              59mlnx44-1.59056                                      amd64        InfiniBand diagnostic programs
ii  knem-modules                                  1.1.4.90mlnx1-OFED.4.9.6.0.0.1.kver.5.15.0-71-generic amd64        kernel module for high-performance intra-node MPI communication for large messages
ii  libdapl-dev                                   2.1.10.1.mlnx-OFED.4.9.0.1.5.59056                    amd64        Development files for the DAPL libraries
ii  libdapl2                                      2.1.10.1.mlnx-OFED.4.9.0.1.5.59056                    amd64        The Direct Access Programming Library (DAPL)
ii  libibmad-dev:amd64                            59mlnx44-1.59056                                      amd64        Development files for libibmad
ii  libibmad5:amd64                               59mlnx44-1.59056                                      amd64        Infiniband Management Datagram (MAD) library
ii  libibnetdisc5:amd64                           59mlnx44-1.59056                                      amd64        InfiniBand diagnostics library
ii  libibumad-dev:amd64                           59mlnx44-1.59056                                      amd64        Development files for libibumad
ii  libibumad3:amd64                              59mlnx44-1.59056                                      amd64        InfiniBand Userspace Management Datagram (uMAD) library
ii  libibverbs-dev:amd64                          59mlnx44-1.59056                                      amd64        Development files for the libibverbs library
ii  libibverbs1:amd64                             59mlnx44-1.59056                                      amd64        Library for direct userspace use of RDMA (InfiniBand/iWARP)
ii  libibverbs1-dbg:amd64                         59mlnx44-1.59056                                      amd64        Debug symbols for the libibverbs library
ii  libopensm                                     5.14.0.MLNX20230123.8bab53ab-0.1.59056                amd64        Infiniband subnet manager libraries
ii  libopensm-devel                               5.14.0.MLNX20230123.8bab53ab-0.1.59056                amd64        Developement files for OpenSM
ii  librdmacm-dev:amd64                           59mlnx44-1.59056                                      amd64        Development files for the librdmacm library
ii  librdmacm1:amd64                              59mlnx44-1.59056                                      amd64        Library for managing RDMA connections
ii  mlnx-ethtool                                  6.0-1.59056                                           amd64        This utility allows querying and changing settings such as speed,
ii  mlnx-iproute2                                 6.0.0-1.59056                                         amd64        This utility allows querying and changing settings such as speed,
ii  mlnx-ofed-kernel-utils                        5.9-OFED.5.9.0.5.5.1.kver.5.15.0-71-generic           amd64        Userspace tools to restart and tune mlnx-ofed kernel modules
ii  mlnx-tools                                    5.2.0-0.59056                                         amd64        Userspace tools to restart and tune MLNX_OFED kernel modules
ii  mpitests                                      3.2.20-de56b6b.59056                                  amd64        Set of popular MPI benchmarks and tools IMB 2018 OSU benchmarks ver 4.0.1 mpiP-3.3 IPM-2.0.6
ii  mstflint                                      4.21.0-7                                              amd64        mstflint - Mellanox firmware burning tools
ii  openmpi                                       4.1.5rc2-1.59056                                      all          Open MPI
ii  opensm                                        5.14.0.MLNX20230123.8bab53ab-0.1.59056                amd64        An Infiniband subnet manager
ii  opensm-doc                                    5.14.0.MLNX20230123.8bab53ab-0.1.59056                amd64        Documentation for opensm
ii  perftest                                      4.5-0.20.gac7cca5.59056                               amd64        Infiniband verbs performance tests
ii  rdma-core                                     59mlnx44-1.59056                                      amd64        RDMA core userspace infrastructure and documentation
ii  rdmacm-utils                                  59mlnx44-1.59056                                      amd64        Examples for the librdmacm library
ii  rshim                                         2.0.8+debian-1~exp1~22.04.1                           amd64        Userspace (fuse) host driver for Mellanox BlueField devices
ii  sharp                                         3.2.0.MLNX20230122.a97f1d1c-1.59056                   amd64        SHArP switch collectives
ii  sockperf                                      3.7-1                                                 amd64        Network benchmarking utility for testing latency and throughput
ii  srptools                                      59mlnx44-1.59056                                      amd64        Tools for Infiniband attached storage (SRP)
ii  ucx                                           1.15.0-1.59056                                        amd64        Unified Communication X
3

Hi

Before you load the parameter (p2p) ,please remove the driver
modprobe -rv nvme
And load again

Thx