How to run NVMe over RDMA target offload

Hello, I want to run NVMe over RDMA target offload with:
1: a X86 PC;
2: 2 Mellanox cx-6 cards;
3: An arm server installed with Linux 6.1;
4: an Intel D4800x nvme ssd; which can set io command queues and completion queues in its CMB;

I want to do the NVMe over RDMA target offload, and I try to connect the test platform like this:

  1. a CX-6 will installed in PC;
    2: a CX-6 and the D4800x are installed in the arm server;

I want to set the D4800x io queues in its CMB, and map the io queues to CX6’ receive and send queue;

I have a few confusion about how to enable my case:
1: Is there any limitation about the Arm server and its linux kernel ,such as the arm server should support P2P and it should be in the "host bridge whitelist "?
2: Should the Linux kernel of the arm server support “p2pmem”?
3: Should the “nvme” driver will be loaded automatically when linux boot up, in which case, the D4800 IO queues will be setup in cmb automatically.


Nvidia NvmeOF offload don’t support CMB however we showed it as POC for selective customer.
Also Arm server is not official support as a NIC (We have Bluefield controller-DPU which base on Arm ) .

Best Regards

Thanks for your apply!
Then , I am wondering if it does not support CMB, then where are the NVMe IO queues loacated in target offload?
And how does peer-to-peer occur? The data is moved from NIC to NVMe directly or
rdma queues is directly mapped to NVMe queues inside NVMe, which are not in the host?

The Data move direct to the target DDR . In the target we handle three queue. I/O queue, completion queue and staging buffer . The p2p is between the host DDR and the target DDR.

Best Regards

Thanks for your clarification, so this p2p is not that one of PCIe, which moves data from NIC directly to CMB/DDR insid NVMe

I/O moves directly from initiator DDR to target DDR .NIC FW “offload” will take control on the PCI and move the data from target DDR to the SSD and opposite. In this way the CPU is “offload”

Best Regards


In the nvme connect you have -i parameter . If is not set (default) is per core (the minimum between initiator and target) . If is set (ex -I =8) you have 8 queues

Bet Regards


So, when I use “nvme connect …” to create NVMe IO queues;
How does target NVMe setup its IO queues, and where does it create the queues?
I’ve read the “NVMe over fabric” spec, there is no answer.

If is without -i parameter, the default is the queue per core. If the number of core in the targe/initator is not equal it is the minimum between initiator/target

Best Regards


So, Do you mean:
1) the “nvme connect …" command will not acturelly create IO queues? It just use the IO Queues that NVMe driver created when it setup?and “nvme connect …” just do something like “mapping”?

Best Regards

The nvme driver don’t set up queue in case of fabric (NVMeOF). The nvme connect setup the queues per core as default and per -i parameter if you set this parameter.

Best Regards

Hi gilh,

I found the discussion on target offload intriguing and have some follow-up questions.

Firstly, could you please elaborate on the statement “Nvidia NvmeOF offload don’t support CMB”? My understanding was that NVMeOF with CMB was supported, as indicated by Stephen Bates and Mellanox presentations [1,2,3]. They mentioned the approach involving NVMeOF from Mellanox CX5 and NVMe CMB device, suggesting support for NVMeOF offload with CMB.

Additionally, I’m curious about why you mentioned the number of queues from the -i parameter. How does the number of queues relate to target offload? Why is it necessary to consider the -i parameter instead of just using the default setting?




We have POC which show the capability of offload with CMB however is never got to production.
About the - i parameter, I just explain regard to num of queue. However, you can use the default.

Best Regards


Hi Gilh,

Thank you for explaining the queue parameter; it’s now clear to me.

However, I’m a bit confused about the POC versus production-ready status. From the slides and talks I referenced, they did present CMB-target-offloading experiments with performance measurements on Mellanox ConnectX. Could you clarify if these experiments are not publicly production-ready on Mellanox ConnectX NIC + current Linux?

It plan in the past (to make it production)
Nvidia Nic don’t support CMB with offload

Best Regards


1 Like

Thanks for the prompt reply. I look forward to seeing NVIDIA Mellanox NICs being production-ready for CMB target offloading in the future.