Hello, I want to run NVMe over RDMA target offload with:
1: a X86 PC;
2: 2 Mellanox cx-6 cards;
3: An arm server installed with Linux 6.1;
4: an Intel D4800x nvme ssd； which can set io command queues and completion queues in its CMB;
I want to do the NVMe over RDMA target offload, and I try to connect the test platform like this:
a CX-6 will installed in PC;
2: a CX-6 and the D4800x are installed in the arm server;
I want to set the D4800x io queues in its CMB, and map the io queues to CX6’ receive and send queue;
I have a few confusion about how to enable my case:
1: Is there any limitation about the Arm server and its linux kernel ,such as the arm server should support P2P and it should be in the "host bridge whitelist "?
2: Should the Linux kernel of the arm server support “p2pmem”?
3: Should the “nvme” driver will be loaded automatically when linux boot up, in which case, the D4800 IO queues will be setup in cmb automatically.
Nvidia NvmeOF offload don’t support CMB however we showed it as POC for selective customer.
Also Arm server is not official support as a NIC (We have Bluefield controller-DPU which base on Arm ) .
Thanks for your apply!
Then , I am wondering if it does not support CMB, then where are the NVMe IO queues loacated in target offload?
And how does peer-to-peer occur? The data is moved from NIC to NVMe directly or
rdma queues is directly mapped to NVMe queues inside NVMe, which are not in the host?
Hi
The Data move direct to the target DDR . In the target we handle three queue. I/O queue, completion queue and staging buffer . The p2p is between the host DDR and the target DDR.
I/O moves directly from initiator DDR to target DDR .NIC FW “offload” will take control on the PCI and move the data from target DDR to the SSD and opposite. In this way the CPU is “offload”
In the nvme connect you have -i parameter . If is not set (default) is per core (the minimum between initiator and target) . If is set (ex -I =8) you have 8 queues
So, when I use “nvme connect …” to create NVMe IO queues;
How does target NVMe setup its IO queues， and where does it create the queues?
I’ve read the “NVMe over fabric” spec, there is no answer.
If is without -i parameter, the default is the queue per core. If the number of core in the targe/initator is not equal it is the minimum between initiator/target
So, Do you mean：
1） the “nvme connect …" command will not acturelly create IO queues? It just use the IO Queues that NVMe driver created when it setup？and “nvme connect …” just do something like “mapping”?
The nvme driver don’t set up queue in case of fabric (NVMeOF). The nvme connect setup the queues per core as default and per -i parameter if you set this parameter.
