Maximizing IOPS for NVME emulation using Bluefield 3 SNAP

Infrastructure & Networking BlueField
1

Setup details:

Platform 1 (Storage Target):

  1. SSD
  2. ConnectX-6 VPI; 100Gb/s (HDR100; EDR IB and 100GbE); single-port QSFP56; PCIe3.0 x16 Adapter
  3. Optical cable
  4. Ubuntu 20.04 server

Configured the setup with details from ESPCommunity using Infiniband.

Platform 2: DPU with Host Server:

  1. NVIDIA BlueField-3 B3140L E-Series FHHL SuperNIC
    with parameters NVME_EMULATION_ENABLE=1,NVME_EMULATION_NUM_PF=1 for NVME emulation.
  2. Ubuntu 20.04 on Host OS

Configured SNAP with SNAP_RDMA_ZCOPY_ENABLE for single controller with example from SNAP Deployment - NVIDIA Docs for single controller and able to see the controller on the host and perform IO operations successfully on the SSD sitting on the storage target. I am looking to optimize for maximum iops and throughput .

Things tried for maximizing throughput:

  1. Switched NVME_EMU_provider = dpu instead of dpa . Received the error on host: NVMExpressPassThru: Timeout occurs for an NVMe command. NvmExpress: Failed to identify active namespace.
  2. Increased memory limits in doca_snap.yaml. No difference in IOPS.

Is there any other DPU configuration or SNAP configuration that would help increase / maximum throughput/IOPS for doing IO operations?

2

Hi,

Thanks for your question.
To reach maximum performance with SNAP default settings are good enough.

You can run the below tests:

N_CORES BS RWMIXREAD NUMJOBS QD
8 4096 100 31 512
8 4096 0 31 512

You may reach around 2.6MIOPS with Null device in the target

To test Max performance please use Null device in the Target, means DRAM (otherwise a single SSD may be a bottleneck in these tests)

If you’d like to debug a particular scenario, opening a new support case in Nvidia will be required (or just send an email to enterprisesupport@nvidia.com), and it will be handled according to support entitlement.

Best Regards,
Anatoly