Low NVMe-oF Target Offload performance when size is small


I ran some NVMe-oF Target Offload tests using ConnectX-5 and compared with NVMe-oF results using the same environment.

Very low CPU utilization showed that target offload was done correctly, but target offload showed higher latency compared to NVMe-oF when the value size was smaller than 32K.

For the tests, I ran fio and the my connectX-5 is Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5]


You haven’t provided enough details about the setup and I would assume that you’ve used FIO with libio, that goes thought kernel, thus latency fluctuation is expected. Using different library that is based on the polling and not events is recommended. For best results use SPDK and run perf tool, that is part of SPDK.