Question regarding RNR NAK handling with a ConnectX-4

#1

Hello,

I ran into an issue with a simple configuration that uses the NVME-oF protocol between a Host server with a ConnectX-4 and our NVME-oF target connected via a 100G switch MLNX switch. Here are the detail on teh configuration and the issue:

Configuration:

Host:

  1. Linux kernel version:
  • root@host139# uname -a

Linux host139 4.8.0-22-generic #24-Ubuntu SMP Sat Oct 8 09:15:00 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

  • root@host139:# cat /etc/os-release

NAME=“Ubuntu”

VERSION=“16.10 (Yakkety Yak)”

ID=ubuntu

ID_LIKE=debian

PRETTY_NAME=“Ubuntu 16.10”

VERSION_ID=“16.10”

HOME_URL="http://www.ubuntu.com/ http://www.ubuntu.com/ "

SUPPORT_URL="http://help.ubuntu.com/ http://help.ubuntu.com/ "

BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/ http://bugs.launchpad.net/ubuntu/ "

PRIVACY_POLICY_URL="http://www.ubuntu.com/legal/terms-and-policies/privacy-policy http://www.ubuntu.com/legal/terms-and-policies/privacy-policy "

VERSION_CODENAME=yakkety

UBUNTU_CODENAME=yakkety

  • MLNX RNIC:

Product Name: CX415A - ConnectX-4 QSFP28

Part number: MCX415A-CCAT

FW Version: firmware-version: 12.16.1020

Switch:

MSN 2410

  • Host connected to a 100G switch port
  • Target connected to a 25G switch port

Target:

Target that support NVME-oF protocol

Summary of the issue:

  1. Host is running an FIO script that sends workload to a NVME-oF target across the MLNX switch to the target.
  2. FIO workload is 100% Writes – 4 Jobs / 32 Queue Depth / 8K Random Writes
  3. A few seconds into the test the Target sends the MLNX RNIC an RNR NAK with a “wait time” of 0.12ms for a SEND (NVME Write) frame from the Host. The Host has sent other SEND frames after the one that got NAKd which the target does not acknowledge as per the spec
  4. MLNX RNIC re-transmits the NAK’d frame after 1.5ms and that IO compltetes successfully
  5. However, the Host does not re-transmit any of the SEND frames that were sent after the one that got NAKd so the FIO job just hangs.

This seems to be a fundamental behavior of RNR NAK which I think the MLNX RNIC would handle – so I am hoping that there could be some configuration setting or something similar that either the target should have done or should have been configured on the Host for this to work.

Please let me know if there is any additional information is needed to understand the issue.

Thanks,

Narayan Ayalasomayajula

Kazan Networks

#2

Hi Narayan,

The support for “inbox” driver is provided by the OS vendor and not by Mellanox.

Did you test with our Mellanox MLNX_OFED driver?

Also, can you please emphasized what you are trying to accomplish and how you are determining that other frames that should be transmitted are not seen from the MLNX RNIC upon reception of an RNR NAK?

Sophie.

#3

Hi Sophie,

Thanks for the pointer to the Performance Tuning Guide. I had downloaded this guide a while back and have been following relevant recommendations. The specific question I had was with regards to the behavior of the MLNX RNIC on reception of an RNR NAK from the target. The MLNX RNIC does retransmit the frame in question but other frames that should be transmitted are not seen from the MLNX RNIC. My query primarily to see if there is anything that needs to be configured in the RNIC for this functionality to work.

Thanks,

Narayan

#4

Hi Narayan,

To inquire and have additional information for a support contract, please email "contracts@mellanox.com mailto:contracts@mellanox.com ".

In the meantime, we will review your inquiry posted in our Community website.

Sophie.

#5

Hi Narayan,

Can you please confirm if you are using our MLNX_OFED driver and if so, what release? (ofed_info -s or ofed_info | head -1).

Sophie.

#6

Hi Narayan,

Indeed running MLNX_OFED driver is not requirement though, the Inbox support is provided through the OS vendor and not through Mellanox.

Also please make sure you have consulted our Performance Tuning Guide for our HCA’s. (BIOS & OS settings).

The performance tests bandwidth & latency are performed against our driver.

Sophie.

#7

Hi Sophie,

I am not using the MLNX_OFED driver. The Linux 4.8.0-r22 kernel has native driver support for NVMEoF, so MLNX_OFED is not needed.

Thanks,

Narayan