Interoperability problems of the Linux iSER initiator with Windows target over RoCE

Hi All,

we are working on our StarWind iSER target for Windows.

Testing the target with Linux iSER initiator from RHEL we faced a strange problem:

It works perfectly in the IB mode, but in the RoCE mode any connection from Linux fails with status 0xC000009A at Accept() at Windows side.

The Linux kernel is 3.10.0-327.el7.x86_64 #1 SMP Thu Oct 29 17:29:29 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux with the recent MLNX drivers.

At Windows 2012R2 side we use the recent MLNX drivers v5.10 and the NDv2 API to access the HBA.

The cards are ConnectX-3 VPI dual port (MCX354A-QCBT). Updating of the firmware to the recent version 2.36.5000 has changed nothing.

The problem persists either with a direct link of HBAs and with a switch.

Our test Windows client connects to the target over RoCE without problems.

The Linux iSER initiator also works with the Linux iSER target over both IB/RoCE.

Is this a misconfiguration of our RoCE network or maybe there is some interoperability issue of RDMA stacks in Linux and Windows?

Thank you,


Thank you guys for the help and the ideas!

The problem with RoCEv1 was rather trivial and it is described in the WinOF User Manual v.5.10. → RoCE Options

We just needed to set the roce_mode = 5 to set the mode to “RoCE IP Based” to let it work with Linux RoCEv1 implementation.

What about standard utils where Linux is the client and Windows is the server?

Is there ucmtaose, rping, ib_read_bw, etc for Windows?

Are those tools working?

Hi Ophir,

recently we’ve managed to get Linux iSER initiator working with our target over a pair of directly linked ConnectX-4 EN 100Gbit cards with RoCEv2. But still can’t have it working with ConnectX-3 with RoCEv1.

It seems that the recent versions of Mellanox test tools for Windows (v5.10) and for Linux are completely incompatible.

Most of the Windows tools use the NetDirect API (they are named nd_*.exe) and can’t work with Linux tools that use the older APIs.

There is a couple of older test tools for Windows, but that ib_read_bw.exe says “This tool is no longer supported.”, other tools work only in IB, not EN…

Are there any interoperable test tools for Linux and Windows?

Did you try with Mellanox OFED? Is the firmware on linux and windows hosts the same? The RoCE version?

Did you use the same MTU? ( check WinOF release notes)