RoCE dramatic loss in throughput.

I am running ConnectX5-VPI cards in two Redhad 7.6 Linux boxes with direct fiber connection. When I run in InfiniBand mode I am am seeing ~92GBits/Sec consistent. When I switch to RoCE is see ~92Gbits on first run but then drops down to ~8GBits/sec after that. Resetting the card allows the 1st run to be ~92GBits/sec then back to ~8. A run exchanges QPairs, 1000 1MB RDMA sends, tears down QPairs. The code between IB and RoCE is virtually the same except when transitioning QP to RTR where I use LID for IB and GID for RoCE. I am not doing anything different during tear down.

Hello Michael,

Thank you for posting your inquiry to the Mellanox Community.

We see that your organization currently has a valid support contract with Mellanox.

As such, we’ve opened a Mellanox support ticket on your behalf.

We will continue to track this issue via our support portal, and will provide further information to you directly.

Thank you,

Mellanox Technical Support