I am running ConnectX5-VPI cards in two Redhad 7.6 Linux boxes with direct fiber connection. When I run in InfiniBand mode I am am seeing ~92GBits/Sec consistent. When I switch to RoCE is see ~92Gbits on first run but then drops down to ~8GBits/sec after that. Resetting the card allows the 1st run to be ~92GBits/sec then back to ~8. A run exchanges QPairs, 1000 1MB RDMA sends, tears down QPairs. The code between IB and RoCE is virtually the same except when transitioning QP to RTR where I use LID for IB and GID for RoCE. I am not doing anything different during tear down.
Hello Michael,
Thank you for posting your inquiry to the Mellanox Community.
We see that your organization currently has a valid support contract with Mellanox.
As such, we’ve opened a Mellanox support ticket on your behalf.
We will continue to track this issue via our support portal, and will provide further information to you directly.
Thank you,
Mellanox Technical Support