Hi all,
I am new to the Mellanox community and would appreciate some help/advice. I have 2 Connectx-3 adapters (MCX353A-FCBT) between two systems and am not getting the speeds I believe I should be getting. I am utilizing ethernet mode for both cards, and I want to use RoCE.
- The first system is a Dell R620 with 2 x E5-2660 CPU’s, 192gb of RAM, and 2 * 1TB 850 EVO SSD’s in RAID 1.
- The second system is a custom built ITX system with an i7-8700k, 32gb RAM, 1 * Samsung 970 evo M.2 NVMe drive, and 1 * 1TB 850 EVO SSD.
Both systems are dual boot, so I can use Windows Server 2016 or Linux (CentOS 7) at any given point.
For Windows, I made sure that SMB was enabled on both cards through powershell and made sure that RDMA capable and the RSS capable were true. Afterwards, I then followed the Mellanox guide to configuring RoCE below:
The only thing that was changed was to remove the ETS bandwidth limiter, as it isn’t necessary. Everything else was followed and changed where necessary (i.e: different name for the card in Windows).
For Linux, I had followed the steps on redhat’s site to configuring RoCE (I’m using CentOS 7 for both systems, so the steps are the same):
Networking Guide Red Hat Enterprise Linux 7 | Red Hat Customer Portal
However, I get the same results as I would in Windows.
For the card info:
- Both cards are the same model number and have the same firmware (2.42.500)
- Both are in Ethernet mode
- Both are in PCIe 3.0 x8 slots
I can provide any outputs necessary from commands/screenshots of any testing that can be done, so long as I know what I need to provide. I’m just at a standstill and am trying to figure out where the problem lies or if I have a bottleneck somewhere.
A help is greatly appreciated.
Thank you in advance.