ConnectX-6 connection with Mellanox ConnectX-4 is very slow to send files

Hello, my MT2894 Family [ConnectX-6 Lx] Ethernet Card is installed on my dell poweredge 7625 server with Redhat 8.8 operating system installed.

In my other Nas Server, MT27710 Family [ConnectX-4 Lx] Openmedia Vault (Debian 11 Linux) is installed on my installed device.

2 Server 10 gbe Cards are connected to HP Aruba 2930f 10gb Switch port with DAC cable. I made MTU 9000 on 2 sides, but the file transfer between the two servers is very slow, I can even say that I can not even do it, it waits for a long time. what is the way to overcome this problem? What should I check? I need your help in this matter.

The result I get with iperf3 between 2 servers is as follows, what could be the reason?

[ 5] 0.00-1.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 1.00-2.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 2.00-3.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 3.00-4.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 4.00-5.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 5.00-6.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 6.00-7.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 7.00-8.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 8.00-9.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 9.00-10.00 sec 0.00 Bytes 0.00 bits/sec


[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.04 sec 0.00 Bytes 0.00 bits/sec receiver

When I connected the Mellanox Cards directly to each other with a dac cable, the speed froze to normal. I need to make some adjustments to the 10gbe port on the Hp Aruba 2930F, but what exactly? I am waiting for support on this issue.

Hello alperaykut80,

Thank you for posting your inquiry on the NVIDIA Developer Forum - Infrastructure and Networking - Section.

Based on your last update, you isolated the issue to your switch as the throughput is as expected when connection is back-2-back.

Recommendation is to set the MTU on the switch port the same as on the adapter. In general MTU needs the be same from end-2-end.

As this switch is an HPE Aruba switch, we recommend to reach out to HPE for assistance in configuring the switch properly for achieving the maximum throughput.

Thank you and regards,
~NVIDIA Networking Technical Support

1 Like