Is this Normal? Mellanox ConnectX-DX 6 slow performance

Infrastructure & Networking Adapters and Cables Ethernet Adapter Cards
1

Hello guys,

i have two servers with 1 Mellanox ConnectX-DX 6 with 2x 100Gbits Ports each. So when i connect them via Crossover and test each port alone then i get around 99Gbits for each port. But when i bench both ports in parallel then i get only 130Gbits for both

iperf -c 198.168.10.10 -t 30 -p 4321 -P 16 -w 2M
And for bidirectional around 180 Gbits/sec for

I already updated the firmware and did many tunings.

So basicly i just wanna know if this is normal. In theory i expected having around 200Gbits for non-bidirectional and 400Gbits for bidirectional (because of fullduplex).
saturn_mellanox_summary.pdf (2.0 MB)

What are you experiences with that? So if this is not normal i will invest more time to fix the issue.

I have attached the summary from Mellanox/linux-sysinfo-snapshot

BR

2

You need turn CPU max performance and PCIE BW enough, and bind iperf to local numa cpu core.

3

cpupower frequency-set --governor performance already set.
And also tried with setting numa node for iperf

What do you exactly mean with PCI BW?
So it is using x16 Lanes on PCIE 4

4

x16 gen4 should enough for 2 x 100GE

5

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.