Hello!
We have 2 Mellanox MCX416A-CCAT(100G Ethernet NIC Card) installed on two Dell Server R720xd. We have used iperf software to generate traffic from server A to server B (6 parallel thread). The maximum achievable bandwidth is near 64Gb/s. My question is why we cannot achieve 100Gb/s bandwidth?
We have the latest driver version and the NICs have been optimised to use multiple cpus.
Here is our complementary information for debugging
Thanks!
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Cable:
- TE QSFP28 26AWG 3m
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Server:
- 2×INTEL XEON 8CORE E5-2690 2.90GHZ 20MB L3 CACHE PROCESSORS
- 128 GB (4×32) MEMORY
- PCIE3x16lanes
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
NIC:
- 2x MCX416A-CCAT
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
OS:
Linux 3.11.0-22-generic #38~precise1-Ubuntu SMP Fri May 16 20:47:57 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
*-network:0
description: Ethernet interface
product: MT27620 Family
vendor: Mellanox Technologies
physical id: 0
bus info: pci@0000:05:00.0
logical name: eth12
version: 00
serial: e4:1d:2d:e9:81:9e
width: 64 bits
clock: 33MHz
capabilities: bus_master cap_list ethernet physical fibre
configuration: autonegotiation=off broadcast=yes driver=mlx5_core driverversion=3.0-1.0.1 (03 Mar 2015) duplex=full firmware=12.100.6440 ip=192.168.12.112 latency=0 link=yes multicast=yes
resources: irq:48 memory:d0000000-d1ffffff
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Settings for eth12:
Supported ports: [ FIBRE ]
Supported link modes: Not reported
Supported pause frame use: Symmetric Receive-only
Supports auto-negotiation: No
Advertised link modes: Not reported
Advertised pause frame use: No
Advertised auto-negotiation: No
Speed: 100000Mb/s
Duplex: Full
Port: Direct Attach Copper
PHYAD: 0
Transceiver: internal
Auto-negotiation: off
Cannot get wake-on-lan settings: Operation not permitted
Link detected: yes
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
ethtool -i eth12:
driver: mlx5_core
version: 3.0-1.0.1 (03 Mar 2015)
firmware-version: 12.100.6440
bus-info: 0000:05:00.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
ifconfig:
eth12 Link encap:Ethernet HWaddr …
inet addr:192.168.12.112 Bcast:192.168.12.255 Mask:255.255.255.0
inet6 addr: fe80::e61d:2dff:fee9:819e/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1
RX packets:38207802 errors:0 dropped:0 overruns:0 frame:0
TX packets:427379787 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:2292477406 (2.2 GB) TX bytes:3849436789889 (3.8 TB)
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
05:00.0 Network controller [0207]: Mellanox Technologies MT27620 Family
Subsystem: Mellanox Technologies Device 0008
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 48
Region 0: Memory at d0000000 (64-bit, prefetchable) [size=32M]
Capabilities:
Kernel driver in use: mlx5_core
Kernel modules: mlx5_core
Designation: PCI6
Type: x16 PCI Express 3
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Discovered irqs for eth12: 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222
Assign irq 191 core_id 15
Assign irq 192 core_id 16
Assign irq 193 core_id 17
Assign irq 194 core_id 18
Assign irq 195 core_id 15
Assign irq 196 core_id 16
Assign irq 197 core_id 17
Assign irq 198 core_id 18
Assign irq 199 core_id 15
Assign irq 200 core_id 16
Assign irq 201 core_id 17
Assign irq 202 core_id 18
Assign irq 203 core_id 15
Assign irq 204 core_id 16
Assign irq 205 core_id 17
Assign irq 206 core_id 18
Assign irq 207 core_id 15
Assign irq 208 core_id 16
Assign irq 209 core_id 17
Assign irq 210 core_id 18
Assign irq 211 core_id 15
Assign irq 212 core_id 16
Assign irq 213 core_id 17
Assign irq 214 core_id 18
Assign irq 215 core_id 15
Assign irq 216 core_id 16
Assign irq 217 core_id 17
Assign irq 218 core_id 18
Assign irq 219 core_id 15
Assign irq 220 core_id 16
Assign irq 221 core_id 17
Assign irq 222 core_id 18
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
cpuinfo: 32 cpu
processor : 31
vendor_id : GenuineIntel
cpu family : 6
model : 45
model name : Intel(R) Xeon(R) CPU E5-2690 0 @ 2.90GHz
stepping : 7
microcode : 0x710
cpu MHz : 2900.109
cache size : 20480 KB
physical id : 1
siblings : 16
core id : 7
cpu cores : 8
apicid : 47
initial apicid : 47
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx lahf_lm ida arat xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips : 5801.33
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Thanks!