Decrease latency

I have three identical clusters using

mlx4_core: Mellanox ConnectX core driver v2.2-1 (Feb, 2014)

mlx4_en: Mellanox ConnectX HCA Ethernet driver v2.2-1 (Feb 2014)

<mlx4_ib> mlx4_ib_add: mlx4_ib: Mellanox ConnectX InfiniBand driver v2.2-1 (Feb 2014) 40G

My cards are single port

CA ‘mlx4_0’

CA type: MT4099

Number of ports: 1

Firmware version: 2.31.5050

Hardware version: 0

Node GUID: 0xf45214030057a450

System image GUID: 0xf45214030057a453

Port 1:

State: Active

Physical state: LinkUp

Rate: 40

Base lid: 3

LMC: 0

SM lid: 1

Capability mask: 0x02514868

Port GUID: 0xf45214030057a451

Link layer: InfiniBand

Two of my systems give me an average of .17ms latency. The third system is .38ms. There is no load on any of the systems when I am running the ibping test. Does anyone have an idea on what may be causing this or a suggestion on where/what to look at?

Thanks,

Dave

I would check BIOS settings of the third system? Could you check sections (2.4.2-2.4.5) of the performance tuning guide?

http://www.mellanox.com/related-docs/prod_software/Performance_Tuning_Guide_for_Mellanox_Network_Adapters.pdf http://www.mellanox.com/related-docs/prod_software/Performance_Tuning_Guide_for_Mellanox_Network_Adapters.pdf

Are all systems the same system architecture? Same slot for IB on motherboard all system?

Maybe also try unplug the cables from switch and start opensm on the node and test the direct connect (see might be the switch port?)

The MT4099 is the ConnectX-3 VPI HCA adapter, I believe it should get better latency.

Will do.

Thanks

Dave