Slow speeds and PortXmitWait - Any tips?

Infrastructure & Networking Software And Drivers WinOF Driver
1

Hi Everyone,

I have a new setup using 3 windows server hosts running WinOF with dual port Connectx-2 cards. These connect to two 4036 switches (one port to each switch and two links between switches). Here is the netdiscover:

Topology file: generated on Thu Jun 30 10:32:28 2016

Initiated from node 0008f10500203b28 port 0008f10500203b28

vendid=0x8f1

devid=0x5a5a

sysimgguid=0x8f10500109553

switchguid=0x8f10500109552(8f10500109552)

Switch 36 “S-0008f10500109552” # “Mellanox 4036 # 4036-SW2” enhanced port 0 lid 6 lmc 0

[1] “S-0008f10500203b28”[1] # “Mellanox 4036 # 4036-SW1” lid 1 4xQDR

[2] “S-0008f10500203b28”[2] # “Mellanox 4036 # 4036-SW1” lid 1 4xQDR

[34] "H-0002c903004e445a"1 # “IGA-S2D1” lid 2 4xQDR

[35] "H-0008f104039a3c1c"2 # “IGA-S2D2” lid 5 4xQDR

[36] "H-0008f104039a4e3c"2 # “IGA-S2D3” lid 8 4xQDR

vendid=0x8f1

devid=0x5a5a

sysimgguid=0x8f10500203b29

switchguid=0x8f10500203b28(8f10500203b28)

Switch 36 “S-0008f10500203b28” # “Mellanox 4036 # 4036-SW1” enhanced port 0 lid 1 lmc 0

[1] “S-0008f10500109552”[1] # “Mellanox 4036 # 4036-SW2” lid 6 4xQDR

[2] “S-0008f10500109552”[2] # “Mellanox 4036 # 4036-SW2” lid 6 4xQDR

[34] "H-0002c903004e445a"2 # “IGA-S2D1” lid 3 4xQDR

[35] "H-0008f104039a3c1c"1 # “IGA-S2D2” lid 4 4xQDR

[36] "H-0008f104039a4e3c"1 # “IGA-S2D3” lid 7 4xQDR

vendid=0x2c9

devid=0x673c

sysimgguid=0x8f104039a4e3f

caguid=0x8f104039a4e3c

Ca 2 “H-0008f104039a4e3c” # “IGA-S2D3”

1 “S-0008f10500203b28”[36] # lid 7 lmc 0 “Mellanox 4036 # 4036-SW1” lid 1 4xQDR

2 “S-0008f10500109552”[36] # lid 8 lmc 0 “Mellanox 4036 # 4036-SW2” lid 6 4xQDR

vendid=0x2c9

devid=0x673c

sysimgguid=0x8f104039a3c1f

caguid=0x8f104039a3c1c

Ca 2 “H-0008f104039a3c1c” # “IGA-S2D2”

1 “S-0008f10500203b28”[35] # lid 4 lmc 0 “Mellanox 4036 # 4036-SW1” lid 1 4xQDR

2 “S-0008f10500109552”[35] # lid 5 lmc 0 “Mellanox 4036 # 4036-SW2” lid 6 4xQDR

vendid=0x2c9

devid=0x673c

sysimgguid=0x2c903004e445d

caguid=0x2c903004e445a

Ca 2 “H-0002c903004e445a” # “IGA-S2D1”

1 “S-0008f10500109552”[34] # lid 2 lmc 0 “Mellanox 4036 # 4036-SW2” lid 6 4xQDR

2 “S-0008f10500203b28”[34] # lid 3 lmc 0 “Mellanox 4036 # 4036-SW1” lid 1 4xQDR

4036-SW1(utilities)#

Everything us up and transmitting data, but I have two concerns. ntttcp tests are only getting 1500-1800 MB/Sec, I would expect a bit more from QDR even with overhead. Can anyone tell me if this is normal or what could be tuned to improve? Additionally ibqueryerrorsis showing an increasing increasing PortXmitWait cound on any sending interface while I test. From what I can see some increase here is normal, but these seem high.

PS C:> ibqueryerrors

Errors for “IGA-S2D3”

GUID 0x8f104039a4e3d port 1: [PortXmitWait == 1]

GUID 0x8f104039a4e3e port 2: [PortXmitWait == 541647202]

Errors for 0x8f10500203b28 “Mellanox 4036 # 4036-SW1”

GUID 0x8f10500203b28 port ALL: [PortXmitWait == 86266712]

GUID 0x8f10500203b28 port 1: [PortXmitWait == 33430055]

GUID 0x8f10500203b28 port 34: [PortXmitWait == 52836657]

Errors for 0x8f10500109552 “Mellanox 4036 # 4036-SW2”

GUID 0x8f10500109552 port ALL: [PortXmitWait == 2344169726]

GUID 0x8f10500109552 port 0: [PortXmitWait == 261]

GUID 0x8f10500109552 port 34: [PortXmitWait == 2344169465]

Errors for “IGA-S2D1”

GUID 0x2c903004e445b port 1: [PortXmitWait == 59]

1500MB/sec is not terrible, but it is closer to a 15gbps connection than the 32gbps that this should be. Anyone have any hints?

Thanks!

2

Hi Adam,

First I would like to explain what this “PortXmitWait” counters mean.

This is not an error counter, It is a very high-resolution counter that is incremented every internal clock tick of the device whenever there is a packet queued on an

output port, and this packet cannot be sent.

This can happen whenever there is some temporary congestion in the fabric. For instance, if two nodes are sending to one node you can see this counter increment, or if there

Is a node mismatch in the rates you can see this counter increment.

When the counter increments it does not mean that any packets dropped, Packets are just stored in the buffer of the port until they can be sent.

Regarding the performance improvement, I suggest to get the latest WinOf driver for your HCA,

including upgrading the Connectx-2 firmware level to the latest (2.9.1000).

This can be found on the following link:

http://www.mellanox.com/page/products_dyn?product_family=32&mtag=windows_sw_drivers http://www.mellanox.com/page/products_dyn?product_family=32&mtag=windows_sw_drivers

You should also follow the tuning recommendations for performance on windows on the following tuning guide:

http://www.mellanox.com/related-docs/prod_software/Performance_Tuning_Guide_for_Mellanox_Network_Adapters.pdf http://www.mellanox.com/related-docs/prod_software/Performance_Tuning_Guide_for_Mellanox_Network_Adapters.pdf

Thanks,

Karen.