Confusion with Packet and Byte counters for RoCE over ConnectX-3

We are using RoCE over a CX3 (RHEL 6.4)

driver: mlx4_en

version: 2.1 (Apr 3 2013)

firmware-version: 2.11.500

We are confused by the RoCE packet and byte counters. Eg,

cat /sys/class/infiniband/mlx4_0/ports/1/counters_ext/port_xmit_packets_64

cat /sys/class/infiniband/mlx4_0/ports/1/counters_ext/port_rcv_data_64

The packet counters seem be about a factor of 4 smaller… Ie, if we multiple the number by 4, it seems to match the switch counters. The byte counters have a larger multiplication factor.

Are we interpreting these counters correctly?

Thanks,

Dan

Message was edited by: Brian Sparks Changed “CX3” to ConnectX-3 to ensure folks knew which product.

The man page for perfquery states that only the data counters are divided by 4. It doesn’t say anything about packets counters.

Note: In PortCounters, PortCountersExtended, PortXmitDataSL, and PortRcvDataSL, components that represent Data (e.g. PortXmitData and PortRcvData) indicate octets divided by 4 rather than just octets.

For RoCE adapters the counters do not count regular Ethernet frames, they only count Infiniband. However, the switch will count everything. So maybe that’s the reason why port_rcv_data is smaller than the switch counters?

Hi Yairi,

I am not sure I know what you mean… ethtool does not reports counters for packets that have been sent or received via RDMA.

thanks,

Dan

Thank you for the confirmation that the byte counters are a factor of 4. I suspect you are right about the packet counters as well… I will double check.

It’s too bad “perfquery” doesn’t work in a RoCE network. We are not running a subnet agent on this linux server. (Should we be running a subnet agent? But that would require a subnet manager somewhere, right?)

perfquery

ibwarn: [8397] mad_rpc_open_port: can’t open UMAD port ((null):0)

perfquery: iberror: failed: Failed to open ‘(null)’ port ‘0’

well… perfquery would only work for InfiniBand ports as it send InfiniBand MAD queries. you mentioned RoCE so i am guessing you use the cards in Ethernet mode… right?

Correct, the cards are in Ethernet mode.

It would be nice to have a tool to display counters for a RoCE interface.

currently whatever that is reported by ethtool (for standard Eth interface) is probably available.

you are correct. those are not going to know how many RoCE and how many other counters.

ethtool -S will provide you with bytes, packets for Rx and Tx data.

for Infiniband case, things are the same - the counters pulled out from the silicon are packet counters and you can’t really say to which ULP they associates.