How to get port counters separately for each channel ?

Hello all

Are investigated with factory monitoring.

As is known each switch port is connected to other ports of this switch by full matrix ASIC (internal channel) and can be connected to another switch by cable (external channel).

Can I get port counters statistic separately for each internal and external channel by Mellanox-OFED ?

I think there may be a terminology issue here. When you write switch, do you mean some chassis based switch ? Chassis based switches (commonly anything larger than 36 or 38 4x switch ports) is a conglomeration of individual switches. Chassis based switches have spine cards (internal connectivity) and line cards (external connectivity). Internally connected switches are interconnected by etch between switches on a single PCB or by backplane traces. External ports are connected via cables of some sort. You should be able to see all this with ibnetdiscover. If the chassis based switch is supported by ibnetdiscover, you can even try -g option. Regardless of whether it’s an internal or external connection, you just need to find the right port numbers on the peer IB devices to query the port counters.

There are no standardized counters for traffic statistics by (source, destination) so that you can see specific counters only for traffic between say n1 port 1 and n3 port 1.

In terms of determining the loaded links in the fabric, this is doable by comparing the traffic counters for a link as compared to link rate as well as monitoring PortXmitDiscards and PortXmitWait counters which are indicate of congested links.

You can query individual ports using infiniband-diags perfquery or look at ibqueryerrors (–counters | --errors)

Sorry, my english is bad, maybe I was not able to clearly explain

I understand how to get the counters between two ports of two different switches

Can I get a counters between the two ports of the same switch (ports in the same switch connected by ASIC of THIS switch (connected by electric circuit path of THIS switch) ) ?

Sorry for unclear explain

Hi Serge,

AFAIK, there is no such option based on customer tools.

Hello Hal. Thanks for answer.

What you describe - this is clear.

Apparently I’m very bad explain. I try again to describe my problem.

By the switch i mean, for example, SX6025

If i have one (for example and for simplify) 36 ports switch SX6025 and three HCA (n1, n2, n3) connected to port1, port2, port3 in one switch (n1<->sw1.p1, n2<->sw1.p3, n3<->sw1.p3).

HCA | cable | switch1 ASIC

n1.p1 Xmt buf | cable | Rcv buf p1 - full 36 ports matrix communication env

Rcv buf | | Xmit buf

| |

n2.p1 Xmt buf | cable | Rcv buf p2 - full 36 ports matrix communication env

Rcv buf | | Xmit buf

| |

n3.p1 Xmt buf | cable | Rcv buf p3 - full 36 ports matrix communication env

Rcv buf | | Xmit buf

Global challenge: to identify the most loaded sections of the path in all of communication fabric.

I can get statistics of switch1.p1 by ibnetdiscover (PortXmitPkts, PortRcvPkts).

I want to get counters by the connection n1.p1 <-> switch1.p1.

But switch1.p1.PortXmitPkts, switch1.p1.PortRsvPkts may contain Pkts not only from n1.p1 ports but also have packets from switch1.p2.PortXmitPkts, switch1.p2.PortRsvPkts and switch1.p3.PortXmitPkts, switch1.p3.PortRsvPkts.

Do I have the ability to have counters by switch1.p1<->switch1.p2 and switch1.p1<->switch1.p3 ?

it is necessary for the accurate collection of counters all the way between n1<->sw1.p1<->n2 and

It need for correctly collect data from full path

I do exactly as you say

I had thought that there is perhaps there are still other ways

Thanks for your reply