How to determine tick duration for port_xmit_wait on ConnectX-6?

Infrastructure & Networking Software And Drivers Mellanox OFED
,
1

Hello,

According to the guide “Understanding mlx5 Linux Counters and Status Parameters” and other related resources, the port_xmit_wait counter measures the number of clock ticks during which the port had data to transmit, but was unable to do so due to insufficient credits or lack of arbitration.

As I understand it, this counter effectively accounts for the amount of time the port was stalled while data was available to send, a useful metric for diagnosing network congestion or flow control issues.

If the duration of a tick were known, it would be possible to convert port_xmit_wait into an actual time (e.g., seconds), which would allow comparison against job runtime and help evaluate whether the value is significant.

My test platform uses:

> lspci | grep InfiniBand
InfiniBand Controller: MT28908 Family [ConnectX-6]

> ibstatus | grep rate
rate:	         100 Gb/sec (2X HDR)

However, I have been unable to find any official documentation or method that specifies the tick duration for this adapter. The only source I’ve found so far is ChatGPT, which suggests that 1 tick = 4 ns, based on a presumed 250 MHz internal clock. While this value is mentioned in some sources, particularly for SDR adapters, I have found no reference regarding HDR.

My question:

Is there any official documentation, support article, or query method that defines or reveals the tick duration used for port_xmit_wait on a ConnectX-6 HDR adapter?

This would be extremely helpful for interpreting counter values in a meaningful way.

Thank you in advance for any clarification or guidance!

Best regards,

– G.

2

Hi,

Unfortunately, there doesn’t seem to be any official documentation mentioning this, as this is HW counter gets interpreted within the FW code to adhere to the IB spec.

In general,

The concept is this:

  • XmitWait is not a “latency” property, but a “BW loss” property.

  • Each tick (as explained in the IB spec) is a loss of a single Byte (8 bits)

  • The XmitWait HW counter counts each tick as 8 Bytes (64 bits) and the FW multiplies it by 8 to align to spec definition.

  • For example, to understand what exactly the counter means:

    • Assuming there are 2 samples in a 10 seconds diff, and the counter delta in this time frame was 12.5G (12,500,000,000)

    • The BW loss (in bits) for the full period is 12.5*8 = 100Gbps

    • The BW loss per sec is 100 / 10 = 10Gbps

I hope this helps clear things up a bit.

If you have any additional questions, please open a support case, and it will be handled based on entitlement.

Thanks,

Jonathan.