Is there any software tool or benchmark tools for PCIe performance check?

[jeff_jein]

Is there any software tool or benchmark tools for PCIe performance check ?

[kayccc]

Many developers use iperf3 for PCIE network card performance testing, not sure if this meets your requirement.
You may also google to find some references.

[linuxdev]

This is not performance measurement per se, but do take a look at the output of “sudo lspci -vvv”. This will tell you about speed (version) capabilities in GT/s; you can then compare this to the actual GT/s setting. You will at least know if your PCIe card is actually running at its max spec. If you are thinking it is PCIe v3, but it is really running at PCIe v2, then your testing won’t really tell you what you want to know.

[jeff_jein]

Could you guide me how to know the speed(version) capabilities in GT/s from log file?
sudo lspci -vvv.log (45.6 KB)

[linuxdev]

Which device? There can be more than one device associated with a single PCIe slot, and it can be useful to see a tree view with “lspci -t”. This also means there might be a chain of devices to reach the device in question (and then your benchmark is the weakest point in the chain).

As an example, you might be looking at:
0000:01:00.0 Ethernet controller: Aquantia Corp. Device 04c0 (rev 03)

This device is at PCI slot “0000:01:00.0”. We could limit our query to that exact device:
sudo lspci -s 0000:01:00.0 -vvv

Searching through that for “GT/s” to find various speed listings, you will notice either a “capability” or a “status”. Capability is how fast that link can work (this does not include overhead, actual throughput is always lower), and how fast that link is actually running. So for the above device, this is of interest:

LnkCap:	Port #0, Speed 16GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <4us, L1 <64us
...
LnkCtl2: Target Link Speed: 16GT/s, EnterCompliance- SpeedDis-
...
Capabilities: [198 v1] Physical Layer 16.0 GT/s <?>

I also see an error, but this might not really be an error:

pcilib: sysfs_read_vpd: read failed: Input/output error

Regarding that error, there is an attempt to read some specification regarding this device. It could be that this specification does not apply to your device. It could be that there is some sort of permission error, e.g., due to the driver not exporting something to the sysfs (the files in “/sys” are being read by lspci). Maybe there is a driver associated with this file and the hardware for creating this does not exist on your PCIe card. Don’t know. If it really is a problem, then the driver might be incorrect for the hardware, or the hardware might be failing, or some other error is causing hardware read issues. I don’t see Advanced Error Reporting (AER) capability on this card, so it isn’t possible to conclude much about this (to see this in action, search for “AERCap”; if the first error pointer is 00, NULL, then there are no reported errors in the linked list of errors).

The important part on this one node is that the link status (“LnkSta”, the actual operating speed standard) is the same as the link capability. This node is running at the full speed it is capable of. Signal quality is probably good.

Note that raw bit throughput for PCIe v1 is 2.5 GT/s, for v2 it is 5 GT/s, for v3 it is 8 GT/s, and for v4 it is 16 GT/s. v5 is 32 GT/s. This node is running at a raw bit rate from PCIe gen. 4.

You can go through the chain of devices, where appropriate as seen from the tree view, and see if each is doing well. Anything leading to this particular device is working well, but because a PCIe card might have more than one device, e.g., if it were dual ethernet controllers, then one device could work and the other could fail or run at slower speeds. Also, if AER is present (it is optional), always check if the first error pointer is NULL (0x00), which means no outstanding uncorrected errors.