Hi, we ran into some issues while using the 4x10Gbit/s X710PCIe ethernet card with optical port. We got a rate of 39.5G+ when using iperf2 between two PCs, but only got 36.5G+ between an Orin and a PC. We don’t know what causes such a difference. We did such tests as following:
1, Optical port PCIe card, between pc and orin, got 36.5G+;
2, Optical port PCIe card, between two PCs, got 39.5G+;
3, Another 4x10Gbit/s x710 PCIe card with electrical port, between pc and orin, got 38.5+G, which can also meet our requirements in terms of rate, but we want to use optical port if possible;
4, electrical port PCIe card, between two PCs, also got 39.5G+;
What factors could have led to this result? And how can we increase the rate when using our optical port PCIe card? We would appreciate if got any suggestion.
You might find this of interest:
https://forums.developer.nvidia.com/t/network-packet-loss-for-xavier-nx-vs-orin-nx/253466/2
I’m guessing that if you examine “/proc/interrupts” that you’ll find all of the hardware IRQs for your hardware/driver will be running on CPU0. I’m not positive about IRQ routing on an Orin though, so you could try giving the driver its own core. Beware that even if hardware IRQs cannot route to another core, that you can still set affinity there…but the scheduler will migrate it back to CPU0 if that isn’t allowed. After changing affinity you might see if the driver now shows on both CPU0 and some other chosen core (indicating the scheduler migrated it back).
Orin is a beast in terms of embedded processing power, but it still has less bandwidth than a desktop PC, so it is also possible you are being limited by the core in cases where DMA is not used. At some point though you will get to things like checksums, and if the core itself is computing that, then that part is something you can’t get around without a faster core. Many of the higher end NICs compute their own checksums, but might need some sort of driver or configuration to enable it. I know nothing about your NIC, but you might specifically look up what configuration options are available.
Yes, a PCIe device could probably be put on its own IRQ. What I would try is putting this entirely on the last CPU core, and running at max performance mode and maxed out clock settings. Compare the performance with no change from default affinity.