In our lab we use CX6 NICs as directly-connected Ethernet link partners to various FPGA designs. The FPGA side is the DUT. The CX6 is sending and receiving packets to/from the FPGA DUT. The CX6 side is usually Ubuntu 22 with up-to-date Nvidia drivers and firmware. Scapy is one of many tools we use on the CX6 side to programmatically produce and consume packets. No surprise this works great, as it should, most of the time.
Recently we’ve come across this strange case where we are trying to send packets from the CX6 to the FPGA/DUT. We “see” the packets counted by the driver (eg counts from “ifconfig”). We “see” them leave in Wireshark. But are not seeing them arrive on the FPGA-side MAC. And yes, we see RX Link and the various PMA stats on the FPGA OK. And data flows fine the other direction from FPGA to CX6.
This is likely a defect in our FPGA RX we need to fix, but it begs a question: Are there lower-level stats we can look at on the CX6, MAC level and lower, so that we can prove the packets went out on the wire? Is there a go-to tool to use to check that a CX6 egress packet didn’t get discarded for some reason on the way to the wire?
Thanks in advance. We always seem to learn something here!
Shep Siegel