I have a host machine running Ubuntu 22.04 with a ConnectX-5 (MCX515A-CCAT, firmware 16.35.4506-LTS) installed into a x16 Gen5 PCI slot.
A separate machine transmits a sequence of RDMA Write Only packets each with PMTU 1024 bytes. The packets populate a buffer on the host machine. This works correctly below about 400Mbps. Above about 3Gbps we see consistent packet drops on the host machine. I can confirm that rx_discards_phy reports exactly the number of packets dropped in the data
It is consistently the case that the first ~512kB of data is received correctly after which packets are regularly dropped. The attached image indicates the behavior (ignore the axis values). The zero values on the right hand side of the plot indicate the dropped packets. Observe that a bulk of packets are correctly received at first (left hand size of the plot)
My question is regards to the repeated use of the RDMA Write Only opcode (instead of RDMA Write First/Middle/Last). Does the ConnectX-5 support this? If not, do later cards support this? If this is supported, then any assistance that help understand why packets are dropped in this case would be much appreciated.