Are you using Xavier DMA or some other ep DMA to initiate the reads?
An FPGA configured as a PCIe endpoint is performing the memory read transactions.
What is the memory controller configuration ? How about the DRAM freq? Is ECC enabled?
This is a standard 8GB or 16GB Xavier (the problem happens in both, but I’m focusing on the 8GB module right now). With the EMC frequency fixed at 1333 and everything else set to max in nvpmodel and jetson_clocks the problem occurs, albeit less frequently than at lower EMC frequencies.
What is the cpu +memory load bandwidth other than PCIe read? How much load is cpu reads?
I can easily aggravate the issue by lowering the EMC clock to 666 MHz and then using stress-ng --stream 2 (~15 GB/sec), which will contend with the PCIe writes enough for the PCIe stream to start missing deadlines periodically.
It’s not at all surprising that this kind of aggregate load is the maximum the bus can handle at this rate, and that’s the point. No matter what happens to the rest of the system, I’d like the PCIe transactions to take priority by some means, or perhaps like an ISOMMU client (such as the display) that seems to be resilient to this sort of thing.
What is your expected memory bandwidth and latency? which pcie controller is being used?
In these tests, I’m moving about 2 GB/sec over Gen 3 x4 on C5. The load on the host is going to be pretty unpredictable, so I just want to have some guarantee that PCIe traffic does not get contended with.
. These are DMA reads of a fixed location in host memory
What do you mean by “fixed location” here? Is the start location a fixed addr?
Yes, it’s a video framebuffer, with a fixed start and size that are not moving during these tests. The FPGA is streaming this data through a short fifo and out to its own display (hence the deadlines).
Thanks for your help,