Jetson TX1 and CycloneV FPGA with PCI memory mapped


I have a CycloneV FPGA connected to the Jetson TX1 with PCIe bus. A memory range (64MiB) of the CycloneV memory is mapped through the PCIe bus, which is accessible from the Jetson.
My application writes data from Jetson to the mapped memory. I found that sometimes the data transfer is delayed: the maximum measured delay was ~0.3 second.
How can I flush this memory, or make sure all the pending transactions have been completed? I tried to use fsync() and msync() without success.

I use Linux kernel 3.10.96 on the Jetson.

If I understand it correctly, your FPGA based end point has 64MB of internal memory which is being exposed to host system through BAR. Your application must be writing data from some location in Tegra’s system memory to BAR, which is typically carried out by CPU.
When you say 0.3 second delay, Is this the time difference between best case data copy time Vs worst case data copy time?
Since it is CPU that is doing the job of copying, it is possible that it gets pre-empted in some cases resulting in slightly higher copy times.

Hi Vidyas,

The delay means that the data appears in the FPGA memory later, after the memcpy() on Tegra finished.
It is very interesting that most of the data is copied immediately, but some small chunks are missing (the chunks are in 64-byte increments, containing the previous memory content, having before memcpy() call). The full data size was 6MiB (handled by memcpy()).

I analyzed the PCI transfers with SignalTap, and found that the whole memory content is copied continuously, but there are some chunks of previous memory content (was before memcpy(), 64 bytes, as mentioned above). I don’t understand this behavior, especially why the problematic size is 64 bytes.
Probably these wrong chunks will be re-sent later, but I could not catch these transfers. At least dumping the FPGA memory later (>0.3s) shows proper content.

If there is any cache between them, there must be some kind of flush system call. The standard ones do not work for me, as I mentioned before.

you mean to say that you see those 64-byte re-transfers over PCIe link? If so, it is possible that the link is bad and data got corrupted when they were sent first time. Also, the offset of this 64-byte data (that gets re-transmitted) is same always or it keeps changing?

Hi Vidyas,

Thanks for the answer!
The offset of the wrong 64-byte chunks are random and there are a lot of transfers without any wrong chunk. As far as i could, the PCI link is tested, and did not find any problem - at least the PCI signals do not show any error.
Unfortunately, i could not catch the re-transmission with SignalTap (could not make trigger for them), but i see the proper memory content a little later.