I am trying to figure out if I create a PCIe Bus Master DMA endpoint in an FPGA and connect it to the Jetson TX2 via the PCIe gen2 4x bus will I be able to get this data into the GPU directly via DMA or do I have to go through the CPU complex?
The GPU and CPU share the same physical DRAM. Thus, if you can write to the DRAM, then you can write to the GPU.
However, it may be that the DRAM write signals will travel through the CPU bus interface, and it will certainly use bandwidth that the CPU is contending for. I don’t know if this qualifies for “going through the CPU complex.”
The GPU may also need some kind of cache invalidation from a driver to make sure it reads the freshest data.
Thanks for the reply. I agree with you from a hardware perspective. The Parker top level block diagram confirms that the CPU’s and GPU can access the DRAM through the System MMU as you have said. Do you know if this is supported in the CUDA API’s?
Yes, I believe this is supported in CUDA, although the way it’s supported is simply by making “mapping” and “locking” use the same memory buffer for GPU/CPU, rather than having a hidden copy step. (They still need to invalidate caches, of course.)
One thing to watch out for: There is also a separate PCI-express-domain memory management unit, which I don’t know much about. This unit makes PCI express memory addresses different from physical addresses, AND different from application/kernel level addresses. I know it can be turned off (because that worked around some driver that was having problems with this API) but if you’re bus mastering from the PCI Express side, you may need to worry about this.
At that point, you’d probably want to have a direct contact into NVIDIA developer relations / engineering support, rather than going through the bulletin board, if possible, because those bits are not well documented publicly, and having someone explain the system for 30 minutes would probably save a week of trial-and-error head-desk-banging.
I read a post describing issues with enabling the PCIe memory manager. Thanks for the heads up. Our company is in the Inception program so I should be able to get some detailed help from Nvidia. Thanks for pointing me in the right direction