# GPUDirect RDMA - use Jetson's DMA **URL:** \ **Category:** Jetson AGX Orin\ **Tags:** gpu-computing\ **Created:** [June 5, 2024, 10:26am UTC](https://forums.developer.nvidia.com/t/gpudirect-rdma-use-jetsons-dma/295236 "2024-06-05T10:26:45Z")\ **Posts on this page:** 16\ **Page:** 1
**Author:** ![bsp\_dev](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@bsp\_dev](https://forums.developer.nvidia.com/u/bsp_dev)\ **Post date:** [June 5, 2024, 10:26am UTC](https://forums.developer.nvidia.com/t/gpudirect-rdma-use-jetsons-dma/295236/1 "2024-06-05T10:26:45Z")
Hi, In the GPUDirect RDMA Github demo: > **[GitHub - NVIDIA/jetson-rdma-picoevb: Minimal HW-based demo of GPUDirect RDMA on NVIDIA...](https://github.com/NVIDIA/jetson-rdma-picoevb)** > > Minimal HW-based demo of GPUDirect RDMA on NVIDIA Jetson AGX Xavier running L4T You use the FPGA’s DMA engine to copy the data from the GPU’s memory space into the FPGA memory. Is it possible to use the Jetson’s DMA engine instead? That it will copy the data into the FPGA memory? Thanks ---
**Author:** ![AastaLLL](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/aastalll/32/14043_2.png) [@AastaLLL](https://forums.developer.nvidia.com/u/AastaLLL)\ **Post date:** [June 6, 2024, 4:33am UTC](https://forums.developer.nvidia.com/t/gpudirect-rdma-use-jetsons-dma/295236/4 "2024-06-06T04:33:28Z")
Hi, RDMA is about direct access, no memory copy is involved. GPUDirect RDMA’s target is to share the cudaMalloc buffer with FPGA. It can not used in the counterpart. Thanks. ---
**Author:** ![bsp\_dev](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@bsp\_dev](https://forums.developer.nvidia.com/u/bsp_dev)\ **Post date:** [June 6, 2024, 1:21pm UTC](https://forums.developer.nvidia.com/t/gpudirect-rdma-use-jetsons-dma/295236/5 "2024-06-06T13:21:08Z")
@AastaLLL Thank you for your reply. Just to make sure, what do you mean by “counterpart”? Thanks ---
**Author:** ![AastaLLL](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/aastalll/32/14043_2.png) [@AastaLLL](https://forums.developer.nvidia.com/u/AastaLLL)\ **Post date:** [June 7, 2024, 4:41am UTC](https://forums.developer.nvidia.com/t/gpudirect-rdma-use-jetsons-dma/295236/6 "2024-06-07T04:41:03Z")
Hi, FPGA can access CUDA memory. But CUDA cannot access FPGA’s memory. This requires importing/wrapping the CPU memory into GPU space. Which is not supported. Thanks. ---
**Author:** ![bsp\_dev](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@bsp\_dev](https://forums.developer.nvidia.com/u/bsp_dev)\ **Post date:** [June 7, 2024, 5:23am UTC](https://forums.developer.nvidia.com/t/gpudirect-rdma-use-jetsons-dma/295236/7 "2024-06-07T05:23:01Z")
Thank you for your reply @AastaLLL There is something small I didn’t understand: In the demo the FPGA can access CUDA memory. “Someone” has to program the FPGA’s DMA engine with appropriate data: - physical source address, corresponds to CUDA virtual space - physical destination address, in the FPGA - length 1. Who does this? - the CUDA code? 2. Why he can’t program the Jetson DMA to do the same work instead of the FPGA DMA? Thanks for the help ---
**Author:** ![AastaLLL](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/aastalll/32/14043_2.png) [@AastaLLL](https://forums.developer.nvidia.com/u/AastaLLL)\ **Post date:** [June 11, 2024, 7:55am UTC](https://forums.developer.nvidia.com/t/gpudirect-rdma-use-jetsons-dma/295236/8 "2024-06-11T07:55:09Z")
Hi, Could you share the demo you mentioned with us? So we can know more about the questions. Thanks. ---
**Author:** ![bsp\_dev](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@bsp\_dev](https://forums.developer.nvidia.com/u/bsp_dev)\ **Post date:** [June 11, 2024, 7:57am UTC](https://forums.developer.nvidia.com/t/gpudirect-rdma-use-jetsons-dma/295236/9 "2024-06-11T07:57:47Z")
Sure: > **[GitHub - NVIDIA/jetson-rdma-picoevb: Minimal HW-based demo of GPUDirect RDMA on NVIDIA...](https://github.com/NVIDIA/jetson-rdma-picoevb)** > > Minimal HW-based demo of GPUDirect RDMA on NVIDIA Jetson AGX Xavier running L4T ---
**Author:** ![bsp\_dev](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@bsp\_dev](https://forums.developer.nvidia.com/u/bsp_dev)\ **Post date:** [June 24, 2024, 4:44pm UTC](https://forums.developer.nvidia.com/t/gpudirect-rdma-use-jetsons-dma/295236/10 "2024-06-24T16:44:56Z")
Hi @AastaLLL , Any new regarding my question? After I sent you the demo link, maybe you know to answer my questions? ---
**Author:** ![AastaLLL](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/aastalll/32/14043_2.png) [@AastaLLL](https://forums.developer.nvidia.com/u/AastaLLL)\ **Post date:** [June 25, 2024, 7:27am UTC](https://forums.developer.nvidia.com/t/gpudirect-rdma-use-jetsons-dma/295236/11 "2024-06-25T07:27:22Z")
Hi, Thanks for your patience. **1.** No, it is ioctl call rather than a CUDA code: For example: [https://github.com/NVIDIA/jetson-rdma-picoevb/blob/master/client-applications/rdma-cuda-c2h-perf.cu#L105](https://github.com/NVIDIA/jetson-rdma-picoevb/blob/master/client-applications/rdma-cuda-c2h-perf.cu#L105) **2.** The sample provides an interface for FPGA to access CUDA memory. If you want to access FPGA memory on Jetson, please check with the FPGA provider to see if they have a similar API/mechanism. Thanks. ---
**Author:** ![bsp\_dev](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@bsp\_dev](https://forums.developer.nvidia.com/u/bsp_dev)\ **Post date:** [June 25, 2024, 9:41am UTC](https://forums.developer.nvidia.com/t/gpudirect-rdma-use-jetsons-dma/295236/12 "2024-06-25T09:41:34Z")
Hi @AastaLLL , Thank you for your reply. I apologies but I didn’t understand your answer. You said that the code being executed on the GPU uses an ioctl to program the FPGA’s DMA core for executing the transfer from the Jetson’s memory (GPU address space) into the FPGA’s memory. My question was if its possible to use another method , maybe another ioctl , to program the Jetson’s DMA core for executing the transfer from the Jetson’s memory (GPU address space) into the FPGA’s memory. In other words, instead of the FPGA’s DMA core will do the transfers, we want the Jetson’s DMA core to do the transfers. ---
**Author:** ![AastaLLL](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/aastalll/32/14043_2.png) [@AastaLLL](https://forums.developer.nvidia.com/u/AastaLLL)\ **Post date:** [June 26, 2024, 7:38am UTC](https://forums.developer.nvidia.com/t/gpudirect-rdma-use-jetsons-dma/295236/13 "2024-06-26T07:38:59Z")
Hi, CUDA doesn’t support importing a pre-allocated memory into CUDA memory space. But if you don’t need this for GPU, the memory mapping should work with a standard Linux DMA way. Thanks. ---
**Author:** ![bsp\_dev](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@bsp\_dev](https://forums.developer.nvidia.com/u/bsp_dev)\ **Post date:** [June 26, 2024, 9:19am UTC](https://forums.developer.nvidia.com/t/gpudirect-rdma-use-jetsons-dma/295236/14 "2024-06-26T09:19:15Z")
I apologize, I don’t understand your answer. Maybe I’m not familiar enough with the details. I’ll appreciate if you could elaborate a little. In the demo we see the ioctl that utilize the FPGA’s DMA core to perform the transfer. The code configures the FPGA’s DMA core with the suitable physicals addresses. Why can’t it do the same with the Jetson’s DMA core? Using another ioctl that will load the same data physical address into the Jetsons DMA core instead of the FPGA’s DMA core and initiate the transfer with the Jetson’s DMA. ---
**Author:** ![AastaLLL](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/aastalll/32/14043_2.png) [@AastaLLL](https://forums.developer.nvidia.com/u/AastaLLL)\ **Post date:** [June 27, 2024, 9:47am UTC](https://forums.developer.nvidia.com/t/gpudirect-rdma-use-jetsons-dma/295236/15 "2024-06-27T09:47:54Z")
Hi, Do you mean PCIe DMA? If yes, it can work just as the Linux standard way. If not, could you tell us more about the “Jetson’s DMA” you indicated? Thanks. ---
**Author:** ![AastaLLL](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/aastalll/32/14043_2.png) [@AastaLLL](https://forums.developer.nvidia.com/u/AastaLLL)\ **Post date:** [June 27, 2024, 9:54am UTC](https://forums.developer.nvidia.com/t/gpudirect-rdma-use-jetsons-dma/295236/16 "2024-06-27T09:54:26Z")
Hi, Not sure if this helps, but please check the below topic discussing DMA engine. > [@How to Carry out DMA transfer when sending data using PCIe from NVIDIA Root Port to a custom end point](https://forums.developer.nvidia.com/t/how-to-carry-out-dma-transfer-when-sending-data-using-pcie-from-nvidia-root-port-to-a-custom-end-point/259610/13): > > Hi, Custom Endpoint hardware should have DMA support and get the DMA programming guidelines from vendor. Then you have to write a client driver by following the guidelines. This is not related to Tegra. Thanks, Manikanta Thanks. ---
**Author:** ![bsp\_dev](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@bsp\_dev](https://forums.developer.nvidia.com/u/bsp_dev)\ **Post date:** [July 12, 2024, 7:20pm UTC](https://forums.developer.nvidia.com/t/gpudirect-rdma-use-jetsons-dma/295236/18 "2024-07-12T19:20:44Z")
Yes, I mean the general DMA engine in the Jetson (not a specific device DMA) What I meant was that instead of activating the FPG DMA from the CUDA code, could we just activate the PCIe DMA from the CUDA code on the GPU? ---
**Author:** ![system](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/system/32/68080_2.png) [@system](https://forums.developer.nvidia.com/u/system)\ **Post date:** [July 31, 2024, 2:47am UTC](https://forums.developer.nvidia.com/t/gpudirect-rdma-use-jetsons-dma/295236/19 "2024-07-31T02:47:58Z")
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.