# GPU Direct RDMA for Jetson TK1

**URL:** <https://forums.developer.nvidia.com/t/gpu-direct-rdma-for-jetson-tk1/41116>\
**Category:** Jetson TK1\
**Created:** [January 24, 2016, 12:40pm UTC](https://forums.developer.nvidia.com/t/gpu-direct-rdma-for-jetson-tk1/41116 "2016-01-24T12:40:07Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![kumar81](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@kumar81](https://forums.developer.nvidia.com/u/kumar81)\
**Post date:** [January 24, 2016, 12:40pm UTC](https://forums.developer.nvidia.com/t/gpu-direct-rdma-for-jetson-tk1/41116/1 "2016-01-24T12:40:07Z")

</div>

Hi,

We are exploring the possibility of using the GPUDirect RDMA feature to transfer data from GPU to PCIe devices directly. Is it possible? If so, can anyone provide some pointers to the same?

Regards,  
Kumar.

---

<div class="post-metadata">

**Author:** ![kayccc](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/kayccc/32/14043_2.png) [@kayccc](https://forums.developer.nvidia.com/u/kayccc)\
**Post date:** [January 25, 2016, 8:10am UTC](https://forums.developer.nvidia.com/t/gpu-direct-rdma-for-jetson-tk1/41116/2 "2016-01-25T08:10:40Z")

</div>

Hi kumar81,

Regarding GPUdirect, it’s not supported in current BSP.

Thanks

---

<div class="post-metadata">

**Author:** ![kumar81](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@kumar81](https://forums.developer.nvidia.com/u/kumar81)\
**Post date:** [January 25, 2016, 11:11am UTC](https://forums.developer.nvidia.com/t/gpu-direct-rdma-for-jetson-tk1/41116/3 "2016-01-25T11:11:40Z")

</div>

Hello Kayccc,

```
Thanks lot for your reply. 

Actually we want to transfer data generated by GPU to PCIe device. As Tegra K1 has unified physical memory, all we need is a DMA'able (contingious and physical) memory to be allocated (either by ARM or by GPU) and accessible at the GPU over CUDA. How is it possible? Is there a mechanism for that in CUDA (like cudaMAllocHost or extensions)? Or is it possible using the NVMAP driver? 

I am sure there must be a similar kind mechanism being used by GRALLOC in Android. But there are no references for the same anywhere :( 

Looking forward for answers.

Thanks in advance.

```

Regards,  
Kumar.

---

<div class="post-metadata">

**Author:** ![dusty\_nv](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/dusty_nv/32/14043_2.png) [@dusty\_nv](https://forums.developer.nvidia.com/u/dusty_nv)\
**Post date:** [January 26, 2016, 5:40am UTC](https://forums.developer.nvidia.com/t/gpu-direct-rdma-for-jetson-tk1/41116/4 "2016-01-26T05:40:12Z")

</div>

You probably need to use a kernel function like vmalloc\_to\_page() or walk\_page\_range() to trace the userspace virtual pointer to the physical bus address (similarly, you might find virt\_to\_page() or get\_user\_pages() useful). Then once you obtain this physical address of the cudaMallocHost() buffer, you can send it over to your PCIe device. See [url][http://stackoverflow.com/a/28987409[/url]](http://stackoverflow.com/a/28987409%5B/url%5D) for an example from userspace.

---

<div class="post-metadata">

**Author:** ![kumar81](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@kumar81](https://forums.developer.nvidia.com/u/kumar81)\
**Post date:** [February 9, 2016, 12:17pm UTC](https://forums.developer.nvidia.com/t/gpu-direct-rdma-for-jetson-tk1/41116/5 "2016-02-09T12:17:07Z")

</div>

Hi,

Thanks a lot for the info and we could succesfully work with cudaMallocHost memory.

When we allocate an address using cudaMalloc, we are getting a device pointer. Since this is mapped to a different virtual space outside the processor’s, it will not be accessible in the CPU space (will return Bus error on access).

Now how can we get the physical address of a cudaMalloc-ated memory? And who does this virtual to physical mapping? SMMU? Is it how the current implementation works?

Regards,  
Kumar.

---

<div class="post-metadata">

**Author:** ![kayccc](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/kayccc/32/14043_2.png) [@kayccc](https://forums.developer.nvidia.com/u/kayccc)\
**Post date:** [February 15, 2016, 8:14am UTC](https://forums.developer.nvidia.com/t/gpu-direct-rdma-for-jetson-tk1/41116/6 "2016-02-15T08:14:39Z")

</div>

Hi kumar81,

All memory accessible to TK1 GPU needs to be page-locked  
Please check the CUDA doc to get more details for how to use it, see:  
[url][http://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#page-locked-host-memory[/url]](http://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#page-locked-host-memory%5B/url%5D)

Thanks
