I have a few questions about GPU Direct RDMA

_JJANG · November 22, 2022, 6:30am

I want to use gpu direct rdma on jetson agx xavier.
So I am making a Linux kernel source by looking at an example.

But I have a few questions.

Is it correct to proceed in the following order to use cuda memory as dma?
Assign with cudaHostAlloc → cuPointerSetAttribute → nvidia_p2p_get_page → nvidia_p2p_dma_map_page
If I use the nvidia_p2p_dma_map_page function, can I get the physical address of the memory allocated by cudaHostAlloc?
(Is the hw_address variable in the struct nvidia_p2p_dma_mapping the physical address of the memory allocated by cudaHostAlloc?)
If hw_address is the physical address of the memory allocated by cudaHostAlloc, is it possible to write data to hw_address to write data to cudaDMA?
Why dmam_alloc_coherent in the GPUDirectRDMA example?
Shouldn’t nvidia_p2p_dma_map_page be used to use cudaDMA?

Help me. thank you

AastaLLL · November 22, 2022, 7:41am

Hi,

Have you checked the below document before?

Thanks.

_JJANG · November 22, 2022, 7:50am

Hi, AstaLLL
I have read the documentation.
But I can’t find the answer to question 4.

AastaLLL · December 1, 2022, 7:24am

Hi

Did you mean the code below?

github.com

NVIDIA/jetson-rdma-picoevb/blob/master/kernel-module/picoevb-rdma.c#L1164


      
          	 * sizes by dma_map_sg(), we cannot rely on dma_map_sg()'s chunking in
          	 * this case. Instead, this driver must handle the chunking itself, and
          	 * so can accept arbitrarily long IOVA chunks in sg lists.
          	 */
          	pevb->dma_params.max_segment_size = UINT_MAX;
          	pdev->dev.dma_parms = &pevb->dma_params;
          
          
	sema_init(&pevb->sem, 1);
          	init_completion(&pevb->dma_xfer_cmpl);
          
          
	pevb->descs_ptr = dmam_alloc_coherent(&pdev->dev, SZ_4K,
          		&pevb->descs_dma_addr, GFP_KERNEL);
          	if (!pevb->descs_ptr) {
          		dev_err(&pdev->dev,
          			"dma_alloc_coherent(descriptors): failed\n");
          		return -ENOMEM;
          	}
          
          
	ret = alloc_chrdev_region(&pevb->devt, 0, 1, MODULENAME);
          	if (ret < 0) {
          		dev_err(&pdev->dev, "alloc_chrdev_region(): %d\n", ret);

The function is for the PCIe device.

Thanks.

system · December 21, 2022, 5:33am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Is GPU Direct RDMA supported on Xavier? (Solved) Jetson AGX Xavier	23	3954	October 18, 2021
Issues porting desktop RDMA app to Tegra: mmap hangs kernel Jetson AGX Xavier cuda	11	1489	April 1, 2022
How to use memory allocated by dma_alloc_coherent() in cuda Jetson TX2 cuda	6	964	October 18, 2021
GPUdirect RDMA in Jetson Xavier - cudaHostAlloc() Jetson AGX Xavier cuda	2	441	January 5, 2022
GPU direct access to DMA memory over PCIe Jetson Xavier NX pcie , cuda	4	2258	April 22, 2022
GPUDirect RDMA on NVIDIA Jetson AGX Xavier Technical Blog	1	839	June 12, 2019
GPUDirect RDMA - use Jetson's DMA Jetson AGX Orin gpu-computing	15	390	July 12, 2024
Error when trying to write data to GPU DMA memory (using GPU Direct RDMA) Jetson AGX Xavier pcie , kernel , fpga	8	1468	May 30, 2023
GPUDirect RDMA - Multithreading Jetson AGX Xavier cuda , kernel	4	1170	October 18, 2021
GDS and Jetson Xavier Jetson AGX Xavier gds	5	843	June 7, 2023

I have a few questions about GPU Direct RDMA

Related topics