How to enable GPU Direct RDMA for Jetson Orin NX

[DerekL]

Hi,

請問要如何開啟 GPU Direct RDMA 的相關功能?
我們目前已經修改以下參數

…/configs/tegra_defconfig :
CONFIG_NVME_RDMA=m
CONFIG_TERGA_RDMA=y

$make menuconfig
Device Drivers—>
<*>TEGRA_RDMA_DRIVER —

原本螢幕可以正常顯示，修改完後螢幕無法顯示。

How to enable GPU Direct RDMA ?

We have modified some parameters below.

…/configs/tegra_defconfig :
CONFIG_NVME_RDMA=m
CONFIG_TERGA_RDMA=y

$make menuconfig
Device Drivers—>
<*>TEGRA_RDMA_DRIVER —

The original screen can be displayed normally, but after modification, the screen cannot be displayed.

[AastaLLL]

Hi,

Please try to restart gdm3.

$ systemctl daemon-reload
$ sudo service gdm3 start

Thanks.

[DerekL]

Hi,

Thanks for your help.
But It doesn’t seem to work.

By the way, my capture card is YUAN SC710N1.
Then I use HDMI to connect my PC as input of the capture card. And I use V4L2 to capture my PC screen on Orin NX screen. Before I modified the RDMA configuration, the monitor can display from Orin NX.

Regards,
Derek Lin

[AastaLLL]

Hi,

Could you share the detailed steps about how you setup the RDMA environment?
We want to check if we can reproduce the same in our environment.

More, which JetPack version do you use?
Thanks.

[DerekL]

Hi,

I had solved this problem with this GPUDirect RDMA - Module can not be insert into kernel - #27 by AastaLLL

$ sudo modprobe nvidia_p2p 
$ lsmod | grep nv

The display drivers and nvidia_p2p module have been loaded.

Is RDMA enabled?

Because the image transmission delay has not improved, I’m confused.

Here is my testing method :

I use V4L2 to capture my PC HDMI output on Orin NX.

PC ----> HDMI ----> capture card ----> PCIe ----> Orin NX ----> HDMI ----> monitor 1
PC ----> Displayport ----> monitor 2

Then my PC run 1ms timer and I take photos to compare the time difference between two monitor. I fond that the time difference almost unchanged when RDMA is enabled.

Is there a better way to test the transmission delay?

Regards,
Derek Lin

[DerekL]

Hi,

I use Jetpack 5.1

Regards.

[AastaLLL]

Hi,

Could you share more about your use case?
Is the display connected to the Orin?

Thanks.

[DerekL]

I use my PC as HDMI input to connect capture card.
Then use V4L2 to capture image from PC on ORIN. So I can display my PC screen on ORIN.
Left monitor : Orin NX
Right monitor : x86 PC

Regards,
Derek Lin

[AastaLLL]

Hi,

Could you also share a screenshot of connecting Jetson with an FPGA?
Thanks.

[DerekL]

Hi,

How can I do that?

But We are trying to use the API that supports RDMA provided by YUAN.
Then I will test this use case again. And We will confirm whether we are not using YUAN’s API for optimization.

Thanks.

[AastaLLL]

Hi,

Could you share the expected output with us?

RDMA provides some API for PCIe devices to use the buffer allocated on the GPU.
But it won’t reduce the delay by default.

Thanks.