DMA Engines for PCIe

Please provide the following info (check/uncheck the boxes after creating this topic):
Software Version
DRIVE OS Linux 5.2.6
DRIVE OS Linux 5.2.0
DRIVE OS Linux 5.2.0 and DriveWorks 3.5
NVIDIA DRIVE™ Software 10.0 (Linux)
NVIDIA DRIVE™ Software 9.0 (Linux)
other DRIVE OS version
other

Target Operating System
Linux
QNX
other

Hardware Platform
NVIDIA DRIVE™ AGX Xavier DevKit (E3550)
NVIDIA DRIVE™ AGX Pegasus DevKit (E3550)
other

SDK Manager Version
1.6.0.8170
other

Host Machine Version
native Ubuntu 18.04
other

Hi All,

Could general purpose DMA engines in Xavier or Orins be used to schedule a large buffer transfer over PCIe to another Xavier/Orin ? Would this path be more efficient and fast compared to TCP based path taken by DW socket APIs ? Is there an example or sample that illustrates this ?

Thanks.

Dear @ai.trucker,
Is the ask: Xavier to Xavier communication is better over PCIe switch or ethernet?

No. The ask is whether there is a DMA engine which can drive data over PCIe without much of CPU intervention ?
Thanks.

Hi SivaRamaKrishnaNV - any update ?
Thanks.

Dear @ai.trucker,
We are checking internally on your query and update you as soon as I hear from team.

Hi @ai.trucker,

C2C supports DMA over PCIe and will be available in DRIVE Orin releases. For now you may need to use 10 GbE for large transfers. Sorry for any inconvenience.

Thanks VickNV. For now, on Xaviers dev kit and Pegasus platforms I take it that there is no support for DMA based PCIe transfer functionality.
Can you confirm whether PCIe x4 lane based connectivity for xavier to xavier communication (on Pegasus) is functional at all ? My current understanding is that it is not functional and has not been validated. Please confirm.

Thanks.

Isn’t NTB working on your side? What do you mean PCIe x4 lane based connectivity for xavier to xavier communication (on Pegasus)? Thanks.

Dear @ai.trucker,
Just to clarify, as Vick stated, 10gbe is the only way to send data across two Xavier.

Its very confusing. VickNV seem to suggest or ask with the allusion that communication over PCIe, NTB should work. Now SivaRamaKrishnaNV says that 10gbe is the only way to send data across xavier.

Contradiction ?

Yes VickNV we are able to go over path B (of attached pic) with ntb_netdev. However we are not able to get the data transfer rates (we observe 2.2 Gb/s) which seems way below theoretical limit of 4 GB/s for pcie g3 x4. So SivaRamaKrishnaNV I am not sure what to make of “10gbe is the only way to send data across two Xavier” ?

Thanks

Regarding the DMA access, it is my understanding that Nvidia does not support DMA through PCIe in Xavier to Xavier communications.
That said, I suspect we can use the Linux Kernel’s NTB drivers to gain access to the DMA capabilities of the PCIe port using ntb_transport. Moreover, a quick modinfo suggests that the ntb_transport driver supports DMA (see last line)

nvidia@xavier1a:~$ modinfo ntb_transport
filename:       /lib/modules/4.14.102-rt53-tegra/kernel/drivers/ntb/ntb_transport.ko
author:         Intel Corporation
license:        Dual BSD/GPL
version:        4
description:    Software Queue-Pair Transport over NTB
srcversion:     D93FD561B2CAE8732631245
depends:        ntb
intree:         Y
name:           ntb_transport
vermagic:       4.14.102-rt53-tegra SMP preempt mod_unload modversions aarch64
parm:           max_mw_size:Limit size of large memory windows (ulong)
parm:           transport_mtu:Maximum size of NTB transport packets (uint)
parm:           max_num_clients:Maximum number of NTB transport clients (byte)
parm:           copy_bytes:Threshold under which NTB will use the CPU to copy instead of DMA (uint)
parm:           use_dma:Use DMA engine to perform large data copy (bool)

… which is also in line with what we saw at Kernel.org’s NTB Drivers page

Only PIO mode will work. DMA isn’t supported.
We support PCIe DMA in C2C as NTB client.
Thanks.

1 Like

Thanks a lot for your reply @VickNV, this aligns with our understanding of the Pegasus system.

Could you help us clarify @SivaRamaKrishnaNV 's answer?

I strongly believe that @SivaRamaKrishnaNV means that Nvidia does not provides a specific API + Docs for NTB-based PCIe communication, and that it is up to the engineer to enable and write the software for this type of communication. Is my assumption correct?

Thanks in advance for your clarification

Dear @erick.vieyra,
So the question: is there any way NTB-based PCIe communication possible between two Xavier’s?

Hello @SivaRamaKrishnaNV
thanks for your reply.
We are just a little confused about your statement here
We would like to know whether or not “10Gbe is the only way to send data across two Xavier” since this contradicts a lot of the docs and answers received from Nvidia.