Jetson Xavier Low bandwidth on PCIe

[michaelfink]

I am trying to connect a Jetson Xavier (as host) to a Xilinx ZCU106 using PCIe. I am using the Xilinx DMA drivers (dma_ip_drivers/XDMA/linux-kernel at master · Xilinx/dma_ip_drivers · GitHub).

Although I did not expect to reach the theoretical maximum speeds possible my bandwidth seems very low. I am achieving roughly 1.8 GB/s write speeds and 1.4 GB/s read speeds. Is there anything I can do increase my throughput?

[Manikanta]

Hi,

I don’t see any issue with PCIe, all capabilities are set to max of either RP or EP.

You can further check,

• CPU: Set nvpmodel to MaxN and run jetsonclocks.
• SMMU: Frequent dma_map & dma_unmap can increase CPU utilization and can hit perf. Try to avoid, if not disable SMMU for PCIe.

There is lot of documentation/forum threads on above two, please check.

Thanks,
Manikanta

[michaelfink]

Thanks for your reply!

From going through other forum posts I have been able to find information on what to edit in the device tree file in order to disable the SMMU but I have not been able to find where that file is, would you be able to provide this information? Also I am a bit confused by the editing process. Would I be able to edit the device tree file directly on the Jetson or will I need to make the change (according to the Kernel Customization section of the docs), rebuild, and reflash?

Would it be possible to avoid doing this altogether with the use of GPUDirect?

Thanks in advance,
Michael

[Manikanta]

Hi,

file: hardware/nvidia/soc/t19x/kernel-dts/tegra194-soc/tegra194-soc-pcie.dtsi

Yes, please follow the Kernel Customization section of the docs, rebuild, and reflash.

No, GPUDirect will not help here.

Thanks,
Manikanta