Hi NOPUFF,
Thanks for your sharing!
- As we have talked, due to linux kernel network stack, the PCIe virtual network not match the 10Gbps requirement.
- For exchanging data of “10 cameras, 1920x1080 NV12 30fps, 5 lidars”, you can base on our reference code to implement you own feature.
Below patch verified Jetpack4.4
0001-gathered-all-dma-performance-test-patches.patch (11.2 KB)
RP Mode DMA
In the below procedure, x being the number of the root port controller whose DMA is being used for perf test
Write:
Go to the debugfs directory of the root port controller
#cd /sys/kernel/debug/pcie-x/
Set channel number (set it to one of 0,1,2,3)
#echo 1 > channel
Set size to 512MB
#echo 0x20000000 > size
Set source address for DMA.
For this, grep for the string “—> Allocated memory for DMA” in dmesg log and use whatever address comes up in the grep output
#dmesg | grep " ---> Allocated memory for DMA"
example output would be something like
[ 7.102149] tegra-pcie-dw 141a0000.pcie: —> Allocated memory for DMA @ 0xC0000000
So, use 0xC0000000 as the source address
#echo 0xC0000000 > src
Note : - don’t forget to replace 0xC0000000 with your grep output value. In case it is not found in grep output, save full kernel boot log and search in it
Set destination address for DMA
For this, execute the following command
#lspci -vv | grep -i “region 0”
an example output would be something like
Region 0: Memory at 1f40000000 (32-bit, non-prefetchable) [size=512M]
So, use 1f40000000 as destination address
#echo 0x1f40000000 > dst
Note : - don’t forget to replace 0x1f40000000 with your grep output value. In case it is not found in grep output, save full kernel boot log and search in it
Execute write test
#cat write
It prints the output in the following format(use ‘dmesg |tail’ to get the output)
tegra-pcie-dw 14100000.pcie_c1_rp: DMA write. Size: 536870912 bytes, Time diff: 316519776 ns
Read test can be performed by interchanging ‘src’ and ‘dst’ and executing ‘cat read’ command.
EP Mode DMA
Note: Most of steps operate in RP Xavier, except extract information from EP Xavier.
Write:
In the RP console, go to the debugfs directory of the end point client driver
#cd /sys/kernel/debug/tegra_pcie_ep/
Set channel number (set it to one of 0,1,2,3)
#echo 1 > channel
Set size to 512 MB
#echo 0x20000000 > size
Set source address for EP’s DMA.
For this, grep for the string "BAR0 RAM IOVA” in dmesg log of endpoint system console and use whatever address comes up in the grep output
#dmesg | grep “BAR0 RAM IOVA”
an example output would be something like
pci_epf_nv_test pci_epf_nv_test.0: BAR0 RAM IOVA: 0xc0000000
So, use 0xc0000000 as source address
#echo 0xc0000000 > src
Note : - don’t forget to replace 0xe0000000 with your grep output value. In case it is not found in grep output, save full kernel boot log and search in it
Set destination address for DMA
For this, grep for the string “Allocated memory for DMA operation” in dmesg log of host system console (i.e. current system) and use whatever address comes up in the grep output
#dmesg | grep " Allocated memory for DMA"
an example output would be something like
tegra_ep_mem 0005:01:00.0: Allocated memory for DMA operation @ 0x80000000, size=0x20000000
So, use 0x80000000 as source address
#echo 0x80000000 > dst
Note : - don’t forget to replace 0xC0000000 with your grep output value. In case it is not found in grep output, save full kernel boot log and search in it
Execute write test
#cat write
It prints the output in the following format
tegra_ep_mem 0000:01:00.0: DMA write: Size: 536870912 bytes, Time diff: 296565536 ns
Read test can be performed by interchanging ‘src’ and ‘dst’ and executing ‘cat read’ command.