How to use pci-epf-dma-test between two Nvidia AGX Xavier

I have two Xavier AGX Industrial connected via an external cable, providing PCIe Gen 1 speed. One is RP while the other is EP. I checked the nvidia/drivers/pci/endpoint/functions/ folder and I thought pci-epf-dma-test.c would be the best choice.
On the EP side, I applied the differences on 0001-gathered-all-dma-performance-test-patches.patch. When I check pci-epf-dma-test.c the rp phy address was missing, so I added a debug folder to add rp, ep and size from debug folders. As a result, I made following settings;
cd /sys/kernel/config/pci_ep/
mkdir functions/tegra_pcie_dma_epf/func1
echo 16 > functions/tegra_pcie_dma_epf/func1/msi_interrupts
ln -s functions/tegra_pcie_dma_epf/func1 controllers/141a0000.pcie_ep/
echo 1 > controllers/141a0000.pcie_ep/start
echo 0xe0000000 > /sys/kernel/debug/141a0000.pcie_ep_epf_dma_test/rp_dest
echo 0xFF00000 > /sys/kernel/debug/141a0000.pcie_ep_epf_dma_test/rp_size
echo 0x10000 > /sys/kernel/debug/141a0000.pcie_ep_epf_dma_test/dma_size
echo 0xf0000000 > /sys/kernel/debug/141a0000.pcie_ep_epf_dma_test/ep_src
On RP side there was a line as below. So I thought RP DMA address is 0xe0000000.
0xf0000000 was the BAR address assigned on EP side.
When RP side is up, It successfully found the enpoint. You can find the details of lspci output in attachments.
lspci.txt (4.2 KB)
I called following commands;
setpci -s 0005:01:00.0 COMMAND=0x02
setpci -s 0005:01:00.0 4.l=0x07

When I call perf_test on RP Side following output appears;
/sys/kernel/debug/pcie@141a0000# cat perf_test
[ 224.505511] tegra194-pcie 141a0000.pcie: perf_test: DD WR, CH: 0 SZ: 267386880 B, time: 297553184 ns
[ 224.989784] tegra194-pcie 141a0000.pcie: perf_test: DD RD, CH: 0 SZ: 267386880 B, time: 484098944 ns
[ 225.287079] tegra194-pcie 141a0000.pcie: perf_test: LL WR, CH: 0 N: 1 SZ: 267386880 B, time: 297080192 ns
[ 225.681459] tegra194-pcie 141a0000.pcie: perf_test: LL RD, CH: 0 N: 1 SZ: 267386880 B, time: 394207680 ns

When I call perf_test on EP side following output appears;
/sys/kernel/debug/141a0000.pcie_ep_epf_dma_test# cat perf_test
[ 413.798509] pcie_dma_epf tegra_pcie_dma_epf.0: DDMA TX EP:f0100000 RP:e0100000 SZ:ff00000
[ 414.091929] pcie_dma_epf tegra_pcie_dma_epf.0: perf_test: DD WR, CH: 0 SZ: 267386880 B, time: 293288512 ns
[ 414.091946] pcie_dma_epf tegra_pcie_dma_epf.0: DDMA RX RP:e0100000 EP:f0100000 SZ:ff00000
[ 414.450522] pcie_dma_epf tegra_pcie_dma_epf.0: perf_test: DD RD, CH: 0 SZ: 267386880 B, time: 358539776 ns
[ 414.450541] pcie_dma_epf tegra_pcie_dma_epf.0: LL TX EP:f0100000 RP:e0100000 SZ:ff00000
[ 414.744122] pcie_dma_epf tegra_pcie_dma_epf.0: perf_test: LL WR, CH: 0 N: 1 SZ: 267386880 B, time: 293349184 ns
[ 414.744168] pcie_dma_epf tegra_pcie_dma_epf.0: LL RX RP:e0100000 EP:f0100000 SZ:ff00000
[ 415.090287] pcie_dma_epf tegra_pcie_dma_epf.0: perf_test: LL RD, CH: 0 N: 1 SZ: 267386880 B, time: 346072064 ns

But, I have a problem at this point;
When I manually change f0100000 on EP side or 1c00000000 or e0100000 on RP side, I cant see it on the other side.
If I set a wrong address I always got decode address error or mc_err.
What I couldnt understand why I cant see my modifications on other side. I think my transaction is done, but why I cant see my modifications on other side.
Can you please guide/comment me on how to use pci-epf-dma-test?
Best Regards

I just wanted to add more information.
Both RP and EP board has JetPack 5.2 and Linux version l4t-r35.4.1. The boards are identical custom carrier board with Nvidia AGX Xavier Industrial Module. RP ODMDATA is set to be RootComplex, EP side is set as endpoint device.
Boards are connected via external cables. On RP side lspci output shows they are successfully connected.
I used the source in the path nvidia/drivers/pci/endpoint/functions/pci-epf-dma-test.c to make a performance test using perf_test. As I shared it gives output that data is delivered to other side, but I cant see the updated data on the other side.
On RP side I follower the pcie_tegra194.c file, I understood that the line depicted below shows the DMAable address of RP side.
[ 6.822584] tegra194-pcie 141a0000.pcie: RP host DMA buf: 0xe0000000 size: 268435456n
On the other hand on EP side the following dmesg output I thpught is the EP DMAable region, which is also mapped to BAR address;
[ 59.619784] pcie_dma_epf tegra_pcie_dma_epf.0: BAR0 phy_addr: f0000000 size: 10000000
I used devmem function to change some data so as to see it on other side.
Accordingly, I used these addresses with perf_test/pci-epf-dma-test.c to start transaction from 0xf0000000 to 0xe0000000 on EP side. When it is completed it doesnt give any error, it looks like data is transfered successfully. But on RP side I couldnt see at the physical address I inserted. I also tried to use the read command, there was no mc_err or smmu error, but I couldn’t see the data on EP side.
I will really appreciate, If you can give a quick response.***
Thank you very much.
Best Regards

Please help to provide full uart logs.

Please use sanity_test test instead of perf_test. This does CRC check, so we will know if transfer happened correctly or not.

Please let us know how do you check that transfer happened on other side or not, and how do you conclude that data is not transferred correctly.

Hello @kayccc
Thank you for your response. I attached the dmesg and console output for both RP and EP side. And for the EP side, rp_phy_addr parameter wasn’t assigned to any value. Because of that I made some code changes in pci-epf-dma-test.c file, which is also attached.
From RP side, it seems sanity also works. From EP side, even if I set the correct addresses, I received errors even during dma_rx/tx transaction. You can examine the attached files.
What is interesting is when sanity fails on EP side, I saw mc_err output on EP side. And I am sure I didn’t set 0x0000000 address on EP.
I initiated also the perf_test again.
Best Regards
console-ep.txt (5.6 KB)
console-rp.txt (4.2 KB)
dmesg-ep.txt (64.8 KB)
dmesg-rp.txt (66.9 KB)
console-ep.txt (5.6 KB)
pci-epf-dma-test.c.txt (49.7 KB)

I also inserted one more print in edma_submit_direct_tx and edma_submit_direct_rx in order to be sure if the addresses copied correctly. As you can see below, addresses seem correct. Best Regards

root@jetson-agx-xavier-industrial:~# cat /sys/kernel/debug/141a0000.pcie_ep_epf_dma_test/sanity_test
[ 245.056486] pcie_dma_epf tegra_pcie_dma_epf.0: sanity test EP:f0100000 RP:e0100000 SZ:10000
[ 245.057321] pcie_dma_epf tegra_pcie_dma_epf.0: sanity test TX EP:f0100000 RP:e0100000 SZ:10000 ch:0
[ 245.057358] pcie_dma_epf tegra_pcie_dma_epf.0: edma_submit_direct_tx src low:f0100000 high:0 dst low:e0100000 high:0 size:10000 ch:0
root@jetson-agx-xavier-industrial:~# [ 250.080881] pcie_dma_epf tegra_pcie_dma_epf.0: edma_submit_direct_tx: DD WR CH: 0 TO
[ 250.080915] pcie_dma_epf tegra_pcie_dma_epf.0: sanity_test: DD WR CH: 0 failed


Hello @kayccc ,
I made a little proceed. I want to share it.
I noticed the line below. All other test initializes edma with lie=1.
edma_init(epfnv, 1/0/);
So when I changed to lie=1, now edma_submit_direct_tx is completed. But now in crc, written data crc is totally different than BAR data. (4bf364eb!=0)
I think bar0_virt and epf_bar0->ep_phy_addr (or epf_bar->phys_addr) might point to different addresses.
Best Regards

root@jetson-agx-xavier-industrial:~# cat /sys/kernel/debug/141a0000.pcie_ep_epf_dma_test/sanity_test
[ 182.730545] pcie_dma_epf tegra_pcie_dma_epf.0: sanity test EP:f0100000 RP:e0100000 SZ:10000
[ 182.730777] pcie_dma_epf tegra_pcie_dma_epf.0: sanity test TX EP:f0100000 RP:e0100000 SZ:10000 ch:0
[ 182.730783] pcie_dma_epf tegra_pcie_dma_epf.0: edma_submit_direct_tx src low:f0100000 high:0 dst low:e0100000 high:0 size:10000 ch:0
[ 182.730978] pcie_dma_epf tegra_pcie_dma_epf.0: sanity_test: DD WR, SZ: 65536 B CH: 0 CRC failed 4bf364eb!=0

0x1c00000000 is the RP side BAR address. 0xf0000000 is the ep_phy_addr. For nv_test, devmem can be used to modify it and the modification could be observed on other side. In our case all data is different.

The other question is how sanity works? epf_bar0->wr_data[i].crc isn’t set. So it returns 0 for crc.

Is it able to work with the test method mentioned here?

Hi @WayneWWW,
Yes it works as can be seen below. I tried from EP side.
I set RP phy address to 0xe0000000.
Do you have any further comment?
Best Regards

root@jetson-agx-xavier-industrial:/sys/kernel/debug/141a0000.pcie_ep_epf_dma_test# cat edmalib_test
[ 185.542631] pcie_dma_epf tegra_pcie_dma_epf.0: edmalib_common_test: re-init edma lib prev_ch(0) != current chans(11)
[ 185.544906] tegra194-pcie 141a0000.pcie_ep: tegra_pcie_edma_initialize: success
[ 185.546092] pcie_dma_epf tegra_pcie_dma_epf.0: edmalib_common_test: EDMA LIB WR started for 1 chans, size 65536 Bytes, iterations: 1000 of descriptors 4
[ 185.547275] pcie_dma_epf tegra_pcie_dma_epf.0: edmalib_common_test: EDMA LIB submit done
[ 185.868507] pcie_dma_epf tegra_pcie_dma_epf.0: edma_final_complete: WR-local-Async complete for chan 0 with status 0. Total desc 4000 of Sz 65536 Bytes done in time 322399168 nsec. Perf is 6504 Mbps
[ 185.868526] pcie_dma_epf tegra_pcie_dma_epf.0: edma_final_complete: All Async channels. Cumulative Perf 6504 Mbps, time 322399264 nsec

My other question is why BAR on RP is not same as EP BAR memory area;

On the RP side pcie-tegra194.c, due to line below, the data is randomly created.
get_random_bytes(pcie->dma_virt, BAR0_SIZE);

root@jetson-agx-xavier-industrial:~# devmem 0x1c00100000
root@jetson-agx-xavier-industrial:~# lspci -v
0005:01:00.0 RAM memory: NVIDIA Corporation Device 229a
Flags: bus master, fast devsel, latency 0, IRQ 255
Memory at 1c00000000 (64-bit, prefetchable) [size=256M]
Memory at 1c10000000 (64-bit, prefetchable) [size=128K]
Memory at 1f40000000 (64-bit, non-prefetchable) [size=1M]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/16 Maskable+ 64bit+
Capabilities: [70] Express Endpoint, MSI 00
Capabilities: [b0] MSI-X: Enable- Count=8 Masked-
Capabilities: [100] Advanced Error Reporting
Capabilities: [148] Secondary PCI Express
Capabilities: [168] Physical Layer 16.0 GT/s <?> Capabilities: [190] Lane Margining at the Receiver <?>
Capabilities: [1b8] Latency Tolerance Reporting
Capabilities: [1c0] L1 PM Substates
Capabilities: [1d0] Vendor Specific Information: ID=0002 Rev=4 Len=100 <?> Capabilities: [2d0] Vendor Specific Information: ID=0001 Rev=1 Len=038 <?>
Capabilities: [308] Data Link Feature <?> Capabilities: [314] Precision Time Measurement Capabilities: [320] Vendor Specific Information: ID=0003 Rev=1 Len=054 <?>

On EP side;
root@jetson-agx-xavier-industrial:/sys/kernel/debug/141a0000.pcie_ep_epf_dma_test# devmem 0xf0100000

Any update pon both side, doesn’t have any effect on other.

I did all the settings mentioned in the code and documentation.
Is there any missing settings?

Best regards

For the CRC issue,

There is no RP → EP interrupt support in dma test driver, that’s why CRC information from RP is not received by EP. This caused wait timeout and CRC check failure.

In Tegra vnet driver, we are using host1x interrupts for RP → EP.

Thus, please move to use Tegra vnet driver.