Hello,
I have two Xavier AGX Industrial connected via an external cable, providing PCIe Gen 1 speed. One is RP while the other is EP. I checked the nvidia/drivers/pci/endpoint/functions/ folder and I thought pci-epf-dma-test.c would be the best choice.
On the EP side, I applied the differences on 0001-gathered-all-dma-performance-test-patches.patch. When I check pci-epf-dma-test.c the rp phy address was missing, so I added a debug folder to add rp, ep and size from debug folders. As a result, I made following settings;
cd /sys/kernel/config/pci_ep/
mkdir functions/tegra_pcie_dma_epf/func1
echo 16 > functions/tegra_pcie_dma_epf/func1/msi_interrupts
ln -s functions/tegra_pcie_dma_epf/func1 controllers/141a0000.pcie_ep/
echo 1 > controllers/141a0000.pcie_ep/start
echo 0xe0000000 > /sys/kernel/debug/141a0000.pcie_ep_epf_dma_test/rp_dest
echo 0xFF00000 > /sys/kernel/debug/141a0000.pcie_ep_epf_dma_test/rp_size
echo 0x10000 > /sys/kernel/debug/141a0000.pcie_ep_epf_dma_test/dma_size
echo 0xf0000000 > /sys/kernel/debug/141a0000.pcie_ep_epf_dma_test/ep_src
On RP side there was a line as below. So I thought RP DMA address is 0xe0000000.
0xf0000000 was the BAR address assigned on EP side.
When RP side is up, It successfully found the enpoint. You can find the details of lspci output in attachments. lspci.txt (4.2 KB)
I called following commands;
setpci -s 0005:01:00.0 COMMAND=0x02
setpci -s 0005:01:00.0 4.l=0x07
But, I have a problem at this point;
When I manually change f0100000 on EP side or 1c00000000 or e0100000 on RP side, I cant see it on the other side.
If I set a wrong address I always got decode address error or mc_err.
What I couldnt understand why I cant see my modifications on other side. I think my transaction is done, but why I cant see my modifications on other side.
Can you please guide/comment me on how to use pci-epf-dma-test?
Best Regards
Hello,
I just wanted to add more information.
Both RP and EP board has JetPack 5.2 and Linux version l4t-r35.4.1. The boards are identical custom carrier board with Nvidia AGX Xavier Industrial Module. RP ODMDATA is set to be RootComplex, EP side is set as endpoint device.
Boards are connected via external cables. On RP side lspci output shows they are successfully connected.
I used the source in the path nvidia/drivers/pci/endpoint/functions/pci-epf-dma-test.c to make a performance test using perf_test. As I shared it gives output that data is delivered to other side, but I cant see the updated data on the other side.
On RP side I follower the pcie_tegra194.c file, I understood that the line depicted below shows the DMAable address of RP side.
[ 6.822584] tegra194-pcie 141a0000.pcie: RP host DMA buf: 0xe0000000 size: 268435456n
On the other hand on EP side the following dmesg output I thpught is the EP DMAable region, which is also mapped to BAR address;
[ 59.619784] pcie_dma_epf tegra_pcie_dma_epf.0: BAR0 phy_addr: f0000000 size: 10000000
I used devmem function to change some data so as to see it on other side.
Accordingly, I used these addresses with perf_test/pci-epf-dma-test.c to start transaction from 0xf0000000 to 0xe0000000 on EP side. When it is completed it doesnt give any error, it looks like data is transfered successfully. But on RP side I couldnt see at the physical address I inserted. I also tried to use the read command, there was no mc_err or smmu error, but I couldn’t see the data on EP side. I will really appreciate, If you can give a quick response.***
Thank you very much.
Best Regards
Hello @kayccc
Thank you for your response. I attached the dmesg and console output for both RP and EP side. And for the EP side, rp_phy_addr parameter wasn’t assigned to any value. Because of that I made some code changes in pci-epf-dma-test.c file, which is also attached.
From RP side, it seems sanity also works. From EP side, even if I set the correct addresses, I received errors even during dma_rx/tx transaction. You can examine the attached files.
What is interesting is when sanity fails on EP side, I saw mc_err output on EP side. And I am sure I didn’t set 0x0000000 address on EP.
I initiated also the perf_test again.
Best Regards console-ep.txt (5.6 KB) console-rp.txt (4.2 KB) dmesg-ep.txt (64.8 KB) dmesg-rp.txt (66.9 KB) console-ep.txt (5.6 KB) pci-epf-dma-test.c.txt (49.7 KB)
I also inserted one more print in edma_submit_direct_tx and edma_submit_direct_rx in order to be sure if the addresses copied correctly. As you can see below, addresses seem correct. Best Regards
Hello @kayccc ,
I made a little proceed. I want to share it.
I noticed the line below. All other test initializes edma with lie=1.
edma_init(epfnv, 1/0/);
So when I changed to lie=1, now edma_submit_direct_tx is completed. But now in crc, written data crc is totally different than BAR data. (4bf364eb!=0)
I think bar0_virt and epf_bar0->ep_phy_addr (or epf_bar->phys_addr) might point to different addresses.
Best Regards
0x1c00000000 is the RP side BAR address. 0xf0000000 is the ep_phy_addr. For nv_test, devmem can be used to modify it and the modification could be observed on other side. In our case all data is different.
The other question is how sanity works? epf_bar0->wr_data[i].crc isn’t set. So it returns 0 for crc.
Hi @WayneWWW,
Yes it works as can be seen below. I tried from EP side.
I set RP phy address to 0xe0000000.
Do you have any further comment?
Best Regards
root@jetson-agx-xavier-industrial:/sys/kernel/debug/141a0000.pcie_ep_epf_dma_test# cat edmalib_test
[ 185.542631] pcie_dma_epf tegra_pcie_dma_epf.0: edmalib_common_test: re-init edma lib prev_ch(0) != current chans(11)
[ 185.544906] tegra194-pcie 141a0000.pcie_ep: tegra_pcie_edma_initialize: success
[ 185.546092] pcie_dma_epf tegra_pcie_dma_epf.0: edmalib_common_test: EDMA LIB WR started for 1 chans, size 65536 Bytes, iterations: 1000 of descriptors 4
[ 185.547275] pcie_dma_epf tegra_pcie_dma_epf.0: edmalib_common_test: EDMA LIB submit done
[ 185.868507] pcie_dma_epf tegra_pcie_dma_epf.0: edma_final_complete: WR-local-Async complete for chan 0 with status 0. Total desc 4000 of Sz 65536 Bytes done in time 322399168 nsec. Perf is 6504 Mbps
[ 185.868526] pcie_dma_epf tegra_pcie_dma_epf.0: edma_final_complete: All Async channels. Cumulative Perf 6504 Mbps, time 322399264 nsec
There is no RP → EP interrupt support in dma test driver, that’s why CRC information from RP is not received by EP. This caused wait timeout and CRC check failure.
In Tegra vnet driver, we are using host1x interrupts for RP → EP.