Fail to Testing Bidirectional Data Transfer

I followed the steps in this document to test
https://docs.nvidia.com/jetson/archives/r35.1/DeveloperGuide/text/SD/Communications/PcieEndpointMode.html

When it comes to Testing Bidirectional Data Transfer, and i execute this command:

# Endpoint
busybox devmem 0x106896000 32 0x98765432
# Root port
busybox devmem 0x94534000

The result displayed by the Root port is

0x00000000

My PCIe EP device is A Jetson AGX Orin series device, Related information is as follows:

# uname -a
Linux autra-orin 5.10.104-tegra #2 SMP PREEMPT Wed Nov 2 17:24:30 CST 2022 aarch64 aarch64 aarch64 GNU/Linux

# head -1 /etc/nv_tegra_release 
# R35 (release), REVISION: 1.0, GCID: 31346300, BOARD: t186ref, EABI: aarch64, DATE: Thu Aug 25 18:41:45 UTC 2022

# dmesg | grep pci_epf_nv_test
[  737.357899] pci_epf_nv_test pci_epf_nv_test.0: BAR0 RAM phys: 0x106896000
[  737.357928] pci_epf_nv_test pci_epf_nv_test.0: BAR0 RAM IOVA: 0xffff0000
[  737.357975] pci_epf_nv_test pci_epf_nv_test.0: BAR0 RAM virt: 0x00000000d8c93981

My PCIe Root Port device is a standard x86-64 PC that is running Linux:

# uname -a
Linux autra-Nuvo-8208GC-Series 5.13.0-52-generic #59~20.04.1-Ubuntu SMP Thu Jun 16 21:21:28 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

# lspci
00:00.0 Host bridge: Intel Corporation 8th Gen Core 8-core Desktop Processor Host Bridge/DRAM Registers [Coffee Lake S] (rev 0d)
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x16) (rev 0d)
00:01.1 PCI bridge: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x8) (rev 0d)
00:12.0 Signal processing controller: Intel Corporation Cannon Lake PCH Thermal Controller (rev 10)
00:14.0 USB controller: Intel Corporation Cannon Lake PCH USB 3.1 xHCI Host Controller (rev 10)
00:14.2 RAM memory: Intel Corporation Cannon Lake PCH Shared SRAM (rev 10)
00:16.0 Communication controller: Intel Corporation Cannon Lake PCH HECI Controller (rev 10)
00:16.3 Serial controller: Intel Corporation Cannon Lake PCH Active Management Technology - SOL (rev 10)
00:17.0 SATA controller: Intel Corporation Cannon Lake PCH SATA AHCI Controller (rev 10)
00:1b.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #17 (rev f0)
00:1c.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #4 (rev f0)
00:1c.4 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #5 (rev f0)
00:1d.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #15 (rev f0)
00:1f.0 ISA bridge: Intel Corporation Cannon Point-LP LPC Controller (rev 10)
00:1f.3 Audio device: Intel Corporation Cannon Lake PCH cAVS (rev 10)
00:1f.4 SMBus: Intel Corporation Cannon Lake PCH SMBus Controller (rev 10)
00:1f.5 Serial bus controller [0c80]: Intel Corporation Cannon Lake PCH SPI Controller (rev 10)
00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (7) I219-LM (rev 10)
01:00.0 VGA compatible controller: NVIDIA Corporation Device 2216 (rev a1)
01:00.1 Audio device: NVIDIA Corporation Device 1aef (rev a1)
07:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd Device a809
08:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
09:00.0 Memory controller: Xilinx Corporation Device 7024
0a:00.0 Network controller: PEAK-System Technik GmbH Device 0018 (rev 01)

# lspci -v -s 00:14.2   (RAM Memory)
00:14.2 RAM memory: Intel Corporation Cannon Lake PCH Shared SRAM (rev 10)
	Subsystem: Intel Corporation Cannon Lake PCH Shared SRAM
	Flags: bus master, fast devsel, latency 0
	Memory at 94534000 (64-bit, non-prefetchable) [size=8K]
	Memory at 94539000 (64-bit, non-prefetchable) [size=4K]
	Capabilities: [80] Power Management version 3

These two devices are also connected normally with PCIe cable:

So, is there any way I can debug it now?

please check

Thank you first.

I repeat what you did on x86 host.

but i can’t update value at RAM address:

root@autra-Nuvo-8208GC-Series:~# uname -a
Linux autra-Nuvo-8208GC-Series 5.13.0-52-generic #59~20.04.1-Ubuntu SMP Thu Jun 16 21:21:28 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
root@autra-Nuvo-8208GC-Series:~# devmem2 0x94534000 w 0x67891234
/dev/mem opened.
Memory mapped at address 0x7f6182cdc000.
Value at address 0x94534000 (0x7f6182cdc000): 0x0
Written 0x67891234; readback 0x0

Could you give me some advice, how to debug?

Does your Orin side disable CONFIG_STRICT_DEVMEMM config in kernel?

Yes

root@autra-orin:/sys/kernel/config/pci_ep# zcat /proc/config.gz |grep CONFIG_STRICT_DEVMEMM
root@autra-orin:/sys/kernel/config/pci_ep# 
root@autra-orin:/sys/kernel/config/pci_ep# dmesg | grep pci_epf_nv_test
[  737.357899] pci_epf_nv_test pci_epf_nv_test.0: BAR0 RAM phys: 0x106896000
[  737.357928] pci_epf_nv_test pci_epf_nv_test.0: BAR0 RAM IOVA: 0xffff0000
[  737.357975] pci_epf_nv_test pci_epf_nv_test.0: BAR0 RAM virt: 0x00000000d8c93981
root@autra-orin:/sys/kernel/config/pci_ep# devmem2 0x106896000 w 0x67891234
/dev/mem opened.
Memory mapped at address 0xffffb3c0f000.
Value at address 0x6896000 (0xffffb3c0f000): 0x55AA5A5A
Written 0x67891234; readback 0x67891234
root@autra-orin:/sys/kernel/config/pci_ep# devmem2 0x106896000 w 0x67891235
/dev/mem opened.
Memory mapped at address 0xffff8ca1f000.
Value at address 0x6896000 (0xffff8ca1f000): 0x67891234
Written 0x67891235; readback 0x67891235
root@autra-orin:/sys/kernel/config/pci_ep# 

I am a little confused here. Why is your device here to be

00:14.2 RAM memory: Intel Corporation Cannon Lake PCH Shared SRAM (rev 10)

As another user’s test and our local test, even the x86 host side shall see the jetson as RAM memory: NVIDIA Corporation Device 0001…

Yes, I noticed this too, I’ll check this first

By the way, did the RAM * Nvidia information you show increase after the two devices are connected, or is it the x86 itself?

Another question, does the x86 host need some additional adaptation, or can it be directly connected to the orin through a PCIe cable?

I found the reason, ignore my question above…

after run this command:

# echo 1 > controllers/141a0000.pcie_ep/start

I need to reboot the x86 machine, and then i found the correct RAM memory.

thanks man !

1 Like

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.