The ethernet interface over PCIe(endpoint mode) not work

Hi @WayneWWW

I borrowed a AGX Orin dev kit and replaced the AGX Xavier RC side with it. Then I have a comparison test, you can refer the test set-2 in the test cases below.

Summary:

  1. The ethernet communication between EP and RC is workable in test set-2
  2. The bitrate from RC to EP is less than 4Gbits/s
  3. The bitrate from EP to RC is less than 7Gbits/s

Question:

  1. Why the result is different between AGX Xavier and AGX Orin with the same JP 5.1?
  2. The throughput seems not to meet PCIe Gen.3 x8(Less than 10%), even though I set the power mode to MAXN.

Test Cases:
PCIe_InterCon_Compare

EP Side test log:

root@tegra-ubuntu:/sys/kernel/config/pci_ep# ifconfig eth1
eth1: flags=4098<BROADCAST,MULTICAST>  mtu 64512
        ether fa:a6:fa:3a:19:bf  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

root@tegra-ubuntu:/sys/kernel/config/pci_ep# ifconfig eth1 up
root@tegra-ubuntu:/sys/kernel/config/pci_ep# [  333.694349] NOHZ tick-stop error: Non-RCU local softirq work is pending, handler #08!!!
[  333.694576] NOHZ tick-stop error: Non-RCU local softirq work is pending, handler #08!!!
[  333.831660] NOHZ tick-stop error: Non-RCU local softirq work is pending, handler #08!!!
[  333.831938] NOHZ tick-stop error: Non-RCU local softirq work is pending, handler #08!!!
[  333.866746] NOHZ tick-stop error: Non-RCU local softirq work is pending, handler #08!!!
[  333.867004] NOHZ tick-stop error: Non-RCU local softirq work is pending, handler #08!!!
[  334.370709] NOHZ tick-stop error: Non-RCU local softirq work is pending, handler #08!!!
[  334.370985] NOHZ tick-stop error: Non-RCU local softirq work is pending, handler #08!!!
[  334.594560] NOHZ tick-stop error: Non-RCU local softirq work is pending, handler #08!!!
[  334.594821] NOHZ tick-stop error: Non-RCU local softirq work is pending, handler #08!!!

root@tegra-ubuntu:/sys/kernel/config/pci_ep# ifconfig eth1
eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 64512
        inet 10.10.10.2  netmask 255.255.255.0  broadcast 10.10.10.255
        inet6 fe80::f8a6:faff:fe3a:19bf  prefixlen 64  scopeid 0x20<link>
        ether fa:a6:fa:3a:19:bf  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

root@tegra-ubuntu:/sys/kernel/config/pci_ep# ping 10.10.10.1
PING 10.10.10.1 (10.10.10.1) 56(84) bytes of data.
64 bytes from 10.10.10.1: icmp_seq=1 ttl=64 time=0.597 ms
64 bytes from 10.10.10.1: icmp_seq=2 ttl=64 time=1.27 ms
64 bytes from 10.10.10.1: icmp_seq=3 ttl=64 time=1.27 ms
64 bytes from 10.10.10.1: icmp_seq=4 ttl=64 time=1.21 ms
^C
--- 10.10.10.1 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3005ms
rtt min/avg/max/mdev = 0.597/1.085/1.272/0.283 ms
root@tegra-ubuntu:/sys/kernel/config/pci_ep# iperf3 -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 10.10.10.1, port 47622
[  5] local 10.10.10.2 port 5201 connected to 10.10.10.1 port 47624
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   418 MBytes  3.51 Gbits/sec                  
[  5]   1.00-2.00   sec   414 MBytes  3.47 Gbits/sec                  
[  5]   2.00-3.00   sec   412 MBytes  3.45 Gbits/sec                  
[  5]   3.00-4.00   sec   409 MBytes  3.43 Gbits/sec                  
[  5]   4.00-5.00   sec   408 MBytes  3.43 Gbits/sec                  
[  5]   5.00-6.00   sec   408 MBytes  3.42 Gbits/sec                  
[  5]   6.00-7.00   sec   411 MBytes  3.45 Gbits/sec                  
[  5]   7.00-8.00   sec   407 MBytes  3.41 Gbits/sec                  
[  5]   8.00-9.00   sec   406 MBytes  3.41 Gbits/sec                  
[  5]   9.00-10.00  sec   406 MBytes  3.41 Gbits/sec                  
[  5]  10.00-10.00  sec   831 KBytes  3.30 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  4.00 GBytes  3.44 Gbits/sec                  receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
^Ciperf3: interrupt - the server has terminated
root@tegra-ubuntu:/sys/kernel/config/pci_ep# iperf3 -c 10.10.10.1
Connecting to host 10.10.10.1, port 5201
[  5] local 10.10.10.2 port 60002 connected to 10.10.10.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   782 MBytes  6.56 Gbits/sec    0   3.07 MBytes       
[  5]   1.00-2.00   sec   742 MBytes  6.23 Gbits/sec    0   3.07 MBytes       
[  5]   2.00-3.00   sec   739 MBytes  6.19 Gbits/sec    0   3.07 MBytes       
[  5]   3.00-4.00   sec   750 MBytes  6.30 Gbits/sec    0   3.07 MBytes       
[  5]   4.00-5.00   sec   756 MBytes  6.34 Gbits/sec    0   3.07 MBytes       
[  5]   5.00-6.00   sec   780 MBytes  6.54 Gbits/sec    0   3.07 MBytes       
[  5]   6.00-7.00   sec   791 MBytes  6.64 Gbits/sec    0   3.07 MBytes       
[  5]   7.00-8.00   sec   798 MBytes  6.69 Gbits/sec    0   3.07 MBytes       
[  5]   8.00-9.00   sec   784 MBytes  6.58 Gbits/sec    0   3.07 MBytes       
[  5]   9.00-10.00  sec   772 MBytes  6.48 Gbits/sec    0   3.07 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  7.51 GBytes  6.45 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  7.51 GBytes  6.45 Gbits/sec                  receiver

iperf Done.
root@tegra-ubuntu:/sys/kernel/config/pci_ep# 

RC Side test log:

root@tegra-ubuntu:/home/denso# ifconfig eth0
eth0: flags=4098<BROADCAST,MULTICAST>  mtu 64512
        ether f6:a1:f9:f6:21:a6  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

root@tegra-ubuntu:/home/denso# ifconfig eth0 up
root@tegra-ubuntu:/home/denso# ifconfig eth0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 64512
        inet 10.10.10.1  netmask 255.255.255.0  broadcast 10.10.10.255
        inet6 fe80::f4a1:f9ff:fef6:21a6  prefixlen 64  scopeid 0x20<link>
        ether f6:a1:f9:f6:21:a6  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

root@tegra-ubuntu:/home/denso# ping 10.10.10.2
PING 10.10.10.2 (10.10.10.2) 56(84) bytes of data.
64 bytes from 10.10.10.2: icmp_seq=1 ttl=64 time=3.18 ms
64 bytes from 10.10.10.2: icmp_seq=2 ttl=64 time=1.66 ms
64 bytes from 10.10.10.2: icmp_seq=3 ttl=64 time=1.65 ms
64 bytes from 10.10.10.2: icmp_seq=4 ttl=64 time=1.60 ms
^C
--- 10.10.10.2 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3006ms
rtt min/avg/max/mdev = 1.599/2.023/3.180/0.668 ms
root@tegra-ubuntu:/home/denso# iperf3 -c 10.10.10.2
Connecting to host 10.10.10.2, port 5201
[  5] local 10.10.10.1 port 47624 connected to 10.10.10.2 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   418 MBytes  3.51 Gbits/sec    0   1.35 MBytes       
[  5]   1.00-2.00   sec   414 MBytes  3.47 Gbits/sec    0   1.35 MBytes       
[  5]   2.00-3.00   sec   412 MBytes  3.45 Gbits/sec    0   1.35 MBytes       
[  5]   3.00-4.00   sec   408 MBytes  3.43 Gbits/sec    0   1.35 MBytes       
[  5]   4.00-5.00   sec   409 MBytes  3.43 Gbits/sec    0   1.35 MBytes       
[  5]   5.00-6.00   sec   409 MBytes  3.42 Gbits/sec    0   2.09 MBytes       
[  5]   6.00-7.00   sec   410 MBytes  3.45 Gbits/sec    0   2.09 MBytes       
[  5]   7.00-8.00   sec   408 MBytes  3.41 Gbits/sec    0   2.09 MBytes       
[  5]   8.00-9.00   sec   406 MBytes  3.41 Gbits/sec    0   2.09 MBytes       
[  5]   9.00-10.00  sec   406 MBytes  3.41 Gbits/sec    0   2.09 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  4.00 GBytes  3.44 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  4.00 GBytes  3.44 Gbits/sec                  receiver

iperf Done.
root@tegra-ubuntu:/home/denso# iperf3 -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 10.10.10.2, port 60000
[  5] local 10.10.10.1 port 5201 connected to 10.10.10.2 port 60002
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   774 MBytes  6.49 Gbits/sec                  
[  5]   1.00-2.00   sec   749 MBytes  6.29 Gbits/sec                  
[  5]   2.00-3.00   sec   738 MBytes  6.19 Gbits/sec                  
[  5]   3.00-4.00   sec   750 MBytes  6.29 Gbits/sec                  
[  5]   4.00-5.00   sec   756 MBytes  6.34 Gbits/sec                  
[  5]   5.00-6.00   sec   780 MBytes  6.54 Gbits/sec                  
[  5]   6.00-7.00   sec   791 MBytes  6.64 Gbits/sec                  
[  5]   7.00-8.00   sec   798 MBytes  6.69 Gbits/sec                  
[  5]   8.00-9.00   sec   785 MBytes  6.58 Gbits/sec                  
[  5]   9.00-10.00  sec   772 MBytes  6.48 Gbits/sec                  
[  5]  10.00-10.00  sec  1.50 MBytes  6.08 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  7.51 GBytes  6.45 Gbits/sec                  receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
^Ciperf3: interrupt - the server has terminated

root@tegra-ubuntu:/home/denso# dmesg | grep -i pcie
[    7.591710] tegra194-pcie 14100000.pcie: Adding to iommu group 9
[    7.604338] tegra194-pcie 14100000.pcie: Using GICv2m MSI allocator
[    7.611451] tegra194-pcie 14160000.pcie: Adding to iommu group 10
[    7.623846] tegra194-pcie 14160000.pcie: Using GICv2m MSI allocator
[    7.630651] tegra194-pcie 141a0000.pcie: Adding to iommu group 11
[    7.642535] tegra194-pcie 141a0000.pcie: Using GICv2m MSI allocator
[    7.649500] tegra194-pcie 141a0000.pcie: Failed to get slot regulators: -517
[    9.929852] tegra194-pcie 14100000.pcie: Using GICv2m MSI allocator
[    9.944360] tegra194-pcie 14100000.pcie: host bridge /pcie@14100000 ranges:
[    9.951534] tegra194-pcie 14100000.pcie:       IO 0x0030100000..0x00301fffff -> 0x0030100000
[    9.963943] tegra194-pcie 14100000.pcie:      MEM 0x20a8000000..0x20afffffff -> 0x0040000000
[    9.980207] tegra194-pcie 14100000.pcie:      MEM 0x2080000000..0x20a7ffffff -> 0x2080000000
[   10.099975] tegra194-pcie 14100000.pcie: Link up
[   10.106675] tegra194-pcie 14100000.pcie: PCI host bridge to bus 0001:00
[   10.318073] pcieport 0001:00:00.0: Adding to iommu group 9
[   10.330202] pcieport 0001:00:00.0: PME: Signaling with IRQ 51
[   10.336442] pcieport 0001:00:00.0: AER: enabled with IRQ 51
[   10.343009] tegra194-pcie 14160000.pcie: Using GICv2m MSI allocator
[   10.350672] tegra194-pcie 14160000.pcie: host bridge /pcie@14160000 ranges:
[   10.364574] tegra194-pcie 14160000.pcie:       IO 0x0036100000..0x00361fffff -> 0x0036100000
[   10.373269] tegra194-pcie 14160000.pcie:      MEM 0x2428000000..0x242fffffff -> 0x0040000000
[   10.381970] tegra194-pcie 14160000.pcie:      MEM 0x2140000000..0x2427ffffff -> 0x2140000000
[   11.496264] tegra194-pcie 14160000.pcie: Phy link never came up
[   11.502439] tegra194-pcie 14160000.pcie: PCI host bridge to bus 0004:00
[   11.578175] pcieport 0004:00:00.0: Adding to iommu group 10
[   11.584176] pcieport 0004:00:00.0: PME: Signaling with IRQ 53
[   11.590456] pcieport 0004:00:00.0: AER: enabled with IRQ 53
[   11.615584] tegra194-pcie 141a0000.pcie: Using GICv2m MSI allocator
[   11.622152] tegra194-pcie 141a0000.pcie: Failed to get slot regulators: -517
[   11.622247] vdd-3v3-pcie: supplied by vdd-3v3-sys
[   11.681432] tegra194-pcie 141a0000.pcie: Using GICv2m MSI allocator
[   11.797188] tegra194-pcie 141a0000.pcie: host bridge /pcie@141a0000 ranges:
[   11.804375] tegra194-pcie 141a0000.pcie:       IO 0x003a100000..0x003a1fffff -> 0x003a100000
[   11.813055] tegra194-pcie 141a0000.pcie:      MEM 0x2b28000000..0x2b2fffffff -> 0x0040000000
[   11.821733] tegra194-pcie 141a0000.pcie:      MEM 0x2740000000..0x2b27ffffff -> 0x2740000000
[   11.936120] tegra194-pcie 141a0000.pcie: Link up
[   11.943017] tegra194-pcie 141a0000.pcie: PCI host bridge to bus 0005:00
[   12.036410] pci 0005:01:00.0: 63.008 Gb/s available PCIe bandwidth, limited by 8.0 GT/s PCIe x8 link at 0005:00:00.0 (capable of 126.024 Gb/s with 16.0 GT/s PCIe x8 link)
[   12.134372] pcieport 0005:00:00.0: Adding to iommu group 11
[   12.140347] pcieport 0005:00:00.0: PME: Signaling with IRQ 55
[   12.146620] pcieport 0005:00:00.0: AER: enabled with IRQ 55
root@tegra-ubuntu:/home/denso#