PCIE BandWidth test Between Agx Orin and Orin NX

Robotics & Edge Computing Jetson Systems Jetson AGX Orin
,
1

Hi,

We use our Custom design Carrier Board and successfully used PCIE connection between AGX orin and Orin nx.

The Jetpack version is 6.0 with l4t 36.3.0. The Jetson Orin Nx 16G is the EP (Use C4). And The Jetson AGX orin 64G is the RP(C5). The connection use 4 * pcie4.0 lane.

After successfully load the pcie-ep_epf_dma_test, we could see the log below 54116MB which is a reasonable value of the 4 lane PCIE bandwidth.

root@tegra-ubuntu:/sys/kernel/debug/14160000.pcie-ep_epf_dma_test# cat edmalib_test
root@tegra-ubuntu:/sys/kernel/debug/14160000.pcie-ep_epf_dma_test# dmesg |tail
[  559.146931] r8168: eth0: link up
[  574.447729] tegra194-pcie 14160000.pcie-ep: iATU unroll: enabled
[  574.447742] tegra194-pcie 14160000.pcie-ep: Detected iATU regions: 8 outbound, 2 inbound
[  574.447836] pcie_dma_epf tegra_pcie_dma_epf.0: BAR0 phy_addr: f0000000 size: 10000000
[  908.543849] pcie_dma_epf tegra_pcie_dma_epf.0: edmalib_common_test: re-init edma lib prev_ch(0) != current chans(11)
[  908.544236] tegra194-pcie 14160000.pcie-ep: tegra_pcie_edma_initialize: success
[  908.749046] pcie_dma_epf tegra_pcie_dma_epf.0: edmalib_common_test: EDMA LIB WR started for 1 chans, size 16777216 Bytes, iterations: 1000 of descriptors 4
[  908.751127] pcie_dma_epf tegra_pcie_dma_epf.0: edmalib_common_test: EDMA LIB submit done
[  918.669790] pcie_dma_epf tegra_pcie_dma_epf.0: edma_final_complete: WR-local-Async complete for chan 0 with status 0. Total desc 4000 of Sz 16777216 Bytes done in time 9920729312 nsec. Perf is 54116 Mbps
[  918.669807] pcie_dma_epf tegra_pcie_dma_epf.0: edma_final_complete: All Async channels. Cumulative Perf 54116 Mbps, time 9920729440 nsec

With the help of the nvidia doc, I tried the pcie-epf-tegra-vnet to use virtual network upon pcie bus.

On EP

modprobe pci-epf-tegra-vnet

1. Endpoint Setup (On the Endpoint System):
Enable PCIe Endpoint Mode:
The first step is to enable PCIe endpoint mode on the system that will act as the endpoint. This involves configuring the PCIe controller and enabling the endpoint functionality.
Create the pci_epf_tvnet function:
Navigate to the configuration directory for PCIe endpoints and create a directory for the pci_epf_tvnet function: 

    cd /sys/kernel/config/pci_ep/
    mkdir -p functions/pci_epf_tvnet/func1
Configure MSI Interrupts: The pci_epf_tvnet function uses MSI interrupts. Configure the number of MSI interrupts to be used:

    echo 16 > functions/pci_epf_tvnet/func1/msi_interrupts
Bind to the Controller: Link the pci_epf_tvnet function to the PCIe controller:

    ln -s functions/pci_epf_tvnet/func1 controllers/14160000.pcie-ep/ （NX）
Start the Endpoint: Enable the endpoint functionality:

    echo 1 > controllers/14160000.pcie-ep/start （NX）
Alternative Setup: The endpoint can also be set up by using the pci_epf_nv_test driver and then switching the pci_epf_nv_test directory to pci_epf_tvnet. This method is useful if the pci_epf_tvnet driver is not available directly:

After RP power on, I could see the log below.

# [  310.269013] tegra194-pcie 14160000.pcie-ep: iATU unroll: enabled
# [  310.269063] tegra194-pcie 14160000.pcie-ep: Detected iATU regions: 8 outbound, 2 inbound
# [  317.918948] r8168: eth0: link down
# [  321.037724] r8168: eth0: link up
# [  336.134924] tegra194-pcie 14160000.pcie-ep: iATU unroll: enabled
# [  336.134954] tegra194-pcie 14160000.pcie-ep: Detected iATU regions: 8 outbound, 2 inbound
# [ 1409.187771] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready

And then on RP, modprobe tegra-vnet

# [  330.999755] nvethernet 6800000.ethernet: [poll_check][42][type:0x4][loga-0x0] poll_check: timeout
# [  330.999767] nvethernet 6800000.ethernet: ether_open: failed to initialize MAC HW core with reason -1
# [ 1083.182795] tvnet 0005:01:00.0: Adding to iommu group 6
# [ 1083.182974] tvnet 0005:01:00.0: enabling device (0000 -> 0002)
# [ 1204.193715] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready

After that, I setup the static ip for both eth1, with NX 192.168.1.100, and AGX 192.168.1.2

ip addr add 192.168.1.100/24 dev eth1
ip link set dev eth1 up
ifconfig

# eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 64512
#        inet 192.168.1.100  netmask 255.255.255.0  broadcast 0.0.0.0
#        ether ca:53:cd:85:c3:8f  txqueuelen 1000  (Ethernet)
#        RX packets 0  bytes 0 (0.0 B)
#        RX errors 0  dropped 0  overruns 0  frame 0
#        TX packets 0  bytes 0 (0.0 B)
#        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

And then the command ping 192.168.1.2 on NX is success.

root@tegra-ubuntu:/home/hihonor# ping 192.168.1.2
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=64 time=4.81 ms
64 bytes from 192.168.1.2: icmp_seq=2 ttl=64 time=3.58 ms
64 bytes from 192.168.1.2: icmp_seq=3 ttl=64 time=1.67 ms
64 bytes from 192.168.1.2: icmp_seq=4 ttl=64 time=3.39 ms
64 bytes from 192.168.1.2: icmp_seq=5 ttl=64 time=1.53 ms
64 bytes from 192.168.1.2: icmp_seq=6 ttl=64 time=3.90 ms
64 bytes from 192.168.1.2: icmp_seq=7 ttl=64 time=2.63 ms

By using the iperf3 command to test the bandwidth and the speed of the PCIE as vnet, I found that the value of bandwidth is unreasonable. The log shows below. It shows that the bandwidth is only 5Gbps only 10% of the PCIE bandwith. And the latency is high. Is there anything wrong in my experiment?

root@tegra-ubuntu:/sys/kernel/config/pci_ep# iperf3 -c 192.168.1.100 -P 1 -u -b 100000M
Connecting to host 192.168.1.100, port 5201
[  5] local 192.168.1.2 port 59488 connected to 192.168.1.100 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec   525 MBytes  4.41 Gbits/sec  17092
[  5]   1.00-2.00   sec   546 MBytes  4.58 Gbits/sec  17775
[  5]   2.00-3.00   sec   546 MBytes  4.58 Gbits/sec  17754
[  5]   3.00-4.00   sec   548 MBytes  4.59 Gbits/sec  17814
[  5]   4.00-5.00   sec   546 MBytes  4.58 Gbits/sec  17761
[  5]   5.00-6.00   sec   547 MBytes  4.59 Gbits/sec  17800
[  5]   6.00-7.00   sec   546 MBytes  4.58 Gbits/sec  17766
[  5]   7.00-8.00   sec   547 MBytes  4.58 Gbits/sec  17780
[  5]   8.00-9.00   sec   546 MBytes  4.58 Gbits/sec  17764
[  5]   9.00-10.00  sec   547 MBytes  4.59 Gbits/sec  17805
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  5.32 GBytes  4.57 Gbits/sec  0.000 ms  0/177111 (0%)  sender
[  5]   0.00-10.06  sec  2.80 GBytes  2.39 Gbits/sec  0.065 ms  83750/177107 (47%)  receiver

iperf Done.
root@tegra-ubuntu:/sys/kernel/config/pci_ep# iperf3 -c 192.168.1.100 -P 4 -u -b 100000M
Connecting to host 192.168.1.100, port 5201
[  5] local 192.168.1.2 port 46702 connected to 192.168.1.100 port 5201
[  7] local 192.168.1.2 port 33955 connected to 192.168.1.100 port 5201
[  9] local 192.168.1.2 port 38219 connected to 192.168.1.100 port 5201
[ 11] local 192.168.1.2 port 48866 connected to 192.168.1.100 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec   133 MBytes  1.12 Gbits/sec  4333
[  7]   0.00-1.00   sec   132 MBytes  1.11 Gbits/sec  4300
[  9]   0.00-1.00   sec   131 MBytes  1.10 Gbits/sec  4269
[ 11]   0.00-1.00   sec   130 MBytes  1.09 Gbits/sec  4239
[SUM]   0.00-1.00   sec   527 MBytes  4.42 Gbits/sec  17141
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.00-2.00   sec   137 MBytes  1.15 Gbits/sec  4448
[  7]   1.00-2.00   sec   136 MBytes  1.14 Gbits/sec  4419
[  9]   1.00-2.00   sec   135 MBytes  1.13 Gbits/sec  4378
[ 11]   1.00-2.00   sec   134 MBytes  1.12 Gbits/sec  4345
[SUM]   1.00-2.00   sec   541 MBytes  4.54 Gbits/sec  17590
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   2.00-3.00   sec   137 MBytes  1.15 Gbits/sec  4442
[  7]   2.00-3.00   sec   136 MBytes  1.14 Gbits/sec  4414
[  9]   2.00-3.00   sec   134 MBytes  1.13 Gbits/sec  4374
[ 11]   2.00-3.00   sec   133 MBytes  1.12 Gbits/sec  4338
[SUM]   2.00-3.00   sec   540 MBytes  4.53 Gbits/sec  17568
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   3.00-4.00   sec   137 MBytes  1.15 Gbits/sec  4455
[  7]   3.00-4.00   sec   135 MBytes  1.14 Gbits/sec  4407
[  9]   3.00-4.00   sec   135 MBytes  1.13 Gbits/sec  4378
[ 11]   3.00-4.00   sec   133 MBytes  1.12 Gbits/sec  4341
[SUM]   3.00-4.00   sec   540 MBytes  4.53 Gbits/sec  17581
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   4.00-5.00   sec   137 MBytes  1.15 Gbits/sec  4448
[  7]   4.00-5.00   sec   136 MBytes  1.14 Gbits/sec  4412
[  9]   4.00-5.00   sec   135 MBytes  1.13 Gbits/sec  4378
[ 11]   4.00-5.00   sec   133 MBytes  1.12 Gbits/sec  4341
[SUM]   4.00-5.00   sec   540 MBytes  4.53 Gbits/sec  17579
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   5.00-6.00   sec   137 MBytes  1.15 Gbits/sec  4443
[  7]   5.00-6.00   sec   135 MBytes  1.14 Gbits/sec  4405
[  9]   5.00-6.00   sec   134 MBytes  1.13 Gbits/sec  4370
[ 11]   5.00-6.00   sec   133 MBytes  1.12 Gbits/sec  4341
[SUM]   5.00-6.00   sec   540 MBytes  4.53 Gbits/sec  17559
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   6.00-7.00   sec   136 MBytes  1.14 Gbits/sec  4439
[  7]   6.00-7.00   sec   135 MBytes  1.13 Gbits/sec  4401
[  9]   6.00-7.00   sec   134 MBytes  1.13 Gbits/sec  4370
[ 11]   6.00-7.00   sec   133 MBytes  1.12 Gbits/sec  4334
[SUM]   6.00-7.00   sec   539 MBytes  4.52 Gbits/sec  17544
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   7.00-8.00   sec   137 MBytes  1.15 Gbits/sec  4450
[  7]   7.00-8.00   sec   135 MBytes  1.14 Gbits/sec  4407
[  9]   7.00-8.00   sec   134 MBytes  1.13 Gbits/sec  4368
[ 11]   7.00-8.00   sec   133 MBytes  1.12 Gbits/sec  4336
[SUM]   7.00-8.00   sec   540 MBytes  4.53 Gbits/sec  17561
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   8.00-9.00   sec   137 MBytes  1.15 Gbits/sec  4458
[  7]   8.00-9.00   sec   136 MBytes  1.14 Gbits/sec  4419
[  9]   8.00-9.00   sec   135 MBytes  1.13 Gbits/sec  4384
[ 11]   8.00-9.00   sec   134 MBytes  1.12 Gbits/sec  4345
[SUM]   8.00-9.00   sec   541 MBytes  4.54 Gbits/sec  17606
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   9.00-10.00  sec   137 MBytes  1.15 Gbits/sec  4463
[  7]   9.00-10.00  sec   136 MBytes  1.14 Gbits/sec  4422
[  9]   9.00-10.00  sec   135 MBytes  1.13 Gbits/sec  4381
[ 11]   9.00-10.00  sec   133 MBytes  1.12 Gbits/sec  4341
[SUM]   9.00-10.00  sec   541 MBytes  4.54 Gbits/sec  17607
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  1.33 GBytes  1.14 Gbits/sec  0.000 ms  0/44379 (0%)  sender
[  5]   0.00-10.05  sec  1.33 GBytes  1.14 Gbits/sec  0.171 ms  0/44379 (0%)  receiver
[  7]   0.00-10.00  sec  1.32 GBytes  1.13 Gbits/sec  0.000 ms  0/44006 (0%)  sender
[  7]   0.00-10.05  sec  1.32 GBytes  1.13 Gbits/sec  0.173 ms  0/44006 (0%)  receiver
[  9]   0.00-10.00  sec  1.31 GBytes  1.13 Gbits/sec  0.000 ms  0/43650 (0%)  sender
[  9]   0.00-10.05  sec  1.31 GBytes  1.12 Gbits/sec  0.170 ms  0/43650 (0%)  receiver
[ 11]   0.00-10.00  sec  1.30 GBytes  1.12 Gbits/sec  0.000 ms  0/43301 (0%)  sender
[ 11]   0.00-10.05  sec  1.30 GBytes  1.11 Gbits/sec  0.180 ms  0/43300 (0%)  receiver
[SUM]   0.00-10.00  sec  5.26 GBytes  4.52 Gbits/sec  0.000 ms  0/175336 (0%)  sender
[SUM]   0.00-10.05  sec  5.26 GBytes  4.50 Gbits/sec  0.174 ms  0/175335 (0%)  receiver

iperf Done.
root@tegra-ubuntu:/sys/kernel/config/pci_ep# iperf3 -c 192.168.1.100 -P 10 -u -b 100000M
Connecting to host 192.168.1.100, port 5201
[  5] local 192.168.1.2 port 44490 connected to 192.168.1.100 port 5201
[  7] local 192.168.1.2 port 44744 connected to 192.168.1.100 port 5201
[  9] local 192.168.1.2 port 50537 connected to 192.168.1.100 port 5201
[ 11] local 192.168.1.2 port 39873 connected to 192.168.1.100 port 5201
[ 13] local 192.168.1.2 port 42957 connected to 192.168.1.100 port 5201
[ 15] local 192.168.1.2 port 60653 connected to 192.168.1.100 port 5201
[ 17] local 192.168.1.2 port 39441 connected to 192.168.1.100 port 5201
[ 19] local 192.168.1.2 port 45164 connected to 192.168.1.100 port 5201
[ 21] local 192.168.1.2 port 49658 connected to 192.168.1.100 port 5201
[ 23] local 192.168.1.2 port 60853 connected to 192.168.1.100 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec  52.7 MBytes   442 Mbits/sec  1713
[  7]   0.00-1.00   sec  52.4 MBytes   439 Mbits/sec  1704
[  9]   0.00-1.00   sec  51.8 MBytes   434 Mbits/sec  1684
[ 11]   0.00-1.00   sec  51.5 MBytes   432 Mbits/sec  1674
[ 13]   0.00-1.00   sec  51.1 MBytes   428 Mbits/sec  1661
[ 15]   0.00-1.00   sec  50.5 MBytes   423 Mbits/sec  1642
[ 17]   0.00-1.00   sec  50.1 MBytes   420 Mbits/sec  1629
[ 19]   0.00-1.00   sec  49.8 MBytes   417 Mbits/sec  1619
[ 21]   0.00-1.00   sec  49.1 MBytes   412 Mbits/sec  1598
[ 23]   0.00-1.00   sec  48.8 MBytes   409 Mbits/sec  1588
[SUM]   0.00-1.00   sec   508 MBytes  4.26 Gbits/sec  16512
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.00-2.00   sec  56.0 MBytes   470 Mbits/sec  1821
[  7]   1.00-2.00   sec  55.4 MBytes   465 Mbits/sec  1804
[  9]   1.00-2.00   sec  55.0 MBytes   462 Mbits/sec  1791
[ 11]   1.00-2.00   sec  54.5 MBytes   457 Mbits/sec  1774
[ 13]   1.00-2.00   sec  53.8 MBytes   451 Mbits/sec  1751
[ 15]   1.00-2.00   sec  53.5 MBytes   448 Mbits/sec  1739
[ 17]   1.00-2.00   sec  53.1 MBytes   445 Mbits/sec  1727
[ 19]   1.00-2.00   sec  52.7 MBytes   442 Mbits/sec  1716
[ 21]   1.00-2.00   sec  52.2 MBytes   438 Mbits/sec  1699
[ 23]   1.00-2.00   sec  51.7 MBytes   434 Mbits/sec  1683
[SUM]   1.00-2.00   sec   538 MBytes  4.51 Gbits/sec  17505
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   2.00-3.00   sec  56.0 MBytes   470 Mbits/sec  1821
[  7]   2.00-3.00   sec  55.4 MBytes   465 Mbits/sec  1803
[  9]   2.00-3.00   sec  55.1 MBytes   462 Mbits/sec  1793
[ 11]   2.00-3.00   sec  54.6 MBytes   458 Mbits/sec  1775
[ 13]   2.00-3.00   sec  54.1 MBytes   454 Mbits/sec  1761
[ 15]   2.00-3.00   sec  53.8 MBytes   451 Mbits/sec  1750
[ 17]   2.00-3.00   sec  53.4 MBytes   448 Mbits/sec  1737
[ 19]   2.00-3.00   sec  53.0 MBytes   445 Mbits/sec  1725
[ 21]   2.00-3.00   sec  52.5 MBytes   441 Mbits/sec  1709
[ 23]   2.00-3.00   sec  51.9 MBytes   435 Mbits/sec  1688
[SUM]   2.00-3.00   sec   540 MBytes  4.53 Gbits/sec  17562
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   3.00-4.00   sec  56.0 MBytes   470 Mbits/sec  1822
[  7]   3.00-4.00   sec  55.4 MBytes   464 Mbits/sec  1801
[  9]   3.00-4.00   sec  55.0 MBytes   461 Mbits/sec  1788
[ 11]   3.00-4.00   sec  54.6 MBytes   458 Mbits/sec  1775
[ 13]   3.00-4.00   sec  54.2 MBytes   455 Mbits/sec  1764
[ 15]   3.00-4.00   sec  53.6 MBytes   450 Mbits/sec  1745
[ 17]   3.00-4.00   sec  53.3 MBytes   447 Mbits/sec  1735
[ 19]   3.00-4.00   sec  52.8 MBytes   443 Mbits/sec  1717
[ 21]   3.00-4.00   sec  52.5 MBytes   440 Mbits/sec  1707
[ 23]   3.00-4.00   sec  51.9 MBytes   435 Mbits/sec  1688
[SUM]   3.00-4.00   sec   539 MBytes  4.52 Gbits/sec  17542
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   4.00-5.00   sec  55.5 MBytes   466 Mbits/sec  1807
[  7]   4.00-5.00   sec  55.2 MBytes   463 Mbits/sec  1795
[  9]   4.00-5.00   sec  55.0 MBytes   461 Mbits/sec  1790
[ 11]   4.00-5.00   sec  54.4 MBytes   456 Mbits/sec  1769
[ 13]   4.00-5.00   sec  54.1 MBytes   453 Mbits/sec  1759
[ 15]   4.00-5.00   sec  53.5 MBytes   449 Mbits/sec  1740
[ 17]   4.00-5.00   sec  53.1 MBytes   445 Mbits/sec  1726
[ 19]   4.00-5.00   sec  52.5 MBytes   440 Mbits/sec  1707
[ 21]   4.00-5.00   sec  52.2 MBytes   438 Mbits/sec  1699
[ 23]   4.00-5.00   sec  51.9 MBytes   435 Mbits/sec  1687
[SUM]   4.00-5.00   sec   537 MBytes  4.51 Gbits/sec  17479
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   5.00-6.00   sec  56.1 MBytes   471 Mbits/sec  1825
[  7]   5.00-6.00   sec  55.5 MBytes   466 Mbits/sec  1806
[  9]   5.00-6.00   sec  55.1 MBytes   462 Mbits/sec  1792
[ 11]   5.00-6.00   sec  54.5 MBytes   457 Mbits/sec  1772
[ 13]   5.00-6.00   sec  54.0 MBytes   453 Mbits/sec  1757
[ 15]   5.00-6.00   sec  53.8 MBytes   452 Mbits/sec  1751
[ 17]   5.00-6.00   sec  53.4 MBytes   448 Mbits/sec  1736
[ 19]   5.00-6.00   sec  52.8 MBytes   443 Mbits/sec  1718
[ 21]   5.00-6.00   sec  52.4 MBytes   439 Mbits/sec  1704
[ 23]   5.00-6.00   sec  52.0 MBytes   437 Mbits/sec  1693
[SUM]   5.00-6.00   sec   540 MBytes  4.53 Gbits/sec  17554
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   6.00-7.00   sec  55.7 MBytes   467 Mbits/sec  1813
[  7]   6.00-7.00   sec  55.3 MBytes   464 Mbits/sec  1799
[  9]   6.00-7.00   sec  54.9 MBytes   460 Mbits/sec  1785
[ 11]   6.00-7.00   sec  54.4 MBytes   457 Mbits/sec  1771
[ 13]   6.00-7.00   sec  54.0 MBytes   453 Mbits/sec  1756
[ 15]   6.00-7.00   sec  53.5 MBytes   449 Mbits/sec  1740
[ 17]   6.00-7.00   sec  53.0 MBytes   445 Mbits/sec  1725
[ 19]   6.00-7.00   sec  52.5 MBytes   441 Mbits/sec  1709
[ 21]   6.00-7.00   sec  52.1 MBytes   437 Mbits/sec  1695
[ 23]   6.00-7.00   sec  51.8 MBytes   434 Mbits/sec  1685
[SUM]   6.00-7.00   sec   537 MBytes  4.51 Gbits/sec  17478
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   7.00-8.00   sec  56.2 MBytes   472 Mbits/sec  1830
[  7]   7.00-8.00   sec  55.8 MBytes   468 Mbits/sec  1814
[  9]   7.00-8.00   sec  55.1 MBytes   462 Mbits/sec  1793
[ 11]   7.00-8.00   sec  54.6 MBytes   458 Mbits/sec  1776
[ 13]   7.00-8.00   sec  54.2 MBytes   455 Mbits/sec  1763
[ 15]   7.00-8.00   sec  53.8 MBytes   451 Mbits/sec  1749
[ 17]   7.00-8.00   sec  53.4 MBytes   448 Mbits/sec  1737
[ 19]   7.00-8.00   sec  52.8 MBytes   443 Mbits/sec  1718
[ 21]   7.00-8.00   sec  52.5 MBytes   441 Mbits/sec  1709
[ 23]   7.00-8.00   sec  51.9 MBytes   435 Mbits/sec  1688
[SUM]   7.00-8.00   sec   540 MBytes  4.53 Gbits/sec  17577
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   8.00-9.00   sec  56.3 MBytes   473 Mbits/sec  1833
[  7]   8.00-9.00   sec  55.6 MBytes   466 Mbits/sec  1809
[  9]   8.00-9.00   sec  55.1 MBytes   463 Mbits/sec  1794
[ 11]   8.00-9.00   sec  54.8 MBytes   460 Mbits/sec  1783
[ 13]   8.00-9.00   sec  54.1 MBytes   454 Mbits/sec  1761
[ 15]   8.00-9.00   sec  53.7 MBytes   451 Mbits/sec  1748
[ 17]   8.00-9.00   sec  53.1 MBytes   446 Mbits/sec  1729
[ 19]   8.00-9.00   sec  52.9 MBytes   443 Mbits/sec  1720
[ 21]   8.00-9.00   sec  52.3 MBytes   438 Mbits/sec  1700
[ 23]   8.00-9.00   sec  51.8 MBytes   434 Mbits/sec  1685
[SUM]   8.00-9.00   sec   540 MBytes  4.53 Gbits/sec  17562
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   9.00-10.00  sec  56.3 MBytes   473 Mbits/sec  1833
[  7]   9.00-10.00  sec  55.8 MBytes   468 Mbits/sec  1816
[  9]   9.00-10.00  sec  55.3 MBytes   464 Mbits/sec  1798
[ 11]   9.00-10.00  sec  54.6 MBytes   458 Mbits/sec  1775
[ 13]   9.00-10.00  sec  54.1 MBytes   454 Mbits/sec  1759
[ 15]   9.00-10.00  sec  53.6 MBytes   449 Mbits/sec  1743
[ 17]   9.00-10.00  sec  53.2 MBytes   446 Mbits/sec  1730
[ 19]   9.00-10.00  sec  52.8 MBytes   443 Mbits/sec  1719
[ 21]   9.00-10.00  sec  52.3 MBytes   439 Mbits/sec  1703
[ 23]   9.00-10.00  sec  52.0 MBytes   437 Mbits/sec  1693
[SUM]   9.00-10.00  sec   540 MBytes  4.53 Gbits/sec  17569
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec   557 MBytes   467 Mbits/sec  0.000 ms  0/18118 (0%)  sender
[  5]   0.00-10.03  sec   557 MBytes   466 Mbits/sec  0.413 ms  0/18118 (0%)  receiver
[  7]   0.00-10.00  sec   552 MBytes   463 Mbits/sec  0.000 ms  0/17951 (0%)  sender
[  7]   0.00-10.03  sec   552 MBytes   462 Mbits/sec  0.414 ms  0/17951 (0%)  receiver
[  9]   0.00-10.00  sec   547 MBytes   459 Mbits/sec  0.000 ms  0/17808 (0%)  sender
[  9]   0.00-10.03  sec   547 MBytes   458 Mbits/sec  0.426 ms  0/17808 (0%)  receiver
[ 11]   0.00-10.00  sec   542 MBytes   455 Mbits/sec  0.000 ms  0/17644 (0%)  sender
[ 11]   0.00-10.03  sec   542 MBytes   454 Mbits/sec  0.423 ms  0/17644 (0%)  receiver
[ 13]   0.00-10.00  sec   538 MBytes   451 Mbits/sec  0.000 ms  0/17492 (0%)  sender
[ 13]   0.00-10.03  sec   538 MBytes   450 Mbits/sec  0.438 ms  0/17492 (0%)  receiver
[ 15]   0.00-10.00  sec   533 MBytes   447 Mbits/sec  0.000 ms  0/17347 (0%)  sender
[ 15]   0.00-10.03  sec   533 MBytes   446 Mbits/sec  0.406 ms  0/17347 (0%)  receiver
[ 17]   0.00-10.00  sec   529 MBytes   444 Mbits/sec  0.000 ms  0/17211 (0%)  sender
[ 17]   0.00-10.03  sec   529 MBytes   443 Mbits/sec  0.364 ms  0/17211 (0%)  receiver
[ 19]   0.00-10.00  sec   525 MBytes   440 Mbits/sec  0.000 ms  0/17068 (0%)  sender
[ 19]   0.00-10.03  sec   525 MBytes   439 Mbits/sec  0.361 ms  0/17068 (0%)  receiver
[ 21]   0.00-10.00  sec   520 MBytes   436 Mbits/sec  0.000 ms  0/16923 (0%)  sender
[ 21]   0.00-10.03  sec   520 MBytes   435 Mbits/sec  0.371 ms  0/16922 (0%)  receiver
[ 23]   0.00-10.00  sec   516 MBytes   433 Mbits/sec  0.000 ms  0/16778 (0%)  sender
[ 23]   0.00-10.03  sec   516 MBytes   431 Mbits/sec  0.404 ms  0/16777 (0%)  receiver
[SUM]   0.00-10.00  sec  5.23 GBytes  4.50 Gbits/sec  0.000 ms  0/174340 (0%)  sender
[SUM]   0.00-10.03  sec  5.23 GBytes  4.48 Gbits/sec  0.402 ms  0/174338 (0%)  receiver

iperf Done.
root@tegra-ubuntu:/sys/kernel/config/pci_ep# iperf3 -c 192.168.1.100 -P 2 -u -b 100000M
Connecting to host 192.168.1.100, port 5201
[  5] local 192.168.1.2 port 43453 connected to 192.168.1.100 port 5201
[  7] local 192.168.1.2 port 51072 connected to 192.168.1.100 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec   266 MBytes  2.23 Gbits/sec  8667
[  7]   0.00-1.00   sec   264 MBytes  2.22 Gbits/sec  8595
[SUM]   0.00-1.00   sec   531 MBytes  4.45 Gbits/sec  17262
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.00-2.00   sec   272 MBytes  2.28 Gbits/sec  8836
[  7]   1.00-2.00   sec   269 MBytes  2.26 Gbits/sec  8760
[SUM]   1.00-2.00   sec   541 MBytes  4.54 Gbits/sec  17596
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   2.00-3.00   sec   273 MBytes  2.29 Gbits/sec  8880
[  7]   2.00-3.00   sec   271 MBytes  2.27 Gbits/sec  8806
[SUM]   2.00-3.00   sec   544 MBytes  4.56 Gbits/sec  17686
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   3.00-4.00   sec   273 MBytes  2.29 Gbits/sec  8872
[  7]   3.00-4.00   sec   270 MBytes  2.26 Gbits/sec  8777
[SUM]   3.00-4.00   sec   542 MBytes  4.55 Gbits/sec  17649
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   4.00-5.00   sec   272 MBytes  2.28 Gbits/sec  8857
[  7]   4.00-5.00   sec   270 MBytes  2.26 Gbits/sec  8775
[SUM]   4.00-5.00   sec   542 MBytes  4.55 Gbits/sec  17632
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   5.00-6.00   sec   272 MBytes  2.28 Gbits/sec  8852
[  7]   5.00-6.00   sec   270 MBytes  2.26 Gbits/sec  8772
[SUM]   5.00-6.00   sec   542 MBytes  4.54 Gbits/sec  17624
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   6.00-7.00   sec   272 MBytes  2.28 Gbits/sec  8851
[  7]   6.00-7.00   sec   270 MBytes  2.26 Gbits/sec  8779
[SUM]   6.00-7.00   sec   542 MBytes  4.55 Gbits/sec  17630
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   7.00-8.00   sec   273 MBytes  2.29 Gbits/sec  8873
[  7]   7.00-8.00   sec   270 MBytes  2.27 Gbits/sec  8786
[SUM]   7.00-8.00   sec   543 MBytes  4.55 Gbits/sec  17659
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   8.00-9.00   sec   272 MBytes  2.28 Gbits/sec  8860
[  7]   8.00-9.00   sec   270 MBytes  2.27 Gbits/sec  8792
[SUM]   8.00-9.00   sec   543 MBytes  4.55 Gbits/sec  17652
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   9.00-10.00  sec   272 MBytes  2.29 Gbits/sec  8864
[  7]   9.00-10.00  sec   270 MBytes  2.26 Gbits/sec  8777
[SUM]   9.00-10.00  sec   542 MBytes  4.55 Gbits/sec  17641
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  2.65 GBytes  2.28 Gbits/sec  0.000 ms  0/88412 (0%)  sender
[  5]   0.00-10.06  sec  2.64 GBytes  2.26 Gbits/sec  0.123 ms  406/88412 (0.46%)  receiver
[  7]   0.00-10.00  sec  2.63 GBytes  2.26 Gbits/sec  0.000 ms  0/87619 (0%)  sender
[  7]   0.00-10.06  sec  2.63 GBytes  2.25 Gbits/sec  0.119 ms  17/87619 (0.019%)  receiver
[SUM]   0.00-10.00  sec  5.28 GBytes  4.54 Gbits/sec  0.000 ms  0/176031 (0%)  sender
[SUM]   0.00-10.06  sec  5.27 GBytes  4.50 Gbits/sec  0.121 ms  423/176031 (0.24%)  receiver

iperf Done.

64 bytes from 192.168.1.100: icmp_seq=1000 ttl=64 time=4.02 ms

--- 192.168.1.100 ping statistics ---
1000 packets transmitted, 1000 received, 0% packet loss, time 21356ms
rtt min/avg/max/mdev = 0.274/2.562/4.854/1.263 ms
6

hello dingfeng1,

please using the lspci -v command on both devices for checking PCIE lane configuration.
PCIe signal integrity issues can cause bandwidth and latency problems. you may check the signal integrity by running the pcie_ep_fdmatest module with the -s option to enable signal integrity testing, adding -d option to enable debug mode for more details.

7

Hi Jerry,

Thanks for your quick response. With the pcie_epf_dma_test, the log shows fine. From the log above the bandwidth of pcie_epf_dma_test is 54116Mbps, which is 85% of the 4 lane pcie4.0 theoretically bandwidth(64Gbps). It is only the vnet mode(pci-epf-tegra-vnet) which looks not good. Btw, I will paste the lspci -v result in the next comment.

root@tegra-ubuntu:/sys/kernel/debug/14160000.pcie-ep_epf_dma_test# cat edmalib_test
root@tegra-ubuntu:/sys/kernel/debug/14160000.pcie-ep_epf_dma_test# dmesg |tail
[  559.146931] r8168: eth0: link up
[  574.447729] tegra194-pcie 14160000.pcie-ep: iATU unroll: enabled
[  574.447742] tegra194-pcie 14160000.pcie-ep: Detected iATU regions: 8 outbound, 2 inbound
[  574.447836] pcie_dma_epf tegra_pcie_dma_epf.0: BAR0 phy_addr: f0000000 size: 10000000
[  908.543849] pcie_dma_epf tegra_pcie_dma_epf.0: edmalib_common_test: re-init edma lib prev_ch(0) != current chans(11)
[  908.544236] tegra194-pcie 14160000.pcie-ep: tegra_pcie_edma_initialize: success
[  908.749046] pcie_dma_epf tegra_pcie_dma_epf.0: edmalib_common_test: EDMA LIB WR started for 1 chans, size 16777216 Bytes, iterations: 1000 of descriptors 4
[  908.751127] pcie_dma_epf tegra_pcie_dma_epf.0: edmalib_common_test: EDMA LIB submit done
[  918.669790] pcie_dma_epf tegra_pcie_dma_epf.0: edma_final_complete: WR-local-Async complete for chan 0 with status 0. Total desc 4000 of Sz 16777216 Bytes done in time 9920729312 nsec. Perf is 54116 Mbps
[  918.669807] pcie_dma_epf tegra_pcie_dma_epf.0: edma_final_complete: All Async channels. Cumulative Perf 54116 Mbps, time 9920729440 nsec
8

hello dingfeng1,

is there show any different results with UDP/TCP?
for instance,

Server:
$ iperf3 -s

Client:
[UDP]
$ iperf3 -c <ip> -u -b 0 -l 16K -t 120 -i 1
[TCP]
$ iperf3 -c <ip> -b 0 -l 16K -t 120 -i 1
10

The log of lspci -v are shown below.

agx.log (4.8 KB)
nx.log (4.5 KB)

11

Hi Jerry,

It looks like it has difference between UDP and TCP but not too much. Still have gap to the theoretical value.


root@tegra-ubuntu:/data# iperf3 -c 192.168.1.100 -u -b 0 -l 16K -t 120 -i 1
Connecting to host 192.168.1.100, port 5201
[  5] local 192.168.1.2 port 49441 connected to 192.168.1.100 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec   235 MBytes  1.97 Gbits/sec  15010
[  5]   1.00-2.00   sec   235 MBytes  1.97 Gbits/sec  15010
[  5]   2.00-3.00   sec   235 MBytes  1.97 Gbits/sec  15010
[  5]   3.00-4.00   sec   235 MBytes  1.97 Gbits/sec  15010
[  5]   4.00-5.00   sec   235 MBytes  1.97 Gbits/sec  15020
[  5]   5.00-6.00   sec   235 MBytes  1.97 Gbits/sec  15020
[  5]   6.00-7.00   sec   235 MBytes  1.97 Gbits/sec  15030
[  5]   7.00-8.00   sec   235 MBytes  1.97 Gbits/sec  15030
[  5]   8.00-9.00   sec   235 MBytes  1.97 Gbits/sec  15030
[  5]   9.00-10.00  sec   235 MBytes  1.97 Gbits/sec  15020
[  5]  10.00-11.00  sec   235 MBytes  1.97 Gbits/sec  15020
[  5]  11.00-12.00  sec   235 MBytes  1.97 Gbits/sec  15030
[  5]  12.00-13.00  sec   235 MBytes  1.97 Gbits/sec  15020
[  5]  13.00-14.00  sec   235 MBytes  1.97 Gbits/sec  15020
[  5]  14.00-15.00  sec   235 MBytes  1.97 Gbits/sec  15040
[  5]  15.00-16.00  sec   235 MBytes  1.97 Gbits/sec  15020
[  5]  16.00-17.00  sec   235 MBytes  1.97 Gbits/sec  15030
[  5]  17.00-18.00  sec   235 MBytes  1.97 Gbits/sec  15040
[  5]  18.00-19.00  sec   235 MBytes  1.97 Gbits/sec  15030
[  5]  19.00-20.00  sec   235 MBytes  1.97 Gbits/sec  15030
[  5]  20.00-21.00  sec   235 MBytes  1.97 Gbits/sec  15030
[  5]  21.00-22.00  sec   235 MBytes  1.97 Gbits/sec  15030
[  5]  22.00-23.00  sec   235 MBytes  1.97 Gbits/sec  15030
[  5]  23.00-24.00  sec   235 MBytes  1.97 Gbits/sec  15030
[  5]  24.00-25.00  sec   235 MBytes  1.97 Gbits/sec  15030
[  5]  25.00-26.00  sec   235 MBytes  1.97 Gbits/sec  15030
[  5]  26.00-27.00  sec   235 MBytes  1.97 Gbits/sec  15020
[  5]  27.00-28.00  sec   235 MBytes  1.97 Gbits/sec  15030
[  5]  28.00-29.00  sec   235 MBytes  1.97 Gbits/sec  15030
[  5]  29.00-30.00  sec   235 MBytes  1.97 Gbits/sec  15030
[  5]  30.00-31.00  sec   235 MBytes  1.97 Gbits/sec  15030
[  5]  31.00-32.00  sec   235 MBytes  1.97 Gbits/sec  15020
[  5]  32.00-33.00  sec   235 MBytes  1.97 Gbits/sec  15030
[  5]  33.00-34.00  sec   235 MBytes  1.97 Gbits/sec  15030
[  5]  34.00-35.00  sec   235 MBytes  1.97 Gbits/sec  15030
[  5]  35.00-36.00  sec   235 MBytes  1.97 Gbits/sec  15030
[  5]  36.00-37.00  sec   235 MBytes  1.97 Gbits/sec  15030
[  5]  37.00-38.00  sec   235 MBytes  1.97 Gbits/sec  15020
[  5]  38.00-39.00  sec   235 MBytes  1.97 Gbits/sec  15030
[  5]  39.00-40.00  sec   235 MBytes  1.97 Gbits/sec  15030
[  5]  40.00-41.00  sec   235 MBytes  1.97 Gbits/sec  15030
[  5]  41.00-42.00  sec   235 MBytes  1.97 Gbits/sec  15020
[  5]  42.00-43.00  sec   235 MBytes  1.97 Gbits/sec  15040
[  5]  43.00-44.00  sec   235 MBytes  1.97 Gbits/sec  15030
[  5]  44.00-45.00  sec   235 MBytes  1.97 Gbits/sec  15030
[  5]  45.00-46.00  sec   235 MBytes  1.97 Gbits/sec  15040
[  5]  46.00-47.00  sec   235 MBytes  1.97 Gbits/sec  15030
[  5]  47.00-48.00  sec   235 MBytes  1.97 Gbits/sec  15030
[  5]  48.00-49.00  sec   235 MBytes  1.97 Gbits/sec  15040
[  5]  49.00-50.00  sec   235 MBytes  1.97 Gbits/sec  15040
[  5]  50.00-51.00  sec   235 MBytes  1.97 Gbits/sec  15060
[  5]  51.00-52.00  sec   235 MBytes  1.97 Gbits/sec  15050
[  5]  52.00-53.00  sec   235 MBytes  1.97 Gbits/sec  15050
[  5]  53.00-54.00  sec   235 MBytes  1.97 Gbits/sec  15060
[  5]  54.00-55.00  sec   235 MBytes  1.97 Gbits/sec  15050
[  5]  55.00-56.00  sec   235 MBytes  1.97 Gbits/sec  15050
[  5]  56.00-57.00  sec   235 MBytes  1.97 Gbits/sec  15060
[  5]  57.00-58.00  sec   235 MBytes  1.97 Gbits/sec  15050
[  5]  58.00-59.00  sec   235 MBytes  1.97 Gbits/sec  15050
[  5]  59.00-60.00  sec   235 MBytes  1.97 Gbits/sec  15060
[  5]  60.00-61.00  sec   235 MBytes  1.97 Gbits/sec  15060
[  5]  61.00-62.00  sec   235 MBytes  1.97 Gbits/sec  15050
[  5]  62.00-63.00  sec   235 MBytes  1.97 Gbits/sec  15060
[  5]  63.00-64.00  sec   235 MBytes  1.97 Gbits/sec  15060
[  5]  64.00-65.00  sec   235 MBytes  1.97 Gbits/sec  15060
[  5]  65.00-66.00  sec   235 MBytes  1.97 Gbits/sec  15050
[  5]  66.00-67.00  sec   235 MBytes  1.97 Gbits/sec  15060
[  5]  67.00-68.00  sec   235 MBytes  1.97 Gbits/sec  15060
[  5]  68.00-69.00  sec   235 MBytes  1.97 Gbits/sec  15060
[  5]  69.00-70.00  sec   235 MBytes  1.97 Gbits/sec  15050
[  5]  70.00-71.00  sec   235 MBytes  1.97 Gbits/sec  15060
[  5]  71.00-72.00  sec   235 MBytes  1.97 Gbits/sec  15060
[  5]  72.00-73.00  sec   235 MBytes  1.97 Gbits/sec  15050
[  5]  73.00-74.00  sec   235 MBytes  1.97 Gbits/sec  15060
[  5]  74.00-75.00  sec   235 MBytes  1.97 Gbits/sec  15050
[  5]  75.00-76.00  sec   235 MBytes  1.97 Gbits/sec  15050
[  5]  76.00-77.00  sec   235 MBytes  1.97 Gbits/sec  15050
[  5]  77.00-78.00  sec   235 MBytes  1.97 Gbits/sec  15060
[  5]  78.00-79.00  sec   235 MBytes  1.97 Gbits/sec  15050
[  5]  79.00-80.00  sec   235 MBytes  1.97 Gbits/sec  15050
[  5]  80.00-81.00  sec   235 MBytes  1.97 Gbits/sec  15050
[  5]  81.00-82.00  sec   235 MBytes  1.97 Gbits/sec  15060
[  5]  82.00-83.00  sec   235 MBytes  1.97 Gbits/sec  15060
[  5]  83.00-84.00  sec   235 MBytes  1.97 Gbits/sec  15060
[  5]  84.00-85.00  sec   235 MBytes  1.97 Gbits/sec  15050
[  5]  85.00-86.00  sec   235 MBytes  1.97 Gbits/sec  15060
[  5]  86.00-87.00  sec   235 MBytes  1.97 Gbits/sec  15060
[  5]  87.00-88.00  sec   235 MBytes  1.97 Gbits/sec  15050
[  5]  88.00-89.00  sec   235 MBytes  1.97 Gbits/sec  15060
[  5]  89.00-90.00  sec   235 MBytes  1.97 Gbits/sec  15050
[  5]  90.00-91.00  sec   235 MBytes  1.97 Gbits/sec  15060
[  5]  91.00-92.00  sec   235 MBytes  1.97 Gbits/sec  15050
[  5]  92.00-93.00  sec   235 MBytes  1.97 Gbits/sec  15050
[  5]  93.00-94.00  sec   235 MBytes  1.97 Gbits/sec  15070
[  5]  94.00-95.00  sec   235 MBytes  1.97 Gbits/sec  15050
[  5]  95.00-96.00  sec   235 MBytes  1.97 Gbits/sec  15060
[  5]  96.00-97.00  sec   235 MBytes  1.97 Gbits/sec  15060
[  5]  97.00-98.00  sec   235 MBytes  1.97 Gbits/sec  15050
[  5]  98.00-99.00  sec   235 MBytes  1.97 Gbits/sec  15060
[  5]  99.00-100.00 sec   235 MBytes  1.97 Gbits/sec  15050
[  5] 100.00-101.00 sec   235 MBytes  1.97 Gbits/sec  15060
[  5] 101.00-102.00 sec   235 MBytes  1.97 Gbits/sec  15060
[  5] 102.00-103.00 sec   235 MBytes  1.97 Gbits/sec  15050
[  5] 103.00-104.00 sec   235 MBytes  1.97 Gbits/sec  15060
[  5] 104.00-105.00 sec   235 MBytes  1.97 Gbits/sec  15050
[  5] 105.00-106.00 sec   235 MBytes  1.97 Gbits/sec  15060
[  5] 106.00-107.00 sec   235 MBytes  1.97 Gbits/sec  15060
[  5] 107.00-108.00 sec   235 MBytes  1.97 Gbits/sec  15060
[  5] 108.00-109.00 sec   236 MBytes  1.98 Gbits/sec  15080
[  5] 109.00-110.00 sec   236 MBytes  1.98 Gbits/sec  15080
[  5] 110.00-111.00 sec   235 MBytes  1.97 Gbits/sec  15060
[  5] 111.00-112.00 sec   235 MBytes  1.97 Gbits/sec  15060
[  5] 112.00-113.00 sec   235 MBytes  1.97 Gbits/sec  15060
[  5] 113.00-114.00 sec   235 MBytes  1.97 Gbits/sec  15050
[  5] 114.00-115.00 sec   235 MBytes  1.97 Gbits/sec  15050
[  5] 115.00-116.00 sec   235 MBytes  1.97 Gbits/sec  15050
[  5] 116.00-117.00 sec   235 MBytes  1.97 Gbits/sec  15050
[  5] 117.00-118.00 sec   235 MBytes  1.97 Gbits/sec  15050
[  5] 118.00-119.00 sec   235 MBytes  1.97 Gbits/sec  15060
[  5] 119.00-120.00 sec   235 MBytes  1.97 Gbits/sec  15050
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-120.00 sec  27.5 GBytes  1.97 Gbits/sec  0.000 ms  0/1805300 (0%)  sender
[  5]   0.00-120.04 sec  27.5 GBytes  1.97 Gbits/sec  0.062 ms  683/1805300 (0.038%)  receiver

iperf Done.
root@tegra-ubuntu:/data# iperf3 -c 192.168.1.100 -b 0 -l 16K -t 120 -i 1
Connecting to host 192.168.1.100, port 5201
[  5] local 192.168.1.2 port 43338 connected to 192.168.1.100 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   475 MBytes  3.99 Gbits/sec    0   2.15 MBytes
[  5]   1.00-2.00   sec   475 MBytes  3.98 Gbits/sec    0   2.15 MBytes
[  5]   2.00-3.00   sec   475 MBytes  3.99 Gbits/sec    0   2.15 MBytes
[  5]   3.00-4.00   sec   475 MBytes  3.99 Gbits/sec    0   2.15 MBytes
[  5]   4.00-5.00   sec   475 MBytes  3.98 Gbits/sec    0   2.15 MBytes
[  5]   5.00-6.00   sec   475 MBytes  3.99 Gbits/sec    0   2.15 MBytes
[  5]   6.00-7.00   sec   476 MBytes  3.99 Gbits/sec    0   2.15 MBytes
[  5]   7.00-8.00   sec   477 MBytes  4.00 Gbits/sec    0   2.15 MBytes
[  5]   8.00-9.00   sec   476 MBytes  4.00 Gbits/sec    0   2.15 MBytes
[  5]   9.00-10.00  sec   477 MBytes  4.00 Gbits/sec    0   2.15 MBytes
[  5]  10.00-11.00  sec   476 MBytes  4.00 Gbits/sec    0   2.15 MBytes
[  5]  11.00-12.00  sec   475 MBytes  3.99 Gbits/sec    0   2.15 MBytes
[  5]  12.00-13.00  sec   475 MBytes  3.99 Gbits/sec    0   2.15 MBytes
[  5]  13.00-14.00  sec   476 MBytes  4.00 Gbits/sec    0   2.15 MBytes
[  5]  14.00-15.00  sec   479 MBytes  4.02 Gbits/sec    0   2.15 MBytes
[  5]  15.00-16.00  sec   479 MBytes  4.02 Gbits/sec    0   2.15 MBytes
[  5]  16.00-17.00  sec   479 MBytes  4.02 Gbits/sec    0   2.15 MBytes
[  5]  17.00-18.00  sec   479 MBytes  4.02 Gbits/sec    0   2.15 MBytes
[  5]  18.00-19.00  sec   478 MBytes  4.01 Gbits/sec    0   2.15 MBytes
[  5]  19.00-20.00  sec   478 MBytes  4.01 Gbits/sec    0   2.15 MBytes
[  5]  20.00-21.00  sec   479 MBytes  4.02 Gbits/sec    0   2.15 MBytes
[  5]  21.00-22.00  sec   478 MBytes  4.01 Gbits/sec    0   2.15 MBytes
[  5]  22.00-23.00  sec   477 MBytes  4.00 Gbits/sec    0   2.15 MBytes
[  5]  23.00-24.00  sec   478 MBytes  4.01 Gbits/sec    0   2.15 MBytes
[  5]  24.00-25.00  sec   478 MBytes  4.01 Gbits/sec    0   2.15 MBytes
[  5]  25.00-26.00  sec   478 MBytes  4.01 Gbits/sec    0   2.15 MBytes
[  5]  26.00-27.00  sec   479 MBytes  4.02 Gbits/sec    0   2.15 MBytes
[  5]  27.00-28.00  sec   479 MBytes  4.01 Gbits/sec    0   2.15 MBytes
[  5]  28.00-29.00  sec   478 MBytes  4.01 Gbits/sec    0   2.15 MBytes
[  5]  29.00-30.00  sec   478 MBytes  4.01 Gbits/sec    0   2.15 MBytes
[  5]  30.00-31.00  sec   478 MBytes  4.01 Gbits/sec    0   2.15 MBytes
[  5]  31.00-32.00  sec   478 MBytes  4.01 Gbits/sec    0   2.15 MBytes
[  5]  32.00-33.00  sec   479 MBytes  4.01 Gbits/sec    0   2.15 MBytes
[  5]  33.00-34.00  sec   478 MBytes  4.01 Gbits/sec    0   2.15 MBytes
[  5]  34.00-35.00  sec   478 MBytes  4.01 Gbits/sec    0   2.15 MBytes
[  5]  35.00-36.00  sec   478 MBytes  4.01 Gbits/sec    0   2.15 MBytes
[  5]  36.00-37.00  sec   476 MBytes  4.00 Gbits/sec    0   2.15 MBytes
[  5]  37.00-38.00  sec   476 MBytes  3.99 Gbits/sec    0   2.15 MBytes
[  5]  38.00-39.00  sec   476 MBytes  3.99 Gbits/sec    0   2.15 MBytes
[  5]  39.00-40.00  sec   476 MBytes  4.00 Gbits/sec    0   2.15 MBytes
[  5]  40.00-41.00  sec   476 MBytes  4.00 Gbits/sec    0   2.15 MBytes
[  5]  41.00-42.00  sec   476 MBytes  3.99 Gbits/sec    0   2.15 MBytes
[  5]  42.00-43.00  sec   475 MBytes  3.98 Gbits/sec    0   2.15 MBytes
[  5]  43.00-44.00  sec   476 MBytes  3.99 Gbits/sec    0   2.15 MBytes
[  5]  44.00-45.00  sec   475 MBytes  3.99 Gbits/sec    0   2.15 MBytes
[  5]  45.00-46.00  sec   475 MBytes  3.99 Gbits/sec    0   2.15 MBytes
[  5]  46.00-47.00  sec   475 MBytes  3.98 Gbits/sec    0   2.15 MBytes
[  5]  47.00-48.00  sec   475 MBytes  3.98 Gbits/sec    0   2.15 MBytes
[  5]  48.00-49.00  sec   475 MBytes  3.99 Gbits/sec    0   2.15 MBytes
[  5]  49.00-50.00  sec   476 MBytes  3.99 Gbits/sec    0   2.15 MBytes
[  5]  50.00-51.00  sec   477 MBytes  4.00 Gbits/sec    0   2.15 MBytes
[  5]  51.00-52.00  sec   476 MBytes  3.99 Gbits/sec    0   2.15 MBytes
[  5]  52.00-53.00  sec   476 MBytes  3.99 Gbits/sec    0   2.15 MBytes
[  5]  53.00-54.00  sec   477 MBytes  4.00 Gbits/sec    0   2.15 MBytes
[  5]  54.00-55.00  sec   477 MBytes  4.00 Gbits/sec    0   2.15 MBytes
[  5]  55.00-56.00  sec   476 MBytes  4.00 Gbits/sec    0   2.15 MBytes
[  5]  56.00-57.00  sec   477 MBytes  4.00 Gbits/sec    0   2.15 MBytes
[  5]  57.00-58.00  sec   476 MBytes  3.99 Gbits/sec    0   2.15 MBytes
[  5]  58.00-59.00  sec   477 MBytes  4.00 Gbits/sec    0   2.15 MBytes
[  5]  59.00-60.00  sec   477 MBytes  4.00 Gbits/sec    0   2.15 MBytes
[  5]  60.00-61.00  sec   476 MBytes  3.99 Gbits/sec    0   2.15 MBytes
[  5]  61.00-62.00  sec   475 MBytes  3.98 Gbits/sec    0   2.15 MBytes
[  5]  62.00-63.00  sec   475 MBytes  3.98 Gbits/sec    0   2.15 MBytes
[  5]  63.00-64.00  sec   475 MBytes  3.99 Gbits/sec    0   2.15 MBytes
[  5]  64.00-65.00  sec   475 MBytes  3.98 Gbits/sec    0   2.15 MBytes
[  5]  65.00-66.00  sec   475 MBytes  3.99 Gbits/sec    0   2.15 MBytes
[  5]  66.00-67.00  sec   475 MBytes  3.99 Gbits/sec    0   2.15 MBytes
[  5]  67.00-68.00  sec   475 MBytes  3.99 Gbits/sec    0   2.15 MBytes
[  5]  68.00-69.00  sec   475 MBytes  3.98 Gbits/sec    0   2.15 MBytes
[  5]  69.00-70.00  sec   475 MBytes  3.98 Gbits/sec    0   2.15 MBytes
[  5]  70.00-71.00  sec   475 MBytes  3.98 Gbits/sec    0   2.15 MBytes
[  5]  71.00-72.00  sec   475 MBytes  3.99 Gbits/sec    0   2.15 MBytes
[  5]  72.00-73.00  sec   475 MBytes  3.99 Gbits/sec    0   2.15 MBytes
[  5]  73.00-74.00  sec   475 MBytes  3.99 Gbits/sec    0   2.15 MBytes
[  5]  74.00-75.00  sec   475 MBytes  3.99 Gbits/sec    0   2.15 MBytes
[  5]  75.00-76.00  sec   475 MBytes  3.98 Gbits/sec    0   2.15 MBytes
[  5]  76.00-77.00  sec   476 MBytes  3.99 Gbits/sec    0   2.15 MBytes
[  5]  77.00-78.00  sec   475 MBytes  3.98 Gbits/sec    0   2.15 MBytes
[  5]  78.00-79.00  sec   475 MBytes  3.98 Gbits/sec    0   2.15 MBytes
[  5]  79.00-80.00  sec   475 MBytes  3.99 Gbits/sec    0   2.15 MBytes
[  5]  80.00-81.00  sec   475 MBytes  3.99 Gbits/sec    0   2.15 MBytes
[  5]  81.00-82.00  sec   475 MBytes  3.99 Gbits/sec    0   2.15 MBytes
[  5]  82.00-83.00  sec   476 MBytes  4.00 Gbits/sec    0   2.15 MBytes
[  5]  83.00-84.00  sec   477 MBytes  4.00 Gbits/sec    0   2.15 MBytes
[  5]  84.00-85.00  sec   476 MBytes  3.99 Gbits/sec    0   2.15 MBytes
[  5]  85.00-86.00  sec   477 MBytes  4.00 Gbits/sec    0   2.15 MBytes
[  5]  86.00-87.00  sec   477 MBytes  4.00 Gbits/sec    0   2.15 MBytes
[  5]  87.00-88.00  sec   477 MBytes  4.00 Gbits/sec    0   2.15 MBytes
[  5]  88.00-89.00  sec   476 MBytes  3.99 Gbits/sec    0   2.15 MBytes
[  5]  89.00-90.00  sec   477 MBytes  4.00 Gbits/sec    0   2.15 MBytes
[  5]  90.00-91.00  sec   477 MBytes  4.00 Gbits/sec    0   2.15 MBytes
[  5]  91.00-92.00  sec   476 MBytes  4.00 Gbits/sec    0   2.15 MBytes
[  5]  92.00-93.00  sec   476 MBytes  4.00 Gbits/sec    0   2.15 MBytes
[  5]  93.00-94.00  sec   476 MBytes  3.99 Gbits/sec    0   2.15 MBytes
[  5]  94.00-95.00  sec   475 MBytes  3.98 Gbits/sec    0   2.15 MBytes
[  5]  95.00-96.00  sec   475 MBytes  3.98 Gbits/sec    0   2.15 MBytes
[  5]  96.00-97.00  sec   475 MBytes  3.98 Gbits/sec    0   2.15 MBytes
[  5]  97.00-98.00  sec   476 MBytes  3.99 Gbits/sec    0   2.15 MBytes
[  5]  98.00-99.00  sec   476 MBytes  3.99 Gbits/sec    0   2.15 MBytes
[  5]  99.00-100.00 sec   477 MBytes  4.00 Gbits/sec    0   2.15 MBytes
[  5] 100.00-101.00 sec   476 MBytes  3.99 Gbits/sec    0   2.15 MBytes
[  5] 101.00-102.00 sec   477 MBytes  4.00 Gbits/sec    0   2.15 MBytes
[  5] 102.00-103.00 sec   477 MBytes  4.00 Gbits/sec    0   2.15 MBytes
[  5] 103.00-104.00 sec   476 MBytes  3.99 Gbits/sec    0   2.15 MBytes
[  5] 104.00-105.00 sec   476 MBytes  3.99 Gbits/sec    0   2.15 MBytes
[  5] 105.00-106.00 sec   477 MBytes  4.00 Gbits/sec    0   2.15 MBytes
[  5] 106.00-107.00 sec   476 MBytes  3.99 Gbits/sec    0   2.15 MBytes
[  5] 107.00-108.00 sec   477 MBytes  4.00 Gbits/sec    0   2.15 MBytes
[  5] 108.00-109.00 sec   477 MBytes  4.00 Gbits/sec    0   2.15 MBytes
[  5] 109.00-110.00 sec   476 MBytes  3.99 Gbits/sec    0   2.15 MBytes
[  5] 110.00-111.00 sec   476 MBytes  3.99 Gbits/sec    0   2.15 MBytes
[  5] 111.00-112.00 sec   476 MBytes  3.99 Gbits/sec    0   2.15 MBytes
[  5] 112.00-113.00 sec   476 MBytes  3.99 Gbits/sec    0   2.15 MBytes
[  5] 113.00-114.00 sec   476 MBytes  4.00 Gbits/sec    0   2.15 MBytes
[  5] 114.00-115.00 sec   476 MBytes  3.99 Gbits/sec    0   2.15 MBytes
[  5] 115.00-116.00 sec   476 MBytes  3.99 Gbits/sec    0   2.15 MBytes
[  5] 116.00-117.00 sec   476 MBytes  3.99 Gbits/sec    0   2.15 MBytes
[  5] 117.00-118.00 sec   476 MBytes  3.99 Gbits/sec    0   2.15 MBytes
[  5] 118.00-119.00 sec   476 MBytes  3.99 Gbits/sec    0   2.15 MBytes
[  5] 119.00-120.00 sec   477 MBytes  4.00 Gbits/sec    0   2.15 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-120.00 sec  55.8 GBytes  4.00 Gbits/sec    0             sender
[  5]   0.00-120.04 sec  55.8 GBytes  3.99 Gbits/sec                  receiver

iperf Done.
root@tegra-ubuntu:/data#
12

And I think the difference is because the length setting. After change the -l to 48K the Bandwidth of UDP and TCP are the same.

root@tegra-ubuntu:/data# iperf3 -c 192.168.1.100 -u -b 0 -l 48K -t 10 -i 1
warning: UDP block size 49152 exceeds TCP MSS 32230, may result in fragmentation / drops
Connecting to host 192.168.1.100, port 5201
[  5] local 192.168.1.2 port 60107 connected to 192.168.1.100 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec   510 MBytes  4.28 Gbits/sec  10890
[  5]   1.00-2.00   sec   511 MBytes  4.29 Gbits/sec  10900
[  5]   2.00-3.00   sec   511 MBytes  4.29 Gbits/sec  10900
[  5]   3.00-4.00   sec   511 MBytes  4.29 Gbits/sec  10910
[  5]   4.00-5.00   sec   511 MBytes  4.29 Gbits/sec  10900
[  5]   5.00-6.00   sec   511 MBytes  4.29 Gbits/sec  10900
[  5]   6.00-7.00   sec   511 MBytes  4.29 Gbits/sec  10910
[  5]   7.00-8.00   sec   511 MBytes  4.29 Gbits/sec  10900
[  5]   8.00-9.00   sec   511 MBytes  4.29 Gbits/sec  10910
[  5]   9.00-10.00  sec   511 MBytes  4.29 Gbits/sec  10900
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  4.99 GBytes  4.29 Gbits/sec  0.000 ms  0/109020 (0%)  sender
[  5]   0.00-10.04  sec  4.99 GBytes  4.27 Gbits/sec  0.008 ms  12/109020 (0.011%)  receiver

iperf Done.
13

hello dingfeng1,

please refer to developer guide, Ethernet Controller Configuration to check you’ve correct configuration.
you may also check UPHY Lane Configuration by updating ODMDATA variable in the flash configuration.

15

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.