Hi,
Following PCIe Endpoint Mode — Jetson Linux Developer Guide documentation and using latest JetPack 5.1, getting low speeds ~5Gb/s relative to theoretical for PCIe 4.0 x8 which is ~128Gb/s. Uplink and downlink speeds initiated from either end are mostly symmetrical. Orin is set to max power.
iperf3 -c 192.168.2.1
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 482 MBytes 4.04 Gbits/sec 0 1.97 MBytes
[ 5] 1.00-2.00 sec 481 MBytes 4.03 Gbits/sec 0 1.97 MBytes
[ 5] 2.00-3.00 sec 476 MBytes 3.99 Gbits/sec 0 1.97 MBytes
[ 5] 3.00-4.00 sec 479 MBytes 4.02 Gbits/sec 0 1.97 MBytes
[ 5] 4.00-5.00 sec 476 MBytes 3.99 Gbits/sec 0 1.97 MBytes
[ 5] 5.00-6.00 sec 475 MBytes 3.99 Gbits/sec 0 1.97 MBytes
[ 5] 6.00-7.00 sec 425 MBytes 3.57 Gbits/sec 0 1.97 MBytes
[ 5] 7.00-8.00 sec 438 MBytes 3.67 Gbits/sec 0 1.97 MBytes
[ 5] 8.00-9.00 sec 465 MBytes 3.91 Gbits/sec 0 1.97 MBytes
[ 5] 9.00-10.00 sec 470 MBytes 3.94 Gbits/sec 0 1.97 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 4.56 GBytes 3.91 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 4.56 GBytes 3.91 Gbits/sec receiver
iperf3 -s
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 749 MBytes 6.28 Gbits/sec
[ 5] 1.00-2.00 sec 523 MBytes 4.38 Gbits/sec
[ 5] 2.00-3.00 sec 436 MBytes 3.66 Gbits/sec
[ 5] 3.00-4.00 sec 453 MBytes 3.80 Gbits/sec
[ 5] 4.00-5.00 sec 447 MBytes 3.75 Gbits/sec
[ 5] 5.00-6.00 sec 457 MBytes 3.84 Gbits/sec
[ 5] 6.00-7.00 sec 666 MBytes 5.59 Gbits/sec
[ 5] 7.00-8.00 sec 785 MBytes 6.58 Gbits/sec
[ 5] 8.00-9.00 sec 795 MBytes 6.67 Gbits/sec
[ 5] 9.00-10.00 sec 785 MBytes 6.58 Gbits/sec
[ 5] 10.00-10.00 sec 1.19 MBytes 2.94 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 5.95 GBytes 5.11 Gbits/sec receiver
From lspci logs it looks like RP is downgraded to 8GT/s but EP is also downgraded to x1?
RP.eth0.log (4.3 KB) LnkSta: Speed 8GT/s (downgraded), Width x8 (ok)
EP.eth1.log (4.7 KB) LnkSta: Speed 2.5GT/s (downgraded), Width x1 (ok)
Does virtual Ethernet use DMA?
Do higher speeds require patching Orin EP or PC RP kernel or tvnet, or PCIe or Ethernet buffer sizes?
Thanks.
1 Like
Hi,
Following up on Enabling Orin Dev Kit Ethernet over PCIe - #8 by DaneLLL, @DaneLLL are these the max speeds that can be expected over virtual Ethernet?
If this is max or close to max speed, comparing to AGX Endpoint PCIe DMA speed - #6 by kayccc, Write 13.52GB/s=108Gb/s and Read 4.625GB/s=37Gb/s, tvnet is still more than magnitude slower than DMA. Tried to reproduce DMA test but having two issues.
– On EP side, /sys/kernel/debug/tegra_pcie_ep/ doesn’t show up after enabling CONFIG_PCIE_TEGRA_DW_DMA_TEST=y in tegra_defconfig, rebuilding kernel and rebooting? Running make menuconfig shows PCI support->PCI Endpoint->PCI Endpoint Test driver checked, ->Tegra PCIe Endpoint Test driver checked, but ->Tegra PCIe DMA test driver unchecked.
– One RP side, what would need to be changed in pci-epf-nv-test.c and Makefile to build and show /sys/kernel/debug/pcie-x/ on a PC?
Thanks.
Hi,
The PCIe interfaces on Orin developer kit are enabled by default. Please connect your device in poweroff and then power on the device. It shall be detected automatically if the device is valid. It should not be required to enable certain config, unless there is additional driver needs for the device.
Hi,
The basic PCIe connectivity between Orin EP and PC RP is not a problem based on lspci and devmem2, even though details indicate only 2.5GT/s and x1. But performance of virtual Ethernet over PCIe is an issue and is not clear if there is an underlying DMA issue since configuring separate DMA test is a problem.
– The virtual Ethernet test was enabled by Bringing up an Ethernet Interface over PCIe on EP and by compiling tvnet on RP. Assuming tvnet driver is required and is up to date, is 5Gb/s measured, max speed that can be expected?
– The DMA test was attempted by Connecting and Configuring the Devices and What is the actual maximum speed of Jetson AGX Xavier PCIE Ethernet? - #8 by Jasonm on EP. Assuming CONFIG_PCIE_TEGRA_DW_DMA_TEST=y is required and recompiling kernel and reboot Orin, reboot PC, /sys/kernel/debug/tegra_pcie_ep/ still does not show up on EP. Also not clear how to compile pci-epf-nv-test.c on PC that should enable /sys/kernel/debug/pcie-x/ to show up on RP side?
Thanks.
Hi @DaneLLL,
Updated to Jetpack 5.1.1 and its tegra_vnet.c on PC RP, speeds have not increased, still 4-6Gb/s.
– Has Nvidia done any internal testing if this is max expected which seem pretty low both relative to just cabled Ethernet and particularly PCIe x8, and are there any additional patches available for faster speed?
– If this is the max, and direct DMA is only way to get faster EP->RP speeds, have commands for EP Mode DMA and RP Mode DMA changed since Xavier, since they don’t seem to work for Orin, are there any updated instructions?
Thanks.
iperf3 -c 192.168.2.1
Connecting to host 192.168.2.1, port 5201
[ 5] local 192.168.2.2 port 37792 connected to 192.168.2.1 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 472 MBytes 3.96 Gbits/sec 0 1.11 MBytes
[ 5] 1.00-2.00 sec 471 MBytes 3.95 Gbits/sec 0 1.11 MBytes
[ 5] 2.00-3.00 sec 471 MBytes 3.96 Gbits/sec 0 1.11 MBytes
[ 5] 3.00-4.00 sec 478 MBytes 4.00 Gbits/sec 0 1.11 MBytes
[ 5] 4.00-5.00 sec 471 MBytes 3.96 Gbits/sec 0 1.11 MBytes
[ 5] 5.00-6.00 sec 471 MBytes 3.95 Gbits/sec 0 1.11 MBytes
[ 5] 6.00-7.00 sec 471 MBytes 3.96 Gbits/sec 0 1.11 MBytes
[ 5] 7.00-8.00 sec 472 MBytes 3.95 Gbits/sec 0 1.11 MBytes
[ 5] 8.00-9.00 sec 471 MBytes 3.96 Gbits/sec 0 1.11 MBytes
[ 5] 9.00-10.00 sec 471 MBytes 3.95 Gbits/sec 0 1.11 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 4.61 GBytes 3.96 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 4.61 GBytes 3.96 Gbits/sec receiver
iperf3 -c 192.168.2.2
Connecting to host 192.168.2.2, port 5201
[ 5] local 192.168.2.1 port 34704 connected to 192.168.2.2 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 519 MBytes 4.35 Gbits/sec 0 3.01 MBytes
[ 5] 1.00-2.00 sec 786 MBytes 6.59 Gbits/sec 0 3.01 MBytes
[ 5] 2.00-3.00 sec 682 MBytes 5.73 Gbits/sec 0 3.14 MBytes
[ 5] 3.00-4.00 sec 562 MBytes 4.72 Gbits/sec 0 3.14 MBytes
[ 5] 4.00-5.00 sec 490 MBytes 4.10 Gbits/sec 0 3.14 MBytes
[ 5] 5.00-6.00 sec 482 MBytes 4.05 Gbits/sec 0 3.14 MBytes
[ 5] 6.00-7.00 sec 555 MBytes 4.65 Gbits/sec 0 3.14 MBytes
[ 5] 7.00-8.00 sec 789 MBytes 6.61 Gbits/sec 0 3.14 MBytes
[ 5] 8.00-9.00 sec 785 MBytes 6.59 Gbits/sec 0 3.14 MBytes
[ 5] 9.00-10.00 sec 801 MBytes 6.71 Gbits/sec 0 3.14 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 6.30 GBytes 5.41 Gbits/sec 0 sender
[ 5] 0.00-10.01 sec 6.30 GBytes 5.41 Gbits/sec receiver