Orin NX 16GB experiencing low throughput on Ethernet connection

[pgonzalez3]

We have an Orin installed on a custom carrier board and connected to a KSZ9897R ethernet switch. Due to the physical connections we are limited to 100Mbps. In the process of testing the connections (we are using iperf), we have found that every feasible connection gives us the throughput we expect (near 100Mbps). However, as soon as we run the test with the Orin, we see a significantly reduced amount of throughput.

Getting data FROM the Orin does not seem to be a problem. The issue seems to be when trying to transfer data TO the Orin, whether by iperf or by scp, sftp, etc.

We’re already looking into any potential issues with the network switch.

Is there an Orin setting we’re missing? Currently configured to 100Mbps, full duplex, autonegotiate off. No firewall.

The command we run:

iperf3.exe -c <dest_ip> --bind <src_ip> --bidir

Sample output to/from another laptop endpoint:

iperf3.exe -c <dest_ip> --bind <src_ip> --bidir
Connecting to host <dest_ip>, port 5201
[  5] local <src_ip> port 51531 connected to <dest_ip> port 5201
[  7] local <src_ip> port 51532 connected to <dest_ip> port 5201
[ ID][Role] Interval           Transfer     Bitrate
[  5][TX-C]   0.00-1.01   sec  8.50 MBytes  70.6 Mbits/sec
[  7][RX-C]   0.00-1.01   sec  9.00 MBytes  74.7 Mbits/sec
[  5][TX-C]   1.01-2.00   sec  9.38 MBytes  79.2 Mbits/sec
[  7][RX-C]   1.01-2.00   sec  10.5 MBytes  88.7 Mbits/sec
[  5][TX-C]   2.00-3.01   sec  11.0 MBytes  91.7 Mbits/sec
[  7][RX-C]   2.00-3.01   sec  11.0 MBytes  91.7 Mbits/sec
[  5][TX-C]   3.01-4.01   sec  11.0 MBytes  91.9 Mbits/sec
[  7][RX-C]   3.01-4.01   sec  10.2 MBytes  85.6 Mbits/sec
[  5][TX-C]   4.01-5.01   sec  10.8 MBytes  90.9 Mbits/sec
[  7][RX-C]   4.01-5.01   sec  9.38 MBytes  79.3 Mbits/sec
[  5][TX-C]   5.01-6.01   sec  10.9 MBytes  90.7 Mbits/sec
[  7][RX-C]   5.01-6.01   sec  9.50 MBytes  79.3 Mbits/sec
[  5][TX-C]   6.01-7.01   sec  10.9 MBytes  91.7 Mbits/sec
[  7][RX-C]   6.01-7.01   sec  9.25 MBytes  78.0 Mbits/sec
[  5][TX-C]   7.01-8.01   sec  10.9 MBytes  90.6 Mbits/sec
[  7][RX-C]   7.01-8.01   sec  9.38 MBytes  78.1 Mbits/sec
[  5][TX-C]   8.01-9.00   sec  10.6 MBytes  90.0 Mbits/sec
[  7][RX-C]   8.01-9.00   sec  9.00 MBytes  76.3 Mbits/sec
[  5][TX-C]   9.00-10.01  sec  10.9 MBytes  90.7 Mbits/sec
[  7][RX-C]   9.00-10.01  sec  9.75 MBytes  81.3 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID][Role] Interval           Transfer     Bitrate
[  5][TX-C]   0.00-10.01  sec   105 MBytes  87.8 Mbits/sec                  sender
[  5][TX-C]   0.00-10.02  sec   105 MBytes  87.7 Mbits/sec                  receiver
[  7][RX-C]   0.00-10.01  sec  97.4 MBytes  81.6 Mbits/sec                  sender
[  7][RX-C]   0.00-10.02  sec  97.0 MBytes  81.2 Mbits/sec                  receiver

iperf Done.

Sample output to/from the Orin:

Connecting to host <orin_ip>, port 5201
[  5] local <src_ip> port 53376 connected to <orin_ip> port 5201
[  7] local <src_ip> port 53377 connected to <orin_ip> port 5201
[ ID][Role] Interval           Transfer     Bitrate
[  5][TX-C]   0.00-1.01   sec   256 KBytes  2.07 Mbits/sec
[  7][RX-C]   0.00-1.01   sec  4.88 MBytes  40.4 Mbits/sec
[  5][TX-C]   1.01-2.01   sec  0.00 Bytes  0.00 bits/sec
[  7][RX-C]   1.01-2.01   sec  4.12 MBytes  34.7 Mbits/sec
[  5][TX-C]   2.01-3.01   sec  0.00 Bytes  0.00 bits/sec
[  7][RX-C]   2.01-3.01   sec  4.38 MBytes  36.8 Mbits/sec
[  5][TX-C]   3.01-4.01   sec   128 KBytes  1.05 Mbits/sec
[  7][RX-C]   3.01-4.01   sec  3.50 MBytes  29.4 Mbits/sec
[  5][TX-C]   4.01-5.00   sec  0.00 Bytes  0.00 bits/sec
[  7][RX-C]   4.01-5.00   sec  4.50 MBytes  37.9 Mbits/sec
[  5][TX-C]   5.00-6.01   sec  0.00 Bytes  0.00 bits/sec
[  7][RX-C]   5.00-6.01   sec  4.25 MBytes  35.3 Mbits/sec
[  5][TX-C]   6.01-7.01   sec  0.00 Bytes  0.00 bits/sec
[  7][RX-C]   6.01-7.01   sec  4.38 MBytes  36.8 Mbits/sec
[  5][TX-C]   7.01-8.01   sec  0.00 Bytes  0.00 bits/sec
[  7][RX-C]   7.01-8.01   sec  5.12 MBytes  42.8 Mbits/sec
[  5][TX-C]   8.01-9.00   sec  0.00 Bytes  0.00 bits/sec
[  7][RX-C]   8.01-9.00   sec  4.38 MBytes  37.0 Mbits/sec
[  5][TX-C]   9.00-10.02  sec  0.00 Bytes  0.00 bits/sec
[  7][RX-C]   9.00-10.02  sec  4.25 MBytes  35.2 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID][Role] Interval           Transfer     Bitrate         Retr
[  5][TX-C]   0.00-10.02  sec   384 KBytes   314 Kbits/sec                  sender
[  5][TX-C]   0.00-10.05  sec   164 KBytes   134 Kbits/sec                  receiver
[  7][RX-C]   0.00-10.02  sec  44.0 MBytes  36.8 Mbits/sec  2441            sender
[  7][RX-C]   0.00-10.05  sec  43.8 MBytes  36.5 Mbits/sec                  receiver

iperf Done.

[WayneWWW]

Do you have other kind of switch that can test? I didn’t hear about such issue before.

[pgonzalez3]

The problem does not appear to be present with a different switch, or even on a direct connection to an Orin Nano dev kit.

[pgonzalez3]

The problem is present when the physical connection is limited to 100Mbps, but only occurs when connected to an Orin, either the Orin NX 16GB we have onboard OR an Orin Nano Developer kit.
RX from the Orin is OK.
TX to the Orin is on the order of kbps.

[sgidel]

Are you using the DSA driver for that switch with JP 5.x? There is a bug that causes the RX performance to be terrible due it not handling TSO properly. Try updating to JP6.x or you can patch the dsa-core driver.

KSZ DSA driver transmission speed issue

If this is your issue, running a UDP iperf test would show the expected speed in both directions.

[pgonzalez3]

We are running jetson Linux 36.3 (Jetpack 6.0). Running iperf with UDP gives us the expected throughput in both directions. I’ll look into the driver issue you mentioned.

[pgonzalez3]

Took some time to implement the fix as described in this post: KSZ DSA driver transmission speed issue · Issue #86 · Microchip-Ethernet/EVB-KSZ9477 · GitHub
The issue is still present. Strictly when we are trying to transfer large files or running iperf with TCP.
But what’s different here is the driver. The output of lspci | grep -i eth is:

0001:01:00.0 Ethernet controller: Intel Corporation Device 125c (rev 04)
0008:01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15)

So it looks like we’re not using the DSA driver…

[WayneWWW]

Just to clarify. Please make a clear statement for what is the exact scenario that would hit this error.

You mentioned 100Mbps and also KSZ9897R. Do you mean you need both these two things together to reproduce error or even setting 100Mbps on Orin + other kind of switch could reproduce this?

[pgonzalez3]

The problem occurs when we connect to an Orin through the KSZ9897R switch AND the link speed is physically or logically limited to 100Mbps.

For the purposes of condensing the data captured below:

• CG = CAT6 or CAT5e cable limited to 1Gbps
• CM = Only 4 wires (two twisted pairs), limited to 100Mbps
• PC = Windows laptop running iperf3 as client
• PC100 = Windows laptop configured to 100Mbps Full Duplex
• Orin = Either Orin NX 16GB or Orin Nano 8GB dev kit running iperf3 as server. The error was reproducible on both types of devices.
• Orin100 = Orin configured to 100Mbps with ethtool -s eth0 speed 100 duplex full autoneg off
• KSZ = KSZ9897R
• KSZ100 = Resistor in place for strapping option to peg all interfaces to 100Mbps

All tests were run at least for 30 seconds. UDP returned expected throughput (between 93 and 96 Mbps), but not TCP.

Here’s what we’ve tried:

Configuration	iperf TCP TX to Orin	iperf TCP RX from Orin
[PC] ← CG → [KSZ] ← CG → [Orin]	940 Mbps	949 Mbps
[PC] ← CG → [KSZ] ← CG → [Orin100]	1.29 Mbps	38.1 Mbps
[PC] ← CM → [KSZ] ← CG → [Orin]	104 kbps	94.9 Mbps
[PC] ← CM → [KSZ] ← CG → [Orin100]	415 kbps	49 Mbps
[PC] ← CG → [KSZ100] ← CG → [Orin]	45 Mbps	66.8 Mbps
[PC] ← CM → [KSZ100] ← CG → [Orin]	17 Mbps	70 Mbps

What’s interesting is that when operating at Gigabit speeds there doesn’t seem to be an issue. However, the intent is to place this on a remotely controlled or autonomous vehicle with a C2 link via a radio for which the connection consists of 4 wires and is limited to 100Mbps, which is why we are trying to make this work.

Also, when the KSZ switch was strapped down to 100Mbps, the Orin was reporting 100Mbps half duplex

We could not reproduce the issue using a different Ethernet switch and Orin Dev Kit, neither by picking a 100Mbps-limited cable nor by setting the ethtool configuration. In both those cases throughput was reporting a nominal 100Mbps as expected.

[WayneWWW]

Hi,

It is unlikely for us to check as we don’t have such switch on our side.

Is it possible to change to other kind of switch?

[pgonzalez3]

Hi, unfortunately not at this point, no, we cannot switch to another one.

[pgonzalez3]

Hi, we’ve performed additional testing and have identified that the issue is present for this specific hardware setup:

• Ubuntu 22.04 or 24.04 running the r8169 driver
• Gigabit link (CAT5e or better)
• KSZ9897R ethernet switch
• 100BaseT link (only two twisted pairs)
• Windows or Linux host transmitting data (iperf or video stream) to the Ubuntu.

We’ve tested the exact same setup with a Raspbery Pi or a Windows laptop in place of the Orin (Ubunu 22.04) or Ubunu 24.04 Host and see 100Mbps throughput.