Jetson Thor T5000 MGBE

Robotics & Edge Computing Jetson Systems Jetson Thor
1

================================================================================

Jetson T5000 QSFP28 (MGBE) Throughput Below Expected Line Rate

Document purpose : Problem description for NVIDIA Developer Forum submission
Date : 2026-06-18
Author : (fill in your name / organization)

FORUM SUBJECT LINE

Jetson T5000 QSFP28 (MGBE) throughput far below line rate — ~12.7 Gbps single
stream, ~3 Gbps per stream with 4 parallel flows

================================================================================

  1. ISSUE SUMMARY

We are benchmarking QSFP28 optical Ethernet on NVIDIA Jetson T5000 and observe
throughput well below the expected 25 Gbps per port.

Test scenario Observed throughput Expected
Single TCP stream (1 link) ~12.6–12.7 Gbps ~25 Gbps (QSFP28)
Four parallel TCP streams (4 links) ~2.8–3.5 Gbps per stream ~25 Gbps per link 
                                   | (~12 Gbps aggregate)         | independently

The combined behavior suggests an aggregate bottleneck around ~12 Gbps shared
across ports, rather than independent 25 Gbps per interface.

Question: Is this the current expected performance on T5000, or is additional
driver/kernel configuration required to reach full QSFP28 line rate?

================================================================================

  1. PLATFORM AND ENVIRONMENT
Item Details
Device NVIDIA Jetson T5000
Interfaces QSFP28 optical ports: mgbe0_0, mgbe1_0, mgbe2_0, mgbe3_0
OS Ubuntu on Jetson (tegra-ubuntu)
Tool iperf3
Power/clocks sudo nvpmodel -m 0 and sudo jetson_clocks on all test nodes
MGBE config threaded mode enabled on mgbe0_0–mgbe3_0 (server and client)

Network topology:

Role IP addresses Notes
Server 192.168.1.100–103 One iperf3 -s instance per port
Client 192.168.1.200–203 One iperf3 -c instance per port

Each client stream is bound to its own source IP and targets the matching server
IP/port (1:1 mapping across four links).

================================================================================

  1. TEST PROCEDURE

3.1 Server side (192.168.1.100 – 103)

sudo nvpmodel -m 0
sudo jetson_clocks
echo 1 > /sys/devices/platform/bus@0/a808a10000.ethernet/net/mgbe0_0/threaded
echo 1 > /sys/devices/platform/bus@0/a808b10000.ethernet/net/mgbe1_0/threaded
echo 1 > /sys/devices/platform/bus@0/a808d10000.ethernet/net/mgbe2_0/threaded
echo 1 > /sys/devices/platform/bus@0/a808e10000.ethernet/net/mgbe3_0/threaded

iperf3 -s -B 192.168.1.100 -p 5010 &
iperf3 -s -B 192.168.1.101 -p 5011 &
iperf3 -s -B 192.168.1.102 -p 5012 &
iperf3 -s -B 192.168.1.103 -p 5013 &

3.2 Client side (192.168.1.200 – 203)

sudo nvpmodel -m 0
sudo jetson_clocks
echo 1 > /sys/devices/platform/bus@0/a808a10000.ethernet/net/mgbe0_0/threaded
echo 1 > /sys/devices/platform/bus@0/a808b10000.ethernet/net/mgbe1_0/threaded
echo 1 > /sys/devices/platform/bus@0/a808d10000.ethernet/net/mgbe2_0/threaded
echo 1 > /sys/devices/platform/bus@0/a808e10000.ethernet/net/mgbe3_0/threaded

iperf3 -c 192.168.1.100 -B 192.168.1.200 -i 1 -t 300 -p 5010
iperf3 -c 192.168.1.101 -B 192.168.1.201 -i 1 -t 300 -p 5011
iperf3 -c 192.168.1.102 -B 192.168.1.202 -i 1 -t 300 -p 5012
iperf3 -c 192.168.1.103 -B 192.168.1.203 -i 1 -t 300 -p 5013

3.3 MGBE threaded mode (configured and enabled on all test nodes)

echo 1 > /sys/devices/platform/bus@0/a808a10000.ethernet/net/mgbe0_0/threaded
echo 1 > /sys/devices/platform/bus@0/a808b10000.ethernet/net/mgbe1_0/threaded
echo 1 > /sys/devices/platform/bus@0/a808d10000.ethernet/net/mgbe2_0/threaded
echo 1 > /sys/devices/platform/bus@0/a808e10000.ethernet/net/mgbe3_0/threaded

Note: The above threaded mode was applied on both server and client sides before
running iperf3. Throughput results in Section 4 were obtained with this
configuration already in place.

================================================================================

  1. TEST RESULTS

4.1 Single stream (baseline)

Command:
iperf3 -c 192.168.1.100 -B 192.168.1.200 -i 1 -t 30 -p 5010

Result:
Connecting to host 192.168.1.100, port 5010
[ 5] local 192.168.1.200 port 54357 connected to 192.168.1.100 port 5010
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 1.21 GBytes 10.4 Gbits/sec 124 794 KBytes
[ 5] 1.00-2.00 sec 1.47 GBytes 12.7 Gbits/sec 21 1.14 MBytes
[ 5] 2.00-3.00 sec 1.48 GBytes 12.7 Gbits/sec 1 1.46 MBytes
…
[ 5] 29.00-30.00 sec 1.48 GBytes 12.7 Gbits/sec 0 1.72 MBytes

[ 5] 0.00-30.00 sec 44.0 GBytes 12.6 Gbits/sec 162 sender
[ 5] 0.00-30.00 sec 44.0 GBytes 12.6 Gbits/sec receiver

iperf Done.

Summary:

  • Steady-state throughput: ~12.7 Gbits/sec after TCP slow start
  • Total retransmissions: 162 over 30 seconds (124 in the first second)

4.2 Four simultaneous streams

When all four client/server pairs run in parallel, each stream averages only
~3 Gbps, with highly variable per-second rates. Several runs showed 0.00
bits/sec on the receiver summary line (tests interrupted with Ctrl-C).

Stream (client → server) Duration Avg throughput (sender)
192.168.1.200 → .100 (port 5010) ~262 s 3.32 Gbits/sec
192.168.1.201 → .101 (port 5011) ~256 s 3.45 Gbits/sec
192.168.1.202 → .102 (port 5012) ~248 s 2.78 Gbits/sec
192.168.1.203 → .103 (port 5013) ~223 s 3.09 Gbits/sec

Observation: Total aggregate throughput across 4 links is approximately
~12 Gbps, consistent with a shared bottleneck rather than 4 x 25 Gbps.

Example excerpt (stream 1, interrupted):
[ 5] 258.00-259.00 sec 417 MBytes 3.50 Gbits/sec 0 2.06 MBytes
[ 5] 259.00-260.00 sec 572 MBytes 4.80 Gbits/sec 0 2.06 MBytes
[ 5] 260.00-261.00 sec 754 MBytes 6.32 Gbits/sec 0 2.06 MBytes
[ 5] 261.00-262.00 sec 1.34 GBytes 11.5 Gbits/sec 2 2.06 MBytes
[ 5] 0.00-262.09 sec 101 GBytes 3.32 Gbits/sec 28 sender
[ 5] 0.00-262.09 sec 0.00 Bytes 0.00 bits/sec receiver
iperf3: interrupt - the client has terminated

================================================================================

  1. ADDITIONAL TESTS (NO IMPROVEMENT)

The following were also tried without meaningful improvement (MGBE threaded
mode was already enabled during all tests above):

  • Changing iperf3 parallel stream count (-P)
  • UDP tests
  • Setting target bitrate with -b 25G

None of these raised throughput above the ~12 Gbps ceiling (single stream) or
~12 Gbps aggregate (four streams).

================================================================================

  1. LINK-LAYER OBSERVATIONS (OPTIONAL)

On client interface mgbe0_0, ethtool -S showed non-zero receive error counters
during testing, for example:

mmc_rx_crc_error: 125
mmc_rx_packet_smd_err_cnt: 158606

TCP retransmission count (162 over 30 s in single-stream test) may correlate
with physical-layer receive errors. Full ethtool output can be provided on
request.

================================================================================

  1. QUESTIONS FOR NVIDIA

  2. Is ~12.7 Gbps per MGBE port (TCP, single stream) the current expected
    maximum on Jetson T5000, or should we expect near 25 Gbps per QSFP28 link?

  3. When using all four MGBE ports concurrently, should each port achieve
    independent line rate, or is there a known platform/DMA/PCIe aggregate
    bandwidth limit around ~12 Gbps?

  4. Are there recommended kernel/driver tunings for MGBE on T5000 (e.g.
    threaded sysfs, IRQ affinity, ring buffer sizes, MTU, firmware settings) to
    reach full QSFP28 performance?

  5. Could retransmissions / link-layer errors limit TCP throughput in this
    setup, and what ethtool -S / debug steps do you recommend?

================================================================================

END OF DOCUMENT

3

They did mention something about this in the Release Notes for the latest Jetson Linux.

image
image1230×714 198 KB