Jetson AGX Thor MGBE: TX packet flood (~200k pkt/s) with UDP frames larger than 1500 bytes (USRP X310 / UHD)

Robotics & Edge Computing Jetson Systems Jetson Thor
1

Hi,

I’m streaming TX samples from a Jetson AGX Thor to an Ettus USRP X310 over the QSFP port at 10GbE. With UDP frames larger than the standard 1500-byte size, tpackets than the applicahe MGBE interface starts sending far more tion sends, at a fixed rate of about 200k packets/s. Frames of 1472 bytes work correctly.

Setup

  • Jetson AGX Thor Developer Kit, JetPack 7.2.0
  • QSFP port in the default 10GbE mode, interface mgbe0_0
  • Connected to the USRP X310 SFP+ port via QSFP-to-4xSFP+ breakout cable
  • UHD 4.5.0.0, tested with benchmark_rate (TX only)
  • MTU 8932, threaded NAPI enabled, jetson_clocks applied

Symptom With send_frame_size above 1472, TX starts at the expected rate. Begins with a bunch of num sequence errors(tx) SSSSSSS, After a few seconds - the TX packet count in /sys/class/net/mgbe0_0/statistics/tx_packets jumps to about 190–200k packets/s and stays there. UHD then reports continuous S (sequence) errors and eventually IOError: x300 fw poke32 - reply timed out.

send_frame_size TX rate Expected pkt/s (approx.) Observed after jump
1472 10–50 MS/s ~27k–137k no jump, works correctly
2000 10 MS/s ~20k ~3000 Mb/s (~190k pkt/s)
3000 10 MS/s ~13k ~4500–4600 Mb/s (~190k pkt/s)
4000 10 / 20 / 50 MS/s ~10k / 20k / 50k ~6000 Mb/s (~200k pkt/s) in all three cases

Observations

  • After the jump, the packet rate stays at about 200k pkt/s whatever the frame size or sample rate. That is several times what the application sends, so it looks like the driver is sending the same packets more than once.
  • It only happens with frames larger than a standard 1500-byte Ethernet frame. 1472-byte frames run without errors.
  • ethtool -L returns “Operation not supported” on MGBE. Steering all TX traffic to queue 0 with XPS did not change the behavior.
  • Turning on threaded NAPI did not help.

How to reproduce

# terminal 1
benchmark_rate --args "addr=<usrp_ip>,send_frame_size=4000" --tx_rate 10e6 --duration 60

# terminal 2
watch -n1 cat /sys/class/net/mgbe0_0/statistics/tx_packets

Questions

  1. Is this a known issue in the MGBE TX path with jumbo or multi-buffer frames?
  2. Is there a driver setting or patch to work around it?
  3. Can you reproduce it on a dev kit, for example with iperf3 -u -l 4000 -b 320M to another host?

Thanks

4

Not any known issue so not able to provide any patch yet.

We will check if we could reproduce with NV devkit to NV devkit first.

7

Tried with Thor to Thor connection with MTU size to 8932.

However, it looks like the tx_packets not increase as that fast.

Could you confirm if below result means not reproduce the issue?

tx_packets results pre-second: after iperf starts

$ cat /sys/class/net/mgbe0_0/statistics/tx_packets
9412820
$ cat /sys/class/net/mgbe0_0/statistics/tx_packets
9427095
$ cat /sys/class/net/mgbe0_0/statistics/tx_packets
9438735
$ cat /sys/class/net/mgbe0_0/statistics/tx_packets
9451525
$ cat /sys/class/net/mgbe0_0/statistics/tx_packets
9464195
$ cat /sys/class/net/mgbe0_0/statistics/tx_packets
9475815
$ cat /sys/class/net/mgbe0_0/statistics/tx_packets
9487385
$ cat /sys/class/net/mgbe0_0/statistics/tx_packets
9503815
$ cat /sys/class/net/mgbe0_0/statistics/tx_packets
9513685
$ cat /sys/class/net/mgbe0_0/statistics/tx_packets
9526105
8

Hi Wayne,

Thanks for testing. More results from my side, and a correction to my first post: comparing OutDatagrams in /proc/net/snmp with tx_packets, both rise to ~200k/s together during the flood. So the extra packets come from UHD, not from the driver duplicating them. UHD probably stops limiting its send rate after the sequence errors start.

Results:

  • iperf 2 UDP, 4000-byte packets, Thor → X310: rate stays normal
  • UHD RX, X310 → Thor, 4000-byte frames: works
  • ping -M do -s 3972 Thor → X310: replies received
  • UHD TX, send_frame_size=1472: works
  • UHD TX, send_frame_size 2000–4000: sequence errors from the start
  • sg and TX checksum offload off: no change
  • Same UHD TX test from an x86 PC with a 10G SFP+ NIC to the same X310: works

So it only fails with UHD TX from the Thor using frames above 1472 bytes.

Are there any MGBE counters or debug options worth checking during the first seconds of streaming?

9

Is this issue only happened to USRP? Are you able to reproduce issue on Devkit so that we could proceed?

10

Currently i wasn’t able to reproduce with iperf. Thanks for your time, I’ll write if there will progress.

11

Hi Wayne,

I now have an iperf3 result without the USRP involved. I tested UDP from the Thor to another Ubuntu PC over a QSFP-to-SFP+ breakout cable, with MTU 9000 on both interfaces, at 320 Mbit/s for 20 seconds.

The PC reports increasing numbers of out-of-order datagrams as I increase -l:

  • 1472: a few
  • 2000: ~1,300
  • 4000: ~25,000
  • 8000: ~45,000

In the reverse direction, I don’t see an out-of-order report. Could this be related to the UHD TX sequence errors with larger frames? What counters or debug logs would help investigate?

Thanks

12

Are there any ideas?