Reduced bandwidth at 10Gbps on Orin and mgbe_payload_cs_err correlation

Robotics & Edge Computing Jetson & Embedded Systems Jetson AGX Orin
5

It seems that the patch I created to allow a packet size of 9000 (see RGMII issue with Jetpack 5.1.1 (KSZ9131) - #7 by waterbear) is inducing the issue. I decided to try different MTU sizes to see what effects those had (e.g. 1500 → 3000 → 6000 → 9000). At 9000 the troubles appeared. I backed things off to 8966 and don’t see retransmit / mgbe_payload_cs_errs. Here’s an example run:

orin@orin64-alpha:~/projects/romulan/scripts$ sudo ./update-mtu.sh eth0 9000
orin@orin64-alpha:~/projects/romulan/scripts$ ethtool -S eth0 | grep payload
     payload_cs_error: 0
     mgbe_payload_cs_err: 610
orin@orin64-alpha:~/projects/romulan/scripts$ iperf3 -Z --client 10.0.3.1 -t 5
Connecting to host 10.0.3.1, port 5201
[  5] local 10.0.3.10 port 39528 connected to 10.0.3.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   509 MBytes  4.27 Gbits/sec  232   1.02 MBytes
[  5]   1.00-2.00   sec   646 MBytes  5.42 Gbits/sec  117   1.04 MBytes
[  5]   2.00-3.00   sec   584 MBytes  4.90 Gbits/sec  231   1.05 MBytes
[  5]   3.00-4.00   sec   952 MBytes  7.99 Gbits/sec    0   1.05 MBytes
[  5]   4.00-5.00   sec   952 MBytes  7.99 Gbits/sec    0   1.05 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-5.00   sec  3.56 GBytes  6.11 Gbits/sec  580             sender
[  5]   0.00-5.00   sec  3.56 GBytes  6.11 Gbits/sec                  receiver

iperf Done.
orin@orin64-alpha:~/projects/romulan/scripts$ ethtool -S eth0 | grep payload
     payload_cs_error: 0
     mgbe_payload_cs_err: 615
orin@orin64-alpha:~/projects/romulan/scripts$ sudo ./update-mtu.sh eth0 8966
orin@orin64-alpha:~/projects/romulan/scripts$ ethtool -S eth0 | grep payload
     payload_cs_error: 0
     mgbe_payload_cs_err: 615
orin@orin64-alpha:~/projects/romulan/scripts$ iperf3 -Z --client 10.0.3.1 -t 5
Connecting to host 10.0.3.1, port 5201
[  5] local 10.0.3.10 port 56760 connected to 10.0.3.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   935 MBytes  7.84 Gbits/sec    0   1.03 MBytes
[  5]   1.00-2.00   sec   936 MBytes  7.85 Gbits/sec    0   1.03 MBytes
[  5]   2.00-3.00   sec   936 MBytes  7.85 Gbits/sec    0   1.03 MBytes
[  5]   3.00-4.00   sec   936 MBytes  7.85 Gbits/sec    0   1.03 MBytes
[  5]   4.00-5.00   sec   938 MBytes  7.86 Gbits/sec    0   1.03 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-5.00   sec  4.57 GBytes  7.85 Gbits/sec    0             sender
[  5]   0.00-5.00   sec  4.57 GBytes  7.85 Gbits/sec                  receiver

iperf Done.
orin@orin64-alpha:~/projects/romulan/scripts$ ethtool -S eth0 | grep payload
     payload_cs_error: 0
     mgbe_payload_cs_err: 615

I can’t update the other thread since that’s locked, so hopefully anyone stumbling on that will see this one as well.

I’m still not sure what’s up and assume something funky is happening in the MGBE controller when a packet size of 9000 is defined.