QSFP can not reach 100G

Robotics & Edge Computing Jetson Systems Jetson Thor
,
1

Hello NVIDIA experts, I have now upgraded to version 7.1, but after testing, I found that the single 25Gb connection is still around 11Gb, and the 4-link bond together is 33Gb. Previously, with version r38.2.1, the speed was able to reach over 40Gb. It was mentioned that this issue would be addressed in this release version, but the speed still remains at these levels after testing. Could you help explain why this is happening? Additionally, I have already set nvpmodel to the maxN mode, set jetson_clocks to the maximum frequency, and enabled the threaded multi-threading for MGBE. and udp is about 6Gbps.Besides the above, are there any additional configurations I need to make? Did I miss anything?

mgbe0_0

root@tegra-ubuntu:/etc/netplan# iperf3 -c 192.168.139.12 -b 25G -P 10
Connecting to host 192.168.139.12, port 5201
[  5] local 192.168.139.13 port 43138 connected to 192.168.139.12 port 5201
[  7] local 192.168.139.13 port 43142 connected to 192.168.139.12 port 5201
[  9] local 192.168.139.13 port 43150 connected to 192.168.139.12 port 5201
[ 11] local 192.168.139.13 port 43156 connected to 192.168.139.12 port 5201
[ 13] local 192.168.139.13 port 43168 connected to 192.168.139.12 port 5201
[ 15] local 192.168.139.13 port 43170 connected to 192.168.139.12 port 5201
[ 17] local 192.168.139.13 port 43178 connected to 192.168.139.12 port 5201
[ 19] local 192.168.139.13 port 43184 connected to 192.168.139.12 port 5201
[ 21] local 192.168.139.13 port 43190 connected to 192.168.139.12 port 5201
[ 23] local 192.168.139.13 port 43206 connected to 192.168.139.12 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  93.9 MBytes   787 Mbits/sec   16    228 KBytes
[  7]   0.00-1.00   sec  78.4 MBytes   657 Mbits/sec   14    236 KBytes
[  9]   0.00-1.00   sec   162 MBytes  1.35 Gbits/sec   18    325 KBytes
[ 11]   0.00-1.00   sec   216 MBytes  1.81 Gbits/sec   11    290 KBytes
[ 13]   0.00-1.00   sec   130 MBytes  1.09 Gbits/sec   13    238 KBytes
[ 15]   0.00-1.00   sec   121 MBytes  1.02 Gbits/sec   20    312 KBytes
[ 17]   0.00-1.00   sec   165 MBytes  1.38 Gbits/sec   16    269 KBytes
[ 19]   0.00-1.00   sec   186 MBytes  1.56 Gbits/sec   17    384 KBytes
[ 21]   0.00-1.00   sec  98.2 MBytes   824 Mbits/sec   17    195 KBytes
[ 23]   0.00-1.00   sec   132 MBytes  1.11 Gbits/sec   10    260 KBytes
[SUM]   0.00-1.00   sec  1.35 GBytes  11.6 Gbits/sec  152
Thor T5000's QSFP interface can not reach 100G
3

It is a known issue that x4 25MGBE won’t have fully performance and we are still checking this internally.

4

@wpceswpces Did you do following? If so what does this show? cat /sys/class/net/mgbe*_*/speed

This document shows 4 mgbe devices all at 25G in Jetpack.

BPMP=bootloader/generic/tegra264-bpmp-3834-0008-4071-xxxx.dtb

dtc -I dtb -O dts -o bpmp.dts "$BPMP"

edit bpmp.dts to change to, I believe, these values

        uphy {
                status = "okay";
                uphy0-config = <0x06>;
                uphy1-config = <0x08>;
                mgbe0-speed = <0x03>;   /* 0=2.5G,1=5G,2=10G,3=25G */
                mgbe1-speed = <0x03>;
                mgbe2-speed = <0x03>;
                mgbe3-speed = <0x03>;
        };

Rebuild dtb and cp it home.

dtc -I dts -O dtb -o tegra264-bpmp-3834-0008-4071-xxxx.dtb bpmp.dts

cp -v tegra264-bpmp-3834-0008-4071-xxxx.dtb "$BPMP"

Then Enable 25 Gigabit Ethernet on QSFP Port

https://docs.nvidia.com/jetson/archives/r38.4/DeveloperGuide/SD/Kernel/Enable25GbEthernetOnQSFP.html
5

Hi，whitesscott

i have do that ,One MGBE port link is 25Gb, and after bonding, the four links show as 100Gb.

1 Like
6

Okay, I previously received feedback saying that the next version after 7.0 would fix it, so I waited for 7.1, but I found that there hasn’t been any change.

7

Hello, I would like to ask another question. I don’t need the QSFP for camera-related tasks, and only want to use it for general network purposes. I noticed that some of the dmesg messages are related to camera-related drivers. In my case, if I disable the mgbe and camera-related components, will it improve the network speed?

[   14.576655] camrtc-coe tegra-capture-coe0: Camera Over Eth controller a808a10000.ethernet num_chans=5 IRQ=4
[   15.103475] camrtc-coe tegra-capture-coe1: netdev event 5 dev mgbe1_0
[   15.110623] camrtc-coe tegra-capture-coe1: Ch4->PDMA4
[   15.114749] camrtc-coe tegra-capture-coe1: Ch5->PDMA5
[   15.119950] camrtc-coe tegra-capture-coe1: Ch6->PDMA6
[   15.124832] camrtc-coe tegra-capture-coe1: Ch7->PDMA7
[   15.130064] camrtc-coe tegra-capture-coe1: Ch8->PDMA7
[   15.135072] camrtc-coe tegra-capture-coe1: Camera Over Eth controller a808b10000.ethernet num_chans=5 IRQ=4
[   15.144938] camrtc-coe tegra-capture-coe2: netdev event 5 dev mgbe2_0
[   15.151428] camrtc-coe tegra-capture-coe2: Ch4->PDMA4
[   15.156259] camrtc-coe tegra-capture-coe2: Ch5->PDMA5
[   15.161532] camrtc-coe tegra-capture-coe2: Ch6->PDMA6
[   15.161750] camrtc-coe tegra-capture-coe2: Ch7->PDMA7
[   15.161795] camrtc-coe tegra-capture-coe2: Ch8->PDMA7
[   15.161952] camrtc-coe tegra-capture-coe2: Camera Over Eth controller a808d10000.ethernet num_chans=5 IRQ=4
[   15.186492] camrtc-coe tegra-capture-coe3: netdev event 5 dev mgbe3_0
[   15.192987] camrtc-coe tegra-capture-coe3: Ch4->PDMA4
[   15.197844] camrtc-coe tegra-capture-coe3: Ch5->PDMA5
[   15.203054] camrtc-coe tegra-capture-coe3: Ch6->PDMA6
[   15.207955] camrtc-coe tegra-capture-coe3: Ch7->PDMA7
[   15.213184] camrtc-coe tegra-capture-coe3: Ch8->PDMA7
[   15.218151] camrtc-coe tegra-capture-coe3: Camera Over Eth controller a808e10000.ethernet num_chans=5 IRQ=4
8

I did following just because a lot of dmesg log writes about coe and figured it was using resources I could use.

cat /etc/modprobe.d/jetson-no-camera.conf

# Disable Jetson camera stack (no sensors attached)
blacklist camchar
blacklist camera_diagnostics
blacklist rtcpu_debug
blacklist tegra_camera
blacklist tegra_camera_rtcpu
blacklist tegra_capture_isp
blacklist tegra_capture_coe
blacklist tegra_camera_platform
# DO NOT blacklist core V4L2 modules like videodev/videobuf2_*
9

hi，whitesscott

After you blacklist the camera-related drivers, is there any indication that the mgbe rate has improved compared to before? What is your maximum rate now?

10

Did not notice any change and haven’t iperf3 it. I am not approaching your implementation; still just at 10gb with only a 10gb switch.

11

Hi,whitesscott

Okay, thank you very much!

12

Hi,

Could you please clarify how you aggregated the four 25G MGBE interfaces into 100G?

Thanks.

13

Hi,cray-III

I aggregated them together using bonding.

14

hi,whitesscott

root@tegra-ubuntu:~# lsmod |grep camera
root@tegra-ubuntu:~#
root@tegra-ubuntu:~#

I disabled the camera-related modules, but unfortunately, there was no improvement in the speed.

15

Hi wpceswpces

Sorry that didn’t help.

# If you haven't you could try multi-stream iperf3  "-P, --parallel # number of parallel client streams to run"

# Receiver (server)
iperf3 -s

# Sender: 16 parallel TCP streams (aggregate test)
iperf3 -c <server_ip> -P 16 -t 20

# And reverse direction.
iperf3 -c <server_ip> -P 16 -t 20 -R

I’ve not looked into mptcp but intend to. https://www.mptcp.dev/

# Following to confirm each mgbe link is at 25G individually.

ip -br link
ethtool <ifname> | egrep -i 'Speed|Duplex|Auto-neg'
ethtool -S <ifname>

# If using LACP

cat /proc/net/bonding/bond0
echo layer3+4 | sudo tee /sys/class/net/bond0/bonding/xmit_hash_policy

echo fast | sudo tee /sys/class/net/bond0/bonding/lacp_rate
16

Hello,whitesscott

On my end, the xmit_hash_policy is already set to layer3+4, but the lacp_rate isn’t set to fast—I checked and it’s currently slow. However, this seems to be the negotiation rate, which is the response speed for link aggregation detection, so it probably doesn’t affect the iperf throughput rate. It feels more like the polling interval for checking the status of each aggregated port. I’ll try changing it to fast later and compare the results. Thank you very much!

17

Hi wpceswpces, and set bond mtu equal to members, or pick a higher MTU that switch supports.

sudo ip link set dev bond0 mtu 1466
sudo ip link set dev mgbe0_0 mtu 1466
sudo ip link set dev mgbe1_0 mtu 1466
sudo ip link set dev mgbe2_0 mtu 1466
sudo ip link set dev mgbe3_0 mtu 1466
19

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.