使用两个NV devkit板子的mgbe2_0接口对测，iperf3速率未达到预期（预期9.4Gbits/sec以上），如何继续提高？

[1516775545]

英伟达团队您好，在使用nv devkit板卡对测mgbe2_0接口，iperf3速率如何提高？

系统版本：

[1516775545]

具体操作步骤：

使用文档上的优化（我们这里改成了mgbe2_0）

echo 1 > /sys/devices/platform/bus@0/a808d10000.ethernet/net/mgbe2_0/threaded

1. 
2. 使用iperf3测试，测试结果

[whitesscott]

This might help speeds.

cat /sys/devices/platform/bus@0/a808d10000.ethernet/net/mgbe2_0/mtu

sudo ip link set dev mgbe2_0 mtu 1500
ip -br link show mgbe2_0
ip link show dev mgbe2_0 | grep -i mtu
ping -M do -s 1472 <peer_ip>

# and or try:
sudo ip link set dev mgbe2_0 mtu 9000
ping -M do -s 8972 <peer_ip>


# Receiver (server)
iperf3 -s

# Sender: 16 parallel TCP streams (aggregate test) or use "-P 8"
iperf3 -c <server_ip> -P 16 -t 20

# Also try reverse direction
iperf3 -c <server_ip> -P 16 -t 20 -R

[1516775545]

@whitesscott

cat /sys/devices/platform/bus@0/a808d10000.ethernet/net/mgbe2_0/mtu

MTU value cannot be modified.

[whitesscott]

temporary MTU change until the next reconnect/reboot:

sudo ip link set dev mgbe2_0 down
sudo ip link set dev mgbe2_0 mtu 1500
sudo ip link set dev mgbe2_0 up

If mgbe2_0 is managed by NetworkManager:

sudo nmcli connection down mgbe2_0
sudo nmcli connection modify mgbe2_0 802-3-ethernet.mtu 1500
sudo nmcli connection up mgbe2_0

[1516775545]

@whitesscott

nmcli connection up mgbe2_0

Still unsuccessful.

[whitesscott]

Based on output of following commands, it appears my 8966 and probably your 1466 mtu is caused by nvethernet.ko reserving 34 bytes for macsec. The reduction appears unconditional once macsec resources are present/probed.

I have mgbe*_0 mtu = 9000 for ipv4 and ipv6 in /etc/netplan/90-NM-*.yaml

sudo dmesg | grep -i macsec
sudo ethtool -k mgbe2_0
ip -d link show mgbe2_0

[ 9.661934] nvethernet a808b10000.ethernet mgbe1_0: Macsec: Reduced MTU: 8966 Max: 9000                                                                                                      [ 9.666967] nvethernet a808a10000.ethernet mgbe0_0: Macsec: Reduced MTU: 8966 Max: 9000

[whitesscott]

nvidia-oot/drivers/net/ethernet/nvidia/nvethernet/macsec.h

/**
 * @brief MACSEC SECTAG + ICV + 2B 
          ethertype adds up to 34B
 */
#define MACSEC_TAG_ICV_LEN              34U

[1516775545]

I successfully modified the MTU via netplan, but the iperf transmit performance on one DK board is slightly low.

[1516775545]

The systems on the two DK boards are the same.