Missing packets when receving Jumbo Multicast packets on MGBE

Robotics & Edge Computing Jetson & Embedded Systems Jetson Thor
1

Hi,

I’m trying to send ~8000 byte multicast UDP packets that are not fragmented to one of the mgbe’s on the Thor and for some strange reason I do not receive any data.

Running tcpdump yields no received packets. The MTU is set to 8966 bytes on the interface. What I find strange is if I reduce the multicast packet size to ~1940 bytes, I can see packets received in tcpdump, and on the NIC’s counters. Anything higher than that, the packets stop.

If I instead send unicast data of ~8000 bytes, the NIC is able to receive the packets without issue.

The Thor is capable of sending ~8000 byte multicast packets; the issue I’m observing is just with receive.

Is this a known issue ?

A few more bits of information:

  • The data being sent as been validated already. Other machines can receive the same data without issue. I am also in possession of an AGX Orin and it works as expected.
  • I’m using the Thor Dev kit using the QSFP28 running at 10Gbe.
  • Currently running R38, stock kernel 6.8.12-tegra.
  • In my experiments I do see some of the mmc_rx_* counters contain non-zero values
  • mmc_rx_fifo_overflow has been non-zero as well.
3

please share your commands for how you observe the problem.

4

Sure. In my scenario I have a piece of test equipment that is generating multicast traffic directly to the Thor, but I’ve been able to simplify the scenario. In simplifying things to answer your question I have determined that sending multicast packets of 1943 bytes is the threshold where it is received or not on the Thor.

All you need is another computer directly connected to the Thor:

 [PC]——10GBE—–>[THOR Dev Kit]

With this setup I will run 3 tests:

  • UDP Multicast traffic with packet sizes < 1943 bytes
  • UDP Multicast traffic with packet sizes >= 1943 bytes
  • UDP Unicast traffic with packet size of 8000 bytes.

Using tcpdump on the Thor, we will be able to validate if the packets were received or not.

On the PC I have a unicast address of 192.168.3.1. On the Thor in mgbe0_0 I have a unicast address of 192.168.3.2 . The Thor MTU on mgbe0_0 is 8966.

root@jetson-thor:/home/mbush# ifconfig mgbe0_0
mgbe0_0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 8966
        inet 192.168.3.2  netmask 255.255.255.0  broadcast 192.168.3.255
        ether 3c:6d:66:e3:fe:03  txqueuelen 1000  (Ethernet)
        RX packets 27  bytes 21300 (21.3 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 52  bytes 15398 (15.3 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

To generate packets of various sizes I will use ncat.

head -c <size> /dev/zero | ncat -vu <dest addr> <dest port> -s <sending interface addr>

Test 1 - UDP Multicast < 1943 bytes:

PC Command:

head -c 1900 /dev/zero | ncat -vu 239.192.1.1 20001 -s 192.168.3.1

tcpdump output:

root@jetson-thor:/home/mbush# tcpdump -i mgbe0_0
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on mgbe0_0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
08:57:11.721364 IP jetson-thor.53200 > 239.192.1.1.20001: UDP, length 1900

Here we see the Thor recieved the packet.

Test 2 - UDP Multicast >= 1943 bytes:

PC Command:

head -c 2000 /dev/zero | ncat -vu 239.192.1.1 20001 -s 192.168.3.1

tcpdump output:

root@jetson-thor:/home/mbush# tcpdump -i mgbe0_0
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on mgbe0_0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
 
0 packets captured
0 packets received by filter
0 packets dropped by kernel

Even though the packet was sent, it was not captured by tcpdump.

Test 3- UDP Unicast traffic with packet size of 8000 bytes.

PC Command:

head -c 8000 /dev/zero | ncat -vu 192.168.3.2 20001 -s 192.168.3.1

tcpdump output:

root@jetson-thor:/home/mbush# tcpdump -i mgbe0_0
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on mgbe0_0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
09:04:51.863214 IP 192.168.3.1.40518 > jetson-thor.20001: UDP, length 8000
09:04:51.863350 IP jetson-thor > 192.168.3.1: ICMP jetson-thor udp port 20001 unreachable, length 556

2 packets captured
2 packets received by filter
0 packets dropped by kernel

Here we can see that the Thor is capable of receiving UDP packets > 1943 bytes when the destination address is a unicast address. So I can say the Thor is capable of receiving jumbo frames, but for some reason it wont if it’s a multicast packet.

6

@WayneWWW Any update on this?

8

While I wait from someone from Nvidia to respond, I’ll mention that if I repeat the above experiments using the 5Gbe RJ-45 interface this does not exhibit the same issues as the mgbe interface. Using the 5GBe interface (I was connected at 1G) I can receive 8000 byte multicast packets.

Taking a quick glance to compare the differences between the nvethernet driver on the Orin and Thor, it looks like there were changes to support Camera Over Ethernet. I tried disabling COE on the Thor nvethernet driver but still, my large multicast packets are lost.

Also tried disabling macsec. Made no difference.

Also, one thing to note is when receiving large multicast packets on the mgbe interfaces, there are no interrupts received on the mgbe interface IRQs.

If anyone reading this knows the another way to get support from Nvidia could you please let me know. I am just assuming the only way to get support if you aren’t a deep pocketed vendor is via the forums. Heck, I’m not even sure I’m in the right sub-forum or not.

I believe this to be a serious networking issue, and would have expected to have some form of interest/acknowledgement/interaction at the very least.

9

we are reproducing this issue internally.

10

Thanks for the update. I appreciate that.