Problems with recieving multicast on startup.

Autonomous Machines Jetson & Embedded Systems Jetson TX2
#1

I’m working on an application to capture and display 4 multicast streams. The video
processing is done with a gstreamer pipeline, using udpsrc to capture multicast.

The problem is that the first time the application starts, the gstreamer pipeline does
not receive any of the multicast datagram packets. If I stop and restart the application,
the data is received and the video is displayed.

I am waiting until the ethernet interface is up, I make sure it will ping.

I know the packets are on the ethernet port, I am using a managed switch with
port mirroring (more below) and I see all four packet streams. I also see the
expected igmp join messages.

netstat -gn shows the group memberships.

Interestingly “ethtool -S eth0 | egrep ‘rx_multi|rx_broad’” shows some broadcast
traffic, but not the volume I expect (see below). Note that this is when I can
see all the streams on the port from the monitor port. The Ethernet packets
are there.

# ethtool -S eth0 | egrep 'rx_multi|rx_broad'
     mmc_rx_broadcastframe_g: 8380
     mmc_rx_multicastframe_g: 6
# ethtool -S eth0 | egrep 'rx_multi|rx_broad'
     mmc_rx_broadcastframe_g: 8405
     mmc_rx_multicastframe_g: 6

The most confusing part in debugging is that if I run tcpdump, it shows
all the packet streams and all the streams start almost instantly. That’s
why I set up the managed switch with port mirroring.

Interstingly, if I don’t run tcpdump in promiscious mode, the video streams
don’t start.

Even more interestingly, after running tcpdump in promiscous mode, ethtool
shows that the multicast frames are being received at the expected rate.

# tcpdump -n -i eth0
[  596.822526] Unsupported IOCTL call
[  596.846401] Unsupported IOCTL call
[  596.865935] device eth0 entered promiscuous mode
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
15:50:26.682731 IP 10.3.15.12.36250 > 239.3.15.1.4001: UDP, length 1316
15:50:26.682911 IP 10.3.15.12.36250 > 239.3.15.1.4001: UDP, length 1316
15:50:26.682961 IP 10.3.15.12.36250 > 239.3.15.1.4001: UDP, length 1316
...
# ethtool -S eth0 | egrep 'rx_multi|rx_broad'
     mmc_rx_broadcastframe_g: 10456
     mmc_rx_multicastframe_g: 55855
# ethtool -S eth0 | egrep 'rx_multi|rx_broad'
     mmc_rx_broadcastframe_g: 10490
     mmc_rx_multicastframe_g: 57772

Note many more multicast frames.

It seems as if somehow running tcpdump kicks the system into properly categorizing
the streams as multicast.

This situation can also be cleared by stopping ALL the multicast rx streams and
restarting them.

No clue where “Unsupported IOCTL call” comes from.

I can probably fix the problem by detecting video loss and restarting the gstreamer
pipelines, but this will slow startup, which is not desirable in some of the
expected use cases. I could also probably fix the problem by periodically running
tcpdump, but that’s a path rather than a fix. I’d rather have a solid understanding
of the problem and the best fix.

Note currently all the streams are on the same multicast group. The problem is also
observed when they are on different multicast groups.

Ideas? Suggestions?

Thanks, Cary

#2

When you ping first are you using a named address, or a dotted-decimal format address? When your application accesses this address, does it use a named address, or does it use a dotted-decimal address? Named addresses have an additional requirement of the DNS server being set…dotted-decimal addresses have no such requirement. If any of the addresses are named, perhaps try with dotted-decimal…this may get around any slow startup of DNS.

FYI, a missing/invalid IOCTL implies the driver does not support that feature.

In the case of a managed switch I think not only about ports being bridged, I also think about firewall features. In the case of seeing some traffic, yet not all you expected, make sure filtering is not taking place on the switch. Keep in mind that the switch may have some activity the first time it see’s the Jetson’s MAC address which changes the configuration for the packets which follow initial discovery.

You might attempt to use something like “netcat” just to see bytes flowing, and only then start the Jetson…if it is something in the link layer setting up, then having netcat trigger that port/protocol combination might cause the Jetson to stream correctly on first startup…this would give you a better idea of where the startup issue is.

Also make sure your Jetson and the managed switch are not set to use some default odd value requiring renegotiating something…e.g., jumbo frames as one example…I don’t know off hand what the Jetson supports, nor what your switch is trying to use.

#3

Thanks,

It’s all numeric ip addresses.
I’ve tried several stand-alone applications that have the same problem.
It’s just small 1.3k transport stream video packets from a gstreamer encoder.

I did finally find one page that suggested running (once at startup)

ip link set dev eth0 allmulticast on

This seemed to solve the problem.

I’m still wondering if there’s a better solution, since with this all of
the multicast groups are processed by the network stack, rather than just
the few that are being received, the rest supposedly dropped by the NIC
hardware.

If nothing else shows up I’ll mark this as the solution in a day or two.

Thanks again,
Cary

#4

I marked the ip link config step as the answer, because it works, but I’m surprised nobody else has run into this.

#5

I had the same problem.
In order to receive data from a multicast socket I need to start my application, kill it and start it again.
The socket initialization doesn’t produce any error on the first run.

#6

Experiencing the same issue. The command referenced above works, but it would be nice to have a more permanent solution. Can you provide a link to where you found the command?

#7

Not sure exactly, it was a while ago.

Possibly here: https://serverfault.com/questions/321245/what-is-the-difference-between-multicast-and-allmulti-ifconfig-ifconfig-flags

Some other references:

Multicast uses special mac destination addresses, so if possible the hardware
can be configured to only receive the ethernet packets for the specified
multicast group (well, all multicast groups that end the same way). But that
depends on the hardware having multiple match registers.

Interestingly, I’m struggling with this again with a totally different project.