Mmc_rx_fifo_overflow Meaning

rseward · May 16, 2023, 5:57pm

Currently running JetPack 4.4 - L4T R32.4.3 on a ConnectTech Jetson Xavier AGX. We have our own custom application accepting data on one of the ethernet ports via UDP. Occasionally we’ll notice that our software (which monitors the incoming data) reports missing packets. Looking at the the mmc_rx_fifo_overflow parameter from ethtool -S eth0 we see a direct correlation in this value increasing and the number of packets we see missing from our datastream.

My question, what exactly does this parameter (mmc_rx_fifo_overflow) represent and what could potentially cause it to increase?

Thanks!

WayneWWW · May 17, 2023, 3:19am

Hi,

What is the result of ifconfig when you see rx fifo overflow happened?

How is the CPU loading when this issue happens? The fifo overflow mostly means the kernel side speed is unable to catch the coming frames from NIC.

Rel-32.4.3 is a little bit old so maybe you can upgrade to rel-32.7.3 and see if this gets enhanced.

rseward · May 17, 2023, 8:59am

So ifconfig doesn’t show anything weird, no errors or overruns:

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.0.100  netmask 255.255.255.0  broadcast 192.168.0.255
        inet6 fe80::4ab0:2dff:fe18:5040  prefixlen 64  scopeid 0x20<link>
        ether 48:b0:2d:18:50:40  txqueuelen 1000  (Ethernet)
        RX packets 3350223287  bytes 1862710247149 (1.8 TB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 28800  bytes 6908942 (6.9 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 41

How exactly could I check the CPU loading? That’s not something I’m familiar with.

WayneWWW · May 17, 2023, 9:21am

Run sudo tegrastats and it will show.

rseward · May 17, 2023, 2:29pm

Here’s some of the output from tegrastats around the time of a network drop. I’ll try to put something together that lines up with timestamps if this isn’t useful though:

RAM 1634/31919MB (lfb 193x4MB) SWAP 0/15959MB (cached 0MB) CPU [57%@1267,53%@1267,2%@1267,0%@1267,0%@1267,4%@1267,0%@1267,1%@1267] EMC_FREQ 0% GR3D_FREQ 0% AO@30C GPU@30C Tdiode@34C PMIC@100C AUX@30.5C CPU@30C thermal@30.3C Tboard@31C GPU 0/0 CPU 849/884 SOC 2358/2380 CV 0/0 VDDRQ 377/377 SYS5V 4245/4244
RAM 1634/31919MB (lfb 193x4MB) SWAP 0/15959MB (cached 0MB) CPU [57%@1267,8%@1267,6%@1267,7%@1267,0%@1267,3%@1267,0%@1267,3%@1267] EMC_FREQ 0% GR3D_FREQ 0% AO@30C GPU@30.5C Tdiode@34C PMIC@100C AUX@30C CPU@30.5C thermal@30.3C Tboard@31C GPU 0/0 CPU 849/884 SOC 2358/2380 CV 0/0 VDDRQ 377/377 SYS5V 4245/4244
RAM 1635/31919MB (lfb 193x4MB) SWAP 0/15959MB (cached 0MB) CPU [58%@1036,18%@1036,0%@1036,1%@1036,5%@1036,6%@1036,3%@1036,0%@1036] EMC_FREQ 0% GR3D_FREQ 0% AO@30.5C GPU@30C Tdiode@34C PMIC@100C AUX@30C CPU@30C thermal@30.5C Tboard@31C GPU 0/0 CPU 849/884 SOC 2452/2380 CV 0/0 VDDRQ 377/377 SYS5V 4245/4244
RAM 1634/31919MB (lfb 193x4MB) SWAP 0/15959MB (cached 0MB) CPU [58%@1497,50%@1497,0%@1497,0%@1497,13%@1497,2%@1497,4%@1497,3%@1497] EMC_FREQ 0% GR3D_FREQ 0% AO@30C GPU@30.5C Tdiode@34C PMIC@100C AUX@30C CPU@30.5C thermal@30.2C Tboard@31C GPU 0/0 CPU 849/884 SOC 2358/2380 CV 0/0 VDDRQ 377/377 SYS5V 4245/4244
RAM 1635/31919MB (lfb 193x4MB) SWAP 0/15959MB (cached 0MB) CPU [58%@1267,39%@1267,2%@1267,4%@1267,0%@1267,4%@1267,2%@1267,1%@1267] EMC_FREQ 0% GR3D_FREQ 0% AO@30.5C GPU@30.5C Tdiode@34C PMIC@100C AUX@30C CPU@30.5C thermal@30.3C Tboard@31C GPU 0/0 CPU 849/884 SOC 2358/2380 CV 0/0 VDDRQ 377/377 SYS5V 4238/4244
RAM 1635/31919MB (lfb 193x4MB) SWAP 0/15959MB (cached 0MB) CPU [58%@1497,18%@1497,2%@1497,5%@1497,1%@1497,5%@1497,4%@1497,0%@1497] EMC_FREQ 0% GR3D_FREQ 0% AO@30.5C GPU@30.5C Tdiode@34C PMIC@100C AUX@30C CPU@30.5C thermal@30.2C Tboard@31C GPU 0/0 CPU 849/884 SOC 2358/2380 CV 0/0 VDDRQ 377/377 SYS5V 4245/4244
RAM 1633/31919MB (lfb 193x4MB) SWAP 0/15959MB (cached 0MB) CPU [57%@1497,59%@1497,3%@1497,1%@1497,0%@1497,3%@1497,1%@1497,2%@1497] EMC_FREQ 0% GR3D_FREQ 0% AO@30.5C GPU@30.5C Tdiode@34C PMIC@100C AUX@30C CPU@30.5C thermal@30.2C Tboard@31C GPU 0/0 CPU 849/884 SOC 2358/2380 CV 0/0 VDDRQ 377/377 SYS5V 4245/4244
RAM 1635/31919MB (lfb 193x4MB) SWAP 0/15959MB (cached 0MB) CPU [57%@1267,24%@1267,0%@1267,0%@1267,1%@1267,9%@1267,2%@1267,4%@1267] EMC_FREQ 0% GR3D_FREQ 0% AO@30.5C GPU@30.5C Tdiode@34C PMIC@100C AUX@30C CPU@30.5C thermal@30.5C Tboard@31C GPU 0/0 CPU 849/884 SOC 2452/2380 CV 0/0 VDDRQ 377/377 SYS5V 4245/4244
RAM 1635/31919MB (lfb 193x4MB) SWAP 0/15959MB (cached 0MB) CPU [56%@1651,17%@1651,0%@1651,0%@1651,1%@1651,4%@1651,2%@1651,4%@1651] EMC_FREQ 0% GR3D_FREQ 0% AO@30.5C GPU@30.5C Tdiode@34C PMIC@100C AUX@30C CPU@30.5C thermal@30.3C Tboard@31C GPU 0/0 CPU 849/883 SOC 2358/2380 CV 0/0 VDDRQ 377/377 SYS5V 4245/4244
RAM 1634/31919MB (lfb 193x4MB) SWAP 0/15959MB (cached 0MB) CPU [61%@1190,41%@1497,1%@1497,1%@1497,12%@1497,3%@1497,7%@1497,1%@1497] EMC_FREQ 0% GR3D_FREQ 0% AO@30.5C GPU@30.5C Tdiode@34C PMIC@100C AUX@30.5C CPU@30.5C thermal@30.5C Tboard@31C GPU 0/0 CPU 849/883 SOC 2358/2380 CV 0/0 VDDRQ 377/377 SYS5V 4238/4244
RAM 1634/31919MB (lfb 193x4MB) SWAP 0/15959MB (cached 0MB) CPU [58%@1190,14%@1190,1%@1190,2%@1190,1%@1190,8%@1190,0%@1190,2%@1190] EMC_FREQ 0% GR3D_FREQ 0% AO@30C GPU@30.5C Tdiode@34C PMIC@100C AUX@30.5C CPU@30.5C thermal@30.5C Tboard@31C GPU 0/0 CPU 849/883 SOC 2358/2380 CV 0/0 VDDRQ 377/377 SYS5V 4245/4244
RAM 1634/31919MB (lfb 193x4MB) SWAP 0/15959MB (cached 0MB) CPU [59%@1113,21%@1113,2%@1113,3%@1113,1%@1113,5%@1113,0%@1113,2%@1113] EMC_FREQ 0% GR3D_FREQ 0% AO@30C GPU@30C Tdiode@34C PMIC@100C AUX@30.5C CPU@30C thermal@30C Tboard@31C GPU 0/0 CPU 849/883 SOC 2358/2380 CV 0/0 VDDRQ 377/377 SYS5V 4245/4244
RAM 1634/31919MB (lfb 193x4MB) SWAP 0/15959MB (cached 0MB) CPU [58%@1190,60%@1190,1%@1190,0%@1190,0%@1190,2%@1190,3%@1190,1%@1190] EMC_FREQ 0% GR3D_FREQ 0% AO@30.5C GPU@30C Tdiode@34C PMIC@100C AUX@30.5C CPU@30C thermal@30C Tboard@31C GPU 0/0 CPU 849/883 SOC 2452/2380 CV 0/0 VDDRQ 377/377 SYS5V 4245/4244
RAM 1635/31919MB (lfb 193x4MB) SWAP 0/15959MB (cached 0MB) CPU [60%@1036,20%@1036,0%@1036,5%@1036,0%@1036,1%@1036,0%@1036,24%@1036] EMC_FREQ 0% GR3D_FREQ 0% AO@30C GPU@30C Tdiode@34C PMIC@100C AUX@30C CPU@30C thermal@30.3C Tboard@31C GPU 0/0 CPU 754/883 SOC 2358/2380 CV 0/0 VDDRQ 377/377 SYS5V 4238/4244
RAM 1635/31919MB (lfb 193x4MB) SWAP 0/15959MB (cached 0MB) CPU [60%@1728,30%@1728,4%@1728,2%@1728,1%@1728,3%@1728,1%@1728,22%@1728] EMC_FREQ 0% GR3D_FREQ 0% AO@30C GPU@30.5C Tdiode@34C PMIC@100C AUX@30C CPU@30.5C thermal@30.8C Tboard@31C GPU 0/0 CPU 1125/883 SOC 2437/2380 CV 0/0 VDDRQ 374/377 SYS5V 4245/4244
RAM 1636/31919MB (lfb 193x4MB) SWAP 0/15959MB (cached 0MB) CPU [57%@1651,19%@1651,0%@1651,4%@1651,0%@1651,9%@1651,0%@1651,0%@1651] EMC_FREQ 0% GR3D_FREQ 0% AO@30C GPU@30C Tdiode@34C PMIC@100C AUX@30C CPU@30C thermal@30.3C Tboard@31C GPU 0/0 CPU 941/884 SOC 2353/2380 CV 0/0 VDDRQ 376/377 SYS5V 4245/4244
RAM 1635/31919MB (lfb 193x4MB) SWAP 0/15959MB (cached 0MB) CPU [56%@1574,10%@1574,2%@1574,2%@1574,1%@1574,2%@1574,0%@1574,0%@1574] EMC_FREQ 0% GR3D_FREQ 0% AO@30C GPU@30.5C Tdiode@34.25C PMIC@100C AUX@30.5C CPU@30.5C thermal@30.5C Tboard@31C GPU 0/0 CPU 849/883 SOC 2358/2380 CV 0/0 VDDRQ 377/377 SYS5V 4245/4244
RAM 1635/31919MB (lfb 193x4MB) SWAP 0/15959MB (cached 0MB) CPU [57%@1267,56%@1267,1%@1267,1%@1267,1%@1267,3%@1267,2%@1267,3%@1267] EMC_FREQ 0% GR3D_FREQ 0% AO@30.5C GPU@30.5C Tdiode@34C PMIC@100C AUX@30.5C CPU@30.5C thermal@30.5C Tboard@31C GPU 0/0 CPU 849/883 SOC 2358/2380 CV 0/0 VDDRQ 377/377 SYS5V 4245/4244
RAM 1635/31919MB (lfb 193x4MB) SWAP 0/15959MB (cached 0MB) CPU [57%@1574,45%@1574,1%@1574,0%@1574,0%@1574,4%@1574,1%@1574,4%@1574] EMC_FREQ 0% GR3D_FREQ 0% AO@30C GPU@30.5C Tdiode@34C PMIC@100C AUX@30C CPU@30.5C thermal@30.3C Tboard@31C GPU 0/0 CPU 849/883 SOC 2358/2380 CV 0/0 VDDRQ 377/377 SYS5V 4245/4244
RAM 1636/31919MB (lfb 193x4MB) SWAP 0/15959MB (cached 0MB) CPU [56%@1267,36%@1267,1%@1267,2%@1267,0%@1267,4%@1267,1%@1267,0%@1267] EMC_FREQ 0% GR3D_FREQ 0% AO@30.5C GPU@30.5C Tdiode@34C PMIC@100C AUX@30.5C CPU@30.5C thermal@30.5C Tboard@31C GPU 0/0 CPU 849/883 SOC 2358/2380 CV 0/0 VDDRQ 377/377 SYS5V 4245/4244
RAM 1635/31919MB (lfb 193x4MB) SWAP 0/15959MB (cached 0MB) CPU [56%@1497,33%@1497,0%@1497,0%@1497,0%@1497,1%@1497,3%@1497,5%@1497] EMC_FREQ 0% GR3D_FREQ 0% AO@30C GPU@30.5C Tdiode@34C PMIC@100C AUX@30C CPU@30.5C thermal@30.5C Tboard@31C GPU 0/0 CPU 849/883 SOC 2358/2380 CV 0/0 VDDRQ 377/377 SYS5V 4245/4244

rseward · May 18, 2023, 4:06pm

So nothing weird from tegrastats whenever we see the dropped packets. One thing I also noticed is that rx_buf_unavailable_irq_n[0] seems to be increasing as well.

Anything else we can try outside of updating to a newer release? I’m planning on testing that, however, we have existing systems out in the wild and it would be nice if we could push a simple configuration fix.

WayneWWW · May 18, 2023, 4:09pm

Hi,

There is actually not much you can check. I also cannot remember every detail in each release.
Please be aware that we have about 20 release software branches out there. Each branch may have their own problems.

The actual point is we don’t debug on old release. If your issue is still there after you upgrade, then we can help you check.

Also, what is the point to check the overflow here? Is your throughput not ideal so you want to check something out?

rseward · May 18, 2023, 4:15pm

Fair point about debugging old releases.

Just for some context, we have an application that listens on a UDP port for traffic coming from a custom FPGA at a rate of about 25MB/s in 568 byte snapshots. Each snapsnot contains a timestamp which we check upon reception. If the delta between snapshots isn’t what we expect (200ms) we log this as a “missing snapshot”. We’ve noticed that the number of missing snapshots corresponds with the overflow here.

So really I’m mainly looking to see if there is anything that can be done in the way of configuration.

I’ll upgrade to 32.7.3 to see if the issue is still present.

Thanks for the help!

WayneWWW · May 18, 2023, 4:18pm

It would be great if you can do more about below

Try if rel-35.3.1 can reproduce this, but the gap between rel-32 and rel-35 are large. I know it would be difficult to upgrade. But rel-35 may have more chance to get highlight. rel-32 will gradually become LTS branch.
Please try to reproduce on devkit with similar scenario since we won’t have that FPGA you are using.

system · June 14, 2023, 1:58am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Unstable performance across multiple Jetson AGX Xavier devices DeepStream SDK fps , jetson , deepstream	5	818	October 30, 2023
Jetson AGX Xavier eqos ethernet driver causing kernel panic Jetson AGX Xavier ethernet	5	1006	August 8, 2022
Jetson TX2 mmc_rx_fifo_overflow errors Jetson TX2 ethernet , networking	4	1433	December 8, 2021
Jetson Xavier AGX nvgpu_timeout_expired Jetson AGX Xavier	30	1660	December 29, 2020
High CPU usage on Jetson TX2 with GigE fully loaded Jetson TX2 hw , kernel , performance	12	2506	October 18, 2021
overlayfs for Jetson Jetson Nano	39	12579	October 13, 2021
ResourceExhaustedError: Running TF-TRT integration on Jetson AGX Jetson AGX Xavier	10	1137	October 18, 2021
AGX Xavier easy to crash when ethernet network connected Jetson AGX Xavier ethernet	38	4705	October 18, 2021
CPU Throttling more on 4.9 than 4.4 Jetson TX2 kernel	15	1224	October 18, 2021
Using ONNX Runtime with TensorRT on Jetson Devices Jetson AGX Xavier tensorrt	5	1106	October 18, 2021

Mmc_rx_fifo_overflow Meaning

Related topics