Frames skip seriously on retrieving frames through nvvideosink's EGL stream in mailbox mode

I use 8 IP cameras to do panoramic stitching, each camera with a pipeline “rtspsrc ! decodebin ! nvvidconv ! nvvideosin”, and retrieve each camera’s video frames through nvvideosink’s EGL stream.

When the application starts, some cameras may going to play earlier than the others. If I use FIFO mode to retrieve each camera’s frame, they are not synchronize with each other. So I use mailbox mode just to let the earlier old frames dropped.

At the first begins, it works fine, and the panoramic video plays smoothly. But after a period of time, the EGL stream skip frames seriously, the panoramic video begin Caton (see attachment), and the nvvideosink continuously report warning messages:

TVMR: FrameRate = 25.000000 
TVMR: FrameRate = 25.000000 
TVMR: FrameRate = 25.000000 
TVMR: FrameRate = 25.000000 
TVMR: FrameRate = 25.000000 
<../PanoramicStitching/gstreamerutils.cpp, 42>: Warning from /GstPipeline:IPCVideo6/GstNvVideoSink:nvvideosink70: A lot of buffers are being dropped.
<../PanoramicStitching/gstreamerutils.cpp, 44>: debug info:
gstbasesink.c(2854): gst_base_sink_is_too_late (): /GstPipeline:IPCVideo6/GstNvVideoSink:nvvideosink70:
There may be a timestamping problem, or this computer is too slow.
<../PanoramicStitching/gstreamerutils.cpp, 42>: Warning from /GstPipeline:IPCVideo4/GstNvVideoSink:nvvideosink68: A lot of buffers are being dropped.
<../PanoramicStitching/gstreamerutils.cpp, 44>: debug info:
gstbasesink.c(2854): gst_base_sink_is_too_late (): /GstPipeline:IPCVideo4/GstNvVideoSink:nvvideosink68:
There may be a timestamping problem, or this computer is too slow.
<../PanoramicStitching/gstreamerutils.cpp, 42>: Warning from /GstPipeline:IPCVideo6/GstNvVideoSink:nvvideosink70: A lot of buffers are being dropped.
<../PanoramicStitching/gstreamerutils.cpp, 44>: debug info:
gstbasesink.c(2854): gst_base_sink_is_too_late (): /GstPipeline:IPCVideo6/GstNvVideoSink:nvvideosink70:
There may be a timestamping problem, or this computer is too slow.
<../PanoramicStitching/gstreamerutils.cpp, 42>: Warning from /GstPipeline:IPCVideo4/GstNvVideoSink:nvvideosink68: A lot of buffers are being dropped.
<../PanoramicStitching/gstreamerutils.cpp, 44>: debug info:
gstbasesink.c(2854): gst_base_sink_is_too_late (): /GstPipeline:IPCVideo4/GstNvVideoSink:nvvideosink68:
There may be a timestamping problem, or this computer is too slow.
TVMR: FrameRate = 25.000000 
TVMR: FrameRate = 25.000000 
TVMR: FrameRate = 25.000000 
<../PanoramicStitching/gstreamerutils.cpp, 42>: Warning from /GstPipeline:IPCVideo6/GstNvVideoSink:nvvideosink70: A lot of buffers are being dropped.
<../PanoramicStitching/gstreamerutils.cpp, 44>: debug info:
gstbasesink.c(2854): gst_base_sink_is_too_late (): /GstPipeline:IPCVideo6/GstNvVideoSink:nvvideosink70:
There may be a timestamping problem, or this computer is too slow.
<../PanoramicStitching/gstreamerutils.cpp, 42>: Warning from /GstPipeline:IPCVideo4/GstNvVideoSink:nvvideosink68: A lot of buffers are being dropped.
<../PanoramicStitching/gstreamerutils.cpp, 44>: debug info:
gstbasesink.c(2854): gst_base_sink_is_too_late (): /GstPipeline:IPCVideo4/GstNvVideoSink:nvvideosink68:
There may be a timestamping problem, or this computer is too slow.
<../PanoramicStitching/gstreamerutils.cpp, 42>: Warning from /GstPipeline:IPCVideo6/GstNvVideoSink:nvvideosink70: A lot of buffers are being dropped.
<../PanoramicStitching/gstreamerutils.cpp, 44>: debug info:
gstbasesink.c(2854): gst_base_sink_is_too_late (): /GstPipeline:IPCVideo6/GstNvVideoSink:nvvideosink70:
There may be a timestamping problem, or this computer is too slow.
<../PanoramicStitching/gstreamerutils.cpp, 42>: Warning from /GstPipeline:IPCVideo4/GstNvVideoSink:nvvideosink68: A lot of buffers are being dropped.
<../PanoramicStitching/gstreamerutils.cpp, 44>: debug info:
gstbasesink.c(2854): gst_base_sink_is_too_late (): /GstPipeline:IPCVideo4/GstNvVideoSink:nvvideosink68:
There may be a timestamping problem, or this computer is too slow.
<../PanoramicStitching/gstreamerutils.cpp, 42>: Warning from /GstPipeline:IPCVideo6/GstNvVideoSink:nvvideosink70: A lot of buffers are being dropped.
<../PanoramicStitching/gstreamerutils.cpp, 44>: debug info:
gstbasesink.c(2854): gst_base_sink_is_too_late (): /GstPipeline:IPCVideo6/GstNvVideoSink:nvvideosink70:
There may be a timestamping problem, or this computer is too slow.
TVMR: FrameRate = 25.000000 
TVMR: FrameRate = 25.000000 
TVMR: FrameRate = 25.000000 
<../PanoramicStitching/gstreamerutils.cpp, 42>: Warning from /GstPipeline:IPCVideo4/GstNvVideoSink:nvvideosink68: A lot of buffers are being dropped.
<../PanoramicStitching/gstreamerutils.cpp, 44>: debug info:
gstbasesink.c(2854): gst_base_sink_is_too_late (): /GstPipeline:IPCVideo4/GstNvVideoSink:nvvideosink68:
There may be a timestamping problem, or this computer is too slow.
TVMR: FrameRate = 25.000000 
TVMR: FrameRate = 25.000000

and here is the tegrastats report, it’s not overloaded:

RAM 3548/7851MB (lfb 513x4MB) cpu [26%@2036,0%@2035,0%@2035,18%@2036,22%@2034,14%@2036] EMC 33%@1866 APE 150 NVDEC 1164 MSENC 166 GR3D 0%@1300
RAM 3548/7851MB (lfb 513x4MB) cpu [26%@2035,0%@2035,0%@2036,19%@2036,17%@2034,18%@2033] EMC 35%@1866 APE 150 NVDEC 1164 MSENC 166 GR3D 0%@1300
RAM 3548/7851MB (lfb 513x4MB) cpu [22%@2036,0%@2035,0%@2035,15%@2034,20%@2034,18%@2034] EMC 34%@1866 APE 150 NVDEC 1164 MSENC 166 GR3D 0%@1300
RAM 3548/7851MB (lfb 513x4MB) cpu [23%@2006,0%@2034,1%@2036,13%@2021,18%@2025,11%@2030] EMC 33%@1866 APE 150 NVDEC 1164 MSENC 166 GR3D 10%@1300
RAM 3548/7851MB (lfb 513x4MB) cpu [26%@2035,0%@2035,0%@2035,15%@2034,16%@2034,16%@2034] EMC 32%@1866 APE 150 NVDEC 1164 MSENC 166 GR3D 0%@1300
RAM 3548/7851MB (lfb 513x4MB) cpu [23%@2035,0%@2035,0%@2034,19%@2033,21%@2035,19%@2035] EMC 32%@1866 APE 150 NVDEC 1164 MSENC 166 GR3D 0%@1300
RAM 3548/7851MB (lfb 513x4MB) cpu [25%@2033,0%@2034,0%@2035,22%@2035,17%@2035,19%@2034] EMC 32%@1866 APE 150 NVDEC 1164 MSENC 166 GR3D 3%@1300
RAM 3574/7851MB (lfb 513x4MB) cpu [36%@2020,0%@2035,30%@2035,23%@2029,25%@2027,34%@2026] EMC 38%@1866 APE 150 NVDEC 1164 MSENC 166 GR3D 19%@1300
RAM 3574/7851MB (lfb 513x4MB) cpu [17%@2022,15%@2035,23%@2035,16%@2035,15%@2030,19%@2028] EMC 37%@1866 APE 150 NVDEC 1164 MSENC 166 GR3D 4%@1300
RAM 3574/7851MB (lfb 513x4MB) cpu [26%@2034,0%@2035,0%@2034,20%@2035,17%@2034,26%@2034] EMC 35%@1866 APE 150 NVDEC 1164 MSENC 166 GR3D 31%@1300
RAM 3575/7851MB (lfb 513x4MB) cpu [23%@2024,1%@2035,2%@2035,17%@2029,24%@2018,23%@2024] EMC 36%@1866 APE 150 NVDEC 1164 MSENC 166 GR3D 9%@1300
RAM 3575/7851MB (lfb 513x4MB) cpu [18%@2035,0%@2035,1%@2035,16%@2034,11%@2036,15%@2035] EMC 34%@1866 APE 150 NVDEC 1164 MSENC 166 GR3D 0%@1300
RAM 3575/7851MB (lfb 513x4MB) cpu [23%@2035,0%@2035,0%@2035,15%@2034,14%@2034,15%@2034] EMC 32%@1866 APE 150 NVDEC 1164 MSENC 166 GR3D 19%@1300
RAM 3575/7851MB (lfb 513x4MB) cpu [17%@2027,0%@2035,0%@2035,15%@2032,9%@2027,13%@2030] EMC 32%@1866 APE 150 NVDEC 1164 MSENC 166 GR3D 0%@1300
RAM 3575/7851MB (lfb 513x4MB) cpu [18%@2034,0%@2034,0%@2035,15%@2034,14%@2035,16%@2034] EMC 32%@1866 APE 150 NVDEC 1164 MSENC 166 GR3D 0%@1300
RAM 3575/7851MB (lfb 513x4MB) cpu [19%@2034,0%@2034,0%@2035,14%@2034,16%@2033,14%@2035] EMC 32%@1866 APE 150 NVDEC 1164 MSENC 166 GR3D 0%@1300
RAM 3575/7851MB (lfb 513x4MB) cpu [19%@2006,0%@2035,0%@2036,12%@2024,17%@2025,15%@2026] EMC 31%@1866 APE 150 NVDEC 1164 MSENC 166 GR3D 1%@1300

SkipFrames.avi (4.72 MB)

Hi SongjianSu,
Please share sample code and steps to reproduce the issue on r28.1.

Hi SongjianSu,

Have you found the cause and resolved the problem?
Or please share sample code and steps to reproduce the issue then can help to investigate where might be wrong.

Thanks

It may be the video decoder issue, I change the gstreamer pipeline to rtspsrc ! rtph264depay ! appsink, and then use the tegra_multimedia_api decoder and converter to decode frames and convert to RGBA format. The issue still remain exists.

We probably cannot support 8 IP cameras. It might be performance limitation.