What I see is somewhere some frames are dropped, most probably because nvinfer can’t handle 90 FPS (3 streams x 30 FPS). The postdecode_queue_{X} never grow.
If I change the decodebin to rtph264depay ! h264parse ! avdec_h264 ! nvvideoconvert ! video/x-raw(memory:NVMM), the postdecode_queue_{X} start to grow (and I expect this behavior). The problem is, that then the hw-accelerated h264 decoder is not used. If I use the nvv4l2decoder instead of avdec_h264, the pipeline does not launch. I Simplify the pipeline to gst-launch-1.0 rtspsrc location=rtsp://192.168.100.109:554/cam0_0 ! rtph264depay ! h264parse ! nvv4l2decoder ! fakesink and get (gst-launch-1.0:25118): GStreamer-CRITICAL **: 07:43:12.443: gst_mini_object_unref: assertion 'mini_object != NULL' failed
In case with decodebin I also added a bus message listener and I expect QOS messages to appear, but there is no messages of that type (and seems that there is no messages related to frame dropping)
So, I have 2 questions:
how to disable frame dropping when decodebin is used
how to correctly run the pipeline with nvv4l2decoder instead of avdec_h264
Setting pipeline to PAUSED ...
Opening in BLOCKING MODE
Opening in BLOCKING MODE
Pipeline is live and does not need PREROLL ...
Progress: (open) Opening Stream
Progress: (connect) Connecting to rtsp://192.168.100.109:554/cam0_0
Progress: (open) Retrieving server options
Progress: (open) Retrieving media info
Progress: (request) SETUP stream 0
Progress: (request) SETUP stream 1
Progress: (open) Opened Stream
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Progress: (request) Sending PLAY request
Progress: (request) Sending PLAY request
Progress: (request) Sent PLAY request
NvMMLiteOpen : Block : BlockType = 261
NVMEDIA: Reading vendor.tegra.display-size : status: 6
NvMMLiteBlockCreate : Block : BlockType = 261
(gst-launch-1.0:10935): GStreamer-CRITICAL **: 04:05:28.071: gst_mini_object_unref: assertion 'mini_object != NULL' failed
NvMMLiteOpen : Block : BlockType = 261
NVMEDIA: Reading vendor.tegra.display-size : status: 6
NvMMLiteBlockCreate : Block : BlockType = 261
But I have switched the camera to h265 mode ant it works out of the box rtspsrc location=rtsp://192.168.100.109:554/cam0_0 ! rtph265depay ! h265parse ! nvv4l2decoder ! fakesink sync=1
What do you mean by slowing down? If a stream produces 30 FPS, and the performance after nvifer is 20 FPS, where is the rest? Can gstreamer slow down the camera stream (like make the camera produce 20 FPS instead of 30)?
I set up 3 streams and add FPS count before and after the nvstreammux using the src buffer callbacks or the gst-perf element. I expect to have 30 FPS (total 90) on each input stream (before inference) and some slow down after the inference (in my case, total about 70). But what I see is that FPS is the same on both postdecode_queue_0 and the queue after demux and it is about 70. That’s why I think there is some kind of drop somewhere, but I can’t understand where exactly.
Also, if I add a leaky queue before the nvstreammux, I will have total FPS 90 (as produces by the camera)
You can try removing the nvinfer plugin and test the fps.
It is possible that the buffer in the rtspsrc is full and it drops packet directly. For nv-plugin, if you do not configure related parameters, it will not actively drop frames.
In this case you can only improve performance by setting the interval parameter for nvinfer. This paramter specifies the number of consecutive batches to be skipped for inference.
That’s what I’m actually doing. I dynamically increase the interval when any of the the postdecode queues starts to grow constantly, and reset the interval when queues are almost empty. This way I can get average 30 FPS (without losing frames but skipping the inference sometimes).
But I still have the questions
How can I programmaticaly determine that rtspsrc or whatever is dropping frames? Like connect to some signal or listen to some event on event bus? I have found no way unfortunately. I tried to listen to the bus messages and looked on rtspsrc documentation. I think it’s something more gstreamer-related
How can I run something like rtspsrc ! rtph264depay ! h264parse ! nvv4l2decoder on deepstream 5.1 jetpack 4.5.1 on Nano (without decodebin)? It leads to crash now, while using decodebin is working fine
Unfortunately, I have found nothing related to frame drop. I even looked throgh the code and found some log entries that should be shown in case of frame drop, but they are not. Maybe the deepstream plugin version is different from the code version I looked into. Anyway, I will ask at gstreamer forum if I’m still interested in finding this out. Maybe have to dig here
Enabling drop-on-latency=1 for rtspsrc makes the element output messages like Queue full, dropping old packet to the log. But the behaviour is still the same (or I think it’s the same) - with or without drop-on-latency=1, some frames seems to disappear implicitely
So, IMO, the solution to control the framerate is to put a queue before streammux and to monitor it’s size, and if it grows - then change the nvinfer’s interval property.
I failed to find out where exactly the frames are lost/dropped (maybe I’m mistaken, but I beleive I’m not =). To find it out I think it’s necessary to spend more time on gstreamer elements source code reading (and probably debugging).
I’m going to investigate this issue later and open another thread if necessary
Due to the performance of your board, the fps can not be increased after the nvinfer plugin. If you just want to stabilize the frame rate, I suggest you consider using the videorate plugin. But this does not improve the performance, it just stabilizes the frame rate.
There may be no frame drop, just accumulated the latency of the rtsp stream.