Here is the content of mux_config.txt:
[property]
algorithm-type=1
batch-size=1
overall-max-fps-n=30
overall-max-fps-d=1
overall-min-fps-n=10
overall-min-fps-d=1
adaptive-batching=1
Note that live-source, batch-size, batched-push-timeout, width and height in [streammux] are N/A (irrelevant) since we are using adaptive-batching mode.
I’ll adapt runtime_source_and_delete code to demonstrate the issue. In the meanwhile, can you forward the issue to your developers? This is basically a show-stop issue. It is quite likely that it is just a simple mistake in sink_pad_delete in nvstreammux source code, which meant to decrease the batchSize, instead, accidentally increment the batchSize. As I mentioned, DS6.0.1 does not have this issue. Our application basically has no changes from DS6.0.1 to DS6.2.
I am able to reproduce the issue after modify the source code “runtime_source_add_delete”, you can follow the steps:
create 2 virtual rtsp sources camera with rtsp server with “rtsp://127.0.0.1:19000/stream” and “rtsp://127.0.0.1:19001/stream”
export USE_NEW_NVSTREAMMUX=yes
make & make install (which will install
to directory: /opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream_reference_apps/runtime_source_add_delete)
run ./deepstream-test-rt-src-add-del rtsp://127.0.0.1:19000/stream 1 nveglglessink 0 which will add the two sources and display on nvelgglesink
While two sources are playing, you hit key “d” and then “return” key, which will delete source_id 1.
You will see, the remaining source_id (0) playing will become so choppy with very low fps on nvelgglesink.
You can enable tiler_sink_pad_buffer_probe (by changing #if 0 to #if 1 at line 636 ) in the source code to observe nvbufSurface->batchSize keeps at 2 (after deletion) although I did not observe the batchSize increasing in this case. Here is the adapted source code. runtime_source_add_delete.zip (38.6 KB)
I hardcoded 3 rtsp source cameras in the source code deepstream_test_rt_src_add_del.c (22.9 KB)
in case you want to test 3-source. You will notice the same choppy with very low fps on nvelgglesink after deleting source_id 2 and source_id 1 while batchSize keeps printing the same value 3.
In case you don’t have rtsp cameras available for testing, you can try to create rtsp server with the following example:
a: docker run --rm -it --network=host aler9/rtsp-simple-server
b: cd /opt/nvidia/deepstream/deepstream/samples/streams
c: ffmpeg -re -stream_loop -1 -i sample_1080p_h264.mp4 -c:v copy -an -f rtsp -rtsp_transport tcp rtsp://127.0.0.1:8554/stream0
d: ffmpeg -re -stream_loop -1 -i sample_1080p_h264.mp4 -c:v copy -an -f rtsp -rtsp_transport tcp rtsp://127.0.0.1:8554/stream1
e: ffmpeg -re -stream_loop -1 -i sample_1080p_h264.mp4 -c:v copy -an -f rtsp -rtsp_transport tcp rtsp://127.0.0.1:8554/stream2
Make sure that you change rtsp sources to “rtsp://127.0.0.1:8554/stream0” “rtsp://127.0.0.1:8554/stream1” and “rtsp://127.0.0.1:8554/stream2” in deepstream_test_rt_src_add_del.c. And run command run ./deepstream-test-rt-src-add-del rtsp://127.0.0.1:8554/stream0 1 nveglglessink 0
Thanks for your primary analysis. I use 2 real rtsp sources with your code attached. When deleting one source, the fps of the other does indeed decrease significantly. We will reply promptly once we have a conclusion.
I don’t think so. Normally I set overall-max-fps-n=30, I got the same problem.
Even in you case, it does not make sense that when overall-max-fps-n=60, you only get throughput fps about 7 and after you delete one source, it only gets about fps 15.
Also, does not make sense to set overall-max-fps-n = 60 since most of source rtsp cameras fps is 25/30.
We have been tried many different settings based on deepstream doc with DS6.2 and very sure that something is wrong in nvstreammux.
Wondering if there is chance that you can make gst plugin nvstreammux to be open source. We can certainly know how fix the issue.
In my case I had very strange similar problem, that I finally “solved” giving overall-max-fps-n an arbitrary value like overall-min-fps-n * number_of_sources * 10… Very arbitrary, but this is because the doc doesn’t help, I have no idea of what a good value is for this parameter…
I agree that it would be nice to have nvstreammux open source…
That is true. We had tried overall-max-fps-n = number_of_sources * some_value and overall-min-fps-n = number_of_sources * some_other_value too. But it is not always working since number_of_sources may change dynamically and wildly.
From we have experienced in deepstream sdk based application, we found that nvstreammux/nvstreamdemux determines the fundamentals of gst pipeline stability. We do hope that NVIDIA can make it to be open source.
Note that we had to develop dsstreamdemux to replace nvstreamdemux due to its unexpected video scaling in the case of mixed video resolutions. We may have to develop nvstreammux if you can’t make it open source. But that’s a waste of time and effort.
OK. We’ll check the following 2 issues. If there are any results, we will reply in time.
1.How to set parameters to nvstreammux in the scenarios of dynamically adding and deleting sources.
2.Could the nvstreammux code be open source.
Why donn’t you send eos message when stop the source in your demo code? You should send an eos message when you dynamically delete a source. I try to send the eos message, it works fine.(The fps of my rtsp stream is 15.)
I updated the code to add back “gst_pad_send_event (sinkpad, gst_event_new_eos ());” and surprisingly the call “gst_pad_unlink” has to be removed before calling gst_element_release_request_pad. Note that “gst_pad_unlink” is being called in deepstream_common.c in DS6.2. This did improve and most of times, the fps maintains after deleting a source. But sometimes, the fps still hits near zero after deleting a source depending on timing (This can be more likely reproduced if you hit “d” quickly as soon as you are seeing all videos are displayed).
I have updated the deepstream_test_rt_src_add_del.c and command line options to make it easier to reproduce the issue:
have fixed the list of 3 rtsp sources uri in the code char * SRC_URI_LIST = {
“rtsp://127.0.0.1:19000/stream”,
“rtsp://127.0.0.1:19001/stream”,
“rtsp://127.0.0.1:19002/stream”,
“”
};
command line is “./deepstream-test-rt-src-add-del 1 nveglglessink 1” (I removed uri option from command line options)
Now you can add source back by hitting ‘a’ and return. (note that ‘d’ and return for deleting ) Here is updated code: runtime_source_add_delete.zip (38.7 KB)
The other issue is that with the exact same 3 video sources, the fps throughput can easily be 30 in DS6.0 (even with more than 20 sources on RTX 30380). But it can only reach 24 in DS6.2 with just 3 video sources.
I use your code and ffmpeg rtsp server to test the fps. Add perf log as follows: Enable Perf measurement. Because there is no monitor on my device, I chose filesink.
The printing frequency is 1 second. After adding or deleting source, the nvstreammux batch algorithm requires some time to converge. But after stabilization, the frame rate is around 30.
Log:
Let’s not to concern throughput for now. The major problem is still the near-zero fps issue if I repeat with “d” then “a” serveral times quickly. I got the following message:
WARNING from element nveglglessink: A lot of buffers are being dropped.
Warning: A lot of buffers are being dropped.
WARNING from element nveglglessink: A lot of buffers are being dropped.
Warning: A lot of buffers are being dropped.
WARNING from element nveglglessink: A lot of buffers are being dropped.
Warning: A lot of buffers are being dropped.
WARNING from element nveglglessink: A lot of buffers are being dropped.
Warning: A lot of buffers are being dropped.
WARNING from element nveglglessink: A lot of buffers are being dropped.
Warning: A lot of buffers are being dropped.
WARNING from element nveglglessink: A lot of buffers are being dropped.
I did not add demux in deepstream-test-rt-src-add-del for fps output, I will try to add it tomorrow.
Right now I just watch tiler displaying, it doesn’t seem to be recovering to normal fps sometimes. But I will test more.
I retried many times with nveglglessink. I confirmed that normal fps throughput can never recover to normal fps if you try “d”/“a” several times and video display will show near freezing on nveglglessink. I haven’t have chance to add demux yet. Note that I used 3 rtspsrcs with different video resolutions (1920x1080, 3840x2160 and 2592x1944)
Have you ask if it is possible to make the nvstreammux to be open source?
One other major issue (in both DS6.0 and DS6.2) is that if the all fps will be set to the lowest fps of rtspsrcs. For example, if one camera is 30 fps, second camera is 15 fps and third camera is 5 fps, then the throughput fps is only 5 fps in adaptive batching mode. This is very bad since no reliable detection is going to be obtained for any camera because fps is too low. Although you can use gst videorate plugin to make the all input fps to be 30 after decoder, but that simply wastes GPU computation powers. So at this point, I really hope that nvstreammux will be open source so that we have chance to customize it.
If we open source the code, it will be stated in our release notes. So please pay attention to our updates.
About the “d”/“a” operation, I use your code attached to operate serveral times, it can stabilize to 30 eventually.
Let’s put aside some hypothetical scenarios. Could you provide a specific description of your product and the problems encountered in actual use?
We have finally developed our customized gst plugin mux/demux sucessfully to address the issues. So we no longer need to use nvstreammux/nvstreamdemux anymore. The issues in nvstreammux/nvstreamdemu remain there.