Runtime addition of sources for streammux new fails

[eslam.ahmed]

Please provide complete information as applicable to your setup.

• Hardware Platform (GPU)
• DeepStream Version (6.1.1)
• TensorRT Version (8.4.3.1)
• NVIDIA GPU Driver Version (515.105.01)
• Issue Type( bug?)
• How to reproduce the issue ? (Runtime addition of streams while pipeline is in playing state)

We recently switched to using streammux new and we encountered issues when adding more sources at runtime without pausing the pipeline (by setting the state for the newly added elements to Playing). We used this link as reference.

We collected the following stacktrace:

0:50:16.131440654 75643 0x563ff43f84c0 ERROR         nvvideoconvert gstnvvideoconvert.c:3750:gst_nvvideoconvert_transform: buffer transform failed
[ERROR push 334] push failed [-5]
 2# 0x00007F56757FE3C0 in /lib/x86_64-linux-gnu/libpthread.so.0
 3# BatchPolicy::update_with_source(Batch*, unsigned int) in /usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_multistream.so
 4# BatchPolicy::form_batch(Batch*, unsigned int) in /usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_multistream.so
 5# NvStreamMux::push_loop(NvDsBatchBufferWrapper*, _NvDsBatchMeta*) in /usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_multistream.so
 6# 0x00007F565D2D835E in /usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_multistream.so
 7# 0x00007F5676AFA1E7 in /lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
 8# 0x00007F567699B374 in /lib/x86_64-linux-gnu/libglib-2.0.so.0
 9# 0x00007F567699AAD1 in /lib/x86_64-linux-gnu/libglib-2.0.so.0
10# start_thread at /build/glibc-sMfBJT/glibc-2.31/nptl/pthread_create.c:478
11# __clone in /lib/x86_64-linux-gnu/libc.so.6

streammux new config file is as follows:

[property]
algorithm-type=1
batch-size=1
adaptive-batching=1
max-fps-control=0
enable-source-rate-control=0
overall-max-fps-n=900
overall-max-fps-d=1
overall-min-fps-n=30
overall-min-fps-d=1
max-same-source-frames=1
max-latency=34482758
sync-inputs=1

Notes:

• This happens when adding 3 or more sources at once. However when adding 2 sources, we encounter the nvvideoconvert error of buffer transform failed [ERROR push 334] push failed [-5] randomly. At time it works, other times it doesn’t.
• All streams are of identical resolution/codec (same rtsp link actually)
• We didn’t experience this when using the old muxer.

Please advise, Thanks!

[yuweiw]

Is there a problem with using the original code you referenced? If not, could you attach the minimized code that can run with your problem?

[eslam.ahmed]

Hello, I have uploaded the modified code for usage with streammux new.
USE_NEW_NVSTREAMMUX=yes ./deepstream-test-rt-src-add-del rtsp://127.0.0.1:5003/test 0 nveglglessink 0
runtime_source_add_delete.zip (17.1 KB)

However I ran into a tracker problem, probably a wrong configuration just from switching to the streammux new.

~~ CLOG[src/modules/NvMultiObjectTracker/context.cpp, processFrame() @line 166]: !!![WARNING] params->numFrames = 2, m_Config.maxStreams = 1

!![Exception] [NvMOTContext::processFrame()] params->numFrames > m_Config.maxStreams !!
An exception occurred. [NvMOTContext::processFrame()] params->numFrames > m_Config.maxStreams !!
gstnvtracker: Low-level tracker lib returned error 1
0:01:04.710787713 33419 0x5612af9ce180 WARN                 nvinfer gstnvinfer.cpp:2300:gst_nvinfer_output_loop:<primary-nvinference-engine> error: Internal data stream error.
0:01:04.710801429 33419 0x5612af9ce180 WARN                 nvinfer gstnvinfer.cpp:2300:gst_nvinfer_output_loop:<primary-nvinference-engine> error: streaming stopped, reason error (-5)
ERROR from element nvtracker: Failed to submit input to tracker
Error details: gstnvtracker.cpp(610): gst_nv_tracker_submit_input_buffer (): /GstPipeline:dstest-pipeline/GstNvTracker:nvtracker
Returned, stopping playback
free(): invalid next size (fast)
Aborted (core dumped)

To move on, I removed the tracker from the pipeline and didn’t encounter the issue mentioned above. Will see if I can reproduce it.

[eslam.ahmed]

I think I managed to produce it. nvstreammux new config file remains unchanged.
runtime_source_add_delete.zip (17.1 KB)

Changes made include (but not limited to):

• usage of streammux new USE_NEW_NVSTREAMMUX=yes which required removing some of the old properties.
• removal of nvtracker - previously explained why.
• removal of nvmultistreamtiler - will be explained in detail
• usage of fakesink - can’t use nveglglessink while batch size > 1 since nvmultistreamtiler is removed.

With reference to the following image (both apps use nvstreammux new):

Investigation Results

• In our app, the issue mainly happen when adding one source upon the exiting source. note that the batch size after the muxer is > 1.
• In deepstream-test-rt-src-add-del, the batch size remains 1 after the muxer even when having multiple sources. I believe this is why I couldn’t immediately reproduce. This also can’t be right because I can clearly see more sources being added and nvmultistreamtiler dynamically expanding.
• Hence I decided to remove nvmultistreamtiler and replace nveglglessink with fakesink and the issue was reproduced. here’s the log for it

0:00:29.176232148 79517 0x55863eb93550 INFO                 nvinfer gstnvinfer_impl.cpp:328:notifyLoadModelStatus:<primary-nvinference-engine> [UID 1]: Load new model:dstest_pgie_config.txt sucessfully
decodebin child added source
Now playing: rtsp://127.0.0.1:5003/test
Running...
decodebin child added decodebin0
decodebin child added rtph265depay0
decodebin child added h265parse0
decodebin child added capsfilter0
decodebin child added nvv4l2decoder0
decodebin new pad video/x-raw
Decodebin linked to pipeline
[Error while parsing streammux config file: Key file does not have key ?batch-size? in group ?property?]
max_fps_dur 1.11111e+06 min_fps_dur 3.33333e+07
Calling Start 3 
creating uridecodebin for [rtsp://127.0.0.1:5003/test]
decodebin child added source
STATE CHANGE NO PREROLL

decodebin child added decodebin1
decodebin child added rtph265depay1
decodebin child added h265parse1
decodebin child added capsfilter1
decodebin child added nvv4l2decoder1
decodebin new pad video/x-raw
Decodebin linked to pipeline
0:00:31.917685609 79517 0x55863e42e640 ERROR         nvvideoconvert gstnvvideoconvert.c:3750:gst_nvvideoconvert_transform: buffer transform failed
0:00:31.951175678 79517 0x55863e42e6a0 WARN                 nvinfer gstnvinfer.cpp:2300:gst_nvinfer_output_loop:<secondary-nvinference-engine3> error: Internal data stream error.
0:00:31.951188983 79517 0x55863e42e6a0 WARN                 nvinfer gstnvinfer.cpp:2300:gst_nvinfer_output_loop:<secondary-nvinference-engine3> error: streaming stopped, reason error (-5)
ERROR from element secondary-nvinference-engine3: Internal data stream error.
Error details: gstnvinfer.cpp(2300): gst_nvinfer_output_loop (): /GstPipeline:dstest-pipeline/GstNvInfer:secondary-nvinference-engine3:
streaming stopped, reason error (-5)
Returned, stopping playback
free(): invalid next size (fast)
Aborted (core dumped)

• Bonus Round: The issue immediately disappears for both apps with turning off adaptive-batching and setting a proper batch-size.

Hope that this proves useful and assists in finding a solution.
Best Regards.

[yuweiw]

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks

Yes. We have relevant instructions in our guide about Dynamic addition/removal of input stream. This is supported by adaptive-batching.