Dynamic Add/Remove Stream Issues with DeepStream nvmultiurisrcbin and nvstreammux

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Please provide complete information as applicable to your setup.

• Hardware Platform (GPU)
• DeepStream Version 7.1
• JetPack Version (valid for Jetson only)
• TensorRT Version
• NVIDIA GPU Driver Version (valid for GPU only)
• Issue Type( questions)
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)

When using the DeepStream framework with nvstreammux New enabled, dynamically adding a new stream through nvmultiurisrcbin causes a brief glitch/blur in the currently playing streams, but they recover to normal.

However, dynamically removing a stream through nvmultiurisrcbin causes glitches/blur in the currently playing streams that do not recover.

• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

3

I tested the following command line and found no unrecoverable add/remove issues. You can check if adding/removing cameras causes network fluctuations, which results in packet loss that cannot be recovered.

USE_NEW_NVSTREAMMUX=yes  gst-launch-1.0 nvmultiurisrcbin \
port=9000 ip-address=localhost \
batched-push-timeout=33333 max-batch-size=8 \
drop-pipeline-eos=0 live-source=1 \
uri-list=rtsp://xxx:8554/mystream3,rtsp://xx:8554/mystream2,rtsp://xxxx:8554/mystream1,rtsp://xx:8554/mystream0 width=720 height=480 ! nvmultistreamtiler width=1920 height=1080 square-seq-grid=1 ! nv3dsink


curl -XPOST 'http://localhost:9000/api/v1/stream/add' -d '{
  "key": "sensor",
  "value": {
     "camera_id": "uniqueSensorID1",
     "camera_name": "front_door",
     "camera_url": "rtsp://xxxx:8554/mystream6",
     "change": "camera_add",
     "metadata": {
         "resolution": "1920 x1080",
         "codec": "h264",
         "framerate": 30
     }
 },
 "headers": {
     "source": "vst",
     "created_at": "2021-06-01T14:34:13.417Z"
 }
}'


curl -XPOST 'http://localhost:9000/api/v1/stream/remove' -d '{
    "key": "sensor",
    "value": {
        "camera_id": "uniqueSensorID1",
        "camera_name": "",
        "camera_url": "rtsp://xxx:8554/mystream6",
        "change": "camera_remove",
        "metadata": {
            "resolution": "1920 x1080",
            "codec": "h264",
            "framerate": 30
        }
    },
    "headers": {
        "source": "vst",
        "created_at": "2021-06-01T14:34:13.417Z"
    }
}'
4

Pipeline diagram

nvmultiurisrcbin → capsfilter → queue → nvstreamdemux →  
    ├─ nvvideoconvert0 → nvdsosd0 → nvvideoconvert_0 → capsfilter0 → nvv4l2h264enc0 → h264parse0 → flvmux0 → rtmpsink0  
    └─ nvvideoconvert1 → nvdsosd1 → nvvideoconvert_1 → capsfilter1 → nvv4l2h264enc1 → h264parse1 → flvmux1 → rtmpsink1

The inference elements have been removed, keeping only the elements shown above for testing.

5

How do you observe glitch/blur using rtmpsink?

This appears to be a issue in nvmultiurisrcbin. I can reproduce it using the following pipeline.

USE_NEW_NVSTREAMMUX=yes 
gst-launch-1.0 nvmultiurisrcbin \
port=9000 ip-address=localhost \
batched-push-timeout=33333 max-batch-size=8 \
drop-pipeline-eos=0 live-source=1 \
uri-list=rtsp://xxxx:8554/mystream0,rtsp://xxx:8554/mystream1 width=640 height=480 ! nvstreamdemux name=demux \
demux.src_0 ! queue ! nvvideoconvert ! nv3dsink sync=false \
demux.src_1 ! queue ! nvvideoconvert ! nv3dsink sync=false

Please use the legacy nvstreamux; I’ve tested it and it works fine.

Alternatively, if you’re using rtspsrc with the new nvstreammux, this may be a synchronization issue. We will discuss this internally.

6

We’re using the new nvstreammux because the pipeline also needs to handle audio.

7

How do you handle audio with nvmultiurisrcbin ? I think nvmultiurisrcbin can’t form audio into batch.

What does your complete pipeline look like? Do you need audio inference?

8

Our official documentation does mention that audio batching is supported, and in our tests it indeed works correctly. We do need audio inference — the inference results are passed as input tensors to the video inference. At the same time, the demuxed audio is mixed with the demuxed video in flvmux for output.

9
10

So if I understand you correctly, you want to build the following pipeline right? If my understanding is incorrect, please describe your complete pipeline (including the audio pipeline)

USE_NEW_NVSTREAMMUX=yes nvmultiurisrcbin mode=0(video only) uri-list=xxxx  --> infer --> demux --> demux.src_x   -->   |
                                                                                                                       |  --> flvmux --> rtmpsink
USE_NEW_NVSTREAMMUX=yes nvmultiurisrcbin mode=1(audio only) uri-list=xxxx --> audioinfer --> demux --> demux.src_x --> |

New nvstreammux requires the config-file-path property to be specified, but since nvmultiurisrcbin cannot set this value to new nvstreammux. This results in some parameters of new nvstreammux being unable to be adjusted, which may be the problem.

As a workaround, you can consider adding this property to nvmultiurisrcbin, which is open source.

11

Your understanding is basically correct. But could you clarify — which specific properties set via config-file-path for the new nvstreammux would actually resolve the issue mentioned in the post?

12

This depends on the max-fps/min-fps and other parameters of the source you add/remove. Please refer to this FAQ.

We have verified that deepstream-test-rt-src-add-del with new nvstreammux works properly for add/remove sources scenario.

So, I believe that this problem is caused by nvmultiurisrcbin not correctly setting some parameters of new nvstreammux.

13

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks.