DeepStream 7.1 nvstreammux (new) sinkpad link error

Accelerated Computing Intelligent Video Analytics DeepStream SDK
, ,
1

• Hardware Platform (Jetson / GPU) Linux w/ dGPU (RTX3090)
• DeepStream Version 7.1
• TensorRT Version 10.3.0.26-1 + CUDA 12.5
• NVIDIA GPU Driver Version 565.77
• Issue Type question + bug

Hi,
I’m trying to request + link a sinkpad for nvstreammux (new) in DS 7.1 (linux with dGPU via Docker) during runtime.

I receive a GST_PAD_LINK_NOFORMAT error, indicating a formatting mismatch (I assume).
We start with 3 sources linked to muxer pads (they all have the same caps as the 4th source and are fully connected + handling buffers).
We request a new sinkpad for the 4th source (during runtime) but cannot seem to link it (re: above error).
This occurs when the muxer is in PLAYING state, but not when in READY state.

I’m not sure which element (src or muxer-sink) is responsible for this (if any), please advise?
Is it even possible to dynamically link to a newly-requested sink pad whilst nvstreammux is PLAYING?

Any help would be much appreciated, thanks.

Python snippet:

newqueue = Gst.ElementFactory.make("queue",f"queue1_{index}")
pipeline.add(newqueue)

muxtee_src = newmuxtee.get_static_pad("src_1")

if muxtee_src.link(newqueue.get_static_pad("sink")) != Gst.PadLinkReturn.OK:
    print("Error linking to queue1")

newqueue.sync_state_with_parent()
new_mux_pad = streammux_dynamic.request_pad_simple(SINKTEMP + index)

for pad in streammux_dynamic.sinkpads:
    pad_name = pad.get_name()
    pad_caps = pad.get_current_caps()
    query_caps = pad.query_caps(None)
    if pad == new_mux_pad:
        print(f"\n(NEW SINK) CAPS for {pad_name}:\ncurrent     -> {pad_caps}\ntemplate    -> {query_caps}\n")
    else:
        print(f"\nCAPS for {pad_name}:\ncurrent    -> {pad_caps}\ntemplate    -> {query_caps}\n")

src_caps_query = newqueue.get_static_pad("src").get_current_caps()
src_caps_temp = newqueue.get_static_pad("src").query_caps(None)

print(f"\nCAPS for SRC pad:\ncurrent    -> {src_caps_temp}\ntemplate    -> {src_caps_query}\n\n")

evt_start = Gst.Event.new_stream_start(str(index_int))
newqueue.get_static_pad("sink").send_event(evt_start)
evt_start_mux = Gst.Event.new_stream_start(str(index_int))
new_mux_pad.send_event(evt_start_mux)

ret = newqueue.get_static_pad("src").link(new_mux_pad)

terminal output:

CAPS for sink_0000:
current     -> video/x-raw(memory:NVMM), format=(string)NV12, width=(int)800, height=(int)600, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, framerate=(fraction)25/1, nvbuf-memory-type=(string)nvbuf-mem-cuda-device, gpu-id=(int)0, batch-size=(uint)1, num-surfaces-per-frame=(int)1
template    -> video/x-raw, format=(string)NV12, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, nvbuf-memory-type=(string)nvbuf-mem-cuda-device, gpu-id=(int)0, batch-size=(int)4, num-surfaces-per-frame=(int)1, framerate=(fraction)[ 0/1, 2147483647/1 ], width=(int)[ 0, 2147483647 ], height=(int)[ 0, 2147483647 ]; video/x-raw(memory:NVMM), format=(string)NV12, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, nvbuf-memory-type=(string)nvbuf-mem-cuda-device, gpu-id=(int)0, batch-size=(int)4, num-surfaces-per-frame=(int)1, framerate=(fraction)[ 0/1, 2147483647/1 ], width=(int)[ 0, 2147483647 ], height=(int)[ 0, 2147483647 ]

CAPS for sink_0002:
current     -> video/x-raw(memory:NVMM), format=(string)NV12, width=(int)800, height=(int)600, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, framerate=(fraction)25/1, nvbuf-memory-type=(string)nvbuf-mem-cuda-device, gpu-id=(int)0, batch-size=(uint)1, num-surfaces-per-frame=(int)1
template    -> video/x-raw, format=(string)NV12, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, nvbuf-memory-type=(string)nvbuf-mem-cuda-device, gpu-id=(int)0, batch-size=(int)4, num-surfaces-per-frame=(int)1, framerate=(fraction)[ 0/1, 2147483647/1 ], width=(int)[ 0, 2147483647 ], height=(int)[ 0, 2147483647 ]; video/x-raw(memory:NVMM), format=(string)NV12, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, nvbuf-memory-type=(string)nvbuf-mem-cuda-device, gpu-id=(int)0, batch-size=(int)4, num-surfaces-per-frame=(int)1, framerate=(fraction)[ 0/1, 2147483647/1 ], width=(int)[ 0, 2147483647 ], height=(int)[ 0, 2147483647 ]

CAPS for sink_0004:
current     -> video/x-raw(memory:NVMM), format=(string)NV12, width=(int)800, height=(int)600, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, framerate=(fraction)25/1, nvbuf-memory-type=(string)nvbuf-mem-cuda-device, gpu-id=(int)0, batch-size=(uint)1, num-surfaces-per-frame=(int)1
template    -> video/x-raw, format=(string)NV12, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, nvbuf-memory-type=(string)nvbuf-mem-cuda-device, gpu-id=(int)0, batch-size=(int)4, num-surfaces-per-frame=(int)1, framerate=(fraction)[ 0/1, 2147483647/1 ], width=(int)[ 0, 2147483647 ], height=(int)[ 0, 2147483647 ]; video/x-raw(memory:NVMM), format=(string)NV12, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, nvbuf-memory-type=(string)nvbuf-mem-cuda-device, gpu-id=(int)0, batch-size=(int)4, num-surfaces-per-frame=(int)1, framerate=(fraction)[ 0/1, 2147483647/1 ], width=(int)[ 0, 2147483647 ], height=(int)[ 0, 2147483647 ]

(NEW SINK) CAPS for sink_0001:
current     -> None
template    -> video/x-raw, format=(string)NV12, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, nvbuf-memory-type=(string)nvbuf-mem-cuda-device, gpu-id=(int)0, batch-size=(int)4, num-surfaces-per-frame=(int)1, framerate=(fraction)[ 0/1, 2147483647/1 ], width=(int)[ 0, 2147483647 ], height=(int)[ 0, 2147483647 ]; video/x-raw(memory:NVMM), format=(string)NV12, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, nvbuf-memory-type=(string)nvbuf-mem-cuda-device, gpu-id=(int)0, batch-size=(int)4, num-surfaces-per-frame=(int)1, framerate=(fraction)[ 0/1, 2147483647/1 ], width=(int)[ 0, 2147483647 ], height=(int)[ 0, 2147483647 ]

CAPS for SRC pad:
current     -> video/x-raw(memory:NVMM), format=(string)NV12, width=(int)800, height=(int)600, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, framerate=(fraction)25/1, nvbuf-memory-type=(string)nvbuf-mem-cuda-device, gpu-id=(int)0, batch-size=(uint)1, num-surfaces-per-frame=(int)1
template    -> None

[ERROR] Unexpected error for source 0001: <enum GST_PAD_LINK_NOFORMAT of type Gst.PadLinkReturn>
3

Please refer to our sample code deepstream_rt_src_add_del.py to learn how to dynamically add and remove sources.

4

Thanks for the sound advice !

The sample code completely destroys the source bins but we need to keep ours alive. Our pipeline roughly goes:
nvstreammux_0 → nvinfer → nvstreamdemux → nvstreammux_1.

The specific issue I have is with how we dynamically link our chosen (based on logic) nvstreamdemux src pads to freshly requested nvstreammux sink pads. Do nvstreamdemux src pads need to be requested before the pipeline moves into PLAYING, or is this impossible?
When trying to request the pads before they are needed, and then later linking them we have the above caps issue.

Is it posssible to manually reset the internal table used by nvmultistreamtiler (for source tracking) or is seq-square-grid the only applicaple property we have available?

Is there an NVMM alternative to gst compositor? I’ve found nvcompositor information but there doesn’t seem to be any actual plugin available for x86 + dGPU.

Finally, the deepstream 7.1 documentation and pyds sample apps are very limited, is this the correct platform for finding out about how specific nv plugins work?

thanks

5

No. It doesn’t destroy the sourcebins but only create a new urisourcebin.

Your pipeline is a bit complicated. We suggest that you gradually achieve the function of dynamically adding and deleting sources in the following sequence.

  1. nvstreammux->nvinfer
  2. nvstreammux->nvinfer->nvstreamdemux
  3. nvstreammux->nvinfer->nvstreamdemux->nvstreammux

You can try the custom-tile-config parameter.

Currently, there is no such plugin available for use on the dGPU.

6

Thanks

7

is there any information about the CustomTileConfig enum?

8

Sorry, this is reserved for future use. The current function has not yet been implemented. We will discuss internally on whether to provide support for dGPU in the uncoming versions. Thanks