DeepStream Python bindings with new Streammux outputs nothing

Zygislu · September 30, 2023, 8:59am

• Hardware Platform (Jetson / GPU)
GPU
• DeepStream Version
6.2
• NVIDIA GPU Driver Version (valid for GPU only)
535
• Issue Type( questions, new requirements, bugs)
Bugs
• How to reproduce the issue?

I have setup DeepStream 6.2 and installed python bindings for it. I’m using USE_NEW_NVSTREAMMUX=yes and have modified deepstream-imagedata-multistream python script to use New Streammux as such:

    if not sink:
        sys.stderr.write(" Unable to create egl sink \n")

    if os.environ.get('USE_NEW_NVSTREAMMUX') != 'yes':  # Only set these properties if not using new gst-nvstreammux
        if is_live:
            print("Atleast one of the sources is live")
            streammux.set_property('live-source', 1)

        streammux.set_property('width', STREAMMUX_WIDTH)
        streammux.set_property('height', STREAMMUX_HEIGHT)
        streammux.set_property('batched-push-timeout', 40000)
    streammux.set_property('batch-size', number_sources)
    pgie.set_property('config-file-path', "inference_config.txt")

and modified this code part aswell:

    if not is_aarch64():
        # Use CUDA unified memory in the pipeline so frames
        # can be easily accessed on CPU in Python.
        mem_type = int(pyds.NVBUF_MEM_CUDA_UNIFIED)
        if os.environ.get('USE_NEW_NVSTREAMMUX') != 'yes':
            streammux.set_property("nvbuf-memory-type", mem_type)
        nvvidconv.set_property("nvbuf-memory-type", mem_type)
        nvvidconv1.set_property("nvbuf-memory-type", mem_type)
        tiler.set_property("nvbuf-memory-type", mem_type)

So now when i don’t use New Streammux everything works as intended, however if I set USE_NEW_NVSTREAMMUX to yes then the pipeline doesn’t work and sources don’t get processed. I suspect it’s because of not using nvbuf-memory-type with New Streammux, or maybe it’s something else?

yingliu · October 1, 2023, 9:17am

Should be other reason as nvbuf-memory-type doesn’t exist in the new streammux. (refer the comparation here: Gst-nvstreammux New — DeepStream 6.3 Release documentation (nvidia.com))
You can set GST_DEBUG=3 to get more information before the issue happens.

Zygislu · October 1, 2023, 9:36am

I have set GST_DEBUG=3 and this is what i get when i run the pipeline:

deepstream-python | 0:00:06.235296306     1 0x7fd5a00674c0 WARN                 qtdemux qtdemux.c:3250:qtdemux_parse_trex:<qtdemux0> failed to find fragment defaults for stream 1
deepstream-python | 0:00:06.250728709     1 0x7fd5a0067de0 WARN                    v4l2 gstv4l2object.c:3057:gst_v4l2_object_get_nearest_size:<nvv4l2decoder0:sink> Unable to try format: Unknown error -1
deepstream-python | 0:00:06.250763256     1 0x7fd5a0067de0 WARN                    v4l2 gstv4l2object.c:2942:gst_v4l2_object_probe_caps_for_format:<nvv4l2decoder0:sink> Could not probe minimum capture size for pixelformat MJPG
deepstream-python | 0:00:06.250771312     1 0x7fd5a0067de0 WARN                    v4l2 gstv4l2object.c:3057:gst_v4l2_object_get_nearest_size:<nvv4l2decoder0:sink> Unable to try format: Unknown error -1
deepstream-python | 0:00:06.250778342     1 0x7fd5a0067de0 WARN                    v4l2 gstv4l2object.c:2948:gst_v4l2_object_probe_caps_for_format:<nvv4l2decoder0:sink> Could not probe maximum capture size for pixelformat MJPG
deepstream-python | 0:00:06.250815328     1 0x7fd5a0067de0 WARN                    v4l2 gstv4l2object.c:3057:gst_v4l2_object_get_nearest_size:<nvv4l2decoder0:sink> Unable to try format: Unknown error -1
deepstream-python | 0:00:06.250822672     1 0x7fd5a0067de0 WARN                    v4l2 gstv4l2object.c:2942:gst_v4l2_object_probe_caps_for_format:<nvv4l2decoder0:sink> Could not probe minimum capture size for pixelformat AV10
deepstream-python | 0:00:06.250828807     1 0x7fd5a0067de0 WARN                    v4l2 gstv4l2object.c:3057:gst_v4l2_object_get_nearest_size:<nvv4l2decoder0:sink> Unable to try format: Unknown error -1
deepstream-python | 0:00:06.250836709     1 0x7fd5a0067de0 WARN                    v4l2 gstv4l2object.c:2948:gst_v4l2_object_probe_caps_for_format:<nvv4l2decoder0:sink> Could not probe maximum capture size for pixelformat AV10
deepstream-python | 0:00:06.250852199     1 0x7fd5a0067de0 WARN                    v4l2 gstv4l2object.c:3057:gst_v4l2_object_get_nearest_size:<nvv4l2decoder0:sink> Unable to try format: Unknown error -1
deepstream-python | 0:00:06.250885063     1 0x7fd5a0067de0 WARN                    v4l2 gstv4l2object.c:2942:gst_v4l2_object_probe_caps_for_format:<nvv4l2decoder0:sink> Could not probe minimum capture size for pixelformat DVX5
deepstream-python | 0:00:06.250891687     1 0x7fd5a0067de0 WARN                    v4l2 gstv4l2object.c:3057:gst_v4l2_object_get_nearest_size:<nvv4l2decoder0:sink> Unable to try format: Unknown error -1
deepstream-python | 0:00:06.250898792     1 0x7fd5a0067de0 WARN                    v4l2 gstv4l2object.c:2948:gst_v4l2_object_probe_caps_for_format:<nvv4l2decoder0:sink> Could not probe maximum capture size for pixelformat DVX5
deepstream-python | 0:00:06.250918528     1 0x7fd5a0067de0 WARN                    v4l2 gstv4l2object.c:3057:gst_v4l2_object_get_nearest_size:<nvv4l2decoder0:sink> Unable to try format: Unknown error -1
deepstream-python | 0:00:06.250935246     1 0x7fd5a0067de0 WARN                    v4l2 gstv4l2object.c:2942:gst_v4l2_object_probe_caps_for_format:<nvv4l2decoder0:sink> Could not probe minimum capture size for pixelformat DVX4
deepstream-python | 0:00:06.250961846     1 0x7fd5a0067de0 WARN                    v4l2 gstv4l2object.c:3057:gst_v4l2_object_get_nearest_size:<nvv4l2decoder0:sink> Unable to try format: Unknown error -1
deepstream-python | 0:00:06.250979191     1 0x7fd5a0067de0 WARN                    v4l2 gstv4l2object.c:2948:gst_v4l2_object_probe_caps_for_format:<nvv4l2decoder0:sink> Could not probe maximum capture size for pixelformat DVX4
deepstream-python | 0:00:06.251011225     1 0x7fd5a0067de0 WARN                    v4l2 gstv4l2object.c:3057:gst_v4l2_object_get_nearest_size:<nvv4l2decoder0:sink> Unable to try format: Unknown error -1
deepstream-python | 0:00:06.251043500     1 0x7fd5a0067de0 WARN                    v4l2 gstv4l2object.c:2942:gst_v4l2_object_probe_caps_for_format:<nvv4l2decoder0:sink> Could not probe minimum capture size for pixelformat MPG4
deepstream-python | 0:00:06.251050352     1 0x7fd5a0067de0 WARN                    v4l2 gstv4l2object.c:3057:gst_v4l2_object_get_nearest_size:<nvv4l2decoder0:sink> Unable to try format: Unknown error -1
deepstream-python | 0:00:06.251057649     1 0x7fd5a0067de0 WARN                    v4l2 gstv4l2object.c:2948:gst_v4l2_object_probe_caps_for_format:<nvv4l2decoder0:sink> Could not probe maximum capture size for pixelformat MPG4
deepstream-python | 0:00:06.251070342     1 0x7fd5a0067de0 WARN                    v4l2 gstv4l2object.c:3057:gst_v4l2_object_get_nearest_size:<nvv4l2decoder0:sink> Unable to try format: Unknown error -1
deepstream-python | 0:00:06.251078831     1 0x7fd5a0067de0 WARN                    v4l2 gstv4l2object.c:2942:gst_v4l2_object_probe_caps_for_format:<nvv4l2decoder0:sink> Could not probe minimum capture size for pixelformat MPG2
deepstream-python | 0:00:06.251085505     1 0x7fd5a0067de0 WARN                    v4l2 gstv4l2object.c:3057:gst_v4l2_object_get_nearest_size:<nvv4l2decoder0:sink> Unable to try format: Unknown error -1
deepstream-python | 0:00:06.251092834     1 0x7fd5a0067de0 WARN                    v4l2 gstv4l2object.c:2948:gst_v4l2_object_probe_caps_for_format:<nvv4l2decoder0:sink> Could not probe maximum capture size for pixelformat MPG2
deepstream-python | 0:00:06.251106423     1 0x7fd5a0067de0 WARN                    v4l2 gstv4l2object.c:3057:gst_v4l2_object_get_nearest_size:<nvv4l2decoder0:sink> Unable to try format: Unknown error -1
deepstream-python | 0:00:06.251114912     1 0x7fd5a0067de0 WARN                    v4l2 gstv4l2object.c:2942:gst_v4l2_object_probe_caps_for_format:<nvv4l2decoder0:sink> Could not probe minimum capture size for pixelformat H265
deepstream-python | 0:00:06.251121884     1 0x7fd5a0067de0 WARN                    v4l2 gstv4l2object.c:3057:gst_v4l2_object_get_nearest_size:<nvv4l2decoder0:sink> Unable to try format: Unknown error -1
deepstream-python | 0:00:06.251129550     1 0x7fd5a0067de0 WARN                    v4l2 gstv4l2object.c:2948:gst_v4l2_object_probe_caps_for_format:<nvv4l2decoder0:sink> Could not probe maximum capture size for pixelformat H265
deepstream-python | 0:00:06.251141244     1 0x7fd5a0067de0 WARN                    v4l2 gstv4l2object.c:3057:gst_v4l2_object_get_nearest_size:<nvv4l2decoder0:sink> Unable to try format: Unknown error -1
deepstream-python | 0:00:06.251148874     1 0x7fd5a0067de0 WARN                    v4l2 gstv4l2object.c:2942:gst_v4l2_object_probe_caps_for_format:<nvv4l2decoder0:sink> Could not probe minimum capture size for pixelformat VP90
deepstream-python | 0:00:06.251155639     1 0x7fd5a0067de0 WARN                    v4l2 gstv4l2object.c:3057:gst_v4l2_object_get_nearest_size:<nvv4l2decoder0:sink> Unable to try format: Unknown error -1
deepstream-python | 0:00:06.251165011     1 0x7fd5a0067de0 WARN                    v4l2 gstv4l2object.c:2948:gst_v4l2_object_probe_caps_for_format:<nvv4l2decoder0:sink> Could not probe maximum capture size for pixelformat VP90
deepstream-python | 0:00:06.251194881     1 0x7fd5a0067de0 WARN                    v4l2 gstv4l2object.c:3057:gst_v4l2_object_get_nearest_size:<nvv4l2decoder0:sink> Unable to try format: Unknown error -1
deepstream-python | 0:00:06.251213401     1 0x7fd5a0067de0 WARN                    v4l2 gstv4l2object.c:2942:gst_v4l2_object_probe_caps_for_format:<nvv4l2decoder0:sink> Could not probe minimum capture size for pixelformat VP80
deepstream-python | 0:00:06.251221056     1 0x7fd5a0067de0 WARN                    v4l2 gstv4l2object.c:3057:gst_v4l2_object_get_nearest_size:<nvv4l2decoder0:sink> Unable to try format: Unknown error -1
deepstream-python | 0:00:06.251228902     1 0x7fd5a0067de0 WARN                    v4l2 gstv4l2object.c:2948:gst_v4l2_object_probe_caps_for_format:<nvv4l2decoder0:sink> Could not probe maximum capture size for pixelformat VP80
deepstream-python | 0:00:06.251242939     1 0x7fd5a0067de0 WARN                    v4l2 gstv4l2object.c:3057:gst_v4l2_object_get_nearest_size:<nvv4l2decoder0:sink> Unable to try format: Unknown error -1
deepstream-python | 0:00:06.251261926     1 0x7fd5a0067de0 WARN                    v4l2 gstv4l2object.c:2942:gst_v4l2_object_probe_caps_for_format:<nvv4l2decoder0:sink> Could not probe minimum capture size for pixelformat H264
deepstream-python | 0:00:06.251289747     1 0x7fd5a0067de0 WARN                    v4l2 gstv4l2object.c:3057:gst_v4l2_object_get_nearest_size:<nvv4l2decoder0:sink> Unable to try format: Unknown error -1
deepstream-python | 0:00:06.251324932     1 0x7fd5a0067de0 WARN                    v4l2 gstv4l2object.c:2948:gst_v4l2_object_probe_caps_for_format:<nvv4l2decoder0:sink> Could not probe maximum capture size for pixelformat H264
deepstream-python | 0:00:06.252196724     1 0x7fd5a0067de0 WARN                    v4l2 gstv4l2object.c:3057:gst_v4l2_object_get_nearest_size:<nvv4l2decoder0:src> Unable to try format: Unknown error -1
deepstream-python | 0:00:06.252228989     1 0x7fd5a0067de0 WARN                    v4l2 gstv4l2object.c:2942:gst_v4l2_object_probe_caps_for_format:<nvv4l2decoder0:src> Could not probe minimum capture size for pixelformat NM12
deepstream-python | 0:00:06.252262626     1 0x7fd5a0067de0 WARN                    v4l2 gstv4l2object.c:3057:gst_v4l2_object_get_nearest_size:<nvv4l2decoder0:src> Unable to try format: Unknown error -1
deepstream-python | 0:00:06.252277025     1 0x7fd5a0067de0 WARN                    v4l2 gstv4l2object.c:2948:gst_v4l2_object_probe_caps_for_format:<nvv4l2decoder0:src> Could not probe maximum capture size for pixelformat NM12
deepstream-python | 0:00:06.252316156     1 0x7fd5a0067de0 WARN                    v4l2 gstv4l2object.c:2395:gst_v4l2_object_add_interlace_mode:0x7fd598019ff0 Failed to determine interlace mode
deepstream-python | 0:00:06.256055046     1 0x7fd5a0067de0 FIXME              h265parse gsth265parse.c:1850:gst_h265_parse_parse_frame:<h265parse0> Implement timestamp/duration interpolation based on SEI message
deepstream-python | 0:00:06.368502102     1 0x7fd5a0067de0 WARN             nvstreammux gstnvstreammux.cpp:1051:configure_module:<Stream-muxer> No config-file provided; falling back to default streammux config 1
deepstream-python | 
deepstream-python | 0:00:06.372027769     1 0x7fd5a0067de0 WARN            v4l2videodec gstv4l2videodec.c:1836:gst_v4l2_video_dec_decide_allocation:<nvv4l2decoder0> Duration invalid, not setting latency
deepstream-python | 0:00:06.372114377     1 0x7fd5a0067de0 WARN          v4l2bufferpool gstv4l2bufferpool.c:1082:gst_v4l2_buffer_pool_start:<nvv4l2decoder0:pool:src> Uncertain or not enough buffers, enabling copy threshold
deepstream-python | 0:00:06.372505305     1 0x7fd5a0067de0 FIXME              h265parse gsth265parse.c:1850:gst_h265_parse_parse_frame:<h265parse0> Implement timestamp/duration interpolation based on SEI message
deepstream-python | 0:00:06.373082126     1 0x7fd5a0067de0 FIXME              h265parse gsth265parse.c:1850:gst_h265_parse_parse_frame:<h265parse0> Implement timestamp/duration interpolation based on SEI message
deepstream-python | 0:00:06.373202620     1 0x7fd5a0067de0 FIXME              h265parse gsth265parse.c:1850:gst_h265_parse_parse_frame:<h265parse0> Implement timestamp/duration interpolation based on SEI message
deepstream-python | 0:00:06.373293192     1 0x7fd5a0067de0 FIXME              h265parse gsth265parse.c:1850:gst_h265_parse_parse_frame:<h265parse0> Implement timestamp/duration interpolation based on SEI message
deepstream-python | 0:00:06.488436676     1 0x7fd598027c00 WARN          v4l2bufferpool gstv4l2bufferpool.c:1533:gst_v4l2_buffer_pool_dqbuf:<nvv4l2decoder0:pool:src> Driver should never set v4l2_buffer.field to ANY
deepstream-python | 0:00:06.488553900     1 0x7fd5a0067de0 FIXME              h265parse gsth265parse.c:1850:gst_h265_parse_parse_frame:<h265parse0> Implement timestamp/duration interpolation based on SEI message
deepstream-python | 0:00:06.489604991     1 0x7fd5a0067de0 FIXME              h265parse gsth265parse.c:1850:gst_h265_parse_parse_frame:<h265parse0> Implement timestamp/duration interpolation based on SEI message
deepstream-python | 0:00:06.490558220     1 0x7fd5a0067de0 FIXME              h265parse gsth265parse.c:1850:gst_h265_parse_parse_frame:<h265parse0> Implement timestamp/duration interpolation based on SEI message
deepstream-python | 0:00:06.490755246     1 0x7fd5a0067de0 FIXME              h265parse gsth265parse.c:1850:gst_h265_parse_parse_frame:<h265parse0> Implement timestamp/duration interpolation based on SEI message
deepstream-python | 0:00:06.491985916     1 0x7fd5a0067de0 FIXME              h265parse gsth265parse.c:1850:gst_h265_parse_parse_frame:<h265parse0> Implement timestamp/duration interpolation based on SEI message
deepstream-python | 0:00:06.492204384     1 0x7fd5a0067de0 FIXME              h265parse gsth265parse.c:1850:gst_h265_parse_parse_frame:<h265parse0> Implement timestamp/duration interpolation based on SEI message
deepstream-python | 0:00:06.492520567     1 0x7fd5a0067de0 FIXME              h265parse gsth265parse.c:1850:gst_h265_parse_parse_frame:<h265parse0> Implement timestamp/duration interpolation based on SEI message
deepstream-python | 0:00:06.492856508     1 0x7fd5a0067de0 FIXME              h265parse gsth265parse.c:1850:gst_h265_parse_parse_frame:<h265parse0> Implement timestamp/duration interpolation based on SEI message

fanzh · October 8, 2023, 6:33am

sorry for the late reply.

what do you mean about “sources don’t get processed”?
what is your start commandline?

fanzh · October 8, 2023, 7:20am

using the same code, I can’t reproduce the issue on T4+DS6.2. here are the code and log.
deepstream_imagedata-multistream.py (17.5 KB)
log.txt (6.5 KB)

Zygislu · October 8, 2023, 11:00am

Thanks for the reply, your code works, thanks for it as well.
In the main question I forgot to mention that I’m using additionally these lines to crop ROI from input source:

        if uri_name.find("rtsp://") == 0:
            is_live = True
        source_bin = create_source_bin(i, uri_name)
        if not source_bin:
            sys.stderr.write("Unable to create source bin \n")

        nvvidconv_crop = Gst.ElementFactory.make("nvvideoconvert", f'nvvideoconvert{i}')
        nvvidconv_crop.set_property('src-crop', '14:26:819:288')

        pipeline.add(source_bin)
        pipeline.add(nvvidconv_crop)

        padname = "sink_%u" % i
        sinkpad = streammux.get_request_pad(padname)
        if not sinkpad:
            sys.stderr.write("Unable to create sink pad bin \n")
        srcpad = source_bin.get_static_pad("src")
        if not srcpad:
            sys.stderr.write("Unable to create src pad bin \n")

        nvvidconv_srcpad = nvvidconv_crop.get_static_pad("src")
        nvvidconv_sinkpad = nvvidconv_crop.get_static_pad("sink")

        srcpad.link(nvvidconv_sinkpad)
        nvvidconv_srcpad.link(sinkpad)

    print("Creating Pgie \n ")
    pgie = Gst.ElementFactory.make("nvinfer", "primary-inference")

When I comment them out, everything works, however when I enable them, the pipeline doesn’t work.
Maybe you have an example or suggestion how I could fix this, so that ROI cropping would work with new streammux, because with old it works.

fanzh · October 10, 2023, 9:13am

you can work around by adding a NV12 capsfilter after nvvideoconvert. please refer to this code
1.diff (7.1 KB)

Zygislu · October 14, 2023, 8:21am

Thank you for all the help and the code above, the pipeline runs now. However when i save frames in tiler_sink_pad_buffer_probe function, more specifically when I save n_frame it saves it at the same resolution as my video, e.g. 1920x1080, but not cropped, e.g. 416x256.
What other changes should I make so that I could save n_frame as cropped?

fanzh · October 15, 2023, 2:47pm

please refer to this sample test, the resolution of 7.jpeg is still 720p.
gst-launch-1.0 filesrc location= /opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.jpg ! jpegdec ! ‘video/x-raw,format=I420’ ! nvvideoconvert src-crop=“0:0:300:300” ! jpegenc ! filesink location=7.jpeg

Zygislu · October 17, 2023, 4:37am

Yeah you are right. Is there anything I can add to maybe the end of the pipeline so that the output image would be the same resolution as the crop?
Or should I use something else instead of nvvideoconvert src-crop?

fanzh · October 17, 2023, 6:32am

please refer to this command, the output resolution is the same with crop’s.
gst-launch-1.0 filesrc location= /opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.jpg ! jpegdec ! ‘video/x-raw,format=I420’ ! nvvideoconvert ! ‘video/x-raw,format=I420,width=300,height=300’ ! jpegenc ! filesink location=7.jpeg

system · October 31, 2023, 6:33am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.