How can appsrc properly retrieve data from realsense so that it can be used by nvinfer

**• Hardware Platform: jetson orin nx
**• DeepStream Version: 7.0
**• JetPack Version: 6.0GA
**• TensorRT Version: TRT 8.6.2.3

Hi!
When I run the following code,an error occurs.I want to know why this happened and how to deal with it.

ERROR: gst-stream-error-quark: Internal data stream error. (1): …/libs/gst/base/gstbasesrc.c(3127): gst_base_src_loop (): /GstPipeline:pipeline0/GstAppSrc:realsense-source

The code is:
def on_need_data(source, user_data):
frames = pipeline_rs.wait_for_frames()
color_frame = frames.get_color_frame()
color_image = np.asanyarray(color_frame.get_data())

if color_frame:
    buf = Gst.Buffer.new_wrapped_bytes(GLib.Bytes.new(color_image.tobytes()))
    
    flag = source.emit("push-buffer", buf)
    if flag != Gst.FlowReturn.OK:
        print("error")
    return flag

def main():
platform_info = PlatformInfo()
# Standard GStreamer initialization
Gst.init(None)

# Create gstreamer elements
# Create Pipeline element that will form a connection of other elements
print("Creating Pipeline \n ")
pipeline = Gst.Pipeline()

if not pipeline:
    sys.stderr.write(" Unable to create Pipeline \n")

# Source element for reading from intel realsense
print("Creating Source \n ")
source = Gst.ElementFactory.make("appsrc", "realsense_source")
if not source:
    sys.stderr.write(" Unable to create Source \n")

# Use videoconvertor to convert from BGR8 to RGBA as required by nvinfer
print("Creating Video Converter \n")
vidconv = Gst.ElementFactory.make("videoconvert", "convertor_src")
if not vidconv:
    sys.stderr.write(" Unable to create videoconvert \n")

# Create nvstreammux instance to form batches from one or more sources.
streammux = Gst.ElementFactory.make("nvstreammux", "Stream-muxer")
if not streammux:
    sys.stderr.write(" Unable to create NvStreamMux \n")

# Use nvinfer to run inferencing on decoder's output,
# behaviour of inferencing is set through config file
pgie = Gst.ElementFactory.make("nvinfer", "primary-inference")
if not pgie:
    sys.stderr.write(" Unable to create pgie \n")

# Use convertor to convert from NV12 to RGBA as required by nvosd
nvvidconv = Gst.ElementFactory.make("nvvideoconvert", "convertor")
if not nvvidconv:
    sys.stderr.write(" Unable to create nvvidconv \n")

# Create OSD to draw on the converted RGBA buffer
nvosd = Gst.ElementFactory.make("nvdsosd", "onscreendisplay")

if not nvosd:
    sys.stderr.write(" Unable to create nvosd \n")

# Finally render the osd output
if platform_info.is_integrated_gpu():
    print("Creating nv3dsink \n")
    sink = Gst.ElementFactory.make("nv3dsink", "nv3d-sink")
    if not sink:
        sys.stderr.write(" Unable to create nv3dsink \n")
else:
    if platform_info.is_platform_aarch64():
        print("Creating nv3dsink \n")
        sink = Gst.ElementFactory.make("nv3dsink", "nv3d-sink")
    else:
        print("Creating EGLSink \n")
        sink = Gst.ElementFactory.make("nveglglessink", "nvvideo-renderer")
    if not sink:
        sys.stderr.write(" Unable to create egl sink \n")

# set appsrc element
caps_string = "video/x-raw, format=BGR, width=1280, height=720, framerate=30/1"
source.set_property("is-live", True)
source.set_property("format", Gst.Format.TIME)
source.set_property("caps", Gst.caps_from_string(caps_string))
source.connect("need-data", on_need_data)

# set nvstreammux element
if os.environ.get('USE_NEW_NVSTREAMMUX') != 'yes': # Only set these properties if not using new gst-nvstreammux
    streammux.set_property('width', 1920)
    streammux.set_property('height', 1080)
    streammux.set_property('batched-push-timeout', MUXER_BATCH_TIMEOUT_USEC)

streammux.set_property('batch-size', 1)
pgie.set_property('config-file-path', "dstest1_pgie_config.txt")

print("Adding elements to Pipeline \n")
pipeline.add(source)
pipeline.add(vidconv)
pipeline.add(streammux)
pipeline.add(pgie)
pipeline.add(nvvidconv)
pipeline.add(nvosd)
pipeline.add(sink)

# link the elements together
print("Linking elements in the Pipeline \n")
source.link(vidconv)

sinkpad = streammux.request_pad_simple("sink_0")
if not sinkpad:
    sys.stderr.write(" Unable to get the sink pad of streammux \n")

vidconv.link(streammux)
streammux.link(pgie)
pgie.link(nvvidconv)
nvvidconv.link(nvosd)
nvosd.link(sink)

# create an event loop and feed gstreamer bus mesages to it
loop = GLib.MainLoop()
bus = pipeline.get_bus()
bus.add_signal_watch()
bus.connect ("message", bus_call, loop)

osdsinkpad = nvosd.get_static_pad("sink")
if not osdsinkpad:
    sys.stderr.write(" Unable to get sink pad of nvosd \n")

osdsinkpad.add_probe(Gst.PadProbeType.BUFFER, osd_sink_pad_buffer_probe, 0)

# start play back and listen to events
print("Starting pipeline \n")
pipeline.set_state(Gst.State.PLAYING)
try:
    loop.run()
except:
    pass
# cleanup
pipeline.set_state(Gst.State.NULL)
  1. can you simplify the code to narrow down this issue? for example,
    can "appsrc → vidconv → fakesink " run well?
    can “appsrc → vidconv → nvstreammux → fakesink” run well?

  2. Noticing vidconv linked with streammux, streammux only can accept NV12, RGBA, I420 format, please make sure vidconv outputs these format. you can add a casfilter after vidconv .

Hi fanzh!
Thank you for your reply!

the issue is :
can appsrc->vidconv->capsfilter->streammux->nvinfer->nvvidconv->nvosd->nv3dsink run well?

from your log, this pipeline “appsrc->vidconv->capsfilter->streammux->nvinfer->nvvidconv->nvosd->nv3dsink” can’t run.
please refer to my last comment.

  1. from the code, there is no capsfilter between appsrc and streammux.
  2. after adding capsfilter, the app still can’t run. please simplify the pipeline to narrow down this issue.

I have done the following tests:

 1.  "appsrc->vidconv->capsfilter->xvimagesink" can run well.
 2.  "filesrc->h264parse->decoder->streammux->nvinfer->nvvidconv->nvosd->nv3dsink" can run well.This is based on deepstream-test1 and I have tested it myself.
 3.  "appsrc->vidconv->capsfilter->streammux->nvinfer->nvvidconv->nvosd->nv3dsink" can't run.

Does it mean that there is an error between capsfilter and streammux?
Is this an issue with the data format?

By the way,the data from realsense is input by this way:

    pipeline_rs = rs.pipeline()
    config = rs.config()
    config.enable_stream(rs.stream.color, 1280, 720, rs.format.bgr8, 30)
    pipeline_rs.start(config)

    frames = pipeline_rs.wait_for_frames()
    color_frame = frames.get_color_frame()
    color_image = np.asanyarray(color_frame.get_data())

    if color_frame:
        buf = Gst.Buffer.new_wrapped_bytes(GLib.Bytes.new(color_image.tobytes()))
      
        flag = source.emit("push-buffer", buf)
        if flag != Gst.FlowReturn.OK:
            print("error")

And the capsfilter is set to:

    caps_string2 = Gst.caps_from_string("format=NV12, width=1280, height=720,framerate=30/1")
    caps.set_property("caps", caps_string2)

nvstreammux requires hardware format. vidconv is videoconvert type, which can’t output hardware format. so please try “appsrc->vidconv->capsfilter->nvvidconv->streammux->fakesink”.
Then try “appsrc->vidconv->capsfilter->nvvidconv->streammux->nvinfer->nvvidconv->nvosd->nv3dsink”.

Thank you for your reply!

According to your advice,I have tried:

  1. “appsrc->vidconv->capsfilter->nvvidconv->streammux->fakesink” can run well.
  2. “appsrc->vidconv->capsfilter->nvvidconv->streammux->nvinfer->nvvidconv->nvosd->nv3dsink” can’t run,and it reported an error “gi.overrides.Gst.AddError: <gi.Gstnvvideoconvert object at 0xffff9d9043c0 (Gstnvvideoconvert at 0xaaaaf91361f0)”.I don’t quite understand what this means.
  1. you can add plugin step by step, for exmaple, can * “appsrc->vidconv->capsfilter->nvvidconv->streammux->nvinfer->fakesink” run well?
  2. please refer to this deepstream_test_1.py, only the source is different between test1 and your application. the pipeline after streammux is similar.
  3. if still can’t work, could you share the whole log?

Sorry,the error was caused by my oversight.There were two plugins using the same name.
The issue has been solved.
Thank you very much.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.