[SOLVED] Trouble running inference on RealSense RGB stream: GStreamer-CRITICAL **: gst_meta_api_type_has_tag: assertion 'tag != 0' failed

I’m trying to create a GStreamer/DeepStream pipeline on a Jetson Xavier to detect, track, and classify objects, and have gotten stuck with the message, "GStreamer-CRITICAL **: gst_meta_api_type_has_tag: assertion 'tag != 0' failed" . Can anyone provide insight on what the error means and/or how to resolve it? I can run the pipeline with just background subtraction fine (excluding detector0 , tracker0 , and classifier0 ), and can plug the object detector and classifier into another example, making me believe there is a problem with the RGB stream, but I’ve had a hard time determining what that is. Could somebody give me guidance?

Below is a synopsis of the network I’m trying to build:

(realsense) -> (background subtraction) -> (object detection) -> (object tracking) -> (object classification) -> (rendition)

The error appears to originate from gst/gstmeta.c#L120, which is invoked by the NvDsMeta API. Beyond that, I’m not sure, specifically, what is causing the failure.

This is an example run of the pipeline I’m trying to execute, with the error message:

$ gst-launch-1.0 realsensesrc serial="$RS_SERIAL" enable-color=true enable-depth=false framerate=6 \
! rgbddemux name=realsense_demux realsense_demux.src_color \
! videoconvert \
! segmentation test-mode=false method=2 \
! nvvidconv \
! "video/x-raw(memory:NVMM)" \
! nvinfer name=detector0 config-file-path="$DETECTOR_CONFIG" batch-size=1 unique-id=1 \
! nvtracker name=tracker0 ll-lib-file=/opt/nvidia/deepstream/deepstream-4.0/lib/libnvds_mot_klt.so \
! nvinfer name=classifier0 config-file-path="$CLASSIFIER_CONFIG" unique-id=2 \
! nvdsosd \
! nvegltransform \
! nveglglessink
Setting pipeline to PAUSED ...

Using winsys: x11
Creating LL OSD context new
gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream-4.0/lib/libnvds_mot_klt.so
gstnvtracker: Optional NvMOT_RemoveStreams not implemented
gstnvtracker: Batch processing is OFF
Deserialize yoloLayerV3 plugin: yolo_83
Deserialize yoloLayerV3 plugin: yolo_95
Deserialize yoloLayerV3 plugin: yolo_107
Pipeline is live and does not need PREROLL ...
Got context from element 'eglglessink0': gst.egl.EGLDisplay=context, display=(GstEGLDisplay)NULL;
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Creating LL OSD context new
 15/04 16:38:47,267 ERROR [545301447152] (types.h:307) get_device_time_ms() took too long (more then 2 mSecs)
 15/04 16:38:47,419 ERROR [545301447152] (types.h:307) get_device_time_ms() took too long (more then 2 mSecs)
 15/04 16:38:47,573 ERROR [545301447152] (types.h:307) get_device_time_ms() took too long (more then 2 mSecs)
 15/04 16:38:47,725 ERROR [545301447152] (types.h:307) get_device_time_ms() took too long (more then 2 mSecs)
 15/04 16:38:47,877 ERROR [545301447152] (types.h:307) get_device_time_ms() took too long (more then 2 mSecs)
 15/04 16:38:48,030 ERROR [545301447152] (types.h:307) get_device_time_ms() took too long (more then 2 mSecs)
 15/04 16:38:48,120 ERROR [545058189808] (types.h:307) get_device_time_ms() took too long (more then 2 mSecs)
 15/04 16:38:48,189 ERROR [545301447152] (types.h:307) get_device_time_ms() took too long (more then 2 mSecs)

(gst-launch-1.0:22288): GStreamer-CRITICAL **: 16:38:48.191: gst_meta_api_type_has_tag: assertion 'tag != 0' failed

(gst-launch-1.0:22288): GStreamer-CRITICAL **: 16:38:48.191: gst_meta_api_type_has_tag: assertion 'tag != 0' failed
0:00:08.801922496 22288   0x5597b8d280 WARN                 nvinfer gstnvinfer.cpp:1201:gst_nvinfer_process_full_frame:<detector0> error: NvDsBatchMeta not found for input buffer.
ERROR: from element /GstPipeline:pipeline0/GstNvInfer:detector0: NvDsBatchMeta not found for input buffer.
Additional debug info:
/dvs/git/dirty/git-master_linux/deepstream/sdk/src/gst-plugins/gst-nvinfer/gstnvinfer.cpp(1201): gst_nvinfer_process_full_frame (): /GstPipeline:pipeline0/GstNvInfer:detector0
Execution ended after 0:00:01.052650098
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
 15/04 16:38:48,284 ERROR [545058189808] (types.h:307) get_device_time_ms() took too long (more then 2 mSecs)
 15/04 16:38:48,345 ERROR [545301447152] (types.h:307) get_device_time_ms() took too long (more then 2 mSecs)
Setting pipeline to NULL ...
Freeing pipeline ...

Here is an example where the object detector and classifier work:

$ gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deepstream-4.0/samples/streams/sample_1080p_h264.mp4 ! \
qtdemux ! \
h264parse ! \
nvv4l2decoder ! \
m.sink_0 nvstreammux name=m batch-size=1 width=1920 height=1080 ! \
nvinfer config-file-path="$DETECTOR_CONFIG" batch-size=1 unique-id=1 ! \
nvtracker ll-lib-file=/opt/nvidia/deepstream/deepstream-4.0/lib/libnvds_mot_klt.so ! \
nvinfer config-file-path="$CLASSIFIER_CONFIG" batch-size=16 unique-id=2 infer-on-gie-id=1 infer-on-class-ids=0 ! \
nvmultistreamtiler rows=1 columns=1 width=1280 height=720 ! \
nvvideoconvert ! \
nvdsosd ! \
nvegltransform ! \
nveglglessink

A big difference that I see is that the latter example uses nvstreammux, but when I plug it into my pipeline I get the error:

...
ERROR: from element /GstPipeline:pipeline0/GstNvStreamMux:nvmuxer: Input buffer number of surfaces (0) must be equal to mux->num_surfaces_per_frame (1)
        Set nvstreammux property num-surfaces-per-frame appropriately
...

I am able to fix this error by inserting the nvstreammux expression before running object detection and by using nvvideoconvert instead of nvvidconv, but the rendered frames have not had their backgrounds subtracted:

$ gst-launch-1.0 realsensesrc serial="$RS_SERIAL" enable-color=true enable-depth=false framerate=6 \
! rgbddemux name=realsense_demux realsense_demux.src_color \
! videoconvert \
! segmentation test-mode=false method=2 \
! nvvideoconvert \
! "video/x-raw(memory:NVMM)" \
! nvmuxer.sink_0 nvstreammux name=nvmuxer batch-size=1 width=1280 height=720 \
! nvinfer name=detector0 config-file-path="$DETECTOR_CONFIG" batch-size=1 unique-id=1 \
! nvtracker name=tracker0 ll-lib-file=/opt/nvidia/deepstream/deepstream-4.0/lib/libnvds_mot_klt.so \
! nvinfer name=classifier0 config-file-path="$CLASSIFIER_CONFIG" unique-id=2 \
! nvdsosd \
! nvegltransform \
! nveglglessink

Hi,
CPU buffers in BGR format is not optimal on Jetson platforms. The optimal solution is to have video frames in NVMM buffer and you will need nvvideoconvert for the conversion. Besides, nvinfer has to be linked with nvstreammux. It relies on nvstreammux for organizing the source(s).

Your observation look expected and matches the implementation. The conversion can take certain CPU loading and please execute sudo jetson_clocks to get max performance.

It looks like the segmentation plugin adds an alpha channel to the RGB frames from the RealSense camera. Rather than applying the background-subtracted mask to the RGB frame, it embeds it into the alpha channel. My object detector works on RGB frames and not RGBA, so I believe once it hits the first nvinfer plugin the inference is run on the original RGB frame (ignoring the alpha channel) and the RGB frame is then passed through the rest of the pipeline without the alpha channel. My object detector depends on background subtraction to reduce noise, do you know of a way to merge the alpha channel into the RGB frames before the DS segment of the pipeline? I can drop the alpha channel with videoconvert but that doesn’t do me any good. I could write a custom filter that uses OpenCV’s bitwise_and to apply the alpha channel mask to the other channels but would prefer to stick with built-in plugins.