Get image gst-buffer created by from URIDECODEBIN

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

• Hardware Platform (Jetson / GPU) :- GPU
• DeepStream Version :- 6.4
• TensorRT Version
• NVIDIA GPU Driver Version (valid for GPU only) :- 545
• Issue Type( questions, new requirements, bugs) :- Question

I have a deepstream application that is using URIDECODEBIN as the source element. The pipeline looks like
uridecodebin -> nvvideoconvert -> streammux -> rest-of-the-pipeline

the uridecodebin element is reading an RTSP stream. I want the pipeline to work only on some part of the video so, I’m using src-crop property of nvvideoconvert element to crop the incoming RTSP stream and feed only the required ROI to the pipeline. But because of this I loose the full FOV image coming from the camera.

My Question is :- is there any way to do that ? or whats the best way to do that ?

What I have tried so far :-

  • Trial One
    I thought of attaching a probe function at sink pad of nvvideoconvert and access the full fov image and attach it as a custom gst-metadata to the buffer. but the linking of uridecodebin -> jpegenc -> nvvideoconvert creates some linking issues (probably related to caps). Im uploading
    error_logs.txt (695 Bytes)
    if you’d want to check that.

  • Trial Two
    I create two queues. One containing Full FOV images another containing the cropped ones. Cropped ones are feeded to the pipeline, after inference is over, I try to map the inference results from the cropped images to the full FOV images. I dont like this approach as The mapping is often not right.

More information if it helps :- The elements created inside/by uridecodebin for the type of stream that Im using are (in order) :- depay-> parser-> tee_rtsp_pre_decode-> dec_que-> tee_rtsp_post_decode-> decodebin-> queue-> nvvidconv-> src_cap_filter_nvvidconv-> h264parse0-> capsfilter0-> nvv4l2decoder0

2

I think It depends on your task, if you are just using nvinfer or nvinferserver, you can write your own postprocess logic to filter results that are out of your ROI, but if your pipeline does some operation on the original image, this may not work as you expected.

Your solution 1 is not good but 2 seems better (but still can’t meet your requirement). Anyway i prefer do postprocess inside nvinferserver or nvinfer

3

You don’t need to use nvvideoconvert to crop the video. I suggest you use nvdspreprocess. You only need to specify the ROI.

Please refer to the document below

https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvdspreprocess.html#id4

5

cant use nvdspreprocess because we have multiple sources and the cropping needs to happen before the streammux.

7

nvdspreprocess can support multiple sources. Just set roi-params-src-<id> in nvdspreprocess configuration file for every source.

Is cropping necessary?

8

yes cropping is necessary. I want the full resolution (whatever the camera supports) image, to be given to the pipeline for inference. so we take the full resolution image from camera, crop the required ROI and then feed it to streammux.

if we feed the image coming from camera without cropping, in most of the cases, camera is capturing regions where we dont really have any object.

Also, we have to run the pipeline on different cameras having different resolutions, if we dont crop the ROI, the streammux will rescale all of them to one resoution, which may result in out objects getting very small in size.

9

Do you need a pipeline similar to the one below?

                     |-----------> FOV image  --> sink
uridecodebin --> tee |
                     |-----------> nvvideoconvert (do crop) --> nvstreammux --> infer --> osd --> sink
10

yes.

                     |-----------> FOV image  --> sink
uridecodebin --> tee |
                     |-----------> nvvideoconvert (do crop) --> nvstreammux --> infer --> osd --> sink

the task is to obtain the full image before we crop it. if it can be done without splitting the pipeline we are okay.

so maybe,

                     
uridecodebin ---------> nvvideoconvert (do crop) --> nvstreammux --> infer --> osd --> sink\
                   |
                   | 
       we can also put a probe here,
        to obtain the full image before we crop it.

once we obtain the full image, we plan on attaching it as custom get-meta to the gst-buffers.

11

why we were thinking of spllitting the pipeline in the first place is because we thought in the second head of the pipeplien we would put nvjpegenc or somehing that will encode the iamge from raw data to jpeg. But thats okay, splitting the pipeline is not something that is compelled upon us.

what matters is, getting the full frame before it gets cropped. and attaching it as custom gst meta.

we are open to do it in any way that works.

12

If the requirement is to save images in the probe function, you can refer to deepstream-image-meta-test. nvds_obj_enc_process can help you complete this work very well.

13

we already have that probe in our pipeline.
that WONT work here as i want to get the full frame BEFORE streammux.

14

can you please read the question, understand it and then reply?

15

Do you understand what I said?

If this is the pipeline you need, add a probe function in the src pad of uridecodebin, nvds_obj_enc_process can save the full frame as an image and add a custom gst-meta (NVDS_CROP_IMAGE_META). Cropping and inference will also work correctly

If not, please share your pipeline

16 
pgie_src_pad_buffer_probe (GstPad * pad, GstPadProbeInfo * info, gpointer ctx)
{
  GstBuffer *buf = (GstBuffer *) info->data;
  GstMapInfo inmap = GST_MAP_INFO_INIT;
  if (!gst_buffer_map (buf, &inmap, GST_MAP_READ)) {
    GST_ERROR ("input buffer mapinfo failed");
    return GST_PAD_PROBE_DROP;
  }
  NvBufSurface *ip_surf = (NvBufSurface *) inmap.data;
  gst_buffer_unmap (buf, &inmap);

  NvDsObjectMeta *obj_meta = NULL;
  guint vehicle_count = 0;
  guint person_count = 0;
  NvDsMetaList *l_frame = NULL;
  NvDsMetaList *l_obj = NULL;
  NvDsBatchMeta *batch_meta = gst_buffer_get_nvds_batch_meta (buf);
  const gchar *calc_enc_str = g_getenv ("CALCULATE_ENCODE_TIME");
  gboolean calc_enc = !g_strcmp0 (calc_enc_str, "yes");

  for (l_frame = batch_meta->frame_meta_list; l_frame != NULL;
      l_frame = l_frame->next) {
    NvDsFrameMeta *frame_meta = (NvDsFrameMeta *) (l_frame->data);

this is a part of the probe function that you are talking about. My point is, if I put it before streammux, it will not work as there is no NVDS meta. am I right ??

17

Ok, I got your point, you can modify the pipeline to look like the following, which is similar to this GitHub - NVIDIA-AI-IOT/deepstream_parallel_inference_app: A project demonstrating how to use nvmetamux to run multiple models in parallel..

I think it should work.

uridecodebin --> nvstreammux ----> nvstreamdemux --> nvviddeoconvert (do crop) --> nvstreammux --> .....
							|
							|
                       probe at here

You can also add custom gst-meta using gst_buffer_add_meta. This does not depend on nvstreammux and is not compatible with NvDsMetaType. Of course nvds_obj_enc_process will not work either. You can use software encoding if you want.

https://gstreamer.freedesktop.org/documentation/gstreamer/gstbuffer.html?gi-language=c#gst_buffer_add_meta

18 
> uridecodebin --> nvstreammux(first) ----> nvstreamdemux --> 
---> nvviddeoconvert (do crop) --> nvstreammux(second) --> .....
> 							|
> 							|
>                        probe at here

this seems interesting. but since I have different cameras running at different resolution, I would still have to give some width and height to streammux(one).

Im thinking of creating a bin, an Input bin containing uridecodebin->streammux->nvstreamdemux->nvvideoconvert

each source will have this unique bin and streammux in this bin will get the stream resolution so it doesnt rescale the image.

I think that should work. what do you think?

19

You can try it. If you need the rtsp reconnection feature, you can consider using nvurisrcbin instead of uridecodebin.

20

yes. thanks.

21

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks

22

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.