New Nv* Data Structures in custom plugin to be pushed downstream

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Hello everyone,

I am currently implementing custom Deepstream plugin for video analytics. This plugin will come after “nvstreammux” element in pipeline. The logic of the plugin is as follow:

  1. Map input buffer (GstBuffer) to NvBufSurface.
  2. Obtain NvDsBatchMeta of input buffer.
  3. Create new NvBufSurface and new NvDsBatchMeta, both empty.
  4. Fill new NvBufSurface and NvDsBatchMeta, from point 3, with modified data based on original input buffer objects, from point 1 and 2.
  5. Convert filled NvBufSurface, from point 4, to GstBuffer.
  6. Convert filled NvDsBatchMeta, from point 4, to GstMeta.
  7. Attach metadata, from point 6, to buffer, from point 5.
  8. Discard original buffer along with its metadata, from point 1 and 2.
  9. Push modified buffer, from point 7, to srcpad.

In short I want to create brand new buffer based on NvBufSurface along with brand new metadata based on NvDsBatchMeta and push it downstream, while discarding original buffer.

The thing is I cannot find appropriate way to “convert” NvDsBatchMeta to GstBuffer and NvDsBatchMeta to GstMeta, so that the next element in pipeline “nvinfer” will handle them.

I would appreciate any help in this subject. I just hope I am not missing some crucial point.

Best,
Tom

2

Hi
May I know why you discard nvstreammux original buffer and recreate them?

3

Hi,

The plugin I mentioned have to basically clone and process frames incoming from “nvstreammux” element. On the output buffer I would like to have not only the original frames I got but also their copies, three for each, with some filter applied. Therefore there are 3 times more frames on the output than on the input. Processing is done purely by OpenCV cuda contrib modules.

Currently I am stuck at the very end where new NvBufSurface was created and filled, along with NvDsBatchMeta, via filling individual NvDsFrameMeta’s. NvBufSurface is copied to new, mapped data of GstBuffer. NvDsBatchMeta is added to output buffer via gst_buffer_add_nvds_meta. Still, the response from the “nvinfer” element, which is next, is:

0:00:05.699618690 29995   0x556fd3a990 WARN                 nvinfer gstnvinfer.cpp:1830:gst_nvinfer_output_loop:<primary-nvinference-engine> error: Internal data stream error.
0:00:05.699828480 29995   0x556fd3a990 WARN                 nvinfer gstnvinfer.cpp:1830:gst_nvinfer_output_loop:<primary-nvinference-engine> error: streaming stopped, reason error (-5)
Error: Internal data stream error.
Returned, stopping playback
Deleting pipeline
4

nvinfer batch-size should be greater or equal than source numbers or streammux numbers, or it will encounter error, how about your nvinfer batch-size, since you clone 3 more frames from streammux, nvinfer batch-size should change accordingly.

5

Hi @amycao,

batch-size property of “nvinfer” element, controlled via config.txt file, is set correctly, that’s sure. For batch-size property of “nvstreammux” element equal to 2, I am setting 6 for batch-size of “nvinfer”.

I imagine the problem lies in one of those places:

  • NvBufSurface for individual frames
  • NvFrameMeta for individual frames
  • NvDsBatchMeta parameters
  • The way I am converting NvBufSurface to GstBuffer and the way I am adding NvDsBatchMeta to this buffer.

As a sanity check I have created additional dummy element, which I have placed after my plugin element in the pipeline. There, I have read the buffer and the contents looks exactly the same as in my plugin, therefore I suspect that the problem is not in the last point.

I imagine that the only place where the example how new Nv* structures are created and pushed is precisely “nvstreammux” element, for which source code isn’t public, is it?

6

As of now there is no plan to open source existing streammux in ds 5.0

Will check internally, and get back to you.

7

HI
Could you please share your code which can build and run on our environments for recreating your issue and a further check?

8

Hi @amycao,

Since the code I am working on is proprietary I have PM You.

Thank You in advance for Your help.

10

Hi,
Sorry for a late response,
Here are some points you need to implement:

// Create NvBufSurface
NvBufSurface *surf;
NvBufSurfaceCreate(&surf, …);
//Create a GstBuf wrapper over the NvBufSurface
gst_buffer_new_wrapped_full (0, *surf, sizeof(NvBufSurface), 0, sizeof(NvBufSurface), NULL, NULL);
// Create batch meta
batch_meta = nvds_create_batch_meta(mux->batch_size);
meta = gst_buffer_add_nvds_meta (out_buf, batch_meta, NULL,
nvds_batch_meta_copy_func, nvds_batch_meta_release_func);
meta->meta_type = NVDS_BATCH_GST_META;
batch_meta->base_meta.batch_meta = batch_meta;
batch_meta->base_meta.copy_func = nvds_batch_meta_copy_func;
batch_meta->base_meta.release_func = nvds_batch_meta_release_func;
batch_meta->max_frames_in_batch = batch_size;
// Create and add frame meta
NvDsFrameMeta *frame_meta = nvds_acquire_frame_meta_from_pool(batch_meta);
nvds_add_frame_meta_to_batch(batch_meta, frame_meta);
// Some important parameters to fill
frame_meta->pad_index
frame_meta->source_id
frame_meta->buf_pts
frame_meta->ntp_timestamp
frame_meta->frame_num
frame_meta->batch_id
frame_meta->source_frame_width
frame_meta->source_frame_height
frame_meta->num_surfaces_per_frame
// members for dewarper like elements
frame_meta->surface_type
frame_meta->surface_index

11

Hi @Amycao,

Thanks for Your reply. This actually helped in some limited regard. I hope You don’t mind more questions to come.

Thanks once more for Your help.

Best,
Tom

12

Please do not hestinate and reach out to me if you need help.

14

hi i have a similar problem

i linked pipeline → 'Src → Streamux → dsexample → overlaysink ’

Streammux send to dsexample more than two batched meta+frames

i want to the out buffer have single batch meta + frame after processing with transform_ip

i read by accident this topic, and write some code follow the amycao’s answer,

but I don’t think it’s a complete code.

i don’t know how to change over from input Gstbuffer to my new GstBuffer

  1. i made pipeline (src(v4l2) → streammux → dsexample → overlaysink)
  2. want to make new GstBuffer for src pad in dsexample plugin
  3. new GstBuffer is 1batched composited data(menu ui + N camera src(input gstbuffer))
  4. how to make new GstBuffer and change over from input gstbuffer to my new gstbuffer in transform_ip func ?

can you give some guide for me ??