Deepstream can run async mode?

• Hardware Platform (Jetson / GPU) Jetson Nano Module (Custom Board)
• DeepStream Version 5.0
• JetPack Version (valid for Jetson only) 4.4
• TensorRT Version 7
• Issue Type( questions, new requirements, bugs) questions

Hi we made an application based on Deepstream SDK (Gstreamer)

i have a question.

i know, deepstream does not support async pgie(object detector).

by any chance, can i make async object detector??

this is i need

I want to use a higher performance model to reduce misdetection,

This slows down the output, i want video shows the original speed and I want to make inference as a async mode.

any idea???

thank you

Hi,
It is not supported by default, but you can check source code of nvinfer plugin and do customization. The code is in

/opt/nvidia/deepstream/deepstream-5.0/sources/gst-plugins/gst-nvinfer

We also have dsexample plugin. Please check

/opt/nvidia/deepstream/deepstream-5.0/sources/gst-plugins/gst-dsexample

You can modify metadata in the plugin. May implement the osd handler block based on it.

hi DaneLLL

thank your reply.

i read source code of gstnvinfer.

I understand, maybe gstnvinfer follow this flow

  1. init and start two thread
    first thread [gst_nvinfer_input_queue_loop] = waiting ‘input queue’ and add process queue
    second thread [gst_nvinfer_input_queue_loop] = waiting ‘process queue’

  2. call gst_nvinfer_submit_input_buffer() when buffer input from up stream

  3. do convert_mat() [maybe resize for network info] and make converted_batch

  4. add converted_batch to input queue and broadcast to first thread

  5. in first thread, make new input_batch and call queueInputBatch < - maybe for inferece ?

  6. and converted_batch to process queue and broadcast to second thread

  7. in second thread, call queueOutputBatch ← maybe inference completed

  8. attach metadata.

question.

  1. can you tell me where i modify code for pgie async inference mode?

  2. now (nvinfer default), while nvinfer do inference. will the upstream data be ignored or not coming?

If classification async mode and solution are similar, I would like to get help.

+add

Hi DaneLLL

While waiting for an answer, I added asynchronous mode.

i modified some function and add var.

this code seems to work well.

result video [ yolo V4-CrowdHuman Model On DS5.0 Nano]

code
gst-nvinfer.zip (49.5 KB)

but i have some question.

in “gst_nvinfer_process_full_frame”

 for (guint i = 0; i < num_filled; i++) {
    guint idx;		
...
...
...

}

If you look at the video above, usually the box is drawn in the normal position but sometimes a box is drawn toward an incorrect frame.

I thought the variable ‘i’ was ‘source_id’ but it doesn’t work like that.

when use ‘pad_index’ in ‘frame_meta’ is also.
[my input source is two but sometimes show the pad_index higher than 1]

complpete pgie pad0 continued
complpete pgie pad1 continued
frame.frame_meta->pad_index 0
frame.frame_meta->pad_index 1
stop infer
complpete pgie pad0 continued
complpete pgie pad1 continued
frame.frame_meta->pad_index 0
frame.frame_meta->pad_index 1
stop infer
complpete pgie pad0 continued
complpete pgie pad1 continued
frame.frame_meta->pad_index 0
frame.frame_meta->pad_index 1
stop infer
complpete pgie pad0 continued
complpete pgie pad1 continued
frame.frame_meta->pad_index 1  <=
frame.frame_meta->pad_index 2  <=

I think ‘classification async mode’ uses “SourceInfo”. Will this be the answer?

How can I determine “source_id” in the loop statement above?

Hi,
For source id, please check
DeepStream SDK FAQ
How to get/update source_id

And please make sure you set batch-size=2 to nvstreammux.

already i set batch-size =2 to nvstreammux

and i got the ‘source_id’ of ‘frame_meta’ at your suggestion

and i tested to gstnvinfer, but this looks strange when after inference

  1. source_id seems to correct in gst_nvinfer_process_full_frame

code

        GstNvInferFrame frame;
        frame.converted_frame_ptr = memory->frame_memory_ptrs[idx];
        frame.scale_ratio_x = scale_ratio_x;
        frame.scale_ratio_y = scale_ratio_y;
        frame.obj_meta = nullptr;
        frame.frame_meta = nvds_get_nth_frame_meta (batch_meta->frame_meta_list, i);
        frame.frame_num = frame.frame_meta->frame_num;
        frame.batch_index = i;
        frame.input_surf_params = in_surf->surfaceList + i;
        g_print("before infer frame source_id %d var i = %d\n",frame.frame_meta->source_id, i);

output =

before infer frame source_id 0 var i = 1
before infer frame source_id 1 var i = 0

before infer frame source_id 0 var i = 1
before infer frame source_id 1 var i = 0

  1. source_id same to “1.” in gst_nvinfer_input_queue_loop

code

for (i = 0; i < batch->frames.size (); i++) {
  input_frames.push_back (batch->frames[i].converted_frame_ptr);
	  g_print("input loop source_id = %d var i = %d\n", batch->frames[i].frame_meta->source_id, i);
}
...

output =

before infer frame source_id 1 var i = 0
before infer frame source_id 0 var i = 1
input loop source_id = 1 var i = 0
input loop source_id = 0 var i = 1

before infer frame source_id 1 var i = 0
before infer frame source_id 0 var i = 1
input loop source_id = 1 var i = 0
input loop source_id = 0 var i = 1
  1. source_id seems to incorrect when after inference in gst_nvinfer_output_loop

code

  for (guint i = 0; i < batch->frames.size (); i++) {
  GstNvInferFrame & frame = batch->frames[i];
  NvDsInferFrameOutput &frame_output = batch_output->frames[i];
  auto obj_history = frame.history.lock ();

  /* If we have an object's history and the buffer PTS is same as last
   * inferred PTS mark the object as not being inferred. This check could be
   * useful if object is inferred multiple times before completion of an
   * existing inference. */
  if (obj_history != nullptr) {
    if (obj_history->last_inferred_frame_num == frame.frame_num)
      obj_history->under_inference = FALSE;
  }
  	 
  guint id = frame.frame_meta->source_id;
  g_print("output source_id %d var i = %d\n", id, i);

.....

output =

before infer frame source_id 1 var i = 0
before infer frame source_id 0 var i = 1
input loop source_id = 1 var i = 0
input loop source_id = 0 var i = 1
output source_id 0 var i = 0    < incorrect
output source_id -738111664 var i = 1  <incorrect

before infer frame source_id 1 var i = 0
before infer frame source_id 0 var i = 1
input loop source_id = 1 var i = 0
input loop source_id = 0 var i = 1
output source_id 0 var i = 0  < incorrect
output source_id 0 var i = 1  < maybe incorrect

I think there is a problem during the inference. Where can I get the information?


temporarily, I hypothesized that the frame order after inference is the same as the frame order at the time of entry.
So I’m processing the process by buffering the “source_id” at point “2.”

Hi,
Not sure what it wrong. Ideally you should see frames with correct source_id. We have discussion and would like to suggest set the interval property in the nvinfer config file. After nvinfer, nvtracker can be added which will help track and maintain current bounding box around the detected objects.
You can check and try the config file:

/opt/nvidia/deepstream/deepstream-5.0/samples/configs/deepstream-app/source8_1080p_dec_infer-resnet_tracker_tiled_display_fp16_nano.txt

More information is in document:
DeepStream Reference Application - deepstream-app — DeepStream 5.1 Release documentation

Hi DaneLLL

I think interval option is not proper to async output,
because when during inference the mainthread[pipeline] is blocked.

and i tested my custom nvinfer(pgie async), maybe two weeks
it still works without any problems.

However, the way I identified “source id” is not normal.

later, please let me know if your team meeting reveals why the “source id” was broken after inference.

thank you.