Secondary inference using IInferCustomProcessor: inferenceDone() called AFTER gstBufffer pushed

**• Hardware Platform (Jetson / GPU) : dGPU
**• DeepStream Version : 6.0.1
**• JetPack Version (valid for Jetson only) : None
**• TensorRT Version : 8.0.1
**• NVIDIA GPU Driver Version (valid for GPU only) : 495.29.05
**• Issue Type( questions, new requirements, bugs) : questions
**• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing) : None
**• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description) : None


background

I am working on the emotion net app which uses following pipeline:
pgie(nvinfer, face) → sgie1(nvinfer, landmark recognition) → sgie2(nvdsvideotemplate, emotionNet which uses sgie1 result as input)

I am replacing nvinfer with inferserver+IInferCustomProcessor

landmark metadata (which is used by sgie2) is attached by sgie1’s IInferCustomProcessor::inferenceDone function.

problem

The problem is,
during the processing of sgie2, the metadata that should be inserted by sgie1 cannot be found.
It seems that the sgie2 process is running before the sgie1 metadata-attaching is completed.

Everything works fine and emotion-result is obtained If I insert std::this_thread::sleep_for(std::chrono::seconds(1)) in the sgie2 processing-loop (before accessing to obj_meta->obj_user_meta_list).
But this type of hack is unacceptable for the real-world applications.

How can I address this issue?

what I tried

disabling tracking did not help.
setting async_mode: false of sgie1 did not help (nvinferserver document link)

1 about “I am replacing nvinfer with inferserver+IInferCustomProcessor”, do you means all three nvinfer were replaced with replacing nvinfer with inferserver+IInferCustomProcessor"? please provide your media pipeline?
2 about “during the processing of sgie2, the metadata that should be inserted by sgie1 cannot be found.”, if there is no face and landmark, there will be no landmark meta.
3 about “It seems that the sgie2 process is running before the sgie1 metadata-attaching is completed.”, there is only one async mode sgie , that is sgie + tracker, your case is not this mode, so landmark meta will come with buffer if found face and landmark. please refer to Gst-nvinferserver — DeepStream 6.1.1 Release documentation

Hi,
thank you for the reply.

1

the pipeline is
(sources) → nvstreammux → nvinferserver(facedetect, default-detector-post-process) → queue → nvinferserver(landmark, IInferCustomProcessor) → queue → nvdsvideotemplate(emotionNet, customlib)

emotionnet custo-lib is from deepstream_tao_apps/apps/tao_others/deepstream-emotion-app/emotion_impl at master · NVIDIA-AI-IOT/deepstream_tao_apps · GitHub

2

I used a movie looping a single image .
the image has 2 faces.

I also confirmed faces are detected by the app.

3

Everything works fine and emotion-result is obtained If I insert std::this_thread::sleep_for(std::chrono::seconds(1)) in the sgie2 processing-loop (before accessing to obj_meta->obj_user_meta_list).

The results of the above experiment suggest that it is a matter of timing.

Could you confirm that the gstbuffer is pushed AFTER IInferCustomProcessor::inferenceDone() completed?


PS:

In nvdsvideotemplate, I implemented a sleep loop so that it waits until the landmark metadata is obtained, and it works.
However, the sleep loop is not an ideal solution and I would like to solve this problem if possible.

1 did you set interval in pgie and sgie? it will skip some frames instead of inference.
2 about " nsert std::this_thread::sleep_for(std::chrono::seconds(1)) in the sgie2 processing-loop (before accessing to obj_meta->obj_user_meta_list).", where did you add that sleep? which function?

1
I did not set interval:
interval: 0 in config-file

2
in void EmotionAlgorithm::OutputThread(void) before framemeta loopping:

/* Output Processing Thread */
void EmotionAlgorithm::OutputThread(void)
{
  GstFlowReturn flow_ret;
  GstBuffer *outBuffer = NULL;
  NvBufSurface *outSurf = NULL;
  NvDsBatchMeta *batch_meta = NULL;
  std::unique_lock<std::mutex> lk(m_processLock);
  /* Run till signalled to stop. */
  while (1) {
    /* Wait if processing queue is empty. */
    if (m_processQ.empty()) {
      if (m_stop == TRUE) {
        break;
      }
      m_processCV.wait(lk);
      continue;
    }

    PacketInfo packetInfo = m_processQ.front();
    m_processQ.pop();

    m_processCV.notify_all();
    lk.unlock();

    // DEBUG
    // wait 1 sec first
    printf("[DEBUG] emotion: wait for a while\n");
    std::this_thread::sleep_for(std::chrono::seconds(1));

    // Add custom algorithm logic here
    // Once buffer processing is done, push the buffer to the downstream by using gst_pad_push function
    batch_meta = gst_buffer_get_nvds_batch_meta (packetInfo.inbuf);

    //First getting the bbox of faces and eyes
    NvDsMetaList * l_frame = NULL;
    NvDsMetaList * l_obj = NULL;
    NvDsObjectMeta *obj_meta = NULL;
    uint8_t *imagedataPtr = NULL;

    nvds_acquire_meta_lock (batch_meta);

    for (l_frame = batch_meta->frame_meta_list; l_frame != NULL;
    ..........

1 About your “disabling tracking did not help.”, did your pipeline have tracker? where is position?
2 Only not in async mode, the gstbuffer is pushed AFTER IInferCustomProcessor::inferenceDone() completed.

1

tracker is after sgie, if enabled
Actually, my code is based on deepstream-app and almost everything is the same with deepstream-app

2

I have set async_mode: false in the config, but the problem persists…

please provide the full terminal log, thanks, for example: execute export GST_DEBUG=4, then execute: app >1.log 2>1.log
1 set async_mode = true, save the log.
2 set async_mode = false, save the log.

I can’t share the logs with you because the production logs are included here and there.
My apologies.

I would like to activatethis issue again when I can create clean code that reproduces the problem, but I don’t have time right now.

Thank you for your support.

I would appreciate it if you could contact me if you find out anything.

Is this still an issue to support? could you share your clean code that reproduces the problem?

Yes it is.
However, I will not be working on it for the time being.

Thanks.

I’m closing this topic due to there is no update from you for a period, assuming this issue was resolved.
If still need the support, please open a new topic. Thanks

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.