How to know which object in nvinfer (classifier) belongs to which object in nvinfer (detection)

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

• Hardware Platform: GPU
• DeepStream Version: 6.3
• TensorRT Version: 8.5.3-1+cuda11.8
• NVIDIA GPU Driver Version: 525.89.02

Hello, I have completed the deepstream pipeline as follows:
pgie (detect vehicle) → sgie (detect license plate one row or two row) → sgie 2 (recognize character)

Currently I want to get the class information of one row or two rows in phase sgie 1 to optimize the merging of characters in phase sgie 2. My ParseCustom for (pgie, sgie 1, sgie 2) as follows:

# Vehicle detect
extern "C" bool NvDsInferParseCustomYolor(std::vector<NvDsInferLayerInfo> const& outputLayersInfo,
    NvDsInferNetworkInfo const& networkInfo,
    NvDsInferParseDetectionParams const& detectionParams,
    std::vector<NvDsInferParseObjectInfo>& objectList)
{
  // my code
}

# License Plate detect (one row or two row)
extern "C" bool NvDsInferParseCustomYoloV8LPD(std::vector<NvDsInferLayerInfo> const& outputLayersInfo,
NvDsInferNetworkInfo const& networkInfo, 
NvDsInferParseDetectionParams const& detectionParams,
std::vector<NvDsInferParseObjectInfo>& objectList)
{
   // my code
}

# recognize (list character -> string)
extern "C" bool NvDsInferParseCustomYoloV8LPR(std::vector<NvDsInferLayerInfo> const &outputLayersInfo,
                                 NvDsInferNetworkInfo const &networkInfo, float classifierThreshold,
                                 std::vector<NvDsInferAttribute> &attrList, std::string &attrString)
{
 // my code
}

How can I know which outputLayersInfo in NvDsInferParseCustomYoloV8LPR function belongs to the object with bounding box and which class in outputLayersInfo part of NvDsInferParseCustomYoloV8LPD. Because of the asynchronous deepstream issue it becomes difficult in this case.

Hope everyone will review and give some comments, thanks you !

2

yeah, its a problem. Async processing makes it tough. I am interested in a official solution

1 Like
4

The parsed class string will be attached to the NvDsObjectMeta. You can get the relationship with NvDsObjectMeta. deepstream_lpr_app/deepstream-lpr-app/deepstream_lpr_app.c at master · NVIDIA-AI-IOT/deepstream_lpr_app (github.com)

6

It seems that the answer is not correct, what you send is the result returned when the pipeline (pgie → sgie 1 → sgie 2) one or more frames is complete. What I need is how to know during the pipeline processing one or more frames, which outputLayersInfo parameter in NvDsInferParseCustomYoloV8LPR function comes from which outputLayersInfo in NvDsInferParseCustomYoloV8LPD

7

You can’t get any extra information inside the model parsing callback function. It is not supported. Please do your job in other places.

8

Is there a solution to my idea?

9

The gst-nvinfer is open source, you may try to modify it to transfer NvDsObjectMeta to the callback.

Why do you need the object information in the model output parsing stage?

10

I need the class of the object in the license plate detector (one row or two row) phase (sgie1) to serve the character merging in the recognize character phase (sgie2).

11

Then you may do your postprocessing in the app level, please refer to /opt/nvidia/deepstream/deepstream-/sources/apps/sample_apps/deepstream-infer-tensor-meta-test, then you can get NvDsObjectMeta when you do the postprocessing(including model output parsing).

12

Thanks for your suggestion, I will look into it and if it works I will share with everyone

13

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks

14

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.