Tertairy mode for nvinfer?

[tom_s]

• Hardware Platform: Jetson
• DeepStream : 6.0.1
• Issue Type question

I have a pipeline that detects faces and then facial landmarks (running 2 nvinfer, one in primary and one in secondary mode).
The output of the facial landmarks is a list of facial landmarks (x,y) in the meta data of the stream

As a third step i would like to pass the facial landmarks (that are in the meta data) to a different model (using nvinfer ? ) so it can start doing some inference on the facial landmarks, but from what i can see is that nvinfer only works on image data ?

Q1: is it possible to load an onnx that only works on the meta data added to the stream by previous nvinfer plugins ?

Q2: What do i need to do to make sure this third model uses the meta data ?

Q3: are there any example floating around where i can see what needs to be done ?

[Fiona.Chen]

@tom_s

is it possible to load an onnx that only works on the meta data added to the stream by previous nvinfer plugins ?

Yes. We alreay have such sample deepstream_tao_apps/apps/tao_others/deepstream-emotion-app at master · NVIDIA-AI-IOT/deepstream_tao_apps (github.com)

@tom_s

What do i need to do to make sure this third model uses the meta data ?

Please implement the inferencing by yourself, you can implement with gst-nvdsvideotemplate Gst-nvdsvideotemplate — DeepStream documentation 6.4 documentation , just as what we implement for the emotion model here deepstream_tao_apps/apps/tao_others/deepstream-emotion-app at master · NVIDIA-AI-IOT/deepstream_tao_apps (github.com)

@tom_s

are there any example floating around where i can see what needs to be done ?

Please refer to deepstream_tao_apps/apps/tao_others/deepstream-emotion-app at master · NVIDIA-AI-IOT/deepstream_tao_apps (github.com)

[tom_s]

I just got this running.
this is actually what i was trying to detect (smiling/laughing) and the ‘happy’ emotion seems to cover it.

I notice in the modelcard for emotionNet that it is trained on

INPUT

68 points (X, Y) of Human facial landmarks (1 x 136 x 1)

The training pipeline can accept more input points, but the pre-train model is trained with 68 points input.

but the FpeNet has more… it looks like FpeNet is a superset of the Dlib’s 68 Model…
My facial landmark model has a different enumeration of the points so i guess at

NVIDIA-AI-IOT/deepstream_tao_apps/blob/73c5184b854fe6d076f652f64fd8ec984223b491/apps/tao_others/deepstream-emotion-app/emotion_impl/emotion_impl.cpp#L767

i need to do a translation of the mmpose face 2d enumeration to the Dlib 68 enumeration.

[Fiona.Chen]

This is just a sample for how to integrate the models together. You can customize as your requirement.