Face alignment using Facial Landmarks

Accelerated Computing Intelligent Video Analytics DeepStream SDK
, , ,
1

I am working on a face recognition project by setting this pipeline:
face detection → face alignment → face recognition

And for this i am using jetson agx orin

Currently, I am facing issues related to face alignment, as for face detection i am using an open source insightface SCRFD model and it is working as pgie by giving the bounding boxes now i want to implement another model that provides facial landmark points for face alignment and for that i am using the following model 2d106det from the following repo:

The model works fine in onnxruntime so i want to know how to implement this in deepstream sdk-7.0 because as i think only detection classification and segmentation based task are possible till now so i just wanted to know that whether facial landmarks are possible or not?

for example here is the image of landmarks

WhatsApp Image 2024-07-29 at 11.24.51 AM
WhatsApp Image 2024-07-29 at 11.24.51 AM850×686 52.4 KB

and here is the implementation output

test_out (2)
test_out (2)736×736 57.3 KB

3

Please refer to deepstream_tao_apps/apps/tao_others/deepstream-emotion-app at master · NVIDIA-AI-IOT/deepstream_tao_apps (github.com)

4

but i want to do the inference on my jetson hardware only and not using TAO so is it possible or not?

5

This is the sample code to show how to customize the no image input for some models. DeepStream is a SDK, there are many samples to show how to use the DeepStream APIs.

You can customize your own application according to your own model.

6

okay thank you, can you provide me some example link for landmarks using deepstream SDK only and not TAO

7

The link deepstream_tao_apps/apps/tao_others/deepstream-emotion-app at master · NVIDIA-AI-IOT/deepstream_tao_apps (github.com) is the DeepStream SDK sample. It uses the TAO pretrained models, it uses no TAO code. If you have read the document and the code, you will know it.

8

Thanks for the help, i tried the given sample application of facial landmarks and it worked but the results were not accurate like the landmarks were spilled on the whole face randomly, and also i want to implement my custom facial landmark model which works on whole input frame so can you tell me how can i do it or like can you share any resources?

9

The sample code has shown all key APIs and flow for the customization. The landmarks generated by the faciallandmakrs model are transferred to the emotion model by the customized object user meta deepstream_tao_apps/apps/tao_others/deepstream-faciallandmark-app/ds_facialmark_meta.h at master · NVIDIA-AI-IOT/deepstream_tao_apps (github.com). Please read the code carefully.

10

For model training, please raise topic in TAO forum. Latest Intelligent Video Analytics/TAO Toolkit topics - NVIDIA Developer Forums

11

i don’t want to train any model i have a pre trained model and i want to use it for facial landmarks application so and the main issue is that my model works on full frame so it will not work on the meta data generated by the face detection models

12

Do you mean your model works on the whole frame which contains multiple faces and the model can output all the faciallandmarks for all faces in the frame?

13

No my face detection model takes full frame as input and then the landmark model takes those faces as input and then provides landmarks on all the faces for each frame…and that’s what i want to do using deepstream sdk…so tell me how should i configure the config files or should i do something else

14

The sample deepstream_tao_apps/apps/tao_others/deepstream-faciallandmark-app at master · NVIDIA-AI-IOT/deepstream_tao_apps (github.com) uses the FaceNet model to take the faces out, and then use the Faciallandmarks model to inference landmarks from the faces.

15

okay i will go through the TAO sample code and let you know if i have any questions

16

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.