DeepStream 8.0 SCRFD + ArcFace: How to Pass Facial Landmark Metadata for Warp Affine Before SGIE?

Accelerated Computing Intelligent Video Analytics DeepStream SDK
,
1

• Hardware Platform GPU
• DeepStream Version :- nvcr.io/nvidia/deepstream:8.0-gc-triton-devel
** Triton Inference Server Version: nvcr.io/nvidia/tritonserver:26.04-py3
• NVIDIA GPU Driver Version (valid for GPU only) 595.58.03
• Issue Type( questions, new requirements )

Pipeline Architecture:
nvstreammux → nvinferserver (PGIE: SCRFD) → nvtracker → nvinferserver (SGIE: ArcFace) → fakesink / Kafka Broker

Issue

I am building a high-performance face recognition pipeline using SCRFD + ArcFace.

SCRFD custom tensor parsing is working correctly through libnvds_infercustomparser.so , and facial landmarks are extracted successfully.

The current blocker is that I am unable to properly pass the 5-point facial keypoints downstream as NvDsUserMeta for preprocessing before the SGIE.

ArcFace requires a strictly aligned 112x112 face tensor generated using 5-point Warp Affine alignment, but I cannot find a proper DeepStream-native way to:

  • propagate landmark metadata safely
  • access it before SGIE
  • perform custom Warp Affine alignment
  • replace the default SGIE crop tensor with aligned input

Looking for the recommended DeepStream 8.0 approach for SCRFD + ArcFace pipelines using Triton.

3

DeepStream SDK provides general framework and interfaces to help the video inferencing. Some items you listed here are model related but not framework related.

Please refer to deepstream_tao_apps/apps/tao_others/deepstream-pose-classification at release/tao_ds8.0ga · NVIDIA-AI-IOT/deepstream_tao_apps. The sample shows how to transfer bodypose points with user meta and use the bodypose points to generate tensor needed by the pose classifier model.

This is a algorithm which is model related. You may refer to this sample for the affine algorithm. deepstream_tao_apps/apps/tao_others/deepstream_custom_preprocessing_app at release/tao_ds8.0ga · NVIDIA-AI-IOT/deepstream_tao_apps

Can you explain this? What do you mean by aligned input?

4

replace the default nvinfer preprocess to use custom nvpreprocess for crop using warpaffine method send sgie

5

The nvdspreprocess is a template for the customers to customize different preprocess algorithms. You need to consult the model author for the details of preprocess algorithm and implement the algorithm by yourself.

We have already provide samples for how to implement nvdspreprocess lib interfaces and even a sample with warp affine.

6

can you share the link of nvpreprocess lib interfaces and sample of warp affine

7

I’ve shared here