Multi-task Age/Gender Model - How to get both outputs in DeepStream?

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

• Hardware Platform (Jetson / GPU) GPU
• DeepStream Version 6.4
• Issue Type( questions, new requirements, bugs) question

Hello,

I’m working on integrating a multi-task (multi-head) model that needs to classify both age and gender simultaneously.

labels.txt file:

text

children
teenagers
adults
seniors
male
female

Problem:
Currently, DeepStream only detects the first output (age). The default parser seems to ignore the “gender” output.

Question:
Does DeepStream natively support multi-head models with multiple outputs? If not, what is the recommended approach to parse both outputs simultaneously?

3

Is this topic the same as Multi-Head Output Model in DeepStream Pipeline - Intelligent Video Analytics / DeepStream SDK - NVIDIA Developer Forums?

Let’s discuss in one topic if they are the same.

4

Yes, it is the same topic. Let’s discuss it here.

5

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU)
• DeepStream Version
• JetPack Version (valid for Jetson only)
• TensorRT Version
• NVIDIA GPU Driver Version (valid for GPU only)

6

• Hardware Platform ( GPU)
• DeepStream Version 6.4
• TensorRT Version 8.6.1
• NVIDIA GPU Driver Version (550.144.03)

7

I’m using a custom model with multiple output heads deepstream classifier parser doesn’t seem to handle this structure and fails to parse the results correctly.

8

From your configuration file

There are two output layers. Are they all softmax?

9

Can you post the labels_age_gender.txt file?

11

Please answer the question

12

this is the file :

labels_age_gender.txt (47 Bytes)

13

There are two output layers. Are they all softmax?

14

  1. We don’t know about your model, please check whether the output layers are softmax layers. If they are not softmax layers, you may need to customize the postprocessing function by yourself.

  2. Your label file is wrong. Please check with the function ClassifyPostprocessor::parseAttributesFromSoftmaxLayers() in /opt/nvidia/deepstream/deepstream/sources/libs/nvdsinfer/nvdsinfer_context_impl_output_parsing.cpp and the function InferPostprocessor::parseLabelsFile() in /opt/nvidia/deepstream/deepstream/sources/libs/nvdsinfer/nvdsinfer_context_impl.cpp

There is multi-task sample for DeepStream 6.4.

deepstream_tao_apps/configs/nvinfer/multi_task_tao/multi_task_labels.txt at release/tao5.1_ds6.4ga · NVIDIA-AI-IOT/deepstream_tao_apps

15

my output layers are not softmax layers - they are linear layers producing raw logits. I use CrossEntropyLoss and FocalLoss during training, which internally apply softmax.

16

You may need to customize the postprocessing function by yourself.

17

I have already added softmax to the output layers and modified the label file as follows:

enfants;adolescents;adultes;seniors
male;female

However, I’m now getting incorrect results where all age predictions are classified as either “enfants” or “adultes”.

My question: How can I determine if this is a model issue or a DeepStream mapping problem? Is there a way to debug how DeepStream maps the model outputs to the labels in the label.txt file?

18

For a multi-task model in DeepStream, does the parser:

Return all predictions with their probabilities for both tasks?

Or return only the top label with best score for each task?

19

How did you get the model? Normally the model will be provided with the training and inferencing tools by the author. If the model itself can work with the original tool, you need to check whether your configurations in DeepStream is aligned with the parameters used in training.

The gst-nvinfer is totally open source, you can debug with the source code. I’ve pointed out the related code in the previous post Multi-task Age/Gender Model - How to get both outputs in DeepStream? - #14 by Fiona.Chen.

20

Please check with your model provider for whether the model will output all probabilities for both types.

According to the information you provided in Multi-Head Output Model in DeepStream Pipeline - Intelligent Video Analytics / DeepStream SDK - NVIDIA Developer Forums, the layer order is different to the label order

The gender layer is before age layer as you configured with the nvinfer configuration, while you put the age labels before the gender labels in the label file. Please make sure they are provided by the same order. We don’t know anything about your model, please check the correct order by yourself.

21

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks.