Running a muli head output secondary classifier model in Deepstream

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Please provide complete information as applicable to your setup.

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) GPU
• DeepStream Version 6.3
• NVIDIA GPU Driver Version (valid for GPU only) 535
• Issue Type( questions, new requirements, bugs) question

I am trying to run a secondary classifier in DeepStream with multiple output heads (age and gender). Below is my configuration:

[property]  
gpu-id=0  
model-engine-file=age_gender.trt  
labelfile-path=class_labels.txt  
batch-size=16  
network-type=1  # Classifier  
process-mode=2  # Secondary GIE  
gie-unique-id=2  
output-blob-names=age_output;gender_output  
input-object-min-width=10  
input-object-min-height=10  

#[class-attrs-all]  
#threshold=0.5

I followed this discussion: How to use metadata from the secondary classifier in Python?.

However, in my implementation, class_meta = pyds.NvDsClassifierMeta.cast(class_obj.data) returns None.
Is there something I might be missing in my configuration or metadata parsing?

3

what is the whole media pipeline? please refer to deepstream_test_2.py for how to run secondary classifier in DeepStream.

4

But those are single cascaded models, i am talking about single classifier models with multiple output head

5

i wrote a custom parser and it worked for me

#include <cstring>
#include <iostream>
#include "nvdsinfer_custom_impl.h"

// C-linkage to prevent name-mangling
extern "C"
bool NvDsInferClassiferParseCustomSoftmax(
    std::vector<NvDsInferLayerInfo> const& outputLayersInfo,
    NvDsInferNetworkInfo const& networkInfo,
    float classifierThreshold,
    std::vector<NvDsInferAttribute>& attrList,
    std::string& descString);

// Custom parser function
extern "C"
bool NvDsInferClassiferParseCustomSoftmax(
    std::vector<NvDsInferLayerInfo> const& outputLayersInfo,
    NvDsInferNetworkInfo const& networkInfo,
    float classifierThreshold,
    std::vector<NvDsInferAttribute>& attrList,
    std::string& descString)
{
    // Print the number of output layers
    //std::cout << "Number of output layers: " << outputLayersInfo.size() << std::endl;

    // Loop through each output layer
    for (size_t i = 0; i < outputLayersInfo.size(); i++) {
        const NvDsInferLayerInfo& layerInfo = outputLayersInfo[i];
        float* outputBuffer = (float*)layerInfo.buffer;

        // Get dimensions of the output layer
        NvDsInferDimsCHW dims;
        getDimsCHWFromDims(dims, layerInfo.inferDims);
        unsigned int numClasses = dims.c;

        // Find the class with the maximum probability
        float maxProb = 0;
        int maxClass = -1;

        for (unsigned int c = 0; c < numClasses; c++) {
            //std::cout << "attribute " << i<< " index " << c <<" prob " << outputBuffer[c]<< std::endl;
            if (outputBuffer[c] > maxProb) {
                maxProb = outputBuffer[c];
                maxClass = c;
            }
        }

        // If a valid class is found, add it to the attribute list
        if (maxClass != -1) {
            NvDsInferAttribute attr;
            attr.attributeIndex = i;  // Use the output layer index as attributeIndex
            attr.attributeValue = maxClass;  // Class index
            attr.attributeConfidence = maxProb;  // Confidence score
            attr.attributeLabel = nullptr;  // No label, just index
            attrList.push_back(attr);

            // Append to the description string
            descString.append("Output Layer ").append(std::to_string(i))
                      .append(": Class ").append(std::to_string(maxClass))
                      .append(" (").append(std::to_string(maxProb)).append(") ");
        }
    }

    return true;
}

// Check that the custom function has been defined correctly
CHECK_CUSTOM_CLASSIFIER_PARSE_FUNC_PROTOTYPE(NvDsInferClassiferParseCustomSoftmax);
6

there is a dissimilarity in the results from the pytorch model and model deployed in deepstream. In order to debug, I decided to dump the output tensor meta.

output-tensor-meta=1

but tensor meta is not generated for all the objects. Do I have to add something in the configuration file to get output tensor for all the frame?

frame_4.json:

{
    "objects": [
        {
            "obj_id": 8,
            "y": 0.2782819324069553,
            "x": 0.3682586193084717,
            "w": 0.06006125211715698,
            "h": 0.2447621875339084,
            "class_id": "customer",
            "trackingId": 8,
            "confidence": 0.9627491235733032,
            "attributes": {
                "gender": {
                    "label": "kids",
                    "confidence": 0.9992684721946716
                },
                "tensor": [
                    0.00047103501856327057,
                    0.0002605192130431533,
                    0.9992684721946716,
                    0.0,
                    0.0
                ]
            }
        },
        {
            "obj_id": 7,
            "y": 0.8316291809082031,
            "x": 0.8612545967102051,
            "w": 0.09726865887641907,
            "h": 0.16837081909179688,
            "class_id": "customer",
            "trackingId": 7,
            "confidence": 0.6339417695999146,
            "attributes": {
                "gender": {
                    "label": "male",
                    "confidence": 0.9948983788490295
                }
            }
        },
        {
            "obj_id": 6,
            "y": 0.08715483347574869,
            "x": 0.6873117923736572,
            "w": 0.012521107494831086,
            "h": 0.08300283749898275,
            "class_id": "customer",
            "trackingId": 6,
            "confidence": 0.2878338694572449,
            "attributes": {
                "gender": {
                    "label": "kids",
                    "confidence": 0.999718964099884
                }
            }
        },
        {
            "obj_id": 5,
            "y": 0.2864137437608507,
            "x": 0.3040372371673584,
            "w": 0.0571000337600708,
            "h": 0.243203247918023,
            "class_id": "staff",
            "trackingId": 5,
            "confidence": 0.9287275671958923,
            "attributes": {
                "gender": {
                    "label": "kids",
                    "confidence": 0.9995505213737488
                }
            }
        }
    ],
    "time": "2025-03-12 06:32:26",
    "height": 720,
    "width": 1280,
    "source_time": "2025-03-12 06:32:26.409714"
}

                        if user_meta and user_meta.base_meta.meta_type == pyds.NvDsMetaType.NVDSINFER_TENSOR_OUTPUT_META:
                            try:
                                tensor_meta = pyds.NvDsInferTensorMeta.cast(user_meta.user_meta_data)
                            except StopIteration:
                                break
                            layer = pyds.get_nvds_LayerInfo(tensor_meta, 0)
                            output = []
                            for i in range(5):
                                output.append(pyds.get_detections(layer.buffer, i))
                            res = np.array(output)
                            obj_attributes['tensor'] = res.tolist()
7

output-tensor-meta is used to add inference results to meta data. please refer to the sample for how to use NvDsInferTensorMeta. do you want add output tensor for sending message broker? how did you set msg2p-newapi?

8

following this link https://forums.developer.nvidia.com/t/how-to-use-metadata-from-the-secondary-classifier-in-python/199679/6 I tried to extract the data from classifier meta for a single head classification model (without any custom parser). But the model outputs only object of class at first index.

                while l_obj is not None:
                    try:
                        obj_meta = pyds.NvDsObjectMeta.cast(l_obj.data)
                    except StopIteration:
                        continue

                    classifier_meta_list = obj_meta.classifier_meta_list
                    while classifier_meta_list is not None:
                        classifier_meta = pyds.NvDsClassifierMeta.cast(classifier_meta_list.data)
                        label_info_list = classifier_meta.label_info_list

                        while label_info_list is not None:
                            label_info = pyds.NvDsLabelInfo.cast(label_info_list.data)
                            print({"label":label_info.result_label , "confidence": label_info.result_prob})

                            label_info_list = label_info_list.next
                        classifier_meta_list = classifier_meta_list.next

config file

[property]
gpu-id=0
model-engine-file=GenderMFK0903.trt
labelfile-path=people_attribute/labels.txt
batch-size=16
network-type=1  # classifier
process-mode=2    # Secondary GIE
model-color-format=0
gie-unique-id=2
output-blob-names=gender_output
operate-on-class-ids=0;1;2;4
infer-dims=3;288;288
#custom-lib-path=classifier_customparser/libnvds_infercustomparser.so
#parse-classifier-func-name=NvDsInferClassiferParseCustomSoftmax
net-scale-factor= 0.01735207357279195
offsets=123.675; 116.28; 103.53
#input-object-min-width=10
#input-object-min-height=10
#output-tensor-meta=1

#[class-attrs-all]
#threshold=0.5
9

i am just printing it out as python dict (to debug)

10

earlier in the labels.txt it was

men
women
kids

changing that to

men;women;kids

it worked

11

Thanks for the update! From the opensource code InferPostprocessor::parseLabelsFile, the label string should be splitted by “\n”. Is this still an DeepStream issue to support? Thanks!