How to set the config file of sgie of multi label classifier?

**• Hardware Platform : GPU
**• DeepStream Version: deepstream 4.0.1

Hi, here is my config file of sgie of multi label classifier, which works for deepstream 3.0:

###################
[property]
gpu-id=0
net-scale-factor=1
#model-engine-file=resnet18_alldatasets_50000.caffemodel_b1_fp32.engine

model-file=sgie1_person_marketnx_29884_it100000_190618_20190825.caffemodel
proto-file=deploy.prototxt

model-color-format=1
batch-size=10
network-mode=0

gie-unique-id=2
operate-on-gie-id=1
operate-on-class-ids=2
is-classifier=1
offsets=128;128;128

input-object-min-width=64
input-object-min-height=64

output-blob-names=pro_age;pro_backpack;pro_bag;pro_handbag;pro_clothes;pro_down;pro_up;pro_hair;pro_hat;pro_gender;pro_upcolor;pro_downcolor;pro_suitcase;pro_scarf;pro_sunglasses;pro_mask;pro_umbrella;pro_carrybox;pro_carrykid;pro_direction

classifier-async-mode=0
classifier-threshold=0.00001
interval=25
gie-mode=2

###################

Now, we want to migrate to deepstream 4.0, but when done, it just didn’t work,
no classification result was acquired. when we change the sgie to single label classifier, it worked well. and i find in our config file of sgie of multi label classifier, there set no label file.

could anyone give me some advices? Thanks!

Hi @489536658,
The latest DS is DS5.0, can you upgrade to DS5.0 directly instead of DS4.0?

operate-on-class-ids=2 ==> indicates this classifier only operate on the 2nd class, so if you want it to do multi label, you can change it to multiple classes, e.g. operate-on-class-ids=1;2,

refer to doc - https://docs.nvidia.com/metropolis/deepstream/dev-guide/index.html#page/DeepStream%20Plugins%20Development%20Guide/deepstream_plugin_details.html#wwpID0E0YFB0HA

Hi, mchi, thanks for your reply!
I’m sorry for that i didn’t make it clear, the multi label classifier here means the sgie can classify the detecte objects (person or car) into multi labels.
for example, if a sgie can recognize the car’s type, color, maker and other labels simultaneously, we call it a multi label classifier.

In our procedure, we set operate-on-class-ids=2, because our sgie of multi label classifier operates on the detected object of person, and can recognize the person’s hair type, age, sex and so on.

Ok, how about if you give it label file?

Hi, mchi, i didn’t give it a try, because the label file’s format of a multi label classifier is unknown, and i found the labelfile-path is a optional properties in config file .

“”"
Other optional properties:
net-scale-factor(Default=1), network-mode(Default=0 i.e FP32),
model-color-format(Default=0 i.e. RGB) model-engine-file, labelfile-path,
mean-file, gie-unique-id(Default=0), offsets, gie-mode (Default=1 i.e. primary),
custom-lib-path, network-mode(Default=0 i.e FP32)
“”"

Hi, mchi, have you ever been use a sgie of multi label classifier? could you send me an example config file?

Hi @489536658,
Sorry for late! I tried to find out a multi label classifier sgie sample, but I can’t find it.
i think the best way is to look into the post-processing code of your multi label classifier sgie to find out the root cause.
I think you should use DeepStream post-processing code, which have been open source.

Thanks!

Thank you, mchi, could you tell me where to find a deepstream post-processing code exmaple?

for exmaple, the function defined by CHECK_CUSTOM_PARSE_FUNC_PROTOTYPE() marco in https://github.com/NVIDIA-AI-IOT/deepstream_tlt_apps or DeepStream package

Hi @489536658,
Did you find the root cause?
Please take a look /opt/nvidia/deepstream/deepstream-5.0/sources/apps/sample_apps/deepstream-test2 sample, are the sgies under the sample multi label classifier you mentioned above?

Thanks!

Hi, mchi, i still didn’t find the root cause. The /opt/nvidia/deepstream/deepstream-5.0/sources/apps/sample_apps/deepstream-test2 sample is cascade of sgie of single label classifier, not sgie of multi label classifier.

we set the sgie1 in /opt/nvidia/deepstream/deepstream-5.0/sources/apps/sample_apps/deepstream-test2 sample with our multi label classifier, and acquired following debug info, the line 25 to line 50 is about our multi label classifier.


log.txt (4.3 KB)

mchi, could you get some useful infomation from the debug text?

Hi @489536658
In deepstream-test2, the network of sgie1 - samples/models/Secondary_CarColor/resnet18.prototxt is the similar as your network, the output layer is softmax, and it can detect different car colors - black;blue;brown;gold;green;grey;maroon;orange;red;silver;white;yellow as you can find in samples/models/Secondary_CarColor/labels.txt.

it has the similar log as yours :

deserialized trt engine from :/opt/nvidia/deepstream/deepstream-5.0/samples/models/Secondary_CarColor/resnet18.caffemodel_b16_gpu0_int8.engine
INFO: [Implicit Engine Info]: layers num: 2
0   INPUT  kFLOAT input_1         3x224x224
1   OUTPUT kFLOAT predictions/Softmax 12x1x1

So, could you enlighten me why you said it’s not multi label classifier?

Hi, mchi, here we named the “color” as label, while named the “black” or others as attribute value, so the model samples/models/Secondary_CarColor is single label classifier.

however, our multi label classifier as described in the following picture from line 25 to line 50, has one input and 20 output, general single label classifie only has one input and one output. our multi label classifier has the labels “age”(attribue values: young/middle/old), “hair type”(attribute values: short/long/no hair), “gender”(attribute values: male/female), “dress color” (attribute values: black/red/green/blue) and so on.

Ok, so I think you need customize your own output parsing.

refer to https://docs.nvidia.com/metropolis/deepstream/dev-guide/index.html#page/DeepStream_Development_Guide/deepstream_custom_model.html#wwpID0E0RB0HA

Hi, mchi, i’m just confused about that why the multi label classifier works fine in deepstream 3.0?
And i think deepstream 4.0 and 5.0 also support multi label classifier like deepstream 3.0 as i find this in deepstream 4.0 and 5.0 header file

as in deepstream 3.0,

Hi @489536658,
I’m not sure. As below function in sources/libs/nvdsinfer/nvdsinfer_context_impl.cpp, I hink you could add some prints in the ClassifyPostprocessor to check if the postprocessor process the output corectly.

NvDsInferStatus
NvDsInferContextImpl::preparePostprocess(const NvDsInferContextInitParams& initParams)
{
    assert(!m_Postprocessor);
    std::unique_ptr<InferPostprocessor> processor;
    switch (initParams.networkType)
    {
        case NvDsInferNetworkType_Detector:
            processor = std::make_unique<DetectPostprocessor>(m_UniqueID, m_GpuID);
            break;
        case NvDsInferNetworkType_Classifier:
            processor =
                std::make_unique<ClassifyPostprocessor>(m_UniqueID, m_GpuID);
            break;
        case NvDsInferNetworkType_Segmentation:
            processor = std::make_unique<SegmentPostprocessor>(m_UniqueID, m_GpuID);
            break;
   ...
}

Ok, thanks, mchi

Hi, mchi, can i send my procedure and models to you for debug?

Hi @489536658,
Thanks for sharing! You could put it somewhere and share the link to me by mail.
If it’s a DS issue, we will fix it, otherwise, we will try to help you to solve it.

Hi, mchi, our test procedure is based on deepstream-5.0/sources/apps/sample_apps/deepstream-test2, and we replaced the first sgie “Secondary_CarColor” with our sgie of multi label classifier, the model files are placed in deepstream-multilabelSgieTest_1501/sgie1_person, the config file is deepstream-multilabelSgieTest_1501/config_infer_secondary_person.txt, the test video is deepstream-5.0/samples/streams/sample_720p.h264.

The procedure link:

Thanks for your help!