TAO Classifier predicting only one label when deployed to DS

Accelerated Computing Intelligent Video Analytics TAO Toolkit
1

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU)
GPU
• DeepStream Version
6.1
• JetPack Version (valid for Jetson only)
• TensorRT Version
8.2.5.1
• NVIDIA GPU Driver Version (valid for GPU only)
512.95
• Issue Type( questions, new requirements, bugs)
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

Hi,

I have trained a classifier model (darknet53 backbone) with the newest version of TAO (3.22.05), to classify detections from PeopleNet (v2.6) into gender (Female/Male). The classifier works as expected when running inference on some test data with tao classifier inference.

However, when the model is exported as .etlt format, and deployed to DS, the outputs are completely broken with only one label getting predicted with 100% confidence;

The model is deployed with the nvinfer_config.txt file generated from the --gen_ds_config flag in the tao classification export command as below:

[property]
net-scale-factor=1.0
batch-size=1
offsets=103.939;116.779;123.68
infer-dims=3;224;224
tlt-encoded-model=gender.etlt
tlt-model-key=06052019
labelfile-path=gender_label.txt
network-type=1
num-detected-classes=2
uff-input-order=0
output-blob-names=predictions/Softmax
uff-input-blob-name=input_1
model-color-format=1
maintain-aspect-ratio=0
output-tensor-meta=0
process-mode=2
network-mode=0
gie-unique-id=3
operate-on-gie-id=1
classifier-threshold=0.0

When inspecting the raw output tensor by setting output-tensor-meta=1 the results are the same, with only one label being predicted by 100%.

Model .etlt file at the following link: https://transfer.sh/(/ELcTKy/gender.etlt).zip
Config and label file attached.
gender_config.txt (461 Bytes)
gender_labels.txt (12 Bytes)

Any help is greatly appreciated,

/M

5

Moving to TAO forum.

6

Please refer to Issue with image classification tutorial and testing with deepstream-app - #21 by Morganh

7

Hi @Morganh

Changing the scaling-filter or flipping from BGR to RGB with offsets and input color or vica versa unfortunately didn’t help. Do you have any other suggestions?

Thanks in advance,

/M

8

From the link I shared, actually the classification model is working as primary trt engine.
How about you? Did you set it as primary trt engine directly?

9

The classifier works as SGIE on bounding boxes produced by PeopleNet PGIE. Setting the classifier as PGIE does not make sense to me since it will then do inference on the entire frame instead of just the bounding boxes?

/M

10

Got it. And actually I also provide the way in Issue with image classification tutorial and testing with deepstream-app - #12 by Morganh when work as secondary trt engine. Please check.
And please modify to
offsets=103.939;116.779;123.68
model-color-format=1

11

Hi @Morganh,

I’ve changed the network input dimensions to 3;256;128 in the training phase (instead of the default 3;224;224) - the model is now producing correct results when deployed to DS.

I’m using custom image means when training the model since our images are quite “dark” - should I use these means as offsets when deploying to DS as well or the default offsets=103.939;116.779;123.68 as suggested by the resulting nvinfer_config.txt ?

Thanks in advance,

/M

12

OK, how did you train your model? You set above in input_image_size,right?

Yes.

13

Correct,

experiment settings as below:

…
input_image_size: “3,256,128”
image_mean {
key: ‘b’
value: 81.94
}
image_mean {
key: ‘g’
value: 80.46
}
image_mean {
key: ‘r’
value: 82.85
}
…

I will change the nvinfer_config.txt from

offsets=103.939;116.779;123.68

to

offsets=81.94;80.46;82.85

Correct?

Thanks,

/M

14

Yes.

15

Perfect - appreciate the swift response.

Perhaps for future releases of TAO you can consider if the generated nvinfer_config.txt file when exporting should be generated with the custom image means used in the spec file (currently it always outputs offsets=103.939;116.779;123.68)

Thanks,

/M

16

Currently, there is “--gen_ds_config” option when exporting. See Image Classification (TF1) - NVIDIA Docs