Classification model using TLT 3.0

rohitnairkp · June 14, 2021, 4:02am

Hi all,

I have trained a classification sample model using the steps provided in the TLT Jupyter Notebook → TLT Quick Start Guide — Transfer Learning Toolkit 3.0 documentation

The model training is complete and I have generated and exported the model which creates the following model files,

I want to know how to run this model as a secondary classifier for testing it out. My machine successfully runs the deepstream app for classification of vehicle types and make. I used the same config and changed the model paths and tried running it and it results in an error.

Here is my config files and app file,

################################################################################
# Copyright (c) 2020, NVIDIA CORPORATION. All rights reserved.
#
# Permission is hereby granted, free of charge, to any person obtaining a
# copy of this software and associated documentation files (the "Software"),
# to deal in the Software without restriction, including without limitation
# the rights to use, copy, modify, merge, publish, distribute, sublicense,
# and/or sell copies of the Software, and to permit persons to whom the
# Software is furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
# DEALINGS IN THE SOFTWARE.
################################################################################

[property]
gpu-id=0
net-scale-factor=1
offsets=124;117;104
tlt-model-key=nvidia_tlt
tlt-encoded-model=/home/glueck/Downloads/Classification/final_model.etlt
labelfile-path=/home/glueck/Downloads/Classification/labels.txt
model-engine-file=/home/glueck/Downloads/Classification/classification.engine
input-dims=3;224;224;0
uff-input-blob-name=input_1
batch-size=4
process-mode=2
model-color-format=0
operate-on-gie-id=1
network-mode=2
network-type=1
num-detected-classes=37
interval=0
gie-unique-id=6
output-blob-names=predictions/Softmax
classifier-threshold=0.2


################################################################################
# Copyright (c) 2020, NVIDIA CORPORATION. All rights reserved.
#
# Permission is hereby granted, free of charge, to any person obtaining a
# copy of this software and associated documentation files (the "Software"),
# to deal in the Software without restriction, including without limitation
# the rights to use, copy, modify, merge, publish, distribute, sublicense,
# and/or sell copies of the Software, and to permit persons to whom the
# Software is furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
# DEALINGS IN THE SOFTWARE.
################################################################################

[application]
enable-perf-measurement=1
perf-measurement-interval-sec=1

[tiled-display]
enable=1
rows=1
columns=1
width=1280
height=720
gpu-id=0

[source0]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=3
num-sources=1
#uri=file://../../streams/sample_1080p_h265.mp4
uri=file://../../streams/test/Glueck/2021-05-20-150357 (convert-video-online.com).mp4
#uri=rtsp://root:Glueck321@10.0.1.36/axis-media/media.amp?streamprofile=H264
#drop-frame-interval=2
gpu-id=0

[streammux]
gpu-id=0
batch-size=1
batched-push-timeout=40000
## Set muxer output width and height
width=1920
height=1080

[sink0]
enable=1
#Type - 1=FakeSink 2=EglSink 3=File
type=2
sync=0
source-id=0
gpu-id=0

[osd]
enable=1
gpu-id=0
border-width=3
text-size=15
text-color=1;1;1;1;
text-bg-color=0.3;0.3;0.3;1
font=Arial

[primary-gie]
enable=1
gpu-id=0
# Modify as necessary
model-engine-file=../../models/Primary_FaceDetector/resnet18_detector.etlt_b1_gpu0_fp16.engine
batch-size=1
#Required by the app for OSD, not a plugin property
bbox-border-color0=1;0;0;1
bbox-border-color1=0;1;1;1
bbox-border-color2=0;0;1;1
bbox-border-color3=0;1;0;1
gie-unique-id=1
config-file=config_infer_primary_facedetectir.txt

[sink1]
enable=0
type=3
#1=mp4 2=mkv
container=1
#1=h264 2=h265 3=mpeg4
codec=1
#encoder type 0=Hardware 1=Software
enc-type=0
sync=0
bitrate=2000000
#H264 Profile - 0=Baseline 2=Main 4=High
#H265 Profile - 0=Main 1=Main10
profile=0
output-file=out.mp4
source-id=0

[sink2]
enable=0
#Type - 1=FakeSink 2=EglSink 3=File 4=RTSPStreaming 5=Overlay
type=4
#1=h264 2=h265
codec=1
#encoder type 0=Hardware 1=Software
enc-type=0
sync=0
bitrate=4000000
#H264 Profile - 0=Baseline 2=Main 4=High
#H265 Profile - 0=Main 1=Main10
profile=0
# set below properties in case of RTSPStreaming
rtsp-port=8554
udp-port=5400

[tracker]
enable=0
tracker-width=640
tracker-height=384
#ll-lib-file=/opt/nvidia/deepstream/deepstream-5.1/lib/libnvds_mot_iou.so
ll-lib-file=/opt/nvidia/deepstream/deepstream-5.1/lib/libnvds_nvdcf.so
#ll-lib-file=/opt/nvidia/deepstream/deepstream-5.1/lib/libnvds_mot_klt.so
#ll-config-file required for DCF/IOU only
ll-config-file=../deepstream-app/tracker_config.yml
#ll-config-file=iou_config.txt
gpu-id=0
#enable-batch-process applicable to DCF only
enable-batch-process=1

[secondary-gie0]
enable=0
#model-engine-file=../../models/Secondary_Gender/gender11.caffemodel_b16_gpu0_fp16.engine
gpu-id=0
batch-size=16
gie-unique-id=2
operate-on-gie-id=1
operate-on-class-ids=0;
config-file=config_infer_gender_classifier.txt

[secondary-gie1]
enable=0
#model-engine-file=../../models/Secondary_Age/age.model_b16_gpu0_fp16.engine
gpu-id=0
batch-size=16
gie-unique-id=3
operate-on-gie-id=1
operate-on-class-ids=0;
config-file=config_infer_age_classifier.txt

[secondary-gie2]
enable=0
model-engine-file=../../models/Secondary_Emotion/emotion_2convs_iter_3000.caffemodel_b16_gpu0_fp16.engine
gpu-id=0
batch-size=16
gie-unique-id=4
operate-on-gie-id=1
operate-on-class-ids=0;
config-file=config_infer_emotion_classifier.txt

[secondary-gie3]
enable=0
model-engine-file=../../models/Secondary_Ethnicity/ethnicity_generic.caffemodel_b16_gpu0_fp16.engine
gpu-id=0
batch-size=16
gie-unique-id=5
operate-on-gie-id=1
operate-on-class-ids=0;
config-file=config_infer_ethnicity_classifier.txt

[secondary-gie4]
enable=0
model-engine-file=../../models/Secondary_FaceMask/caffe_model_1_iter_100000.caffemodel_b16_gpu0_fp16.engine
gpu-id=0
batch-size=16
gie-unique-id=6
operate-on-gie-id=1
operate-on-class-ids=0;
config-file=config_infer_mask_classifier.txt

[secondary-gie5]
enable=1
model-engine-file=/home/glueck/Downloads/Classification/classification.engine
gpu-id=0
batch-size=16
gie-unique-id=6
operate-on-gie-id=1
operate-on-class-ids=0;
config-file=config_classificationtest.txt

[tests]
file-loop=1

I have added the current trained classification model as [secondary-gie5] to the working app config file which was created and tested. The test video I added as input consists of the classes required for the model like car,boat,people etc.

Please help and let me know where I am going wrong.

Morganh · June 14, 2021, 6:38am

What is the error? Please share full command line along with the full log.

rohitnairkp · June 14, 2021, 7:31am

Hi @Morganh,

There was some path issue which caused the issue now I set the paths correctly the app works.

The primary detector here is Face
The secondary detector is the sample model from classification jupyter notebook. I am planning to train this classification model for gender and age, before that tried to run the sample model training and now want to test that.

I have added the trained model to the exisiting primary face detector to check if it working. There is no label change in the case (attached video) and is there any other code to check this classification model after training if we are training for custom classes.

Labels file is as follows:

boat;motorbike;sofa;chair;bicycle;train;pottedplant;horse;cat;bottle;dog;cow;diningtable;tvmonitor;sheep;car;person;aeroplane;bus;bird

Morganh · June 14, 2021, 4:02pm

The label is in alphabetic order.

Refer to Image Classification — Transfer Learning Toolkit 3.0 documentation

Topic		Replies	Views
Issue with image classification tutorial and testing with deepstream-app TAO Toolkit tensorrt , jetson-inference	34	5788	October 12, 2021
Deepstream: create custom age gender model with TLT TAO Toolkit jetson-inference	14	3107	October 12, 2021
FaceDetect IR Training using TLT 3.0 and Custom Dataset TAO Toolkit tensorrt , ai-training , deep-learning	13	1602	October 12, 2021
[Secondary GIE] Custom Classifier in sgie outputs only random entry in label.txt DeepStream SDK	30	2605	September 4, 2021
TLT V2.0 Classification TAO Toolkit	26	2787	August 3, 2021
Error on tags while running deepstream-app with a classifier as primary-gie TAO Toolkit	13	1359	October 12, 2021
How to load existing caffe model (Gender) as a secondary classifier in deepstream DeepStream SDK tensorrt , cuda , ubuntu , python	20	1907	October 12, 2021
Integrating the classification model with deepstream (6.0). No classes getting detected TAO Toolkit deepstream	2	763	September 30, 2022
How to use a classification video as primarye gieon deepstream-app DeepStream SDK	2	451	October 12, 2021
Integrating Tao Models (detectnet_v2) into Deepstream SDK TAO Toolkit tao , deepstream , jetson-nano	11	974	March 24, 2023

Classification model using TLT 3.0

Related topics