ONNX and ETLT model integration in one of the branch in deepstream parallel inference app

[pradyyadav]

Setup information

• Hardware Platform: GPU
• DeepStream Version: 6.2
• TensorRT Version: 8.5.2.2
• NVIDIA GPU Driver Version: 525.85.12
• Issue Type: Question

Hi,
I want to develop a video analytics solution using deepstream parallel inference app as shown in the deepstream parallel inference app repo.

I want to develop a single branch for which I have a trained ONNX model file, but I am unsure how to integrate it with the pipeline.

I could only figure out that one needs to change the config file path in case of sample use-cases provided with the deepstream-app.

pgie.set_property('config-file-path', './pgie_config_peoplenet.txt')

I believe one needs to create these config files from custom models to build it. Let me know if I am right here.

Thus I want a clear explanation what is the required work flow if one has ONNX model file. Also it would be great if you can mention a similar process for etlt model file.

Regards,
Pradyumna

[yuweiw]

We have many demos for ONNX model. You can refer to the link below: deepstream_tao_apps yolo_deepstream.

[pradyyadav]

Thanks for the reply. I found Integrating classification model in deepstream, which seems to be somewhat similar to what I want, I have trained the classification model using TAO Toolkit, I got the following things as output, nvinfer_config.txt, an etlt model file , and a labels.txt, the doc I shared above says I need to modify a config_infer_*.txt, I am unable to find it.

[yuweiw]

The config_infer_*.txt is nvinfer_config.txt. it’s used for configuring the nvinfer element in DeepStream, like pgie_yolov3_tao_config.txt.

[pradyyadav]

This is the content in the generated nvinfer_config.txt

net-scale-factor=1.0
offsets=103.939;116.779;123.68
infer-dims=3;224;224
tlt-model-key=nvidia_tlt
network-type=1
num-detected-classes=2
uff-input-order=0
output-blob-names=predictions/Softmax
uff-input-blob-name=input_1
model-color-format=1
maintain-aspect-ratio=0
output-tensor-meta=0

and I can see most of the config files look like:

[property]
gpu-id=0
# preprocessing parameters
net-scale-factor=1.0
offsets=103.939;116.779;123.68
model-color-format=1
batch-size=30

# Model specific paths. These need to be updated for every classification model.
int8-calib-file=<Path to optional INT8 calibration cache>
labelfile-path=<Path to classification_labels.txt>
tlt-encoded-model=<Path to Classification etlt model>
tlt-model-key=<Key to decrypt model>
infer-dims=c;h;w # where c = number of channels, h = height of the model input, w = width of model input
uff-input-blob-name=input_1
uff-input-order=0
output-blob-names=predictions/Softmax

## 0=FP32, 1=INT8, 2=FP16 mode
network-mode=0
# process-mode: 2 - inferences on crops from primary detector, 1 - inferences on whole frame
process-mode=2
interval=0
network-type=1 # defines that the model is a classifier.
gie-unique-id=1
classifier-threshold=0.2

Please observe the fields from the sample config and compare. I feel I need to change the common fields, please correct me if I am wrong.

Please note: Since my trained model is a classification model, so I am expecting my nvinfer_config.txt should have all the important fields present in config_infer_secondary_*.txt.

[yuweiw]

At present, it seems that only some parameter configurations are automatically generated, and you need to add other configuration items according to your own environment. You can refer to the link below to check all the parameters: https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvinfer.html#id2

[pradyyadav]

Thanks, @yuweiw; I can now integrate TAO-trained ETLT models (I will focus on ONNX model file integration, too. Hence I require this query to be open till I complete it). I am attaching my config file here for community reference.

[property]
gpu-id=0
# preprocessing parameters
net-scale-factor=1.0
offsets=103.939;116.779;123.68
model-color-format=1
batch-size=30
maintain-aspect-ratio=0

# Model specific paths. These need to be updated for every classification model.
# int8-calib-file=<Path to optional INT8 calibration cache>
labelfile-path=<Path to labels.txt>
tlt-encoded-model=<.etlt model file>
tlt-model-key=<KEY> ## automatically generated after TAO training.
## input-dims=3;224;224;0
infer-dims=3;224;224  
uff-input-blob-name=input_1
uff-input-order=0
output-blob-names=predictions/Softmax
num-detected-classes=2

## 0=FP32, 1=INT8, 2=FP16 mode
network-mode=0

process-mode=2  # Mode (primary or secondary) in which the element is to operate on 
network-type=1
gie-unique-id=5            # mention this ID manually, 
operate-on-gie-id=1    # Unique ID of the GIE on whose metadata (bounding boxes) this GIE is to operate on
classifier-threshold=0.2  
operate-on-class-ids=0    # Class IDs of the parent GIE on which this GIE is to operate on

As mentioned in the above answers, the content in nvinfer_config.txt is required to write your own config file

PS: gie ids were of great help for my case as I utilized a primary model (detection) along with my secondary (classification).

[yuweiw]

Glad to hear that. When you meet new problems, we suggest that you create a new topic, which is more convenient for others to refer to.