Deepstream Onnx inference no output

Accelerated Computing Intelligent Video Analytics TAO Toolkit
1

• Hardware: ubuntu 20 x86 with RTX 3060
• Network Type: Classification tf1 (VehicleMakeNet - Resnet18)
• TLT Version: http://nvcr.io/nvidia/tao/tao-toolkit:5.0.0-tf1.15.5
• Training spec file:
retrain_car_make.txt (1.0 KB)

• How to reproduce the issue ?
I have a vehiclemakenet modified that classify 35 car brands. I have exported the model after trained in tao classification tf1:

# Generate .onnx file using tao container
!tao model classification_tf1 export \
            -m $USER_EXPERIMENT_DIR/retrain_pruned/weights/resnet_010.hdf5 \
            -o $USER_EXPERIMENT_DIR/export/final_model \
            -e $SPECS_DIR/retrain_car_make.cfg \
            --classmap_json $USER_EXPERIMENT_DIR/retrain_pruned/classmap.json \
            --gen_ds_config

I use this Onnx file to inference the net in Deepstream, but I do not have any output from this net. (If I use the catalog pretrained model it works, but I need more output classes). When deepstream generate the engine, I obtain that:

INFO: [FullDims Engine Info]: layers num: 2
0   INPUT  kFLOAT input_1         3x224x224       min: 1x3x224x224     opt: 2x3x224x224     Max: 2x3x224x224     
1   OUTPUT kFLOAT predictions     35              min: 0               opt: 0               Max: 0

But when I run the pretrained model I obtain that:

INFO: [FullDims Engine Info]: layers num: 2
0   INPUT  kFLOAT input_1         3x224x224   
1   OUTPUT kFLOAT predictions/Softmax    35x1x1

So, I do not get the softmax output in my model engine. This is my model:

final_model.onnx
final_model.onnx460×5476 355 KB

3

Hi @josemiad
After trained, may I know the tao model classification_tf1 inference xxx can run inference as expected? You can run inference against your training images or test images.

4

Yes, my inference works good:

!tao model classification_tf1 inference -e $SPECS_DIR/retrain_car_make.cfg \
                          -m $USER_EXPERIMENT_DIR/retrain_pruned/weights/resnet_$EPOCH.hdf5 \
                          -b 32 -d $DATA_DOWNLOAD_DIR/test_set \
                          -cm $USER_EXPERIMENT_DIR/retrain_pruned/classmap.json

and I get:

INFO: Processing /data/test_set...

INFO: Inference complete. Result is saved at /data/test_set/result.csv
Execution status: PASS
5

How about the result? Is the inference expected?

6

Yes, my inference results looks good. I think the problem occur when I use the TAO export.

7

After tao model classification_tf1 export, you will get the onnx file. To narrow down, please run tao deploy classification_tf1 gen_trt_engine xxx and then tao deploy classification_tf1 inference to check if it is expected.

8

I run:

!tao deploy classification_tf1 inference \
                            -m $USER_EXPERIMENT_DIR/export/final_model_trt.engine \
                            -r $USER_EXPERIMENT_DIR/results_trt \
                            -e $SPECS_DIR/retrain_car_make.cfg \
                            -i $DATA_DOWNLOAD_DIR/test_set \
                            -b 32 -c $USER_EXPERIMENT_DIR/retrain_pruned/classmap.json

and it went well.

It looks like when I load my engine in deepstream does not take the correct output node. It must be predictions/Softmax but only take predictions.

10

My results went so bad. When tao deploy generate the trt engine take the output layer as predictions too.

Output 'predictions' with shape (-1, 35) and dtype DataType.FLOAT
11

May I know if it is good or bad for tao deploy classification_tf1 inference?

12

The inference run good but my results were bad.

13

Do you mean tao deploy classification_tf1 inference is good but “your jetson deepstream inference” is bad?

14

No, I mean the inference looks like went good but when I check the results in the CSV are incorrect. (All in my ubuntu PC).

15

I think the error is there:
when I export the model it print the output node is Using output nodes: ['predictions/Softmax'] but when I generate the tensorRT engine I get Output 'predictions' with shape (-1, 35).

16

I used fp64 engine instead of int8 and now my inference results are well.

17

Is it fp32 tensorrt engine?

18

sorry, fp32 yes, it went good on my PC. I take this onnx to deepstream and I change the labels txt because the tao export a wrong format. Now, my net run on deepstream and get output labels but the results are wrong. I do not now if this is because it is not taken the predictions/Softmax output instead of predictions layer only.

19

You can modify output-blob-names in the config file to match the onnx file.

20

I modified it and my results are wrong yet. I get practically all time the same output.

21

How do you run with deepstream? What is the command and all the config files?

22

I have my own app. I run it in a nvinfer plugin. This is my config file:

property:
  gpu-id: 0
  net-scale-factor: 1
  offsets: 103.939;116.779;123.68
  onnx-file: ../models/vehicle_make_net_2/final_model.onnx
  #model-engine-file: ../models/vehicle_make_net_2/final_model.onnx_b2_gpu0_int8.engine
  labelfile-path: ../models/vehicle_make_net_2/labels_corr.txt
  int8-calib-file: ../models/vehicle_make_net_2/final_model_int8_cache.bin
  #force-implicit-batch-dim: 1
  batch-size: 2
  # 0: FP32 and 1=INT8 mode
  network-mode: 1
  input-object-min-width: 64
  input-object-min-height: 64
  process-mode: 2
  model-color-format: 1
  gpu-id: 0
  gie-unique-id: 2
  operate-on-gie-id: 1
  operate-on-class-ids: 0
  is-classifier: 1
  uff-input-blob-name: input_1
  network-type: 1
  output-blob-names: predictions
  classifier-async-mode: 0
  classifier-threshold: 0.7
  infer-dims: 3;224;224