Deepstream infrence gives no detection

Accelerated Computing Intelligent Video Analytics TAO Toolkit
1

cuda-11.3 + cuDNN-8.2
Quadro RTX 5000 dual GPU
Driver Version: 470.82.00
CUDA Version: 11.4
Ubuntu 18.04
python 3.6
Yolo_v4

nvidia/tao/tao-toolkit-tf:

i am using
deepstream_tao_apps](GitHub - NVIDIA-AI-IOT/deepstream_tao_apps: Sample apps to demonstrate how to deploy models trained with TAO on DeepStream)

i can run the python_tao_apps for inference on the default models.
when i run custom model on a image there is no detection
I get the same image ,i got detection on image while using TOA toolkit but not on deepstream

./apps/tao_classifier/ds-tao-classifier -c /home/vaaan/Downloads/deepstream_tao_apps/configs/yolov4_tao/pgie_yolov4_tao_config.txt -i /home/vaaan/Frame_2_473.jpg

Now playing: /home/vaaan/Downloads/deepstream_tao_apps/configs/yolov4_tao/pgie_yolov4_tao_config.txt
WARNING: [TRT]: TensorRT was linked against cuDNN 8.2.1 but loaded cuDNN 8.1.0
WARNING: [TRT]: TensorRT was linked against cuDNN 8.2.1 but loaded cuDNN 8.1.0
0:00:01.206504402 34162 0x55ba302f02a0 INFO nvinfer gstnvinfer.cpp:638:gst_nvinfer_logger: NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1900> [UID = 1]: deserialized trt engine from :/home/vaaan/Downloads/cuda11.3-trt8.0-20210820T231234Z-001/cuda11.3-trt8.0/export_0.1_prune/trt2.engine
INFO: …/nvdsinfer/nvdsinfer_model_builder.cpp:610 [FullDims Engine Info]: layers num: 5
0 INPUT kFLOAT Input 3x608x608 min: 1x3x608x608 opt: 8x3x608x608 Max: 16x3x608x608
1 OUTPUT kINT32 BatchedNMS 1 min: 0 opt: 0 Max: 0
2 OUTPUT kFLOAT BatchedNMS_1 200x4 min: 0 opt: 0 Max: 0
3 OUTPUT kFLOAT BatchedNMS_2 200 min: 0 opt: 0 Max: 0
4 OUTPUT kFLOAT BatchedNMS_3 200 min: 0 opt: 0 Max: 0

0:00:01.206571377 34162 0x55ba302f02a0 INFO nvinfer gstnvinfer.cpp:638:gst_nvinfer_logger: NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2004> [UID = 1]: Use deserialized engine model: /home/vaaan/Downloads/cuda11.3-trt8.0-20210820T231234Z-001/cuda11.3-trt8.0/export_0.1_prune/trt2.engine
0:00:01.214861350 34162 0x55ba302f02a0 INFO nvinfer gstnvinfer_impl.cpp:313:notifyLoadModelStatus: [UID 1]: Load new model:/home/vaaan/Downloads/deepstream_tao_apps/configs/yolov4_tao/pgie_yolov4_tao_config.txt sucessfully
Running…
End of stream
Returned, stopping playback
Deleting pipeline

my config file

################################################################################

Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved.

Permission is hereby granted, free of charge, to any person obtaining a

copy of this software and associated documentation files (the “Software”),

to deal in the Software without restriction, including without limitation

the rights to use, copy, modify, merge, publish, distribute, sublicense,

and/or sell copies of the Software, and to permit persons to whom the

Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in

all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR

IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,

FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL

THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER

LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING

FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER

DEALINGS IN THE SOFTWARE.

################################################################################

[property]
gpu-id=0
net-scale-factor=1.0
offsets=103.939;116.779;123.68
model-color-format=1
labelfile-path=/home/vaaan/Downloads/deepstream_tao_apps/configs/yolov4_tao/yolov4_labels.txt
#labelfile-path=/home/vaaan/Downloads/deepstream_tao_apps/configs/yolov4_tao/yolov4_labels_orginal.txt

#model-engine-file=…/…/models/yolov4/yolov4_resnet18.etlt_b1_gpu0_int8.engine
#int8-calib-file=…/…/models/yolov4/yolov4nv.trt8.cal.bin
#tlt-encoded-model=…/…/models/yolov4/yolov4_resnet18.etlt

#model-engine-file=/home/vaaan/Downloads/deepstream_tao_apps/trt1.engine
model-engine-file=/home/vaaan/Downloads/cuda11.3-trt8.0-20210820T231234Z-001/cuda11.3-trt8.0/export_0.1_prune/trt2.engine
#int8-calib-file=/home/vaaan/Downloads/deepstream_tao_apps/models/yolov4/cal.bin
#tlt-encoded-model=/home/vaaan/Downloads/deepstream_tao_apps/models/yolov4/yolov4_resnet18_epoch_080.etlt

tlt-model-key=nvidia_tlt
#infer-dims=3;544;960
infer-dims=3;608;608
maintain-aspect-ratio=1
uff-input-order=0
uff-input-blob-name=Input
batch-size=1

0=FP32, 1=INT8, 2=FP16 mode

network-mode=1
num-detected-classes=12
interval=0
gie-unique-id=1
is-classifier=0
#network-type=0
cluster-mode=3
output-blob-names=BatchedNMS
parse-bbox-func-name=NvDsInferParseCustomBatchedNMSTLT
custom-lib-path=…/…/post_processor/libnvds_infercustomparser_tao.so

[class-attrs-all]
pre-cluster-threshold=0.8
roi-top-offset=0
roi-bottom-offset=0
detected-min-w=0
detected-min-h=0
detected-max-w=0
detected-max-h=0

2

Why you are running with classification command “ds-tao-classifier” and yolov4 config file?

Please ds-tao-detection if you run yolov4.

3

I have tried that, Same problem

./apps/tao_detection/ds-tao-detection -c /home/vaaan/Downloads/deepstream_tao_apps/configs/yolov4_tao/pgie_yolov4_tao_config.txt -i /home/vaaan/Desktop/download.jpeg

image resolution changes but no detection

4

The /home/vaaan/Downloads/deepstream_tao_apps/models/yolov4/yolov4_resnet18_epoch_080.etlt is your etlt file, right?

What is the size? 608x608?

5

And how did you generate /home/vaaan/Downloads/cuda11.3-trt8.0-20210820T231234Z-001/cuda11.3-trt8.0/export_0.1_prune/trt2.engine ? Seems that it is not correct.

6

Yes thats my etlt file
i have trained with
output_width: 608
output_height: 608
in specs file and even training data set was of the same size.

7

i did the conversion using this ,the cuda i have is 11.4
and tensort is 8.0.1-1+cuda11.3,for deepstream 6 it required cuda 11.4 and the tensorrt installation was done using the documentation

the converter i used was for cuda 11.3 ,11.4 wasn’t available.

./tao-converter /home/vaaan/Downloads/cuda11.3-trt8.0-20210820T231234Z-001/cuda11.3-trt8.0/export_0.1_prune/yolov4_resnet18_epoch_080.etlt -k key -c /home/vaaan/Downloads/cuda11.3-trt8.0-20210820T231234Z-001/cuda11.3-trt8.0/export_0.1_prune/cal.bin -o BatchedNMS -d 3,384,1248 -m 16 -i nchw -t int8 -e /home/vaaan/Downloads/cuda11.3-trt8.0-20210820T231234Z-001/cuda11.3-trt8.0/export_0.1_prune/cal.bin/trt2.engine -b 8 -p Input,1x3x608x608,8x3x608x608,16x3x608x608

8

How did you download tao-converter ? Can you share the link?

9

And your command is not correct. Please modify above to 3,608,608

10

i did the conversion using the version specifying 11.3 ,the cuda i have is 11.4 and tensort is 8.0.1-1+cuda11.3,for deepstream 6 it required cuda 11.4 and the tensorrt installation was done using the documentation specified on deepstream 6.0

the converter i used was for cuda 11.3 as converter 11.4 wasn’t available.

And your command is not correct. Please modify above to 3,608,608

i changed the argument as you said and converted again still the same result

I trained the model on different cuda and tensorrt version?
is that the problem

11

Please comment out engine file as below.
#model-engine-file=/home/vaaan/Downloads/cuda11.3-trt8.0-20210820T231234Z-001/cuda11.3-trt8.0/export_0.1_prune/trt2.engine

Then enable “tlt-encoded-model” in order to let deepstream generate the tensorrt engine instead.

Try again.

12

I commented out the the engine file i am getting this error, i read somewhere yolov4 won’t run on tlt-encoded-model

./apps/tao_detection/ds-tao-detection -c /home/vaaan/Downloads/deepstream_tao_apps/configs/yolov4_tao/pgie_yolov4_tao_config.txt -i /home/vaaan/Desktop/download.jpeg
Now playing: /home/vaaan/Downloads/deepstream_tao_apps/configs/yolov4_tao/pgie_yolov4_tao_config.txt
0:00:00.220707738 3953 0x561b02445aa0 INFO nvinfer gstnvinfer.cpp:638:gst_nvinfer_logger: NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1914> [UID = 1]: Trying to create engine from model files
parseModel: Failed to parse ONNX model
ERROR: tlt/tlt_decode.cpp:389 Failed to build network, error in model parsing.
Segmentation fault (core dumped)

13

No, it should be working.
Can you modify your key?. You are set a wrong key as below. It should be your key.

tlt-model-key=nvidia_tlt

14

still no detection only output image

./apps/tao_detection/ds-tao-detection -c /home/vaaan/Downloads/deepstream_tao_apps/configs/yolov4_tao/pgie_yolov4_tao_config.txt -i /home/vaaan/Desktop/download.jpeg
Now playing: /home/vaaan/Downloads/deepstream_tao_apps/configs/yolov4_tao/pgie_yolov4_tao_config.txt
0:00:00.219165062 4299 0x55ac4f1040a0 INFO nvinfer gstnvinfer.cpp:638:gst_nvinfer_logger: NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1914> [UID = 1]: Trying to create engine from model files
WARNING: [TRT]: onnx2trt_utils.cpp:364: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
WARNING: [TRT]: onnx2trt_utils.cpp:390: One or more weights outside the range of INT32 was clamped
WARNING: [TRT]: onnx2trt_utils.cpp:390: One or more weights outside the range of INT32 was clamped
WARNING: [TRT]: onnx2trt_utils.cpp:390: One or more weights outside the range of INT32 was clamped
WARNING: [TRT]: onnx2trt_utils.cpp:390: One or more weights outside the range of INT32 was clamped
WARNING: [TRT]: onnx2trt_utils.cpp:390: One or more weights outside the range of INT32 was clamped
WARNING: [TRT]: onnx2trt_utils.cpp:390: One or more weights outside the range of INT32 was clamped
WARNING: [TRT]: onnx2trt_utils.cpp:390: One or more weights outside the range of INT32 was clamped
WARNING: [TRT]: onnx2trt_utils.cpp:390: One or more weights outside the range of INT32 was clamped
WARNING: [TRT]: onnx2trt_utils.cpp:390: One or more weights outside the range of INT32 was clamped
WARNING: [TRT]: onnx2trt_utils.cpp:390: One or more weights outside the range of INT32 was clamped
WARNING: [TRT]: onnx2trt_utils.cpp:390: One or more weights outside the range of INT32 was clamped
WARNING: [TRT]: onnx2trt_utils.cpp:390: One or more weights outside the range of INT32 was clamped
WARNING: [TRT]: onnx2trt_utils.cpp:390: One or more weights outside the range of INT32 was clamped
WARNING: [TRT]: onnx2trt_utils.cpp:390: One or more weights outside the range of INT32 was clamped
WARNING: [TRT]: onnx2trt_utils.cpp:390: One or more weights outside the range of INT32 was clamped
WARNING: [TRT]: onnx2trt_utils.cpp:390: One or more weights outside the range of INT32 was clamped
WARNING: [TRT]: onnx2trt_utils.cpp:390: One or more weights outside the range of INT32 was clamped
WARNING: [TRT]: onnx2trt_utils.cpp:390: One or more weights outside the range of INT32 was clamped
WARNING: [TRT]: onnx2trt_utils.cpp:390: One or more weights outside the range of INT32 was clamped
WARNING: [TRT]: onnx2trt_utils.cpp:390: One or more weights outside the range of INT32 was clamped
WARNING: [TRT]: onnx2trt_utils.cpp:390: One or more weights outside the range of INT32 was clamped
WARNING: [TRT]: Missing scale and zero-point for tensor (Unnamed Layer* 161) [Constant]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
WARNING: [TRT]: Missing scale and zero-point for tensor (Unnamed Layer* 165) [Constant]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
WARNING: [TRT]: Missing scale and zero-point for tensor (Unnamed Layer* 170) [Constant]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
WARNING: [TRT]: Missing scale and zero-point for tensor (Unnamed Layer* 271) [Constant]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
WARNING: [TRT]: Missing scale and zero-point for tensor (Unnamed Layer* 275) [Constant]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
WARNING: [TRT]: Missing scale and zero-point for tensor (Unnamed Layer* 280) [Constant]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
WARNING: [TRT]: Missing scale and zero-point for tensor (Unnamed Layer* 369) [Constant]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
WARNING: [TRT]: Missing scale and zero-point for tensor (Unnamed Layer* 372) [Constant]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
WARNING: [TRT]: Missing scale and zero-point for tensor (Unnamed Layer* 376) [Constant]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
WARNING: [TRT]: Missing scale and zero-point for tensor (Unnamed Layer* 609) [Constant]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
WARNING: [TRT]: Missing scale and zero-point for tensor (Unnamed Layer* 613) [Constant]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
WARNING: [TRT]: TensorRT was linked against cuDNN 8.2.1 but loaded cuDNN 8.1.0
WARNING: [TRT]: Detected invalid timing cache, setup a local cache instead
WARNING: [TRT]: TensorRT was linked against cuDNN 8.2.1 but loaded cuDNN 8.1.0
0:03:33.279739852 4299 0x55ac4f1040a0 INFO nvinfer gstnvinfer.cpp:638:gst_nvinfer_logger: NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1947> [UID = 1]: serialize cuda engine to file: /home/vaaan/Downloads/deepstream_tao_apps/models/yolov4/yolov4_resnet18_epoch_080.etlt_b1_gpu0_int8.engine successfully
WARNING: [TRT]: TensorRT was linked against cuDNN 8.2.1 but loaded cuDNN 8.1.0
INFO: …/nvdsinfer/nvdsinfer_model_builder.cpp:610 [Implicit Engine Info]: layers num: 5
0 INPUT kFLOAT Input 3x608x608
1 OUTPUT kINT32 BatchedNMS 1
2 OUTPUT kFLOAT BatchedNMS_1 200x4
3 OUTPUT kFLOAT BatchedNMS_2 200
4 OUTPUT kFLOAT BatchedNMS_3 200

0:03:33.291213898 4299 0x55ac4f1040a0 INFO nvinfer gstnvinfer_impl.cpp:313:notifyLoadModelStatus: [UID 1]: Load new model:/home/vaaan/Downloads/deepstream_tao_apps/configs/yolov4_tao/pgie_yolov4_tao_config.txt sucessfully
Running…
End of stream
Returned, stopping playback
Deleting pipeline

15

Can you set below to run with fp16 mode and retry?
network-mode=2

1 Like
16

Thanks its working now ,Please can i also know

1.whats the problem with int8
2.why does deepstream-tao-app take h264 file and not mp4 format ,how can i inference on mp4 format.
3 where can i get inference speed and meta data

17

Can you upload your cal.bin file? And did you have log how did you generate the cal.bin file?

For mp4, you can leverage the code in deepstream_tao_apps/apps/tao_others at master · NVIDIA-AI-IOT/deepstream_tao_apps · GitHub

For speed test, please see GitHub - NVIDIA-AI-IOT/deepstream_tao_apps: Sample apps to demonstrate how to deploy models trained with TAO on DeepStream

18

cal.bin (7.0 KB)

# tlt-export will fail if .etlt already exists. So we clear the export folder before tlt-export
!rm -rf $LOCAL_EXPERIMENT_DIR/export
!mkdir -p $LOCAL_EXPERIMENT_DIR/export
# Export in FP32 mode. Change --data_type to fp16 for FP16 mode
#tlt yolo_v4 export -m $USER_EXPERIMENT_DIR/experiment_dir_retrain/weights/yolov4_resnet18_epoch_070.tlt \
#                  -k $KEY \
#                   -o $USER_EXPERIMENT_DIR/export/yolov4_resnet18_epoch_070.tlt \
#                   -e $SPECS_DIR/yolo_v4_retrain_resnet18_kitti.txt \
#                   --batch_size 16 \
#                   --data_type fp32

# Uncomment to export in INT8 mode (generate calibration cache file). 
!tlt yolo_v4 export -m $USER_EXPERIMENT_DIR/experiment_dir_retrain/weights/yolov4_resnet18_epoch_080.tlt \
                    -o $USER_EXPERIMENT_DIR/export/yolov4_resnet18_epoch_080.etlt \
                    -e $SPECS_DIR/yolo_v4_retrain_resnet18_kitti.txt \
                    -k $KEY \
                    --cal_image_dir  $USER_EXPERIMENT_DIR/data/testing/image_2 \
                    --data_type int8 \
                    --batch_size 8 \
                    --batches 1 \
                    --cal_cache_file $USER_EXPERIMENT_DIR/export/cal.bin  \
                    --cal_data_file $USER_EXPERIMENT_DIR/export/cal.tensorfile

/home/vaaan/.local/lib/python3.7/site-packages/tlt/init.py:20: DeprecationWarning:
The nvidia-tlt package will be deprecated soon. Going forward please migrate to using the nvidia-tao package.

warnings.warn(message, DeprecationWarning)
~/.tao_mounts.json wasn’t found. Falling back to obtain mount points and docker configs from ~/.tlt_mounts.json.
Please note that this will be deprecated going forward.
2021-11-15 13:18:17,563 [INFO] root: Registry: [‘nvcr.io’]
2021-11-15 13:18:17,718 [WARNING] tlt.components.docker_handler.docker_handler:
Docker will run the commands as root. If you would like to retain your
local host permissions, please add the “user”:“UID:GID” in the
DockerOptions portion of the “/home/vaaan/.tlt_mounts.json” file. You can obtain your
users UID and GID by using the “id -u” and “id -g” commands on the
terminal.
Using TensorFlow backend.
Using TensorFlow backend.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
2021-11-15 07:48:25,754 [INFO] iva.common.export.keras_exporter: Using input nodes: [‘Input’]
2021-11-15 07:48:25,754 [INFO] iva.common.export.keras_exporter: Using output nodes: [‘BatchedNMS’]
The ONNX operator number change on the optimization: 513 → 241
2021-11-15 07:48:53,999 [INFO] keras2onnx: The ONNX operator number change on the optimization: 513 → 241
2021-11-15 07:48:54,137 [INFO] iva.common.export.base_exporter: Generating a tensorfile with random tensor images. This may work well as a profiling tool, however, it may result in inaccurate results at inference. Please generate a tensorfile using the tlt-int8-tensorfile, or provide a custom directory of images for best performance.
100%|█████████████████████████████████████████████| 1/1 [00:01<00:00, 1.91s/it]
2021-11-15 07:48:56,052 [INFO] iva.common.export.keras_exporter: Calibration takes time especially if number of batches is large.
2021-11-15 07:49:14,024 [INFO] iva.common.export.base_calibrator: Saving calibration cache (size 7136) to /workspace/tlt-experiments/yolo_v4/export/cal.bin
2021-11-15 13:20:55,923 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.

19

Thanks. You were generating the cal.bin inside the docker. Could you share which tlt/tao docker you were running? Please check with
$ tlt info --verbose
or
$ tao info --verbose

20

yes i was using a docker
nvcr.io/nvidia/tlt-streamanalytics:v3.0-py3