TAO Toolkit Yolov4

motorwaytosafety · November 28, 2022, 6:26am

• Hardware Platform (Jetson AGX)
• DeepStream 5.1
• JetPack Version 4.6
• Issue Type: bugs

I just trained a yolov4 model on the TAO Toolkit, then referenced the ‘.etlt’, calibration and labels file in my Deepstream python application. The application failed to create model engine file. I was able to run a ‘detectnet_v2’ resnet18 model on the same deepstream application without any issues. Can anyone point out the cause of the issue?

This is what my config file looks like.

[property]
gpu-id=0
net-scale-factor=1.0
tlt-model-key=tlt_encode
tlt-encoded-model=/opt/nvidia/deepstream/deepstream-5.1/sources/python/apps/deepstream-imagedata-multistream-tlt/tlt_files/resnet18_detector.etlt
labelfile-path=/opt/nvidia/deepstream/deepstream-5.1/sources/python/apps/deepstream-imagedata-multistream-tlt/tlt_files/labels.txt
int8-calib-file=/opt/nvidia/deepstream/deepstream-5.1/sources/python/apps/deepstream-imagedata-multistream-tlt/tlt_files/calibration.bin
# model-engine-file=/opt/nvidia/deepstream/deepstream-5.1/sources/python/apps/deepstream-imagedata-multistream-tlt/tlt_files/resnet18_detector.trt.int8
offsets=103.939;116.779;123.68
infer-dims=3;384;1248
network-type=0
num-detected-classes=3
uff-input-order=0
output-blob-names=output_cov/Sigmoid;output_bbox/BiasAdd
uff-input-blob-name=input_1
model-color-format=1
maintain-aspect-ratio=0
output-tensor-meta=0

batch-size=1
process-mode=1
network-mode=1
interval=0
gie-unique-id=1
## 0=Group Rectangles, 1=DBSCAN, 2=NMS, 3 = None(No clustering)
cluster-mode=1 

[class-attrs-all]
threshold=0.2
eps=0.7
minBoxes=1

If there is a way to run the model without the INT8 calibration file, I’ll take that.

yingliu · November 28, 2022, 7:14am

What’s type is your Jetson AGX device? Xavier, NX or Nano?

motorwaytosafety · November 28, 2022, 7:15am

The device is AGX Xavier.

yingliu · November 28, 2022, 7:31am

Xavier supports INT8 inference.
So the etlt model can work with the same configuration, but the exported INT8 model and the corresponding calibration file cannot work with the above config? Can you provide the log of the issue?

Also, I have a question on your jetpack&deepstream version. Jetpack4.6 should work together with DeepStream6.0, and deepstream5.1 corresponds to Jetpack4.5.1, your version seems to be mixed. Please consider upgrading to the latest version of Jetpack and DeepStream.

motorwaytosafety · November 28, 2022, 8:02am

The ‘.etlt’ model doesn’t work without the INT8 configuration file. If there is a way to do that, kindly let me know.

Here is the console output.

The output folder frames already exists. Removing...
Frames will be saved in  frames
Creating Pipeline 
 
Creating streamux 
 
Creating source_bin  0  
 
Creating source bin
source-bin-00
Creating Pgie 
 
Creating nvvidconv1 
 
Creating filter1 
 
Creating tiler 
 
Creating nvvidconv 
 
Creating nvosd 
 
Creating transform 
 
Creating EGLSink 

Atleast one of the sources is live
Warn: 'threshold' parameter has been deprecated. Use 'pre-cluster-threshold' instead.
Adding elements to Pipeline 

Linking elements in the Pipeline 

Now playing...
1 :  rtsp://admin:abc12345@10.50.15.182
Starting pipeline 


Using winsys: x11 
gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream-5.1/lib/libnvds_mot_klt.so
gstnvtracker: Optional NvMOT_RemoveStreams not implemented
gstnvtracker: Batch processing is OFF
gstnvtracker: Past frame output is OFF
0:00:00.308914840 15475     0x2db33840 INFO                 nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1716> [UID = 1]: Trying to create engine from model files
ERROR: [TRT]: UffParser: Could not read buffer.
parseModel: Failed to parse UFF model
ERROR: failed to build network since parsing model errors.
ERROR: Failed to create network using custom network creation function
ERROR: Failed to get cuda engine from custom library API
0:00:01.849316101 15475     0x2db33840 ERROR                nvinfer gstnvinfer.cpp:613:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Error in NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1736> [UID = 1]: build engine file failed
Segmentation fault (core dumped)

motorwaytosafety · November 28, 2022, 8:03am

Also, I don’t think the there is a Deepstream-Jetpack version conflict since I was able to train a ‘detectnet_v2’ resnet18 model on the TAO Toolkit and run it on the same deepstream application without any issues.

yingliu · November 28, 2022, 8:17am

This is expected behavior, calibration file is part of the export from TAO when exporting data type as INT8.
Any specific reason you don’t want it?

motorwaytosafety · November 28, 2022, 8:36am

Not that I don’t want it. I was wondering if there was a way to run the application without it since it is causing the issue.

motorwaytosafety · November 28, 2022, 8:39am

Were you able to find the problem from looking at the console output?

yingliu · November 28, 2022, 8:52am

The calibration file is needed for INT8 type, I think we should focus on the issue when INT8 model + calibration is used.

motorwaytosafety · November 28, 2022, 9:10am

Understood.

Were you able to find out why the application failed to load the engine file?

The console output I have uploaded is for the same configuration file. As you can see, I have referenced the calibration file in the config file.

fanzh · November 28, 2022, 9:28am

TRT parsed model failed, did you use TAO tool to test the model? please make sure the model can work.

motorwaytosafety · November 28, 2022, 9:34am

That I didn’t do. I’ll do that and get back to you.

motorwaytosafety · November 29, 2022, 3:45am

I just checked and the ‘!tao yolo_v4 inference’ cell in the training notebook ran without any problems. The predictions were clearly visible in the images inside the ‘yolo_infer_images’ folder.

motorwaytosafety · November 29, 2022, 6:57am

Maybe, I am missing something in the config file. Is there a standard deepstream config file for yolov4 out there?

fanzh · November 29, 2022, 7:25am

please refer to TAO yolov4 sample:deepstream_tao_apps/configs/yolov4_tao at master · NVIDIA-AI-IOT/deepstream_tao_apps · GitHub

motorwaytosafety · November 30, 2022, 5:32am

These samples are for the C++ version of deepstream. Are there any sample configs for deepstream python?

fanzh · November 30, 2022, 6:35am

did you try that C++ version sample? it there still that “ERROR: [TRT]: UffParser: Could not read buffer.” error?
here is no yolov4 python sample, c++ and python have the same configuration file because python will use deepstream c++ SDK by python binding.

motorwaytosafety · November 30, 2022, 10:26am

According to your suggestion here, I installed Deepstream 6.0 and tried to run my yolov4 model in one of the applications. The engine file was successfully created by the sample application. But, the application stops at the first frame due to a bounding box parsing error.

Here is the console output.

Frames will be saved in  frames
Creating Pipeline 
 
Creating streamux 
 
Creating source_bin  0  
 
Creating source bin
source-bin-00
Creating Pgie 
 
Creating nvvidconv1 
 
Creating filter1 
 
Creating tiler 
 
Creating nvvidconv 
 
Creating nvosd 
 
Creating transform 
 
Creating EGLSink 

Atleast one of the sources is live
Warn: 'threshold' parameter has been deprecated. Use 'pre-cluster-threshold' instead.
Adding elements to Pipeline 

Linking elements in the Pipeline 

Now playing...
1 :  rtsp://admin:abc12345@10.50.15.182
Starting pipeline 


Using winsys: x11 
0:00:03.788714178  9630      0x6321d00 INFO                 nvinfer gstnvinfer.cpp:638:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1900> [UID = 1]: deserialized trt engine from :/opt/nvidia/deepstream/deepstream-6.0/sources/deepstream_python_apps/apps/deepstream-imagedata-multistream-edit/tao/tlt_files_lums_engine/yolov4_resnet18_epoch_005.etlt_b1_gpu0_int8.engine
INFO: [Implicit Engine Info]: layers num: 5
0   INPUT  kFLOAT Input           3x384x1248      
1   OUTPUT kINT32 BatchedNMS      1               
2   OUTPUT kFLOAT BatchedNMS_1    200x4           
3   OUTPUT kFLOAT BatchedNMS_2    200             
4   OUTPUT kFLOAT BatchedNMS_3    200             

ERROR: [TRT]: 3: Cannot find binding of given name: boxes
0:00:03.789028661  9630      0x6321d00 WARN                 nvinfer gstnvinfer.cpp:635:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::checkBackendParams() <nvdsinfer_context_impl.cpp:1868> [UID = 1]: Could not find output layer 'boxes' in engine
ERROR: [TRT]: 3: Cannot find binding of given name: scores
0:00:03.789153500  9630      0x6321d00 WARN                 nvinfer gstnvinfer.cpp:635:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::checkBackendParams() <nvdsinfer_context_impl.cpp:1868> [UID = 1]: Could not find output layer 'scores' in engine
ERROR: [TRT]: 3: Cannot find binding of given name: classes
0:00:03.789343880  9630      0x6321d00 WARN                 nvinfer gstnvinfer.cpp:635:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::checkBackendParams() <nvdsinfer_context_impl.cpp:1868> [UID = 1]: Could not find output layer 'classes' in engine
0:00:03.789476752  9630      0x6321d00 INFO                 nvinfer gstnvinfer.cpp:638:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2004> [UID = 1]: Use deserialized engine model: /opt/nvidia/deepstream/deepstream-6.0/sources/deepstream_python_apps/apps/deepstream-imagedata-multistream-edit/tao/tlt_files_lums_engine/yolov4_resnet18_epoch_005.etlt_b1_gpu0_int8.engine
0:00:03.811989524  9630      0x6321d00 INFO                 nvinfer gstnvinfer_impl.cpp:313:notifyLoadModelStatus:<primary-inference> [UID 1]: Load new model:dstest_imagedata_config.txt sucessfully
Decodebin child added: source 

Decodebin child added: decodebin0 

Decodebin child added: rtppcmadepay0 

Decodebin child added: alawdec0 

In cb_newpad


**PERF:  {'stream0': 0.0} 

Decodebin child added: decodebin1 

Decodebin child added: rtph264depay0 

Decodebin child added: h264parse0 

Decodebin child added: capsfilter0 

Decodebin child added: nvv4l2decoder0 

Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 261 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteBlockCreate : Block : BlockType = 261 
In cb_newpad

0:00:06.539014075  9630      0x63178a0 ERROR                nvinfer gstnvinfer.cpp:632:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Error in NvDsInferContextImpl::parseBoundingBox() <nvdsinfer_context_impl_output_parsing.cpp:59> [UID = 1]: Could not find output coverage layer for parsing objects
0:00:06.539097664  9630      0x63178a0 ERROR                nvinfer gstnvinfer.cpp:632:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Error in NvDsInferContextImpl::fillDetectionOutput() <nvdsinfer_context_impl_output_parsing.cpp:735> [UID = 1]: Failed to parse bboxes
Segmentation fault (core dumped)

motorwaytosafety · November 30, 2022, 11:07am

I just referenced the parser ‘.so’ file in the config file and the application is running without any issues. Thanks for your help.

Topic		Replies	Views
Error loading model TAO 4.0 and DeepStream Python Apps DeepStream SDK tensorrt , python	23	1195	February 1, 2023
Use re trainned model in deepstream TAO Toolkit deepstream	5	455	January 12, 2023
How to run deepstream-app without building engine? DeepStream SDK	6	455	March 11, 2024
Tao Yolo_v4 transfer learning is not working with deepstream 5.0 DeepStream SDK	4	775	December 11, 2021
Error loading model TAO 4.0 and DeepStream Python Apps TAO Toolkit	5	296	December 30, 2022
Using deepstream_python_app test3 in deepstream to load etlt model prompts configuration file parsing failure TAO Toolkit	25	949	August 17, 2021
Instructions to integrate TAO 3.0 YoloV4 model into DeepStream produce no output on Jetson NX DeepStream SDK	10	382	December 5, 2023
I ported the yolo sample engine to test4, and an error occurred DeepStream SDK	6	1188	October 12, 2021
Running yolo4v-tiny custom model DeepStream SDK	14	883	December 13, 2022
How use TAO tool kit trained model in deepstream-python-app? DeepStream SDK	9	447	January 24, 2023

TAO Toolkit Yolov4

Related topics