Failed to generate TRT .engine from ONNX model generated using TAO

Accelerated Computing Intelligent Video Analytics TAO Toolkit
1

I converted a .etlt model file to .onnx with Object Detection using TAO DetectNet_v2, but when i am trying to build its tensorrt .engine file on orin nano running:

/usr/src/tensorrt/bin/trtexec --onnx=resnet18_detector.onnx --saveEngine=/home/orin/Documents/resnet_detector.engine --workspace=2048 --verbose

it raised the following error.I was wondering if you could help me to fix it.

[06/18/2024-02:02:08] [I] === Model Options ===
[06/18/2024-02:02:08] [I] Format: ONNX
[06/18/2024-02:02:08] [I] Model: resnet18_detector_fp32.onnx
[06/18/2024-02:02:08] [I] Output:
[06/18/2024-02:02:08] [I] === Build Options ===
[06/18/2024-02:02:08] [I] Max batch: explicit batch
[06/18/2024-02:02:08] [I] Memory Pools: workspace: 2048 MiB, dlaSRAM: default, dlaLocalDRAM: default, dlaGlobalDRAM: default
[06/18/2024-02:02:08] [I] minTiming: 1
[06/18/2024-02:02:08] [I] avgTiming: 8
[06/18/2024-02:02:08] [I] Precision: FP32
[06/18/2024-02:02:08] [I] LayerPrecisions:
[06/18/2024-02:02:08] [I] Calibration:
[06/18/2024-02:02:08] [I] Refit: Disabled
[06/18/2024-02:02:08] [I] Sparsity: Disabled
[06/18/2024-02:02:08] [I] Safe mode: Disabled
[06/18/2024-02:02:08] [I] DirectIO mode: Disabled
[06/18/2024-02:02:08] [I] Restricted mode: Disabled
[06/18/2024-02:02:08] [I] Build only: Disabled
[06/18/2024-02:02:08] [I] Save engine: /home/orin/Documents/resnet_detector.engine
[06/18/2024-02:02:08] [I] Load engine:
[06/18/2024-02:02:08] [I] Profiling verbosity: 0
[06/18/2024-02:02:08] [I] Tactic sources: Using default tactic sources
[06/18/2024-02:02:08] [I] timingCacheMode: local
[06/18/2024-02:02:08] [I] timingCacheFile:
[06/18/2024-02:02:08] [I] Heuristic: Disabled
[06/18/2024-02:02:08] [I] Preview Features: Use default preview flags.
[06/18/2024-02:02:08] [I] Input(s)s format: fp32:CHW
[06/18/2024-02:02:08] [I] Output(s)s format: fp32:CHW
[06/18/2024-02:02:08] [I] Input build shapes: model
[06/18/2024-02:02:08] [I] Input calibration shapes: model
[06/18/2024-02:02:08] [I] === System Options ===
[06/18/2024-02:02:08] [I] Device: 0
[06/18/2024-02:02:08] [I] DLACore:
[06/18/2024-02:02:08] [I] Plugins:
[06/18/2024-02:02:08] [I] === Inference Options ===
[06/18/2024-02:02:08] [I] Batch: Explicit
[06/18/2024-02:02:08] [I] Input inference shapes: model
[06/18/2024-02:02:08] [I] Iterations: 10
[06/18/2024-02:02:08] [I] Duration: 3s (+ 200ms warm up)
[06/18/2024-02:02:08] [I] Sleep time: 0ms
[06/18/2024-02:02:08] [I] Idle time: 0ms
[06/18/2024-02:02:08] [I] Streams: 1
[06/18/2024-02:02:08] [I] ExposeDMA: Disabled
[06/18/2024-02:02:08] [I] Data transfers: Enabled
[06/18/2024-02:02:08] [I] Spin-wait: Disabled
[06/18/2024-02:02:08] [I] Multithreading: Disabled
[06/18/2024-02:02:08] [I] CUDA Graph: Disabled
[06/18/2024-02:02:08] [I] Separate profiling: Disabled
[06/18/2024-02:02:08] [I] Time Deserialize: Disabled
[06/18/2024-02:02:08] [I] Time Refit: Disabled
[06/18/2024-02:02:08] [I] NVTX verbosity: 0
[06/18/2024-02:02:08] [I] Persistent Cache Ratio: 0
[06/18/2024-02:02:08] [I] Inputs:
[06/18/2024-02:02:08] [I] === Reporting Options ===
[06/18/2024-02:02:08] [I] Verbose: Enabled
[06/18/2024-02:02:08] [I] Averages: 10 inferences
[06/18/2024-02:02:08] [I] Percentiles: 90,95,99
[06/18/2024-02:02:08] [I] Dump refittable layers:Disabled
[06/18/2024-02:02:08] [I] Dump output: Disabled
[06/18/2024-02:02:08] [I] Profile: Disabled
[06/18/2024-02:02:08] [I] Export timing to JSON file:
[06/18/2024-02:02:08] [I] Export output to JSON file:
[06/18/2024-02:02:08] [I] Export profile to JSON file:
[06/18/2024-02:02:08] [I]
[06/18/2024-02:02:08] [I] === Device Information ===
[06/18/2024-02:02:08] [I] Selected Device: Orin
[06/18/2024-02:02:08] [I] Compute Capability: 8.7
[06/18/2024-02:02:08] [I] SMs: 8
[06/18/2024-02:02:08] [I] Compute Clock Rate: 0.918 GHz
[06/18/2024-02:02:08] [I] Device Global Memory: 14485 MiB
[06/18/2024-02:02:08] [I] Shared Memory per SM: 164 KiB
[06/18/2024-02:02:08] [I] Memory Bus Width: 64 bits (ECC disabled)
[06/18/2024-02:02:08] [I] Memory Clock Rate: 0.918 GHz
[06/18/2024-02:02:08] [I]
[06/18/2024-02:02:08] [I] TensorRT version: 8.5.2
[06/18/2024-02:02:08] [V] [TRT] Registered plugin creator - ::BatchedNMSDynamic_TRT version 1
[06/18/2024-02:02:08] [V] [TRT] Registered plugin creator - ::BatchedNMS_TRT version 1
[06/18/2024-02:02:08] [V] [TRT] Registered plugin creator - ::BatchTilePlugin_TRT version 1
[06/18/2024-02:02:08] [V] [TRT] Registered plugin creator - ::Clip_TRT version 1
[06/18/2024-02:02:08] [V] [TRT] Registered plugin creator - ::CoordConvAC version 1
[06/18/2024-02:02:08] [V] [TRT] Registered plugin creator - ::CropAndResizeDynamic version 1
[06/18/2024-02:02:08] [V] [TRT] Registered plugin creator - ::CropAndResize version 1
[06/18/2024-02:02:08] [V] [TRT] Registered plugin creator - ::DecodeBbox3DPlugin version 1
[06/18/2024-02:02:08] [V] [TRT] Registered plugin creator - ::DetectionLayer_TRT version 1
[06/18/2024-02:02:08] [V] [TRT] Registered plugin creator - ::EfficientNMS_Explicit_TF_TRT version 1
[06/18/2024-02:02:08] [V] [TRT] Registered plugin creator - ::EfficientNMS_Implicit_TF_TRT version 1
[06/18/2024-02:02:08] [V] [TRT] Registered plugin creator - ::EfficientNMS_ONNX_TRT version 1
[06/18/2024-02:02:08] [V] [TRT] Registered plugin creator - ::EfficientNMS_TRT version 1
[06/18/2024-02:02:08] [V] [TRT] Registered plugin creator - ::FlattenConcat_TRT version 1
[06/18/2024-02:02:08] [V] [TRT] Registered plugin creator - ::GenerateDetection_TRT version 1
[06/18/2024-02:02:08] [V] [TRT] Registered plugin creator - ::GridAnchor_TRT version 1
[06/18/2024-02:02:08] [V] [TRT] Registered plugin creator - ::GridAnchorRect_TRT version 1
[06/18/2024-02:02:08] [V] [TRT] Registered plugin creator - ::GroupNorm version 1
[06/18/2024-02:02:08] [V] [TRT] Registered plugin creator - ::InstanceNormalization_TRT version 1
[06/18/2024-02:02:08] [V] [TRT] Registered plugin creator - ::InstanceNormalization_TRT version 2
[06/18/2024-02:02:08] [V] [TRT] Registered plugin creator - ::LayerNorm version 1
[06/18/2024-02:02:08] [V] [TRT] Registered plugin creator - ::LReLU_TRT version 1
[06/18/2024-02:02:08] [V] [TRT] Registered plugin creator - ::MultilevelCropAndResize_TRT version 1
[06/18/2024-02:02:08] [V] [TRT] Registered plugin creator - ::MultilevelProposeROI_TRT version 1
[06/18/2024-02:02:08] [V] [TRT] Registered plugin creator - ::MultiscaleDeformableAttnPlugin_TRT version 1
[06/18/2024-02:02:08] [V] [TRT] Registered plugin creator - ::NMSDynamic_TRT version 1
[06/18/2024-02:02:08] [V] [TRT] Registered plugin creator - ::NMS_TRT version 1
[06/18/2024-02:02:08] [V] [TRT] Registered plugin creator - ::Normalize_TRT version 1
[06/18/2024-02:02:08] [V] [TRT] Registered plugin creator - ::PillarScatterPlugin version 1
[06/18/2024-02:02:08] [V] [TRT] Registered plugin creator - ::PriorBox_TRT version 1
[06/18/2024-02:02:08] [V] [TRT] Registered plugin creator - ::ProposalDynamic version 1
[06/18/2024-02:02:08] [V] [TRT] Registered plugin creator - ::ProposalLayer_TRT version 1
[06/18/2024-02:02:08] [V] [TRT] Registered plugin creator - ::Proposal version 1
[06/18/2024-02:02:08] [V] [TRT] Registered plugin creator - ::PyramidROIAlign_TRT version 1
[06/18/2024-02:02:08] [V] [TRT] Registered plugin creator - ::Region_TRT version 1
[06/18/2024-02:02:08] [V] [TRT] Registered plugin creator - ::Reorg_TRT version 1
[06/18/2024-02:02:08] [V] [TRT] Registered plugin creator - ::ResizeNearest_TRT version 1
[06/18/2024-02:02:08] [V] [TRT] Registered plugin creator - ::ROIAlign_TRT version 1
[06/18/2024-02:02:08] [V] [TRT] Registered plugin creator - ::RPROI_TRT version 1
[06/18/2024-02:02:08] [V] [TRT] Registered plugin creator - ::ScatterND version 1
[06/18/2024-02:02:08] [V] [TRT] Registered plugin creator - ::SeqLen2Spatial version 1
[06/18/2024-02:02:08] [V] [TRT] Registered plugin creator - ::SpecialSlice_TRT version 1
[06/18/2024-02:02:08] [V] [TRT] Registered plugin creator - ::SplitGeLU version 1
[06/18/2024-02:02:08] [V] [TRT] Registered plugin creator - ::Split version 1
[06/18/2024-02:02:08] [V] [TRT] Registered plugin creator - ::VoxelGeneratorPlugin version 1
[06/18/2024-02:02:08] [I] [TRT] [MemUsageChange] Init CUDA: CPU +220, GPU +0, now: CPU 246, GPU 3029 (MiB)
[06/18/2024-02:02:09] [V] [TRT] Trying to load shared library libnvinfer_builder_resource.so.8.5.2
[06/18/2024-02:02:09] [V] [TRT] Loaded shared library libnvinfer_builder_resource.so.8.5.2
[06/18/2024-02:02:10] [I] [TRT] [MemUsageChange] Init builder kernel library: CPU +302, GPU +429, now: CPU 571, GPU 3480 (MiB)
[06/18/2024-02:02:10] [I] Start parsing network model
[libprotobuf ERROR google/protobuf/text_format.cc:298] Error parsing text-format onnx2trt_onnx.ModelProto: 1:5: Invalid control characters encountered in text.
[libprotobuf ERROR google/protobuf/text_format.cc:298] Error parsing text-format onnx2trt_onnx.ModelProto: 1:9: Interpreting non ascii codepoint 232.
[libprotobuf ERROR google/protobuf/text_format.cc:298] Error parsing text-format onnx2trt_onnx.ModelProto: 1:9: Message type “onnx2trt_onnx.ModelProto” has no field named “ptrt”.
[06/18/2024-02:02:10] [E] [TRT] ModelImporter.cpp:688: Failed to parse ONNX model from file: resnet18_detector_fp32.onnx
[06/18/2024-02:02:10] [E] Failed to parse onnx file
[06/18/2024-02:02:10] [I] Finish parsing network model
[06/18/2024-02:02:10] [E] Parsing model failed
[06/18/2024-02:02:10] [E] Failed to create engine from model or file.
[06/18/2024-02:02:10] [E] Engine set up failed
&&&& FAILED TensorRT.trtexec [TensorRT v8502] # /usr/src/tensorrt/bin/trtexec --onnx=resnet18_detector.onnx --saveEngine=/home/orin/Documents/resnet_detector.engine --workspace=2048 --verbose

Thanks!

2

How did you convert? Actually the .etlt model of detectent_v2 is a .uff file instead of .onnx file. So, after conversion, it is uff file.
So, for detectnet_v2 network, you can not convert ngc’s etlt file to onnx file.
You can use TAO 5.0 docker to export .tlt file to .onnx file.

3

Hii @Morganh ,
this is how i converted:

!tao converter $USER_EXPERIMENT_DIR/experiment_dir_final/resnet18_detector.etlt
-k $KEY
-o output_cov/Sigmoid,output_bbox/BiasAdd
-d 3,384,1248
-i nchw
-m 128
-t fp32
-e $USER_EXPERIMENT_DIR/experiment_dir_final/resnet18_detector.onnx
-b 4 \

Do note i was working on TAO 4.0 with Python 3.6.9 but then as you suggested i tried TAO 5.0 docker with Python 3.7.0 but after training, the file generated is resnet18_detector.hdf5 and not .tlt file.

Can you explain why?

4

It is expected. For detectnet_v2 network, the .tlt is actually encrypted .hdf5 file. Since TAO5.0, the source code is open. So, the trained result will be .hdf5 file now.
You can export it to onnx file.

5

It worked! The engine file was built successfully. However, when I test it with DeepStream sample apps, there is no inference being performed.

this is the config file:
[property]
gpu-id=0
net-scale-factor=0.0039215697906911373
model-color-format=0
onnx-file=/home/orin/Documents/TAO/resnet18_detector.onnx
model-engine-file=/home/orin/Documents/resnet_detector.engine
#int8-calib-file=calib.table
labelfile-path=/home/orin/Documents/TAO/labels.txt
batch-size=1
network-mode=0
num-detected-classes=3
interval=0
gie-unique-id=1
process-mode=1
network-type=0
cluster-mode=2
maintain-aspect-ratio=1
symmetric-padding=1
#workspace-size=2000
parse-bbox-func-name=NvDsInferParseYolo
#parse-bbox-func-name=NvDsInferParseYoloCuda
custom-lib-path=nvdsinfer_custom_impl_Yolo/libnvdsinfer_custom_impl_Yolo.so
engine-create-func-name=NvDsInferYoloCudaEngineGet

[class-attrs-all]
nms-iou-threshold=0.5
pre-cluster-threshold=0.2
topk=20

and this is the command:

gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! m.sink_0 nvstreammux name=m batch-size=1 width=640 height=640 ! nvinfer config-file-path=/opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-test1/dstest1_pgie_config.txt ! nvvideoconvert ! nvv4l2h264enc ! rtph264pay ! udpsink host=192.168.55.100 port=4000 sync=false

6

Seems that your spec file is not correct. Please refer to deepstream-tao-apps github. Similar model is in Facenet, peoplenet_tao, etc. They are all based on detectnet_v2 network.

7

Yes, i got it corrected and it worked!
Thanks @Morganh

8

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.