Running trtexec on Google's EfficientDet-D4 in ONNX Format

Niklas.Flindt · March 29, 2021, 2:01pm

Description

I am trying to run the official EfficientDet-D4 in TensorRT. It just won’t work.
Prior to that, I am using tf2onnx.convert to convert the TF saved-model to onnx.

When running trtexec on the onnx file it results in no traceback at all.
If I am using “verbose” logging, I at least get the information where the import of the model stops but there it still no real traceback.
It seems that he “Unsqueeze” is the problem, but I cannot see why.

Any help would be appreciated. I really want to use TensorRT, but for that it is necessary that the model is working, ofc.

Let me know if any additional information is needed!
I had to crop the traceback/verbose output to match the maximal thread size…

Environment

TensorRT Version: 7.2.3.4
GPU Type: RTX3090
Nvidia Driver Version: 27.21.14.5148
CUDA Version: 11.2
CUDNN Version: 8.0
Operating System + Version: Windows
Python Version: 3.8.8
TensorFlow Version (if applicable): 2.4.1
ONNX_Tensorflow Version: 1.9
Onnx Version: 1.8.1

Relevant Files

The onnx file can be found here:

*Updated File: Dropbox - File Deleted

Steps To Reproduce

Simply use
trtexec.exe --onnx=onnx_model.onnx --verbose

Please include:

Exact steps/commands to build your repro
Exact steps/commands to run your repro
Full traceback of errors encountered

I encounter the following traceback:

(tf24) C:\Repos\TensorRT-7.2.3.4\bin>trtexec.exe --onnx=…\saved_model\onnx_float.onnx --verbose
&&&& RUNNING TensorRT.trtexec # trtexec.exe --onnx=…\saved_model\onnx_float.onnx --verbose
[03/29/2021-15:20:06] [I] === Model Options ===
[03/29/2021-15:20:06] [I] Format: ONNX
[03/29/2021-15:20:06] [I] Model: …\saved_model\onnx_float.onnx
[03/29/2021-15:20:06] [I] Output:
[03/29/2021-15:20:06] [I] === Build Options ===
[03/29/2021-15:20:06] [I] Max batch: explicit
[03/29/2021-15:20:06] [I] Workspace: 16 MiB
[03/29/2021-15:20:06] [I] minTiming: 1
[03/29/2021-15:20:06] [I] avgTiming: 8
[03/29/2021-15:20:06] [I] Precision: FP32
[03/29/2021-15:20:06] [I] Calibration:
[03/29/2021-15:20:06] [I] Refit: Disabled
[03/29/2021-15:20:06] [I] Safe mode: Disabled
[03/29/2021-15:20:06] [I] Save engine:
[03/29/2021-15:20:06] [I] Load engine:
[03/29/2021-15:20:06] [I] Builder Cache: Enabled
[03/29/2021-15:20:06] [I] NVTX verbosity: 0
[03/29/2021-15:20:06] [I] Tactic sources: Using default tactic sources
[03/29/2021-15:20:06] [I] Input(s)s format: fp32:CHW
[03/29/2021-15:20:06] [I] Output(s)s format: fp32:CHW
[03/29/2021-15:20:06] [I] Input build shapes: model
[03/29/2021-15:20:06] [I] Input calibration shapes: model
[03/29/2021-15:20:06] [I] === System Options ===
[03/29/2021-15:20:06] [I] Device: 0
[03/29/2021-15:20:06] [I] DLACore:
[03/29/2021-15:20:06] [I] Plugins:
[03/29/2021-15:20:06] [I] === Inference Options ===
[03/29/2021-15:20:06] [I] Batch: Explicit
[03/29/2021-15:20:06] [I] Input inference shapes: model
[03/29/2021-15:20:06] [I] Iterations: 10
[03/29/2021-15:20:06] [I] Duration: 3s (+ 200ms warm up)
[03/29/2021-15:20:06] [I] Sleep time: 0ms
[03/29/2021-15:20:06] [I] Streams: 1
[03/29/2021-15:20:06] [I] ExposeDMA: Disabled
[03/29/2021-15:20:06] [I] Data transfers: Enabled
[03/29/2021-15:20:06] [I] Spin-wait: Disabled
[03/29/2021-15:20:06] [I] Multithreading: Disabled
[03/29/2021-15:20:06] [I] CUDA Graph: Disabled
[03/29/2021-15:20:06] [I] Separate profiling: Disabled
[03/29/2021-15:20:06] [I] Skip inference: Disabled
[03/29/2021-15:20:06] [I] Inputs:
[03/29/2021-15:20:06] [I] === Reporting Options ===
[03/29/2021-15:20:06] [I] Verbose: Enabled
[03/29/2021-15:20:06] [I] Averages: 10 inferences
[03/29/2021-15:20:06] [I] Percentile: 99
[03/29/2021-15:20:06] [I] Dump refittable layers:Disabled
[03/29/2021-15:20:07] [I] Dump output: Disabled
[03/29/2021-15:20:07] [I] Profile: Disabled
[03/29/2021-15:20:07] [I] Export timing to JSON file:
[03/29/2021-15:20:07] [I] Export output to JSON file:
[03/29/2021-15:20:07] [I] Export profile to JSON file:
[03/29/2021-15:20:07] [I]
[03/29/2021-15:20:07] [I] === Device Information ===
[03/29/2021-15:20:07] [I] Selected Device: GeForce RTX 3090
[03/29/2021-15:20:07] [I] Compute Capability: 8.6
[03/29/2021-15:20:07] [I] SMs: 82
[03/29/2021-15:20:07] [I] Compute Clock Rate: 1.86 GHz
[03/29/2021-15:20:07] [I] Device Global Memory: 24576 MiB
[03/29/2021-15:20:07] [I] Shared Memory per SM: 100 KiB
[03/29/2021-15:20:07] [I] Memory Bus Width: 384 bits (ECC disabled)
[03/29/2021-15:20:07] [I] Memory Clock Rate: 9.751 GHz
[03/29/2021-15:20:07] [I]
[03/29/2021-15:20:07] [V] [TRT] Registered plugin creator - ::GridAnchor_TRT version 1
[03/29/2021-15:20:07] [V] [TRT] Registered plugin creator - ::NMS_TRT version 1
[03/29/2021-15:20:07] [V] [TRT] Registered plugin creator - ::Reorg_TRT version 1
[03/29/2021-15:20:07] [V] [TRT] Registered plugin creator - ::Region_TRT version 1
[03/29/2021-15:20:07] [V] [TRT] Registered plugin creator - ::Clip_TRT version 1
[03/29/2021-15:20:07] [V] [TRT] Registered plugin creator - ::LReLU_TRT version 1
[03/29/2021-15:20:07] [V] [TRT] Registered plugin creator - ::PriorBox_TRT version 1
[03/29/2021-15:20:07] [V] [TRT] Registered plugin creator - ::Normalize_TRT version 1
[03/29/2021-15:20:07] [V] [TRT] Registered plugin creator - ::RPROI_TRT version 1
[03/29/2021-15:20:07] [V] [TRT] Registered plugin creator - ::BatchedNMS_TRT version 1
[03/29/2021-15:20:07] [V] [TRT] Registered plugin creator - ::BatchedNMSDynamic_TRT version 1
[03/29/2021-15:20:07] [V] [TRT] Registered plugin creator - ::FlattenConcat_TRT version 1
[03/29/2021-15:20:07] [V] [TRT] Registered plugin creator - ::CropAndResize version 1
[03/29/2021-15:20:07] [V] [TRT] Registered plugin creator - ::DetectionLayer_TRT version 1
[03/29/2021-15:20:07] [V] [TRT] Registered plugin creator - ::Proposal version 1
[03/29/2021-15:20:07] [V] [TRT] Registered plugin creator - ::ProposalLayer_TRT version 1
[03/29/2021-15:20:07] [V] [TRT] Registered plugin creator - ::PyramidROIAlign_TRT version 1
[03/29/2021-15:20:07] [V] [TRT] Registered plugin creator - ::ResizeNearest_TRT version 1
[03/29/2021-15:20:07] [V] [TRT] Registered plugin creator - ::Split version 1
[03/29/2021-15:20:07] [V] [TRT] Registered plugin creator - ::SpecialSlice_TRT version 1
[03/29/2021-15:20:07] [V] [TRT] Registered plugin creator - ::InstanceNormalization_TRT version 1

Input filename: Y:\Flindt\Data\Datasets\Objectdetection\IHFood\Front\Objectdetection\EfficientDet4\models\my_model_dir2\final_model\saved_model\onnx_float.onnx
ONNX IR version: 0.0.7
Opset version: 13
Producer name:
Producer version:
Domain:
Model version: 0
Doc string:

[03/29/2021-15:20:14] [V] [TRT] Plugin creator already registered - ::GridAnchor_TRT version 1
[03/29/2021-15:20:14] [V] [TRT] Plugin creator already registered - ::NMS_TRT version 1
[03/29/2021-15:20:14] [V] [TRT] Plugin creator already registered - ::Reorg_TRT version 1
[03/29/2021-15:20:14] [V] [TRT] Plugin creator already registered - ::Region_TRT version 1
[03/29/2021-15:20:14] [V] [TRT] Plugin creator already registered - ::Clip_TRT version 1
[03/29/2021-15:20:14] [V] [TRT] Plugin creator already registered - ::LReLU_TRT version 1
[03/29/2021-15:20:14] [V] [TRT] Plugin creator already registered - ::PriorBox_TRT version 1
[03/29/2021-15:20:14] [V] [TRT] Plugin creator already registered - ::Normalize_TRT version 1
[03/29/2021-15:20:14] [V] [TRT] Plugin creator already registered - ::RPROI_TRT version 1
[03/29/2021-15:20:14] [V] [TRT] Plugin creator already registered - ::BatchedNMS_TRT version 1
[03/29/2021-15:20:14] [V] [TRT] Plugin creator already registered - ::BatchedNMSDynamic_TRT version 1
[03/29/2021-15:20:14] [V] [TRT] Plugin creator already registered - ::FlattenConcat_TRT version 1
[03/29/2021-15:20:14] [V] [TRT] Plugin creator already registered - ::CropAndResize version 1
[03/29/2021-15:20:14] [V] [TRT] Plugin creator already registered - ::DetectionLayer_TRT version 1
[03/29/2021-15:20:14] [V] [TRT] Plugin creator already registered - ::Proposal version 1
[03/29/2021-15:20:14] [V] [TRT] Plugin creator already registered - ::ProposalLayer_TRT version 1
[03/29/2021-15:20:14] [V] [TRT] Plugin creator already registered - ::PyramidROIAlign_TRT version 1
[03/29/2021-15:20:14] [V] [TRT] Plugin creator already registered - ::ResizeNearest_TRT version 1
[03/29/2021-15:20:14] [V] [TRT] Plugin creator already registered - ::Split version 1
[03/29/2021-15:20:14] [V] [TRT] Plugin creator already registered - ::SpecialSlice_TRT version 1
[03/29/2021-15:20:14] [V] [TRT] Plugin creator already registered - ::InstanceNormalization_TRT version 1
[03/29/2021-15:20:14] [V] [TRT] ModelImporter.cpp:202: Adding network input: input_tensor:0 with dtype: float32, dimensions: (1, 3, 1024, 1024)
[03/29/2021-15:20:14] [V] [TRT] ImporterContext.hpp:120: Registering tensor: input_tensor:0 for ONNX tensor: input_tensor:0
[03/29/2021-15:20:14] [V] [TRT] ModelImporter.cpp:90: Importing initializer: StatefulPartitionedCall/Postprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/ClipToWindow/Greater/y:0
[03/29/2021-15:20:14] [V] [TRT] ModelImporter.cpp:90: Importing initializer: const_fold_opt__3869
[03/29/2021-15:20:15] [V] [TRT] ModelImporter.cpp:90: Importing initializer: StatefulPartitionedCall/map/while/maximum_iterations:0
[03/29/2021-15:20:15] [W] [TRT] onnx2trt_utils.cpp:220: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[03/29/2021-15:20:15] [V] [TRT] ModelImporter.cpp:90: Importing initializer: StatefulPartitionedCall/Postprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/range_2/start:0
[03/29/2021-15:20:15] [V] [TRT] ModelImporter.cpp:90: Importing initializer: StatefulPartitionedCall/map/TensorArrayV2Stack_1/TensorListStack/element_shape:0
[03/29/2021-15:20:15] [V] [TRT] ModelImporter.cpp:90: Importing initializer: const_starts__2943
[03/29/2021-15:20:15] [V] [TRT] ModelImporter.cpp:90: Importing initializer: StatefulPartitionedCall/map/TensorArrayV2Stack/TensorListStack/element_shape:0
[03/29/2021-15:20:15] [V] [TRT] ModelImporter.cpp:90: Importing initializer: StatefulPartitionedCall/EfficientDet-D4/model/stem_conv2d/Conv2D_weights_fused_bn
[03/29/2021-15:20:15] [V] [TRT] ModelImporter.cpp:90: Importing initializer: StatefulPartitionedCall/EfficientDet-D4/model/stem_conv2d/Conv2D_bias_fused_bn
[03/29/2021-15:20:15] [V] [TRT] ModelImporter.cpp:90: Importing initializer: StatefulPartitionedCall/EfficientDet-D4/model/stack_0/block_0/depthwise_conv2d/depthwise_weights_fused_bn
[03/29/2021-15:20:15] [V] [TRT] ModelImporter.cpp:90: Importing initializer: StatefulPartitionedCall/EfficientDet-D4/model/stack_0/block_0/depthwise_conv2d/depthwise_bias_fused_bn
[03/29/2021-15:20:15] [V] [TRT] ModelImporter.cpp:90: Importing initializer: const_fold_opt__3754
[03/29/2021-15:20:15] [V] [TRT] ModelImporter.cpp:90: Importing initializer: new_shape__3682
[03/29/2021-15:20:15] [V] [TRT] ModelImporter.cpp:90: Importing initializer: StatefulPartitionedCall/EfficientDet-D4/model/stack_0/block_0/se_reduce_conv2d/Conv2D/ReadVariableOp:0
[03/29/2021-15:20:15] [V] [TRT] ModelImporter.cpp:90: Importing initializer: StatefulPartitionedCall/EfficientDet-D4/model/stack_0/block_0/se_reduce_conv2d/BiasAdd/ReadVariableOp:0
[03/29/2021-15:20:15] [V] [TRT] ModelImporter.cpp:90: Importing initializer: StatefulPartitionedCall/EfficientDet-D4/model/stack_0/block_0/se_expand_conv2d/Conv2D/ReadVariableOp:0
[03/29/2021-15:20:15] [V] [TRT] ModelImporter.cpp:90: Importing initializer: StatefulPartitionedCall/EfficientDet-D4/model/stack_0/block_0/se_expand_conv2d/BiasAdd/ReadVariableOp:0
[03/29/2021-15:20:15] [V] [TRT] ModelImporter.cpp:90: Importing initializer: StatefulPartitionedCall/EfficientDet-D4/model/stack_0/block_0/project_conv2d/Conv2D_weights_fused_bn
…
[03/29/2021-15:20:39] [V] [TRT] ModelImporter.cpp:90: Importing initializer: const_axes__196
[03/29/2021-15:20:39] [V] [TRT] ModelImporter.cpp:90: Importing initializer: const_fold_opt__3976
[03/29/2021-15:20:39] [V] [TRT] ModelImporter.cpp:90: Importing initializer: const_starts__206
[03/29/2021-15:20:39] [V] [TRT] ModelImporter.cpp:90: Importing initializer: const_ends__207
[03/29/2021-15:20:39] [V] [TRT] ModelImporter.cpp:90: Importing initializer: const_axes__208
[03/29/2021-15:20:39] [V] [TRT] ModelImporter.cpp:90: Importing initializer: const_axes__210
[03/29/2021-15:20:39] [V] [TRT] ModelImporter.cpp:103: Parsing node: map/while/add [Add]
[03/29/2021-15:20:39] [V] [TRT] ModelImporter.cpp:119: Searching for input: map_while_placeholder:0
[03/29/2021-15:20:39] [V] [TRT] ModelImporter.cpp:119: Searching for input: map/while/add/y:0
[03/29/2021-15:20:39] [V] [TRT] ModelImporter.cpp:125: map/while/add [Add] inputs: [map_while_placeholder:0 → ()], [map/while/add/y:0 → ()],
[03/29/2021-15:20:39] [V] [TRT] ImporterContext.hpp:150: Registering constant layer: map/while/add/y:0 for ONNX initializer: map/while/add/y:0
[03/29/2021-15:20:39] [V] [TRT] ImporterContext.hpp:154: Registering layer: map/while/add for ONNX node: map/while/add
[03/29/2021-15:20:39] [V] [TRT] ImporterContext.hpp:120: Registering tensor: map/while/Identity_2:0 for ONNX tensor: map/while/Identity_2:0
[03/29/2021-15:20:39] [V] [TRT] ModelImporter.cpp:179: map/while/add [Add] outputs: [map/while/Identity_2:0 → ()],
[03/29/2021-15:20:39] [V] [TRT] ModelImporter.cpp:103: Parsing node: cond___map/while/Less__6 [Cast]
[03/29/2021-15:20:39] [V] [TRT] ModelImporter.cpp:119: Searching for input: map/while/Identity_2:0
[03/29/2021-15:20:39] [V] [TRT] ModelImporter.cpp:125: cond___map/while/Less__6 [Cast] inputs: [map/while/Identity_2:0 → ()],
[03/29/2021-15:20:39] [V] [TRT] builtin_op_importers.cpp:320: Casting to type: float32
[03/29/2021-15:20:39] [V] [TRT] ImporterContext.hpp:154: Registering layer: cond___map/while/Less__6 for ONNX node: cond___map/while/Less__6
[03/29/2021-15:20:40] [V] [TRT] ImporterContext.hpp:120: Registering tensor: cond___map/while/Less__6:0 for ONNX tensor: cond___map/while/Less__6:0
[03/29/2021-15:20:40] [V] [TRT] ModelImporter.cpp:179: cond___map/while/Less__6 [Cast] outputs: [cond___map/while/Less__6:0 → ()],
[03/29/2021-15:20:40] [V] [TRT] ModelImporter.cpp:103: Parsing node: cond___map/while/Less_1__4 [Cast]
[03/29/2021-15:20:40] [V] [TRT] ModelImporter.cpp:119: Searching for input: map_while_map_while_loop_counter:0
[03/29/2021-15:20:40] [V] [TRT] ModelImporter.cpp:125: cond___map/while/Less_1__4 [Cast] inputs: [map_while_map_while_loop_counter:0 → (1)],
[03/29/2021-15:20:40] [V] [TRT] builtin_op_importers.cpp:320: Casting to type: float32
[03/29/2021-15:20:40] [V] [TRT] ImporterContext.hpp:154: Registering layer: cond___map/while/Less_1__4 for ONNX node: cond___map/while/Less_1__4
[03/29/2021-15:20:40] [V] [TRT] ImporterContext.hpp:120: Registering tensor: cond___map/while/Less_1__4:0 for ONNX tensor: cond___map/while/Less_1__4:0
[03/29/2021-15:20:40] [V] [TRT] ModelImporter.cpp:179: cond___map/while/Less_1__4 [Cast] outputs: [cond___map/while/Less_1__4:0 → (1)],
[03/29/2021-15:20:40] [V] [TRT] ModelImporter.cpp:103: Parsing node: cond___map/while/Less_1 [Less]
[03/29/2021-15:20:40] [V] [TRT] ModelImporter.cpp:119: Searching for input: cond___map/while/Less_1__4:0
[03/29/2021-15:20:40] [V] [TRT] ModelImporter.cpp:119: Searching for input: const_fold_opt__4088
[03/29/2021-15:20:40] [V] [TRT] ModelImporter.cpp:125: cond___map/while/Less_1 [Less] inputs: [cond___map/while/Less_1__4:0 → (1)], [const_fold_opt__4088 → ()],
[03/29/2021-15:20:40] [V] [TRT] ImporterContext.hpp:150: Registering constant layer: const_fold_opt__4088 for ONNX initializer: const_fold_opt__4088
[03/29/2021-15:20:40] [V] [TRT] ImporterContext.hpp:154: Registering layer: cond___map/while/Less_1 for ONNX node: cond___map/while/Less_1
[03/29/2021-15:20:40] [V] [TRT] ImporterContext.hpp:120: Registering tensor: cond___map/while/Less_1:0 for ONNX tensor: cond___map/while/Less_1:0
[03/29/2021-15:20:40] [V] [TRT] ModelImporter.cpp:179: cond___map/while/Less_1 [Less] outputs: [cond___map/while/Less_1:0 → (1)],
[03/29/2021-15:20:40] [V] [TRT] ModelImporter.cpp:103: Parsing node: cond___map/while/Less [Less]
[03/29/2021-15:20:40] [V] [TRT] ModelImporter.cpp:119: Searching for input: cond___map/while/Less__6:0
[03/29/2021-15:20:40] [V] [TRT] ModelImporter.cpp:119: Searching for input: const_fold_opt__3968
[03/29/2021-15:20:40] [V] [TRT] ModelImporter.cpp:125: cond___map/while/Less [Less] inputs: [cond___map/while/Less__6:0 → ()], [const_fold_opt__3968 → ()],
[03/29/2021-15:20:40] [V] [TRT] ImporterContext.hpp:150: Registering constant layer: const_fold_opt__3968 for ONNX initializer: const_fold_opt__3968
[03/29/2021-15:20:40] [V] [TRT] ImporterContext.hpp:154: Registering layer: cond___map/while/Less for ONNX node: cond___map/while/Less
[03/29/2021-15:20:40] [V] [TRT] ImporterContext.hpp:120: Registering tensor: cond___map/while/Less:0 for ONNX tensor: cond___map/while/Less:0
[03/29/2021-15:20:40] [V] [TRT] ModelImporter.cpp:179: cond___map/while/Less [Less] outputs: [cond___map/while/Less:0 → ()],
[03/29/2021-15:20:40] [V] [TRT] ModelImporter.cpp:103: Parsing node: cond___map/while/LogicalAnd [And]
[03/29/2021-15:20:40] [V] [TRT] ModelImporter.cpp:119: Searching for input: cond___map/while/Less_1:0
[03/29/2021-15:20:40] [V] [TRT] ModelImporter.cpp:119: Searching for input: cond___map/while/Less:0
[03/29/2021-15:20:40] [V] [TRT] ModelImporter.cpp:125: cond___map/while/LogicalAnd [And] inputs: [cond___map/while/Less_1:0 → (1)], [cond___map/while/Less:0 → ()],
[03/29/2021-15:20:40] [V] [TRT] ImporterContext.hpp:154: Registering layer: cond___map/while/LogicalAnd for ONNX node: cond___map/while/LogicalAnd
[03/29/2021-15:20:40] [V] [TRT] ImporterContext.hpp:120: Registering tensor: cond___map/while/Identity:0 for ONNX tensor: cond___map/while/Identity:0
[03/29/2021-15:20:40] [V] [TRT] ModelImporter.cpp:179: cond___map/while/LogicalAnd [And] outputs: [cond___map/while/Identity:0 → (1)],
[03/29/2021-15:20:40] [V] [TRT] ModelImporter.cpp:103: Parsing node: Transpose__3107 [Transpose]
[03/29/2021-15:20:40] [V] [TRT] ModelImporter.cpp:119: Searching for input: input_tensor:0
[03/29/2021-15:20:40] [V] [TRT] ModelImporter.cpp:125: Transpose__3107 [Transpose] inputs: [input_tensor:0 → (1, 3, 1024, 1024)],
[03/29/2021-15:20:40] [V] [TRT] ImporterContext.hpp:154: Registering layer: Transpose__3107 for ONNX node: Transpose__3107
[03/29/2021-15:20:40] [V] [TRT] ImporterContext.hpp:120: Registering tensor: Transpose__3107:0 for ONNX tensor: Transpose__3107:0
[03/29/2021-15:20:40] [V] [TRT] ModelImporter.cpp:179: Transpose__3107 [Transpose] outputs: [Transpose__3107:0 → (1, 1024, 1024, 3)],
[03/29/2021-15:20:40] [V] [TRT] ModelImporter.cpp:103: Parsing node: map/while/TensorArrayV2Read/TensorListGetItem [Gather]
[03/29/2021-15:20:40] [V] [TRT] ModelImporter.cpp:119: Searching for input: Transpose__3107:0
[03/29/2021-15:20:40] [V] [TRT] ModelImporter.cpp:119: Searching for input: map_while_placeholder:0
[03/29/2021-15:20:40] [V] [TRT] ModelImporter.cpp:125: map/while/TensorArrayV2Read/TensorListGetItem [Gather] inputs: [Transpose__3107:0 → (1, 1024, 1024, 3)], [map_while_placeholder:0 → ()],
[03/29/2021-15:20:40] [V] [TRT] builtin_op_importers.cpp:996: Using Gather axis: 0
[03/29/2021-15:20:40] [V] [TRT] ImporterContext.hpp:154: Registering layer: map/while/TensorArrayV2Read/TensorListGetItem for ONNX node: map/while/TensorArrayV2Read/TensorListGetItem
[03/29/2021-15:20:40] [V] [TRT] ImporterContext.hpp:120: Registering tensor: map/while/TensorArrayV2Read/TensorListGetItem:0 for ONNX tensor: map/while/TensorArrayV2Read/TensorListGetItem:0
[03/29/2021-15:20:40] [V] [TRT] ModelImporter.cpp:179: map/while/TensorArrayV2Read/TensorListGetItem [Gather] outputs: [map/while/TensorArrayV2Read/TensorListGetItem:0 → (1024, 1024, 3)],
[03/29/2021-15:20:40] [V] [TRT] ModelImporter.cpp:103: Parsing node: map/while/Cast [Cast]
[03/29/2021-15:20:40] [V] [TRT] ModelImporter.cpp:119: Searching for input: map/while/TensorArrayV2Read/TensorListGetItem:0
[03/29/2021-15:20:40] [V] [TRT] ModelImporter.cpp:125: map/while/Cast [Cast] inputs: [map/while/TensorArrayV2Read/TensorListGetItem:0 → (1024, 1024, 3)],
[03/29/2021-15:20:40] [V] [TRT] builtin_op_importers.cpp:320: Casting to type: float32
[03/29/2021-15:20:40] [V] [TRT] ImporterContext.hpp:154: Registering layer: map/while/Cast for ONNX node: map/while/Cast
[03/29/2021-15:20:40] [V] [TRT] ImporterContext.hpp:120: Registering tensor: map/while/Cast:0 for ONNX tensor: map/while/Cast:0
[03/29/2021-15:20:40] [V] [TRT] ModelImporter.cpp:179: map/while/Cast [Cast] outputs: [map/while/Cast:0 → (1024, 1024, 3)],
[03/29/2021-15:20:40] [V] [TRT] ModelImporter.cpp:103: Parsing node: Unsqueeze__80 [Unsqueeze]
[03/29/2021-15:20:40] [V] [TRT] ModelImporter.cpp:119: Searching for input: map/while/Cast:0
[03/29/2021-15:20:40] [V] [TRT] ModelImporter.cpp:119: Searching for input: const_axes__79
[03/29/2021-15:20:40] [V] [TRT] ModelImporter.cpp:125: Unsqueeze__80 [Unsqueeze] inputs: [map/while/Cast:0 → (1024, 1024, 3)], [const_axes__79 → (1)],

NVES · March 29, 2021, 2:37pm

Hi,
Request you to share the ONNX model and the script if not shared already so that we can assist you better.
Alongside you can try few things:

validating your model with the below snippet

check_model.py

import sys
import onnx
filename = yourONNXmodel
model = onnx.load(filename)
onnx.checker.check_model(model).
2) Try running your model with trtexec command.
https://github.com/NVIDIA/TensorRT/tree/master/samples/opensource/trtexec
In case you are still facing issue, request you to share the trtexec “”–verbose"" log for further debugging
Thanks!

spolisetty · March 30, 2021, 7:28am

Hi @Niklas.Flindt,

ONNX opset version 13 support will be available in future releases, we request you to try on opset version 11.

Thank you.

Niklas.Flindt · March 30, 2021, 7:52am

Hi,

I did what you proposed and the initial model I posted did not work with the model_checker.
After some investigation, it appears that I used a TF1 version environment to convert from tf to onnx. That did not work as expected, obviously.

Now, I am using the TF 2.4.1 in my environment and the newest onnx and tf2onnx version.
The conversion to onnx works without errors.
Afterwards, I had to use onnx graphsurgeon to define the input and output shapes and change the input type to float32.

After running the conversion of tf2onnx with --ops=11, I finally got a “nice” error from trtexec.exe:

[03/30/2021-09:40:55] [V] [TRT] ModelImporter.cpp:125: StatefulPartitionedCall/Preprocessor/ResizeToRange/cond [If] inputs: [StatefulPartitionedCall/Preprocessor/ResizeToRange/Less:0 → ()],
ERROR: builtin_op_importers.cpp:1602 In function importIf:
[8] Assertion failed: cond.is_weights() && cond.weights().count() == 1 && “If condition must be a initializer!”

I will continue to look into that, but any help is highly appreciated :)

spolisetty · March 31, 2021, 11:26am

Hi @Niklas.Flindt,

We request you to share latest ONNX model/scripts to reproduce the issue for better assistance.

Thank you.

Niklas.Flindt · March 31, 2021, 11:29am

Hi,

the onnx model can be found as download from dropbox in the initial question.

Thanks!

raphael_zingg · April 16, 2021, 9:22pm

Hi Niklas

I had a similar issue, you can find information about what causes it in this git issue:

What i had to do was to replace the upsampling2d call (i think efficientdet-d4 also has those) with this patched version before converting to .onnx (tf 2.4.1):

from tensorflow.python.ops.gen_image_ops import resize_nearest_neighbor

# define your model here

def patched_UpSampling2D_call(self, inputs):
    _, height, width, _ = inputs.get_shape().as_list()
    if height != None:
        # fixed call in order to avoid trt conversion error
        new_height, new_width = height * 2, width * 2
        size_tensor = tf.convert_to_tensor(np.array([new_height, new_width], dtype=np.int32), dtype=tf.int32)
        inputs = resize_nearest_neighbor(inputs, size=size_tensor, align_corners=False, half_pixel_centers=False)
        inputs.set_shape([None, new_height, new_width, None])
        return inputs
    else:
        # original call
        return backend.resize_images(
        inputs, self.size[0], self.size[1], self.data_format,
        interpolation=self.interpolation)

tf.keras.layers.UpSampling2D.call = patched_UpSampling2D_call
model.save(CHANGE_SAVED_MODEL_PATH, include_optimizer=False)

spolisetty · April 22, 2021, 1:24pm

Hi @Niklas.Flindt,

We tried reproducing the issue, but unfortunately facing some errors related to plugins. Could you please share all files required and steps.

Thank you.

romilaggarwal611 · June 11, 2021, 6:41pm

Hi,
I’m facing similar issues. Do you have any workarounds?
onnx to trt with trtexec failed

thank you!

romilaggarwal611 · June 11, 2021, 6:46pm

Hi @raphael_zingg

This funtion looks like would do the trick. However I’m using a pretrained model. Can you suggest what changes could be made to run it? I think with a combination of multiplication op etc it could be done with onnx_graphsurgeon. But, I’m having problem in figuring out exactly how.
github link to the issue

Thank you!!

Topic		Replies	Views
Trtexec cannot convert QAT onnx model to trt model TensorRT tensorrt	7	1173	January 26, 2023
[onnx->TensorRT]Parser problem about padding TensorRT	5	2124	October 19, 2021
BUG: Output TRT engine from trtexec has completely different inference than input model TensorRT tensorrt , debugging-and-troubleshooting	3	2226	January 4, 2022
Trtexec "Failed to sort the model topologically." TensorRT tensorrt , cudnn , onnx	2	599	March 5, 2024
Tensorrt8.2 convert .onnx model to .trt ,throw "two inputs (data and weights) are allowed only in explicit-quantization mode"error TensorRT	9	1957	April 6, 2022
Trtexec conversion to int8 Jetson AGX Orin tensorrt	2	1062	October 11, 2023
Floor - Cast - Resize(or Slice) cause internal error TensorRT tensorrt	7	2019	January 12, 2022
Whats the shapes in Autotuning format combination part mean using TensorRT with trtexec? TensorRT tensorrt	7	1417	October 12, 2021
Convert onnx to trt format using trtexec TensorRT tensorrt	5	5371	August 17, 2021
Converting onnx model to trt model (multi view CNN) Jetson Nano tensorrt , onnx	4	567	October 15, 2021

Running trtexec on Google's EfficientDet-D4 in ONNX Format

Description

Environment

Relevant Files

Steps To Reproduce

check_model.py

Related topics