Running trtexec on Google's EfficientDet-D4 in ONNX Format

Description

I am trying to run the official EfficientDet-D4 in TensorRT. It just won’t work.
Prior to that, I am using tf2onnx.convert to convert the TF saved-model to onnx.

When running trtexec on the onnx file it results in no traceback at all.
If I am using “verbose” logging, I at least get the information where the import of the model stops but there it still no real traceback.
It seems that he “Unsqueeze” is the problem, but I cannot see why.

Any help would be appreciated. I really want to use TensorRT, but for that it is necessary that the model is working, ofc.

Let me know if any additional information is needed!
I had to crop the traceback/verbose output to match the maximal thread size…

Environment

TensorRT Version: 7.2.3.4
GPU Type: RTX3090
Nvidia Driver Version: 27.21.14.5148
CUDA Version: 11.2
CUDNN Version: 8.0
Operating System + Version: Windows
Python Version: 3.8.8
TensorFlow Version (if applicable): 2.4.1
ONNX_Tensorflow Version: 1.9
Onnx Version: 1.8.1

Relevant Files

The onnx file can be found here:

*Updated File: Dropbox - onnx_nchw_float.onnx - Simplify your life

Steps To Reproduce

Simply use
trtexec.exe --onnx=onnx_model.onnx --verbose

Please include:

  • Exact steps/commands to build your repro
  • Exact steps/commands to run your repro
  • Full traceback of errors encountered

I encounter the following traceback:

(tf24) C:\Repos\TensorRT-7.2.3.4\bin>trtexec.exe --onnx=…\saved_model\onnx_float.onnx --verbose
&&&& RUNNING TensorRT.trtexec # trtexec.exe --onnx=…\saved_model\onnx_float.onnx --verbose
[03/29/2021-15:20:06] [I] === Model Options ===
[03/29/2021-15:20:06] [I] Format: ONNX
[03/29/2021-15:20:06] [I] Model: …\saved_model\onnx_float.onnx
[03/29/2021-15:20:06] [I] Output:
[03/29/2021-15:20:06] [I] === Build Options ===
[03/29/2021-15:20:06] [I] Max batch: explicit
[03/29/2021-15:20:06] [I] Workspace: 16 MiB
[03/29/2021-15:20:06] [I] minTiming: 1
[03/29/2021-15:20:06] [I] avgTiming: 8
[03/29/2021-15:20:06] [I] Precision: FP32
[03/29/2021-15:20:06] [I] Calibration:
[03/29/2021-15:20:06] [I] Refit: Disabled
[03/29/2021-15:20:06] [I] Safe mode: Disabled
[03/29/2021-15:20:06] [I] Save engine:
[03/29/2021-15:20:06] [I] Load engine:
[03/29/2021-15:20:06] [I] Builder Cache: Enabled
[03/29/2021-15:20:06] [I] NVTX verbosity: 0
[03/29/2021-15:20:06] [I] Tactic sources: Using default tactic sources
[03/29/2021-15:20:06] [I] Input(s)s format: fp32:CHW
[03/29/2021-15:20:06] [I] Output(s)s format: fp32:CHW
[03/29/2021-15:20:06] [I] Input build shapes: model
[03/29/2021-15:20:06] [I] Input calibration shapes: model
[03/29/2021-15:20:06] [I] === System Options ===
[03/29/2021-15:20:06] [I] Device: 0
[03/29/2021-15:20:06] [I] DLACore:
[03/29/2021-15:20:06] [I] Plugins:
[03/29/2021-15:20:06] [I] === Inference Options ===
[03/29/2021-15:20:06] [I] Batch: Explicit
[03/29/2021-15:20:06] [I] Input inference shapes: model
[03/29/2021-15:20:06] [I] Iterations: 10
[03/29/2021-15:20:06] [I] Duration: 3s (+ 200ms warm up)
[03/29/2021-15:20:06] [I] Sleep time: 0ms
[03/29/2021-15:20:06] [I] Streams: 1
[03/29/2021-15:20:06] [I] ExposeDMA: Disabled
[03/29/2021-15:20:06] [I] Data transfers: Enabled
[03/29/2021-15:20:06] [I] Spin-wait: Disabled
[03/29/2021-15:20:06] [I] Multithreading: Disabled
[03/29/2021-15:20:06] [I] CUDA Graph: Disabled
[03/29/2021-15:20:06] [I] Separate profiling: Disabled
[03/29/2021-15:20:06] [I] Skip inference: Disabled
[03/29/2021-15:20:06] [I] Inputs:
[03/29/2021-15:20:06] [I] === Reporting Options ===
[03/29/2021-15:20:06] [I] Verbose: Enabled
[03/29/2021-15:20:06] [I] Averages: 10 inferences
[03/29/2021-15:20:06] [I] Percentile: 99
[03/29/2021-15:20:06] [I] Dump refittable layers:Disabled
[03/29/2021-15:20:07] [I] Dump output: Disabled
[03/29/2021-15:20:07] [I] Profile: Disabled
[03/29/2021-15:20:07] [I] Export timing to JSON file:
[03/29/2021-15:20:07] [I] Export output to JSON file:
[03/29/2021-15:20:07] [I] Export profile to JSON file:
[03/29/2021-15:20:07] [I]
[03/29/2021-15:20:07] [I] === Device Information ===
[03/29/2021-15:20:07] [I] Selected Device: GeForce RTX 3090
[03/29/2021-15:20:07] [I] Compute Capability: 8.6
[03/29/2021-15:20:07] [I] SMs: 82
[03/29/2021-15:20:07] [I] Compute Clock Rate: 1.86 GHz
[03/29/2021-15:20:07] [I] Device Global Memory: 24576 MiB
[03/29/2021-15:20:07] [I] Shared Memory per SM: 100 KiB
[03/29/2021-15:20:07] [I] Memory Bus Width: 384 bits (ECC disabled)
[03/29/2021-15:20:07] [I] Memory Clock Rate: 9.751 GHz
[03/29/2021-15:20:07] [I]
[03/29/2021-15:20:07] [V] [TRT] Registered plugin creator - ::GridAnchor_TRT version 1
[03/29/2021-15:20:07] [V] [TRT] Registered plugin creator - ::NMS_TRT version 1
[03/29/2021-15:20:07] [V] [TRT] Registered plugin creator - ::Reorg_TRT version 1
[03/29/2021-15:20:07] [V] [TRT] Registered plugin creator - ::Region_TRT version 1
[03/29/2021-15:20:07] [V] [TRT] Registered plugin creator - ::Clip_TRT version 1
[03/29/2021-15:20:07] [V] [TRT] Registered plugin creator - ::LReLU_TRT version 1
[03/29/2021-15:20:07] [V] [TRT] Registered plugin creator - ::PriorBox_TRT version 1
[03/29/2021-15:20:07] [V] [TRT] Registered plugin creator - ::Normalize_TRT version 1
[03/29/2021-15:20:07] [V] [TRT] Registered plugin creator - ::RPROI_TRT version 1
[03/29/2021-15:20:07] [V] [TRT] Registered plugin creator - ::BatchedNMS_TRT version 1
[03/29/2021-15:20:07] [V] [TRT] Registered plugin creator - ::BatchedNMSDynamic_TRT version 1
[03/29/2021-15:20:07] [V] [TRT] Registered plugin creator - ::FlattenConcat_TRT version 1
[03/29/2021-15:20:07] [V] [TRT] Registered plugin creator - ::CropAndResize version 1
[03/29/2021-15:20:07] [V] [TRT] Registered plugin creator - ::DetectionLayer_TRT version 1
[03/29/2021-15:20:07] [V] [TRT] Registered plugin creator - ::Proposal version 1
[03/29/2021-15:20:07] [V] [TRT] Registered plugin creator - ::ProposalLayer_TRT version 1
[03/29/2021-15:20:07] [V] [TRT] Registered plugin creator - ::PyramidROIAlign_TRT version 1
[03/29/2021-15:20:07] [V] [TRT] Registered plugin creator - ::ResizeNearest_TRT version 1
[03/29/2021-15:20:07] [V] [TRT] Registered plugin creator - ::Split version 1
[03/29/2021-15:20:07] [V] [TRT] Registered plugin creator - ::SpecialSlice_TRT version 1
[03/29/2021-15:20:07] [V] [TRT] Registered plugin creator - ::InstanceNormalization_TRT version 1

Input filename: Y:\Flindt\Data\Datasets\Objectdetection\IHFood\Front\Objectdetection\EfficientDet4\models\my_model_dir2\final_model\saved_model\onnx_float.onnx
ONNX IR version: 0.0.7
Opset version: 13
Producer name:
Producer version:
Domain:
Model version: 0
Doc string:

[03/29/2021-15:20:14] [V] [TRT] Plugin creator already registered - ::GridAnchor_TRT version 1
[03/29/2021-15:20:14] [V] [TRT] Plugin creator already registered - ::NMS_TRT version 1
[03/29/2021-15:20:14] [V] [TRT] Plugin creator already registered - ::Reorg_TRT version 1
[03/29/2021-15:20:14] [V] [TRT] Plugin creator already registered - ::Region_TRT version 1
[03/29/2021-15:20:14] [V] [TRT] Plugin creator already registered - ::Clip_TRT version 1
[03/29/2021-15:20:14] [V] [TRT] Plugin creator already registered - ::LReLU_TRT version 1
[03/29/2021-15:20:14] [V] [TRT] Plugin creator already registered - ::PriorBox_TRT version 1
[03/29/2021-15:20:14] [V] [TRT] Plugin creator already registered - ::Normalize_TRT version 1
[03/29/2021-15:20:14] [V] [TRT] Plugin creator already registered - ::RPROI_TRT version 1
[03/29/2021-15:20:14] [V] [TRT] Plugin creator already registered - ::BatchedNMS_TRT version 1
[03/29/2021-15:20:14] [V] [TRT] Plugin creator already registered - ::BatchedNMSDynamic_TRT version 1
[03/29/2021-15:20:14] [V] [TRT] Plugin creator already registered - ::FlattenConcat_TRT version 1
[03/29/2021-15:20:14] [V] [TRT] Plugin creator already registered - ::CropAndResize version 1
[03/29/2021-15:20:14] [V] [TRT] Plugin creator already registered - ::DetectionLayer_TRT version 1
[03/29/2021-15:20:14] [V] [TRT] Plugin creator already registered - ::Proposal version 1
[03/29/2021-15:20:14] [V] [TRT] Plugin creator already registered - ::ProposalLayer_TRT version 1
[03/29/2021-15:20:14] [V] [TRT] Plugin creator already registered - ::PyramidROIAlign_TRT version 1
[03/29/2021-15:20:14] [V] [TRT] Plugin creator already registered - ::ResizeNearest_TRT version 1
[03/29/2021-15:20:14] [V] [TRT] Plugin creator already registered - ::Split version 1
[03/29/2021-15:20:14] [V] [TRT] Plugin creator already registered - ::SpecialSlice_TRT version 1
[03/29/2021-15:20:14] [V] [TRT] Plugin creator already registered - ::InstanceNormalization_TRT version 1
[03/29/2021-15:20:14] [V] [TRT] ModelImporter.cpp:202: Adding network input: input_tensor:0 with dtype: float32, dimensions: (1, 3, 1024, 1024)
[03/29/2021-15:20:14] [V] [TRT] ImporterContext.hpp:120: Registering tensor: input_tensor:0 for ONNX tensor: input_tensor:0
[03/29/2021-15:20:14] [V] [TRT] ModelImporter.cpp:90: Importing initializer: StatefulPartitionedCall/Postprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/ClipToWindow/Greater/y:0
[03/29/2021-15:20:14] [V] [TRT] ModelImporter.cpp:90: Importing initializer: const_fold_opt__3869
[03/29/2021-15:20:15] [V] [TRT] ModelImporter.cpp:90: Importing initializer: StatefulPartitionedCall/map/while/maximum_iterations:0
[03/29/2021-15:20:15] [W] [TRT] onnx2trt_utils.cpp:220: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[03/29/2021-15:20:15] [V] [TRT] ModelImporter.cpp:90: Importing initializer: StatefulPartitionedCall/Postprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/range_2/start:0
[03/29/2021-15:20:15] [V] [TRT] ModelImporter.cpp:90: Importing initializer: StatefulPartitionedCall/map/TensorArrayV2Stack_1/TensorListStack/element_shape:0
[03/29/2021-15:20:15] [V] [TRT] ModelImporter.cpp:90: Importing initializer: const_starts__2943
[03/29/2021-15:20:15] [V] [TRT] ModelImporter.cpp:90: Importing initializer: StatefulPartitionedCall/map/TensorArrayV2Stack/TensorListStack/element_shape:0
[03/29/2021-15:20:15] [V] [TRT] ModelImporter.cpp:90: Importing initializer: StatefulPartitionedCall/EfficientDet-D4/model/stem_conv2d/Conv2D_weights_fused_bn
[03/29/2021-15:20:15] [V] [TRT] ModelImporter.cpp:90: Importing initializer: StatefulPartitionedCall/EfficientDet-D4/model/stem_conv2d/Conv2D_bias_fused_bn
[03/29/2021-15:20:15] [V] [TRT] ModelImporter.cpp:90: Importing initializer: StatefulPartitionedCall/EfficientDet-D4/model/stack_0/block_0/depthwise_conv2d/depthwise_weights_fused_bn
[03/29/2021-15:20:15] [V] [TRT] ModelImporter.cpp:90: Importing initializer: StatefulPartitionedCall/EfficientDet-D4/model/stack_0/block_0/depthwise_conv2d/depthwise_bias_fused_bn
[03/29/2021-15:20:15] [V] [TRT] ModelImporter.cpp:90: Importing initializer: const_fold_opt__3754
[03/29/2021-15:20:15] [V] [TRT] ModelImporter.cpp:90: Importing initializer: new_shape__3682
[03/29/2021-15:20:15] [V] [TRT] ModelImporter.cpp:90: Importing initializer: StatefulPartitionedCall/EfficientDet-D4/model/stack_0/block_0/se_reduce_conv2d/Conv2D/ReadVariableOp:0
[03/29/2021-15:20:15] [V] [TRT] ModelImporter.cpp:90: Importing initializer: StatefulPartitionedCall/EfficientDet-D4/model/stack_0/block_0/se_reduce_conv2d/BiasAdd/ReadVariableOp:0
[03/29/2021-15:20:15] [V] [TRT] ModelImporter.cpp:90: Importing initializer: StatefulPartitionedCall/EfficientDet-D4/model/stack_0/block_0/se_expand_conv2d/Conv2D/ReadVariableOp:0
[03/29/2021-15:20:15] [V] [TRT] ModelImporter.cpp:90: Importing initializer: StatefulPartitionedCall/EfficientDet-D4/model/stack_0/block_0/se_expand_conv2d/BiasAdd/ReadVariableOp:0
[03/29/2021-15:20:15] [V] [TRT] ModelImporter.cpp:90: Importing initializer: StatefulPartitionedCall/EfficientDet-D4/model/stack_0/block_0/project_conv2d/Conv2D_weights_fused_bn

[03/29/2021-15:20:39] [V] [TRT] ModelImporter.cpp:90: Importing initializer: const_axes__196
[03/29/2021-15:20:39] [V] [TRT] ModelImporter.cpp:90: Importing initializer: const_fold_opt__3976
[03/29/2021-15:20:39] [V] [TRT] ModelImporter.cpp:90: Importing initializer: const_starts__206
[03/29/2021-15:20:39] [V] [TRT] ModelImporter.cpp:90: Importing initializer: const_ends__207
[03/29/2021-15:20:39] [V] [TRT] ModelImporter.cpp:90: Importing initializer: const_axes__208
[03/29/2021-15:20:39] [V] [TRT] ModelImporter.cpp:90: Importing initializer: const_axes__210
[03/29/2021-15:20:39] [V] [TRT] ModelImporter.cpp:103: Parsing node: map/while/add [Add]
[03/29/2021-15:20:39] [V] [TRT] ModelImporter.cpp:119: Searching for input: map_while_placeholder:0
[03/29/2021-15:20:39] [V] [TRT] ModelImporter.cpp:119: Searching for input: map/while/add/y:0
[03/29/2021-15:20:39] [V] [TRT] ModelImporter.cpp:125: map/while/add [Add] inputs: [map_while_placeholder:0 → ()], [map/while/add/y:0 → ()],
[03/29/2021-15:20:39] [V] [TRT] ImporterContext.hpp:150: Registering constant layer: map/while/add/y:0 for ONNX initializer: map/while/add/y:0
[03/29/2021-15:20:39] [V] [TRT] ImporterContext.hpp:154: Registering layer: map/while/add for ONNX node: map/while/add
[03/29/2021-15:20:39] [V] [TRT] ImporterContext.hpp:120: Registering tensor: map/while/Identity_2:0 for ONNX tensor: map/while/Identity_2:0
[03/29/2021-15:20:39] [V] [TRT] ModelImporter.cpp:179: map/while/add [Add] outputs: [map/while/Identity_2:0 → ()],
[03/29/2021-15:20:39] [V] [TRT] ModelImporter.cpp:103: Parsing node: cond___map/while/Less__6 [Cast]
[03/29/2021-15:20:39] [V] [TRT] ModelImporter.cpp:119: Searching for input: map/while/Identity_2:0
[03/29/2021-15:20:39] [V] [TRT] ModelImporter.cpp:125: cond___map/while/Less__6 [Cast] inputs: [map/while/Identity_2:0 → ()],
[03/29/2021-15:20:39] [V] [TRT] builtin_op_importers.cpp:320: Casting to type: float32
[03/29/2021-15:20:39] [V] [TRT] ImporterContext.hpp:154: Registering layer: cond___map/while/Less__6 for ONNX node: cond___map/while/Less__6
[03/29/2021-15:20:40] [V] [TRT] ImporterContext.hpp:120: Registering tensor: cond___map/while/Less__6:0 for ONNX tensor: cond___map/while/Less__6:0
[03/29/2021-15:20:40] [V] [TRT] ModelImporter.cpp:179: cond___map/while/Less__6 [Cast] outputs: [cond___map/while/Less__6:0 → ()],
[03/29/2021-15:20:40] [V] [TRT] ModelImporter.cpp:103: Parsing node: cond___map/while/Less_1__4 [Cast]
[03/29/2021-15:20:40] [V] [TRT] ModelImporter.cpp:119: Searching for input: map_while_map_while_loop_counter:0
[03/29/2021-15:20:40] [V] [TRT] ModelImporter.cpp:125: cond___map/while/Less_1__4 [Cast] inputs: [map_while_map_while_loop_counter:0 → (1)],
[03/29/2021-15:20:40] [V] [TRT] builtin_op_importers.cpp:320: Casting to type: float32
[03/29/2021-15:20:40] [V] [TRT] ImporterContext.hpp:154: Registering layer: cond___map/while/Less_1__4 for ONNX node: cond___map/while/Less_1__4
[03/29/2021-15:20:40] [V] [TRT] ImporterContext.hpp:120: Registering tensor: cond___map/while/Less_1__4:0 for ONNX tensor: cond___map/while/Less_1__4:0
[03/29/2021-15:20:40] [V] [TRT] ModelImporter.cpp:179: cond___map/while/Less_1__4 [Cast] outputs: [cond___map/while/Less_1__4:0 → (1)],
[03/29/2021-15:20:40] [V] [TRT] ModelImporter.cpp:103: Parsing node: cond___map/while/Less_1 [Less]
[03/29/2021-15:20:40] [V] [TRT] ModelImporter.cpp:119: Searching for input: cond___map/while/Less_1__4:0
[03/29/2021-15:20:40] [V] [TRT] ModelImporter.cpp:119: Searching for input: const_fold_opt__4088
[03/29/2021-15:20:40] [V] [TRT] ModelImporter.cpp:125: cond___map/while/Less_1 [Less] inputs: [cond___map/while/Less_1__4:0 → (1)], [const_fold_opt__4088 → ()],
[03/29/2021-15:20:40] [V] [TRT] ImporterContext.hpp:150: Registering constant layer: const_fold_opt__4088 for ONNX initializer: const_fold_opt__4088
[03/29/2021-15:20:40] [V] [TRT] ImporterContext.hpp:154: Registering layer: cond___map/while/Less_1 for ONNX node: cond___map/while/Less_1
[03/29/2021-15:20:40] [V] [TRT] ImporterContext.hpp:120: Registering tensor: cond___map/while/Less_1:0 for ONNX tensor: cond___map/while/Less_1:0
[03/29/2021-15:20:40] [V] [TRT] ModelImporter.cpp:179: cond___map/while/Less_1 [Less] outputs: [cond___map/while/Less_1:0 → (1)],
[03/29/2021-15:20:40] [V] [TRT] ModelImporter.cpp:103: Parsing node: cond___map/while/Less [Less]
[03/29/2021-15:20:40] [V] [TRT] ModelImporter.cpp:119: Searching for input: cond___map/while/Less__6:0
[03/29/2021-15:20:40] [V] [TRT] ModelImporter.cpp:119: Searching for input: const_fold_opt__3968
[03/29/2021-15:20:40] [V] [TRT] ModelImporter.cpp:125: cond___map/while/Less [Less] inputs: [cond___map/while/Less__6:0 → ()], [const_fold_opt__3968 → ()],
[03/29/2021-15:20:40] [V] [TRT] ImporterContext.hpp:150: Registering constant layer: const_fold_opt__3968 for ONNX initializer: const_fold_opt__3968
[03/29/2021-15:20:40] [V] [TRT] ImporterContext.hpp:154: Registering layer: cond___map/while/Less for ONNX node: cond___map/while/Less
[03/29/2021-15:20:40] [V] [TRT] ImporterContext.hpp:120: Registering tensor: cond___map/while/Less:0 for ONNX tensor: cond___map/while/Less:0
[03/29/2021-15:20:40] [V] [TRT] ModelImporter.cpp:179: cond___map/while/Less [Less] outputs: [cond___map/while/Less:0 → ()],
[03/29/2021-15:20:40] [V] [TRT] ModelImporter.cpp:103: Parsing node: cond___map/while/LogicalAnd [And]
[03/29/2021-15:20:40] [V] [TRT] ModelImporter.cpp:119: Searching for input: cond___map/while/Less_1:0
[03/29/2021-15:20:40] [V] [TRT] ModelImporter.cpp:119: Searching for input: cond___map/while/Less:0
[03/29/2021-15:20:40] [V] [TRT] ModelImporter.cpp:125: cond___map/while/LogicalAnd [And] inputs: [cond___map/while/Less_1:0 → (1)], [cond___map/while/Less:0 → ()],
[03/29/2021-15:20:40] [V] [TRT] ImporterContext.hpp:154: Registering layer: cond___map/while/LogicalAnd for ONNX node: cond___map/while/LogicalAnd
[03/29/2021-15:20:40] [V] [TRT] ImporterContext.hpp:120: Registering tensor: cond___map/while/Identity:0 for ONNX tensor: cond___map/while/Identity:0
[03/29/2021-15:20:40] [V] [TRT] ModelImporter.cpp:179: cond___map/while/LogicalAnd [And] outputs: [cond___map/while/Identity:0 → (1)],
[03/29/2021-15:20:40] [V] [TRT] ModelImporter.cpp:103: Parsing node: Transpose__3107 [Transpose]
[03/29/2021-15:20:40] [V] [TRT] ModelImporter.cpp:119: Searching for input: input_tensor:0
[03/29/2021-15:20:40] [V] [TRT] ModelImporter.cpp:125: Transpose__3107 [Transpose] inputs: [input_tensor:0 → (1, 3, 1024, 1024)],
[03/29/2021-15:20:40] [V] [TRT] ImporterContext.hpp:154: Registering layer: Transpose__3107 for ONNX node: Transpose__3107
[03/29/2021-15:20:40] [V] [TRT] ImporterContext.hpp:120: Registering tensor: Transpose__3107:0 for ONNX tensor: Transpose__3107:0
[03/29/2021-15:20:40] [V] [TRT] ModelImporter.cpp:179: Transpose__3107 [Transpose] outputs: [Transpose__3107:0 → (1, 1024, 1024, 3)],
[03/29/2021-15:20:40] [V] [TRT] ModelImporter.cpp:103: Parsing node: map/while/TensorArrayV2Read/TensorListGetItem [Gather]
[03/29/2021-15:20:40] [V] [TRT] ModelImporter.cpp:119: Searching for input: Transpose__3107:0
[03/29/2021-15:20:40] [V] [TRT] ModelImporter.cpp:119: Searching for input: map_while_placeholder:0
[03/29/2021-15:20:40] [V] [TRT] ModelImporter.cpp:125: map/while/TensorArrayV2Read/TensorListGetItem [Gather] inputs: [Transpose__3107:0 → (1, 1024, 1024, 3)], [map_while_placeholder:0 → ()],
[03/29/2021-15:20:40] [V] [TRT] builtin_op_importers.cpp:996: Using Gather axis: 0
[03/29/2021-15:20:40] [V] [TRT] ImporterContext.hpp:154: Registering layer: map/while/TensorArrayV2Read/TensorListGetItem for ONNX node: map/while/TensorArrayV2Read/TensorListGetItem
[03/29/2021-15:20:40] [V] [TRT] ImporterContext.hpp:120: Registering tensor: map/while/TensorArrayV2Read/TensorListGetItem:0 for ONNX tensor: map/while/TensorArrayV2Read/TensorListGetItem:0
[03/29/2021-15:20:40] [V] [TRT] ModelImporter.cpp:179: map/while/TensorArrayV2Read/TensorListGetItem [Gather] outputs: [map/while/TensorArrayV2Read/TensorListGetItem:0 → (1024, 1024, 3)],
[03/29/2021-15:20:40] [V] [TRT] ModelImporter.cpp:103: Parsing node: map/while/Cast [Cast]
[03/29/2021-15:20:40] [V] [TRT] ModelImporter.cpp:119: Searching for input: map/while/TensorArrayV2Read/TensorListGetItem:0
[03/29/2021-15:20:40] [V] [TRT] ModelImporter.cpp:125: map/while/Cast [Cast] inputs: [map/while/TensorArrayV2Read/TensorListGetItem:0 → (1024, 1024, 3)],
[03/29/2021-15:20:40] [V] [TRT] builtin_op_importers.cpp:320: Casting to type: float32
[03/29/2021-15:20:40] [V] [TRT] ImporterContext.hpp:154: Registering layer: map/while/Cast for ONNX node: map/while/Cast
[03/29/2021-15:20:40] [V] [TRT] ImporterContext.hpp:120: Registering tensor: map/while/Cast:0 for ONNX tensor: map/while/Cast:0
[03/29/2021-15:20:40] [V] [TRT] ModelImporter.cpp:179: map/while/Cast [Cast] outputs: [map/while/Cast:0 → (1024, 1024, 3)],
[03/29/2021-15:20:40] [V] [TRT] ModelImporter.cpp:103: Parsing node: Unsqueeze__80 [Unsqueeze]
[03/29/2021-15:20:40] [V] [TRT] ModelImporter.cpp:119: Searching for input: map/while/Cast:0
[03/29/2021-15:20:40] [V] [TRT] ModelImporter.cpp:119: Searching for input: const_axes__79
[03/29/2021-15:20:40] [V] [TRT] ModelImporter.cpp:125: Unsqueeze__80 [Unsqueeze] inputs: [map/while/Cast:0 → (1024, 1024, 3)], [const_axes__79 → (1)],

Hi,
Request you to share the ONNX model and the script if not shared already so that we can assist you better.
Alongside you can try few things:

  1. validating your model with the below snippet

check_model.py

import sys
import onnx
filename = yourONNXmodel
model = onnx.load(filename)
onnx.checker.check_model(model).
2) Try running your model with trtexec command.

In case you are still facing issue, request you to share the trtexec “”–verbose"" log for further debugging
Thanks!

1 Like

Hi @Niklas.Flindt,

ONNX opset version 13 support will be available in future releases, we request you to try on opset version 11.

Thank you.

1 Like

Hi,

I did what you proposed and the initial model I posted did not work with the model_checker.
After some investigation, it appears that I used a TF1 version environment to convert from tf to onnx. That did not work as expected, obviously.

Now, I am using the TF 2.4.1 in my environment and the newest onnx and tf2onnx version.
The conversion to onnx works without errors.
Afterwards, I had to use onnx graphsurgeon to define the input and output shapes and change the input type to float32.

After running the conversion of tf2onnx with --ops=11, I finally got a “nice” error from trtexec.exe:

[03/30/2021-09:40:55] [V] [TRT] ModelImporter.cpp:125: StatefulPartitionedCall/Preprocessor/ResizeToRange/cond [If] inputs: [StatefulPartitionedCall/Preprocessor/ResizeToRange/Less:0 → ()],
ERROR: builtin_op_importers.cpp:1602 In function importIf:
[8] Assertion failed: cond.is_weights() && cond.weights().count() == 1 && “If condition must be a initializer!”

I will continue to look into that, but any help is highly appreciated :)

Hi @Niklas.Flindt,

We request you to share latest ONNX model/scripts to reproduce the issue for better assistance.

Thank you.

Hi,

the onnx model can be found as download from dropbox in the initial question.

Thanks!

Hi Niklas

I had a similar issue, you can find information about what causes it in this git issue:

What i had to do was to replace the upsampling2d call (i think efficientdet-d4 also has those) with this patched version before converting to .onnx (tf 2.4.1):

from tensorflow.python.ops.gen_image_ops import resize_nearest_neighbor

# define your model here

def patched_UpSampling2D_call(self, inputs):
    _, height, width, _ = inputs.get_shape().as_list()
    if height != None:
        # fixed call in order to avoid trt conversion error
        new_height, new_width = height * 2, width * 2
        size_tensor = tf.convert_to_tensor(np.array([new_height, new_width], dtype=np.int32), dtype=tf.int32)
        inputs = resize_nearest_neighbor(inputs, size=size_tensor, align_corners=False, half_pixel_centers=False)
        inputs.set_shape([None, new_height, new_width, None])
        return inputs
    else:
        # original call
        return backend.resize_images(
        inputs, self.size[0], self.size[1], self.data_format,
        interpolation=self.interpolation)

tf.keras.layers.UpSampling2D.call = patched_UpSampling2D_call
model.save(CHANGE_SAVED_MODEL_PATH, include_optimizer=False)

Hi @Niklas.Flindt,

We tried reproducing the issue, but unfortunately facing some errors related to plugins. Could you please share all files required and steps.

Thank you.

Hi,
I’m facing similar issues. Do you have any workarounds?
onnx to trt with trtexec failed

thank you!

Hi @raphael_zingg

This funtion looks like would do the trick. However I’m using a pretrained model. Can you suggest what changes could be made to run it? I think with a combination of multiplication op etc it could be done with onnx_graphsurgeon. But, I’m having problem in figuring out exactly how.
github link to the issue

Thank you!!