Slice & resize operators error: N shape values provided by optimization profile, which is not product of its dimensions [N]

user7278133 · April 6, 2022, 7:06pm

Description

I have a simple ONNX graph which takes input X (1x3x256x256), slice it and resize to output Y (1x3x64x64), attached below. The graph takes starts and ends inputs which are used by the Slice operator, and the operator’s axes input is a graph initializer constant [2,3] to allow slicing only on height & width. Thus, starts and ends are of type int32[2] with constant shape. However:

Running trtexec --onnx=test.onnx produces error: Dynamic dimensions required for input: starts_1234, but no shapes were provided. Automatically overriding shape to: 1x1

Running trtexec --onnx=test.onnx --shapes=starts_1234:2,ends_1234:2 produces error: Error Code 4: Internal Error (starts_1234: 2 shape values provided by optimization profile, which is not product of its dimensions [2].)

Finally, running trtexec --onnx=test.onnx --minShapes=starts_1234:0x0,ends_1234:0x0 --maxShapes=starts_1234:256x256,ends_1234:256x256 --optShapes=starts_1234:0x0,ends_1234:128x128 works fine. I think it wants to know the slice sizes for building optimization profiling. However, I think this will degrade runtime performance since the slices could be arbitrary.

Is this the best way to run this model?

Environment

TensorRT Version: 8.2.2.1
GPU Type: GTX 1080
Nvidia Driver Version: 470.103.01
CUDA Version: 11.1
CUDNN Version: 8.2
Operating System + Version: Ubuntu 20.04
Python Version (if applicable):
TensorFlow Version (if applicable):
PyTorch Version (if applicable):
Baremetal or Container (if container which image + tag):

Relevant Files

ONNX model file: test.onnx - Google Drive.

Steps To Reproduce

Run the trtexec commands above.

NVES · April 6, 2022, 7:37pm

Hi,
Request you to share the ONNX model and the script if not shared already so that we can assist you better.
Alongside you can try few things:

validating your model with the below snippet

check_model.py

import sys
import onnx
filename = yourONNXmodel
model = onnx.load(filename)
onnx.checker.check_model(model).
2) Try running your model with trtexec command.

In case you are still facing issue, request you to share the trtexec “”–verbose"" log for further debugging
Thanks!

user7278133 · April 6, 2022, 9:03pm

Hi, thanks for the quick reply. I have already posted the gdrive link to the ONNX model in the original post above. The model check with onnx.checker.check_model() is also successful. Here are the verbose outputs from trtexec:

./trtexec --onnx=test.onnx --verbose
&&&& RUNNING TensorRT.trtexec [TensorRT v8202] # ./trtexec --onnx=test.onnx --verbose
[04/06/2022-14:00:09] [I] === Model Options ===
[04/06/2022-14:00:09] [I] Format: ONNX
[04/06/2022-14:00:09] [I] Model: test.onnx
[04/06/2022-14:00:09] [I] Output:
[04/06/2022-14:00:09] [I] === Build Options ===
[04/06/2022-14:00:09] [I] Max batch: explicit batch
[04/06/2022-14:00:09] [I] Workspace: 16 MiB
[04/06/2022-14:00:09] [I] minTiming: 1
[04/06/2022-14:00:09] [I] avgTiming: 8
[04/06/2022-14:00:09] [I] Precision: FP32
[04/06/2022-14:00:09] [I] Calibration:
[04/06/2022-14:00:09] [I] Refit: Disabled
[04/06/2022-14:00:09] [I] Sparsity: Disabled
[04/06/2022-14:00:09] [I] Safe mode: Disabled
[04/06/2022-14:00:09] [I] DirectIO mode: Disabled
[04/06/2022-14:00:09] [I] Restricted mode: Disabled
[04/06/2022-14:00:09] [I] Save engine:
[04/06/2022-14:00:09] [I] Load engine:
[04/06/2022-14:00:09] [I] Profiling verbosity: 0
[04/06/2022-14:00:09] [I] Tactic sources: Using default tactic sources
[04/06/2022-14:00:09] [I] timingCacheMode: local
[04/06/2022-14:00:09] [I] timingCacheFile:
[04/06/2022-14:00:09] [I] Input(s)s format: fp32:CHW
[04/06/2022-14:00:09] [I] Output(s)s format: fp32:CHW
[04/06/2022-14:00:09] [I] Input build shapes: model
[04/06/2022-14:00:09] [I] Input calibration shapes: model
[04/06/2022-14:00:09] [I] === System Options ===
[04/06/2022-14:00:09] [I] Device: 0
[04/06/2022-14:00:09] [I] DLACore:
[04/06/2022-14:00:09] [I] Plugins:
[04/06/2022-14:00:09] [I] === Inference Options ===
[04/06/2022-14:00:09] [I] Batch: Explicit
[04/06/2022-14:00:09] [I] Input inference shapes: model
[04/06/2022-14:00:09] [I] Iterations: 10
[04/06/2022-14:00:09] [I] Duration: 3s (+ 200ms warm up)
[04/06/2022-14:00:09] [I] Sleep time: 0ms
[04/06/2022-14:00:09] [I] Idle time: 0ms
[04/06/2022-14:00:09] [I] Streams: 1
[04/06/2022-14:00:09] [I] ExposeDMA: Disabled
[04/06/2022-14:00:09] [I] Data transfers: Enabled
[04/06/2022-14:00:09] [I] Spin-wait: Disabled
[04/06/2022-14:00:09] [I] Multithreading: Disabled
[04/06/2022-14:00:09] [I] CUDA Graph: Disabled
[04/06/2022-14:00:09] [I] Separate profiling: Disabled
[04/06/2022-14:00:09] [I] Time Deserialize: Disabled
[04/06/2022-14:00:09] [I] Time Refit: Disabled
[04/06/2022-14:00:09] [I] Skip inference: Disabled
[04/06/2022-14:00:09] [I] Inputs:
[04/06/2022-14:00:09] [I] === Reporting Options ===
[04/06/2022-14:00:09] [I] Verbose: Enabled
[04/06/2022-14:00:09] [I] Averages: 10 inferences
[04/06/2022-14:00:09] [I] Percentile: 99
[04/06/2022-14:00:09] [I] Dump refittable layers:Disabled
[04/06/2022-14:00:09] [I] Dump output: Disabled
[04/06/2022-14:00:09] [I] Profile: Disabled
[04/06/2022-14:00:09] [I] Export timing to JSON file:
[04/06/2022-14:00:09] [I] Export output to JSON file:
[04/06/2022-14:00:09] [I] Export profile to JSON file:
[04/06/2022-14:00:09] [I]
[04/06/2022-14:00:09] [I] === Device Information ===
[04/06/2022-14:00:09] [I] Selected Device: NVIDIA GeForce GTX 1080
[04/06/2022-14:00:09] [I] Compute Capability: 6.1
[04/06/2022-14:00:09] [I] SMs: 20
[04/06/2022-14:00:09] [I] Compute Clock Rate: 1.7335 GHz
[04/06/2022-14:00:09] [I] Device Global Memory: 8116 MiB
[04/06/2022-14:00:09] [I] Shared Memory per SM: 96 KiB
[04/06/2022-14:00:09] [I] Memory Bus Width: 256 bits (ECC disabled)
[04/06/2022-14:00:09] [I] Memory Clock Rate: 5.005 GHz
[04/06/2022-14:00:09] [I]
[04/06/2022-14:00:09] [I] TensorRT version: 8.2.2
[04/06/2022-14:00:09] [V] [TRT] Registered plugin creator - ::GridAnchor_TRT version 1
[04/06/2022-14:00:09] [V] [TRT] Registered plugin creator - ::GridAnchorRect_TRT version 1
[04/06/2022-14:00:09] [V] [TRT] Registered plugin creator - ::NMS_TRT version 1
[04/06/2022-14:00:09] [V] [TRT] Registered plugin creator - ::Reorg_TRT version 1
[04/06/2022-14:00:09] [V] [TRT] Registered plugin creator - ::Region_TRT version 1
[04/06/2022-14:00:09] [V] [TRT] Registered plugin creator - ::Clip_TRT version 1
[04/06/2022-14:00:09] [V] [TRT] Registered plugin creator - ::LReLU_TRT version 1
[04/06/2022-14:00:09] [V] [TRT] Registered plugin creator - ::PriorBox_TRT version 1
[04/06/2022-14:00:09] [V] [TRT] Registered plugin creator - ::Normalize_TRT version 1
[04/06/2022-14:00:09] [V] [TRT] Registered plugin creator - ::ScatterND version 1
[04/06/2022-14:00:09] [V] [TRT] Registered plugin creator - ::RPROI_TRT version 1
[04/06/2022-14:00:09] [V] [TRT] Registered plugin creator - ::BatchedNMS_TRT version 1
[04/06/2022-14:00:09] [V] [TRT] Registered plugin creator - ::BatchedNMSDynamic_TRT version 1
[04/06/2022-14:00:09] [V] [TRT] Registered plugin creator - ::FlattenConcat_TRT version 1
[04/06/2022-14:00:09] [V] [TRT] Registered plugin creator - ::CropAndResize version 1
[04/06/2022-14:00:09] [V] [TRT] Registered plugin creator - ::DetectionLayer_TRT version 1
[04/06/2022-14:00:09] [V] [TRT] Registered plugin creator - ::EfficientNMS_TRT version 1
[04/06/2022-14:00:09] [V] [TRT] Registered plugin creator - ::EfficientNMS_ONNX_TRT version 1
[04/06/2022-14:00:09] [V] [TRT] Registered plugin creator - ::EfficientNMS_TFTRT_TRT version 1
[04/06/2022-14:00:09] [V] [TRT] Registered plugin creator - ::Proposal version 1
[04/06/2022-14:00:09] [V] [TRT] Registered plugin creator - ::ProposalLayer_TRT version 1
[04/06/2022-14:00:09] [V] [TRT] Registered plugin creator - ::PyramidROIAlign_TRT version 1
[04/06/2022-14:00:09] [V] [TRT] Registered plugin creator - ::ResizeNearest_TRT version 1
[04/06/2022-14:00:09] [V] [TRT] Registered plugin creator - ::Split version 1
[04/06/2022-14:00:09] [V] [TRT] Registered plugin creator - ::SpecialSlice_TRT version 1
[04/06/2022-14:00:09] [V] [TRT] Registered plugin creator - ::InstanceNormalization_TRT version 1
[04/06/2022-14:00:09] [I] [TRT] [MemUsageChange] Init CUDA: CPU +178, GPU +0, now: CPU 189, GPU 246 (MiB)
[04/06/2022-14:00:09] [I] [TRT] [MemUsageSnapshot] Begin constructing builder kernel library: CPU 189 MiB, GPU 246 MiB
[04/06/2022-14:00:09] [I] [TRT] [MemUsageSnapshot] End constructing builder kernel library: CPU 252 MiB, GPU 246 MiB
[04/06/2022-14:00:09] [I] Start parsing network model
[04/06/2022-14:00:09] [I] [TRT] ----------------------------------------------------------------
[04/06/2022-14:00:09] [I] [TRT] Input filename: test.onnx
[04/06/2022-14:00:09] [I] [TRT] ONNX IR version: 0.0.8
[04/06/2022-14:00:09] [I] [TRT] Opset version: 12
[04/06/2022-14:00:09] [I] [TRT] Producer name: onnx-example
[04/06/2022-14:00:09] [I] [TRT] Producer version:
[04/06/2022-14:00:09] [I] [TRT] Domain:
[04/06/2022-14:00:09] [I] [TRT] Model version: 0
[04/06/2022-14:00:09] [I] [TRT] Doc string:
[04/06/2022-14:00:09] [I] [TRT] ----------------------------------------------------------------
[04/06/2022-14:00:09] [V] [TRT] Plugin creator already registered - ::GridAnchor_TRT version 1
[04/06/2022-14:00:09] [V] [TRT] Plugin creator already registered - ::GridAnchorRect_TRT version 1
[04/06/2022-14:00:09] [V] [TRT] Plugin creator already registered - ::NMS_TRT version 1
[04/06/2022-14:00:09] [V] [TRT] Plugin creator already registered - ::Reorg_TRT version 1
[04/06/2022-14:00:09] [V] [TRT] Plugin creator already registered - ::Region_TRT version 1
[04/06/2022-14:00:09] [V] [TRT] Plugin creator already registered - ::Clip_TRT version 1
[04/06/2022-14:00:09] [V] [TRT] Plugin creator already registered - ::LReLU_TRT version 1
[04/06/2022-14:00:09] [V] [TRT] Plugin creator already registered - ::PriorBox_TRT version 1
[04/06/2022-14:00:09] [V] [TRT] Plugin creator already registered - ::Normalize_TRT version 1
[04/06/2022-14:00:09] [V] [TRT] Plugin creator already registered - ::ScatterND version 1
[04/06/2022-14:00:09] [V] [TRT] Plugin creator already registered - ::RPROI_TRT version 1
[04/06/2022-14:00:09] [V] [TRT] Plugin creator already registered - ::BatchedNMS_TRT version 1
[04/06/2022-14:00:09] [V] [TRT] Plugin creator already registered - ::BatchedNMSDynamic_TRT version 1
[04/06/2022-14:00:09] [V] [TRT] Plugin creator already registered - ::FlattenConcat_TRT version 1
[04/06/2022-14:00:09] [V] [TRT] Plugin creator already registered - ::CropAndResize version 1
[04/06/2022-14:00:09] [V] [TRT] Plugin creator already registered - ::DetectionLayer_TRT version 1
[04/06/2022-14:00:09] [V] [TRT] Plugin creator already registered - ::EfficientNMS_TRT version 1
[04/06/2022-14:00:09] [V] [TRT] Plugin creator already registered - ::EfficientNMS_ONNX_TRT version 1
[04/06/2022-14:00:09] [V] [TRT] Plugin creator already registered - ::EfficientNMS_TFTRT_TRT version 1
[04/06/2022-14:00:09] [V] [TRT] Plugin creator already registered - ::Proposal version 1
[04/06/2022-14:00:09] [V] [TRT] Plugin creator already registered - ::ProposalLayer_TRT version 1
[04/06/2022-14:00:09] [V] [TRT] Plugin creator already registered - ::PyramidROIAlign_TRT version 1
[04/06/2022-14:00:09] [V] [TRT] Plugin creator already registered - ::ResizeNearest_TRT version 1
[04/06/2022-14:00:09] [V] [TRT] Plugin creator already registered - ::Split version 1
[04/06/2022-14:00:09] [V] [TRT] Plugin creator already registered - ::SpecialSlice_TRT version 1
[04/06/2022-14:00:09] [V] [TRT] Plugin creator already registered - ::InstanceNormalization_TRT version 1
[04/06/2022-14:00:09] [V] [TRT] Adding network input: X with dtype: float32, dimensions: (1, 3, 256, 256)
[04/06/2022-14:00:09] [V] [TRT] Registering tensor: X for ONNX tensor: X
[04/06/2022-14:00:09] [V] [TRT] Adding network input: starts_1234 with dtype: int32, dimensions: (2)
[04/06/2022-14:00:09] [V] [TRT] Registering tensor: starts_1234 for ONNX tensor: starts_1234
[04/06/2022-14:00:09] [V] [TRT] Adding network input: ends_1234 with dtype: int32, dimensions: (2)
[04/06/2022-14:00:09] [V] [TRT] Registering tensor: ends_1234 for ONNX tensor: ends_1234
[04/06/2022-14:00:09] [V] [TRT] Importing initializer: axes_constant_1234
[04/06/2022-14:00:09] [V] [TRT] Importing initializer: scales_constant_1234
[04/06/2022-14:00:09] [V] [TRT] Importing initializer: roi_constant_1234
[04/06/2022-14:00:09] [V] [TRT] Importing initializer: sizes_constant_1234
[04/06/2022-14:00:09] [V] [TRT] Parsing node: Slice_1234 [Slice]
[04/06/2022-14:00:09] [V] [TRT] Searching for input: X
[04/06/2022-14:00:09] [V] [TRT] Searching for input: starts_1234
[04/06/2022-14:00:09] [V] [TRT] Searching for input: ends_1234
[04/06/2022-14:00:09] [V] [TRT] Searching for input: axes_constant_1234
[04/06/2022-14:00:09] [V] [TRT] Slice_1234 [Slice] inputs: [X → (1, 3, 256, 256)[FLOAT]], [starts_1234 → (2)[INT32]], [ends_1234 → (2)[INT32]], [axes_constant_1234 → (2)[INT32]],
[04/06/2022-14:00:09] [W] [TRT] onnx2trt_utils.cpp:366: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[04/06/2022-14:00:09] [V] [TRT] Registering layer: Slice_1234 for ONNX node: Slice_1234
[04/06/2022-14:00:09] [V] [TRT] Registering tensor: Slice_1234 for ONNX tensor: Slice_1234
[04/06/2022-14:00:09] [V] [TRT] Slice_1234 [Slice] outputs: [Slice_1234 → (1, 3, -1, -1)[FLOAT]],
[04/06/2022-14:00:09] [V] [TRT] Parsing node: Resize_1234 [Resize]
[04/06/2022-14:00:09] [V] [TRT] Searching for input: Slice_1234
[04/06/2022-14:00:09] [V] [TRT] Searching for input: roi_constant_1234
[04/06/2022-14:00:09] [V] [TRT] Searching for input: scales_constant_1234
[04/06/2022-14:00:09] [V] [TRT] Searching for input: sizes_constant_1234
[04/06/2022-14:00:09] [V] [TRT] Resize_1234 [Resize] inputs: [Slice_1234 → (1, 3, -1, -1)[FLOAT]], [roi_constant_1234 → ()[FLOAT]], [scales_constant_1234 → ()[FLOAT]], [sizes_constant_1234 → (4)[INT32]],
[04/06/2022-14:00:09] [V] [TRT] Registering layer: Resize_1234 for ONNX node: Resize_1234
[04/06/2022-14:00:09] [V] [TRT] Registering layer: sizes_constant_1234 for ONNX node: sizes_constant_1234
[04/06/2022-14:00:09] [V] [TRT] Registering tensor: Y_0 for ONNX tensor: Y
[04/06/2022-14:00:09] [V] [TRT] Resize_1234 [Resize] outputs: [Y → (1, 3, 64, 64)[FLOAT]],
[04/06/2022-14:00:09] [V] [TRT] Marking Y_0 as output: Y
[04/06/2022-14:00:09] [I] Finish parsing network model
[04/06/2022-14:00:09] [W] Dynamic dimensions required for input: starts_1234, but no shapes were provided. Automatically overriding shape to: 1x1
[04/06/2022-14:00:09] [W] Dynamic dimensions required for input: ends_1234, but no shapes were provided. Automatically overriding shape to: 1x1
[04/06/2022-14:00:09] [E] Error[4]: [graphShapeAnalyzer.cpp::processCheck::581] Error Code 4: Internal Error (Resize_1234: IResizeLayer requires that if input dimension is zero, output dimension must be zero too (axis = 2 input dimension = 0 output dimension = 64)
)
[04/06/2022-14:00:09] [E] Error[2]: [builder.cpp::buildSerializedNetwork::609] Error Code 2: Internal Error (Assertion enginePtr != nullptr failed. )
[04/06/2022-14:00:09] [E] Engine could not be created from network
[04/06/2022-14:00:09] [E] Building engine failed
[04/06/2022-14:00:09] [E] Failed to create engine from model.
[04/06/2022-14:00:09] [E] Engine set up failed
&&&& FAILED TensorRT.trtexec [TensorRT v8202] # ./trtexec --onnx=test.onnx --verbose

./trtexec --onnx=test.onnx --shapes=starts_1234:2,ends_1234:2 --verbose
&&&& RUNNING TensorRT.trtexec [TensorRT v8202] # ./trtexec --onnx=test.onnx --shapes=starts_1234:2,ends_1234:2 --verbose
[04/06/2022-13:59:09] [I] === Model Options ===
[04/06/2022-13:59:09] [I] Format: ONNX
[04/06/2022-13:59:09] [I] Model: test.onnx
[04/06/2022-13:59:09] [I] Output:
[04/06/2022-13:59:09] [I] === Build Options ===
[04/06/2022-13:59:09] [I] Max batch: explicit batch
[04/06/2022-13:59:09] [I] Workspace: 16 MiB
[04/06/2022-13:59:09] [I] minTiming: 1
[04/06/2022-13:59:09] [I] avgTiming: 8
[04/06/2022-13:59:09] [I] Precision: FP32
[04/06/2022-13:59:09] [I] Calibration:
[04/06/2022-13:59:09] [I] Refit: Disabled
[04/06/2022-13:59:09] [I] Sparsity: Disabled
[04/06/2022-13:59:09] [I] Safe mode: Disabled
[04/06/2022-13:59:09] [I] DirectIO mode: Disabled
[04/06/2022-13:59:09] [I] Restricted mode: Disabled
[04/06/2022-13:59:09] [I] Save engine:
[04/06/2022-13:59:09] [I] Load engine:
[04/06/2022-13:59:09] [I] Profiling verbosity: 0
[04/06/2022-13:59:09] [I] Tactic sources: Using default tactic sources
[04/06/2022-13:59:09] [I] timingCacheMode: local
[04/06/2022-13:59:09] [I] timingCacheFile:
[04/06/2022-13:59:09] [I] Input(s)s format: fp32:CHW
[04/06/2022-13:59:09] [I] Output(s)s format: fp32:CHW
[04/06/2022-13:59:09] [I] Input build shape: starts_1234=2+2+2
[04/06/2022-13:59:09] [I] Input build shape: ends_1234=2+2+2
[04/06/2022-13:59:09] [I] Input calibration shapes: model
[04/06/2022-13:59:09] [I] === System Options ===
[04/06/2022-13:59:09] [I] Device: 0
[04/06/2022-13:59:09] [I] DLACore:
[04/06/2022-13:59:09] [I] Plugins:
[04/06/2022-13:59:09] [I] === Inference Options ===
[04/06/2022-13:59:09] [I] Batch: Explicit
[04/06/2022-13:59:09] [I] Input inference shape: ends_1234=2
[04/06/2022-13:59:09] [I] Input inference shape: starts_1234=2
[04/06/2022-13:59:09] [I] Iterations: 10
[04/06/2022-13:59:09] [I] Duration: 3s (+ 200ms warm up)
[04/06/2022-13:59:09] [I] Sleep time: 0ms
[04/06/2022-13:59:09] [I] Idle time: 0ms
[04/06/2022-13:59:09] [I] Streams: 1
[04/06/2022-13:59:09] [I] ExposeDMA: Disabled
[04/06/2022-13:59:09] [I] Data transfers: Enabled
[04/06/2022-13:59:09] [I] Spin-wait: Disabled
[04/06/2022-13:59:09] [I] Multithreading: Disabled
[04/06/2022-13:59:09] [I] CUDA Graph: Disabled
[04/06/2022-13:59:09] [I] Separate profiling: Disabled
[04/06/2022-13:59:09] [I] Time Deserialize: Disabled
[04/06/2022-13:59:09] [I] Time Refit: Disabled
[04/06/2022-13:59:09] [I] Skip inference: Disabled
[04/06/2022-13:59:09] [I] Inputs:
[04/06/2022-13:59:09] [I] === Reporting Options ===
[04/06/2022-13:59:09] [I] Verbose: Enabled
[04/06/2022-13:59:09] [I] Averages: 10 inferences
[04/06/2022-13:59:09] [I] Percentile: 99
[04/06/2022-13:59:09] [I] Dump refittable layers:Disabled
[04/06/2022-13:59:09] [I] Dump output: Disabled
[04/06/2022-13:59:09] [I] Profile: Disabled
[04/06/2022-13:59:09] [I] Export timing to JSON file:
[04/06/2022-13:59:09] [I] Export output to JSON file:
[04/06/2022-13:59:09] [I] Export profile to JSON file:
[04/06/2022-13:59:09] [I]
[04/06/2022-13:59:09] [I] === Device Information ===
[04/06/2022-13:59:09] [I] Selected Device: NVIDIA GeForce GTX 1080
[04/06/2022-13:59:09] [I] Compute Capability: 6.1
[04/06/2022-13:59:09] [I] SMs: 20
[04/06/2022-13:59:09] [I] Compute Clock Rate: 1.7335 GHz
[04/06/2022-13:59:09] [I] Device Global Memory: 8116 MiB
[04/06/2022-13:59:09] [I] Shared Memory per SM: 96 KiB
[04/06/2022-13:59:09] [I] Memory Bus Width: 256 bits (ECC disabled)
[04/06/2022-13:59:09] [I] Memory Clock Rate: 5.005 GHz
[04/06/2022-13:59:09] [I]
[04/06/2022-13:59:09] [I] TensorRT version: 8.2.2
[04/06/2022-13:59:09] [V] [TRT] Registered plugin creator - ::GridAnchor_TRT version 1
[04/06/2022-13:59:09] [V] [TRT] Registered plugin creator - ::GridAnchorRect_TRT version 1
[04/06/2022-13:59:09] [V] [TRT] Registered plugin creator - ::NMS_TRT version 1
[04/06/2022-13:59:09] [V] [TRT] Registered plugin creator - ::Reorg_TRT version 1
[04/06/2022-13:59:09] [V] [TRT] Registered plugin creator - ::Region_TRT version 1
[04/06/2022-13:59:09] [V] [TRT] Registered plugin creator - ::Clip_TRT version 1
[04/06/2022-13:59:09] [V] [TRT] Registered plugin creator - ::LReLU_TRT version 1
[04/06/2022-13:59:09] [V] [TRT] Registered plugin creator - ::PriorBox_TRT version 1
[04/06/2022-13:59:09] [V] [TRT] Registered plugin creator - ::Normalize_TRT version 1
[04/06/2022-13:59:09] [V] [TRT] Registered plugin creator - ::ScatterND version 1
[04/06/2022-13:59:09] [V] [TRT] Registered plugin creator - ::RPROI_TRT version 1
[04/06/2022-13:59:09] [V] [TRT] Registered plugin creator - ::BatchedNMS_TRT version 1
[04/06/2022-13:59:09] [V] [TRT] Registered plugin creator - ::BatchedNMSDynamic_TRT version 1
[04/06/2022-13:59:09] [V] [TRT] Registered plugin creator - ::FlattenConcat_TRT version 1
[04/06/2022-13:59:09] [V] [TRT] Registered plugin creator - ::CropAndResize version 1
[04/06/2022-13:59:09] [V] [TRT] Registered plugin creator - ::DetectionLayer_TRT version 1
[04/06/2022-13:59:09] [V] [TRT] Registered plugin creator - ::EfficientNMS_TRT version 1
[04/06/2022-13:59:09] [V] [TRT] Registered plugin creator - ::EfficientNMS_ONNX_TRT version 1
[04/06/2022-13:59:09] [V] [TRT] Registered plugin creator - ::EfficientNMS_TFTRT_TRT version 1
[04/06/2022-13:59:09] [V] [TRT] Registered plugin creator - ::Proposal version 1
[04/06/2022-13:59:09] [V] [TRT] Registered plugin creator - ::ProposalLayer_TRT version 1
[04/06/2022-13:59:09] [V] [TRT] Registered plugin creator - ::PyramidROIAlign_TRT version 1
[04/06/2022-13:59:09] [V] [TRT] Registered plugin creator - ::ResizeNearest_TRT version 1
[04/06/2022-13:59:09] [V] [TRT] Registered plugin creator - ::Split version 1
[04/06/2022-13:59:09] [V] [TRT] Registered plugin creator - ::SpecialSlice_TRT version 1
[04/06/2022-13:59:09] [V] [TRT] Registered plugin creator - ::InstanceNormalization_TRT version 1
[04/06/2022-13:59:09] [I] [TRT] [MemUsageChange] Init CUDA: CPU +178, GPU +0, now: CPU 189, GPU 246 (MiB)
[04/06/2022-13:59:10] [I] [TRT] [MemUsageSnapshot] Begin constructing builder kernel library: CPU 189 MiB, GPU 246 MiB
[04/06/2022-13:59:10] [I] [TRT] [MemUsageSnapshot] End constructing builder kernel library: CPU 252 MiB, GPU 246 MiB
[04/06/2022-13:59:10] [I] Start parsing network model
[04/06/2022-13:59:10] [I] [TRT] ----------------------------------------------------------------
[04/06/2022-13:59:10] [I] [TRT] Input filename: test.onnx
[04/06/2022-13:59:10] [I] [TRT] ONNX IR version: 0.0.8
[04/06/2022-13:59:10] [I] [TRT] Opset version: 12
[04/06/2022-13:59:10] [I] [TRT] Producer name: onnx-example
[04/06/2022-13:59:10] [I] [TRT] Producer version:
[04/06/2022-13:59:10] [I] [TRT] Domain:
[04/06/2022-13:59:10] [I] [TRT] Model version: 0
[04/06/2022-13:59:10] [I] [TRT] Doc string:
[04/06/2022-13:59:10] [I] [TRT] ----------------------------------------------------------------
[04/06/2022-13:59:10] [V] [TRT] Plugin creator already registered - ::GridAnchor_TRT version 1
[04/06/2022-13:59:10] [V] [TRT] Plugin creator already registered - ::GridAnchorRect_TRT version 1
[04/06/2022-13:59:10] [V] [TRT] Plugin creator already registered - ::NMS_TRT version 1
[04/06/2022-13:59:10] [V] [TRT] Plugin creator already registered - ::Reorg_TRT version 1
[04/06/2022-13:59:10] [V] [TRT] Plugin creator already registered - ::Region_TRT version 1
[04/06/2022-13:59:10] [V] [TRT] Plugin creator already registered - ::Clip_TRT version 1
[04/06/2022-13:59:10] [V] [TRT] Plugin creator already registered - ::LReLU_TRT version 1
[04/06/2022-13:59:10] [V] [TRT] Plugin creator already registered - ::PriorBox_TRT version 1
[04/06/2022-13:59:10] [V] [TRT] Plugin creator already registered - ::Normalize_TRT version 1
[04/06/2022-13:59:10] [V] [TRT] Plugin creator already registered - ::ScatterND version 1
[04/06/2022-13:59:10] [V] [TRT] Plugin creator already registered - ::RPROI_TRT version 1
[04/06/2022-13:59:10] [V] [TRT] Plugin creator already registered - ::BatchedNMS_TRT version 1
[04/06/2022-13:59:10] [V] [TRT] Plugin creator already registered - ::BatchedNMSDynamic_TRT version 1
[04/06/2022-13:59:10] [V] [TRT] Plugin creator already registered - ::FlattenConcat_TRT version 1
[04/06/2022-13:59:10] [V] [TRT] Plugin creator already registered - ::CropAndResize version 1
[04/06/2022-13:59:10] [V] [TRT] Plugin creator already registered - ::DetectionLayer_TRT version 1
[04/06/2022-13:59:10] [V] [TRT] Plugin creator already registered - ::EfficientNMS_TRT version 1
[04/06/2022-13:59:10] [V] [TRT] Plugin creator already registered - ::EfficientNMS_ONNX_TRT version 1
[04/06/2022-13:59:10] [V] [TRT] Plugin creator already registered - ::EfficientNMS_TFTRT_TRT version 1
[04/06/2022-13:59:10] [V] [TRT] Plugin creator already registered - ::Proposal version 1
[04/06/2022-13:59:10] [V] [TRT] Plugin creator already registered - ::ProposalLayer_TRT version 1
[04/06/2022-13:59:10] [V] [TRT] Plugin creator already registered - ::PyramidROIAlign_TRT version 1
[04/06/2022-13:59:10] [V] [TRT] Plugin creator already registered - ::ResizeNearest_TRT version 1
[04/06/2022-13:59:10] [V] [TRT] Plugin creator already registered - ::Split version 1
[04/06/2022-13:59:10] [V] [TRT] Plugin creator already registered - ::SpecialSlice_TRT version 1
[04/06/2022-13:59:10] [V] [TRT] Plugin creator already registered - ::InstanceNormalization_TRT version 1
[04/06/2022-13:59:10] [V] [TRT] Adding network input: X with dtype: float32, dimensions: (1, 3, 256, 256)
[04/06/2022-13:59:10] [V] [TRT] Registering tensor: X for ONNX tensor: X
[04/06/2022-13:59:10] [V] [TRT] Adding network input: starts_1234 with dtype: int32, dimensions: (2)
[04/06/2022-13:59:10] [V] [TRT] Registering tensor: starts_1234 for ONNX tensor: starts_1234
[04/06/2022-13:59:10] [V] [TRT] Adding network input: ends_1234 with dtype: int32, dimensions: (2)
[04/06/2022-13:59:10] [V] [TRT] Registering tensor: ends_1234 for ONNX tensor: ends_1234
[04/06/2022-13:59:10] [V] [TRT] Importing initializer: axes_constant_1234
[04/06/2022-13:59:10] [V] [TRT] Importing initializer: scales_constant_1234
[04/06/2022-13:59:10] [V] [TRT] Importing initializer: roi_constant_1234
[04/06/2022-13:59:10] [V] [TRT] Importing initializer: sizes_constant_1234
[04/06/2022-13:59:10] [V] [TRT] Parsing node: Slice_1234 [Slice]
[04/06/2022-13:59:10] [V] [TRT] Searching for input: X
[04/06/2022-13:59:10] [V] [TRT] Searching for input: starts_1234
[04/06/2022-13:59:10] [V] [TRT] Searching for input: ends_1234
[04/06/2022-13:59:10] [V] [TRT] Searching for input: axes_constant_1234
[04/06/2022-13:59:10] [V] [TRT] Slice_1234 [Slice] inputs: [X → (1, 3, 256, 256)[FLOAT]], [starts_1234 → (2)[INT32]], [ends_1234 → (2)[INT32]], [axes_constant_1234 → (2)[INT32]],
[04/06/2022-13:59:10] [W] [TRT] onnx2trt_utils.cpp:366: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[04/06/2022-13:59:10] [V] [TRT] Registering layer: Slice_1234 for ONNX node: Slice_1234
[04/06/2022-13:59:10] [V] [TRT] Registering tensor: Slice_1234 for ONNX tensor: Slice_1234
[04/06/2022-13:59:10] [V] [TRT] Slice_1234 [Slice] outputs: [Slice_1234 → (1, 3, -1, -1)[FLOAT]],
[04/06/2022-13:59:10] [V] [TRT] Parsing node: Resize_1234 [Resize]
[04/06/2022-13:59:10] [V] [TRT] Searching for input: Slice_1234
[04/06/2022-13:59:10] [V] [TRT] Searching for input: roi_constant_1234
[04/06/2022-13:59:10] [V] [TRT] Searching for input: scales_constant_1234
[04/06/2022-13:59:10] [V] [TRT] Searching for input: sizes_constant_1234
[04/06/2022-13:59:10] [V] [TRT] Resize_1234 [Resize] inputs: [Slice_1234 → (1, 3, -1, -1)[FLOAT]], [roi_constant_1234 → ()[FLOAT]], [scales_constant_1234 → ()[FLOAT]], [sizes_constant_1234 → (4)[INT32]],
[04/06/2022-13:59:10] [V] [TRT] Registering layer: Resize_1234 for ONNX node: Resize_1234
[04/06/2022-13:59:10] [V] [TRT] Registering layer: sizes_constant_1234 for ONNX node: sizes_constant_1234
[04/06/2022-13:59:10] [V] [TRT] Registering tensor: Y_0 for ONNX tensor: Y
[04/06/2022-13:59:10] [V] [TRT] Resize_1234 [Resize] outputs: [Y → (1, 3, 64, 64)[FLOAT]],
[04/06/2022-13:59:10] [V] [TRT] Marking Y_0 as output: Y
[04/06/2022-13:59:10] [I] Finish parsing network model
[04/06/2022-13:59:10] [E] Error[4]: [network.cpp::validate::2989] Error Code 4: Internal Error (starts_1234: 2 shape values provided by optimization profile, which is not product of its dimensions [2].)
[04/06/2022-13:59:10] [E] Error[2]: [builder.cpp::buildSerializedNetwork::609] Error Code 2: Internal Error (Assertion enginePtr != nullptr failed. )
[04/06/2022-13:59:10] [E] Engine could not be created from network
[04/06/2022-13:59:10] [E] Building engine failed
[04/06/2022-13:59:10] [E] Failed to create engine from model.
[04/06/2022-13:59:10] [E] Engine set up failed
&&&& FAILED TensorRT.trtexec [TensorRT v8202] # ./trtexec --onnx=test.onnx --shapes=starts_1234:2,ends_1234:2 --verbose

spolisetty · April 12, 2022, 12:40pm

Hi,

You can try optimization profiles.

Thank you.

Topic		Replies	Views
Floor - Cast - Resize(or Slice) cause internal error TensorRT tensorrt	7	1985	January 12, 2022
Duplicated reshapes triggers "[graphOptimizer.cpp::findOne::510] Error Code 2: Internal Error (Assertion it != v.end() failed. )" TensorRT tensorrt , onnx	4	918	February 22, 2022
BUG: Output TRT engine from trtexec has completely different inference than input model TensorRT tensorrt , debugging-and-troubleshooting	3	2180	January 4, 2022
PyTorch/ONNX Model Involving Very Large Images: Myelin error: autotuning: CUDA error 2 allocating 0-byte buffer: out of memory TensorRT pytorch	4	1014	February 9, 2023
TensorRT OSS build TensorRT	9	1007	August 31, 2023
How to export the Pytorch model Keypoint R-CNN to onnx and benchmark with trtexec TensorRT	7	1324	July 14, 2022
Trtexec cannot convert QAT onnx model to trt model TensorRT tensorrt	7	1122	January 26, 2023
TensorRT generated QAT engine, why the engine is bigger than pretrained fp16 engine? TensorRT	3	1287	January 4, 2022
Tensorrt8.2 convert .onnx model to .trt ,throw "two inputs (data and weights) are allowed only in explicit-quantization mode"error TensorRT	9	1933	April 6, 2022
tensorRT plan file not working TensorRT	4	1131	September 4, 2023