Assertion isOpConsistent(deconvolution.get()) failed

AI & Data Science Deep Learning (Training & Inference) TensorRT
,
1

Description

I am using the C++ API in a very basic way to generate a serialized trt engine from an .onnx model.
I get an error at buildSerializedNetwork.
The error is the following:

TensorRT: 2: Assertion isOpConsistent(deconvolution.get()) failed. Cask deconvolution isConsistent check failed.
TensorRT: 2: [deconvolutionV2Builder.cpp::nvinfer1::builder::CaskDeconvolutionBuilderBase<class nvinfer1::builder::CaskD
econvolutionV2Builder,class nvinfer1::rt::task::CaskDeconvolutionV2Runner,-2147483603>::createDeconvolution::863] Error
Code 2: Internal Error (Assertion isOpConsistent(deconvolution.get()) failed. Cask deconvolution isConsistent check fail
ed.)

My model is exported with dynamic dimensions so I create an OptimizationProfile with the min, opt and max dimensions.
This is where I don’t understand the error, building the engine works for some dimension, but not for others.
Some examples of working dimension configurations:

"minimum": [45, 41, 27],  
"maximum": [50, 50, 32],  
"optimum": [50, 50, 32]

"minimum": [200, 200, 32],  
"maximum": [200, 200, 32],  
"optimum": [200, 200, 32]

"minimum": [195, 195, 27],  
"maximum": [200, 200, 32],  
"optimum": [200, 200, 32]

Examples of not working dimensions:

"minimum": [50, 48, 15],  
"maximum": [50, 200, 32],  
"optimum": [50, 50, 32]

"minimum": [45, 41, 27],  
"maximum": [50, 100, 32],  
"optimum": [50, 45, 32]

As the assertion is thrown from inside the library implementation I have no chance looking at the condition that is tested in this assertion.

Does anybody know what this could be?

Environment

TensorRT Version: 8.6.1.6
GPU Type: GeForce RTX 3050 Ti Laptop
Nvidia Driver Version: 546.09
CUDA Version: 12.0
CUDNN Version: 8.9.5.29
Operating System + Version: Windows 11
Python Version (if applicable): 3.9
TensorFlow Version (if applicable):
PyTorch Version (if applicable):
Baremetal or Container (if container which image + tag): baremetal

2

The verbose output is:

TensorRT: RunnerBuilder of layer implementation CudnnDeconvolution cannot handle striding for node /_block5/_block5.4_2/
ConvTranspose
TensorRT: --------------- Timing Runner: /_block5/_block5.4_2/ConvTranspose (GemmDeconvolution[0x80000010])
TensorRT: GemmDeconvolution has no valid tactics for this config, skipping
TensorRT: --------------- Timing Runner: /_block5/_block5.4_2/ConvTranspose (CaskDeconvolution[0x8000000a])
TensorRT: CaskDeconvolution has no valid tactics for this config, skipping
TensorRT: Deleting timing cache: 107 entries, served 4 hits since creation.
TensorRT: 2: Assertion isOpConsistent(deconvolution.get()) failed. Cask deconvolution isConsistent check failed.
TensorRT: 2: [deconvolutionV2Builder.cpp::nvinfer1::builder::CaskDeconvolutionBuilderBase<class nvinfer1::builder::CaskD
econvolutionV2Builder,class nvinfer1::rt::task::CaskDeconvolutionV2Runner,-2147483603>::createDeconvolution::863] Error
Code 2: Internal Error (Assertion isOpConsistent(deconvolution.get()) failed. Cask deconvolution isConsistent check fail
ed.)
3

Hi @aron.heck ,
Generally it’s caused by the inconsistent I/O shape of the conv, however i am checking with the team. In the meantime, can you please share the onnx model with us.

Thanks

4

Unfortunately I cannot, the model is proprietary.

5

Heard anything back from the team @AakankshaS ?