Error when trying to convert ONNX to TensorRT

SonTV · March 12, 2021, 9:35am

I use ONNX model that is generated from torch model to TensorRT using command:

/usr/src/tensorrt/bin/trtexec --explicitBatch --onnx=ocr_3-128-256-dynamic.onnx --minShapes=input:1x3x128x256 --optShapes=input:32x3x128x256 --maxShapes=input:32x3x128x256 --saveEngine=model.batch1-32.engine

Output error when run above command:

Input filename:   ocr_3-128-256-dynamic.onnx
ONNX IR version:  0.0.6
Opset version:    10
Producer name:    pytorch
Producer version: 1.6
Domain:           
Model version:    0
Doc string:       
----------------------------------------------------------------
[03/12/2021-08:47:18] [W] [TRT] onnx2trt_utils.cpp:220: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[03/12/2021-08:47:18] [W] [TRT] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[03/12/2021-08:47:18] [W] [TRT] Tensor DataType is determined at build time for tensors not marked as input or output.
[03/12/2021-08:47:18] [W] [TRT] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[03/12/2021-08:47:18] [W] [TRT] Tensor DataType is determined at build time for tensors not marked as input or output.
[03/12/2021-08:47:18] [W] [TRT] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[03/12/2021-08:47:18] [W] [TRT] Tensor DataType is determined at build time for tensors not marked as input or output.
[03/12/2021-08:47:18] [W] [TRT] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[03/12/2021-08:47:18] [W] [TRT] Tensor DataType is determined at build time for tensors not marked as input or output.
[03/12/2021-08:47:18] [W] [TRT] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[03/12/2021-08:47:18] [W] [TRT] Tensor DataType is determined at build time for tensors not marked as input or output.
[03/12/2021-08:47:18] [W] [TRT] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[03/12/2021-08:47:18] [W] [TRT] Tensor DataType is determined at build time for tensors not marked as input or output.
[03/12/2021-08:47:18] [W] [TRT] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[03/12/2021-08:47:18] [W] [TRT] Tensor DataType is determined at build time for tensors not marked as input or output.
[03/12/2021-08:47:19] [W] [TRT] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[03/12/2021-08:47:19] [W] [TRT] Tensor DataType is determined at build time for tensors not marked as input or output.
[03/12/2021-08:47:19] [W] [TRT] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[03/12/2021-08:47:19] [W] [TRT] Tensor DataType is determined at build time for tensors not marked as input or output.
[03/12/2021-08:47:19] [W] [TRT] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[03/12/2021-08:47:19] [W] [TRT] Tensor DataType is determined at build time for tensors not marked as input or output.
[03/12/2021-08:47:19] [W] [TRT] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[03/12/2021-08:47:19] [W] [TRT] Tensor DataType is determined at build time for tensors not marked as input or output.
[03/12/2021-08:47:35] [I] [TRT] Some tactics do not have sufficient workspace memory to run. Increasing workspace size may increase performance, please check verbose output.
[03/12/2021-08:48:17] [E] [TRT] ../builder/myelin/codeGenerator.cpp (401) - Myelin Error in compileGraph: 64 (myelinProgramAnalysisError : Mismatched shape for tensor 2969'_468 _ (Unnamed Layer_ 290) [Shuffle]_Cast_8_Cast_9_Unsqueeze_199_(Unnamed Layer_ 498) [Constant]_700 _ (Unnamed Layer_ 612) [Shuffle]_Unsqueeze_371_(Unnamed Layer_ 878) [Constant]_935 _ (Unnamed Layer_ 990) [Shuffle]_Unsqueeze_548_(Unnamed Layer_ 1256) [Constant]_1170 _ (Unnamed Layer_ 1368) [Shuffle]_Unsqueeze_725_(Unnamed Layer_ 1634) [Constant]_1405 _ (Unnamed Layer_ 1746) [Shuffle]_Unsqueeze_902_(Unnamed Layer_ 2012) [Constant]_1640 _ (Unnamed Layer_ 2124) [Shuffle]_Unsqueeze_1079_(Unnamed Layer_ 2390) [Constant]_1875 _ (Unnamed Layer_ 2502) [Shuffle]_Unsqueeze_1256_(Unnamed Layer_ 2768) [Constant]_2110 _ (Unnamed Layer_ 2880) [Shuffle]_Unsqueeze_1433_(Unnamed Layer_ 3146) [Constant]_2345 _ (Unnamed Layer_ 3258) [Shuffle]_Unsqueeze_1610_(Unnamed Layer_ 3524) [Constant]_2580 _ (Unnamed Layer_ 3636) [Shuffle]_Unsqueeze_1787_(Unnamed Layer_ 3902) [Constant]_2815 _ (Unnamed Layer_ 4014) [Shuffle]_Unsqueeze_1964_(Unnamed Layer_ 4280) [Constant]_3050 _ (Unnamed Layer_ 4392) [Shuffle]_Unsqueeze_2141_Equal_2167_Cast_2168_(Unnamed Layer_ 4281) [Identity]_Cast_2111_Equal_1990_Cast_1991_(Unnamed Layer_ 3903) [Identity]_Cast_1934_Equal_1813_Cast_1814_(Unnamed Layer_ 3525) [Identity]_Cast_1757_Equal_1636_Cast_1637_(Unnamed Layer_ 3147) [Identity]_Cast_1580_Equal_1459_Cast_1460_(Unnamed Layer_ 2769) [Identity]_Cast_1403_Equal_1282_Cast_1283_(Unnamed Layer_ 2391) [Identity]_Cast_1226_Equal_1105_Cast_1106_(Unnamed Layer_ 2013) [Identity]_Cast_1049_Equal_928_Cast_929_(Unnamed Layer_ 1635) [Identity]_Cast_872_Equal_751_Cast_752_(Unnamed Layer_ 1257) [Identity]_Cast_695_Equal_574_Cast_575_(Unnamed Layer_ 879) [Identity]_Cast_518_Equal_397_Cast_398_(Unnamed Layer_ 499) [Identity]_Cast_341_Equal_225_Cast_226_471 _ (Unnamed Layer_ 294) [Shuffle]_466 _ (Unnamed Layer_ 287) [Shuffle]_encoder_layers_0_self_attn_linears_0_bias _ (Unnamed Layer_ 250) [Shuffle]_3133 _ (Unnamed Layer_ 247) [Shuffle]_encoder_layers_0_sublayer_0_norm_b_2 _ (Unnamed Layer_ 235) [Shuffle]_encoder_layers_0_sublayer_0_norm_a_2 _ (Unnamed Layer_ 228) [Shuffle]_Sub_193_Mul_194_418 _ (Unnamed Layer_ 231) [Shuffle]_413 _ (Unnamed Layer_ 223) [Shuffle]_411 _ (Unnamed Layer_ 220) [Shuffle]_Mul_185_Sub_186_Abs_187_Mul_189_Div_191_Sqrt_192_Add_196_Div_197_Add_198_MatMul_203_Add_204_Reshape_207 _ Transpose_208_encoder_layers_0_self_attn_linears_1_bias _ (Unnamed Layer_ 263) [Shuffle]_3137 _ (Unnamed Layer_ 260) [Shuffle]_MatMul_209_Add_210_Reshape_213 _ Transpose_220_MatMul_221_Div_223_620 _ (Unnamed Layer_ 502) [Shuffle]_616 _ (Unnamed Layer_ 496) [Shuffle]_decoder_layers_0_self_attn_linears_0_bias _ (Unnamed Layer_ 459) [Shuffle]_3153 _ (Unnamed Layer_ 456) [Shuffle]_decoder_layers_0_sublayer_0_norm_b_2 _ (Unnamed Layer_ 450) [Shuffle]_decoder_layers_0_sublayer_0_norm_a_2 _ (Unnamed Layer_ 443) [Shuffle]_Sub_310_Mul_311_569 _ (Unnamed Layer_ 446) [Shuffle]_564 _ (Unnamed Layer_ 438) [Shuffle]_562 _ (Unnamed Layer_ 435) [Shuffle]_Mul_302_Sub_303_Abs_304_Mul_306_Div_308_Sqrt_309_Add_313_Div_314_Add_315_MatMul_319_Add_320_Reshape_323 _ Transpose_324_decoder_layers_0_self_attn_linears_1_bias _ (Unnamed Layer_ 472) [Shuffle]_3157 _ (Unnamed Layer_ 469) [Shuffle]_MatMul_325_Add_326_Reshape_329 _ Transpose_336_MatMul_337_Div_339_Where_343_Where_228_, current shape:b,[1,1,11,11], vs. expected shape:b,[].
)
[03/12/2021-08:48:17] [E] [TRT] ../builder/myelin/codeGenerator.cpp (401) - Myelin Error in compileGraph: 64 (myelinProgramAnalysisError : Mismatched shape for tensor 2969'_468 _ (Unnamed Layer_ 290) [Shuffle]_Cast_8_Cast_9_Unsqueeze_199_(Unnamed Layer_ 498) [Constant]_700 _ (Unnamed Layer_ 612) [Shuffle]_Unsqueeze_371_(Unnamed Layer_ 878) [Constant]_935 _ (Unnamed Layer_ 990) [Shuffle]_Unsqueeze_548_(Unnamed Layer_ 1256) [Constant]_1170 _ (Unnamed Layer_ 1368) [Shuffle]_Unsqueeze_725_(Unnamed Layer_ 1634) [Constant]_1405 _ (Unnamed Layer_ 1746) [Shuffle]_Unsqueeze_902_(Unnamed Layer_ 2012) [Constant]_1640 _ (Unnamed Layer_ 2124) [Shuffle]_Unsqueeze_1079_(Unnamed Layer_ 2390) [Constant]_1875 _ (Unnamed Layer_ 2502) [Shuffle]_Unsqueeze_1256_(Unnamed Layer_ 2768) [Constant]_2110 _ (Unnamed Layer_ 2880) [Shuffle]_Unsqueeze_1433_(Unnamed Layer_ 3146) [Constant]_2345 _ (Unnamed Layer_ 3258) [Shuffle]_Unsqueeze_1610_(Unnamed Layer_ 3524) [Constant]_2580 _ (Unnamed Layer_ 3636) [Shuffle]_Unsqueeze_1787_(Unnamed Layer_ 3902) [Constant]_2815 _ (Unnamed Layer_ 4014) [Shuffle]_Unsqueeze_1964_(Unnamed Layer_ 4280) [Constant]_3050 _ (Unnamed Layer_ 4392) [Shuffle]_Unsqueeze_2141_Equal_2167_Cast_2168_(Unnamed Layer_ 4281) [Identity]_Cast_2111_Equal_1990_Cast_1991_(Unnamed Layer_ 3903) [Identity]_Cast_1934_Equal_1813_Cast_1814_(Unnamed Layer_ 3525) [Identity]_Cast_1757_Equal_1636_Cast_1637_(Unnamed Layer_ 3147) [Identity]_Cast_1580_Equal_1459_Cast_1460_(Unnamed Layer_ 2769) [Identity]_Cast_1403_Equal_1282_Cast_1283_(Unnamed Layer_ 2391) [Identity]_Cast_1226_Equal_1105_Cast_1106_(Unnamed Layer_ 2013) [Identity]_Cast_1049_Equal_928_Cast_929_(Unnamed Layer_ 1635) [Identity]_Cast_872_Equal_751_Cast_752_(Unnamed Layer_ 1257) [Identity]_Cast_695_Equal_574_Cast_575_(Unnamed Layer_ 879) [Identity]_Cast_518_Equal_397_Cast_398_(Unnamed Layer_ 499) [Identity]_Cast_341_Equal_225_Cast_226_471 _ (Unnamed Layer_ 294) [Shuffle]_466 _ (Unnamed Layer_ 287) [Shuffle]_encoder_layers_0_self_attn_linears_0_bias _ (Unnamed Layer_ 250) [Shuffle]_3133 _ (Unnamed Layer_ 247) [Shuffle]_encoder_layers_0_sublayer_0_norm_b_2 _ (Unnamed Layer_ 235) [Shuffle]_encoder_layers_0_sublayer_0_norm_a_2 _ (Unnamed Layer_ 228) [Shuffle]_Sub_193_Mul_194_418 _ (Unnamed Layer_ 231) [Shuffle]_413 _ (Unnamed Layer_ 223) [Shuffle]_411 _ (Unnamed Layer_ 220) [Shuffle]_Mul_185_Sub_186_Abs_187_Mul_189_Div_191_Sqrt_192_Add_196_Div_197_Add_198_MatMul_203_Add_204_Reshape_207 _ Transpose_208_encoder_layers_0_self_attn_linears_1_bias _ (Unnamed Layer_ 263) [Shuffle]_3137 _ (Unnamed Layer_ 260) [Shuffle]_MatMul_209_Add_210_Reshape_213 _ Transpose_220_MatMul_221_Div_223_620 _ (Unnamed Layer_ 502) [Shuffle]_616 _ (Unnamed Layer_ 496) [Shuffle]_decoder_layers_0_self_attn_linears_0_bias _ (Unnamed Layer_ 459) [Shuffle]_3153 _ (Unnamed Layer_ 456) [Shuffle]_decoder_layers_0_sublayer_0_norm_b_2 _ (Unnamed Layer_ 450) [Shuffle]_decoder_layers_0_sublayer_0_norm_a_2 _ (Unnamed Layer_ 443) [Shuffle]_Sub_310_Mul_311_569 _ (Unnamed Layer_ 446) [Shuffle]_564 _ (Unnamed Layer_ 438) [Shuffle]_562 _ (Unnamed Layer_ 435) [Shuffle]_Mul_302_Sub_303_Abs_304_Mul_306_Div_308_Sqrt_309_Add_313_Div_314_Add_315_MatMul_319_Add_320_Reshape_323 _ Transpose_324_decoder_layers_0_self_attn_linears_1_bias _ (Unnamed Layer_ 472) [Shuffle]_3157 _ (Unnamed Layer_ 469) [Shuffle]_MatMul_325_Add_326_Reshape_329 _ Transpose_336_MatMul_337_Div_339_Where_343_Where_228_, current shape:b,[1,1,11,11], vs. expected shape:b,[].
)
[03/12/2021-08:48:17] [E] Engine creation failed
[03/12/2021-08:48:17] [E] Engine set up failed
&&&& FAILED TensorRT.trtexec # /usr/src/tensorrt/bin/trtexec --explicitBatch --onnx=ocr_3-128-256-dynamic.onnx --minShapes=input:1x3x128x256 --optShapes=input:32x3x128x256 --maxShapes=input:32x3x128x256 --saveEngine=model.batch1-32.engine

What’s wrong? How to fix it? (source code in attached file)
Thanks!

Environment

• Hardware Platform: Tesla T4
• DeepStream Version: 5.0
• TensorRT Version: 7.2.1
• PyTorch 1.6
• ONNX v6
• NVIDIA GPU: Driver Version 455.32, CUDA Version 11.1
• OS: Ubuntu 18.04

source code: code.zip (23.7 KB)
onnx model: ocr_3-128-256-dynamic.zip - Google Drive

NVES · March 12, 2021, 9:37am

Hi,
Request you to share the ONNX model and the script if not shared already so that we can assist you better.
Alongside you can try few things:

validating your model with the below snippet

check_model.py

import sys
import onnx
filename = yourONNXmodel
model = onnx.load(filename)
onnx.checker.check_model(model).
2) Try running your model with trtexec command.
https://github.com/NVIDIA/TensorRT/tree/master/samples/opensource/trtexec
In case you are still facing issue, request you to share the trtexec “”–verbose"" log for further debugging
Thanks!

spolisetty · March 15, 2021, 7:14am

Hi @SonTV,

I could reproduce the issue when I run command you’ve shared. Facing similar error.
Increasing the workspace resolved the issue. I tried following command. I could successfully generate the TRT engine.
trtexec --explicitBatch --onnx=ocr_3-128-256-dynamic.onnx --minShapes=input:1x3x128x256 --optShapes=input:32x3x128x256 --maxShapes=input:32x3x128x256 --saveEngine=batch1-32.engine --workspace=3000 --verbose

Please try the above command.

Note: I tried on V100 GPU.

Thank you.

SonTV · March 15, 2021, 8:03am

Hi @spolisetty ,

I have added workspace parameter to my command:

/usr/src/tensorrt/bin/trtexec --explicitBatch --onnx=ocr_3-128-256-dynamic.onnx --minShapes=input:1x3x128x256 --optShapes=input:32x3x128x256 --maxShapes=input:32x3x128x256 --saveEngine=batch1-32.engine --workspace=3000

But the same error still occurs :(

spolisetty · March 15, 2021, 8:09am

Hi @SonTV,

Could you please try increasing the workspace to more. Just for confirmation.

Thank you.

SonTV · April 13, 2021, 3:06am

hi @spolisetty ,

Sorry for my late response.
I have tried increasing workspace memory to 7000, but seem that it doesn’t solve this problem.

/usr/src/tensorrt/bin/trtexec --explicitBatch --onnx=ocr_3-128-256-dynamic.onnx --minShapes=input:1x3x128x256 --optShapes=input:32x3x128x256 --maxShapes=input:32x3x128x256 --saveEngine=batch1-32.engine --workspace=7000 --verbose

This is error log file: https://drive.google.com/file/d/1cKzOW9kbMzWeiS7-L4gzInh4fRt2KMMW/view?usp=sharing

Thanks.

spolisetty · April 14, 2021, 9:04am

Hi @SonTV,

Unfortunately we didn’t get chance to try it on Tesla T4 GPU. We verified it on RTX2080 GPU also. We didn’t face any issue.
Also If your T4 GPU is 8GB, may be workspace=7GB is too big then other apps doesn’t have enough space. We suggest you to use smaller size (3 to 6 GB like that) or try a smaller batch-size(8) and see if still have a problem.

Thank you.

SonTV · April 20, 2021, 3:22am

Hi @spolisetty ,

I think the problem I’m facing is not related to GPU memory, because my T4 GPU is 16GB and when I run convert model with workspace=7GB, there are more than 10GB free space GPU at that time.

I also reduced batch size to smaller number (2, 4, 8) but the issue still occurs.