Failed to convert ONNX to TensorRT (myelinProgramAnalysisError)

SonTV · March 12, 2021, 3:07am

Hi,

I have OCR model built in Pytorch, I converted it to onnx model successfully. But when I use this onnx model with Deepstream, it thows an error:

[4] Assertion failed: !_importer_ctx.network()->hasImplicitBatchDimension() && "This version of the ONNX parser only supports TensorRT INetworkDefinitions with an explicit batch dimension. Please ensure the network was created using the EXPLICIT_BATCH NetworkDefinitionCreationFlag."

Then I tried to convert above onnx model to .trt model use --explicitBatch like this:

/usr/src/tensorrt/bin/trtexec --explicitBatch --onnx=ocr_3-128-256-dynamic.onnx --minShapes=input:1x3x128x256 --optShapes=input:32x3x128x256 --maxShapes=input:32x3x128x256 --saveEngine=model.batch1-32.engine

After a while, myelinProgramAnalysisError appears:

Input filename:   ocr_3-128-256-dynamic.onnx
ONNX IR version:  0.0.6
Opset version:    10
Producer name:    pytorch
Producer version: 1.6
Domain:           
Model version:    0
Doc string:       
----------------------------------------------------------------
[03/12/2021-08:47:18] [W] [TRT] onnx2trt_utils.cpp:220: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[03/12/2021-08:47:18] [W] [TRT] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[03/12/2021-08:47:18] [W] [TRT] Tensor DataType is determined at build time for tensors not marked as input or output.
[03/12/2021-08:47:18] [W] [TRT] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[03/12/2021-08:47:18] [W] [TRT] Tensor DataType is determined at build time for tensors not marked as input or output.
[03/12/2021-08:47:18] [W] [TRT] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[03/12/2021-08:47:18] [W] [TRT] Tensor DataType is determined at build time for tensors not marked as input or output.
[03/12/2021-08:47:18] [W] [TRT] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[03/12/2021-08:47:18] [W] [TRT] Tensor DataType is determined at build time for tensors not marked as input or output.
[03/12/2021-08:47:18] [W] [TRT] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[03/12/2021-08:47:18] [W] [TRT] Tensor DataType is determined at build time for tensors not marked as input or output.
[03/12/2021-08:47:18] [W] [TRT] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[03/12/2021-08:47:18] [W] [TRT] Tensor DataType is determined at build time for tensors not marked as input or output.
[03/12/2021-08:47:18] [W] [TRT] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[03/12/2021-08:47:18] [W] [TRT] Tensor DataType is determined at build time for tensors not marked as input or output.
[03/12/2021-08:47:19] [W] [TRT] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[03/12/2021-08:47:19] [W] [TRT] Tensor DataType is determined at build time for tensors not marked as input or output.
[03/12/2021-08:47:19] [W] [TRT] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[03/12/2021-08:47:19] [W] [TRT] Tensor DataType is determined at build time for tensors not marked as input or output.
[03/12/2021-08:47:19] [W] [TRT] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[03/12/2021-08:47:19] [W] [TRT] Tensor DataType is determined at build time for tensors not marked as input or output.
[03/12/2021-08:47:19] [W] [TRT] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[03/12/2021-08:47:19] [W] [TRT] Tensor DataType is determined at build time for tensors not marked as input or output.
[03/12/2021-08:47:35] [I] [TRT] Some tactics do not have sufficient workspace memory to run. Increasing workspace size may increase performance, please check verbose output.
[03/12/2021-08:48:17] [E] [TRT] ../builder/myelin/codeGenerator.cpp (401) - Myelin Error in compileGraph: 64 (myelinProgramAnalysisError : Mismatched shape for tensor 2969'_468 _ (Unnamed Layer_ 290) [Shuffle]_Cast_8_Cast_9_Unsqueeze_199_(Unnamed Layer_ 498) [Constant]_700 _ (Unnamed Layer_ 612) [Shuffle]_Unsqueeze_371_(Unnamed Layer_ 878) [Constant]_935 _ (Unnamed Layer_ 990) [Shuffle]_Unsqueeze_548_(Unnamed Layer_ 1256) [Constant]_1170 _ (Unnamed Layer_ 1368) [Shuffle]_Unsqueeze_725_(Unnamed Layer_ 1634) [Constant]_1405 _ (Unnamed Layer_ 1746) [Shuffle]_Unsqueeze_902_(Unnamed Layer_ 2012) [Constant]_1640 _ (Unnamed Layer_ 2124) [Shuffle]_Unsqueeze_1079_(Unnamed Layer_ 2390) [Constant]_1875 _ (Unnamed Layer_ 2502) [Shuffle]_Unsqueeze_1256_(Unnamed Layer_ 2768) [Constant]_2110 _ (Unnamed Layer_ 2880) [Shuffle]_Unsqueeze_1433_(Unnamed Layer_ 3146) [Constant]_2345 _ (Unnamed Layer_ 3258) [Shuffle]_Unsqueeze_1610_(Unnamed Layer_ 3524) [Constant]_2580 _ (Unnamed Layer_ 3636) [Shuffle]_Unsqueeze_1787_(Unnamed Layer_ 3902) [Constant]_2815 _ (Unnamed Layer_ 4014) [Shuffle]_Unsqueeze_1964_(Unnamed Layer_ 4280) [Constant]_3050 _ (Unnamed Layer_ 4392) [Shuffle]_Unsqueeze_2141_Equal_2167_Cast_2168_(Unnamed Layer_ 4281) [Identity]_Cast_2111_Equal_1990_Cast_1991_(Unnamed Layer_ 3903) [Identity]_Cast_1934_Equal_1813_Cast_1814_(Unnamed Layer_ 3525) [Identity]_Cast_1757_Equal_1636_Cast_1637_(Unnamed Layer_ 3147) [Identity]_Cast_1580_Equal_1459_Cast_1460_(Unnamed Layer_ 2769) [Identity]_Cast_1403_Equal_1282_Cast_1283_(Unnamed Layer_ 2391) [Identity]_Cast_1226_Equal_1105_Cast_1106_(Unnamed Layer_ 2013) [Identity]_Cast_1049_Equal_928_Cast_929_(Unnamed Layer_ 1635) [Identity]_Cast_872_Equal_751_Cast_752_(Unnamed Layer_ 1257) [Identity]_Cast_695_Equal_574_Cast_575_(Unnamed Layer_ 879) [Identity]_Cast_518_Equal_397_Cast_398_(Unnamed Layer_ 499) [Identity]_Cast_341_Equal_225_Cast_226_471 _ (Unnamed Layer_ 294) [Shuffle]_466 _ (Unnamed Layer_ 287) [Shuffle]_encoder_layers_0_self_attn_linears_0_bias _ (Unnamed Layer_ 250) [Shuffle]_3133 _ (Unnamed Layer_ 247) [Shuffle]_encoder_layers_0_sublayer_0_norm_b_2 _ (Unnamed Layer_ 235) [Shuffle]_encoder_layers_0_sublayer_0_norm_a_2 _ (Unnamed Layer_ 228) [Shuffle]_Sub_193_Mul_194_418 _ (Unnamed Layer_ 231) [Shuffle]_413 _ (Unnamed Layer_ 223) [Shuffle]_411 _ (Unnamed Layer_ 220) [Shuffle]_Mul_185_Sub_186_Abs_187_Mul_189_Div_191_Sqrt_192_Add_196_Div_197_Add_198_MatMul_203_Add_204_Reshape_207 _ Transpose_208_encoder_layers_0_self_attn_linears_1_bias _ (Unnamed Layer_ 263) [Shuffle]_3137 _ (Unnamed Layer_ 260) [Shuffle]_MatMul_209_Add_210_Reshape_213 _ Transpose_220_MatMul_221_Div_223_620 _ (Unnamed Layer_ 502) [Shuffle]_616 _ (Unnamed Layer_ 496) [Shuffle]_decoder_layers_0_self_attn_linears_0_bias _ (Unnamed Layer_ 459) [Shuffle]_3153 _ (Unnamed Layer_ 456) [Shuffle]_decoder_layers_0_sublayer_0_norm_b_2 _ (Unnamed Layer_ 450) [Shuffle]_decoder_layers_0_sublayer_0_norm_a_2 _ (Unnamed Layer_ 443) [Shuffle]_Sub_310_Mul_311_569 _ (Unnamed Layer_ 446) [Shuffle]_564 _ (Unnamed Layer_ 438) [Shuffle]_562 _ (Unnamed Layer_ 435) [Shuffle]_Mul_302_Sub_303_Abs_304_Mul_306_Div_308_Sqrt_309_Add_313_Div_314_Add_315_MatMul_319_Add_320_Reshape_323 _ Transpose_324_decoder_layers_0_self_attn_linears_1_bias _ (Unnamed Layer_ 472) [Shuffle]_3157 _ (Unnamed Layer_ 469) [Shuffle]_MatMul_325_Add_326_Reshape_329 _ Transpose_336_MatMul_337_Div_339_Where_343_Where_228_, current shape:b,[1,1,11,11], vs. expected shape:b,[].
)
[03/12/2021-08:48:17] [E] [TRT] ../builder/myelin/codeGenerator.cpp (401) - Myelin Error in compileGraph: 64 (myelinProgramAnalysisError : Mismatched shape for tensor 2969'_468 _ (Unnamed Layer_ 290) [Shuffle]_Cast_8_Cast_9_Unsqueeze_199_(Unnamed Layer_ 498) [Constant]_700 _ (Unnamed Layer_ 612) [Shuffle]_Unsqueeze_371_(Unnamed Layer_ 878) [Constant]_935 _ (Unnamed Layer_ 990) [Shuffle]_Unsqueeze_548_(Unnamed Layer_ 1256) [Constant]_1170 _ (Unnamed Layer_ 1368) [Shuffle]_Unsqueeze_725_(Unnamed Layer_ 1634) [Constant]_1405 _ (Unnamed Layer_ 1746) [Shuffle]_Unsqueeze_902_(Unnamed Layer_ 2012) [Constant]_1640 _ (Unnamed Layer_ 2124) [Shuffle]_Unsqueeze_1079_(Unnamed Layer_ 2390) [Constant]_1875 _ (Unnamed Layer_ 2502) [Shuffle]_Unsqueeze_1256_(Unnamed Layer_ 2768) [Constant]_2110 _ (Unnamed Layer_ 2880) [Shuffle]_Unsqueeze_1433_(Unnamed Layer_ 3146) [Constant]_2345 _ (Unnamed Layer_ 3258) [Shuffle]_Unsqueeze_1610_(Unnamed Layer_ 3524) [Constant]_2580 _ (Unnamed Layer_ 3636) [Shuffle]_Unsqueeze_1787_(Unnamed Layer_ 3902) [Constant]_2815 _ (Unnamed Layer_ 4014) [Shuffle]_Unsqueeze_1964_(Unnamed Layer_ 4280) [Constant]_3050 _ (Unnamed Layer_ 4392) [Shuffle]_Unsqueeze_2141_Equal_2167_Cast_2168_(Unnamed Layer_ 4281) [Identity]_Cast_2111_Equal_1990_Cast_1991_(Unnamed Layer_ 3903) [Identity]_Cast_1934_Equal_1813_Cast_1814_(Unnamed Layer_ 3525) [Identity]_Cast_1757_Equal_1636_Cast_1637_(Unnamed Layer_ 3147) [Identity]_Cast_1580_Equal_1459_Cast_1460_(Unnamed Layer_ 2769) [Identity]_Cast_1403_Equal_1282_Cast_1283_(Unnamed Layer_ 2391) [Identity]_Cast_1226_Equal_1105_Cast_1106_(Unnamed Layer_ 2013) [Identity]_Cast_1049_Equal_928_Cast_929_(Unnamed Layer_ 1635) [Identity]_Cast_872_Equal_751_Cast_752_(Unnamed Layer_ 1257) [Identity]_Cast_695_Equal_574_Cast_575_(Unnamed Layer_ 879) [Identity]_Cast_518_Equal_397_Cast_398_(Unnamed Layer_ 499) [Identity]_Cast_341_Equal_225_Cast_226_471 _ (Unnamed Layer_ 294) [Shuffle]_466 _ (Unnamed Layer_ 287) [Shuffle]_encoder_layers_0_self_attn_linears_0_bias _ (Unnamed Layer_ 250) [Shuffle]_3133 _ (Unnamed Layer_ 247) [Shuffle]_encoder_layers_0_sublayer_0_norm_b_2 _ (Unnamed Layer_ 235) [Shuffle]_encoder_layers_0_sublayer_0_norm_a_2 _ (Unnamed Layer_ 228) [Shuffle]_Sub_193_Mul_194_418 _ (Unnamed Layer_ 231) [Shuffle]_413 _ (Unnamed Layer_ 223) [Shuffle]_411 _ (Unnamed Layer_ 220) [Shuffle]_Mul_185_Sub_186_Abs_187_Mul_189_Div_191_Sqrt_192_Add_196_Div_197_Add_198_MatMul_203_Add_204_Reshape_207 _ Transpose_208_encoder_layers_0_self_attn_linears_1_bias _ (Unnamed Layer_ 263) [Shuffle]_3137 _ (Unnamed Layer_ 260) [Shuffle]_MatMul_209_Add_210_Reshape_213 _ Transpose_220_MatMul_221_Div_223_620 _ (Unnamed Layer_ 502) [Shuffle]_616 _ (Unnamed Layer_ 496) [Shuffle]_decoder_layers_0_self_attn_linears_0_bias _ (Unnamed Layer_ 459) [Shuffle]_3153 _ (Unnamed Layer_ 456) [Shuffle]_decoder_layers_0_sublayer_0_norm_b_2 _ (Unnamed Layer_ 450) [Shuffle]_decoder_layers_0_sublayer_0_norm_a_2 _ (Unnamed Layer_ 443) [Shuffle]_Sub_310_Mul_311_569 _ (Unnamed Layer_ 446) [Shuffle]_564 _ (Unnamed Layer_ 438) [Shuffle]_562 _ (Unnamed Layer_ 435) [Shuffle]_Mul_302_Sub_303_Abs_304_Mul_306_Div_308_Sqrt_309_Add_313_Div_314_Add_315_MatMul_319_Add_320_Reshape_323 _ Transpose_324_decoder_layers_0_self_attn_linears_1_bias _ (Unnamed Layer_ 472) [Shuffle]_3157 _ (Unnamed Layer_ 469) [Shuffle]_MatMul_325_Add_326_Reshape_329 _ Transpose_336_MatMul_337_Div_339_Where_343_Where_228_, current shape:b,[1,1,11,11], vs. expected shape:b,[].
)
[03/12/2021-08:48:17] [E] Engine creation failed
[03/12/2021-08:48:17] [E] Engine set up failed
&&&& FAILED TensorRT.trtexec # /usr/src/tensorrt/bin/trtexec --explicitBatch --onnx=ocr_3-128-256-dynamic.onnx --minShapes=input:1x3x128x256 --optShapes=input:32x3x128x256 --maxShapes=input:32x3x128x256 --saveEngine=model.batch1-32.engine

Please let me know, how to solve this problem?

Model code, convert_torch_to_onnx code are in attached file.

Environment

• Hardware Platform: Tesla T4
• DeepStream Version: 5.0
• TensorRT Version: 7.2.1
• NVIDIA GPU: Driver Version 455.32, CUDA Version 11.1
• OS: Ubuntu 18.04

code.zip (23.7 KB)

mchi · March 12, 2021, 5:48am

Hi @SonTV ,
is your onnx model batch dynamic?
is it possible to share your input screencap in Netron?

Thanks!

SonTV · March 12, 2021, 7:14am

Hi @mchi ,

This is what I got

mchi · March 12, 2021, 7:28am

Did you enable “force-implicit-batch-dim=” option in the DS config file?

SonTV · March 12, 2021, 7:55am

@mchi Yes, I enabled this property.

Base on deepstream-test2 python example, I edited it to use with only 2 models. YoloV4 as primary gie element run ok, but secondary gie - OCR model not run. Below is all deepstream config I’m using:
deepstream-test2.zip (14.1 KB)

mchi · March 12, 2021, 8:16am

In dstest2_sgie1_config.txt, I saw “force-implicit-batch-dim=1”, could you remove it and try again?
Then, maybe, you will see the same error as TRT with DS.

SonTV · March 12, 2021, 8:24am

@mchi Yep, I have the same error as TRT after removing “force-implicit-batch-dim=1” property in dstest2_sgie1_config.txt.

mchi · March 12, 2021, 8:29am

So, seems there is compatbility issue b/w model and TRT.
Could you file a ticket in TRT forum about this error?

Thanks!

SonTV · March 12, 2021, 9:06am

@mchi Ok, I will do it. Thank for your support.