Conversion error of Mask RCNN ONNX model for different types weights

AI & Data Science Deep Learning (Training & Inference) TensorRT
, , ,
1

Description

The Torchvision Mask RCNN model cannot be converted in TensorRT engine for the following error:

[05/09/2023-14:46:31] [E] [TRT] ModelImporter.cpp:731: ERROR: ModelImporter.cpp:172 In function parseGraph:
[6] Invalid Node - /rpn/anchor_generator/ConstantOfShape
[network.cpp::setWeightsName::3366] Error Code 1: Internal Error (Error: Weights of same values but of different types are used in the network!)

Please let me know on how to fix this issue.

Environment

TensorRT Version: 8.5.2.2
GPU Type: Jetson Orin Nano
Nvidia Driver Version: JetPack 5.1.1
CUDA Version: 11.4.19
CUDNN Version: 8.6.0

Operating System + Version: JetPack 5.1.1
Python Version (if applicable): 3.8
TensorFlow Version (if applicable):
PyTorch Version (if applicable): 2.0.0
Baremetal or Container (if container which image + tag): Baremetal

Relevant Files

maskrcnn.onnx
maskrcnn_pth2onnx.py (1.9 KB)

Steps To Reproduce

/usr/src/tensorrt/bin/trtexec --onnx=maskrcnn.onnx --saveEngine=maskrcnn.trt --workspace=2048 --exportProfile=profile.json --verbose

Steps to regenerate the ONNX model on x86 platform

  1. Place the attached maskrcnn_pth2onnx.py in any user path.
  2. Execute the following docker command at the path which contains maskrcnn_pth2onnx.py.
docker run -it --rm --gpus=all --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 --network=host -v ${PWD}:"/maskrcnn_test" -w "/maskrcnn_test" "nvcr.io/nvidia/pytorch:23.04-py3" python maskrcnn_pth2onnx.py

Steps to reproduce the issue on x86 platform

I’d like to have a TensorRT engine for Jetson Orin. But you can reproduce the issue on x86 platform too.

docker run -it --rm --gpus=all --network=host -v ${PWD}:"/maskrcnn_test" -w "/maskrcnn_test" "nvcr.io/nvidia/tensorrt:23.01-py3" trtexec --onnx=maskrcnn.onnx --saveEngine=maskrcnn.trt --workspace=2048 --exportProfile=profile.json --verbose
2

Hi,
Request you to share the ONNX model and the script if not shared already so that we can assist you better.
Alongside you can try few things:

  1. validating your model with the below snippet

check_model.py

import sys
import onnx
filename = yourONNXmodel
model = onnx.load(filename)
onnx.checker.check_model(model).
2) Try running your model with trtexec command.

In case you are still facing issue, request you to share the trtexec “”–verbose"" log for further debugging
Thanks!

4

@AakankshaS
Thank you for your reply.
I have already attached the ONNX model ‘maskrcnn.onnx’ at the 1st post.
I have validated the model with the way you suggested, please find the code in maskrcnn_pth2onnx.py which also attached at the 1st post.

5

Hello. How are things going for this issue?

6

The error message I had, said “Weights of same values but of different types are used in the network!”, so I suspected that the “value” attribute of /rpn/anchor_generator/ConstantOfShape was invalid, because the attribute type was INT64 but TensorRT is supporting FP32 only. Then I removed the “value” attribute with onnx-graphsurgeon so that TensorRT will treat it as FP32 according to the ONNX specification. But I still have the same error from trtexec.
Please let me know the meaning of the error message.

Python code to remove the “value” attribute from the /rpn/anchor_generator/ConstantOfShape node:
modify_onnx_model.py (1.4 KB)

trtexec log for the original model:
trtexec_log_org.txt (507.2 KB)

trtexec log for the modified model:
trtexec_log_mod.txt (507.2 KB)

Thanks.

7

Hi,

We tried the polygraphy tool, trtexec, and faced the following error. It looks like there is a problem with the model input.

polygraphy surgeon sanitize maskrcnn_modified.onnx --fold-constants --output model_folded.onnx

[05/30/2023-06:15:14] [V] [TRT] Registering layer: /roi_heads/Reshape for ONNX node: /roi_heads/Reshape
[05/30/2023-06:15:14] [E] Error[4]: [shapeContext.cpp::operator()::3602] Error Code 4: Shape Error (reshape wildcard -1 has infinite number of solutions or no solution. Reshaping [0,8] to [0,-1].)
[05/30/2023-06:15:14] [E] [TRT] ModelImporter.cpp:771: While parsing node number 633 [Reshape -> "/roi_heads/Reshape_output_0"]:
[05/30/2023-06:15:14] [E] [TRT] ModelImporter.cpp:772: --- Begin node ---
[05/30/2023-06:15:14] [E] [TRT] ModelImporter.cpp:773: input: "/roi_heads/box_predictor/bbox_pred/Gemm_output_0"
input: "/roi_heads/Concat_output_0"
output: "/roi_heads/Reshape_output_0"
name: "/roi_heads/Reshape"
op_type: "Reshape"
attribute {
  name: "allowzero"
  i: 0
  type: INT
}

[05/30/2023-06:15:14] [E] [TRT] ModelImporter.cpp:774: --- End node ---
[05/30/2023-06:15:14] [E] [TRT] ModelImporter.cpp:777: ERROR: ModelImporter.cpp:195 In function parseGraph:
[6] Invalid Node - /roi_heads/Reshape
[shapeContext.cpp::operator()::3602] Error Code 4: Shape Error (reshape wildcard -1 has infinite number of solutions or no solution. Reshaping [0,8] to [0,-1].)
[05/30/2023-06:15:14] [E] Failed to parse onnx file
[05/30/2023-06:15:14] [I] Finished parsing network model. Parse time: 0.473263
[05/30/2023-06:15:14] [E] Parsing model failed
[05/30/2023-06:15:14] [E] Failed to create engine from model or file.
[05/30/2023-06:15:14] [E] Engine set up failed
&&&& FAILED TensorRT.trtexec [TensorRT v8601] # trtexec --onnx=model_folded.onnx --verbose --workspace=20000

Please refer to the similar issues below.

Thank you.