Error Code 1: Internal Error (Error: Weights of same values but of different types are used in the network!)

AI & Data Science Deep Learning (Training & Inference) TensorRT
, ,
1

Description

I’m trying to convert a Pytorch model to onnx then convert it to a tensorRT engine using trtexec and got the following error

[E] Error[1]: [network.cpp::setWeightsName::3366] Error Code 1: Internal Error (Error: Weights of same values but of different types are used in the network!)

Here’s the full output:

[03/21/2023-07:32:23] [W] [TRT] onnx2trt_utils.cpp:375: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[03/21/2023-07:32:23] [W] [TRT] onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[03/21/2023-07:32:26] [E] Error[1]: [network.cpp::setWeightsName::3366] Error Code 1: Internal Error (Error: Weights of same values but of different types are used in the network!)
[03/21/2023-07:32:26] [E] [TRT] ModelImporter.cpp:726: While parsing node number 1288 [ConstantOfShape -> "/anchor_generator/ConstantOfShape_output_0"]:
[03/21/2023-07:32:26] [E] [TRT] ModelImporter.cpp:727: --- Begin node ---
[03/21/2023-07:32:26] [E] [TRT] ModelImporter.cpp:728: input: "/anchor_generator/Constant_12_output_0"
output: "/anchor_generator/ConstantOfShape_output_0"
name: "/anchor_generator/ConstantOfShape"
op_type: "ConstantOfShape"
attribute {
  name: "value"
  t {
    dims: 1
    data_type: 7
    raw_data: "\000\000\000\000\000\000\000\000"
  }
  type: TENSOR
}

[03/21/2023-07:32:26] [E] [TRT] ModelImporter.cpp:729: --- End node ---
[03/21/2023-07:32:26] [E] [TRT] ModelImporter.cpp:731: ERROR: ModelImporter.cpp:172 In function parseGraph:
[6] Invalid Node - /anchor_generator/ConstantOfShape
[network.cpp::setWeightsName::3366] Error Code 1: Internal Error (Error: Weights of same values but of different types are used in the network!)
[03/21/2023-07:32:26] [E] Failed to parse onnx file
[03/21/2023-07:32:26] [E] Parsing model failed
[03/21/2023-07:32:26] [E] Failed to create engine from model or file.
[03/21/2023-07:32:26] [E] Engine set up failed

Environment

Device: Orin AGX
Dockerfile Content:

FROM nvcr.io/nvidia/l4t-tensorrt:r8.5.2.2-devel
RUN apt update
RUN apt --fix-broken install
RUN apt install -y mpich
RUN pip install mpi4py
RUN pip install torchinfo
RUN pip install clearml
RUN pip install segmentation-models-pytorch
RUN pip install transformers
RUN pip install nvidia-tao-deploy

Relevant Files

Link to .onnx file

Steps To Reproduce

Download the .onnx file & Run the command below in terminal
RetinaNet_(1, 3, 1080, 1440)_15.log (5.4 KB)

/usr/src/tensorrt/bin/trtexec --onnx='/workspaces/scratch_ai/onnx/RetinaNet_(1, 3, 1080, 1440)_15.onnx' --saveEngine='/workspaces/scratch_ai/trt/RetinaNet_(1, 3, 1080, 1440)_15.engine' --exportProfile='/workspaces/scratch_ai/trt/RetinaNet_(1, 3, 1080, 1440)_15.json' --allowGPUFallback --useSpinWait --separateProfileRun > '/workspaces/scratch_ai/trt/RetinaNet_(1, 3, 1080, 1440)_15.log'
2

Hi,
Request you to share the ONNX model and the script if not shared already so that we can assist you better.
Alongside you can try few things:

  1. validating your model with the below snippet

check_model.py

import sys
import onnx
filename = yourONNXmodel
model = onnx.load(filename)
onnx.checker.check_model(model).
2) Try running your model with trtexec command.

In case you are still facing issue, request you to share the trtexec “”–verbose"" log for further debugging
Thanks!

4

The ONNX model is in the shared link. Can’t upload due to file size.

5

Were you ever able to resolve this?

6

Hi,

Please try on the latest TensorRT version 8.6.1, and let us know if you still face the same issue.

Thank you.