Error code 4 internal error unnamed layer

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

Description

I successfully converted TensorFlow model to ONNX with tf2onnx. The information about the successful conversion was printed out in the terminal.

Now I want to convert an ONNX model to a TensorRT engine. I downloaded and started the latest docker [2]. Then, I just started trtexec [3] and got an error message “Found invalid input type of UINT8” which I could solve with the help of a small script which I found here [4]. With the help of this scrip I was able to fix this error. So far, so good.

I repeated to use trtexec with the same command [3] and got another error which I am not able so solve. See output below. So what does this error mean and how can I solve it ? I have attached the model for investigation.

Thank you and I hope you have a woundeful start in 2024

[01/01/2024-20:07:38] [W] [TRT] onnx2trt_utils.cpp:374: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[01/01/2024-20:07:38] [W] [TRT] onnx2trt_utils.cpp:400: One or more weights outside the range of INT32 was clamped
[01/01/2024-20:07:38] [E] Error[4]: [graph.cpp::symbolicExecute::539] Error Code 4: Internal Error ((Unnamed Layer* 76) [LoopOutput]: an ILoopOutputLayer cannot be used to compute a shape tensor)
[01/01/2024-20:07:38] [E] [TRT] ModelImporter.cpp:771: While parsing node number 354 [Range -> "StatefulPartitionedCall/Postprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/range_2:0"]:
[01/01/2024-20:07:38] [E] [TRT] ModelImporter.cpp:772: --- Begin node ---
[01/01/2024-20:07:38] [E] [TRT] ModelImporter.cpp:773: input: "StatefulPartitionedCall/Postprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/range_1/start:0"
input: "StatefulPartitionedCall/Postprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/Select_2:0"
input: "StatefulPartitionedCall/Postprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/range_2/delta:0"
output: "StatefulPartitionedCall/Postprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/range_2:0"
name: "StatefulPartitionedCall/Postprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/range_2"
op_type: "Range"

[01/01/2024-20:07:38] [E] [TRT] ModelImporter.cpp:774: --- End node ---
[01/01/2024-20:07:38] [E] [TRT] ModelImporter.cpp:777: ERROR: ModelImporter.cpp:195 In function parseGraph:
[6] Invalid Node - StatefulPartitionedCall/Postprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/range_2
[graph.cpp::symbolicExecute::539] Error Code 4: Internal Error ((Unnamed Layer* 76) [LoopOutput]: an ILoopOutputLayer cannot be used to compute a shape tensor)
[01/01/2024-20:07:38] [E] Failed to parse onnx file
[01/01/2024-20:07:38] [I] Finished parsing network model. Parse time: 0.0658785

[1]
python -m tf2onnx.convert --saved-model /home/playground/export_v1/saved_model/ --output /home/playground/model.onnx

[2]
sudo docker run -v /home/ubuntu/eric:/home --gpus all -it --rm nvcr.io/nvidia/tensorrt:23.12-py3 /bin/bash

[3]
trtexec --onnx=/home/playground/model.onnx --saveEngine=engine.trt

[4]

[5]

Environment

TensorRT Version: 8.6.1
GPU Type:
Nvidia Driver Version:
CUDA Version:
CUDNN Version:
Operating System + Version:
Python Version (if applicable): 3.10.12
TensorFlow Version (if applicable):
PyTorch Version (if applicable):
Baremetal or Container (if container which image + tag):
**PIP

Package Version

appdirs 1.4.4
graphsurgeon 0.4.6
install 1.3.5
Mako 1.3.0
MarkupSafe 2.1.3
numpy 1.23.5
nvidia-pyindex 1.0.9
onnx 1.15.0
onnx-graphsurgeon 0.3.27
Pillow 10.1.0
pip 23.3.1
platformdirs 4.0.0
polygraphy 0.49.3
protobuf 4.25.1
pycuda 2023.1
pytools 2023.1.1
setuptools 59.6.0
tensorrt 8.6.1
typing_extensions 4.8.0
uff 0.6.9
wheel 0.37.1

Relevant Files

Please attach or include links to any models, data, files, or scripts necessary to reproduce your issue. (Github repo, Google Drive, Dropbox, etc.)

Steps To Reproduce

Please include:

  • Exact steps/commands to build your repro
  • Exact steps/commands to run your repro
  • Full traceback of errors encountered
2

Hi @eric_langner ,
You may have to consider the below pointers,

  1. Please create the network with EXPLICIT_BATCH flag.
  2. The input to the FullyConnected layers are expected to have at least 4 dimensions
    Please refer to the link.
    onnx-tensorrt/README.md at main · onnx/onnx-tensorrt · GitHub
    Thanks
3

Hi @AakankshaS

This means I have to retrain my network. I am asking myself if there is also a solution where you don’t have to retrain the network ?

4

I couldn’t get any further at this point and solved my problem in another way.
Thank you.

5

Hi Eric,

How did you solve this another way? I’m getting a similar error.

Would be much appreciated!
Tim

6

@timf34 Unfortunately I didn’t get any further at that point. To avoid the error, I tried modifying the network without success (because I have no experience with it). After that and in the discussion with my colleagues it came out that it could possibly be a buggy version of the TensorRT engine (as far as I can remember). I hope that helps.

1 Like
7

Thanks for replying Eric! I managed to solve my issue thankfully!

8

FYI for anyone else who sees this, my fix involved isolating the line of code in the neural network design via looking at the verbose output when creating the onnx model to isolate where the issue was occurring. And then using GPT4/ online forums to help with replacing that line of code with something suitable

9

@timf34 From your comment I understand that you replaced the problematic layer in your network before training and then you trained it with this change, right? I have the same issue except I don’t have the chance to modify the model before training, so it is crucial for me to understand if this is even possible. Thanks!

10

Hi, yes youre correct, I had to modify the layers in the model (the torch layers) used.
Best,
Tim

1 Like