Conversion to tensorRT error . [graphShapeAnalyzer.cpp::throwIfError::1306] Error Code 9

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

Description

I want to infer a customized SSD on TensorRT at Jetson Nano.
I tried to convert onnx to TensorRT Model using the following command and encountered the following error.

command

trtexec --onnx=xxxx.onnx --explicitBatch --saveEngine=xxxx.trt

Error Message

[08/10/2021-19:20:10] [E] Error[9]: [graphShapeAnalyzer.cpp::throwIfError::1306] Error Code 9: Internal Error (Postprocessor/Reshape: -1 wildcard solution does not fit in int32_t
)
[08/10/2021-19:20:10] [E] [TRT] ModelImporter.cpp:720: While parsing node number 360 [Reshape -> "Postprocessor/Reshape:0"]:
[08/10/2021-19:20:10] [E] [TRT] ModelImporter.cpp:721: --- Begin node ---
[08/10/2021-19:20:10] [E] [TRT] ModelImporter.cpp:722: input: "Postprocessor/Tile:0"
input: "const_fold_opt__5985"
output: "Postprocessor/Reshape:0"
name: "Postprocessor/Reshape"
op_type: "Reshape"

[08/10/2021-19:20:10] [E] [TRT] ModelImporter.cpp:723: --- End node ---
[08/10/2021-19:20:10] [E] [TRT] ModelImporter.cpp:726: ERROR: ModelImporter.cpp:179 In function parseGraph:
[6] Invalid Node - Postprocessor/Reshape
[graphShapeAnalyzer.cpp::throwIfError::1306] Error Code 9: Internal Error (Postprocessor/Reshape: -1 wildcard solution does not fit in int32_t
)
[08/10/2021-19:20:10] [E] Failed to parse onnx file
[08/10/2021-19:20:10] [I] Finish parsing network model
[08/10/2021-19:20:10] [E] Parsing model failed
[08/10/2021-19:20:10] [E] Engine creation failed
[08/10/2021-19:20:10] [E] Engine set up failed

I want to know “-1 wildcard solution does not fit in int32_t” means.

And I also want to know if I need to change the implementation of the model.

This customized SSD is implemented in Tensorflow.

If this is a problem that can be dealt with in the command options when converting from tensorflow to ONNX or ONNX to TensorRT, I would be very happy to get a hint.

If I need to implement the model itself for success converting to TensorRT, I can do that.
However, I would like to avoid changing the implementation if possible because it would have a large impact.

Environment

Jetpack version : 4.6(L4T 6.1)
TensorRT Version: 8.0.1.6
GPU Type: GPU embeded on Jetson nano
Nvidia Driver Version:
CUDA Version: 10.2
CUDNN Version: 8.2.1
Operating System + Version: Jetpack 4.6 (L4T 6.1, Ubuntu18.04)
Python Version (if applicable): 3.6.9
TensorFlow Version (if applicable): 1.15.5
ONNX OPset version: 12
PyTorch Version (if applicable): no use
Baremetal or Container (if container which image + tag):

Full log

[08/10/2021-19:20:06] [I] === Model Options ===
[08/10/2021-19:20:06] [I] Format: ONNX
[08/10/2021-19:20:06] [I] Model: models/person_face_detection_model_frozen_960_540.onnx
[08/10/2021-19:20:06] [I] Output:
[08/10/2021-19:20:06] [I] === Build Options ===
[08/10/2021-19:20:06] [I] Max batch: explicit
[08/10/2021-19:20:06] [I] Workspace: 16 MiB
[08/10/2021-19:20:06] [I] minTiming: 1
[08/10/2021-19:20:06] [I] avgTiming: 8
[08/10/2021-19:20:06] [I] Precision: FP32
[08/10/2021-19:20:06] [I] Calibration: 
[08/10/2021-19:20:06] [I] Refit: Disabled
[08/10/2021-19:20:06] [I] Sparsity: Disabled
[08/10/2021-19:20:06] [I] Safe mode: Disabled
[08/10/2021-19:20:06] [I] Restricted mode: Disabled
[08/10/2021-19:20:06] [I] Save engine: models/person_face_detection_model_frozen_960_540.trt
[08/10/2021-19:20:06] [I] Load engine: 
[08/10/2021-19:20:06] [I] NVTX verbosity: 0
[08/10/2021-19:20:06] [I] Tactic sources: Using default tactic sources
[08/10/2021-19:20:06] [I] timingCacheMode: local
[08/10/2021-19:20:06] [I] timingCacheFile: 
[08/10/2021-19:20:06] [I] Input(s)s format: fp32:CHW
[08/10/2021-19:20:06] [I] Output(s)s format: fp32:CHW
[08/10/2021-19:20:06] [I] Input build shapes: model
[08/10/2021-19:20:06] [I] Input calibration shapes: model
[08/10/2021-19:20:06] [I] === System Options ===
[08/10/2021-19:20:06] [I] Device: 0
[08/10/2021-19:20:06] [I] DLACore: 
[08/10/2021-19:20:06] [I] Plugins:
[08/10/2021-19:20:06] [I] === Inference Options ===
[08/10/2021-19:20:06] [I] Batch: Explicit
[08/10/2021-19:20:06] [I] Input inference shapes: model
[08/10/2021-19:20:06] [I] Iterations: 10
[08/10/2021-19:20:06] [I] Duration: 3s (+ 200ms warm up)
[08/10/2021-19:20:06] [I] Sleep time: 0ms
[08/10/2021-19:20:06] [I] Streams: 1
[08/10/2021-19:20:06] [I] ExposeDMA: Disabled
[08/10/2021-19:20:06] [I] Data transfers: Enabled
[08/10/2021-19:20:06] [I] Spin-wait: Disabled
[08/10/2021-19:20:06] [I] Multithreading: Disabled
[08/10/2021-19:20:06] [I] CUDA Graph: Disabled
[08/10/2021-19:20:06] [I] Separate profiling: Disabled
[08/10/2021-19:20:06] [I] Time Deserialize: Disabled
[08/10/2021-19:20:06] [I] Time Refit: Disabled
[08/10/2021-19:20:06] [I] Skip inference: Disabled
[08/10/2021-19:20:06] [I] Inputs:
[08/10/2021-19:20:06] [I] === Reporting Options ===
[08/10/2021-19:20:06] [I] Verbose: Disabled
[08/10/2021-19:20:06] [I] Averages: 10 inferences
[08/10/2021-19:20:06] [I] Percentile: 99
[08/10/2021-19:20:06] [I] Dump refittable layers:Disabled
[08/10/2021-19:20:06] [I] Dump output: Disabled
[08/10/2021-19:20:06] [I] Profile: Disabled
[08/10/2021-19:20:06] [I] Export timing to JSON file: 
[08/10/2021-19:20:06] [I] Export output to JSON file: 
[08/10/2021-19:20:06] [I] Export profile to JSON file: 
[08/10/2021-19:20:06] [I] 
[08/10/2021-19:20:06] [I] === Device Information ===
[08/10/2021-19:20:06] [I] Selected Device: NVIDIA Tegra X1
[08/10/2021-19:20:06] [I] Compute Capability: 5.3
[08/10/2021-19:20:06] [I] SMs: 1
[08/10/2021-19:20:06] [I] Compute Clock Rate: 0.9216 GHz
[08/10/2021-19:20:06] [I] Device Global Memory: 3956 MiB
[08/10/2021-19:20:06] [I] Shared Memory per SM: 64 KiB
[08/10/2021-19:20:06] [I] Memory Bus Width: 64 bits (ECC disabled)
[08/10/2021-19:20:06] [I] Memory Clock Rate: 0.01275 GHz
[08/10/2021-19:20:06] [I] 
[08/10/2021-19:20:06] [I] TensorRT version: 8001
[08/10/2021-19:20:09] [I] [TRT] [MemUsageChange] Init CUDA: CPU +203, GPU +0, now: CPU 221, GPU 3340 (MiB)
[08/10/2021-19:20:09] [I] Start parsing network model
[08/10/2021-19:20:09] [I] [TRT] ----------------------------------------------------------------
[08/10/2021-19:20:09] [I] [TRT] Input filename:   models/person_face_detection_model_frozen_960_540.onnx
[08/10/2021-19:20:09] [I] [TRT] ONNX IR version:  0.0.7
[08/10/2021-19:20:09] [I] [TRT] Opset version:    12
[08/10/2021-19:20:09] [I] [TRT] Producer name:    tf2onnx
[08/10/2021-19:20:09] [I] [TRT] Producer version: 1.9.1
[08/10/2021-19:20:09] [I] [TRT] Domain:           
[08/10/2021-19:20:09] [I] [TRT] Model version:    0
[08/10/2021-19:20:09] [I] [TRT] Doc string:       
[08/10/2021-19:20:09] [I] [TRT] ----------------------------------------------------------------
[08/10/2021-19:20:09] [W] [TRT] onnx2trt_utils.cpp:364: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[08/10/2021-19:20:09] [W] [TRT] onnx2trt_utils.cpp:390: One or more weights outside the range of INT32 was clamped
[08/10/2021-19:20:09] [W] [TRT] onnx2trt_utils.cpp:390: One or more weights outside the range of INT32 was clamped
[08/10/2021-19:20:10] [E] Error[9]: [graphShapeAnalyzer.cpp::throwIfError::1306] Error Code 9: Internal Error (Postprocessor/Reshape: -1 wildcard solution does not fit in int32_t
)
[08/10/2021-19:20:10] [E] [TRT] ModelImporter.cpp:720: While parsing node number 360 [Reshape -> "Postprocessor/Reshape:0"]:
[08/10/2021-19:20:10] [E] [TRT] ModelImporter.cpp:721: --- Begin node ---
[08/10/2021-19:20:10] [E] [TRT] ModelImporter.cpp:722: input: "Postprocessor/Tile:0"
input: "const_fold_opt__5985"
output: "Postprocessor/Reshape:0"
name: "Postprocessor/Reshape"
op_type: "Reshape"

[08/10/2021-19:20:10] [E] [TRT] ModelImporter.cpp:723: --- End node ---
[08/10/2021-19:20:10] [E] [TRT] ModelImporter.cpp:726: ERROR: ModelImporter.cpp:179 In function parseGraph:
[6] Invalid Node - Postprocessor/Reshape
[graphShapeAnalyzer.cpp::throwIfError::1306] Error Code 9: Internal Error (Postprocessor/Reshape: -1 wildcard solution does not fit in int32_t
)
[08/10/2021-19:20:10] [E] Failed to parse onnx file
[08/10/2021-19:20:10] [I] Finish parsing network model
[08/10/2021-19:20:10] [E] Parsing model failed
[08/10/2021-19:20:10] [E] Engine creation failed
[08/10/2021-19:20:10] [E] Engine set up failed
2

Hi,
Please refer to below links related custom plugin implementation and sample:
https://github.com/NVIDIA/TensorRT/tree/master/samples/opensource/sampleOnnxMnistCoordConvAC

While IPluginV2 and IPluginV2Ext interfaces are still supported for backward compatibility with TensorRT 5.1 and 6.0.x respectively, however, we recommend that you write new plugins or refactor existing ones to target the IPluginV2DynamicExt or IPluginV2IOExt interfaces instead.

Thanks!

4

@NVES
Thanks for the answer.

Later, I found the following thread mentioning a similar error.

I understood about Reshape is not good with the wildcard(-1) specification at TensorRT.
I then used the following script to remove wildcard specified from the Reshape node.

Graph edit code.

import graphsurgeon as gs
.
.
.
graph = gs.DynamicGraph(input_path)

old_node = graph.find_nodes_by_path("Postprocessor/Reshape/shape")
new_node = gs.create_plugin_node(name='Postprocessor/Reshape/shape', op='Const', dtype=tf.int32, value=np.array([2034, 4], dtype=np.int32))
new_node.attr['value'].tensor.dtype = 3  # 3=DT_INT32
graph.collapse_namespaces({"Postprocessor/Reshape/shape": new_shape_node})

graph.write(output_path]

The following images were confirmed by Netron.

Original Graph

Edited Graph

I confirmed that Inference using the edited model was correct.

Then I tried to convert again and got another error.


 [E] Error[9]: [graphShapeAnalyzer.cpp::throwIfError::1306] Error Code 9: Internal Error (Postprocessor/Reshape: reshape changes volume)

The error message has changed, but the error is occurring on the same node (Postprocessor/Reshape).
What does this error mean?

Reshape node not change total size because 1x2034x4(input size) = 2034x4(reshaped size).

I want some advice.

5

Hi @mutsuyuki,

Could you please share with us minimal issue repro ONNX model and verbose logs to try from our end for better debugging.

Thank you.

6

@spolisetty

Thank you for your answer.

I sent you onnx model and verbose log by private message.
Please try to debug it.

I really appreciate your help.

1 Like
7

Hi @mutsuyuki ,
Can you please share your model again with us.

Thanks.

8

Hi @mutsuyuki,

Thank you for sharing the model, we could reproduce the issue. Please allow us sometime to work on this.

9

I’m glad to hear that you were able to reproduce the issue, which is first step to solve.
Can you tell me reason of this issue?

10

I’m facing this same problem with same environment (custom SSD model, Jetson Nano etc…).
I’m also curious what is causing the issue.

11

Hi @spolisetty @AakankshaS @mutsuyuki !

I’m having the same issue, did you solve it?

I’m trying to use this repo:
https://github.com/pskiran1/TensorRT-support-for-Tensorflow-2-Object-Detection-Models

Is that repo official from Nvidia?

Thanks!

12

i am also facing the same issue, is this issue solved?