Description
When converting my onnx model to TensorRT engine (via pre-built trtexec), I ran into a number of issues and eventually the error “kOPT values for profile 0 violate shape constraints: If_525_OutputLayer dimensions not compatible for if-conditional outputs Condition ‘==’ violated”. This error took place not only in my minorly custom model, but also in torchvision’s trained fasterrcnn_resnet50_fpn.
Environment
TensorRT Version: 8.5.3.1+cuda11.8
GPU Type: 1 NVIDIA GeForce GTX 1080
Nvidia Driver Version: 525.78.01
CUDA Version: 11.8
CUDNN Version: 8.6
Operating System + Version: Ubuntu 20.04
Python Version (if applicable): 3.10.8
TensorFlow Version (if applicable):
PyTorch Version (if applicable): 1.13.1 (Torchvision: 0.14.1)
Baremetal or Container (if container which image + tag):
Relevant Files
I am more than happy to provide relevant artifacts (e.g., onnx models) via private messages.
Steps To Reproduce
Context: My model is a torchvision Faster R-CNN model where I used ResNet10 as the backbone, and configured the RCNN head to predict boxes of a single category C. Under pytorch, this means having class “background” and class “C”. The trained pytorch model was first converted into onnx format via torch.onnx.export(…) and then into TensorRT engine. Different opset_version (11-17) all led to the same outcome.
Step1: When converting the onnx model into TensorRT engine (via ./trtexec --onnx=model.onnx --saveEngine=model.trt), I came across the following error:
[02/13/2023-15:54:16] [I] Start parsing network model
[02/13/2023-15:54:16] [I] [TRT] ----------------------------------------------------------------
[02/13/2023-15:54:16] [I] [TRT] Input filename: /home/pryntec/Projects/detectionpytorch/export/model_plate_debug.onnx
[02/13/2023-15:54:16] [I] [TRT] ONNX IR version: 0.0.6
[02/13/2023-15:54:16] [I] [TRT] Opset version: 11
[02/13/2023-15:54:16] [I] [TRT] Producer name: pytorch
[02/13/2023-15:54:16] [I] [TRT] Producer version: 1.13.1
[02/13/2023-15:54:16] [I] [TRT] Domain:
[02/13/2023-15:54:16] [I] [TRT] Model version: 0
[02/13/2023-15:54:16] [I] [TRT] Doc string:
[02/13/2023-15:54:16] [I] [TRT] ----------------------------------------------------------------
[02/13/2023-15:54:16] [W] [TRT] onnx2trt_utils.cpp:377: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[02/13/2023-15:54:16] [W] [TRT] onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[02/13/2023-15:54:16] [W] [TRT] onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[02/13/2023-15:54:16] [W] [TRT] onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[02/13/2023-15:54:16] [W] [TRT] onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[02/13/2023-15:54:16] [W] [TRT] onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[02/13/2023-15:54:16] [E] [TRT] ModelImporter.cpp:726: While parsing node number 407 [TopK -> "/rpn/TopK_output_0"]:
[02/13/2023-15:54:16] [E] [TRT] ModelImporter.cpp:727: --- Begin node ---
[02/13/2023-15:54:16] [E] [TRT] ModelImporter.cpp:728: input: "/rpn/Slice_4_output_0"
input: "/rpn/Reshape_10_output_0"
output: "/rpn/TopK_output_0"
output: "/rpn/TopK_output_1"
name: "/rpn/TopK"
op_type: "TopK"
attribute {
name: "axis"
i: 1
type: INT
}
attribute {
name: "largest"
i: 1
type: INT
}
attribute {
name: "sorted"
i: 1
type: INT
}
[02/13/2023-15:54:16] [E] [TRT] ModelImporter.cpp:729: --- End node ---
[02/13/2023-15:54:16] [E] [TRT] ModelImporter.cpp:732: ERROR: ModelImporter.cpp:168 In function parseGraph:
**[6] Invalid Node - /rpn/TopK**
**This version of TensorRT only supports input K as an initializer. Try applying constant folding on the model using Polygraphy:** https://github.com/NVIDIA/TensorRT/tree/master/tools/Polygraphy/examples/cli/surgeon/02_folding_constants
[02/13/2023-15:54:16] [E] Failed to parse onnx file
[02/13/2023-15:54:17] [I] Finish parsing network model
Step 2: Following the hinted solution, I applied constant folding through Polygraphy surgeon sanitize (the same command as in the official example). When executing the folded onnx with trtexec again, I faced the following:
[02/13/2023-16:04:06] [I] Start parsing network model
[02/13/2023-16:04:06] [I] [TRT] ----------------------------------------------------------------
[02/13/2023-16:04:06] [I] [TRT] Input filename: /home/pryntec/Projects/detectionpytorch/export/model_plate_debug_folded.onnx
[02/13/2023-16:04:06] [I] [TRT] ONNX IR version: 0.0.8
[02/13/2023-16:04:06] [I] [TRT] Opset version: 11
[02/13/2023-16:04:06] [I] [TRT] Producer name: pytorch
[02/13/2023-16:04:06] [I] [TRT] Producer version: 1.13.1
[02/13/2023-16:04:06] [I] [TRT] Domain:
[02/13/2023-16:04:06] [I] [TRT] Model version: 0
[02/13/2023-16:04:06] [I] [TRT] Doc string:
[02/13/2023-16:04:06] [I] [TRT] ----------------------------------------------------------------
[02/13/2023-16:04:06] [W] [TRT] onnx2trt_utils.cpp:377: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[02/13/2023-16:04:06] [W] [TRT] onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[02/13/2023-16:04:06] [W] [TRT] onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[02/13/2023-16:04:06] [W] [TRT] onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[02/13/2023-16:04:06] [W] [TRT] onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[02/13/2023-16:04:06] [W] [TRT] onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[02/13/2023-16:04:06] [W] [TRT] onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[02/13/2023-16:04:06] [W] [TRT] onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[02/13/2023-16:04:06] [W] [TRT] onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[02/13/2023-16:04:06] [W] [TRT] onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[02/13/2023-16:04:06] [W] [TRT] onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[02/13/2023-16:04:06] [W] [TRT] onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[02/13/2023-16:04:06] [W] [TRT] onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[02/13/2023-16:04:06] [W] [TRT] onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[02/13/2023-16:04:06] [W] [TRT] onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[02/13/2023-16:04:06] [W] [TRT] onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[02/13/2023-16:04:06] [W] [TRT] onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[02/13/2023-16:04:06] [E] Error[4]: [graphShapeAnalyzer.cpp::analyzeShapes::1872] **Error Code 4: Miscellaneous (IShuffleLayer /roi_heads/Reshape: reshape wildcard -1 has infinite number of solutions or no solution. Reshaping [0,8] to [0,-1].)**
[02/13/2023-16:04:06] [E] [TRT] ModelImporter.cpp:726: While parsing node number 175 [Reshape -> "/roi_heads/Reshape_output_0"]:
[02/13/2023-16:04:06] [E] [TRT] ModelImporter.cpp:727: --- Begin node ---
[02/13/2023-16:04:06] [E] [TRT] ModelImporter.cpp:728: input: "/roi_heads/box_predictor/bbox_pred/Gemm_output_0"
input: "/roi_heads/Concat_output_0"
output: "/roi_heads/Reshape_output_0"
name: "/roi_heads/Reshape"
op_type: "Reshape"
[02/13/2023-16:04:06] [E] [TRT] ModelImporter.cpp:729: --- End node ---
[02/13/2023-16:04:06] [E] [TRT] ModelImporter.cpp:732: ERROR: ModelImporter.cpp:185 In function parseGraph:
[6] Invalid Node - /roi_heads/Reshape
[graphShapeAnalyzer.cpp::analyzeShapes::1872] Error Code 4: Miscellaneous (IShuffleLayer /roi_heads/Reshape: reshape wildcard -1 has infinite number of solutions or no solution. Reshaping [0,8] to [0,-1].)
[02/13/2023-16:04:06] [E] Failed to parse onnx file
[02/13/2023-16:04:06] [I] Finish parsing network model
Step 3: From the above error and some digging through the internet, it appears that TensorRT does not cope well with wildcard Reshape (i.e., -1). As I could infer the exact tensor shape at roi_heads, I replaced -1s by pre-determined values in the codes of pytorch’s Faster RCNN. This remedy resolved the reshape issue. The newly generated (and then Polygraphy folded) onnx model, however, raised the current issue when being converted into trt engine.
[02/13/2023-16:09:33] [I] Start parsing network model
[02/13/2023-16:09:33] [I] [TRT] ----------------------------------------------------------------
[02/13/2023-16:09:33] [I] [TRT] Input filename: /home/pryntec/Projects/detectionpytorch/export/model_plate_debug_folded.onnx
[02/13/2023-16:09:33] [I] [TRT] ONNX IR version: 0.0.8
[02/13/2023-16:09:33] [I] [TRT] Opset version: 11
[02/13/2023-16:09:33] [I] [TRT] Producer name: pytorch
[02/13/2023-16:09:33] [I] [TRT] Producer version: 1.13.1
[02/13/2023-16:09:33] [I] [TRT] Domain:
[02/13/2023-16:09:33] [I] [TRT] Model version: 0
[02/13/2023-16:09:33] [I] [TRT] Doc string:
[02/13/2023-16:09:33] [I] [TRT] ----------------------------------------------------------------
[02/13/2023-16:09:33] [W] [TRT] onnx2trt_utils.cpp:377: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[02/13/2023-16:09:33] [W] [TRT] onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[02/13/2023-16:09:33] [W] [TRT] onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[02/13/2023-16:09:33] [W] [TRT] onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[02/13/2023-16:09:33] [W] [TRT] onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[02/13/2023-16:09:33] [W] [TRT] onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[02/13/2023-16:09:33] [W] [TRT] onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[02/13/2023-16:09:33] [W] [TRT] onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[02/13/2023-16:09:33] [W] [TRT] onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[02/13/2023-16:09:33] [W] [TRT] onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[02/13/2023-16:09:33] [W] [TRT] onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[02/13/2023-16:09:33] [W] [TRT] onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[02/13/2023-16:09:33] [W] [TRT] onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[02/13/2023-16:09:33] [W] [TRT] onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[02/13/2023-16:09:33] [W] [TRT] onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[02/13/2023-16:09:33] [W] [TRT] onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[02/13/2023-16:09:33] [W] [TRT] onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[02/13/2023-16:09:33] [W] [TRT] onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[02/13/2023-16:09:33] [I] Finish parsing network model
[02/13/2023-16:09:33] [I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +194, GPU +76, now: CPU 720, GPU 1033 (MiB)
[02/13/2023-16:09:33] [I] [TRT] [MemUsageChange] Init cuDNN: CPU +111, GPU +56, now: CPU 831, GPU 1089 (MiB)
[02/13/2023-16:09:33] [I] [TRT] Local timing cache in use. Profiling results in this builder pass will not be stored.
[02/13/2023-16:09:33] [E] Error[4]: [shapeCompiler.cpp::evaluateShapeChecks::1180] **Error Code 4: Internal Error (kOPT values for profile 0 violate shape constraints: If_525_OutputLayer: dimensions not compatible for if-conditional outputs Condition '==' violated: 0 != 125.)**
[02/13/2023-16:09:33] [E] Error[2]: [builder.cpp::buildSerializedNetwork::751] Error Code 2: Internal Error (Assertion engine != nullptr failed. )
[02/13/2023-16:09:33] [E] Engine could not be created from network
After visualizing my onnx model with netron, it appears the above error was raised at which pytorch carries out NMS in RPN (for reducing number of proposals). Specifically, I deduced that the error took place at the line “if boxes.numel() == 0” (where pytorch behaves differently when there remain zero proposals to filter). Nevertheless, I am not able to grasp how this line would raise a shape constraint error for TensorRT, nor did I see a similar post. I confirm that both my pytorch and onnx models generate reasonable outputs.
I appreciate any suggestions and am happy to provide relevant artifacts/logs. Thanks in advance!