Description
I want to convert swin transformer model with dynamic shape to tensorrt. Previously, I tried with static input shape and I could convert the model correctly but, with dynamic shape I’m getting “IShuffleLayer applied to shape tensor must have 0 or 1 reshape dimensions: dimensions were [-1,2])” error.
Environment
TensorRT Version: 8.2.1.8
GPU Type: Jetson TX2
Nvidia Driver Version: 32.7.2
CUDA Version: 10.2.300
CUDNN Version:
Operating System + Version: Ubuntu 18.04.6 LTS
Python Version (if applicable): 3.6.9
TensorFlow Version (if applicable):
PyTorch Version (if applicable): 1.8.0
Baremetal or Container (if container which image + tag):
Relevant Files
Steps To Reproduce
First, I use polygraphy for constant-folding.
polygraphy surgeon sanitize --fold-constants upernet_swin_base_dynamic_1080x608.onnx -o upernet_swin_base_dynamic_1080x608_folded.onnx
[I] Inferring shapes in the model with onnxruntime.tools.symbolic_shape_infer.
Note: To force Polygraphy to use onnx.shape_inference instead, set allow_onnxruntime=False or use the --no-onnxruntime-shape-inference command-line option.
[I] Loading model: /home/nvidia/Documentos/upernet_swin_base_dynamic_1080x608.onnx
Unable to determine if floor(height/4) <= unk__1, treat as equal
Unable to determine if floor(width/4) <= unk__2, treat as equal
Unable to determine if floor(height/4) <= unk__1, treat as equal
Unable to determine if floor(width/4) <= unk__2, treat as equal
Unable to determine if floor(height/4) <= unk__1, treat as equal
Unable to determine if floor(width/4) <= unk__2, treat as equal
Unable to determine if floor(floor(height/4)/2) <= unk__1, treat as equal
Unable to determine if floor(floor(width/4)/2) <= unk__2, treat as equal
[W] ONNX shape inference exited with an error:
[I] Loading model: /home/nvidia/Documentos/upernet_swin_base_dynamic_1080x608.onnx
[I] Original Model:
Name: torch-jit-export | ONNX Opset: 11
---- 1 Graph Input(s) ----
{input [dtype=float32, shape=('batch', 3, 'height', 'width')]}
---- 1 Graph Output(s) ----
{output [dtype=int64, shape=('batch', 1, 'height', 'width')]}
---- 1218 Initializer(s) ----
---- 4534 Node(s) ----
[I] Folding Constants | Pass 1
Unable to determine if floor(height/4) <= unk__1, treat as equal
Unable to determine if floor(width/4) <= unk__2, treat as equal
Unable to determine if floor(height/4) <= unk__1, treat as equal
Unable to determine if floor(width/4) <= unk__2, treat as equal
Unable to determine if floor(height/4) <= unk__1, treat as equal
Unable to determine if floor(width/4) <= unk__2, treat as equal
Unable to determine if floor(floor(height/4)/2) <= unk__1, treat as equal
Unable to determine if floor(floor(width/4)/2) <= unk__2, treat as equal
[W] ONNX shape inference exited with an error:
[I] Total Nodes | Original: 4534, After Folding: 4472 | 62 Nodes Folded
[I] Folding Constants | Pass 2
Unable to determine if floor(height/4) <= unk__1, treat as equal
Unable to determine if floor(width/4) <= unk__2, treat as equal
Unable to determine if floor(height/4) <= unk__1, treat as equal
Unable to determine if floor(width/4) <= unk__2, treat as equal
Unable to determine if floor(height/4) <= unk__1, treat as equal
Unable to determine if floor(width/4) <= unk__2, treat as equal
Unable to determine if floor(floor(height/4)/2) <= unk__1, treat as equal
Unable to determine if floor(floor(width/4)/2) <= unk__2, treat as equal
[W] ONNX shape inference exited with an error:
[I] Total Nodes | Original: 4472, After Folding: 4472 | 0 Nodes Folded
Unable to determine if floor(height/4) <= unk__1, treat as equal
Unable to determine if floor(width/4) <= unk__2, treat as equal
Unable to determine if floor(height/4) <= unk__1, treat as equal
Unable to determine if floor(width/4) <= unk__2, treat as equal
Unable to determine if floor(height/4) <= unk__1, treat as equal
Unable to determine if floor(width/4) <= unk__2, treat as equal
Unable to determine if floor(floor(height/4)/2) <= unk__1, treat as equal
Unable to determine if floor(floor(width/4)/2) <= unk__2, treat as equal
[W] ONNX shape inference exited with an error:
[I] Saving ONNX model to: upernet_swin_base_dynamic_1080x608_folded.onnx
[I] New Model:
Name: torch-jit-export | ONNX Opset: 11
---- 1 Graph Input(s) ----
{input [dtype=float32, shape=('batch', 3, 'height', 'width')]}
---- 1 Graph Output(s) ----
{output [dtype=int64, shape=('batch', 1, 'height', 'width')]}
---- 446 Initializer(s) ----
---- 4472 Node(s) ----
Then I execute trtexec command:
/usr/src/tensorrt/bin/trtexec --onnx=upernet_swin_base_dynamic_1080x608_folded.onnx --saveEngine=upernet_swin_base_dynamic_1080x608.trt --verbose --useCudaGraph --workspace=1000 --explicitBatch
…
[11/14/2022-09:34:47] [V] [TRT] Registering tensor: 552 for ONNX tensor: 552
[11/14/2022-09:34:47] [V] [TRT] Cast_74 [Cast] outputs: [552 → (8)[INT32]],
[11/14/2022-09:34:47] [V] [TRT] Parsing node: Pad_76 [Pad]
[11/14/2022-09:34:47] [V] [TRT] Searching for input: 489
[11/14/2022-09:34:47] [V] [TRT] Searching for input: 552
[11/14/2022-09:34:47] [V] [TRT] Searching for input: 553
[11/14/2022-09:34:47] [V] [TRT] Pad_76 [Pad] inputs: [489 → (-1, -1, -1, 128)[FLOAT]], [552 → (8)[INT32]], [553 → ()[FLOAT]],
[11/14/2022-09:34:47] [V] [TRT] Registering layer: Pad_76 for ONNX node: Pad_76
[11/14/2022-09:34:47] [E] Error[4]: [shuffleNode.cpp::symbolicExecute::387] Error Code 4: Internal Error (Reshape_69: IShuffleLayer applied to shape tensor must have 0 or 1 reshape dimensions: dimensions were [-1,2])
[11/14/2022-09:34:47] [E] [TRT] ModelImporter.cpp:773: While parsing node number 63 [Pad → “554”]:
[11/14/2022-09:34:47] [E] [TRT] ModelImporter.cpp:774: — Begin node —
[11/14/2022-09:34:47] [E] [TRT] ModelImporter.cpp:775: input: “489”
input: “552”
input: “553”
output: “554”
name: “Pad_76”
op_type: “Pad”
attribute {
name: “mode”
s: “constant”
type: STRING
}
[11/14/2022-09:34:47] [E] [TRT] ModelImporter.cpp:776: — End node —
[11/14/2022-09:34:47] [E] [TRT] ModelImporter.cpp:779: ERROR: ModelImporter.cpp:179 In function parseGraph:
[6] Invalid Node - Pad_76
[shuffleNode.cpp::symbolicExecute::387] Error Code 4: Internal Error (Reshape_69: IShuffleLayer applied to shape tensor must have 0 or 1 reshape dimensions: dimensions were [-1,2])
[11/14/2022-09:34:47] [E] Failed to parse onnx file
[11/14/2022-09:34:47] [I] Finish parsing network model
[11/14/2022-09:34:47] [E] Parsing model failed
[11/14/2022-09:34:47] [E] Failed to create engine from model.
[11/14/2022-09:34:47] [E] Engine set up failed
&&&& FAILED TensorRT.trtexec [TensorRT v8201] # /usr/src/tensorrt/bin/trtexec --onnx=upernet_swin_base_dynamic_1080x608_folded.onnx --saveEngine=upernet_swin_base_dynamic_1080x608.trt --verbose --useCudaGraph --workspace=1000