Description
I am migrating from tensortRT 7 to tensorRT 8.4.1, and my input shapes are dynamic. However, I got the following error with tensorRT 8.4.1 while tensorRT 7 run smoothly:
Error Code 10: Internal Error (Could not find any implementation for node {ForeignNode[EdgeDecoder.maxpool:2:SHUFFLE + EdgeDecoder:0:SHUFFLE…ScoreDecoder.sigmoid:0:SIGMOID]}.)
The dotted layers includes multiple torch.linear layers and a torch.matmul before the sigmoid as shown below:
(EdgeDecoder): EdgeDecoder(
(replace_conv_linear1): Linear(in_features=128, out_features=512, bias=True)
(replace_conv_linear2): Linear(in_features=512, out_features=512, bias=True)
(replace_conv_linear3): Linear(in_features=512, out_features=1024, bias=True)
(maxpool): MaxPool1d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
(mha): MultiHeadAttention(
(query_linear): Linear(in_features=512, out_features=512, bias=False)
(key_linear): Linear(in_features=512, out_features=512, bias=False)
(value_linear): Linear(in_features=512, out_features=512, bias=False)
(output_linear): Linear(in_features=512, out_features=512, bias=False)
(relu): ReLU()
(dropout): Dropout(p=0.0, inplace=False)
)
(linear): Linear(in_features=512, out_features=64, bias=True)
)
(ScoreDecoder): ScoreDecoder(
(linear1): Linear(in_features=64, out_features=1024, bias=True)
(linear2): Linear(in_features=1024, out_features=512, bias=True)
(linear3): Linear(in_features=512, out_features=1, bias=True)
####### torch.matmul here
(sigmoid): Sigmoid()
)
Once I removed the matmul operation, the error was gone but the result engine output deviated a lot from the original torch model output under same input. All these problems did not appear in tensorRT 7.
If using static input shapes, the error would be gone.
Can you please help look into the problem?
Environment
TensorRT Version: 8.4.1
GPU Type: tesla T4
Nvidia Driver Version: 10.2
CUDA Version: 10.2
CUDNN Version: 10.2
Operating System + Version: ubuntu 18.04
Python Version (if applicable): 3.6.9
TensorFlow Version (if applicable): N/A
PyTorch Version (if applicable): 1.10.1
Baremetal or Container (if container which image + tag):
Relevant Files
Please attach or include links to any models, data, files, or scripts necessary to reproduce your issue. (Github repo, Google Drive, Dropbox, etc.)
Steps To Reproduce
Please include:
- Exact steps/commands to build your repro
- Exact steps/commands to run your repro
- Full traceback of errors encountered