tensorRT8.4.1 dynamic shape, Could not find any implementation for node {ForeignNode


I am migrating from tensortRT 7 to tensorRT 8.4.1, and my input shapes are dynamic. However, I got the following error with tensorRT 8.4.1 while tensorRT 7 run smoothly:

Error Code 10: Internal Error (Could not find any implementation for node {ForeignNode[EdgeDecoder.maxpool:2:SHUFFLE + EdgeDecoder:0:SHUFFLE…ScoreDecoder.sigmoid:0:SIGMOID]}.)

The dotted layers includes multiple torch.linear layers and a torch.matmul before the sigmoid as shown below:
(EdgeDecoder): EdgeDecoder(
(replace_conv_linear1): Linear(in_features=128, out_features=512, bias=True)
(replace_conv_linear2): Linear(in_features=512, out_features=512, bias=True)
(replace_conv_linear3): Linear(in_features=512, out_features=1024, bias=True)
(maxpool): MaxPool1d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
(mha): MultiHeadAttention(
(query_linear): Linear(in_features=512, out_features=512, bias=False)
(key_linear): Linear(in_features=512, out_features=512, bias=False)
(value_linear): Linear(in_features=512, out_features=512, bias=False)
(output_linear): Linear(in_features=512, out_features=512, bias=False)
(relu): ReLU()
(dropout): Dropout(p=0.0, inplace=False)
(linear): Linear(in_features=512, out_features=64, bias=True)
(ScoreDecoder): ScoreDecoder(
(linear1): Linear(in_features=64, out_features=1024, bias=True)
(linear2): Linear(in_features=1024, out_features=512, bias=True)
(linear3): Linear(in_features=512, out_features=1, bias=True)
####### torch.matmul here
(sigmoid): Sigmoid()

Once I removed the matmul operation, the error was gone but the result engine output deviated a lot from the original torch model output under same input. All these problems did not appear in tensorRT 7.
If using static input shapes, the error would be gone.
Can you please help look into the problem?


TensorRT Version: 8.4.1
GPU Type: tesla T4
Nvidia Driver Version: 10.2
CUDA Version: 10.2
CUDNN Version: 10.2
Operating System + Version: ubuntu 18.04
Python Version (if applicable): 3.6.9
TensorFlow Version (if applicable): N/A
PyTorch Version (if applicable): 1.10.1
Baremetal or Container (if container which image + tag):

Relevant Files

Please attach or include links to any models, data, files, or scripts necessary to reproduce your issue. (Github repo, Google Drive, Dropbox, etc.)

Steps To Reproduce

Please include:

  • Exact steps/commands to build your repro
  • Exact steps/commands to run your repro
  • Full traceback of errors encountered


Could you please try the latest TensorRT version 8.5.2 (if possible, increase some workspace) and let us know if the problem persists?

Please share with us a issue repro ONNX model and complete trtexec --verbose logs for better debugging.

Thank you.