(Could not find any implementation for node {ForeignNode[Transpose_2713 + (Unnamed Layer* 4032) [Shuffle]...MatMul_2714]}.)

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

Hi, I am trying to convert an Onnx model with dynamic inputs to TensorRT format but encounter an error about “Could not find any implementation for node…”.
Can someone please assist with understanding this error message? Here is the model m2m100_418M-decoder.onnx - Google Drive.

This is my error message.

[07/05/2022-11:59:43] [W] [TRT]  (# 1 (RESHAPE (# 0 (SHAPE pkv_44)) 16 2 64 | (* 16 (# 0 (SHAPE input_ids))) -1 64 zeroIsPlaceholder))
[07/05/2022-11:59:43] [W] [TRT] Skipping tactic 0x0000000000000000 due to Myelin error: Incompatible shapes in fully-connected op between: attn_weights_179':f32,trA=false,[128,1,2] and onnx__MatMul_2931':f32,trB=false,[128,8,64].
[07/05/2022-11:59:43] [V] [TRT] Fastest Tactic: 0xd15ea5edd15ea5ed Time: inf
[07/05/2022-11:59:43] [V] [TRT] Deleting timing cache: 186 entries, served 7244 hits since creation.
[07/05/2022-11:59:43] [E] Error[10]: [optimizer.cpp::computeCosts::3628] Error Code 10: Internal Error (Could not find any implementation for node {ForeignNode[Transpose_2713 + (Unnamed Layer* 4032) [Shuffle]...MatMul_2714]}.)
[07/05/2022-11:59:43] [E] Error[2]: [builder.cpp::buildSerializedNetwork::636] Error Code 2: Internal Error (Assertion engine != nullptr failed. )
[07/05/2022-11:59:43] [E] Engine could not be created from network

This is my trtexec command.

trtexec --onnx=m2m100_418M-decoder.onnx --verbose --minShapes=input_ids:5x1,encoder_attention_mask:5x1,pkv_0:5x16x1x64,pkv_1:5x16x1x64,pkv_2:5x16x1x64,pkv_3:5x16x1x64,pkv_4:5x16x1x64,pkv_5:5x16x1x64,pkv_6:5x16x1x64,pkv_7:5x16x1x64,pkv_8:5x16x1x64,pkv_9:5x16x1x64,pkv_10:5x16x1x64,pkv_11:5x16x1x64,pkv_12:5x16x1x64,pkv_13:5x16x1x64,pkv_14:5x16x1x64,pkv_15:5x16x1x64,pkv_16:5x16x1x64,pkv_17:5x16x1x64,pkv_18:5x16x1x64,pkv_19:5x16x1x64,pkv_20:5x16x1x64,pkv_21:5x16x1x64,pkv_22:5x16x1x64,pkv_23:5x16x1x64,pkv_24:5x16x1x64,pkv_25:5x16x1x64,pkv_26:5x16x1x64,pkv_27:5x16x1x64,pkv_28:5x16x1x64,pkv_29:5x16x1x64,pkv_30:5x16x1x64,pkv_31:5x16x1x64,pkv_32:5x16x1x64,pkv_33:5x16x1x64,pkv_34:5x16x1x64,pkv_35:5x16x1x64,pkv_36:5x16x1x64,pkv_37:5x16x1x64,pkv_38:5x16x1x64,pkv_39:5x16x1x64,pkv_40:5x16x1x64,pkv_41:5x16x1x64,pkv_42:5x16x1x64,pkv_43:5x16x1x64,pkv_44:5x16x1x64,pkv_45:5x16x1x64,pkv_46:5x16x1x64,pkv_47:5x16x1x64 --optShapes=input_ids:5x1,encoder_attention_mask:5x100,pkv_0:5x16x1x64,pkv_1:5x16x1x64,pkv_2:5x16x1x64,pkv_3:5x16x1x64,pkv_4:5x16x1x64,pkv_5:5x16x1x64,pkv_6:5x16x1x64,pkv_7:5x16x1x64,pkv_8:5x16x1x64,pkv_9:5x16x1x64,pkv_10:5x16x1x64,pkv_11:5x16x1x64,pkv_12:5x16x1x64,pkv_13:5x16x1x64,pkv_14:5x16x1x64,pkv_15:5x16x1x64,pkv_16:5x16x1x64,pkv_17:5x16x1x64,pkv_18:5x16x1x64,pkv_19:5x16x1x64,pkv_20:5x16x1x64,pkv_21:5x16x1x64,pkv_22:5x16x1x64,pkv_23:5x16x1x64,pkv_24:5x16x1x64,pkv_25:5x16x1x64,pkv_26:5x16x1x64,pkv_27:5x16x1x64,pkv_28:5x16x1x64,pkv_29:5x16x1x64,pkv_30:5x16x1x64,pkv_31:5x16x1x64,pkv_32:5x16x1x64,pkv_33:5x16x1x64,pkv_34:5x16x1x64,pkv_35:5x16x1x64,pkv_36:5x16x1x64,pkv_37:5x16x1x64,pkv_38:5x16x1x64,pkv_39:5x16x1x64,pkv_40:5x16x1x64,pkv_41:5x16x1x64,pkv_42:5x16x1x64,pkv_43:5x16x1x64,pkv_44:5x16x1x64,pkv_45:5x16x1x64,pkv_46:5x16x1x64,pkv_47:5x16x1x64 --maxShapes=input_ids:80x1,encoder_attention_mask:80x200,pkv_0:80x16x1x64,pkv_1:80x16x1x64,pkv_2:80x16x1x64,pkv_3:80x16x1x64,pkv_4:80x16x1x64,pkv_5:80x16x1x64,pkv_6:80x16x1x64,pkv_7:80x16x1x64,pkv_8:80x16x1x64,pkv_9:80x16x1x64,pkv_10:80x16x1x64,pkv_11:80x16x1x64,pkv_12:80x16x1x64,pkv_13:80x16x1x64,pkv_14:80x16x1x64,pkv_15:80x16x1x64,pkv_16:80x16x1x64,pkv_17:80x16x1x64,pkv_18:80x16x1x64,pkv_19:80x16x1x64,pkv_20:80x16x1x64,pkv_21:80x16x1x64,pkv_22:80x16x1x64,pkv_23:80x16x1x64,pkv_24:80x16x1x64,pkv_25:80x16x1x64,pkv_26:80x16x1x64,pkv_27:80x16x1x64,pkv_28:80x16x1x64,pkv_29:80x16x1x64,pkv_30:80x16x1x64,pkv_31:80x16x1x64,pkv_32:80x16x1x64,pkv_33:80x16x1x64,pkv_34:80x16x1x64,pkv_35:80x16x1x64,pkv_36:80x16x1x64,pkv_37:80x16x1x64,pkv_38:80x16x1x64,pkv_39:80x16x1x64,pkv_40:80x16x1x64,pkv_41:80x16x1x64,pkv_42:80x16x1x64,pkv_43:80x16x1x64,pkv_44:80x16x1x64,pkv_45:80x16x1x64,pkv_46:80x16x1x64,pkv_47:80x16x1x64 --shapes=input_ids:5x1,encoder_attention_mask:5x100,pkv_0:5x16x1x64,pkv_1:5x16x1x64,pkv_2:5x16x1x64,pkv_3:5x16x1x64,pkv_4:5x16x1x64,pkv_5:5x16x1x64,pkv_6:5x16x1x64,pkv_7:5x16x1x64,pkv_8:5x16x1x64,pkv_9:5x16x1x64,pkv_10:5x16x1x64,pkv_11:5x16x1x64,pkv_12:5x16x1x64,pkv_13:5x16x1x64,pkv_14:5x16x1x64,pkv_15:5x16x1x64,pkv_16:5x16x1x64,pkv_17:5x16x1x64,pkv_18:5x16x1x64,pkv_19:5x16x1x64,pkv_20:5x16x1x64,pkv_21:5x16x1x64,pkv_22:5x16x1x64,pkv_23:5x16x1x64,pkv_24:5x16x1x64,pkv_25:5x16x1x64,pkv_26:5x16x1x64,pkv_27:5x16x1x64,pkv_28:5x16x1x64,pkv_29:5x16x1x64,pkv_30:5x16x1x64,pkv_31:5x16x1x64,pkv_32:5x16x1x64,pkv_33:5x16x1x64,pkv_34:5x16x1x64,pkv_35:5x16x1x64,pkv_36:5x16x1x64,pkv_37:5x16x1x64,pkv_38:5x16x1x64,pkv_39:5x16x1x64,pkv_40:5x16x1x64,pkv_41:5x16x1x64,pkv_42:5x16x1x64,pkv_43:5x16x1x64,pkv_44:5x16x1x64,pkv_45:5x16x1x64,pkv_46:5x16x1x64,pkv_47:5x16x1x64

These are my specs:
cuda: 11.3
pytorch: 1.11.0
nvidia-tensorrt: 8.4.1.5
onnx: 1.9.0

I really appreciate the help.|

2

Hi,
Request you to share the ONNX model and the script if not shared already so that we can assist you better.
Alongside you can try few things:

  1. validating your model with the below snippet

check_model.py

import sys
import onnx
filename = yourONNXmodel
model = onnx.load(filename)
onnx.checker.check_model(model).
2) Try running your model with trtexec command.

In case you are still facing issue, request you to share the trtexec “”–verbose"" log for further debugging
Thanks!

4

My onnx.checker.check_model(model) doesn’t throw any error.

5

Hi,

We could reproduce the same error.
Please allow us some time to work on this.

Thank you.

6

I met the SAME ERROR, with torch1.11, cuda11.3 and TensorRT-8.0.1.6
when execute the trtexec from CLIP/B-16 model defined from hugging-face;

[09/14/2022-12:02:56] [E] Error[10]: [optimizer.cpp::computeCosts::1855] Error Code 10: Internal Error (Could not find any implementation for node {ForeignNode[onnx::MatMul_1794 + (Unnamed Layer* 105) [Shuffle]…Add_1105]}.)

Any replay will be very helpful, Thanks.

7

Hi,

This issue has been fixed. Please try on the latest TensorRT version 8.4.3.
If it still fails, we recommend you to please wait for future releases to get a fix for this issue.

Thank you.

8

I had the same problem when using version 8.4.3.1

1 Like
9

Hi I am using torch2trt and not onnx, and I have the same issue. Can you fix that as well? I noticed in tensorrt 8.5 it only fix the issue regarding onnx.