Problems with converting Einsum operations to the tensorrt

AI & Data Science Deep Learning (Training & Inference) TensorRT
, , ,
1

Problem

Hello,

I am trying to convert LOFTR pytorch model to tensorrt. The model is just combination of CNN with Transformers, but the implementation uses a lot of einsum operation so this is what I think gives me several errors. Because I tested the entire pipeline on other models and it worked well, so I don’t think I have errors in the code.

This is the process.

I converted model to the onnx and tested it using onnxruntime, it works well. Initially I used opset = 15, but now decreased to 12. Results are same for both versions.

with torch.no_grad():
        torch.onnx.export(model,
                          input_dict, 
                          out_filename, 
                          do_constant_folding=True,
                          verbose=True, 
                          opset_version=12)

This is the code for onnx conversion.

Then I use the generated file for trt conversion. During parsing the onnx file I get the following error,

[TRT] [E] [layers.cpp::EinsumLayer::5821] Error Code 3: API Usage Error (Parameter check failed at: optimizer/api/layers.cpp::EinsumLayer::5821, condition: nbInputs > 0 && nbInputs <= MAX_EINSUM_NB_INPUTS
)

but it finishes the parsing and continues to build the model.

plan = builder.build_serialized_network(network, config)

with trt.Runtime(TRT_LOGGER) as runtime:
	engine = runtime.deserialize_cuda_engine(plan)

I use this script, for generation but it produced error of

network.cpp::validate::2726] Error Code 4: Internal Error (Network must have at least one output

After this I was curious to look if the model is parsed correctly at all and when I print

print(network.num_outputs)

after model parsing, it returns 0. I also added following line to my model

network.mark_output(network.get_layer(network.num_layers - 1).get_output(0))

This produces the following error

[TRT] [E] 2: [checkSanity.cpp::checkSanity::94] Error Code 2: Internal Error (Assertion tensors.size() == g.tensors.size() failed.

I got stuck here, I am not sure which one I have to fix, whether the error is in parsing or in generating step.

Environment

TensorRT Version: 8.4.0.6
GPU Type: RTX 2080
Nvidia Driver Version: 460.73.01
CUDA Version: 11.2
Operating System + Version: “Debian GNU/Linux 10”
Python Version (if applicable): 3.9
PyTorch Version (if applicable): 1.11.0

2

Hi,
Request you to share the ONNX model and the script if not shared already so that we can assist you better.
Alongside you can try few things:

  1. validating your model with the below snippet

check_model.py

import sys
import onnx
filename = yourONNXmodel
model = onnx.load(filename)
onnx.checker.check_model(model).
2) Try running your model with trtexec command.

In case you are still facing issue, request you to share the trtexec “”–verbose"" log for further debugging
Thanks!

4

Thank you for the quick reply.

  1. I tested the model using the 1st approach check_model.py and didnt print any errors.
  2. for the model its uploaded here
    https://drive.google.com/file/d/1Vai-DCioKvd0KYmVWlE6Wyo7w_8RR_Mr/view?usp=sharing
5

Hello,
Sorry for rushing but did you find anything useful ? Maybe you also have some question I could clarify.

6

If anyone also has a similar problem.

torch.einsum("nlhd,nhdv,nlh->nlhv", x1,x2,x3)

this kind of einsum operation is not supported yet and is not parsed by tensorrt exec. I changed this with custom operations and it worked.

7

Hi,

Yes. TRT’s einsum doesn’t support more than two inputs, the above error is trying to convey the same. We will improve this error message in future releases.

Thank you.

2 Likes
8

Hi @madmldev32, I’ve modified the einsum operations, but I am still getting a lot of other issues popping up with other parts of the model implementation when I attempt to convert to ONNX format. Were the einsum replacements really all you changed?

1 Like