Problems with converting Einsum operations to the tensorrt

Problem

Hello,

I am trying to convert LOFTR pytorch model to tensorrt. The model is just combination of CNN with Transformers, but the implementation uses a lot of einsum operation so this is what I think gives me several errors. Because I tested the entire pipeline on other models and it worked well, so I don’t think I have errors in the code.

This is the process.

I converted model to the onnx and tested it using onnxruntime, it works well. Initially I used opset = 15, but now decreased to 12. Results are same for both versions.

with torch.no_grad():
        torch.onnx.export(model,
                          input_dict, 
                          out_filename, 
                          do_constant_folding=True,
                          verbose=True, 
                          opset_version=12)

This is the code for onnx conversion.

Then I use the generated file for trt conversion. During parsing the onnx file I get the following error,

[TRT] [E] [layers.cpp::EinsumLayer::5821] Error Code 3: API Usage Error (Parameter check failed at: optimizer/api/layers.cpp::EinsumLayer::5821, condition: nbInputs > 0 && nbInputs <= MAX_EINSUM_NB_INPUTS
)

but it finishes the parsing and continues to build the model.

plan = builder.build_serialized_network(network, config)

with trt.Runtime(TRT_LOGGER) as runtime:
	engine = runtime.deserialize_cuda_engine(plan)

I use this script, for generation but it produced error of

network.cpp::validate::2726] Error Code 4: Internal Error (Network must have at least one output

After this I was curious to look if the model is parsed correctly at all and when I print

print(network.num_outputs)

after model parsing, it returns 0. I also added following line to my model

network.mark_output(network.get_layer(network.num_layers - 1).get_output(0))

This produces the following error

[TRT] [E] 2: [checkSanity.cpp::checkSanity::94] Error Code 2: Internal Error (Assertion tensors.size() == g.tensors.size() failed.

I got stuck here, I am not sure which one I have to fix, whether the error is in parsing or in generating step.

Environment

TensorRT Version: 8.4.0.6
GPU Type: RTX 2080
Nvidia Driver Version: 460.73.01
CUDA Version: 11.2
Operating System + Version: “Debian GNU/Linux 10”
Python Version (if applicable): 3.9
PyTorch Version (if applicable): 1.11.0

Hi,
Request you to share the ONNX model and the script if not shared already so that we can assist you better.
Alongside you can try few things:

  1. validating your model with the below snippet

check_model.py

import sys
import onnx
filename = yourONNXmodel
model = onnx.load(filename)
onnx.checker.check_model(model).
2) Try running your model with trtexec command.

In case you are still facing issue, request you to share the trtexec “”–verbose"" log for further debugging
Thanks!

Thank you for the quick reply.

  1. I tested the model using the 1st approach check_model.py and didnt print any errors.
  2. for the model its uploaded here
    loftr_.onnx - Google Drive

Hello,
Sorry for rushing but did you find anything useful ? Maybe you also have some question I could clarify.

If anyone also has a similar problem.

torch.einsum("nlhd,nhdv,nlh->nlhv", x1,x2,x3)

this kind of einsum operation is not supported yet and is not parsed by tensorrt exec. I changed this with custom operations and it worked.

Hi,

Yes. TRT’s einsum doesn’t support more than two inputs, the above error is trying to convey the same. We will improve this error message in future releases.

Thank you.

1 Like