[defaultAllocator.cpp::deallocate::35] Error Code 1: Cuda Runtime (invalid argument)

Description

I converted onnx model to Tensorrt, but when I’m trying to load engine:

def get_engine(engine_path):
    # If a serialized engine exists, use it instead of building an engine.
    print("Reading engine from file {}".format(engine_path))
    with open(engine_path, "rb") as f, trt.Runtime(TRT_LOGGER) as runtime:
        return runtime.deserialize_cuda_engine(f.read())

with get_engine("human-pose-estimation-3d.engine") as engine, engine.create_execution_context() as context:

I got error: [defaultAllocator.cpp::deallocate::35] Error Code 1: Cuda Runtime (invalid argument)

I cheked model using:

import sys
import onnx
filename = yourONNXmodel
model = onnx.load(filename)
onnx.checker.check_model(model).

And it’s okay

Also I converted other model the same way, and it works okay.

Environment

TensorRT Version: 8.2.4.2
GPU Type: A100
Nvidia Driver Version: 510.47.03
CUDA Version: 11.4.3
CUDNN Version: 8
Operating System + Version: Ubuntu 20.04
Python Version: 3.8
Baremetal or Container (if container which image + tag): nvidia/cuda:11.4.3-cudnn8-devel-ubuntu20.04

Relevant Files

Here you can find onnx and trt models:

https://drive.google.com/drive/folders/1IWgW66gBHw2hNOwfW6gZXbaVCt3W8ieq?usp=sharing

Hi,
Below link might help you with your query, Kindly check below link for all 3d support layers:

Thanks!

Can you explain please what do you mean 3d layers, it seems to me that i dont have any 3d layers in my model?

Hi,

Could you please share with us the issue repro minimal script and complete verbose logs.

Thank you.