Cuda Runtime error when switching from TensorRT 7.1.3 to TensorRT 8.0.1

[realimposter]

Hi, I basically upgraded from TensorRT 7.1.3 to TensorRT 8.0.1 on AGX Xavier. I run everything in Python 3.6

I converted a face detector model from Pytorch to TensorRT in both with identical code. I then ran this exact same code in both:

#load in the models
stream = cuda.Stream()
TRT_LOGGER = trt.Logger()
explicit_batch = 1 << (int)(trt.NetworkDefinitionCreationFlag.EXPLICIT_BATCH)
#load in the face detector tensorrt model
with open("weights/yolov5s-face-448x800.trt", "rb") as f, trt.Runtime(TRT_LOGGER) as runtime:
	fd_engine = runtime.deserialize_cuda_engine(f.read())
for binding in fd_engine:
	if fd_engine.binding_is_input(binding):
		fd_device_input = cuda.mem_alloc(trt.volume(fd_engine.get_binding_shape(binding)) * fd_engine.max_batch_size * np.dtype(np.float32).itemsize)
	else:
		fd_host_output = cuda.pagelocked_empty(trt.volume(fd_engine.get_binding_shape(binding)) * fd_engine.max_batch_size, dtype=np.float32)
		fd_device_output = cuda.mem_alloc(fd_host_output.nbytes)
fd_context = fd_engine.create_execution_context()

Just testing with this code, I consistently find this error:

This happens when I only load the model in the entire script. It basically shows up at the end of every run. This did not show up with Jetpack 4.4 with TensorRT 7.1.3, only on Jetpack 4.6 with TensorRT 8.0.1.
Could you explain why and how to fix it?

[AastaLLL]

Hi,

There are some changes in destructors that might lead to this behavior.
You can find more details in our release notes:
https://docs.nvidia.com/deeplearning/tensorrt/release-notes/tensorrt-8.html#rel_8-0-1

Currently, please try to set the fd_engine to None before terminating and it should work.
For example:

if __name__ == "__main__":
    engine = PrepareEngine()
    Inference(engine)
    ...
    engine = None

Thanks.

[realimposter]

Thank you, it is fixed.

Setting fd_context = None fixed it. fd_engine = None didn’t do anything.