TensorRT deserialize_cuda_engine() returns a None Object



I’m trying to run Resnet50 inference model using this jupyter notebook

TensorRT/1. Introduction.ipynb at master · NVIDIA/TensorRT · GitHub

I’m running this on a tensorrt container downloaded from ngc

I get the following error on this line in notebook

trt_model = ONNXClassifierWrapper(“resnet_engine_intro.trt”, [BATCH_SIZE, 1000], target_dtype = PRECISION)

On further investigation this line in onnx_helper.py is generating the stack trace:
engine = runtime.deserialize_cuda_engine(f.read())

The function returns None. How can I debug it more?

AttributeError Traceback (most recent call last)
----> 7 trt_model = ONNXClassifierWrapper(“resnet_engine_intro.trt”, [BATCH_SIZE, 1000], target_dtype = PRECISION)

/mnt/TensorRT/quickstart/IntroNotebooks/onnx_helper.py in init(self, file, num_classes, target_dtype)
29 self.target_dtype = target_dtype
30 self.num_classes = num_classes
—> 31 self.load(file)
33 self.stream = None

/mnt/TensorRT/quickstart/IntroNotebooks/onnx_helper.py in load(self, file)
41 engine = runtime.deserialize_cuda_engine(f.read())
42 print(engine)
—> 43 self.context = engine.create_execution_context()
45 def allocate_memory(self, batch):

AttributeError: ‘NoneType’ object has no attribute ‘create_execution_context’


TensorRT Version: nvidia-tensorrt tensorrt

GPU Type: NVIDIA A100-40GB
Nvidia Driver Version: 460.39
CUDA Version: 11.2
CUDNN Version:
Operating System + Version: Docker container, Linux 5.8.0-44
Python Version (if applicable): 3.8.5
TensorFlow Version (if applicable):
PyTorch Version (if applicable):
Baremetal or Container (if container which image + tag):

Relevant Files

Please attach or include links to any models, data, files, or scripts necessary to reproduce your issue. (Github repo, Google Drive, Dropbox, etc.)

Steps To Reproduce

Please include:

  • Exact steps/commands to build your repro
  • Exact steps/commands to run your repro
  • Full traceback of errors encountered

Hi @hkr1990,

Could you please delete resnet_engine_intro.trt and run again trtexec command to generate engine file.
trtexec --onnx=resnet50/model.onnx --saveEngine=resnet_engine_intro.trt --explicitBatch

We cannot use generated resnet_engine_intro.trt to run on different platform.

Thank you.

The trt file I’m using is generated on the same platform using the above mentioned command. I still get the same error. Is there a way I can validate that trt file is good/valid? Also “engine.create_execution_context()” just returns a NoneType object and doesn’t provide any verbose error log. Is there a way I can generate more debug information about the error?

Hi @hkr1990,

Could you please share us ONNX model, we would like to reproduce the issue for better assistance.

Thank you.

Resnet 50 Model

wget https://s3.amazonaws.com/download.onnx/models/opset_8/resnet50.tar.gz tar xzf resnet50.tar.gz

Jupyter Notebook
TensorRT/1. Introduction.ipynb at master · NVIDIA/TensorRT · GitHub

Container: docker pull nvcr.io/nvidia/tensorrt:21.02-py3
GPU : A100 40GB

Hi @hkr1990,

I tried running scripts you’ve shared. I couldn’t reproduce the issue. I am able to run successfully without any errors.
Env -
tensorrt container : nvcr.io/nvidia/tensorrt:21.02-py3
tensorflow : 2.4.1

Thank you.

I downloaded a fresh image of tensorflow container and it works now.