Tensorflow/TensorRT convert fails with ERROR: tensorflow.GraphDef exceeded maximum protobuf size of 2GB

Description

When I attempt to convert a Tensoflow saved model, TrtGraphConverter.convert() log shows the following error:
[libprotobuf ERROR external/com_google_protobuf/src/google/protobuf/message_lite.cc:406] tensorflow.GraphDef exceeded maximum protobuf size of 2GB: 2697675801.
Subsequently TrtGraphConverter.save() produces an empty ‘saved_model.pb’ file.

Environment

TensorRT Version N/A:
GPU Type Tesla K80:
Nvidia Driver Version 450.80.02:
CUDA Version 11.0:
CUDNN Version V10.0.130:
Operating System + Version Amazon Linux AMI release 2018.03:
Python Version (if applicable) 3.7.9:
TensorFlow Version (if applicable) 1.15.5:
PyTorch Version (if applicable):
Baremetal or Container (if container which image + tag):

Relevant Files

tensorrt_compile.log (6.6 KB)
tensorrt_compile.py (816 Bytes)

Steps To Reproduce

Please see the attached files - the script that I ran (tensorrt_compile.py) and its output (tensorrt_compile.log)

Hi, Request you to share the model and script so that we can try reproducing the issue at our end.

Also we recommend you to check the below samples links, as they might answer your concern
https://docs.nvidia.com/deeplearning/frameworks/tf-trt-user-guide/index.html#samples
https://docs.nvidia.com/deeplearning/frameworks/tf-trt-user-guide/index.html#integrate-ovr

Thanks!

Hi @andrei.voinov,

In TF1 the default is a static engine, which is created during conversion and the serialized engine is saved in the graph as a string. This hits the protobuf’s size limitation of 2GB. The solution is to use dynamic engine, which is enabled by the is_dynamic_op=True converter arg.

Also please note that the TF1 converter sometimes has issues in dynamic mode. It is highly recommended to use the TF2, and TRTGraphConverterV2.

Thank you.

@NVES: the script is already attached. I don’t think I can share the model itself (it is proprietary). There was another helpful suggestion to try with TF v2 - I will try this first, thank you @spolisetty !