Tensorflow to TensorRT converted file is too large

Description

I trained a model using tensorflow / keras and tried to convert to TensorRT.
While the tensorflow saved model size was around 19MB, the converted TensorRT file size was around 900MB.
The output file size is too huge; Do you have any suggestions for improvement?
Thanks

My environment,
python 3.7.6
tf 2.3.1
keras 2.4.0
cuda 11.0

This is how I created the model.

cnn_base = MobileNet(input_shape=shape[1:],weights="imagenet",include_top=False)
cnn_reshape = Reshape((28,1024))(cnn_base.output)
cnn = tf.keras.models.Model(cnn_base.input, cnn_reshape)
cnn.trainable = False
timestep_inputs = [Input(shape[1:],name='sinput_'+str(idx)) for idx in range(shape[0])]
conv_outputs = []
for x in timestep_inputs:
   y = cnn(x)
   conv_outputs.append(y)
x = concatenate(conv_outputs, axis = 1)
print('x',x.shape)
y = LSTM(128, return_sequences=False, return_state=False, stateful=False, dropout=0.5,activation='tanh')(x)        
hidden_layer = Dense(128, activation="relu")(y)
outputs = Dense(nbout, activation="softmax")(hidden_layer)
model = tf.keras.models.Model(timestep_inputs, outputs)

This is model summary

Model: “functional_3”


Layer (type) Output Shape Param # Connected to

sinput_0 (InputLayer) [(None, 132, 251, 3) 0
sinput_1 (InputLayer) [(None, 132, 251, 3) 0
.
.
.

sinput_69 (InputLayer) [(None, 132, 251, 3) 0
functional_1 (Functional) (None, 28, 1024) 3228864 sinput_0[0][0]
sinput_1[0][0]
.
.
.
sinput_69[0][0]

concatenate (Concatenate) (None, 1960, 1024) 0 functional_1[0][0]
functional_1[1][0]
.
.
.
functional_1[69][0]

lstm (LSTM) (None, 128) 590336 concatenate[0][0]
dense (Dense) (None, 128) 16512 lstm[0][0]
dense_1 (Dense) (None, 2) 258 dense[0][0]

Total params: 3,835,970
Trainable params: 607,106
Non-trainable params: 3,228,864
None

This is how I converted the model.

import tensorflow as tf
from tensorflow.python.compiler.tensorrt import trt_convert as trt
converter = trt.TrtGraphConverterV2(input_saved_model_dir='drown_model2_m2')
converter.convert()
#converter.build(input_fn=my_input_fn)
converter.save('t')

With our without build the size is around 900MB.

Hi, Request you to share the model and script so that we can try reproducing the issue at our end.

Also we recommend you to check the below samples links, as they might answer your concern

Thanks!

convert.py (5.4 KB) model.tar.gz (19.1 MB)
Hi,
Thank you for your reply.
I have uploaded the model and the script used for conversion.
Please let me know if you need any more info.

Hi @askvelu,
Could you please provide us details of the platform you are using

  • Linux distribution and version
  • GPU type
  • Nvidia driver version
  • CUDA version
  • CUDNN version
  • Python version [if using python]
  • Tensorflow and PyTorch version
  • TensorRT version

Thank you.

Please find the information you have requested.
I could not get information for some of the items.
I tried in two environments,

  1. AWS EC2 instance
  2. Jetson Nano - though it long time, the result was almost same - the output file was around 880MB

This is the environment in AWS EC2 instance

Linux distribution and version
NAME=“Ubuntu”
VERSION=“18.04.5 LTS (Bionic Beaver)”
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME=“Ubuntu 18.04.5 LTS”
VERSION_ID=“18.04”
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

GPU type
±----------------------------------------------------------------------------+
| NVIDIA-SMI 450.80.02 Driver Version: 450.80.02 CUDA Version: 11.0 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 Tesla T4 On | 00000000:00:1E.0 Off | 0 |
| N/A 38C P8 9W / 70W | 0MiB / 15109MiB | 0% Default |
| | | N/A |
±------------------------------±---------------------±---------------------+

Nvidia driver version
CUDA version
nvcc: NVIDIA ® Cuda compiler driver
Copyright © 2005-2018 NVIDIA Corporation
Built on Sat_Aug_25_21:08:01_CDT_2018
Cuda compilation tools, release 10.0, V10.0.130
CUDNN version
I could not find this info
Python version [if using python]
Python 3.7.6
Tensorflow and PyTorch version
Tensorflow - 2.3.1
TensorRT version
I could not find this info

This is the environment in Jetson nano

Ubuntu version
NAME=“Ubuntu”
VERSION=“18.04.4 LTS (Bionic Beaver)”
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME=“Ubuntu 18.04.4 LTS”
VERSION_ID=“18.04”
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic
GPU Type
NVIDIA Jetson Nano(Developer Kit Version) - Jetpack 4.4[L4T 32.4.3]
NVIDIA Driver Version
NVIDIA CUDA version
nvcc: NVIDIA ® Cuda compiler driver
Copyright © 2005-2019 NVIDIA Corporation
Built on Wed_Oct_23_21:14:42_PDT_2019
Cuda compilation tools, release 10.2, V10.2.89
Cudnn
I could not find this info
Python version
3.6.9
Tensorflow
2.3.1
Tensorrt
7.1.3

Please find the information you have requested.(Updated)
I could not get information for some of the items.
I tried in two environments,

  1. AWS EC2 instance
  2. Jetson Nano - though it long time, the result was almost same - the output file was around 880MB

This is the environment in AWS EC2 instance

Linux distribution and version
NAME=“Ubuntu”
VERSION=“18.04.5 LTS (Bionic Beaver)”
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME=“Ubuntu 18.04.5 LTS”
VERSION_ID=“18.04”
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

GPU type
±----------------------------------------------------------------------------+
| NVIDIA-SMI 450.80.02 Driver Version: 450.80.02 CUDA Version: 11.0 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 Tesla T4 On | 00000000:00:1E.0 Off | 0 |
| N/A 38C P8 9W / 70W | 0MiB / 15109MiB | 0% Default |
| | | N/A |
±------------------------------±---------------------±---------------------+

Nvidia driver version
CUDA version
nvcc: NVIDIA ® Cuda compiler driver
Copyright © 2005-2018 NVIDIA Corporation
Built on Sat_Aug_25_21:08:01_CDT_2018
Cuda compilation tools, release 10.0, V10.0.130
CUDNN version - 7.6.5 (cudnnmajor 7, cudnnminor 6, patch level 5)
Python version [if using python]
Python 3.7.6
Tensorflow and PyTorch version
Tensorflow - 2.3.1
TensorRT version - 7.0.0

This is the environment in Jetson nano

Ubuntu version
NAME=“Ubuntu”
VERSION=“18.04.4 LTS (Bionic Beaver)”
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME=“Ubuntu 18.04.4 LTS”
VERSION_ID=“18.04”
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic
GPU Type
NVIDIA Jetson Nano(Developer Kit Version) - Jetpack 4.4[L4T 32.4.3]
NVIDIA Driver Version
NVIDIA CUDA version
nvcc: NVIDIA ® Cuda compiler driver
Copyright © 2005-2019 NVIDIA Corporation
Built on Wed_Oct_23_21:14:42_PDT_2019
Cuda compilation tools, release 10.2, V10.2.89
Cudnn
I could not find this info
Python version - 3.6.9
Tensorflow 2.3.1
Tensorrt 7.1.3

Hi,
Do you have any suggestion/ solution for this issue?
Thanks