Serialize TRT optimized model after being loaded and built in tensorflow

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

Description

I was able to successfully convert/optimize a tensorflow model and can load it in as below:

model = tf.keras.models.load_model('trt_model')
func = model.signatures['serving_default']
frozen_func = convert_variables_to_constants_v2(func)

Inference is very quick; only problem is that loading this model is quite slow. I would like to be able to serialize this frozen_func for quicker loading time but cannot figure out how. Any help is much appreciated!

2

Hi,

Could you please share more details as following for better debugging.

  • Issue repro script, model and steps to try from our end
  • More details of the platform you are using
    TensorRT Version :
    NVIDIA GPU :
    NVIDIA Driver Version :
    CUDA Version :
    CUDNN Version :
    Operating System :
    Python Version :
    Tensorflow Version:

Thank you.

3

Hello,

There is technically not a bug, the above script works with no errors thrown. I am just looking for a way to save a frozen function after it has been loaded in and built. All of the documentation on your site either vaguely mentions serializing or offers TF1 dependent ways of serializing (ie: using freeze_graph function). Hopefully I am just misunderstanding what I have been reading.

Cheers!

4

Hi,

Please try tf-trt. Which may help you.

Thank you.