I converted my tensorflow model to a TF-RT model. On my regular machine, I can load this model pretty easily (5-10 seconds), but loading the model on my Jetson takes nearly 40 minutes. Any reason why that is? Is there any way to reduce this load time?
Tensorflow to TF-RT Model
import numpy as np
import tensorflow as tf
from tensorflow import keras
from tensorflow.python.compiler.tensorrt import trt_convert as trt
def my_input_fn():
inp1 = np.random.normal(size=(1, 432, 614, 3)).astype(np.float32)
yield (inp1,)
conversion_params = trt.DEFAULT_TRT_CONVERSION_PARAMS
conversion_params = conversion_params._replace(max_workspace_size_bytes=(1 << 32))
conversion_params = conversion_params._replace(precision_mode="FP16")
conversion_params = conversion_params._replace(maximum_cached_engines=100)
converter = trt.TrtGraphConverterV2(
input_saved_model_dir="/home/sorozco0612/dev/alan_bot/model_creation/alan_bot_model",
conversion_params=conversion_params,)
converter.convert()
converter.build(input_fn=my_input_fn)
converter.save("/home/sorozco0612/alan_bot_model_trt")
Loading TF-RT Model
# takes nearly 40 minutes on Jetson
model = tf.saved_model.load(
"/home/sorozco/alan_bot_model_trt/",
tags=[tf.compat.v1.saved_model.tag_constants.SERVING],
)
I’m assuming this has something to do with memory limitations, but I’m not sure.