Loading TensorRT model is very slow on Jetson Nano

I converted my tensorflow model to a TF-RT model. On my regular machine, I can load this model pretty easily (5-10 seconds), but loading the model on my Jetson takes nearly 40 minutes. Any reason why that is? Is there any way to reduce this load time?

Tensorflow to TF-RT Model

import numpy as np
import tensorflow as tf
from tensorflow import keras
from tensorflow.python.compiler.tensorrt import trt_convert as trt


def my_input_fn():
   inp1 = np.random.normal(size=(1, 432, 614, 3)).astype(np.float32)
   yield (inp1,)


conversion_params = trt.DEFAULT_TRT_CONVERSION_PARAMS
conversion_params = conversion_params._replace(max_workspace_size_bytes=(1 << 32))
conversion_params = conversion_params._replace(precision_mode="FP16")
conversion_params = conversion_params._replace(maximum_cached_engines=100)
converter = trt.TrtGraphConverterV2(
    input_saved_model_dir="/home/sorozco0612/dev/alan_bot/model_creation/alan_bot_model",
    conversion_params=conversion_params,)
converter.convert()
converter.build(input_fn=my_input_fn)
converter.save("/home/sorozco0612/alan_bot_model_trt")

Loading TF-RT Model

# takes nearly 40 minutes on Jetson

model = tf.saved_model.load(
    "/home/sorozco/alan_bot_model_trt/",
    tags=[tf.compat.v1.saved_model.tag_constants.SERVING],
)

I’m assuming this has something to do with memory limitations, but I’m not sure.

Hi,

Do you think a pure TensorRT model is an option for you?
For TF-TRT, it may take times due to the memory resource from the Nano.

Thanks.

I am currently in the process of converting to pure TensorRT. I will keep you updated with any issues I may have. Thank you for the help!

So I am currently trying to convert my SavedModel to pure TensorRT. I successfully converted my SavedModel to ONNX using the following command:

## convert saved model to ONNX
!python3 -m tf2onnx.convert --saved-model "/home/sorozco0612/dev/alan_bot/model_creation/alan_bot_model" --output "alan_bot_model_onnx.onnx"

I’m not entirely sure where to go from here. Some assistance would be appreciated.

Hi,

First, you can try to convert it into TensorRT engine with trtexec.

$ /usr/src/tensorrt/bin/trtexec --onnx=[your/model]

If all goes good, you can deploy it with the same below:

Thanks.