# Loading TensorRT model is very slow on Jetson Nano

**URL:** <https://forums.developer.nvidia.com/t/loading-tensorrt-model-is-very-slow-on-jetson-nano/175829>\
**Category:** Jetson Nano\
**Tags:** tensorflow, tensorrt, python, jetson-inference\
**Created:** [April 20, 2021, 10:03pm UTC](https://forums.developer.nvidia.com/t/loading-tensorrt-model-is-very-slow-on-jetson-nano/175829 "2021-04-20T22:03:21Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![sorozco0612](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@sorozco0612](https://forums.developer.nvidia.com/u/sorozco0612)\
**Post date:** [April 20, 2021, 10:03pm UTC](https://forums.developer.nvidia.com/t/loading-tensorrt-model-is-very-slow-on-jetson-nano/175829/1 "2021-04-20T22:03:21Z")

</div>

I converted my tensorflow model to a TF-RT model. On my regular machine, I can load this model pretty easily (5-10 seconds), but loading the model on my Jetson takes nearly 40 minutes. Any reason why that is? Is there any way to reduce this load time?

## Tensorflow to TF-RT Model

```
import numpy as np
import tensorflow as tf
from tensorflow import keras
from tensorflow.python.compiler.tensorrt import trt_convert as trt

def my_input_fn():
   inp1 = np.random.normal(size=(1, 432, 614, 3)).astype(np.float32)
   yield (inp1,)

conversion_params = trt.DEFAULT_TRT_CONVERSION_PARAMS
conversion_params = conversion_params._replace(max_workspace_size_bytes=(1 << 32))
conversion_params = conversion_params._replace(precision_mode="FP16")
conversion_params = conversion_params._replace(maximum_cached_engines=100)
converter = trt.TrtGraphConverterV2(
    input_saved_model_dir="/home/sorozco0612/dev/alan_bot/model_creation/alan_bot_model",
    conversion_params=conversion_params,)
converter.convert()
converter.build(input_fn=my_input_fn)
converter.save("/home/sorozco0612/alan_bot_model_trt")

```

## Loading TF-RT Model

```
# takes nearly 40 minutes on Jetson

model = tf.saved_model.load(
    "/home/sorozco/alan_bot_model_trt/",
    tags=[tf.compat.v1.saved_model.tag_constants.SERVING],
)

```

I’m assuming this has something to do with memory limitations, but I’m not sure.

---

<div class="post-metadata">

**Author:** ![AastaLLL](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/aastalll/32/14043_2.png) [@AastaLLL](https://forums.developer.nvidia.com/u/AastaLLL)\
**Post date:** [April 21, 2021, 2:58am UTC](https://forums.developer.nvidia.com/t/loading-tensorrt-model-is-very-slow-on-jetson-nano/175829/3 "2021-04-21T02:58:08Z")

</div>

Hi,

Do you think a pure TensorRT model is an option for you?  
For TF-TRT, it may take times due to the memory resource from the Nano.

Thanks.

---

<div class="post-metadata">

**Author:** ![sorozco0612](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@sorozco0612](https://forums.developer.nvidia.com/u/sorozco0612)\
**Post date:** [April 21, 2021, 5:24pm UTC](https://forums.developer.nvidia.com/t/loading-tensorrt-model-is-very-slow-on-jetson-nano/175829/4 "2021-04-21T17:24:36Z")

</div>

I am currently in the process of converting to pure TensorRT. I will keep you updated with any issues I may have. Thank you for the help!

---

<div class="post-metadata">

**Author:** ![sorozco0612](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@sorozco0612](https://forums.developer.nvidia.com/u/sorozco0612)\
**Post date:** [April 21, 2021, 10:38pm UTC](https://forums.developer.nvidia.com/t/loading-tensorrt-model-is-very-slow-on-jetson-nano/175829/5 "2021-04-21T22:38:40Z")

</div>

So I am currently trying to convert my SavedModel to pure TensorRT. I successfully converted my SavedModel to ONNX using the following command:

```
## convert saved model to ONNX
!python3 -m tf2onnx.convert --saved-model "/home/sorozco0612/dev/alan_bot/model_creation/alan_bot_model" --output "alan_bot_model_onnx.onnx"

```

I’m not entirely sure where to go from here. Some assistance would be appreciated.

---

<div class="post-metadata">

**Author:** ![AastaLLL](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/aastalll/32/14043_2.png) [@AastaLLL](https://forums.developer.nvidia.com/u/AastaLLL)\
**Post date:** [April 23, 2021, 7:08am UTC](https://forums.developer.nvidia.com/t/loading-tensorrt-model-is-very-slow-on-jetson-nano/175829/6 "2021-04-23T07:08:02Z")

</div>

Hi,

First, you can try to convert it into TensorRT engine with _trtexec_.

```auto
$ /usr/src/tensorrt/bin/trtexec --onnx=[your/model]

```

If all goes good, you can deploy it with the same below:

> [@Custom ResNet Jetson Xavier](https://forums.developer.nvidia.com/t/custom-resnet-jetson-xavier/160448/3):
>
> Hi, The error comes from dynamic shape usage. If you don’t need to reshape the model at the runtime, please set EXPLICIT\_BATCH to use static shape instead. Here is an example of onnx model for your reference: import cv2 import time import numpy as np import tensorrt as trt import pycuda.autoinit import pycuda.driver as cuda EXPLICIT\_BATCH = 1 \<\< (int)(trt.NetworkDefinitionCreationFlag.EXPLICIT\_BATCH) TRT\_LOGGER = trt.Logger(trt.Logger.INFO) runtime = trt.Runtime(TRT\_LOGGER) host\_inputs = …

Thanks.

---

<div class="post-metadata">

**Author:** ![nadeemm](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/nadeemm/32/367020_2.png) [@nadeemm](https://forums.developer.nvidia.com/u/nadeemm)\
**Post date:** [October 15, 2021, 3:59am UTC](https://forums.developer.nvidia.com/t/loading-tensorrt-model-is-very-slow-on-jetson-nano/175829/8 "2021-10-15T03:59:21Z")

</div>

Closed on October 15, 2021, 3:59am UTC