Multi view input prediction with tf-trt model in jetson nano

Hi, I have a keras multi view cnn that has the following summary


Layer (type) Output Shape Param # Connected to

input_2 (InputLayer) [(None, 32, 32, 3)] 0


input_3 (InputLayer) [(None, 32, 32, 3)] 0


input_4 (InputLayer) [(None, 32, 32, 3)] 0


input_5 (InputLayer) [(None, 32, 32, 3)] 0


model (Functional) (None, 32) 46400 input_2[0][0]
input_3[0][0]
input_4[0][0]
input_5[0][0]


concatenate (Concatenate) (None, 128) 0 model[0][0]
model[1][0]
model[2][0]
model[3][0]


dense_1 (Dense) (None, 32) 4128 concatenate[0][0]


dropout_4 (Dropout) (None, 32) 0 dense_1[0][0]


dense_2 (Dense) (None, 8) 264 dropout_4[0][0]


dropout_5 (Dropout) (None, 8) 0 dense_2[0][0]


dense_3 (Dense) (None, 2) 18 dropout_5[0][0]

Total params: 50,810
Trainable params: 50,810
Non-trainable params: 0


I’ve tried to convert it to tf-trt model and tflite model like below:

converting to tf-trt :

conversion_params = trt.DEFAULT_TRT_CONVERSION_PARAMS
conversion_params = conversion_params._replace(max_workspace_size_bytes=(1<<32))
conversion_params = conversion_params._replace(precision_mode=“FP16”)
conversion_params = conversion_params._replace(maximum_cached_engines=100)
converter = trt.TrtGraphConverterV2(
input_saved_model_dir=“saved_model_4pic”,
conversion_params=conversion_params)
converter.convert()
converter.save(“trt_FP16_4pic”)
saved model is frozen graph of keras model(pb file)

and for converting to tflite:

model = tf.keras.models.load_model(’./1pic/3class_2019-09-15_0232.h5’)
converter = tf.lite.TFLiteConverter.from_keras_model(model)
tflite_model = converter.convert()
open(“converted_model.tflite”, “wb”).write(tflite_model)

after converting I predict with both models(tf-trt and tflite)
and the result time shows that tflite lite is much faster than tf-trt (prediction time for tf-trt = 250ms and for tflite = 5.8 ms)

but as you can see in the internet tensorrt is faster than tflite in jetson nano

I guess my prediction time isnt correct because of my way to predict
lets see the way of prediction for both models

tf-trt:

saved_model_loaded = tf.saved_model.load(directory, tags=[tag_constants.SERVING])
signature_keys = list(saved_model_loaded.signatures.keys())
graph_func = saved_model_loaded.signatures[‘serving_default’]
graph_func._num_positional_args = 4

graph_func(input_2=x_tensor[0],input_3=x_tensor[1],input_4=x_tensor[2],input_5=x_tensor[3]) #prediction

and for tflite:

self.interpreter = tf.lite.Interpreter(model_path = directory)
self.interpreter.set_tensor(self.input_details[0][‘index’], self.data[0, :, :, :, :])
self.interpreter.set_tensor(self.input_details[1][‘index’], self.data[1, :, :, :, :])
self.interpreter.set_tensor(self.input_details[2][‘index’], self.data[2, :, :, :, :])
self.interpreter.set_tensor(self.input_details[3][‘index’], self.data[3, :, :, :, :])
self.interpreter.invoke()
result = self.interpreter.get_tensor(self.output_details[0][‘index’])

if you have another way for converting this multi view CNN model to tensorRT model I appreciate it .
thank you so much

Environment

TensorRT Version : 7.1.3.0
GPU Type :
Nvidia Driver Version :
CUDA Version : 10.2.89
CUDNN Version : 8.0.0.180
Operating System + Version : ubuntu 18.04
Python Version (if applicable) : 3.6.9
TensorFlow Version (if applicable) : 2.3.1

Hi,

Please check the detailed layer placement from the conversion log.
https://docs.nvidia.com/deeplearning/frameworks/tf-trt-user-guide/index.html#tf-trt-op

Since not all the layers are supported by TF-TRT, if you frequently switch frameworks (TF & TRT) across layers.
It will cause some overhead due to Nano’s limited resources.

If you don’t need the TensorFlow interface, it’s recommended to convert the model into an independent TensorRT engine directly.

Thanks.