Hi, I have a keras multi view cnn that has the following summary
Layer (type) Output Shape Param # Connected to
input_2 (InputLayer) [(None, 32, 32, 3)] 0
input_3 (InputLayer) [(None, 32, 32, 3)] 0
input_4 (InputLayer) [(None, 32, 32, 3)] 0
input_5 (InputLayer) [(None, 32, 32, 3)] 0
model (Functional) (None, 32) 46400 input_2[0][0]
input_3[0][0]
input_4[0][0]
input_5[0][0]
concatenate (Concatenate) (None, 128) 0 model[0][0]
model[1][0]
model[2][0]
model[3][0]
dense_1 (Dense) (None, 32) 4128 concatenate[0][0]
dropout_4 (Dropout) (None, 32) 0 dense_1[0][0]
dense_2 (Dense) (None, 8) 264 dropout_4[0][0]
dropout_5 (Dropout) (None, 8) 0 dense_2[0][0]
dense_3 (Dense) (None, 2) 18 dropout_5[0][0]
Total params: 50,810
Trainable params: 50,810
Non-trainable params: 0
I’ve tried to convert it to tf-trt model and tflite model like below:
converting to tf-trt :
conversion_params = trt.DEFAULT_TRT_CONVERSION_PARAMS
conversion_params = conversion_params._replace(max_workspace_size_bytes=(1<<32))
conversion_params = conversion_params._replace(precision_mode=“FP16”)
conversion_params = conversion_params._replace(maximum_cached_engines=100)
converter = trt.TrtGraphConverterV2(
input_saved_model_dir=“saved_model_4pic”,
conversion_params=conversion_params)
converter.convert()
converter.save(“trt_FP16_4pic”)
saved model is frozen graph of keras model(pb file)
and for converting to tflite:
model = tf.keras.models.load_model(‘./1pic/3class_2019-09-15_0232.h5’)
converter = tf.lite.TFLiteConverter.from_keras_model(model)
tflite_model = converter.convert()
open(“converted_model.tflite”, “wb”).write(tflite_model)
after converting I predict with both models(tf-trt and tflite)
and the result time shows that tflite lite is much faster than tf-trt (prediction time for tf-trt = 250ms and for tflite = 5.8 ms)
but as you can see in the internet tensorrt is faster than tflite in jetson nano
I guess my prediction time isnt correct because of my way to predict
lets see the way of prediction for both models
tf-trt:
saved_model_loaded = tf.saved_model.load(directory, tags=[tag_constants.SERVING])
signature_keys = list(saved_model_loaded.signatures.keys())
graph_func = saved_model_loaded.signatures[‘serving_default’]
graph_func._num_positional_args = 4
graph_func(input_2=x_tensor[0],input_3=x_tensor[1],input_4=x_tensor[2],input_5=x_tensor[3]) #prediction
and for tflite:
self.interpreter = tf.lite.Interpreter(model_path = directory)
self.interpreter.set_tensor(self.input_details[0][‘index’], self.data[0, :, :, :, :])
self.interpreter.set_tensor(self.input_details[1][‘index’], self.data[1, :, :, :, :])
self.interpreter.set_tensor(self.input_details[2][‘index’], self.data[2, :, :, :, :])
self.interpreter.set_tensor(self.input_details[3][‘index’], self.data[3, :, :, :, :])
self.interpreter.invoke()
result = self.interpreter.get_tensor(self.output_details[0][‘index’])
if you have another way for converting this multi view CNN model to tensorRT model I appreciate it .
thank you so much
Environment
TensorRT Version : 7.1.3.0
GPU Type :
Nvidia Driver Version :
CUDA Version : 10.2.89
CUDNN Version : 8.0.0.180
Operating System + Version : ubuntu 18.04
Python Version (if applicable) : 3.6.9
TensorFlow Version (if applicable) : 2.3.1