How to convert keras .h5 model and use it in tensorrt

Autonomous Machines Jetson & Embedded Systems Jetson Nano
1

HI All,

I am pretty new to the DL so please bear with me if I asked anything stupid.

I had defined a custom model using module tensorflow.python.keras. I had serval TimeDistributed Layer and LSTM layer in the custom model and I successfully save it as follow:

cutom_model.h5

When I tried to free the graph following instruction from various the internet (for example this: https://www.dlology.com/blog/how-to-run-keras-model-on-jetson-nano/)

I received the following error message.

File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/util/deprecation.py", line 324, in new_func
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/graph_util_impl.py", line 302, in convert_variables_to_constants
    raise ValueError("Cannot find the variable that is an input "
ValueError: Cannot find the variable that is an input to the ReadVariableOp.

But if I use a model with no LSTM and TimeDistributed(with only ConvNet and Dense), tf is able to freeze the graph and generate the .pb file

Did I do something wrong or I need to wait for the TensorFlow update in order to use tensorrt? Is there any beginner guide to tensorrt using jetson Nano?

2

Hi,

A possible reason is that some of your layers may not be supported from the TensorRT compiler.
Would you mind to follow our official sample to freeze the model first?
/usr/src/tensorrt/samples/python/end_to_end_tensorflow_mnist

Thanks.

3

Hi,

Most of the code is the same. The problem is in this line of code:

frozen_graph = tf.graph_util.convert_variables_to_constants(sess, sess.graph.as_graph_def(), output_names)

for your information, inside tensorflow library ( I am using tf1.14 precompile for jetson Nano)

map_name_to_node[source_op_name].op # 'Enter'
source_op_name # 'lstm/while/split/ReadVariableOp/Enter'

Is it because LSTM is not supported? Does it support GRU / basic RNN?

4

Hi,

This depends on the layers used in your model.
For example, TensorRT support basic LSTM and you can find an example here:
/usr/src/tensorrt/samples/sampleCharRNN
/usr/src/tensorrt/samples/sampleNMT

Thanks.

5

Hi,

The Layer is pretty straight forward and please find the details as follow:
I am able to convert the model from .h5 to pb than uff without LSTM layer.

input_dim = (seq_length,) + img_in
    img_in = Input(shape=input_dim, name='img_in')
    imu_in = Input(shape=(seq_length, imu_in), name="imu_in")

    x = TimeDistributed(Convolution2D(24, (5, 5), strides=(2, 2), activation='relu'))(img_in)
    x = Dropout(drop)(x)
    x = TimeDistributed(Convolution2D(32, (5, 5), strides=(2, 2), activation='relu'))(x)
    x = Dropout(drop)(x)
    x = TimeDistributed(Convolution2D(32, (3, 3), strides=(2, 2), activation='relu'))(x)
    x = Dropout(drop)(x)
    x = TimeDistributed(Convolution2D(32, (3, 3), strides=(2, 2), activation='relu'))(x)
    x = Dropout(drop)(x)
    x = TimeDistributed(Flatten())(x)
    x = Dropout(drop)(x)
    x = TimeDistributed(Dense(64, activation='relu'))(x)
    x = Dropout(drop)(x)

    y = TimeDistributed(Dense(14, activation='relu'))(imu_in)
    y = Dropout(drop)(y)
    y = TimeDistributed(Dense(14, activation='relu'))(y)
    y = Dropout(drop)(y)

    z = concatenate([x, y])

    z = LSTM(128, return_sequences=True, activation='tanh')(z)
    z = LSTM(128, return_sequences=False, activation='tanh')(z)

    z = Dense(64, activation='relu')(z)
    z = Dropout(drop)(z)
    z = Dense(32, activation='relu')(z)
    z = Dropout(drop)(z)

    out_1 = Dense(31, activation='softmax', name='out_1')(z)

    out_2 = Dense(31, activation='softmax', name='out_2')(z)

    model = Model(inputs=[img_in,imu_in], outputs=[out_1, out_2])
6

Hi,

It doesn’t depends on the model complexity but the layer you used.

The TimeDistributed layer is not in our supported matrix.
That’s why you cannot convert it into TensorRT.

Thanks.

7

Hello,

I have the same issue with building TensorRT engine out of tensorflow model which includes TimeDistributed() layers. Is there any solution for that? I was thinking maybe I can replace Timedstributed() layers with something else?
Many thanks

8

Hi mkarami3,

Please help to open a new topic if it’s still an issue.

Thanks