TF-TRT demo for Jetson Nano (TF2)

Hi all,

I’ve made a small demo example that showcases TF-TRT optimization of a small toy model in Keras TF2. You can run it on JetPack 4.4 using three simple commands:

git clone https://github.com/pascal-hwky/tftrt-jetson.git
cd tftrt-jetson
./arm.sh

Please find the repository here. Currently, I’m still receiving a strange error in the optimization stage:

Assertion Error in fromSymbolic: 0 (x must be build-time constant)

Unfortunately, this prevents the conversion of this segment to a fully optimized TRT engine operation. If anybody has seen this error before and knows what’s causing it, I would greatly appreciate any help!

1 Like

Hi,

We are going to reproduce this issue.
Will share more information with you later.

Thanks.

Hi,

We can reproduce this in our environment.
Will let you know once we got a progress.

Thanks.

1 Like

HI @AastaLLL, have you made any progress on this issue?

Thanks.

Hi,

Sorry for keeping you waiting.

This issue is caused from the restrictions of dynamic shapes.
Based on the document below, you will need to specify the constant channel input:
https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html#rest_dynamic_shapes

This indicates that you will need to have at least 4 dimension (including batch) input tensor on NHWC format.
After checking, your sample can work correctly when expand the input tensor to the 2nd or 4th dimension.

    inputs = keras.Input(shape=(1,1,32,))
    ...

def my_input_fn():
  for _ in range(100):
    inp1 = np.random.normal(size=(8,1,1,32)).astype(np.float32)
    yield inp1,

Or

    inputs = keras.Input(shape=(32,1,1,))
    ...
def my_input_fn():
  for _ in range(100):
    inp1 = np.random.normal(size=(8,32,1,1)).astype(np.float32)
    yield inp1,

Thanks.

Hi @AastaLLL, thanks!

In order to have a network segment eligible for TensorRT optimization, I additionally had to change the network architecture to use convolutional layers.

I have updated the repository.