[TensorRT] How to load multiple TF-TRT model?

How can I configure to load multiple models using TF-TRT (the models inference sequentially in a single thread)?

The models are optimized with TF-TRT and I use frozen graph to load model to inference but I cannot load more than one graph when executing. The error is:
tensorflow.python.framework.errors_impl.UnavailableError: Can’t provision more than one single cluster at a time

I am using Jetson TX2, Jetpack 4.3, Tensorflow 1.15.
Thank you.

I would suggest you to look into Triton Inference Server. It should have pretty flexible capabilities for loading several models.

Or you can post question on tf-trt github page: Issues · tensorflow/tensorrt · GitHub

Similar issue:


Thank you.

I have solved the problem by create multiple threads and each thread has one session with one graph.