[TensorRT] How to load multiple TF-TRT model?

tuan.ngo1999 · May 5, 2020, 1:20pm

Hello,
How can I configure to load multiple models using TF-TRT (the models inference sequentially in a single thread)?

The models are optimized with TF-TRT and I use frozen graph to load model to inference but I cannot load more than one graph when executing. The error is:
tensorflow.python.framework.errors_impl.UnavailableError: Can’t provision more than one single cluster at a time

I am using Jetson TX2, Jetpack 4.3, Tensorflow 1.15.
Thank you.

SunilJB · May 6, 2020, 7:24am

I would suggest you to look into Triton Inference Server. It should have pretty flexible capabilities for loading several models.

Or you can post question on tf-trt github page: GitHub · Where software is built

Similar issue:

github.com/tensorflow/tensorrt

UnavailableError: Can't provision more than one single cluster at a time

opened 09:36AM - 29 Sep 19 UTC

leo-XUKANG

my code: ``` FP32_SAVED_MODEL_DIR = SAVED_MODEL_DIR+"_TFTRT_FP32/1" !rm -rf $…FP32_SAVED_MODEL_DIR #Now we create the TFTRT FP32 engine trt.create_inference_graph( input_graph_def=None, outputs=None, max_batch_size=1, input_saved_model_dir=SAVED_MODEL_DIR, output_saved_model_dir=FP32_SAVED_MODEL_DIR, precision_mode="FP32") benchmark_saved_model(FP32_SAVED_MODEL_DIR, BATCH_SIZE=1) ``` and i have set: `import os os.environ["CUDA_DEVICE_ORDER"]="PCI_BUS_ID" os.environ["CUDA_VISIBLE_DEVICES"]="0"` when i run ,i got an error: `InvalidArgumentError: Failed to import metagraph, check error log for more info` and then i add a code: `tf.keras.backend.set_learning_phase(0)` the error is gone ,but one error rasie: `UnavailableError: Can't provision more than one single cluster at a time` emmm....... i just use one GPU,which is RTX 2080ti cuda: Cuda compilation tools, release 10.0, V10.0.130 SOMEONE HELP ME, PLEASE!

Thanks

tuan.ngo1999 · May 14, 2020, 2:29pm

Thank you.

I have solved the problem by create multiple threads and each thread has one session with one graph.

Topic		Replies	Views
Running multiple TensorRT-optimized models in Tensorflow TensorRT tensorrt , tensorflow	1	703	April 7, 2020
Not able to inference multiple input models using TRT TensorRT tensorrt , tensorflow , jetson-inference	1	495	August 12, 2021
How can I infer sequentially using two independent TensorRT models? Jetson Nano tensorrt	2	420	March 30, 2022
Multiple model Inference And Runtime Model Switching Isaac ROS ros , isaac-ros-dnn-inference	3	977	May 13, 2024
Loading a new plan file while running inference TensorRT	1	661	June 9, 2020
Real Time Inference with Multi GPU - Multiple Model Triton Inference Server (archived)	1	1497	January 29, 2020
Is it possible to run multiple TensorRT model inference on a GPU simultaneously and parallelly? TensorRT tensorrt , cuda	3	2218	August 23, 2022
Unable to run two TensorRT models in a cascade manner TensorRT tensorrt , python	7	5155	October 12, 2021
TF-TRT5: How to run tensorflow-tensorrt inferences with multiple GPUs TensorRT	10	3755	September 3, 2019
Run multiple model(engine) with tensorrt without deepstream TensorRT	1	1182	April 20, 2020

[TensorRT] How to load multiple TF-TRT model?

Related topics