Multiple TensorRT models Inference on Jetson Orin

Hi,
I have optimized Tensorflow/Keras Models to TensoRT using TF-TRT and am getting proper outputs.
When I try to run multiple models at the same time - am getting GPU issues.
is there any function in TRT which is similar to Tensorflow’s memory growth function: tf.config.experimental.set_memory_growth(gpus[0], True)?
TensorRT Verison : 8.5.2.2
Jetpack Version : 5.1.2-b104
Tensorflow Version : 2.12.0+nv23.6

Hi,

You can configure the memory maximum with setMemoryPoolLimit at building time.
So TensorRT will pick up the algorithm that can run in the given memory limit.

This is a pure TensorRT API. Please check if this has been exported to the TF-TRT interface.
https://docs.nvidia.com/deeplearning/tensorrt/archives/tensorrt-853/api/c_api/classnvinfer1_1_1_i_builder_config.html#a0a88a9b43bbe47c839ba65de9b40779f

Thanks.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.