Is there any other way besides TensorRT to increase the GPU utilization while doing the inference?

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

we’ve used tensorRT to increase the inference GPU utiliz, it worked, but still the utilization is around 30%,not high enough. Is there any other way? It will be great if it goes up to 70%.
We tried to use MPS（muti-process service）to run several models on the same GPU, but seems no help, our GPU is P40.

2

Hi,

Can you clarify what you mean by using TensorRT to increase the inference GPU utilization?

Also, TensorRT Inference Server ([url]https://github.com/NVIDIA/tensorrt-inference-server[/url] and [url]https://ngc.nvidia.com/catalog/containers/nvidia:tensorrtserver[/url]) can load multiple instances of multiple models on each GPU based on a configuration file.

Have you looked into this?

Thanks,
NVIDIA Enterprise Support