How to get total used gpu memory and set gpu memory limit when tensorrt inferring?

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

Description

Is there any solution to get total used gpu memory and set gpu memory limit when tensorrt inferring? Like torch:
torch.cuda.memory_allocated()
torch.cuda.set_per_process_memory_fraction()

Environment

TensorRT Version:
GPU Type:
Nvidia Driver Version:
CUDA Version:
CUDNN Version:
Operating System + Version:
Python Version (if applicable):
TensorFlow Version (if applicable):
PyTorch Version (if applicable):
Baremetal or Container (if container which image + tag):

2

Hi @450959507 ,
You should be able to see this information while running your onnx model using trtexec by adding ‘–verbose’.
Thanks

3

Is there any api to get this information when i use trt python sdk to infer?

4

Hi,

Currently, we do not have. You can limit the workspace memory size be using following.
https://docs.nvidia.com/deeplearning/tensorrt/api/python_api/infer/Core/BuilderConfig.html#tensorrt.MemoryPoolType

Thank you.