# Peak memory usage during TensorRT execution

**URL:** <https://forums.developer.nvidia.com/t/peak-memory-usage-during-tensorrt-execution/266894>\
**Category:** TensorRT\
**Created:** [September 19, 2023, 1:05pm UTC](https://forums.developer.nvidia.com/t/peak-memory-usage-during-tensorrt-execution/266894 "2023-09-19T13:05:18Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![minkyu.kim2](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@minkyu.kim2](https://forums.developer.nvidia.com/u/minkyu.kim2)\
**Post date:** [September 19, 2023, 1:05pm UTC](https://forums.developer.nvidia.com/t/peak-memory-usage-during-tensorrt-execution/266894/1 "2023-09-19T13:05:18Z")

</div>

## Description

Hi,  
I’m trying to extract peak memory usage information for a TensorRT engine execution.  
I found out that in the log for building the engine with `trtexec`, there is some information like this.

```auto
[I] [TRT] [MemUsageStats] Peak memory usage of TRT CPU/GPU memory allocators: CPU 59 MiB, GPU 13 MiB

```

1. I wonder what exactly the peak memory usage means here and wonder if it can be considered as a reliable data to get peak memory of a TensorRT engine execution. I got confused because with some models, the peak memory reported is too small. For example, running `trtexec` with the [attached ONNX model](https://drive.google.com/drive/folders/1oRbGyqMXZUzRiBlJvKFC2HdAPeMSXCow?usp=sharing), which is Swin\_b with fake quantization, reports only 4MB as GPU peak memory as below, whereas the model size itself is 340M.(approx. 85M considering quantization)

```auto
[I] [TRT] [MemUsageStats] Peak memory usage of TRT CPU/GPU memory allocators: CPU 0 MiB, GPU 4 MiB

```

1. If it’s not actually equivalent to the real peak memory usage, is there any alternatives?

## Environment

**TensorRT Version** : 8.6.1  
**GPU Type** : RTX A6000  
**Nvidia Driver Version** : 510.108.03  
**CUDA Version** : 12.1  
**CUDNN Version** :  
**Operating System + Version** : Ubuntu 20.04  
**Python Version (if applicable)**:  
**TensorFlow Version (if applicable)**:  
**PyTorch Version (if applicable)**:  
**Baremetal or Container (if container which image + tag)**: [nvcr.io/nvidia/tensorrt:23.07-py3](http://nvcr.io/nvidia/tensorrt:23.07-py3)

## Relevant Files

[ONNX model](https://drive.google.com/drive/folders/1oRbGyqMXZUzRiBlJvKFC2HdAPeMSXCow?usp=sharing)

## Reproduce

To reproduce the result of Swin\_b,

```auto
trtexec --onnx=[onnx_path] --int8

```

---

<div class="post-metadata">

**Author:** ![spolisetty](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/spolisetty/32/14047_2.png) [@spolisetty](https://forums.developer.nvidia.com/u/spolisetty)\
**Post date:** [September 21, 2023, 4:15am UTC](https://forums.developer.nvidia.com/t/peak-memory-usage-during-tensorrt-execution/266894/3 "2023-09-21T04:15:07Z")

</div>

Hi,

> [@minkyu.kim2](#):
>
> reports only 4MB as GPU peak memory as below, whereas the model size itself is 340M.(approx. 85M considering quantization)

Peak memory usage reporting is not accurate in TRT 8.6, it will be fixed in upcoming major releases.

Thank you.

---

<div class="post-metadata">

**Author:** ![niloofar.zarif1](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@niloofar.zarif1](https://forums.developer.nvidia.com/u/niloofar.zarif1)\
**Post date:** [October 16, 2024, 12:38am UTC](https://forums.developer.nvidia.com/t/peak-memory-usage-during-tensorrt-execution/266894/6 "2024-10-16T00:38:39Z")

</div>

Are there any updates on this for newer versions of trt?  
I tried using `torch.cuda.max_memory_reserved(device)` and `torch.cuda.max_memory_allocated(device)` for measuring peak memory usage while using a trt engine to do inference but the results are very inaccurate and vastly different than what I’m observing through `nvidia-smi`.