Expected Tensor RT 8 RAM Usage

[ivan24]

Hello, everyone!

We’ve upgraded from L4T R32.4.3 and Tensor RT 7 to L4T R32.6.1 and Tensor RT 8 running on a Jetson TX2 4 GB kit.

It looks like the TRT run-time memory consumption dropped with a measurable 100 MB on some models going from 1.4 GB to 1.3 GB and it did not change on others where it stayed at 1.0 GB.

Can you help me understand why our models cause the TRT run-time to exhibit these two clearly defined patterns of memory usage?

Can we expect more improvements on the memory usage topic? I recall a plan to extract the inference essential part of cuDNN into a separate library (from another user question). Are these 100 MB a step in this direction?

Looking forward to some more information on the topic.
Thank you for your time!

P.S.
The “trtexec --loadEngine” command’s memory usage is spot on with that of loading an engine and running inference with the C++ API.

[AastaLLL]

Hi,

The memory usage depends on the deployed algorithm.
In general, TensorRT will choose a faster one that is within the allowed workspace amount.

In TensorRT-8, you can run the inference without cuDNN.
Please try this feature with trtexec binary first.

$ /usr/src/tensorrt/bin/trtexec -h
&&&& RUNNING TensorRT.trtexec [TensorRT v8001] # /usr/src/tensorrt/bin/trtexec -h
...
=== Build Options ===
  ...
  --tacticSources=tactics     Specify the tactics to be used by adding (+) or removing (-) tactics from the default
                              tactic sources (default = all available tactics).
                              Note: Currently only cuDNN, cuBLAS and cuBLAS-LT are listed as optional tactics.
                              Tactic Sources: tactics ::= [","tactic]
                                              tactic  ::= (+|-)lib
                                              lib     ::= "CUBLAS"|"CUBLAS_LT"|"CUDNN"
                              For example, to disable cudnn and enable cublas: --tacticSources=-CUDNN,+CUBLAS

Thanks.