Tensorrt7 model is double size to Tensorrt4 model

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

Description

I am trying to migrate tensorrt4 to tensorrt7, but I found that the model created with tensorrt7 is about 2 times bigger than tensorrt4, so as their gpu memory usage. Why?

Environment

TensorRT Version: 7.0.0.11, 4.0.0.3
GPU Type: 1080Ti
Nvidia Driver Version: 10.2 and 9.0 tested
CUDA Version: 390.77 and 440.82
CUDNN Version: 7.6.5
Operating System + Version: ubuntu 18.04
Python Version (if applicable): 3.6
TensorFlow Version (if applicable):
PyTorch Version (if applicable):
Baremetal or Container (if container which image + tag):

Relevant Files

Please attach or include links to any models, data, files, or scripts necessary to reproduce your issue. (Github repo, Google Drive, Dropbox, etc.)

Steps To Reproduce

Please include:

  • Exact steps/commands to build your repro
  • Exact steps/commands to run your repro
  • Full traceback of errors encountered
2

The memory usage is dependent on the device and kernel used to optimize the model based on precision and other factors.
To determine the amount of memory a model will use, please below link question How do I determine how much device memory will be required by my network?
https://docs.nvidia.com/deeplearning/tensorrt/archives/tensorrt-710-ea/developer-guide/index.html#faq

In this case it might be possible that more optimized kernel is being used to optimize the performance of model in TRT 7 at cost of additional memory.

You can try to changing the max workspace size while creating engine to reduce the memory consumption, but may degrade performance when setting it too small.

Thanks

1 Like