Potential memory leak in python engine serializer

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

Hi,
I came across the following problem yesterday evening.

Provide details on the platforms you are using:
Jetson Nano with Jetpack 4.2.2
Python version: 3.6.8

Describe the problem

I’m currently trying to build and serialize a couple of hundreds slightly different trt engines für testing and benchmarking. Therefore I have a for-loop for building engines and serializing/saving them afterward.

When running this script I noticed that the memory occupied is constantly increasing. Also, the increase of used memory seems to be related to the size of the engine. I tracked this issue down to

engine.serialize()

Commenting this line out resolves this issue and the memory consumption stays almost constant during the entire runtime of my script.

Do you have any idea how to avoid this?

2

Hi,

engine.serialize() returns A IHostMemory object that contains the serialized engine.

Serialization needs to be performed only once while building the engine. While performing inferencing the saved serialized model should be used.

Please refer to following link for details:
https://docs.nvidia.com/deeplearning/sdk/tensorrt-developer-guide/index.html#serial_model_python

Thanks

3

Hi,

thank you for your answer. Yes I know that an engine needs to be serialized only once - and that’s what I actually do. Maybe I was not clear enough, so I’ll try to explain more precise.

I want to serialize and save some hundreds of (slightly different) engines to evaluate them later with different power modes, clocks, etc…
Because building an engine takes some time, I want to do this only once to save some time.

So I do something like

for engine_config in configurations:
    with build_engine(engine_config.config) as engine:
        with open('test_engines/' + engine_config.name, 'wb') as f:
            f.write(engine.serialize())

Writing random data to the files is not an issue:

open('test_engines/' + engine_config.name, 'wb') as f:
    data = some_random_data()
    f.write(data)

But when I serialize and save the created engine like this (or the first code snipped above)

open('test_engines/' + engine_config.name, 'wb') as f:
    data = engine.serialize()
    f.write(data)

the memory occupied by the script is constantly increasing.

4

Hi,

Can you try deleting the engine and IHostMemory buffer [output of engine.serialize()] using del function at the end of each for loop?
For eg in this case:
del data
del engine

Please refer below link for more details:
https://docs.nvidia.com/deeplearning/sdk/tensorrt-archived/tensorrt-601/tensorrt-api/python_api/migrationGuide.html#create-and-destroy-functions

Thanks

5

Hi,

I installed JetPack SDK 4.4 at Jetson Xavier NX.
The TensorRT version is 7.1.3.
I found a CPU memory leak problem while converting the onnx model using the python API.

When the engine.serialize() method is executed, the memory is increased and not freed.
I added ‘del’, but the CPU memory increase for the process is the same.

    with builder.build_engine(network, config) as engine:
        with open(output_file_path, "wb") as f:
            engine_data = engine.serialize()
            f.write(engine_data)
            f.close()
            del engine_data
            del f
        del engine