Problem with limited memory after loading model - Jetson Nano

Hi,

I am developing a system using a model developed in the PyTorch environment. I noticed that loading the model into the Jetson Nano memory uses a lot of memory.

I have checked several solutions of different possibilities to load the model:

  1. Jetson Nano after startup consumes on average 654 MB of ram (LXDE Desktop version)

  2. After loading the MobileNetV2 model directly from the Trochvision library, the RAM usage increases to 782 MB on average - the loaded model consumes about 128 MB.

  3. After loading the custom mobilenetV2 model in the Pytorch environment the memory usage increases to 2373.78 MB. The model consumes about 1719 MB.

  4. In the case of loading the same model mobilenetV2 converted to TRT using Torch2TRT the RAM consumption is 3079 MB - the model occupies about 2425 MB of memory.
    (I was inspired by this tutorial: torch2trt/conversion.ipynb at master · NVIDIA-AI-IOT/torch2trt · GitHub and torch2trt/live_demo.ipynb at master · NVIDIA-AI-IOT/torch2trt · GitHub )

  5. In the case of converting the model to ONNX and then generating the trt engine, the memory consumption increases to 3313 MB - the model occupies 2659 MB.
    (I followed this tutorial: TensorRT/4. Using PyTorch through ONNX.ipynb at master · NVIDIA/TensorRT · GitHub )

To check the memory I used:
sudo tegrastats

Is it possible to reduce the ram consumption associated with loading the model? Is there only 800MB of RAM left after loading the optimal trt model?

I would appreciate help if I am doing something wrong.

Hi,

This is a known issue for inference on Jetson.
The reason is that these frameworks usually require the cuDNN library.
But it takes 600~800MiB for loading the library.

In TensorRT v8.0, we provide another alternative to run the model with cuBLAS instead.
Could you wait for our next TensorRT package? It’s included in our JetPack 4.6 release:

Thanks.

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.