OpenCV 4.1.1 supplied with Jetpack 4.3 and Jetpack 4.4 DP has a serious memory leak

Robotics & Edge Computing Jetson & Embedded Systems Jetson Nano
1

I’m using a Jetson Nano to run some object detection code which is a combination of some of the original YOLO Darknet code and a TensorRT version of the YOLO model. This was running with no problems with Jetpack 4.2.1 plus OpenCV 4.1.2 built from source. I recently decided to try upgrading to JP 4.3 and JP 4.4 DP and discovered that, when running object detection on a video, all 4G memory on the nano would be consumed within 2-3 mins, causing it to grind to a halt. Experimentation showed that this happened even if I didn’t load the model or run inference and I’ve discovered that the memory leak is in OpenCV. I think it might be related to memory management in the Mat class.

The memory leak occurs in the following cases:

  • using a JP 4.3 image (which includes openCV 4.1.1)
  • building L4T 32.4.2 from source and then installing JP 4.4 (including OpenCV 4.1.1) using apt-get
  • uninstalling openCV from the above JP 4.4 installation and building OpenCV 4.1.1 from source (with or without CUDA support)

The memory leak does not occur in the following cases:

  • JP 4.2.1 plus OpenCV 4.1.2 built from source
  • JP 4.4 plus OpenCV 4.1.2 built from source

I realise that a memory leak in OpenCV is not the responsibility of the Nvidia team but I wanted to post this here to help anyone else who is having the same problem and in the hope that Nvidia will consider moving to a different OpenCV version in Jetpack.

Unfortunately I haven’t been able to come up with a minimal code example that reproduces the problem. I currently have a cut down version of my full application that exhibits the memory leak and a test application that I believe has all the relevant features of my full application and yet doesn’t cause the memory leak. Apologies as I realise that this post would be a lot more useful if I could provide code to reproduce the problem.

3

Hi,

I am little bit confused by this statement.

building L4T 32.4.2 from source and then installing JP 4.4 (including OpenCV 4.1.1) using apt-get

What command did you use here?

4

sudo apt-get install nvidia-jetpack

5

Ok. Looks good.

Could you also try Opencv4.1.1 built from source to verify your assumption?

6

Yes I already have (see my original post).

1 Like
7

This may also be the cause of the problem described here:

8

Hi,

Do you have opencv application other than Darknet that also suffers leakage issue?
I think current info all point to this issue would happen when running Darknet with opencv4.1.1.

9

No. I don’t run any other openCV applications on the Jetson Nano. However, if one application has a memory leak with that version of openCV, presumably others could too. The issue seems to be related to taking a subregion of a Mat object but I can’t nail down exactly how to reproduce it.
I should also say that my current application has loads of changes relative to the original darknet, so I can’t say if the memory leak would occur with that too (although the other post I referenced suggests it might).

10

Ok,

Please use other opencv for your usecase first. We will see if we can update it.
https://github.com/AastaNV/JEP/blob/master/script/install_opencv4.3.0_Jetson.sh

1 Like
12

Hi,

We tested darket with default OpenCV in JetPack4.4 GA and no memory leakage is found.
It 's recommended to upgrade your software and give it a try.

$ ./darknet detector demo cfg/coco.data cfg/yolov3-tiny.cfg yolov3-tiny.weights /home/nvidia/sample_1080p_h264.mp4

Please let us know if the issue goes on in your side.

Thanks.