Nvidia Jetson crashes when inferencing images with cudnn opencv

Robotics & Edge Computing Jetson & Embedded Systems Jetson Xavier NX
1

Hi

I have a Nvidia jetson nx xavier and an application to process images with yolov5. I have 4 models, one for each class.
I use opencv to process the images. Here is the problem:
I’ve been using yolov5m for some time, but I realized the accuracy was not good, so I decided to use yolov5m6. For this model the images should be 1280 for inference. What I do is use an opencv function (blobfromimage) where the size of the image gets stretched to 1280. But when I run the program, after some time, the nvidia crashes. This happens when teamviewer or other remote desktop application is open. I tried run the program and deactivate the teamviewer and the issue doesn’t happen again. But I will need to access the Nvidia through a remote desktop application, so I was wondering if the processing capability of the board is enough for my project. I mean I have to do 4 inferences in the same image. When I was using the models yolov5m this never happened.
Is it capable?

Thank you

3

Hi,

We need more information about this issue.

1. Do you get any errors when the crash happens?
Please also check if there is any error when typing dmesg.

2. Does the crash happen when running only ONE inference?

3. Could you monitor the memory status and share the output when the crash happens?

$ sudo tegrastats

Thanks.

4

Hi,

  1. I used to have a “killed” message
  2. The crash happens when doing the inference of an image through 4 different yolo models (in series). I mean, after a inference with one model another inference occurs in the same image for another model.
  3. I tried that one time and I could see the cpu’s at 100%. The memory was also near 8 gb

I tried the new power mode 20W and apparently the issue is solved. Although the system is a bit slow.

5

Hi,

A killed message usually occurs when running out of memory.
This also meets the status you saw with the tegrastats.

Thanks.

6

Hi,

So is this an hardware limitation? Did you ever saw this issue occur on other models? Can you investigate if this is normal for my model? To reproduce this issue it is necessary at least 4 yolov5 (1280 version) models and use the “recipe” found here Object Detection using YOLOv5 OpenCV DNN in C++ and Python
On the python code. The models should be in series which means the processing is done 4 times (one loop for each model).

Thanks

7

Hi,

Running out of memory indicates that your application allocates too much memory.
This can be solved by reducing the memory usage of the software.
The different models should also require different amounts of memory.
(ex. a deeper model might need more)

XavierNX board has 16GiB memory so you can base on this to find a suitable model and concurrent inference number.

Thanks.

9

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.