Running CNN Model face_recognition on DLIB on Snap from Camera - KILLED

Please help Nvidia, you’re my only hope!

I’m running some very simple code, but it keeps getting killed. This works on a Windows 10 box with DLIB and a GTX 1080. I’m trying to port this to a Jetson Nano to productize a system. Basically I’m snapping a jpeg (~2mb 1080p) from a web cam and checking it against some reference files. I’ve cut the code down the few lines that recognize the face and I’m failing on face_recognition.face_locations(feed_image,number_of_times_to_upsample=1, model=“cnn”) which causes the nano to run out of memory and crash. I’m using the latest DLIB dlib-19.19, with and without code changes mentioned below.

I’ve tried to fix this…

  1. I've rebuilt DLIB using the commenting out of //forward_algo = forward_best_algo; this changed nothing
  2. I've tried removing the swap file, this caused it to fail sooner.
#load file
feed_image = face_recognition.load_image_file(feed_file)
raw_image = cv2.imread(feed_file)
print('file loaded')

#check if a face exists
print('running cnn model')
face_locations = face_recognition.face_locations(feed_image,number_of_times_to_upsample=1, model="cnn")
print('cnn model finished')

This is the result…

root@windranger:/opt/program# python3
file loaded
running cnn model

This is the dmesg.

[  208.499902] Out of memory: Kill process 6296 (python3) score 268 or sacrifice child
[  208.508508] Killed process 6296 (python3) total-vm:10110436kB, anon-rss:618244kB, file-rss:499100kB, shmem-rss:0kB
[  208.608826] nvgpu: 57000000.gpu gk20a_fifo_tsg_unbind_channel_verify_status:2200 [ERR]  Channel 507 to be removed from TSG 4 has NEXT set!
[  208.636342] nvgpu: 57000000.gpu          gk20a_tsg_unbind_channel:164  [ERR]  Channel 507 unbind failed, tearing down TSG 4
[  208.753182] oom_reaper: reaped process 6296 (python3), now anon-rss:0kB, file-rss:499128kB, shmem-rss:0kB

Version Info

root@windranger:/usr/local/src# cat /etc/nv_tegra_release
# R32 (release), REVISION: 3.1, GCID: 18186506, BOARD: t210ref, EABI: aarch64, DATE: Tue Dec 10 06:58:34 UTC 2019

I took a smaller image file, 10kb versus 2 megabytes and it runs. We must be able to run larger files than 10kb?

“Out of memory: Kill process 6296 (python3) score 268 or sacrifice child”

Did u configure swap memory?

at least you should have 4 GB to run.

for more info see jetsonhacks nano setup article/video.

Thank you.

I configured 6GB of swap memory to compile DLIB. I added swap, and removed swap – same error both ways. With swap configured it took about 10 minutes to crash processing the image. Without swap is crashes within 2-3 minutes processing the image with the CNN model.

I’ve configured OPENCV to downsize the image to 480p from 1080p and the model runs now in about 1/2 a second which is fine. I still don’t understand why it uses 4+ GB of ram and crashes with a 1080p camera JPG.


Nano do support 1080p camera decoding.

The problem of OOM is that the model in dlib take too much memory and proportional to the input image size.
In general, we don’t use high resolution image for deep network to reduce unwanted noise.
For example, YOLO only use 608x608 image as input.