Unable to generate tensorrt engine for facenet model on Jetson Nano 2 GB

chitransh.valawat · December 28, 2020, 5:49am

Scenario A

Tried to allocate swap memory of 6GB through the script:

JetsonHacksNano/installSwapfile/blob/master/installSwapfile.sh

#!/bin/bash
# Copyright 2017-2019 JetsonHacks
# MIT License
# Create a swap file and set up permissions
# If a parameter is passed, it should be the place to create the swapfile
set -e
SWAPDIRECTORY="/mnt"
# Ubuntu recommends 6GB for 4GB of memory when using suspend
# You can use 1 or 2 if need be
SWAPSIZE=6
AUTOMOUNT="Y"
function usage
{
    echo "usage: installSwapFile [[[-d directory ] [-s size] -a] | [-h]]"
    echo "  -d | --dir <directoryname>   Directory to place swapfile ( default: /mnt)"
    echo "  -s | --size <gigabytes> (default: 6)"
    echo "  -a | --auto  Enable swap on boot in /etc/fstab (default: Y)"
    echo "  -h | --help  This message"
}

This file has been truncated. show original

As per the “swap_area_modified_killed.png” snap, the process gets killed after using 1915MB RAM area out of 1972MB total RAM and 2017MB of SWAP area from 7144MB(Total ~4GB from ~9GB)

…
UNKNOWN: Retargeting InceptionResnetV1/Repeat_2/block8_5/Branch_1/Conv2d_0c_3x1/Relu to InceptionResnetV1/Repeat_2/block8_5/concat
UNKNOWN: Eliminating concatenation InceptionResnetV1/Block8/concat
UNKNOWN: Generating copy for InceptionResnetV1/Block8/Branch_1/Conv2d_0a_1x1/convolution + InceptionResnetV1/Block8/Branch_1/Conv2d_0a_1x1/Relu || InceptionResnetV1/Block8/Branch_0/Conv2d_1x1/convolution + InceptionResnetV1/Block8/Branch_0/Conv2d_1x1/Relu to InceptionResnetV1/Block8/concat
UNKNOWN: Retargeting InceptionResnetV1/Block8/Branch_1/Conv2d_0c_3x1/Relu to InceptionResnetV1/Block8/concat
UNKNOWN: After concat removal: 137 layers
UNKNOWN: Graph construction and optimization completed in 1.2832 seconds.
Killed

nano@nano-desktop:~$ swapon -s
Filename Type Size Used Priority
/mnt/swapfile file 6291452 0 -1
/dev/zram0 partition 255996 0 5
/dev/zram1 partition 255996 0 5
/dev/zram2 partition 255996 0 5
/dev/zram3 partition 255996 0 5

Scenario B

Tried to allocate swap area 1GB to each CPU ZRAM:

github.com

JetsonHacksNano/resizeSwapMemory/blob/master/setSwapMemorySize.sh

#!/bin/bash
# Copyright (c) 2019 Jetsonhacks 
# MIT License
# Set the zram swap file size on Jetson Nano
# Default is 2GB 


function usage
{
    echo " usage: ./setSwapFileSize [ [-g #gigabytes ] | [ -m #megabytes ] | [ -h ]"
    echo "   -g #gigabytes - #gigabytes total to use for swap area"
    echo "   -m #megabytes - #megabytes total to use for swap area"
    echo "   -h | --help  This message"
}

# Iterate through command line inputs
if [ "$#" -gt 2 ] || [ "$#" == 0 ]  ; then
   usage
   exit 1
fi

This file has been truncated. show original

[zram_modified_killed|690x387]
(https://forums.developer.nvidia.com/uploads/short-url/pv2TwAN8xu1FMGFMN4ch6uPNQIK.jpeg)

As per the “zram_modified_killed.png” snap, the process gets killed after using 1915MB RAM area out of 1972MB total RAM and 2017MB of SWAP area from 7144MB(Total ~4GB from ~9GB)

$ sudo tegrastats
RAM 1904/1972MB (lfb 1x1MB) SWAP 2030/40000MB (cached 0MB) IRAM 0/252kB(lfb 252kB)

nano@nano-desktop:~$ swapon -s
Filename Type Size Used Priority
/dev/zram0 partition 1023996 7056 5
/dev/zram1 partition 1023996 7052 5
/dev/zram2 partition 1023996 7068 5
/dev/zram3 partition 1023996 7064 5

In the both the scenarios, the process got killed after using ~4GB(2GB swap & 2GB RAM) even the swap area is still not fully occupied.

AastaLLL · December 28, 2020, 9:10am

Hi,

It looks like your application is a deep-learning use case.

Please noted that GPU could not use the swap memory.
So to increase swap memory won’t have too much help for CUDA inference.

Thanks.

Topic		Replies	Views
Unable to generate tensorrt engine for facenet model on Jatson Nano 2 GB TensorRT tensorrt , cuda	1	334	December 28, 2020
Script killed Jetson Nano	16	4532	October 18, 2021
Zram vs swapfile Jetson TX2	20	1757	August 17, 2023
Make full use of the swapfile Jetson Nano tensorrt	6	890	June 2, 2023
GPU out of memory when the total ram usage is 2.8G Jetson TX2	28	18540	October 18, 2021
TensorRT process killed with Orin Nano Jetson Orin Nano tensorrt	9	1346	July 27, 2023
Mxnet installation for Jetson Nano Jetson Nano neural-network-framework	5	1558	October 18, 2021
Swapoff optimization for zram (1GB ~10min) Jetson Nano kernel	10	1813	March 14, 2023
Strategy: how to overcome GPU Out-of-Memory? Jetson Nano opencv , cuda , tensorflow	9	3731	October 18, 2021
The perilous adventure of setting up Stable Diffusion A1111 on a Jetson Nano 4GB Jetson Nano cuda , pytorch , generative_ai	4	877	May 2, 2024

Unable to generate tensorrt engine for facenet model on Jetson Nano 2 GB

Scenario A

Scenario B

Related topics