Unable to generate tensorrt engine for facenet model on Jatson Nano 2 GB

chitransh.valawat · December 24, 2020, 4:52pm

Description

Unable to generate tensorrt engine for facenet model on Jatson Nano 2 GB.

Scenario A

Tried to allocate swap memory of 6GB through script:

JetsonHacksNano/installSwapfile/blob/master/installSwapfile.sh

#!/bin/bash
# Copyright 2017-2019 JetsonHacks
# MIT License
# Create a swap file and set up permissions
# If a parameter is passed, it should be the place to create the swapfile
set -e
SWAPDIRECTORY="/mnt"
# Ubuntu recommends 6GB for 4GB of memory when using suspend
# You can use 1 or 2 if need be
SWAPSIZE=6
AUTOMOUNT="Y"
function usage
{
    echo "usage: installSwapFile [[[-d directory ] [-s size] -a] | [-h]]"
    echo "  -d | --dir <directoryname>   Directory to place swapfile ( default: /mnt)"
    echo "  -s | --size <gigabytes> (default: 6)"
    echo "  -a | --auto  Enable swap on boot in /etc/fstab (default: Y)"
    echo "  -h | --help  This message"
}

This file has been truncated. show original

!

As per the “swap_area_modified_killed.png” snap, process gets killed after using 1915MB RAM area out of 1972MB total RAM and 2017MB of SWAP area from 7144MB(Total ~4GB from ~9GB)

…
UNKNOWN: Retargeting InceptionResnetV1/Repeat_2/block8_5/Branch_1/Conv2d_0c_3x1/Relu to InceptionResnetV1/Repeat_2/block8_5/concat
UNKNOWN: Eliminating concatenation InceptionResnetV1/Block8/concat
UNKNOWN: Generating copy for InceptionResnetV1/Block8/Branch_1/Conv2d_0a_1x1/convolution + InceptionResnetV1/Block8/Branch_1/Conv2d_0a_1x1/Relu || InceptionResnetV1/Block8/Branch_0/Conv2d_1x1/convolution + InceptionResnetV1/Block8/Branch_0/Conv2d_1x1/Relu to InceptionResnetV1/Block8/concat
UNKNOWN: Retargeting InceptionResnetV1/Block8/Branch_1/Conv2d_0c_3x1/Relu to InceptionResnetV1/Block8/concat
UNKNOWN: After concat removal: 137 layers
UNKNOWN: Graph construction and optimization completed in 1.2832 seconds.
Killed

nano@nano-desktop:~$ swapon -s
Filename Type Size Used Priority
/mnt/swapfile file 6291452 0 -1
/dev/zram0 partition 255996 0 5
/dev/zram1 partition 255996 0 5
/dev/zram2 partition 255996 0 5
/dev/zram3 partition 255996 0 5

Scenario B

Tried to allocate swap area 1GB to each CPU ZRAM:

github.com

JetsonHacksNano/resizeSwapMemory/blob/master/setSwapMemorySize.sh

#!/bin/bash
# Copyright (c) 2019 Jetsonhacks 
# MIT License
# Set the zram swap file size on Jetson Nano
# Default is 2GB 


function usage
{
    echo " usage: ./setSwapFileSize [ [-g #gigabytes ] | [ -m #megabytes ] | [ -h ]"
    echo "   -g #gigabytes - #gigabytes total to use for swap area"
    echo "   -m #megabytes - #megabytes total to use for swap area"
    echo "   -h | --help  This message"
}

# Iterate through command line inputs
if [ "$#" -gt 2 ] || [ "$#" == 0 ]  ; then
   usage
   exit 1
fi

This file has been truncated. show original

zram_modified_killed|690x387

As per the “zram_modified_killed.png” snap, the process gets killed after using 1915MB RAM area out of 1972MB total RAM and 2017MB of SWAP area from 7144MB(Total ~4GB from ~9GB)

$ sudo tegrastats
RAM 1904/1972MB (lfb 1x1MB) SWAP 2030/40000MB (cached 0MB) IRAM 0/252kB(lfb 252kB)

nano@nano-desktop:~$ swapon -s
Filename Type Size Used Priority
/dev/zram0 partition 1023996 7056 5
/dev/zram1 partition 1023996 7052 5
/dev/zram2 partition 1023996 7068 5
/dev/zram3 partition 1023996 7064 5

In the both the scenarios, the process got killed after using ~4GB(2GB swap & 2GB RAM) even the swap area is still not fully occupied.

A clear and concise description of the bug or issue.

Environment

TensorRT Version: 7
GPU Type: Jetson nano 2Gb
Nvidia Driver Version:
CUDA Version: 10.2
CUDNN Version: 8
Operating System + Version: Jetpack 4.4.1
Python Version (if applicable): 3.6.9
TensorFlow Version (if applicable):
PyTorch Version (if applicable):
Baremetal or Container (if container which image + tag):

Relevant Files

Please attach or include links to any models, data, files, or scripts necessary to reproduce your issue. (Github repo, Google Drive, Dropbox, etc.)

Steps To Reproduce

Please include:

Exact steps/commands to build your repro
Exact steps/commands to run your repro
Full traceback of errors encountered

AakankshaS · December 28, 2020, 4:14am

Hi @chitransh.valawat,
This looks like a Jetson issue, hence recommend you to raise it to the below forum.
Jetson & Embedded Systems - NVIDIA Developer Forums
Thanks!

Topic		Replies	Views
Unable to generate tensorrt engine for facenet model on Jetson Nano 2 GB Jetson Nano tensorrt , cuda , ai-training	2	561	October 15, 2021
Make full use of the swapfile Jetson Nano tensorrt	6	890	June 2, 2023
TensorRT process killed with Orin Nano Jetson Orin Nano tensorrt	9	1346	July 27, 2023
Build TensorRT Engine killed TensorRT tensorrt , board-design	1	503	December 9, 2020
Jetson Nano 2GB, Ram problem Jetson Nano tensorrt	2	450	August 10, 2022
TensorRT Build Killed Jetson Nano tensorrt , nano2gb	4	1966	October 15, 2021
when i run a tensorflow model there is not enough memory ,what shoud i do Jetson TX2	12	7895	September 1, 2021
Jetpack 4.2.3 has a built in 2GB swap file we are not able to increase this to 4 GB Jetson Nano	14	9890	October 14, 2021
Run a UNet segmentation model on Jetson Nano / Convert pb to TensorRT Jetson Nano tensorrt	3	1769	October 18, 2021
Unable to install TensorRT on Jetson Nano Jetson Nano tensorrt	4	927	December 14, 2022