Unable to generate tensorrt engine for facenet model on Jatson Nano 2 GB

Description

Unable to generate tensorrt engine for facenet model on Jatson Nano 2 GB.


Scenario A

Tried to allocate swap memory of 6GB through script:

!

As per the “swap_area_modified_killed.png” snap, process gets killed after using 1915MB RAM area out of 1972MB total RAM and 2017MB of SWAP area from 7144MB(Total ~4GB from ~9GB)


UNKNOWN: Retargeting InceptionResnetV1/Repeat_2/block8_5/Branch_1/Conv2d_0c_3x1/Relu to InceptionResnetV1/Repeat_2/block8_5/concat
UNKNOWN: Eliminating concatenation InceptionResnetV1/Block8/concat
UNKNOWN: Generating copy for InceptionResnetV1/Block8/Branch_1/Conv2d_0a_1x1/convolution + InceptionResnetV1/Block8/Branch_1/Conv2d_0a_1x1/Relu || InceptionResnetV1/Block8/Branch_0/Conv2d_1x1/convolution + InceptionResnetV1/Block8/Branch_0/Conv2d_1x1/Relu to InceptionResnetV1/Block8/concat
UNKNOWN: Retargeting InceptionResnetV1/Block8/Branch_1/Conv2d_0c_3x1/Relu to InceptionResnetV1/Block8/concat
UNKNOWN: After concat removal: 137 layers
UNKNOWN: Graph construction and optimization completed in 1.2832 seconds.
Killed

nano@nano-desktop:~$ swapon -s
Filename Type Size Used Priority
/mnt/swapfile file 6291452 0 -1
/dev/zram0 partition 255996 0 5
/dev/zram1 partition 255996 0 5
/dev/zram2 partition 255996 0 5
/dev/zram3 partition 255996 0 5


Scenario B

Tried to allocate swap area 1GB to each CPU ZRAM:

zram_modified_killed|690x387

As per the “zram_modified_killed.png” snap, the process gets killed after using 1915MB RAM area out of 1972MB total RAM and 2017MB of SWAP area from 7144MB(Total ~4GB from ~9GB)

$ sudo tegrastats
RAM 1904/1972MB (lfb 1x1MB) SWAP 2030/40000MB (cached 0MB) IRAM 0/252kB(lfb 252kB)

nano@nano-desktop:~$ swapon -s
Filename Type Size Used Priority
/dev/zram0 partition 1023996 7056 5
/dev/zram1 partition 1023996 7052 5
/dev/zram2 partition 1023996 7068 5
/dev/zram3 partition 1023996 7064 5

In the both the scenarios, the process got killed after using ~4GB(2GB swap & 2GB RAM) even the swap area is still not fully occupied.

A clear and concise description of the bug or issue.

Environment

TensorRT Version: 7
GPU Type: Jetson nano 2Gb
Nvidia Driver Version:
CUDA Version: 10.2
CUDNN Version: 8
Operating System + Version: Jetpack 4.4.1
Python Version (if applicable): 3.6.9
TensorFlow Version (if applicable):
PyTorch Version (if applicable):
Baremetal or Container (if container which image + tag):

Relevant Files

Please attach or include links to any models, data, files, or scripts necessary to reproduce your issue. (Github repo, Google Drive, Dropbox, etc.)

Steps To Reproduce

Please include:

  • Exact steps/commands to build your repro
  • Exact steps/commands to run your repro
  • Full traceback of errors encountered

Hi @chitransh.valawat,
This looks like a Jetson issue, hence recommend you to raise it to the below forum.
https://forums.developer.nvidia.com/c/agx-autonomous-machines/jetson-embedded-systems/70
Thanks!