TRT Engine creation failure when running pose estimation deepstream sample app

Continuing the discussion from Pose Estimation with DeepStream:

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU)
Jetson Nano 4GB
• DeepStream Version
5.1
• JetPack Version (valid for Jetson only)
JETSON_JETPACK=4.5.1

• TensorRT Version
JETSON_TENSORRT=7.1.3.0

• NVIDIA GPU Driver Version (valid for GPU only)
• Issue Type: Error/Performance/Resources Issue
TRT Engine creation failure when running pose estimation deepstream sample app. The Jetson Nano terminates the process to save compute/memory resources. Blog post mentions that TRT engine creation could take between 4-10 minutes. Do I need to free up some resources for this process to complete successfully?

**• How to reproduce the issue ?

To get started, please follow these steps.

  1. Install DeepStream on your platform, verify it is working by running deepstream-app.
  2. Clone the repository preferably in $DEEPSTREAM_DIR/sources/apps/sample_apps .
  3. Download the TRTPose model, convert it to ONNX using this export utility, and set its location in the DeepStream configuration file.
  4. Replace the OSD binaries (x86 or Jetson) in $DEEPSTREAM_DIR/libs with the ones provided in this repository under bin/ . Please note that these are not inter-compatible across platforms.
  5. Compile the program
 $ cd deepstream-pose-estimation/
 $ sudo make
 $ sudo ./deepstream-pose-estimation-app <file-uri> <output-path>

username@nano:/opt/nvidia/deepstream/deepstream-5.1/sources/apps/sample_apps/deepstream_pose_estimation$ sudo $DEEPSTREAM_ROOT/sources/apps/sample_apps/deepstream_pose_estimation/deepstream-pose-estimation-app /opt/nvidia/deepstream/deepstream-5.1/samples/streams/sample_720p.mp4 /opt/nvidia/deepstream/deepstream-5.1/sources/apps/sample_apps/deepstream_pose_estimation/output/output.mp4
Now playing: /opt/nvidia/deepstream/deepstream-5.1/samples/streams/sample_720p.mp4
Opening in BLOCKING MODE
Opening in BLOCKING MODE
Opening in BLOCKING MODE
Opening in BLOCKING MODE
ERROR: Deserialize engine failed because file path: /opt/nvidia/deepstream/deepstream-5.1/sources/apps/sample_apps/deepstream_pose_estimation/pose_estimation.onnx_b1_gpu0_fp16.engine open error
0:00:04.563512431 22512 0x5581fcfed0 WARN nvinfer gstnvinfer.cpp:616:gst_nvinfer_logger: NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1691> [UID = 1]: deserialize engine from file :/opt/nvidia/deepstream/deepstream-5.1/sources/apps/sample_apps/deepstream_pose_estimation/pose_estimation.onnx_b1_gpu0_fp16.engine failed
0:00:04.563620142 22512 0x5581fcfed0 WARN nvinfer gstnvinfer.cpp:616:gst_nvinfer_logger: NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:1798> [UID = 1]: deserialize backend context from engine from file :/opt/nvidia/deepstream/deepstream-5.1/sources/apps/sample_apps/deepstream_pose_estimation/pose_estimation.onnx_b1_gpu0_fp16.engine failed, try rebuild
0:00:04.563654518 22512 0x5581fcfed0 INFO nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger: NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1716> [UID = 1]: Trying to create engine from model files

Input filename: /opt/nvidia/deepstream/deepstream-5.1/sources/apps/sample_apps/deepstream_pose_estimation/pose_estimation.onnx
ONNX IR version: 0.0.6
Opset version: 9
Producer name: pytorch
Producer version: 1.8
Domain:
Model version: 0
Doc string:

it’s building TensorRT engine from the onnx file, you need wait for some time, maybe 15 ~ 20 mins…

The Jetson Nano is resource constrained. I am running XFCE desktop and have 6GB Swap File, but the the Nano System still killed/terminated the process. Options?

username@nano:/usr/local$ sudo jetson_clocks --show
SOC family:tegra210 Machine:NVIDIA Jetson Nano Developer Kit
Online CPUs: 0-3
cpu0: Online=1 Governor=schedutil MinFreq=1479000 MaxFreq=1479000 CurrentFreq=1479000 IdleStates: WFI=0 c7=0
cpu1: Online=1 Governor=schedutil MinFreq=1479000 MaxFreq=1479000 CurrentFreq=1479000 IdleStates: WFI=0 c7=0
cpu2: Online=1 Governor=schedutil MinFreq=1479000 MaxFreq=1479000 CurrentFreq=1479000 IdleStates: WFI=0 c7=0
cpu3: Online=1 Governor=schedutil MinFreq=1479000 MaxFreq=1479000 CurrentFreq=1479000 IdleStates: WFI=0 c7=0
GPU MinFreq=921600000 MaxFreq=921600000 CurrentFreq=921600000
EMC MinFreq=204000000 MaxFreq=1600000000 CurrentFreq=1600000000 FreqOverride=1
Fan: PWM=0
NV Power Mode: MAXN
--------------------------------------------------

6> [UID = 1]: Trying to create engine from model files
----------------------------------------------------------------
Input filename: /opt/nvidia/deepstream/deepstream-5.1/sources/apps/sample_apps/deepstream_pose_estimation/pose_estimation.onnx
ONNX IR version: 0.0.6
Opset version: 9
Producer name: pytorch
Producer version: 1.8
Domain:
Model version: 0
Doc string:
----------------------------------------------------------------
Killed

after it’s terminated , could you check the kernel log by command - “sudo dmesg” and check if it’s OOM.

It looks like an OOM issue. I ran task manager and the process nears/reaches max available Memory (3956MB), the system terminates the process. I am not sure if the process is using/leveraging available swap memory. Is there a way to verify? Should I use a USB flash drive for swap instead of the SD Card?

username@nano:~$ free -h
total used free shared buff/cache available
Mem: 3.9G 1.6G 1.1G 152M 1.2G 2.0G
Swap: 9.9G 0B 9.9G

username@nano:/opt/nvidia/deepstream/deepstream-5.1/sources/apps/sample_apps/deepstream_pose_estimation$ sudo jetson_clocks --show
SOC family:tegra210 Machine:NVIDIA Jetson Nano Developer Kit
Online CPUs: 0-3
cpu0: Online=1 Governor=schedutil MinFreq=102000 MaxFreq=1479000 CurrentFreq=1479000 IdleStates: WFI=0 c7=0
cpu1: Online=1 Governor=schedutil MinFreq=102000 MaxFreq=1479000 CurrentFreq=1479000 IdleStates: WFI=0 c7=0
cpu2: Online=1 Governor=schedutil MinFreq=102000 MaxFreq=1479000 CurrentFreq=1428000 IdleStates: WFI=0 c7=0
cpu3: Online=1 Governor=schedutil MinFreq=102000 MaxFreq=1479000 CurrentFreq=1479000 IdleStates: WFI=0 c7=0
GPU MinFreq=76800000 MaxFreq=921600000 CurrentFreq=230400000
EMC MinFreq=204000000 MaxFreq=1600000000 CurrentFreq=1600000000 FreqOverride=1
Fan: PWM=0
NV Power Mode: MAXN

-------------------------------------------------------
username@nano:/opt/nvidia/deepstream/deepstream-5.1/sources/apps/sample_apps/deepstream_pose_estimation$ sudo dmesg

[ 1256.804067] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[ 1256.804069] 4388 total pagecache pages
[ 1256.804072] 69 pages in swap cache
[ 1256.804075] Swap cache stats: add 372168, delete 372180, find 35/2993
[ 1256.804077] Free swap = 8948344kB
[ 1256.804079] Total swap = 10414140kB
[ 1256.804081] 1039872 pages RAM
[ 1256.804082] 0 pages HighMem/MovableOnly
[ 1256.804084] 27102 pages reserved
[ 1256.804086] 118784 pages cma reserved
[ 1256.804088] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
[ 1256.804114] [ 2008] 0 2008 10699 2 23 3 245 0 systemd-journal
[ 1256.804119] [ 2157] 0 2157 4030 0 9 4 547 -1000 systemd-udevd
[ 1256.804125] [ 3724] 0 3724 1920 0 7 3 869 0 haveged
[ 1256.804130] [ 3753] 62583 3753 20948 0 11 4 139 0 systemd-timesyn
[ 1256.804135] [ 3756] 102 3756 2627 0 9 4 186 0 systemd-resolve
[ 1256.804139] [ 3757] 0 3757 1602 0 7 3 108 0 rpcbind
[ 1256.804144] [ 3918] 109 3918 54993 0 12 4 387 0 rsyslogd
[ 1256.804160] [ 3919] 0 3919 24415 3 17 3 2318 0 jtop
[ 1256.804164] [ 3951] 103 3951 2115 2 8 4 547 -900 dbus-daemon
[ 1256.804168] [ 4058] 0 4058 100558 0 30 4 1027 0 NetworkManager
[ 1256.804173] [ 4059] 0 4059 96924 3 23 4 1346 0 udisksd
[ 1256.804177] [ 4071] 0 4071 58559 3 17 4 302 0 accounts-daemon
[ 1256.804181] [ 4092] 0 4092 25152 0 18 3 2096 0 networkd-dispat
[ 1256.804193] [ 4094] 114 4094 1538 2 7 4 126 0 avahi-daemon
[ 1256.804197] [ 4153] 0 4153 78225 3 21 4 400 0 ModemManager
[ 1256.804202] [ 4162] 0 4162 2588 2 9 3 173 0 systemd-logind
[ 1256.804206] [ 4171] 0 4171 2415 0 8 4 238 0 wpa_supplicant
[ 1256.804210] [ 4209] 0 4209 1660 3 7 4 69 0 cron
[ 1256.804214] [ 4236] 0 4236 285975 0 53 6 5446 -900 snapd
[ 1256.804218] [ 4368] 114 4368 1491 0 7 4 85 0 avahi-daemon
[ 1256.804222] [ 4483] 0 4483 59230 0 17 4 821 0 polkitd
[ 1256.804228] [ 4787] 111 4787 61950 3 22 3 948 0 whoopsie
[ 1256.804232] [ 4828] 0 4828 277231 0 59 6 7060 -999 containerd
[ 1256.804236] [ 4916] 105 4916 2371 0 8 4 114 0 kerneloops
[ 1256.804241] [ 4959] 105 4959 2371 0 8 3 115 0 kerneloops
[ 1256.804245] [ 5024] 0 5024 1318 0 7 3 33 0 agetty
[ 1256.804250] [ 5042] 0 5042 2605 0 9 3 182 -1000 sshd
[ 1256.804254] [ 5052] 0 5052 1318 0 8 4 36 0 agetty
[ 1256.804259] [ 5054] 0 5054 5375 0 8 3 69 0 nvs-service
[ 1256.804263] [ 5065] 0 5065 1654 3 8 4 76 0 nvmemwarning.sh
[ 1256.804268] [ 5216] 0 5216 25082 0 32 3 1247 0 nvargus-daemon
[ 1256.804272] [ 5229] 0 5229 1654 0 7 3 76 0 nvgetty.sh
[ 1256.804276] [ 5333] 1000 5333 13959 0 20 4 3358 0 Xtightvnc
[ 1256.804281] [ 5350] 0 5350 1318 0 6 3 36 0 getty
[ 1256.804285] [ 5357] 0 5357 58903 4 15 4 293 0 gdm3
[ 1256.804290] [ 5412] 0 5412 40799 4 16 4 320 0 gdm-session-wor
[ 1256.804294] [ 5477] 0 5477 432864 0 94 7 10108 -500 dockerd
[ 1256.804298] [ 5547] 0 5547 786 0 5 3 37 0 nvphsd
[ 1256.804302] [ 5548] 0 5548 3065 0 6 3 92 0 nvphsd
[ 1256.804306] [ 5575] 0 5575 1811 0 7 3 145 0 bluetoothd
[ 1256.804310] [ 5788] 120 5788 3177 2 10 4 351 0 systemd
[ 1256.804315] [ 5811] 120 5811 3914 0 12 3 579 0 (sd-pam)
[ 1256.804319] [ 5985] 0 5985 61248 0 19 4 2362 0 jtop
[ 1256.804323] [ 5992] 1000 5992 61436 6 22 4 393 0 lxsession
[ 1256.804328] [ 6004] 0 6004 42848 2 17 4 2333 0 jtop
[ 1256.804332] [ 6055] 1000 6055 1617 0 7 3 99 0 dbus-launch
[ 1256.804336] [ 6081] 120 6081 39900 4 13 3 168 0 gdm-x-session
[ 1256.804340] [ 6084] 120 6084 1609195 388 38 5 3107 0 Xorg
[ 1256.804345] [ 6139] 1000 6139 1744 2 8 3 203 0 dbus-daemon
[ 1256.804349] [ 6200] 1000 6200 58785 4 16 3 738 0 gvfsd
[ 1256.804353] [ 6207] 1000 6207 77873 0 15 4 732 0 gvfsd-fuse
[ 1256.804357] [ 6232] 1000 6232 14322 4 19 4 749 0 openbox
[ 1256.804361] [ 6244] 1000 6244 45280 0 24 3 1238 0 lxpolkit
[ 1256.804365] [ 6246] 1000 6246 197310 124 44 3 2298 0 lxpanel
[ 1256.804369] [ 6248] 120 6248 1799 2 8 4 232 0 dbus-daemon
[ 1256.804373] [ 6249] 1000 6249 129763 0 40 3 3619 0 pcmanfm
[ 1256.804378] [ 6252] 120 6252 100495 6 29 4 596 0 gnome-session-b
[ 1256.804382] [ 6255] 1000 6255 2472 4 8 3 182 0 xscreensaver
[ 1256.804386] [ 6272] 1000 6272 979 0 6 3 75 0 ssh-agent
[ 1256.804390] [ 6274] 120 6274 76072 0 15 4 193 0 at-spi-bus-laun
[ 1256.804394] [ 6278] 1000 6278 76159 0 16 3 705 0 at-spi-bus-laun
[ 1256.804399] [ 6286] 120 6286 1684 0 8 3 119 0 dbus-daemon
[ 1256.804403] [ 6292] 1000 6292 147995 6 53 5 1613 0 deja-dup-monito
[ 1256.804407] [ 6293] 1000 6293 1651 0 7 4 112 0 dbus-daemon
[ 1256.804411] [ 6295] 120 6295 39810 0 12 5 201 0 at-spi2-registr
[ 1256.804415] [ 6298] 1000 6298 123670 0 42 3 1985 0 nm-applet
[ 1256.804420] [ 6312] 1000 6312 45643 0 25 3 660 0 xfce4-power-man
[ 1256.804423] [ 6316] 1000 6316 39774 0 15 4 702 0 at-spi2-registr
[ 1256.804427] [ 6319] 1000 6319 65997 6 29 4 1555 0 notify-osd
[ 1256.804432] [ 6325] 1000 6325 102893 0 35 3 1666 0 update-notifier
[ 1256.804436] [ 6350] 1000 6350 38829 3 10 4 197 0 menu-cached
[ 1256.804440] [ 6358] 1000 6358 2417 3 9 3 128 0 xfconfd
[ 1256.804444] [ 6368] 1000 6368 133654 0 49 3 4164 0 python3
[ 1256.804448] [ 6374] 1000 6374 155678 0 36 4 1178 0 zeitgeist-datah
[ 1256.804453] [ 6411] 1000 6411 478 0 5 3 22 0 sh
[ 1256.804457] [ 6414] 120 6414 662483 1413 133 5 16252 0 gnome-shell
[ 1256.804462] [ 6422] 0 6422 61885 0 20 3 390 0 upowerd
[ 1256.804478] [ 6457] 1000 6457 76925 4 17 4 265 0 zeitgeist-daemo
[ 1256.804483] [ 6465] 1000 6465 245115 0 40 5 1888 0 pulseaudio
[ 1256.804487] [ 6486] 106 6486 38092 0 9 4 92 0 rtkit-daemon
[ 1256.804491] [ 6623] 1000 6623 58957 0 16 3 543 0 gvfs-udisks2-vo
[ 1256.804495] [ 6636] 1000 6636 64783 0 29 4 1231 0 gnome-screensav
[ 1256.804500] [ 6694] 1000 6694 57958 0 14 3 179 0 gvfs-goa-volume
[ 1256.804510] [ 6734] 1000 6734 129118 6 49 5 1510 0 goa-daemon
[ 1256.804515] [ 6779] 0 6779 2093 2 9 3 324 0 dhclient
[ 1256.804519] [ 6803] 1000 6803 59117 0 18 4 288 0 goa-identity-se
[ 1256.804523] [ 6815] 1000 6815 124777 0 39 3 3868 0 python3
[ 1256.804527] [ 6816] 1000 6816 124777 0 39 3 3868 0 python3
[ 1256.804531] [ 6869] 1000 6869 124780 0 39 3 3874 0 python3
[ 1256.804535] [ 6871] 1000 6871 124780 0 39 3 3874 0 python3
[ 1256.804539] [ 6877] 1000 6877 124780 0 39 3 3876 0 python3
[ 1256.804543] [ 6879] 1000 6879 124780 0 39 3 3874 0 python3
[ 1256.804547] [ 6887] 1000 6887 124780 0 39 3 3875 0 python3
[ 1256.804552] [ 6893] 1000 6893 77987 0 19 4 273 0 gvfs-afc-volume
[ 1256.804556] [ 6975] 1000 6975 58428 4 15 3 229 0 gvfs-gphoto2-vo
[ 1256.804560] [ 7001] 1000 7001 57929 4 14 4 210 0 gvfs-mtp-volume
[ 1256.804565] [ 7069] 1000 7069 77317 0 17 4 301 0 gvfsd-trash
[ 1256.804569] [ 7109] 120 7109 76790 4 18 5 350 0 ibus-daemon
[ 1256.804573] [ 7124] 120 7124 58139 0 14 4 181 0 ibus-dconf
[ 1256.804577] [ 7127] 120 7127 45758 1 25 4 1160 0 ibus-x11
[ 1256.804581] [ 7129] 120 7129 58083 0 15 3 171 0 ibus-portal
[ 1256.804586] [ 7142] 0 7142 2093 2 9 4 329 0 dhclient
[ 1256.804590] [ 7218] 120 7218 329127 0 44 5 2467 0 pulseaudio
[ 1256.804595] [ 7255] 0 7255 58586 4 18 3 222 0 boltd
[ 1256.804599] [ 7292] 0 7292 68461 0 24 4 497 0 packagekitd
[ 1256.804604] [ 7328] 120 7328 82954 0 29 4 1840 0 gsd-xsettings
[ 1256.804608] [ 7331] 120 7331 58042 0 14 3 177 0 gsd-a11y-settin
[ 1256.804612] [ 7337] 120 7337 45665 6 26 3 1161 0 gsd-clipboard
[ 1256.804616] [ 7339] 120 7339 103863 5 36 5 1335 0 gsd-color
[ 1256.804620] [ 7343] 120 7343 53616 0 26 4 477 0 gsd-datetime
[ 1256.804625] [ 7347] 120 7347 58155 0 15 4 164 0 gsd-housekeepin
[ 1256.804629] [ 7349] 120 7349 82939 0 30 3 1249 0 gsd-keyboard
[ 1256.804633] [ 7353] 120 7353 167907 6 36 4 1359 0 gsd-media-keys
[ 1256.804637] [ 7354] 120 7354 39462 0 12 3 149 0 gsd-mouse
[ 1256.804641] [ 7356] 120 7356 83102 0 30 3 1277 0 gsd-power
[ 1256.804645] [ 7359] 120 7359 42283 0 18 5 304 0 gsd-print-notif
[ 1256.804649] [ 7361] 120 7361 39466 0 12 4 146 0 gsd-rfkill
[ 1256.804653] [ 7364] 120 7364 57896 0 15 3 157 0 gsd-screensaver
[ 1256.804657] [ 7371] 120 7371 59681 0 18 5 282 0 gsd-sharing
[ 1256.804661] [ 7376] 120 7376 96082 0 21 4 266 0 gsd-smartcard
[ 1256.804665] [ 7379] 120 7379 59827 0 19 3 1300 0 gsd-sound
[ 1256.804669] [ 7382] 120 7382 64354 0 28 3 1230 0 gsd-wacom
[ 1256.804673] [ 7392] 116 7392 60357 4 20 4 1219 0 colord
[ 1256.804678] [ 7452] 120 7452 38709 3 10 4 158 0 dconf-service
[ 1256.804681] [ 7471] 120 7471 39669 0 12 3 177 0 ibus-engine-sim
[ 1256.804686] [ 7497] 0 7497 40860 4 15 4 351 0 gdm-session-wor
[ 1256.804690] [ 7507] 1000 7507 3175 2 10 4 359 0 systemd
[ 1256.804694] [ 7508] 1000 7508 3974 0 12 3 638 0 (sd-pam)
[ 1256.804698] [ 7521] 1000 7521 58865 0 15 4 284 0 gnome-keyring-d
[ 1256.804702] [ 7532] 1000 7532 39900 4 13 4 168 0 gdm-x-session
[ 1256.804706] [ 7534] 1000 7534 1643918 467 47 5 3790 0 Xorg
[ 1256.804710] [ 7545] 1000 7545 1847 2 7 3 279 0 dbus-daemon
[ 1256.804714] [ 7548] 1000 7548 568 1 5 3 33 0 sh
[ 1256.804718] [ 7690] 1000 7690 58786 4 15 3 233 0 gvfsd
[ 1256.804722] [ 7695] 1000 7695 94257 0 16 4 226 0 gvfsd-fuse
[ 1256.804726] [ 7697] 1000 7697 39546 0 12 3 149 0 gvfsd-metadata
[ 1256.805277] [ 7754] 1000 7754 1046 0 7 3 92 0 ssh-agent
[ 1256.805284] [ 7772] 1000 7772 44434 0 23 3 1055 0 xfce4-session
[ 1256.805289] [ 7773] 1000 7773 76100 0 16 5 196 0 at-spi-bus-laun
[ 1256.805293] [ 7778] 1000 7778 1740 2 7 3 154 0 dbus-daemon
[ 1256.805298] [ 7780] 1000 7780 39800 0 12 3 200 0 at-spi2-registr
[ 1256.805302] [ 7784] 1000 7784 2498 0 8 4 169 0 xfconfd
[ 1256.805306] [ 7793] 1000 7793 9116 0 22 4 1393 0 xfwm4
[ 1256.805310] [ 7797] 1000 7797 55370 6 31 3 2218 0 xfce4-panel
[ 1256.805314] [ 7799] 1000 7799 44692 0 23 4 582 0 Thunar
[ 1256.805318] [ 7801] 1000 7801 113797 0 57 4 9846 0 xfdesktop
[ 1256.805322] [ 7802] 1000 7802 52496 0 37 4 1818 0 polkit-gnome-au
[ 1256.805326] [ 7803] 1000 7803 52757 5 28 3 785 0 xfsettingsd
[ 1256.805330] [ 7805] 1000 7805 148019 6 53 4 2083 0 deja-dup-monito
[ 1256.805334] [ 7808] 1000 7808 155704 3 36 3 643 0 zeitgeist-datah
[ 1256.805338] [ 7815] 1000 7815 104788 0 38 3 1914 0 nm-applet
[ 1256.805342] [ 7819] 1000 7819 102470 0 34 5 1548 0 update-notifier
[ 1256.805346] [ 7822] 1000 7822 46109 0 27 3 1298 0 xfce4-power-man
[ 1256.805351] [ 7824] 1000 7824 245332 0 42 5 1898 0 pulseaudio
[ 1256.805355] [ 7830] 1000 7830 2543 0 9 4 192 0 xscreensaver
[ 1256.805359] [ 7832] 1000 7832 110271 5 25 3 575 0 xfce4-volumed
[ 1256.805363] [ 7833] 1000 7833 95413 4 19 3 785 0 zeitgeist-daemo
[ 1256.805367] [ 7837] 1000 7837 128238 0 50 4 5818 0 blueman-applet
[ 1256.805370] [ 7847] 1000 7847 58984 0 16 3 539 0 gvfs-udisks2-vo
[ 1256.805374] [ 7848] 1000 7848 31842 4 32 3 5336 0 applet.py
[ 1256.805378] [ 7864] 1000 7864 66618 0 18 5 303 0 zeitgeist-fts
[ 1256.805382] [ 7895] 1000 7895 64340 0 27 3 1211 0 notify-osd
[ 1256.805386] [ 7897] 1000 7897 57983 4 14 3 181 0 gvfs-goa-volume
[ 1256.805465] [ 7910] 1000 7910 129141 6 52 4 1510 0 goa-daemon
[ 1256.805470] [ 7921] 1000 7921 7328 0 18 4 509 0 panel-6-systray
[ 1256.805474] [ 7923] 1000 7923 59142 0 17 4 292 0 goa-identity-se
[ 1256.805478] [ 7925] 1000 7925 7880 0 20 3 606 0 panel-2-actions
[ 1256.805482] [ 7926] 1000 7926 78010 0 20 5 274 0 gvfs-afc-volume
[ 1256.805486] [ 7934] 1000 7934 58451 0 15 4 740 0 gvfs-gphoto2-vo
[ 1256.805490] [ 7937] 1000 7937 64337 0 29 3 1237 0 gnome-screensav
[ 1256.805494] [ 7941] 1000 7941 57952 0 14 5 183 0 gvfs-mtp-volume
[ 1256.805498] [ 7986] 1000 7986 10158 0 13 3 171 0 obexd
[ 1256.805503] [ 7989] 1000 7989 77317 0 17 4 285 0 gvfsd-trash
[ 1256.805507] [ 8030] 1000 8030 105267 0 38 5 2344 0 gnome-terminal-
[ 1256.805511] [ 8037] 1000 8037 2203 3 8 3 577 0 bash
[ 1256.805515] [ 8500] 1000 8500 45790 0 26 4 1322 0 lxtask
[ 1256.805520] [ 9223] 0 9223 1190 0 7 4 21 0 sleep
[ 1256.805525] [ 9238] 0 9238 2248 3 8 3 127 0 sudo
[ 1256.805529] [ 9239] 0 9239 2480523 32961 667 6 171508 0 deepstream-pose
[ 1256.805534] Out of memory: Kill process 9239 (deepstream-pose) score 55 or sacrifice child
[ 1256.813882] Killed process 9239 (deepstream-pose) total-vm:9922092kB, anon-rss:0kB, file-rss:131840kB
[ 1257.156417] oom_reaper: reaped process 9239 (deepstream-pose), now anon-rss:0kB, file-rss:133228kB, shmem-rss:0kB

Maybe you could try to create swap on drive… I suspect it does not work, but I’m not sure…

Or, you could try to remove some applications like below (the data below is not from NANO) to have more free memory.

  1. On system freshly installed by SDKManager, after system just boots up, system consumes ~ 849MB (= MemTotal - MemAvailable = 3924576 KB - 3054852 KB )
# cat /proc/meminfo
MemTotal:        3924576 kB
MemFree:          484128 kB
MemAvailable:    3054852 kB
  1. After removing below packages, system memory consumption reducs to ~ 362 MB (= 3924576 KB - 3554064 KB )

sudo apt autoremove sudo apt-get purge gnome-shell // disable ubuntu desktop
sudo apt-get purge containerd // remove Jetson docker container runtime sudo apt-get purge python
// there are still some other applications that can be purged to save memory.

# cat /proc/meminfo
MemTotal:        3924576 kB
MemFree:         3506544 kB
MemAvailable:    3554064 kB

How many memory the application consume are close to ’ Pss ’ of procrank.