Unable to run python sample app deepstream_ssd_parser.py

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) Xavier NX
• DeepStream Version DS5.0dp
• JetPack Version (valid for Jetson only) 4.4

I’m trying to run the SSD python sample app that uses the Triton inference engine:

LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libgomp.so.1 python3 deepstream_ssd_parser.py /opt/nvidia/deepstream/deepstream-5.0/samples/streams/sample_720p.h264

However after the pipeline is created I just see a log output saying KILLED. I have already performed the steps in the deepstream README to make install sources/libs/nvdsinfer_customparser and execute ./prepare_ds_trtis_model_repo.sh

Here is the output:

jason@xavier:/opt/nvidia/deepstream/deepstream-5.0/sources/python/apps/deepstream-ssd-parser$ LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libgomp.so.1 python3 deepstream_ssd_parser.py /opt/nvidia/deepstream/deepstream-5.0/samples/streams/sample_720p.h264 
2020-05-28 11:16:14.789651: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.2
Creating Pipeline 
 
Creating Source
Creating H264Parser
Creating Decoder
Creating NvStreamMux
Creating Nvinferserver
2020-05-28 11:16:17.531846: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.2
Creating Nvvidconv
Creating OSD (nvosd)
Creating Queue
Creating Converter 2 (nvvidconv2)
Creating capsfilter
Creating Encoder
Creating Code Parser
Creating Container
Creating Sink
Playing file /opt/nvidia/deepstream/deepstream-5.0/samples/streams/sample_720p.h264 
Adding elements to Pipeline 

Linking elements in the Pipeline 

Starting pipeline 

Opening in BLOCKING MODE 
I0528 03:16:18.142904 26365 server.cc:120] Initializing Triton Inference Server
I0528 03:16:18.162633 26365 server_status.cc:55] New status tracking for model 'ssd_inception_v2_coco_2018_01_28'
I0528 03:16:18.162842 26365 model_repository_manager.cc:680] loading: ssd_inception_v2_coco_2018_01_28:1
I0528 03:16:18.163660 26365 base_backend.cc:176] Creating instance ssd_inception_v2_coco_2018_01_28_0_0_gpu0 on GPU 0 (7.2) using model.graphdef
2020-05-28 11:16:18.252795: W tensorflow/core/platform/profile_utils/cpu_utils.cc:98] Failed to find bogomips in /proc/cpuinfo; cannot determine CPU frequency
2020-05-28 11:16:18.254292: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x265cf560 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-05-28 11:16:18.254425: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2020-05-28 11:16:18.254739: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcuda.so.1
2020-05-28 11:16:18.255038: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-05-28 11:16:18.255244: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: 
name: Xavier major: 7 minor: 2 memoryClockRate(GHz): 1.109
pciBusID: 0000:00:00.0
2020-05-28 11:16:18.255413: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.2
2020-05-28 11:16:18.255578: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2020-05-28 11:16:18.338158: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2020-05-28 11:16:18.455470: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2020-05-28 11:16:18.600896: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2020-05-28 11:16:18.682709: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2020-05-28 11:16:18.682996: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.8
2020-05-28 11:16:18.683404: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-05-28 11:16:18.684015: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-05-28 11:16:18.684417: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0
2020-05-28 11:16:28.657287: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-05-28 11:16:28.657420: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186]      0 
2020-05-28 11:16:28.657467: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0:   N 
2020-05-28 11:16:28.657798: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-05-28 11:16:28.658142: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-05-28 11:16:28.658388: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-05-28 11:16:28.658671: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4658 MB memory) -> physical GPU (device: 0, name: Xavier, pci bus id: 0000:00:00.0, compute capability: 7.2)
2020-05-28 11:16:28.666698: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7ec85d2390 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-05-28 11:16:28.666839: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Xavier, Compute Capability 7.2
I0528 03:16:30.540033 26365 model_repository_manager.cc:837] successfully loaded 'ssd_inception_v2_coco_2018_01_28' version 1
INFO: TrtISBackend id:5 initialized model: ssd_inception_v2_coco_2018_01_28
Killed

I am running this from the GUI and with Chrome and other apps closed except for a couple of terminal windows.

Hi
Seems your program killed by ubuntu os, hope this helps

Yes, I can repo the issue on the same setup (NX. DS5.0, JP4.4), and from the kernel log, the process is killed by OOM

$ dmesg

[ 12.023700] fuse init (API version 7.26)
[ 262.250505] kworker/u12:1 invoked oom-killer: gfp_mask=0x27080c0(GFP_KERNEL_ACCOUNT|__GFP_ZERO|__GFP_NOTRACK), nodemask=0, order=2, oom_score_adj=0
[ 262.250835] kworker/u12:1 cpuset=/ mems_allowed=0
[ 262.250866] CPU: 0 PID: 46 Comm: kworker/u12:1 Not tainted 4.9.140-tegra #1
[ 262.250870] Hardware name: NVIDIA Jetson Xavier NX Developer Kit (DT)
[ 262.250893] Workqueue: events_unbound call_usermodehelper_exec_work
[ 262.250902] Call trace:
[ 262.250913] [] dump_backtrace+0x0/0x198
[ 262.250921] [] show_stack+0x24/0x30
[ 262.250930] [] dump_stack+0x98/0xc0
[ 262.250938] [] dump_header+0x6c/0x1b8
[ 262.250954] [] oom_kill_process+0x268/0x498
[ 262.250972] [] out_of_memory+0x1e4/0x308
[ 262.250988] [] __alloc_pages_nodemask+0x844/0xd38
[ 262.250999] [] copy_process.isra.5.part.6+0x110/0x1530
[ 262.251005] [] _do_fork+0xd4/0x460
[ 262.251011] [] kernel_thread+0x48/0x58
[ 262.251019] [] call_usermodehelper_exec_work+0x34/0xd0
[ 262.251026] [] process_one_work+0x1e4/0x4b0
[ 262.251033] [] worker_thread+0x50/0x4c8
[ 262.251040] [] kthread+0xec/0xf0
[ 262.251046] [] ret_from_fork+0x10/0x30
[ 262.251060] Mem-Info:
[ 262.251071] active_anon:32 inactive_anon:108 isolated_anon:0
active_file:429 inactive_file:683 isolated_file:0
unevictable:3038 dirty:0 writeback:0 unstable:0
slab_reclaimable:11258 slab_unreclaimable:24902
mapped:407 shmem:0 pagetables:3709 bounce:0
free:17130 free_pcp:104 free_cma:1167
[ 262.251093] Node 0 active_anon:128kB inactive_anon:432kB active_file:1716kB inactive_file:2732kB unevictable:12152kB isolated(anon):0kB isolated(file):0kB mapped:1628kB dirty:0kB writeback:0kB shmem:0kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB pages_scanned:64 all_unreclaimable? no
[ 262.251119] DMA free:32280kB min:10196kB low:12744kB high:15292kB active_anon:12kB inactive_anon:44kB active_file:256kB inactive_file:264kB unevictable:0kB writepending:0kB present:1837812kB managed:1812080kB mlocked:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:220kB local_pcp:0kB free_cma:4668kB
[ 262.251122] lowmem_reserve: 0 6000 6000 6000
[ 262.251162] Normal free:36240kB min:34856kB low:43568kB high:52280kB active_anon:116kB inactive_anon:44kB active_file:1544kB inactive_file:2468kB unevictable:12152kB writepending:0kB present:6291456kB managed:6144748kB mlocked:0kB slab_reclaimable:45032kB slab_unreclaimable:99608kB kernel_stack:7856kB pagetables:14836kB bounce:0kB free_pcp:456kB local_pcp:0kB free_cma:0kB
[ 262.251166] lowmem_reserve: 0 0 0 0
[ 262.251185] DMA: 254kB (UMC) 68kB (UMC) 716kB (UM) 432kB (UM) 764kB (MC) 4128kB (M) 5256kB (UM) 7512kB (UMC) 41024kB (UM) 72048kB (UM) 24096kB (MC) = 32836kB
[ 262.251381] Normal: 418
4kB (UME) 988kB (UME) 96116kB (UM) 44032kB (UM) 4364kB (UM) 9128kB (U) 3256kB (U) 0512kB 01024kB 02048kB 04096kB = 36584kB
[ 262.251446] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[ 262.251450] 4256 total pagecache pages
[ 262.251455] 0 pages in swap cache
[ 262.251459] Swap cache stats: add 802136, delete 802126, find 1289/47678
[ 262.251463] Free swap = 1115016kB
[ 262.251467] Total swap = 3978408kB
[ 262.251471] 2032317 pages RAM
[ 262.251475] 0 pages HighMem/MovableOnly
[ 262.251479] 43110 pages reserved
[ 262.251483] 188416 pages cma reserved
[ 262.251487] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
[ 262.251539] [ 2344] 0 2344 4547 26 13 3 213 0 systemd-journal
[ 262.251553] [ 3059] 0 3059 4052 0 7 3 562 -1000 systemd-udevd
[ 262.251565] [ 3274] 62583 3274 20989 2 11 4 162 0 systemd-timesyn
[ 262.251579] [ 3279] 0 3279 1957 2 7 3 819 0 haveged
[ 262.251593] [ 3310] 0 3310 1601 0 7 3 113 0 rpcbind
[ 262.251603] [ 3321] 102 3321 2641 0 10 4 174 0 systemd-resolve
[ 262.251614] [ 4479] 0 4479 1660 0 7 4 91 0 cron
[ 262.251624] [ 4495] 0 4495 78217 0 20 3 419 0 ModemManager
[ 262.251635] [ 4524] 0 4524 25081 0 18 3 2028 0 networkd-dispat
[ 262.251646] [ 4557] 103 4557 2033 2 8 3 391 -900 dbus-daemon
[ 262.251657] [ 4702] 0 4702 2486 3 9 4 283 0 wpa_supplicant
[ 262.251668] [ 4707] 0 4707 119794 0 34 5 1360 0 NetworkManager
[ 262.251679] [ 4711] 0 4711 59324 0 19 4 355 0 accounts-daemon
[ 262.251690] [ 4724] 0 4724 97384 2 26 4 1173 0 udisksd
[ 262.251737] [ 4732] 0 4732 2646 0 10 4 224 0 systemd-logind
[ 262.251752] [ 4742] 114 4742 1536 0 8 3 136 0 avahi-daemon
[ 262.251760] [ 4773] 109 4773 54992 0 13 4 355 0 rsyslogd
[ 262.251768] [ 4834] 114 4834 1490 0 8 3 84 0 avahi-daemon
[ 262.251777] [ 4955] 0 4955 59197 2 18 4 707 0 polkitd
[ 262.251787] [ 5419] 111 5419 62055 2 23 3 459 0 whoopsie
[ 262.251795] [ 5508] 0 5508 5376 0 8 3 69 0 nvs-service
[ 262.251803] [ 5514] 0 5514 1654 0 7 4 82 0 nvmemwarning.sh
[ 262.251816] [ 5533] 0 5533 287341 0 58 8 5380 0 containerd
[ 262.251824] [ 5563] 105 5563 2368 0 8 4 126 0 kerneloops
[ 262.251852] [ 5578] 0 5578 2603 0 9 4 180 -1000 sshd
[ 262.251862] [ 5611] 105 5611 2368 2 9 3 115 0 kerneloops
[ 262.251870] [ 5764] 0 5764 25219 0 31 3 1247 0 nvargus-daemon
[ 262.251878] [ 5853] 0 5853 58868 0 15 3 276 0 gdm3
[ 262.251886] [ 5894] 0 5894 1318 0 7 4 37 0 agetty
[ 262.251893] [ 5910] 0 5910 40797 2 17 3 383 0 gdm-session-wor
[ 262.251900] [ 6028] 0 6028 787 0 4 3 38 0 nvphsd
[ 262.251906] [ 6030] 0 6030 3046 0 6 3 88 0 nvphsd
[ 262.251914] [ 6141] 0 6141 1654 0 7 3 76 0 nvgetty.sh
[ 262.251922] [ 6172] 0 6172 1318 0 6 4 36 0 getty
[ 262.251929] [ 6207] 0 6207 2094 2 9 4 321 0 dhclient
[ 262.251935] [ 6284] 0 6284 1775 0 7 3 103 0 bluetoothd
[ 262.251942] [ 6301] 120 6301 3185 2 11 4 359 0 systemd
[ 262.251948] [ 6331] 120 6331 4018 0 11 4 670 0 (sd-pam)
[ 262.251955] [ 6650] 0 6650 4378 0 13 4 2417 0 dhcpd
[ 262.251962] [ 6656] 120 6656 39897 2 12 4 167 0 gdm-x-session
[ 262.251969] [ 6671] 120 6671 6327632 436 37 4 2843 0 Xorg
[ 262.251976] [ 6732] 120 6732 1764 2 7 4 222 0 dbus-daemon
[ 262.251982] [ 6736] 120 6736 101008 1 28 3 606 0 gnome-session-b
[ 262.251989] [ 6739] 120 6739 76965 2 18 5 750 0 at-spi-bus-laun
[ 262.251996] [ 6742] 120 6742 58779 2 16 3 250 0 gvfsd
[ 262.252003] [ 6747] 120 6747 78612 0 17 5 274 0 gvfsd-fuse
[ 262.252010] [ 6751] 120 6751 1682 2 7 4 120 0 dbus-daemon
[ 262.252016] [ 6777] 120 6777 39804 0 13 3 204 0 at-spi2-registr
[ 262.252024] [ 6915] 120 6915 782055 1275 138 7 14707 0 gnome-shell
[ 262.252030] [ 6922] 0 6922 61867 0 19 5 331 0 upowerd
[ 262.252037] [ 6977] 120 6977 255406 2 23 4 1816 0 pulseaudio
[ 262.252044] [ 6978] 106 6978 38091 0 9 4 95 0 rtkit-daemon
[ 262.252051] [ 7060] 120 7060 77520 2 19 5 372 0 ibus-daemon
[ 262.252058] [ 7063] 120 7063 58837 0 16 4 209 0 ibus-dconf
[ 262.252065] [ 7066] 120 7066 46285 2 28 3 1180 0 ibus-x11
[ 262.252072] [ 7068] 120 7068 58814 0 16 4 201 0 ibus-portal
[ 262.252078] [ 7121] 0 7121 59434 0 17 3 267 0 boltd
[ 262.252085] [ 7135] 0 7135 68645 0 26 4 464 0 packagekitd
[ 262.252092] [ 7175] 120 7175 83430 2 31 3 1858 0 gsd-xsettings
[ 262.252099] [ 7178] 120 7178 58871 0 17 4 752 0 gsd-a11y-settin
[ 262.252105] [ 7180] 120 7180 46197 2 25 3 1692 0 gsd-clipboard
[ 262.252112] [ 7185] 120 7185 104410 0 36 5 1340 0 gsd-color
[ 262.252119] [ 7188] 120 7188 53615 2 26 4 479 0 gsd-datetime
[ 262.252126] [ 7192] 120 7192 58153 0 14 3 164 0 gsd-housekeepin
[ 262.252132] [ 7193] 120 7193 83428 2 30 5 1340 0 gsd-keyboard
[ 262.252139] [ 7198] 120 7198 149947 2 33 4 1347 0 gsd-media-keys
[ 262.252146] [ 7199] 120 7199 39460 0 12 5 147 0 gsd-mouse
[ 262.252152] [ 7202] 120 7202 83632 0 32 4 1316 0 gsd-power
[ 262.252159] [ 7209] 120 7209 42287 0 17 3 303 0 gsd-print-notif
[ 262.252166] [ 7210] 120 7210 39464 0 12 4 148 0 gsd-rfkill
[ 262.252173] [ 7211] 120 7211 57894 0 14 5 159 0 gsd-screensaver
[ 262.252179] [ 7214] 120 7214 60516 0 19 3 314 0 gsd-sharing
[ 262.252186] [ 7227] 120 7227 78267 0 20 4 294 0 gsd-smartcard
[ 262.252192] [ 7232] 120 7232 60306 0 19 5 829 0 gsd-sound
[ 262.252199] [ 7238] 120 7238 64919 2 28 4 1262 0 gsd-wacom
[ 262.252206] [ 7258] 116 7258 60982 2 21 3 1257 0 colord
[ 262.252212] [ 7318] 120 7318 40367 0 14 3 711 0 ibus-engine-sim
[ 262.252219] [ 7353] 0 7353 1318 0 6 3 36 0 agetty
[ 262.252225] [ 7374] 0 7374 3105 2 9 3 249 0 sshd
[ 262.252232] [ 7388] 1000 7388 3161 2 10 4 337 0 systemd
[ 262.252238] [ 7389] 1000 7389 40884 0 15 4 686 0 (sd-pam)
[ 262.252244] [ 7468] 1000 7468 3105 0 9 3 267 0 sshd
[ 262.252251] [ 7469] 1000 7469 2420 2 10 4 425 0 bash
[ 262.252257] [ 7483] 0 7483 3105 2 10 4 249 0 sshd
[ 262.252264] [ 7530] 1000 7530 3105 2 10 4 249 0 sshd
[ 262.252270] [ 7531] 1000 7531 2420 2 9 4 402 0 bash
[ 262.252276] [ 7563] 1000 7563 4841 2 12 4 780 0 vi
[ 262.252405] [ 7564] 1000 7564 3539143 88632 2177 11 644786 0 python3
[ 262.252415] [ 7937] 0 7937 4052 0 7 3 565 0 systemd-udevd
[ 262.252422] [ 8039] 0 8039 4052 3 7 3 550 0 systemd-udevd
[ 262.252433] Out of memory: Kill process 7564 (python3) score 246 or sacrifice child
[ 262.252794] Killed process 7564 (python3) total-vm:14156572kB, anon-rss:108kB, file-rss:354420kB, shmem-rss:0kB
[ 262.266689] kworker/u12:9 invoked oom-killer: gfp_mask=0x27080c0(GFP_KERNEL_ACCOUNT|__GFP_ZERO|__GFP_NOTRACK), nodemask=0, order=2, oom_score_adj=0
[ 262.266974] kworker/u12:9 cpuset=/ mems_allowed=0
[ 262.267008] CPU: 3 PID: 7933 Comm: kworker/u12:9 Not tainted 4.9.140-tegra #1
[ 262.267013] Hardware name: NVIDIA Jetson Xavier NX Developer Kit (DT)
[ 262.267033] Workqueue: events_unbound call_usermodehelper_exec_work
[ 262.267042] Call trace:
[ 262.267052] [] dump_backtrace+0x0/0x198
[ 262.267059] [] show_stack+0x24/0x30
[ 262.267067] [] dump_stack+0x98/0xc0
[ 262.267075] [] dump_header+0x6c/0x1b8
[ 262.267084] [] oom_kill_process+0x268/0x498
[ 262.267089] [] out_of_memory+0x1e4/0x308
[ 262.267096] [] __alloc_pages_nodemask+0x844/0xd38
[ 262.267104] [] copy_process.isra.5.part.6+0x110/0x1530
[ 262.267110] [] _do_fork+0xd4/0x460
[ 262.267115] [] kernel_thread+0x48/0x58
[ 262.267121] [] call_usermodehelper_exec_work+0x34/0xd0
[ 262.267128] [] process_one_work+0x1e4/0x4b0
[ 262.267134] [] worker_thread+0x50/0x4c8
[ 262.267141] [] kthread+0xec/0xf0
[ 262.267147] [] ret_from_fork+0x10/0x30
[ 262.267152] Mem-Info:
[ 262.267165] active_anon:32 inactive_anon:237 isolated_anon:0
active_file:348 inactive_file:594 isolated_file:0
unevictable:3038 dirty:0 writeback:0 unstable:0
slab_reclaimable:11258 slab_unreclaimable:24902
mapped:708 shmem:0 pagetables:3709 bounce:0
free:19115 free_pcp:352 free_cma:1690
[ 262.267182] Node 0 active_anon:128kB inactive_anon:948kB active_file:1392kB inactive_file:2376kB unevictable:12152kB isolated(anon):0kB isolated(file):0kB mapped:2832kB dirty:0kB writeback:0kB shmem:0kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB pages_scanned:0 all_unreclaimable? no
[ 262.267205] DMA free:38972kB min:10196kB low:12744kB high:15292kB active_anon:12kB inactive_anon:144kB active_file:256kB inactive_file:584kB unevictable:0kB writepending:0kB present:1837812kB managed:1812080kB mlocked:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:720kB local_pcp:0kB free_cma:7060kB
[ 262.267208] lowmem_reserve: 0 6000 6000 6000
[ 262.267243] Normal free:37888kB min:34856kB low:43568kB high:52280kB active_anon:288kB inactive_anon:732kB active_file:1104kB inactive_file:1896kB unevictable:12152kB writepending:0kB present:6291456kB managed:6144748kB mlocked:0kB slab_reclaimable:45032kB slab_unreclaimable:99608kB kernel_stack:7376kB pagetables:14836kB bounce:0kB free_pcp:4kB local_pcp:0kB free_cma:0kB
[ 262.267248] lowmem_reserve: 0 0 0 0
[ 262.267266] DMA: 934kB (UC) 568kB (UMC) 2216kB (UC) 1532kB (UMC) 1464kB (MC) 9128kB (MC) 10256kB (UMC) 9512kB (UMC) 61024kB (UMC) 92048kB (UMC) 14096kB (M) = 39540kB
[ 262.267339] Normal: 321
4kB (UME) 2958kB (UME) 97516kB (UM) 44532kB (UM) 4464kB (UM) 10128kB (U) 3256kB (U) 0512kB 01024kB 02048kB 04096kB = 38348kB
[ 262.267399] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[ 262.267403] 4188 total pagecache pages
[ 262.267407] 6 pages in swap cache
[ 262.267412] Swap cache stats: add 802294, delete 802278, find 1289/47835
[ 262.267416] Free swap = 1135292kB
[ 262.267419] Total swap = 3978408kB
[ 262.267423] 2032317 pages RAM
[ 262.267427] 0 pages HighMem/MovableOnly
[ 262.267431] 43110 pages reserved
[ 262.267435] 188416 pages cma reserved
[ 262.267438] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
[ 262.267469] [ 2344] 0 2344 4547 297 13 3 198 0 systemd-journal
[ 262.267476] [ 3059] 0 3059 4052 0 7 3 562 -1000 systemd-udevd
[ 262.267483] [ 3274] 62583 3274 20989 2 11 4 162 0 systemd-timesyn
[ 262.267490] [ 3279] 0 3279 1957 2 7 3 819 0 haveged
[ 262.267496] [ 3310] 0 3310 1601 0 7 3 113 0 rpcbind
[ 262.267503] [ 3321] 102 3321 2641 0 10 4 174 0 systemd-resolve
[ 262.267510] [ 4479] 0 4479 1660 0 7 4 91 0 cron
[ 262.267516] [ 4495] 0 4495 78217 0 20 3 419 0 ModemManager
[ 262.267522] [ 4524] 0 4524 25081 0 18 3 2028 0 networkd-dispat
[ 262.267528] [ 4557] 103 4557 2033 2 8 3 391 -900 dbus-daemon
[ 262.267534] [ 4702] 0 4702 2486 3 9 4 283 0 wpa_supplicant
[ 262.267540] [ 4707] 0 4707 119794 0 34 5 1360 0 NetworkManager
[ 262.267546] [ 4711] 0 4711 59324 0 19 4 355 0 accounts-daemon
[ 262.267552] [ 4724] 0 4724 97384 2 26 4 1173 0 udisksd
[ 262.267558] [ 4732] 0 4732 2646 0 10 4 224 0 systemd-logind
[ 262.267564] [ 4742] 114 4742 1536 179 8 3 102 0 avahi-daemon
[ 262.267571] [ 4773] 109 4773 54992 0 13 4 355 0 rsyslogd
[ 262.267576] [ 4834] 114 4834 1490 0 8 3 84 0 avahi-daemon
[ 262.267583] [ 4955] 0 4955 59197 2 18 4 707 0 polkitd
[ 262.267591] [ 5419] 111 5419 62055 2 23 3 459 0 whoopsie
[ 262.267596] [ 5508] 0 5508 5376 0 8 3 69 0 nvs-service
[ 262.267602] [ 5514] 0 5514 1654 0 7 4 82 0 nvmemwarning.sh
[ 262.267609] [ 5533] 0 5533 287341 0 58 8 5380 0 containerd
[ 262.267614] [ 5563] 105 5563 2368 0 8 4 126 0 kerneloops
[ 262.267621] [ 5578] 0 5578 2603 0 9 4 180 -1000 sshd
[ 262.267626] [ 5611] 105 5611 2368 2 9 3 115 0 kerneloops
[ 262.267633] [ 5764] 0 5764 25219 0 31 3 1247 0 nvargus-daemon
[ 262.267640] [ 5853] 0 5853 58868 0 15 3 276 0 gdm3
[ 262.267645] [ 5894] 0 5894 1318 0 7 4 37 0 agetty
[ 262.267652] [ 5910] 0 5910 40797 2 17 3 383 0 gdm-session-wor
[ 262.267657] [ 6028] 0 6028 787 0 4 3 38 0 nvphsd
[ 262.267664] [ 6030] 0 6030 3046 0 6 3 88 0 nvphsd
[ 262.267670] [ 6141] 0 6141 1654 0 7 3 76 0 nvgetty.sh
[ 262.267676] [ 6172] 0 6172 1318 0 6 4 36 0 getty
[ 262.267682] [ 6207] 0 6207 2094 2 9 4 321 0 dhclient
[ 262.267687] [ 6284] 0 6284 1775 0 7 3 103 0 bluetoothd
[ 262.267693] [ 6301] 120 6301 3185 2 11 4 359 0 systemd
[ 262.267699] [ 6331] 120 6331 4018 0 11 4 670 0 (sd-pam)
[ 262.267705] [ 6650] 0 6650 4378 0 13 4 2417 0 dhcpd
[ 262.267711] [ 6656] 120 6656 39897 2 12 4 167 0 gdm-x-session
[ 262.267718] [ 6671] 120 6671 6327632 436 37 4 2843 0 Xorg
[ 262.267724] [ 6732] 120 6732 1764 2 7 4 222 0 dbus-daemon
[ 262.267730] [ 6736] 120 6736 101008 1 28 3 606 0 gnome-session-b
[ 262.267736] [ 6739] 120 6739 76965 2 18 5 750 0 at-spi-bus-laun
[ 262.267743] [ 6742] 120 6742 58779 2 16 3 250 0 gvfsd
[ 262.267749] [ 6747] 120 6747 78612 0 17 5 274 0 gvfsd-fuse
[ 262.267755] [ 6751] 120 6751 1682 2 7 4 120 0 dbus-daemon
[ 262.267763] [ 6777] 120 6777 39804 0 13 3 204 0 at-spi2-registr
[ 262.267771] [ 6915] 120 6915 782055 1275 138 7 14707 0 gnome-shell
[ 262.267777] [ 6922] 0 6922 61867 0 19 5 331 0 upowerd
[ 262.267783] [ 6977] 120 6977 255406 2 23 4 1816 0 pulseaudio
[ 262.267790] [ 6978] 106 6978 38091 0 9 4 95 0 rtkit-daemon
[ 262.267796] [ 7060] 120 7060 77520 2 19 5 372 0 ibus-daemon
[ 262.267802] [ 7063] 120 7063 58837 0 16 4 209 0 ibus-dconf
[ 262.267808] [ 7066] 120 7066 46285 2 28 3 1180 0 ibus-x11
[ 262.267814] [ 7068] 120 7068 58814 0 16 4 201 0 ibus-portal
[ 262.267820] [ 7121] 0 7121 59434 0 17 3 267 0 boltd
[ 262.267826] [ 7135] 0 7135 68645 0 26 4 464 0 packagekitd
[ 262.267832] [ 7175] 120 7175 83430 2 31 3 1858 0 gsd-xsettings
[ 262.267839] [ 7178] 120 7178 58871 0 17 4 752 0 gsd-a11y-settin
[ 262.267845] [ 7180] 120 7180 46197 2 25 3 1692 0 gsd-clipboard
[ 262.267851] [ 7185] 120 7185 104410 0 36 5 1340 0 gsd-color
[ 262.267857] [ 7188] 120 7188 53615 2 26 4 479 0 gsd-datetime
[ 262.268936] [ 7192] 120 7192 58153 0 14 3 164 0 gsd-housekeepin
[ 262.268950] [ 7193] 120 7193 83428 2 30 5 1340 0 gsd-keyboard
[ 262.268960] [ 7198] 120 7198 149947 2 33 4 1347 0 gsd-media-keys
[ 262.268968] [ 7199] 120 7199 39460 0 12 5 147 0 gsd-mouse
[ 262.268979] [ 7202] 120 7202 83632 0 32 4 1316 0 gsd-power
[ 262.268987] [ 7209] 120 7209 42287 0 17 3 303 0 gsd-print-notif
[ 262.268997] [ 7210] 120 7210 39464 0 12 4 148 0 gsd-rfkill
[ 262.269005] [ 7211] 120 7211 57894 0 14 5 159 0 gsd-screensaver
[ 262.269012] [ 7214] 120 7214 60516 0 19 3 314 0 gsd-sharing
[ 262.269019] [ 7227] 120 7227 78267 0 20 4 294 0 gsd-smartcard
[ 262.269027] [ 7232] 120 7232 60306 0 19 5 829 0 gsd-sound
[ 262.269034] [ 7238] 120 7238 64919 2 28 4 1262 0 gsd-wacom
[ 262.269041] [ 7258] 116 7258 60982 2 21 3 1257 0 colord
[ 262.269047] [ 7318] 120 7318 40367 0 14 3 711 0 ibus-engine-sim
[ 262.269054] [ 7353] 0 7353 1318 0 6 3 36 0 agetty
[ 262.269061] [ 7374] 0 7374 3105 2 9 3 249 0 sshd
[ 262.269067] [ 7388] 1000 7388 3161 2 10 4 337 0 systemd
[ 262.269074] [ 7389] 1000 7389 40884 0 15 4 686 0 (sd-pam)
[ 262.269080] [ 7468] 1000 7468 3105 0 9 3 267 0 sshd
[ 262.269087] [ 7469] 1000 7469 2420 2 10 4 425 0 bash
[ 262.269093] [ 7483] 0 7483 3105 2 10 4 249 0 sshd
[ 262.269099] [ 7530] 1000 7530 3105 2 10 4 249 0 sshd
[ 262.269105] [ 7531] 1000 7531 2420 2 9 4 402 0 bash
[ 262.269112] [ 7563] 1000 7563 4841 2 12 4 780 0 vi
[ 262.269120] [ 7937] 0 7937 4052 0 7 3 565 0 systemd-udevd
[ 262.269127] [ 8039] 0 8039 4052 3 7 3 550 0 systemd-udevd
[ 262.269132] Out of memory: Kill process 6915 (gnome-shell) score 5 or sacrifice child
[ 262.269378] Killed process 7060 (ibus-daemon) total-vm:310080kB, anon-rss:0kB, file-rss:8kB, shmem-rss:0kB

Thanks @mchi. So where to from here? Maybe the model is too big to run when using Ubuntu desktop (I had no other apps open).

I’ll give it another try headless. The objectdetector_ssd sample with deepstream apps works fine so it should be ok.

we will check and get back… thanks!

Hi @jasonpgf2a,
changing tf_gpu_memory_fraction from 0.6 to 0.5 in /opt/nvidia/deepstream/deepstream/sources/python/apps/deepstream-ssd-parser/dstest_ssd_nopostprocess.txt could help this issue, please give a try.

Thanks!

1 Like

Hi @mchi ,
after performed the steps in the deepstream README : make install sources/libs/nvdsinfer_customparser and execute ./prepare_ds_trtis_model_repo.sh, I run ‘python3 deepstream_ssd_parser.py /opt/nvidia/deepstream/deepstream-5.0/samples/streams/sample_720p.h264’,
I got ERROR output:
Creating Pipeline

Creating Source
Creating H264Parser
Creating Decoder
Creating NvStreamMux
Creating Nvinferserver
2020-08-06 17:00:29.060179: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.2
Creating Nvvidconv
Creating OSD (nvosd)
Creating Queue
Creating Converter 2 (nvvidconv2)
Creating capsfilter
Creating Encoder
Creating Code Parser
Creating Container
Creating Sink
Playing file /opt/nvidia/deepstream/deepstream-5.0/samples/streams/sample_720p.h264
Adding elements to Pipeline

Linking elements in the Pipeline

Starting pipeline

Opening in BLOCKING MODE
I0806 09:00:29.497959 10965 server.cc:120] Initializing Triton Inference Server
I0806 09:00:29.511454 10965 server_status.cc:55] New status tracking for model ‘ssd_inception_v2_coco_2018_01_28’
I0806 09:00:29.511642 10965 model_repository_manager.cc:680] loading: ssd_inception_v2_coco_2018_01_28:1
I0806 09:00:29.512333 10965 base_backend.cc:176] Creating instance ssd_inception_v2_coco_2018_01_28_0_gpu0 on GPU 0 (7.2) using model.graphdef
2020-08-06 17:00:29.596644: W tensorflow/core/platform/profile_utils/cpu_utils.cc:98] Failed to find bogomips in /proc/cpuinfo; cannot determine CPU frequency
2020-08-06 17:00:29.598024: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0xde1cc00 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-08-06 17:00:29.598129: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2020-08-06 17:00:29.598398: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcuda.so.1
2020-08-06 17:00:29.598721: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-08-06 17:00:29.598926: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties:
name: Xavier major: 7 minor: 2 memoryClockRate(GHz): 1.109
pciBusID: 0000:00:00.0
2020-08-06 17:00:29.599004: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.2
2020-08-06 17:00:29.599216: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2020-08-06 17:00:29.603280: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2020-08-06 17:00:29.604506: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2020-08-06 17:00:29.609778: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2020-08-06 17:00:29.613914: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2020-08-06 17:00:29.614128: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.8
2020-08-06 17:00:29.614330: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-08-06 17:00:29.614587: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-08-06 17:00:29.614675: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0
2020-08-06 17:00:32.915790: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-08-06 17:00:32.915956: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186] 0
2020-08-06 17:00:32.916000: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0: N
2020-08-06 17:00:32.916318: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-08-06 17:00:32.916598: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-08-06 17:00:32.916774: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-08-06 17:00:32.916953: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3881 MB memory) -> physical GPU (device: 0, name: Xavier, pci bus id: 0000:00:00.0, compute capability: 7.2)
2020-08-06 17:00:32.923383: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7eb04270a0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-08-06 17:00:32.923555: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Xavier, Compute Capability 7.2
I0806 09:00:33.672094 10965 model_repository_manager.cc:837] successfully loaded ‘ssd_inception_v2_coco_2018_01_28’ version 1
**
ERROR: gie id: 5 model: ssd_inception_v2_coco_2018_01_28 input or output layers are empty, check trtis model config settings
ERROR: failed to initialize backend while ensuring model:ssd_inception_v2_coco_2018_01_28 ready, nvinfer error:NVDSINFER_TRTIS_ERROR
0:00:05.296779474 10965 0xf1dda10 ERROR nvinferserver gstnvinferserver.cpp:362:gst_nvinfer_server_logger: nvinferserver[UID 5]: Error in createNNBackend() <infer_trtis_context.cpp:199> [UID = 5]: failed to initialize trtis backend for model:ssd_inception_v2_coco_2018_01_28, nvinfer error:NVDSINFER_TRTIS_ERROR
I0806 09:00:33.675346 10965 model_repository_manager.cc:708] unloading: ssd_inception_v2_coco_2018_01_28:1
I0806 09:00:33.745982 10965 model_repository_manager.cc:816] successfully unloaded ‘ssd_inception_v2_coco_2018_01_28’ version 1
I0806 09:00:33.746183 10965 server.cc:179] Waiting for in-flight inferences to complete.
I0806 09:00:33.746230 10965 server.cc:194] Timeout 30: Found 0 live models and 0 in-flight requests
0:00:05.368937514 10965 0xf1dda10 ERROR nvinferserver gstnvinferserver.cpp:362:gst_nvinfer_server_logger: nvinferserver[UID 5]: Error in initialize() <infer_base_context.cpp:78> [UID = 5]: create nn-backend failed, check config file settings, nvinfer error:NVDSINFER_TRTIS_ERROR
0:00:05.369028234 10965 0xf1dda10 WARN nvinferserver gstnvinferserver_impl.cpp:439:start: error: Failed to initialize InferTrtIsContext
0:00:05.369100811 10965 0xf1dda10 WARN nvinferserver gstnvinferserver_impl.cpp:439:start: error: Config file path: dstest_ssd_nopostprocess.txt
0:00:05.369810509 10965 0xf1dda10 WARN nvinferserver gstnvinferserver.cpp:460:gst_nvinfer_server_start: error: gstnvinferserver_impl start failed
Error: gst-resource-error-quark: Failed to initialize InferTrtIsContext (1): /dvs/git/dirty/git-master_linux/deepstream/sdk/src/gst-plugins/gst-nvinferserver/gstnvinferserver_impl.cpp(439): start (): /GstPipeline:pipeline0/GstNvInferServer:primary-inference:
Config file path: dstest_ssd_nopostprocess.txt**

why this error come? missing config.pbtxt of model?

As per the README, the c ommand should be

$ LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libgomp.so.1 python3 deepstream_ssd_parser.py …/…/…/…/samples/streams/sample_720p.h264

I don’t think it’s DeepStream bug. can you elaborate your command, and check the model you download ?