DeepStream and HashiCorp Nomad

• NVIDIA Jetson Xavier NX Developer Kit
• L4T 35.3.1

Hello!

I’m trying to use HashiCorp Nomad as orchestrate solution for Docker, especially for running deepstream applications. So, I set runtime to nvidia, start clean deepstream:6.2-samples container and run following code:

cd /opt/nvidia/deepstream/deepstream 
./user_additional_install.sh 
apt install -yy wget zip 
wget --content-disposition https://api.ngc.nvidia.com/v2/models/nvidia/tao/trafficcamnet/versions/pruned_v1.0.3/zip -O trafficcamnet_pruned_v1.0.3.zip 
cd /opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/ 
git clone https://github.com/NVIDIA-AI-IOT/deepstream_reference_apps.git
cd /opt/nvidia/deepstream/deepstream/ 
cp sources/apps/sample_apps/deepstream_reference_apps/deepstream_app_tao_configs/* samples/configs/tao_pretrained_models/ 
cd /opt/nvidia/deepstream/deepstream/samples/configs/tao_pretrained_models/ && ./download_models.sh

deepstream_app_source1_trafficcamnet.txt configuration is default one, only EglSink is disabled, and RTSPStreaming is enabled.

(This configuration works fine in regular docker)

deepstream_app_source1_trafficcamnet.txt
[application]
enable-perf-measurement=1
perf-measurement-interval-sec=1

[tiled-display]
enable=1
rows=1
columns=1
width=1280
height=720
gpu-id=0

[source0]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=3
num-sources=1
uri=file://../../streams/sample_1080p_h265.mp4
gpu-id=0

[streammux]
gpu-id=0
batch-size=1
batched-push-timeout=40000
## Set muxer output width and height
width=1920
height=1080

[osd]
enable=1
gpu-id=0
border-width=3
text-size=15
text-color=1;1;1;1;
text-bg-color=0.3;0.3;0.3;1
font=Arial

[primary-gie]
enable=1
gpu-id=0
# Modify as necessary
model-engine-file=../../models/tao_pretrained_models/trafficcamnet/resnet18_trafficcamnet_pruned.etlt_b1_gpu0_int8.engine
batch-size=1
#Required by the app for OSD, not a plugin property
bbox-border-color0=1;0;0;1
bbox-border-color1=0;1;1;1
bbox-border-color2=0;0;1;1
bbox-border-color3=0;1;0;1
gie-unique-id=1
config-file=config_infer_primary_trafficcamnet.txt

[sink2]
enable=1
#Type - 1=FakeSink 2=EglSink 3=File 4=RTSPStreaming 5=Overlay
type=4
#1=h264 2=h265
codec=1
#encoder type 0=Hardware 1=Software
enc-type=0
sync=0
bitrate=4000000
#H264 Profile - 0=Baseline 2=Main 4=High
#H265 Profile - 0=Main 1=Main10
profile=0
# set below properties in case of RTSPStreaming
rtsp-port=8554
udp-port=5400

[tracker]
enable=1
# For NvDCF and DeepSORT tracker, tracker-width and tracker-height must be a multiple of 32, respectively
tracker-width=640
tracker-height=384
ll-lib-file=/opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so
# ll-config-file required to set different tracker types
# ll-config-file=../deepstream-app/config_tracker_IOU.yml
ll-config-file=../deepstream-app/config_tracker_NvDCF_perf.yml
# ll-config-file=../deepstream-app/config_tracker_NvDCF_accuracy.yml
# ll-config-file=../deepstream-app/config_tracker_DeepSORT.yml
gpu-id=0
enable-batch-process=1
enable-past-frame=1
display-tracking-id=1

[tests]
file-loop=0

Output from Nomad:

deepstream-app -c deepstream_app_source1_trafficcamnet.txt
# deepstream-app -c deepstream_app_source1_trafficcamnet.txt 
(Argus) Error FileOperationFailed: Connecting to nvargus-daemon failed: No such file or directory (in src/rpc/socket/client/SocketClientDispatch.cpp, function openSocketConnection(), line 204)
(Argus) Error FileOperationFailed: Cannot create camera provider (in src/rpc/socket/client/SocketClientDispatch.cpp, function createCameraProvider(), line 106)

Warning: 'input-dims' parameter has been deprecated. Use 'infer-dims' instead.
gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so

gstnvtracker: Batch processing is ON
gstnvtracker: Past frame output is ON
[NvMultiObjectTracker] Initialized
Killed

Detailed output (--gst-debug=4):

debug_output.txt (97.1 KB)

argus camera is only available on Jetson platform, but your config file set source0 as URI, seems the failure is not from the config file you posted, and this error log doesn’t exist in the log file you attached.

That’s correct, I double-checked once again. These errors occur when using the configuration file from my first post, during the initial launch with an empty gstreamer cache.

Look:

# rm -rf ~/.cache/gstreamer-1.0/

# deepstream-app -c deepstream_app_source1_trafficcamnet.txt 
(Argus) Error FileOperationFailed: Connecting to nvargus-daemon failed: No such file or directory (in src/rpc/socket/client/SocketClientDispatch.cpp, function openSocketConnection(), line 204)
(Argus) Error FileOperationFailed: Cannot create camera provider (in src/rpc/socket/client/SocketClientDispatch.cpp, function createCameraProvider(), line 106)

(gst-plugin-scanner:1513): GStreamer-WARNING **: 12:20:37.988: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_udp.so': librivermax.so.0: cannot open shared object file: No such file or directory

(gst-plugin-scanner:1513): GStreamer-WARNING **: 12:20:38.057: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_inferserver.so': libtritonserver.so: cannot open shared object file: No such file or directory

 *** DeepStream: Launched RTSP Streaming at rtsp://localhost:8554/ds-test ***

Warning: 'input-dims' parameter has been deprecated. Use 'infer-dims' instead.
Opening in BLOCKING MODE 
gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so
gstnvtracker: Batch processing is ON
gstnvtracker: Past frame output is ON
[NvMultiObjectTracker] Initialized
Killed

Second start with fulfilled gstreamer cache (no Argus errors):

# deepstream-app -c deepstream_app_source1_trafficcamnet.txt 

 *** DeepStream: Launched RTSP Streaming at rtsp://localhost:8554/ds-test ***

Warning: 'input-dims' parameter has been deprecated. Use 'infer-dims' instead.
Opening in BLOCKING MODE 
gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so
gstnvtracker: Batch processing is ON
gstnvtracker: Past frame output is ON
[NvMultiObjectTracker] Initialized
Killed

My platform is Jetson, BTW

Nomad is running on Xavier, and container for job is native deepstream-6.2:samples with nvidia runtime.

Also module for Argus us present inside container:

# ls -la /usr/lib/aarch64-linux-gnu/tegra/libv4l2_nvargus.so
-rw-r--r-- 1 root root 261744 Mar 19 15:14 /usr/lib/aarch64-linux-gnu/tegra/libv4l2_nvargus.so

I suspect that Argus has nothing to do with this. I see that nvargus-daemon is disabled on the host, and the containers I run directly on it using the same configuration file with pure docker are functioning properly.

I tried to run deepstream-app with same config in pure docker once again, and Argus error is also here. But job is running OK

Please see the output from pure Docker:

# deepstream-app -c deepstream_app_source1_trafficcamnet.txt 
(Argus) Error FileOperationFailed: Connecting to nvargus-daemon failed: No such file or directory (in src/rpc/socket/client/SocketClientDispatch.cpp, function openSocketConnection(), line 204)
(Argus) Error FileOperationFailed: Cannot create camera provider (in src/rpc/socket/client/SocketClientDispatch.cpp, function createCameraProvider(), line 106)

(gst-plugin-scanner:841): GStreamer-WARNING **: 12:44:06.217: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_udp.so': librivermax.so.0: cannot open shared object file: No such file or directory

(gst-plugin-scanner:841): GStreamer-WARNING **: 12:44:06.283: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_inferserver.so': libtritonserver.so: cannot open shared object file: No such file or directory

 *** DeepStream: Launched RTSP Streaming at rtsp://localhost:8554/ds-test ***

Warning: 'input-dims' parameter has been deprecated. Use 'infer-dims' instead.
Opening in BLOCKING MODE 
gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so
gstnvtracker: Batch processing is ON
gstnvtracker: Past frame output is ON
[NvMultiObjectTracker] Initialized
....
< More log and Job output here>
....

So this is definitely not a Argus problem.

OK. Just from the original log, putting aside this Argus error, there are no other errors before you kill it. Could I know how you confirm there was a problem? Could you open more log with GST_DEBUG=3 before the deepstream-app -c deepstream_app_source1_trafficcamnet.txt?

Okay, I figured it out.

The problem was with memory limits, and the OOM Killer was killing the process.

Increasing the memory limit for Nomad Job helped, and the program is working now

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.