Deepstream docker restarts every time in nvidia metropolis

Hi,
I’m trying to use Nvidia Metropolis services, but I’m unable to see the output stream on the overlay. I’ve successfully installed nvstreamer and ai_nvr as per the documentation. I observed that the DeepStream Docker container restarts every time. I’ve attached the log file of the DeepStream Docker.

I’m using Ubuntu 22.04 and the latest JetPack 6.0.

metropolis_deepstream_docker_logs.txt (551.0 KB)

DeepStream keeps crashing due to the following error:

GPUassert_VPI: VPI_ERROR_INTERNAL PVA kernel not loaded due to internal authentication or other PVA driver error /dvs/git/dirty/git-master_linux/deepstream/sdk/src/utils/nvmultiobjecttracker/src/modules/VisualTracker/DCFTracker.cpp 276

Can you please provide information on which device you are using and the specific command you used to launch AI NVR?

Thanks for your reply. I’m using the Nvidia Jetson AGX 64 GB. To launch ai_nvr, I’m using the command sudo docker compose -f compose_agx.yaml up -d --force-recreate. I’m following the Metropolis documentation.

Adding to that, with JetPack 6.0 DP, I did not face these issues. However, with the latest JetPack 6.0 (rev.2), I am encountering problems. When flashing the AGX Orin board, the latest JetPack installs DeepStream 7.0 on host, whereas Metropolis always pulls DeepStream 6.4 in Docker when launching ai_nvr.

Is it an AGX Orin or an older model?

Can you check the VPI version (the libnvvpi3 package) installed on the system? This can be done by running dpkg -l | grep vpi. It should be version 3.1.5 in order to be compatible with the latest version of DeepStream

Yes It’s Agx Orin.

orin@ubuntu:~$ dpkg -l | grep vpi
ii libnvvpi3 3.1.5 arm64 NVIDIA Vision Programming Interface library
ii python3.10-vpi3 3.1.5 arm64 NVIDIA VPI python 3.10 bindings

Are you pulling AI NVR from the latest NGC resource or from SDK Manager?

The docs you linked in your earlier comment is for the previous release which was only for JP6.0 DP. Please follow steps for the latest release, now called Jetson Platform Services (JPS) here: Quick Start Guide — Metropolis on Jetson documentation 0.1.0 documentation . The latest AI NVR bundle can be found on NGC here: https://catalog.ngc.nvidia.com/orgs/nvidia/teams/jps/resources/reference-workflow-and-resources . This new version is meant for JP6.0 GA.

The latest AI NVR will also pull the DeepStream 7.0 container rather than 6.4.

Yes I’m pulling ai_nvr with ngc resource with this link. I’ll go through with latest documentation

I did setup as per new documentation but not getting output stream.
I have tried with below rtsp url on port 8556 and 8555 but not getting output overlay.
deepstream_7.0_log.txt (144.5 KB)

rtsp://10.126.186.57:8556/ds-test

DeepStream RTSP output is no longer supported by default. Rather, you can view gem overlay in the VST web UI.

If you still need bounding box RTSP output from DeepStream, you can enable it following steps from this post: What is the URL and port to see the Deepstream Output on the VLC media player in Jetson platform services? - #6 by hfarooq

Hi @hfarooq ,
I have gone through the post and trying to use below command for my orin agx board that you have mentioned SO I used “ds-config-0_agx.txt” file in that command But I am not getting output.

sh -c 'deepstream-test5-app -c /ds-config-files/pn26/ds-config-0_agx.txt 2>&1 | grep --line-buffered . | tee -a /log/deepstream.log

Can you share the full log of deepstream-test5-app? Can you help to conform below sink3 is enabled in ds-config-0_agx.txt?

[sink3]
enable=1
#Type - 1=FakeSink 2=EglSink 3=File 4=RTSPStreaming 5=Overlay
type=4
#1=h264 2=h265
codec=1
encoder type 0=Hardware 1=Software
enc-type=0
sync=0
bitrate=4000000
#H264 Profile - 0=Baseline 2=Main 4=High
#H265 Profile - 0=Main 1=Main10
profile=0

set below properties in case of RTSPStreaming

rtsp-port=8555
udp-port=5401

  • [sink3]

  • enable=1

  • #Type - 1=FakeSink 2=EglSink 3=File 4=RTSPStreaming 5=Overlay

  • type=4

  • #1=h264 2=h265

  • codec=1

  • encoder type 0=Hardware 1=Software

  • enc-type=0

  • sync=0

  • bitrate=4000000

  • #H264 Profile - 0=Baseline 2=Main 4=High

  • #H265 Profile - 0=Main 1=Main10

  • profile=0

  • set below properties in case of RTSPStreaming

  • rtsp-port=8555

  • udp-port=5401

deepstream-test5-app_logs.txt (89.6 KB)

The full log of deepstream-test5-app is interrupted during TRT build model. Can you wait more time to get the full log?

Suppose you will see below log. The last line show the output RTSP URL.

$ docker logs deepstream
(gst-plugin-scanner:22): GStreamer-WARNING **: 08:09:47.087: Failed to load plugin ‘/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstmpg123.so’: libmpg123.so.0: cannot open shared object file: No such file or directory
(gst-plugin-scanner:22): GStreamer-WARNING **: 08:09:47.175: Failed to load plugin ‘/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstmpeg2enc.so’: libmpeg2encpp-2.1.so.0: cannot open shared object file: No such file or directory
(gst-plugin-scanner:22): GStreamer-WARNING **: 08:09:47.228: Failed to load plugin ‘/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstopenmpt.so’: libmpg123.so.0: cannot open shared object file: No such file or directory
(gst-plugin-scanner:22): GStreamer-WARNING **: 08:09:47.264: Failed to load plugin ‘/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstchromaprint.so’: libavcodec.so.58: cannot open shared object file: No such file or directory
(gst-plugin-scanner:22): GStreamer-WARNING **: 08:09:47.304: Failed to load plugin ‘/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstmpeg2dec.so’: libmpeg2.so.0: cannot open shared object file: No such file or directory
(gst-plugin-scanner:22): GStreamer-WARNING **: 08:09:47.336: adding type GstEvent multiple times
(gst-plugin-scanner:22): GStreamer-WARNING **: 08:09:47.712: Failed to load plugin ‘/usr/lib/aarch64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_udp.so’: librivermax.so.0: cannot open shared object file: No such file or directory
(gst-plugin-scanner:22): GStreamer-WARNING **: 08:09:47.809: Failed to load plugin ‘/usr/lib/aarch64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_inferserver.so’: libtritonserver.so: cannot open shared object file: No such file or directory
(deepstream-test5-app:19): GLib-GObject-CRITICAL **: 08:09:48.698: g_object_set_is_valid_property: object class ‘nvv4l2h264enc’ has no property named ‘gpu-id’
*** DeepStream: Launched RTSP Streaming at rtsp://localhost:8555/ds-test ***

deepstream-test5-app_logs.txt (45.6 KB)
deepstream_docker_logs.txt (274.3 KB)

I’m using rtsp://localhost:8555/ds-test link only to open output video, but its not opening

I see below error log in deepstream-test5-app_logs.txt. Suppose we shouldn’t meet below error. Which DeepStream container are you using? What is the steps to reproduce below error? To enable RTSP output, you just need to modify compose_agx.yaml to sh -c ‘deepstream-test5-app -c /ds-config-files/pn26/ds-config-0_agx.txt 2>&1 | grep --line-buffered . | tee -a /log/deepstream.log’ and then depoly AI NVR.

stnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so
gstnvtracker: Failed to open low-level lib at /opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so
dlopen error: libcufft.so: cannot open shared object file: No such file or directory
gstnvtracker: Failed to initilaize low level lib.
** ERROR: main:1539: Failed to set pipeline to PAUSED
Quitting
Stopping the server…!!
Stopped the server…!!
App run failed

I’m using deestream-7.0. Could you please help me where should I add sh -c ‘deepstream-test5-app -c /ds-config-files/pn26/ds-config-0_agx.txt 2>&1 | grep --line-buffered . | tee -a /log/deepstream.log’ in compose_agx.yaml file.
compose_agx.zip (1.5 KB)

By adding this in last line of compose_agx.yaml file I’m getting below error:
validating /home/orin/ai_nvr/compose_agx.yaml: services.sdr-reprovision-controller.deploy Additional property command is not allowed

Line 32 needs to be replaced with the following:

sh -c ‘deepstream-test5-app -c /ds-config-files/pn26/ds-config-0_agx.txt -c /ds-config-files/pn26/ds-config-1_agx.txt 2>&1 | grep --line-buffered . | tee -a /log/deepstream.log’

Thank you so much @hfarooq and @kesong. Now its working