Deepstream 6.4 problem with gst registry cache

bongkyung.ko · January 4, 2024, 7:58am

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU)
GPU, A6000, RTX2080 Ti, RTX8000
• DeepStream Version
6.4
• JetPack Version (valid for Jetson only)
• TensorRT Version
8.6.1
• NVIDIA GPU Driver Version (valid for GPU only)
545.23.08
• Issue Type( questions, new requirements, bugs)
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)

The second run of deepstream-test1 experiences a hang-up during the source creation phase.

prepare user_deepstream_python_apps_install.sh which is located in nvcr.io/nvidia/deepstream:6.4-gc-triton-devel(/opt/nvidia/deepstream/deepstream/user_deepstream_python_apps_install.sh)
run deepstream 6.4 container.
$ docker run -it --rm --gpus all nvcr.io/nvidia/deepstream:6.4-samples-multiarch
copy user_deepstream_python_apps_install.sh to the container
install python requirements.
$ bash user_deepstream_python_apps_install.sh --version 1.1.10
run the sample app(deepstream-test1). It works.

$ cd /opt/nvidia/deepstream/deepstream-6.4/sources/deepstream_python_apps/apps/deepstream-test1; python3 deepstream_test_1.py /opt/nvidia/deepstream/deepstream-6.4/samples/streams/sample_1080p_h264.mp4

Creating Pipeline

Creating Source

Creating H264Parser

Creating Decoder

Creating EGLSink

Playing file /opt/nvidia/deepstream/deepstream-6.4/samples/streams/sample_1080p_h264.mp4
Adding elements to Pipeline

Linking elements in the Pipeline

/opt/nvidia/deepstream/deepstream-6.4/sources/deepstream_python_apps/apps/deepstream-test1/deepstream_test_1.py:220: DeprecationWarning: Gst.Element.get_request_pad is deprecated
sinkpad = streammux.get_request_pad(“sink_0”)
Starting pipeline

(omitted…)
streaming stopped, reason not-negotiated (-4)
nvstreammux: Successfully handled EOS for source_id=0

(DONE)

run the sample app again(deepstream-test1).
$ cd /opt/nvidia/deepstream/deepstream-6.4/sources/deepstream_python_apps/apps/deepstream-test1; python3 deepstream_test_1.py /opt/nvidia/deepstream/deepstream-6.4/samples/streams/sample_1080p_h264.mp4

Creating Pipeline

Creating Source

(hang forever…)

I found two workarounds.

removing gst registry cache(~/.cache/gstreamer-1.0/registry.x86_64.bin)
disabling the registry with GST_REGISTRY_UPDATE=no

• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

It was okay with the latest cpu(Intel 12th Gen).
It seems the older cpus(Intel 8th, 10th) cannot support some instructions used in deepstream 6.4

yuweiw · January 5, 2024, 3:26am

Could you use the deepstream:6.4-triton-multiarch docker and the latest python demo code? You’d better use a matching version between the demo code and DeepStream.

bongkyung.ko · January 5, 2024, 4:14am

Thanks for reply.

I’ve tried your suggestion. but it didn’t work.

yuweiw · January 8, 2024, 8:59am

I use the deepstream:6.4-triton-multiarch docker.

1.$ cd /opt/nvidia/deepstream/deepstream
2.$ ./user_deepstream_python_apps_install.sh --build-bindings -r master
3.$ cd /opt/nvidia/deepstream/deepstream-6.4/sources/deepstream_python_apps/apps/deepstream-test1
4. $python3 deepstream_test_1.py /opt/nvidia/deepstream/deepstream-6.4/samples/streams/sample_720p.h264

It works well. Are your steps the same as mine?

vtyw93 · January 9, 2024, 12:57am

This seems to be the same issue as this (which I also experience): DeepStream 6.4 - Gst-nvstreammux Plugin Hangs Indefinitely

bongkyung.ko · January 9, 2024, 8:52am

sorry, building bindings was failed with error.

./user_deepstream_python_apps_install.sh --build-bindings -r master

-- Build files have been written to: /opt/nvidia/deepstream/deepstream/sources/deepstream_python_apps/bindings/build
[  6%] Building CXX object CMakeFiles/pyds.dir/src/bindgstnvdsmeta.cpp.o
[ 12%] Building CXX object CMakeFiles/pyds.dir/src/utils.cpp.o
[ 18%] Building CXX object CMakeFiles/pyds.dir/src/pyds.cpp.o
[ 25%] Building CXX object CMakeFiles/pyds.dir/src/bindmeta360.cpp.o
[ 31%] Building CXX object CMakeFiles/pyds.dir/src/bindfunctions.cpp.o
[ 37%] Building CXX object CMakeFiles/pyds.dir/src/bindanalyticsmeta.cpp.o
[ 43%] Building CXX object CMakeFiles/pyds.dir/src/bindnvbufsurface.cpp.o
[ 56%] Building CXX object CMakeFiles/pyds.dir/src/bindnvosd.cpp.o
[ 56%] Building CXX object CMakeFiles/pyds.dir/src/bindnvdsinfer.cpp.o
[ 62%] Building CXX object CMakeFiles/pyds.dir/src/bindnvdsmeta.cpp.o
[ 75%] Building CXX object CMakeFiles/pyds.dir/src/bindschema.cpp.o
[ 75%] Building CXX object CMakeFiles/pyds.dir/src/bindtrackermeta.cpp.o
[ 81%] Building CXX object CMakeFiles/pyds.dir/src/bindopticalflow.cpp.o
[ 87%] Building CXX object CMakeFiles/pyds.dir/src/custom_binding/bindcustom.cpp.o
In file included from /opt/nvidia/deepstream/deepstream/sources/deepstream_python_apps/bindings/src/utils.cpp:26:
/opt/nvidia/deepstream/deepstream/sources/includes/gstnvdsmeta.h:43:10: fatal error: gst/video/video.h: No such file or directory
   43 | #include <gst/video/video.h>
      |          ^~~~~~~~~~~~~~~~~~~
compilation terminated.
make[2]: *** [CMakeFiles/pyds.dir/build.make:90: CMakeFiles/pyds.dir/src/utils.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....
In file included from /opt/nvidia/deepstream/deepstream/sources/deepstream_python_apps/bindings/include/pyds.hpp:29,
                 from /opt/nvidia/deepstream/deepstream/sources/deepstream_python_apps/bindings/include/bind/bindmeta360.hpp:22,
                 from /opt/nvidia/deepstream/deepstream/sources/deepstream_python_apps/bindings/src/bindmeta360.cpp:20:
/opt/nvidia/deepstream/deepstream/sources/includes/gstnvdsmeta.h:43:10: fatal error: gst/video/video.h: No such file or directory
   43 | #include <gst/video/video.h>
      |          ^~~~~~~~~~~~~~~~~~~
compilation terminated.
In file included from /opt/nvidia/deepstream/deepstream/sources/deepstream_python_apps/bindings/include/pyds.hpp:29,
                 from /opt/nvidia/deepstream/deepstream/sources/deepstream_python_apps/bindings/include/bind/bindnvbufsurface.hpp:22,
                 from /opt/nvidia/deepstream/deepstream/sources/deepstream_python_apps/bindings/src/bindnvbufsurface.cpp:20:
/opt/nvidia/deepstream/deepstream/sources/includes/gstnvdsmeta.h:43:10: fatal error: gst/video/video.h: No such file or directory
   43 | #include <gst/video/video.h>
      |          ^~~~~~~~~~~~~~~~~~~
compilation terminated.
In file included from /opt/nvidia/deepstream/deepstream/sources/deepstream_python_apps/bindings/include/pyds.hpp:29,
                 from /opt/nvidia/deepstream/deepstream/sources/deepstream_python_apps/bindings/include/bind/bindnvdsinfer.hpp:22,
                 from /opt/nvidia/deepstream/deepstream/sources/deepstream_python_apps/bindings/src/bindnvdsinfer.cpp:20:
/opt/nvidia/deepstream/deepstream/sources/includes/gstnvdsmeta.h:43:10: fatal error: gst/video/video.h: No such file or directory
   43 | #include <gst/video/video.h>
      |          ^~~~~~~~~~~~~~~~~~~
compilation terminated.
make[2]: *** [CMakeFiles/pyds.dir/build.make:146: CMakeFiles/pyds.dir/src/bindmeta360.cpp.o] Error 1
In file included from /opt/nvidia/deepstream/deepstream/sources/deepstream_python_apps/bindings/include/pyds.hpp:29,
                 from /opt/nvidia/deepstream/deepstream/sources/deepstream_python_apps/bindings/include/bind/bindanalyticsmeta.hpp:23,
                 from /opt/nvidia/deepstream/deepstream/sources/deepstream_python_apps/bindings/src/pyds.cpp:19:
/opt/nvidia/deepstream/deepstream/sources/includes/gstnvdsmeta.h:43:10: fatal error: gst/video/video.h: No such file or directory
   43 | #include <gst/video/video.h>
      |          ^~~~~~~~~~~~~~~~~~~
compilation terminated.
make[2]: *** [CMakeFiles/pyds.dir/build.make:160: CMakeFiles/pyds.dir/src/bindnvbufsurface.cpp.o] Error 1
make[2]: *** [CMakeFiles/pyds.dir/build.make:174: CMakeFiles/pyds.dir/src/bindnvdsinfer.cpp.o] Error 1
make[2]: *** [CMakeFiles/pyds.dir/build.make:76: CMakeFiles/pyds.dir/src/pyds.cpp.o] Error 1
In file included from /opt/nvidia/deepstream/deepstream/sources/deepstream_python_apps/bindings/include/pyds.hpp:29,
                 from /opt/nvidia/deepstream/deepstream/sources/deepstream_python_apps/bindings/include/bind/bindanalyticsmeta.hpp:23,
                 from /opt/nvidia/deepstream/deepstream/sources/deepstream_python_apps/bindings/src/bindanalyticsmeta.cpp:21:
/opt/nvidia/deepstream/deepstream/sources/includes/gstnvdsmeta.h:43:10: fatal error: gst/video/video.h: No such file or directory
   43 | #include <gst/video/video.h>
      |          ^~~~~~~~~~~~~~~~~~~
compilation terminated.
In file included from /opt/nvidia/deepstream/deepstream/sources/deepstream_python_apps/bindings/include/pyds.hpp:29,
                 from /opt/nvidia/deepstream/deepstream/sources/deepstream_python_apps/bindings/src/custom_binding/include/bindcustom.hpp:20,
                 from /opt/nvidia/deepstream/deepstream/sources/deepstream_python_apps/bindings/src/custom_binding/bindcustom.cpp:21:
/opt/nvidia/deepstream/deepstream/sources/includes/gstnvdsmeta.h:43:10: fatal error: gst/video/video.h: No such file or directory
   43 | #include <gst/video/video.h>
      |          ^~~~~~~~~~~~~~~~~~~
compilation terminated.
In file included from /opt/nvidia/deepstream/deepstream/sources/deepstream_python_apps/bindings/include/utils.hpp:30,
                 from /opt/nvidia/deepstream/deepstream/sources/deepstream_python_apps/bindings/src/bindnvosd.cpp:21:
/opt/nvidia/deepstream/deepstream/sources/includes/gstnvdsmeta.h:43:10: fatal error: gst/video/video.h: No such file or directory
   43 | #include <gst/video/video.h>
      |          ^~~~~~~~~~~~~~~~~~~
compilation terminated.
make[2]: *** [CMakeFiles/pyds.dir/build.make:104: CMakeFiles/pyds.dir/src/bindanalyticsmeta.cpp.o] Error 1
make[2]: *** [CMakeFiles/pyds.dir/build.make:258: CMakeFiles/pyds.dir/src/custom_binding/bindcustom.cpp.o] Error 1
make[2]: *** [CMakeFiles/pyds.dir/build.make:202: CMakeFiles/pyds.dir/src/bindnvosd.cpp.o] Error 1
In file included from /opt/nvidia/deepstream/deepstream/sources/deepstream_python_apps/bindings/include/utils.hpp:30,
                 from /opt/nvidia/deepstream/deepstream/sources/deepstream_python_apps/bindings/include/bind/bindfunctions.hpp:26,
                 from /opt/nvidia/deepstream/deepstream/sources/deepstream_python_apps/bindings/src/bindfunctions.cpp:19:
/opt/nvidia/deepstream/deepstream/sources/includes/gstnvdsmeta.h:43:10: fatal error: gst/video/video.h: No such file or directory
   43 | #include <gst/video/video.h>
      |          ^~~~~~~~~~~~~~~~~~~
compilation terminated.
make[2]: *** [CMakeFiles/pyds.dir/build.make:118: CMakeFiles/pyds.dir/src/bindfunctions.cpp.o] Error 1
In file included from /opt/nvidia/deepstream/deepstream/sources/deepstream_python_apps/bindings/include/pyds.hpp:29,
                 from /opt/nvidia/deepstream/deepstream/sources/deepstream_python_apps/bindings/include/bind/bindopticalflow.hpp:22,
                 from /opt/nvidia/deepstream/deepstream/sources/deepstream_python_apps/bindings/src/bindopticalflow.cpp:21:
/opt/nvidia/deepstream/deepstream/sources/includes/gstnvdsmeta.h:43:10: fatal error: gst/video/video.h: No such file or directory
   43 | #include <gst/video/video.h>
      |          ^~~~~~~~~~~~~~~~~~~
compilation terminated.
make[2]: *** [CMakeFiles/pyds.dir/build.make:216: CMakeFiles/pyds.dir/src/bindopticalflow.cpp.o] Error 1
In file included from /opt/nvidia/deepstream/deepstream/sources/deepstream_python_apps/bindings/include/pyds.hpp:29,
                 from /opt/nvidia/deepstream/deepstream/sources/deepstream_python_apps/bindings/include/bind/bindtrackermeta.hpp:22,
                 from /opt/nvidia/deepstream/deepstream/sources/deepstream_python_apps/bindings/src/bindtrackermeta.cpp:21:
/opt/nvidia/deepstream/deepstream/sources/includes/gstnvdsmeta.h:43:10: fatal error: gst/video/video.h: No such file or directory
   43 | #include <gst/video/video.h>
      |          ^~~~~~~~~~~~~~~~~~~
compilation terminated.
In file included from /opt/nvidia/deepstream/deepstream/sources/deepstream_python_apps/bindings/include/pyds.hpp:29,
                 from /opt/nvidia/deepstream/deepstream/sources/deepstream_python_apps/bindings/include/bind/bindschema.hpp:23,
                 from /opt/nvidia/deepstream/deepstream/sources/deepstream_python_apps/bindings/src/bindschema.cpp:21:
/opt/nvidia/deepstream/deepstream/sources/includes/gstnvdsmeta.h:43:10: fatal error: gst/video/video.h: No such file or directory
   43 | #include <gst/video/video.h>
      |          ^~~~~~~~~~~~~~~~~~~
compilation terminated.
make[2]: *** [CMakeFiles/pyds.dir/build.make:244: CMakeFiles/pyds.dir/src/bindtrackermeta.cpp.o] Error 1
In file included from /opt/nvidia/deepstream/deepstream/sources/deepstream_python_apps/bindings/include/pyds.hpp:29,
                 from /opt/nvidia/deepstream/deepstream/sources/deepstream_python_apps/bindings/include/bind/bindnvdsmeta.hpp:22,
                 from /opt/nvidia/deepstream/deepstream/sources/deepstream_python_apps/bindings/src/bindnvdsmeta.cpp:21:
/opt/nvidia/deepstream/deepstream/sources/includes/gstnvdsmeta.h:43:10: fatal error: gst/video/video.h: No such file or directory
   43 | #include <gst/video/video.h>
      |          ^~~~~~~~~~~~~~~~~~~
compilation terminated.
In file included from /opt/nvidia/deepstream/deepstream/sources/deepstream_python_apps/bindings/include/pyds.hpp:29,
                 from /opt/nvidia/deepstream/deepstream/sources/deepstream_python_apps/bindings/include/bind/bindgstnvdsmeta.hpp:20,
                 from /opt/nvidia/deepstream/deepstream/sources/deepstream_python_apps/bindings/src/bindgstnvdsmeta.cpp:21:
/opt/nvidia/deepstream/deepstream/sources/includes/gstnvdsmeta.h:43:10: fatal error: gst/video/video.h: No such file or directory
   43 | #include <gst/video/video.h>
      |          ^~~~~~~~~~~~~~~~~~~
compilation terminated.
make[2]: *** [CMakeFiles/pyds.dir/build.make:230: CMakeFiles/pyds.dir/src/bindschema.cpp.o] Error 1
make[2]: *** [CMakeFiles/pyds.dir/build.make:188: CMakeFiles/pyds.dir/src/bindnvdsmeta.cpp.o] Error 1
make[2]: *** [CMakeFiles/pyds.dir/build.make:132: CMakeFiles/pyds.dir/src/bindgstnvdsmeta.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:85: CMakeFiles/pyds.dir/all] Error 2
make: *** [Makefile:91: all] Error 2
###########################
Installing built PyDS wheel
###########################
WARNING: Requirement './pyds-1*_x86_64.whl' looks like a filename, but the file does not exist
ERROR: pyds-1*_x86_64.whl is not a valid wheel filename.

yuweiw · January 9, 2024, 9:03am

OK. You can just install the whl yourself. Also if you use the deepstream-test1, please use a h264 stream as the source.

python3 deepstream_test_1.py /opt/nvidia/deepstream/deepstream-6.4/samples/streams/sample_720p.h264

bongkyung.ko · January 10, 2024, 10:02am

I could build pyds bindings with your command.

installing to build/bdist.linux-x86_64/wheel
running install
running install_lib
creating build/bdist.linux-x86_64
creating build/bdist.linux-x86_64/wheel
copying build/lib/pyds.so -> build/bdist.linux-x86_64/wheel
running install_egg_info
running egg_info
creating pyds.egg-info
writing pyds.egg-info/PKG-INFO
writing dependency_links to pyds.egg-info/dependency_links.txt
writing requirements to pyds.egg-info/requires.txt
writing top-level names to pyds.egg-info/top_level.txt
writing manifest file 'pyds.egg-info/SOURCES.txt'
reading manifest file 'pyds.egg-info/SOURCES.txt'
writing manifest file 'pyds.egg-info/SOURCES.txt'
Copying pyds.egg-info to build/bdist.linux-x86_64/wheel/pyds-1.1.10-py3.10.egg-info
running install_scripts
creating build/bdist.linux-x86_64/wheel/pyds-1.1.10.dist-info/WHEEL
creating 'dist/pyds-1.1.10-py3-none-linux_x86_64.whl' and adding 'build/bdist.linux-x86_64/wheel' to it
adding 'pyds.so'
adding 'pyds-1.1.10.dist-info/METADATA'
adding 'pyds-1.1.10.dist-info/WHEEL'
adding 'pyds-1.1.10.dist-info/top_level.txt'
adding 'pyds-1.1.10.dist-info/RECORD'
removing build/bdist.linux-x86_64/wheel
make[2]: Leaving directory '/opt/nvidia/deepstream/deepstream-6.4/sources/deepstream_python_apps/bindings/build'
[100%] Built target pip_wheel
make[1]: Leaving directory '/opt/nvidia/deepstream/deepstream-6.4/sources/deepstream_python_apps/bindings/build'
###########################
Installing built PyDS wheel
###########################
Processing ./pyds-1.1.10-py3-none-linux_x86_64.whl
Requirement already satisfied: pgi in /usr/local/lib/python3.10/dist-packages (from pyds==1.1.10) (0.0.11.2)
Requirement already satisfied: PyGObject in /usr/local/lib/python3.10/dist-packages (from pyds==1.1.10) (3.46.0)
Requirement already satisfied: pycairo>=1.16.0 in /usr/local/lib/python3.10/dist-packages (from PyGObject->pyds==1.1.10) (1.25.1)
Installing collected packages: pyds
Successfully installed pyds-1.1.10

however it doesn’t help with same output(hang).

Levi_Pereira · January 10, 2024, 3:53pm

What is your OS Host?
I did two tests:

RTX 4090 (with docker ds-6.4) - OS Host Ubuntu 20.04.6 LTS (Failed)
NVIDIA L4 (with docker ds-6.4) - OS Host Ubuntu 22.04.3 LTS (Works)

Maybe pyds 1.1.10 have some issue on Host Ubuntu 20.04.6 LTS.

yuweiw · January 11, 2024, 1:48am

Hi @bongkyung.ko Have you changed your source in your pipeline?

python3 deepstream_test_1.py \
/opt/nvidia/deepstream/deepstream-6.4/samples/streams/sample_720p.h264

Deepstream_test_1 only supports the h264 stream, not the mp4 stream.

vtyw93 · January 11, 2024, 2:03am

That could be it. I’m all the way back on Ubuntu 18.04.6 LTS.

bongkyung.ko · January 11, 2024, 3:30am

yes I did.

bongkyung.ko · January 11, 2024, 3:32am

It was ubuntu 20.04
~~After upgrade to 22.04, the issue still persists.~~ I was confused sorry.

It worked!
deepstream 6.4 image requires ubuntu 22.04 on host system.

Levi_Pereira · January 11, 2024, 7:46pm

Confirmed. There is an incompatibility when using a Ubuntu Docker image with a version lower than that of the physical host.

system · January 25, 2024, 7:47pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Deepstream setup issues DeepStream SDK	6	2733	February 23, 2022
"reason not-negotiated (-4)" python deepstream-app1 is stopped DeepStream SDK	2	609	December 26, 2022
DeepStream 6.4 - Gst-nvstreammux Plugin Hangs Indefinitely DeepStream SDK	15	961	January 12, 2024
Can not run deepstream_test_1.py? DeepStream SDK samples	7	125	August 9, 2024
Deepstream5.1 issue with RedHat8 DeepStream SDK gstreamer	14	1797	October 12, 2021
DeepStream 2.0 error in running gstreamer，Can anyone solve this problem？ thanks DeepStream SDK	16	2722	September 7, 2023
Deepstream 6.1: deepstream-app not working after install DeepStream SDK deepstream61	9	2026	September 5, 2022
Deepstream_test_1.py doesn`t work DeepStream SDK	23	1514	December 12, 2022
Deepstream 6.2 nvdcf tracker DeepStream SDK	29	1026	April 29, 2024
How to run a sample app? DeepStream SDK	16	2324	October 12, 2021

Deepstream 6.4 problem with gst registry cache

Related topics