Deepstream python app test1 gstreamer error: <nvvideo-renderer> Cuda Init failed

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) Tesla T4
• DeepStream Version 6.1.1
• TensorRT Version 8.4.1.1+cuda11.6
• Gstreamer version: 1.16.3
• NVIDIA GPU Driver Version (valid for GPU only) 515.65.01
• CUDA: 11.7.1
• Issue Type( questions, new requirements, bugs) Question
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
GST_DEBUG=3 python3 deepstream_test_1.py /opt/nvidia/deepstream/deepstream-6.1/samples/streams/sample_720p.h264

Creating Pipeline

Creating Source

Creating H264Parser

Creating Decoder

Creating EGLSink

Playing file …/…/…/…/samples/streams/sample_720p.h264
Adding elements to Pipeline

Linking elements in the Pipeline

Starting pipeline

0:00:00.422970112 38555 0x2ee6aa0 WARN v4l2 gstv4l2object.c:3057:gst_v4l2_object_get_nearest_size:nvv4l2-decoder:sink Unable to try format: Unknown error -1
0:00:00.422990028 38555 0x2ee6aa0 WARN v4l2 gstv4l2object.c:2942:gst_v4l2_object_probe_caps_for_format:nvv4l2-decoder:sink Could not probe minimum capture size for pixelformat MJPG
0:00:00.422994874 38555 0x2ee6aa0 WARN v4l2 gstv4l2object.c:3057:gst_v4l2_object_get_nearest_size:nvv4l2-decoder:sink Unable to try format: Unknown error -1
0:00:00.422998884 38555 0x2ee6aa0 WARN v4l2 gstv4l2object.c:2948:gst_v4l2_object_probe_caps_for_format:nvv4l2-decoder:sink Could not probe maximum capture size for pixelformat MJPG
0:00:00.423009570 38555 0x2ee6aa0 WARN v4l2 gstv4l2object.c:3057:gst_v4l2_object_get_nearest_size:nvv4l2-decoder:sink Unable to try format: Unknown error -1
0:00:00.423013670 38555 0x2ee6aa0 WARN v4l2 gstv4l2object.c:2942:gst_v4l2_object_probe_caps_for_format:nvv4l2-decoder:sink Could not probe minimum capture size for pixelformat AV10
0:00:00.423016945 38555 0x2ee6aa0 WARN v4l2 gstv4l2object.c:3057:gst_v4l2_object_get_nearest_size:nvv4l2-decoder:sink Unable to try format: Unknown error -1
0:00:00.423021976 38555 0x2ee6aa0 WARN v4l2 gstv4l2object.c:2948:gst_v4l2_object_probe_caps_for_format:nvv4l2-decoder:sink Could not probe maximum capture size for pixelformat AV10
0:00:00.423030290 38555 0x2ee6aa0 WARN v4l2 gstv4l2object.c:3057:gst_v4l2_object_get_nearest_size:nvv4l2-decoder:sink Unable to try format: Unknown error -1
0:00:00.423034831 38555 0x2ee6aa0 WARN v4l2 gstv4l2object.c:2942:gst_v4l2_object_probe_caps_for_format:nvv4l2-decoder:sink Could not probe minimum capture size for pixelformat DVX5
0:00:00.423038038 38555 0x2ee6aa0 WARN v4l2 gstv4l2object.c:3057:gst_v4l2_object_get_nearest_size:nvv4l2-decoder:sink Unable to try format: Unknown error -1
0:00:00.423042200 38555 0x2ee6aa0 WARN v4l2 gstv4l2object.c:2948:gst_v4l2_object_probe_caps_for_format:nvv4l2-decoder:sink Could not probe maximum capture size for pixelformat DVX5
0:00:00.423074129 38555 0x2ee6aa0 WARN v4l2 gstv4l2object.c:3057:gst_v4l2_object_get_nearest_size:nvv4l2-decoder:sink Unable to try format: Unknown error -1
0:00:00.423078427 38555 0x2ee6aa0 WARN v4l2 gstv4l2object.c:2942:gst_v4l2_object_probe_caps_for_format:nvv4l2-decoder:sink Could not probe minimum capture size for pixelformat DVX4
0:00:00.423094987 38555 0x2ee6aa0 WARN v4l2 gstv4l2object.c:3057:gst_v4l2_object_get_nearest_size:nvv4l2-decoder:sink Unable to try format: Unknown error -1
0:00:00.423098297 38555 0x2ee6aa0 WARN v4l2 gstv4l2object.c:2948:gst_v4l2_object_probe_caps_for_format:nvv4l2-decoder:sink Could not probe maximum capture size for pixelformat DVX4
0:00:00.423105034 38555 0x2ee6aa0 WARN v4l2 gstv4l2object.c:3057:gst_v4l2_object_get_nearest_size:nvv4l2-decoder:sink Unable to try format: Unknown error -1
0:00:00.423108865 38555 0x2ee6aa0 WARN v4l2 gstv4l2object.c:2942:gst_v4l2_object_probe_caps_for_format:nvv4l2-decoder:sink Could not probe minimum capture size for pixelformat MPG4
0:00:00.423112007 38555 0x2ee6aa0 WARN v4l2 gstv4l2object.c:3057:gst_v4l2_object_get_nearest_size:nvv4l2-decoder:sink Unable to try format: Unknown error -1
0:00:00.423115391 38555 0x2ee6aa0 WARN v4l2 gstv4l2object.c:2948:gst_v4l2_object_probe_caps_for_format:nvv4l2-decoder:sink Could not probe maximum capture size for pixelformat MPG4
0:00:00.423121622 38555 0x2ee6aa0 WARN v4l2 gstv4l2object.c:3057:gst_v4l2_object_get_nearest_size:nvv4l2-decoder:sink Unable to try format: Unknown error -1
0:00:00.423125382 38555 0x2ee6aa0 WARN v4l2 gstv4l2object.c:2942:gst_v4l2_object_probe_caps_for_format:nvv4l2-decoder:sink Could not probe minimum capture size for pixelformat MPG2
0:00:00.423129564 38555 0x2ee6aa0 WARN v4l2 gstv4l2object.c:3057:gst_v4l2_object_get_nearest_size:nvv4l2-decoder:sink Unable to try format: Unknown error -1
0:00:00.423133977 38555 0x2ee6aa0 WARN v4l2 gstv4l2object.c:2948:gst_v4l2_object_probe_caps_for_format:nvv4l2-decoder:sink Could not probe maximum capture size for pixelformat MPG2
0:00:00.423161022 38555 0x2ee6aa0 WARN v4l2 gstv4l2object.c:3057:gst_v4l2_object_get_nearest_size:nvv4l2-decoder:sink Unable to try format: Unknown error -1
0:00:00.423165837 38555 0x2ee6aa0 WARN v4l2 gstv4l2object.c:2942:gst_v4l2_object_probe_caps_for_format:nvv4l2-decoder:sink Could not probe minimum capture size for pixelformat H265
0:00:00.423169929 38555 0x2ee6aa0 WARN v4l2 gstv4l2object.c:3057:gst_v4l2_object_get_nearest_size:nvv4l2-decoder:sink Unable to try format: Unknown error -1
0:00:00.423174319 38555 0x2ee6aa0 WARN v4l2 gstv4l2object.c:2948:gst_v4l2_object_probe_caps_for_format:nvv4l2-decoder:sink Could not probe maximum capture size for pixelformat H265
0:00:00.423181470 38555 0x2ee6aa0 WARN v4l2 gstv4l2object.c:3057:gst_v4l2_object_get_nearest_size:nvv4l2-decoder:sink Unable to try format: Unknown error -1
0:00:00.423185952 38555 0x2ee6aa0 WARN v4l2 gstv4l2object.c:2942:gst_v4l2_object_probe_caps_for_format:nvv4l2-decoder:sink Could not probe minimum capture size for pixelformat VP90
0:00:00.423190032 38555 0x2ee6aa0 WARN v4l2 gstv4l2object.c:3057:gst_v4l2_object_get_nearest_size:nvv4l2-decoder:sink Unable to try format: Unknown error -1
0:00:00.423193602 38555 0x2ee6aa0 WARN v4l2 gstv4l2object.c:2948:gst_v4l2_object_probe_caps_for_format:nvv4l2-decoder:sink Could not probe maximum capture size for pixelformat VP90
0:00:00.423199782 38555 0x2ee6aa0 WARN v4l2 gstv4l2object.c:3057:gst_v4l2_object_get_nearest_size:nvv4l2-decoder:sink Unable to try format: Unknown error -1
0:00:00.423203917 38555 0x2ee6aa0 WARN v4l2 gstv4l2object.c:2942:gst_v4l2_object_probe_caps_for_format:nvv4l2-decoder:sink Could not probe minimum capture size for pixelformat VP80
0:00:00.423207162 38555 0x2ee6aa0 WARN v4l2 gstv4l2object.c:3057:gst_v4l2_object_get_nearest_size:nvv4l2-decoder:sink Unable to try format: Unknown error -1
0:00:00.423225379 38555 0x2ee6aa0 WARN v4l2 gstv4l2object.c:2948:gst_v4l2_object_probe_caps_for_format:nvv4l2-decoder:sink Could not probe maximum capture size for pixelformat VP80
0:00:00.423231716 38555 0x2ee6aa0 WARN v4l2 gstv4l2object.c:3057:gst_v4l2_object_get_nearest_size:nvv4l2-decoder:sink Unable to try format: Unknown error -1
0:00:00.423250764 38555 0x2ee6aa0 WARN v4l2 gstv4l2object.c:2942:gst_v4l2_object_probe_caps_for_format:nvv4l2-decoder:sink Could not probe minimum capture size for pixelformat H264
0:00:00.423254555 38555 0x2ee6aa0 WARN v4l2 gstv4l2object.c:3057:gst_v4l2_object_get_nearest_size:nvv4l2-decoder:sink Unable to try format: Unknown error -1
0:00:00.423258738 38555 0x2ee6aa0 WARN v4l2 gstv4l2object.c:2948:gst_v4l2_object_probe_caps_for_format:nvv4l2-decoder:sink Could not probe maximum capture size for pixelformat H264
0:00:00.423542740 38555 0x2ee6aa0 WARN v4l2 gstv4l2object.c:3057:gst_v4l2_object_get_nearest_size:nvv4l2-decoder:src Unable to try format: Unknown error -1
0:00:00.423551165 38555 0x2ee6aa0 WARN v4l2 gstv4l2object.c:2942:gst_v4l2_object_probe_caps_for_format:nvv4l2-decoder:src Could not probe minimum capture size for pixelformat NM12
0:00:00.423555919 38555 0x2ee6aa0 WARN v4l2 gstv4l2object.c:3057:gst_v4l2_object_get_nearest_size:nvv4l2-decoder:src Unable to try format: Unknown error -1
0:00:00.423586063 38555 0x2ee6aa0 WARN v4l2 gstv4l2object.c:2948:gst_v4l2_object_probe_caps_for_format:nvv4l2-decoder:src Could not probe maximum capture size for pixelformat NM12
0:00:00.423593987 38555 0x2ee6aa0 WARN v4l2 gstv4l2object.c:2395:gst_v4l2_object_add_interlace_mode:0x247bd20 Failed to determine interlace mode
0:00:00.425070988 38555 0x2ee6aa0 WARN nvinfer gstnvinfer.cpp:643:gst_nvinfer_logger: NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::initialize() <nvdsinfer_context_impl.cpp:1170> [UID = 1]: Warning, OpenCV has been deprecated. Using NMS for clustering instead of cv::groupRectangles with topK = 20 and NMS Threshold = 0.5
0:00:01.613483720 38555 0x2ee6aa0 INFO nvinfer gstnvinfer.cpp:646:gst_nvinfer_logger: NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1909> [UID = 1]: deserialized trt engine from :/opt/nvidia/deepstream/deepstream-6.1/samples/models/Primary_Detector/resnet10.caffemodel_b1_gpu0_int8.engine
INFO: …/nvdsinfer/nvdsinfer_model_builder.cpp:610 [Implicit Engine Info]: layers num: 3
0 INPUT kFLOAT input_1 3x368x640
1 OUTPUT kFLOAT conv2d_bbox 16x23x40
2 OUTPUT kFLOAT conv2d_cov/Sigmoid 4x23x40

0:00:01.649419024 38555 0x2ee6aa0 INFO nvinfer gstnvinfer.cpp:646:gst_nvinfer_logger: NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2012> [UID = 1]: Use deserialized engine model: /opt/nvidia/deepstream/deepstream-6.1/samples/models/Primary_Detector/resnet10.caffemodel_b1_gpu0_int8.engine
0:00:01.650983837 38555 0x2ee6aa0 INFO nvinfer gstnvinfer_impl.cpp:328:notifyLoadModelStatus: [UID 1]: Load new model:dstest1_pgie_config.txt sucessfully
0:00:01.652428324 38555 0x2ee6aa0 WARN basesrc gstbasesrc.c:3600:gst_base_src_start_complete: pad not activated yet
cuGraphicsGLRegisterBuffer failed with error(219) gst_eglglessink_cuda_init texture = 1
0:00:01.976295458 38555 0x2d40800 ERROR nveglglessink ext/eglgles/gsteglglessink.c:2851:gst_eglglessink_configure_caps: Cuda Init failed
0:00:01.976303496 38555 0x2d40800 ERROR nveglglessink ext/eglgles/gsteglglessink.c:2862:gst_eglglessink_configure_caps: Configuring caps failed
0:00:01.976347390 38555 0x2d40920 ERROR nveglglessink ext/eglgles/gsteglglessink.c:2907:gst_eglglessink_setcaps: Failed to configure caps
0:00:01.976397530 38555 0x2d40920 ERROR nveglglessink ext/eglgles/gsteglglessink.c:2907:gst_eglglessink_setcaps: Failed to configure caps
0:00:01.976404049 38555 0x2d40920 WARN GST_PADS gstpad.c:4231:gst_pad_peer_query:onscreendisplay:src could not send sticky events
0:00:01.976776615 38555 0x2d40920 WARN v4l2videodec gstv4l2videodec.c:1847:gst_v4l2_video_dec_decide_allocation: Duration invalid, not setting latency
0:00:01.976814599 38555 0x2d40920 WARN v4l2bufferpool gstv4l2bufferpool.c:1082:gst_v4l2_buffer_pool_start:nvv4l2-decoder:pool:src Uncertain or not enough buffers, enabling copy threshold
0:00:01.976916861 38555 0x7f7ca80cbde0 ERROR nveglglessink ext/eglgles/gsteglglessink.c:2907:gst_eglglessink_setcaps: Failed to configure caps
0:00:01.977849772 38555 0x7f7ca80cbe40 WARN v4l2bufferpool gstv4l2bufferpool.c:1533:gst_v4l2_buffer_pool_dqbuf:nvv4l2-decoder:pool:src Driver should never set v4l2_buffer.field to ANY
0:00:01.977914489 38555 0x7f7ca80cbe40 ERROR nveglglessink ext/eglgles/gsteglglessink.c:2907:gst_eglglessink_setcaps: Failed to configure caps
0:00:01.978017861 38555 0x7f7ca80cbde0 ERROR nveglglessink ext/eglgles/gsteglglessink.c:2907:gst_eglglessink_setcaps: Failed to configure caps
0:00:01.978053042 38555 0x7f7ca80cbde0 ERROR nveglglessink ext/eglgles/gsteglglessink.c:2907:gst_eglglessink_setcaps: Failed to configure caps
0:00:01.978061325 38555 0x7f7ca80cbde0 ERROR nveglglessink ext/eglgles/gsteglglessink.c:2907:gst_eglglessink_setcaps: Failed to configure caps
0:00:01.980714848 38555 0x7f7ca80cbde0 ERROR nveglglessink ext/eglgles/gsteglglessink.c:2907:gst_eglglessink_setcaps: Failed to configure caps
0:00:01.980772856 38555 0x7f7ca80cbde0 ERROR nveglglessink ext/eglgles/gsteglglessink.c:2907:gst_eglglessink_setcaps: Failed to configure caps
0:00:01.980781855 38555 0x7f7ca80cbde0 ERROR nveglglessink ext/eglgles/gsteglglessink.c:2907:gst_eglglessink_setcaps: Failed to configure caps
Frame Number=0 Number of Objects=13 Vehicle_count=9 Person_count=4
0:00:02.003288231 38555 0x2d406a0 ERROR nveglglessink ext/eglgles/gsteglglessink.c:2907:gst_eglglessink_setcaps: Failed to configure caps
0:00:02.003322819 38555 0x2d406a0 WARN nvinfer gstnvinfer.cpp:2300:gst_nvinfer_output_loop: error: Internal data stream error.
0:00:02.003327359 38555 0x2d406a0 WARN nvinfer gstnvinfer.cpp:2300:gst_nvinfer_output_loop: error: streaming stopped, reason not-negotiated (-4)
Error: gst-stream-error-quark: Internal data stream error. (1): gstnvinfer.cpp(2300): gst_nvinfer_output_loop (): /GstPipeline:pipeline0/GstNvInfer:primary-inference:
streaming stopped, reason not-negotiated (-4)
Frame Number=1 Number of Objects=11 Vehicle_count=8 Person_count=3
Frame Number=2 Number of Objects=11 Vehicle_count=7 Person_count=4
0:00:02.009235449 38555 0x2d40920 WARN baseparse gstbaseparse.c:3666:gst_base_parse_loop: error: Internal data stream error.
0:00:02.009248409 38555 0x2d40920 WARN baseparse gstbaseparse.c:3666:gst_base_parse_loop: error: streaming stopped, reason not-negotiated (-4)
Frame Number=3 Number of Objects=11 Vehicle_count=7 Person_count=4
Frame Number=4 Number of Objects=11 Vehicle_count=8 Person_count=3
Frame Number=5 Number of Objects=12 Vehicle_count=8 Person_count=4
Frame Number=6 Number of Objects=11 Vehicle_count=7 Person_count=4

Can you double check your environment? The CUDA version should be 11.7 for DeepStream 6.1.1, please refer to the compatibility matrix: Quickstart Guide — DeepStream 6.1.1 Release documentation

I have installed CUDA Version 11.7 with update 1 as mentioned in the document
“Install CUDA Toolkit 11.7.1 (CUDA 11.7 Update 1) and NVIDIA driver 515.65.01”

I misunderstood it when saw tensorRT version as “8.4.1.1+cuda11.6”, sorry for it. Please help provide the output of nvidia-smi on your host.
Your card is Tesla series, please be noted that it’s compute card, not for display, you need use sink (for example Fakesink) or files or RTSP stream other than EGL sink. You can refer below example which uses fakesink sink:

deepstream-app -c /opt/nvidia/deepstream/deepstream-6.1/samples/configs/deepstream-app/source2_1080p_dec_infer-resnet_demux_int8.txt

Thank you Yingliu for the response.
Probably, the display driver for Xorg is installed on Tesla T4 as you mentioned.

I’ll check it out Monday and get back to you.

I’ve removed the xorg display on Tesla T4 and it is used for compute only now.

Without root, deepstream-app fails to start and with root Tensorrt doesn’t work.

The Python bindings of test1 gives me black screen output when I run “python3 deepstream-test1.py …/…/…/…/samples/streams/sample_720p.h264”

Any idea what’s wrong?

from the logs, it can’t find “libnvinfer.so”, can you find it in /usr/lib/x86_64-linux-gnu/libnvinfer.so?
please install dependency libs according to Quickstart Guide — DeepStream 6.1.1 Release documentation

I see the following when I try to install the TensorRT dependencies.
It upgrades to Cuda 11.8, which we don’t want.
Cleaning the apt cache didn’t help. How to avoid the Cuda 11.8 upgrade?

Note:
Because of the above issue, I’ve installed TensorRT 8.4.1.5 from the tar file and added the path of libnvinfer.so files to bashrc
So I am able to import tensorrt from python scripts only.

  • GStreamer 1.16.2
  • NVIDIA driver 515.65.01
  • CUDA 11.7 Update 1
  • TensorRT 8.4.1.5
    please uninstall all the above dependency libs, and install them step by step according to Quickstart Guide — DeepStream 6.1.1 Release documentation
    you can use these command to check the version:
    CUDA version nvcc -V
    dpkg -l |grep TensorRT
    dpkg -l | grep gstreamer
    dpkg -l|grep cudnn

The above issue is seen after I’ve reinstalled Ubuntu 20.04 and tried again by following the steps in the documentation
I need to know how to install TensorRT libraries without upgrading cuda from 11.7.1 to 11.8. Please let me know if you have any suggestions.

that cuda-ccl-11-8 is TensorRT 's dependency lib, not mean cuda will be upgraded to 11.8, you can use “nvcc -V” to get current cuda vesion.
or you try to install TensorRT first, then install cuda11.7?
or we suggest to use deepstream docker, which has installed all dependency libs, here is the link: DeepStream | NVIDIA NGC

Created the container from below command:

docker run -itd --net=host --runtime nvidia -e DISPLAY=$DISPLAY -w /opt/nvidia/deepstream/deepstream-6.1 -v /tmp/.X11-unix/:/tmp/.X11-unix nvcr.io/nvidia/deepstream:6.1.1-samples

Using the docker container, I’m facing the below issues:

deepstream-app -c /opt/nvidia/deepstream/deepstream-6.1/samples/configs/deepstream-app/source2_1080p_dec_infer-resnet_demux_int8.txt

The above command gives the below output
image

However, if other config files are used, they show blank window for a second and then end up with a segmentation fault

deepstream-app -c /opt/nvidia/deepstream/deepstream-6.1/samples/configs/deepstream-app/source30_1080p_dec_infer-resnet_tiled_display_int8.txt

  1. from your screenshot, that sample run successfully, 30 represents fps.
  2. about the second case, you can set sink display type to fakesink, it will not display video, the setting looks like:
    [sink0]
    enable=1
    #Type - 1=FakeSink 2=EglSink/nv3dsink (Jetson only) 3=File
    type=1
1 Like

Thanks Fanzh, deepstream-app runs fine but deepstream-test1-app doesn’t.
Could you tell me what could have caused Segmentation fault?

Similarly, When I try to run Object Detector Yolo, the engine created successfully but no output is seen. The Pipeline seems to run but ends with segmentation fault.

  1. please try to replace nveglglessink with fakesink in deepstream_test1_app.c, then rebuild and run.
  2. if still crash, please do “export GST_DEBUG=6”, then run again, send us the terminal logs.
  3. can you debug it? for example:
    $: gdb ./deepstream-test1-app
    $ set args ./deepstream-test1-app /opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.h264
    $ bt
    then send us the stack information.

I’ve replaced with fakesink and attached the log here
log.txt (5.5 MB)

App is running inside the docker container and I saved the log in a text file.
Let me know if there is a better way to share the log.

did you try this? need to check if it is related to nveglglessink sink.

Yes I have now enabled the fakesink and the previous log was without fakesink.
Please find the new log attached.
log.txt (4.7 MB)

gst_element_factory_create:e[00m creating element “nveglglessink” named “nvvideo-renderer”
from the logs, it still uses nveglglessink, did you rebuild the code after modifying to fakesink? and the command is: ./deepstream-test1-app /opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.h264

Perhaps, I missed to re run the make command inside the docker container. Sorry about that.
The new log for sample_720.h264 is attached here. Now I don’t see the nveglglessink in the log.
log_sample_720.txt (84.2 MB)