Runtime errors from Deepstream 7.1 via GXF on Jetson Xavier NX

Accelerated Computing Intelligent Video Analytics DeepStream SDK
,
1

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU): Jetson Xavier NX
• DeepStream Version: 7.1
• JetPack Version (valid for Jetson only): R35 (release), REVISION: 4.1
• TensorRT Version: 10.3.0
• NVIDIA GPU Driver Version (valid for GPU only): 550.107.02(CUDA version: 12.4)
• Issue Type( questions, new requirements, bugs): bugs
• Docker Version: Docker version 27.0.3-1, build 7d4bcd863a4c863e650eed02a550dfeb98560b83
• OS: Ubuntu 22.04.4 LTS (amd64)
• Graph Composer and Runtime Version: 4.1.0
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)

Whenever I run the deepstream-test1 reference graph example on my Jetson board with the following docker compose file

  deepstream-test1:
    image: <registry>:deepstream-test1-jetson
    runtime: nvidia
    network_mode: host
    environment:
      - DISPLAY=:0
    tty: true
    stdin_open: true
    volumes:
      - /tmp/.X11-unix:/tmp/.X11-unix:rw
      - /etc/localtime:/etc/localtime:ro
      - /etc/timezone:/etc/timezone:ro
      - /tmp/argus_socket:/tmp/argus_socket

I am presented with the following error:

Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: nvidia-container-runtime did not terminate successfully: exit status 1: time="2024-10-23T21:38:11-04:00" level=error msg="failed to create NVIDIA Container Runtime: failed to construct OCI spec modifier: requirements not met: unsatisfied condition: cuda>=12.2 (cuda=11.4)" : unknown

This seems self explanatory, but if I were to build from a prior base image such as nvcr.io/nvidia/deepstream-l4t:6.4-triton-multiarch, I get the following error while running the container:

deepstream-1  | 2024-10-24 04:05:20.643 INFO  gxf/gxe/gxe.cpp@183: Creating context
deepstream-1  | 2024-10-24 04:05:20.790 ERROR gxf/std/extension_loader.cpp@114: Failed to load extension /workspace/deepstream-test1/deepstream/nvdsbase/libgxf_nvdsbase.so Error: libnvbufsurface.so: cannot open shared object file: No such file or directory
deepstream-1  | 2024-10-24 04:05:20.791 ERROR gxf/gxe/gxe.cpp@255: LoadExtensionManifest Error: GXF_EXTENSION_FILE_NOT_FOUND
deepstream-1 exited with code 1

And the following if i use nvcr.io/nvidia/deepstream-l4t:6.2-triton

deepstream-1  | gxe: /lib/aarch64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by gxe)
deepstream-1  | gxe: /lib/aarch64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by gxe)
deepstream-1  | gxe: /lib/aarch64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by gxe)
deepstream-1  | gxe: /lib/aarch64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /opt/nvidia/graph-composer/libgxf_core.so)
deepstream-1  | gxe: /lib/aarch64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by /opt/nvidia/graph-composer/libgxf_core.so)
deepstream-1  | gxe: /lib/aarch64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by /opt/nvidia/graph-composer/libgxf_core.so)
deepstream-1 exited with code 1
3

How did you generate the “deepstream-test1-jetson” image? How did you run the image in your Jetson board?

4

Thanks for the reply. I used the container-builder cli as noted in the README enclosed with the reference graph, i.e. container_builder build -c ds_test1_container_builder_jetson.yaml -d aarch64 -wd $(pwd). I also made sure to update the docker_push.url field in order to push my image to my docker registry.

I ran the image using the compose file snippet in my earlier post via docker compose up deepstream-test1
The docker version on my Jetson board is Docker version 24.0.7, build 24.0.7-0ubuntu2~20.04.1

5

Tried in our AGX Orin board.

xhost +
sudo docker run -it --rm --net=host --runtime nvidia  -e DISPLAY=$DISPLAY -v /tmp/.X11-unix/:/tmp/.X11-unix deepstream-test1-jetson

It works.

6

Would you be kind enough to try on a device with JetPack version R35 (release), REVISION: 4.1 preferably on the Jetson Xavier NX. Also may I also know what base image you are using to build the example/changes to your container builder config?

7

DeepStream 7.1 version does not support Jetson Xavier NX.

Graph Composer 4.1.0 does not support Jetson Xavier NX either.

Please use DeepStream 6.3 and Graph Composer 3.0.0 with Jetson Xavier NX. Container Builder — DeepStream 6.3 Release documentation

8

I get the following error if i use Deepstream 6.3 with Graph Composer 3.0.0

deepstream-1  | gxe: /lib/aarch64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by gxe)
deepstream-1  | gxe: /lib/aarch64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by gxe)
deepstream-1  | gxe: /lib/aarch64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by gxe)
deepstream-1  | gxe: /lib/aarch64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /opt/nvidia/graph-composer/libgxf_core.so)
deepstream-1  | gxe: /lib/aarch64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by /opt/nvidia/graph-composer/libgxf_core.so)
deepstream-1  | gxe: /lib/aarch64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by /opt/nvidia/graph-composer/libgxf_core.so)
deepstream-1 exited with code 1

And the following if i use Deepstream 6.1 with Graph Composer 3.0.0

deepstream-1  | 2024-10-30 15:22:13.370 ERROR extensions/nvdssource/nvurisrcbin.hpp@36: Could not create GStreamer element 'nvurisrcbin'
deepstream-1  | 2024-10-30 15:22:13.370 ERROR gxf/std/program.cpp@519: Couldn't run async. Deactivating...
deepstream-1  | 2024-10-30 15:22:13.373 ERROR gxf/core/runtime.cpp@1392: Graph run failed with error: GXF_FAILURE
deepstream-1  | 2024-10-30 15:22:13.373 ERROR gxf/gxe/gxe.cpp@269: GxfGraphRunAsync Error: GXF_FAILURE
deepstream-1 exited with code 1
9

Is the deepstream-test1-jetson docker generate by Graph Composer 3.0.0 too?

10

Yes it is.

11

Graph Composer 3.0.0 can only work with DeepStream 6.3+JetPack 5.1.2 GA. It means the Jetson host should be installed with JP5.1.2 GA. Why did you mention DeepStream 6.1 + Graph Composer 3.0.0?

Please install your board with JP 5.1.2GA +DeepStream 6.3+ Graph Composer 3.0.0 and build the deepstream-test1 container according to Container Builder — DeepStream 6.3 Release documentation

Such errors show that the needed libraries are not built in the container or the gxf libraries copied to the container are not compatible to the base images.

12

Pardon the delay in responding, I used Deepstream 6.1 as the documentation seems to support its use, although it does not mention the version of the Graph Composer.

The JetPack SDK release notes for 5.1.2 mentions support for Deepstream 6.2. This can be found under Supported SDKs and Tools. I assumed this version should’ve been compatible. I decided to use the targeted release version (Deepstream 6.1) when i encountered errors with 6.2.

Moreover, it was my understanding that the container builder will find and use the appropriate version of the extensions populated via the registry. i.e. If my target device is aarch64 ubuntu_20.04, where my graph uses the NvDsInferenceUtilsExt extension, my minimum Deepstream version should be 6.3 or above as defined in its target.yaml(~/.cache/nvgraph_registry/NvDsInferenceUtilsExt/1.3.0/1.3.0-linux-aarch64-ubuntu_20.04-ds-6.3/target.yaml)

This led me to understand that in order to also get this to work on a graph designed in the latest DS versions, I had to manually change the extension versions in my graph to match the target required for Deepstream 6.3. i.e. Changed the version fields in my graph to 1.3.0.

The example almost works but i encounter the following error:

deepstream-1  | 2024-11-05 12:16:09.543 ERROR extensions/nvdssource/nvurisrcbin.hpp@36: Could not create GStreamer element 'nvurisrcbin'
deepstream-1  | 2024-11-05 12:16:09.543 ERROR gxf/std/program.cpp@519: Couldn't run async. Deactivating...
deepstream-1  | 2024-11-05 12:16:09.545 ERROR gxf/core/runtime.cpp@1392: Graph run failed with error: GXF_FAILURE
deepstream-1  | 2024-11-05 12:16:09.545 ERROR gxf/gxe/gxe.cpp@269: GxfGraphRunAsync Error: GXF_FAILURE
deepstream-1 exited with code 1
13

You may use “gst-inspect-1.0 nvurisrcbin” command to check whether nvurisrcbin is supported in the platform. There is no “nvurisrcbin” in DeepStream 6.1.

14

Pardon the delay. I seem to have fixed the issue. I had to run xhost + before launching my docker service. I made sure the extension versions in my graph were compatible with the Deepstream version(nvcr.io/nvidia/deepstream-l4t:6.3-samples in this case).

After some success with the deepstream-test2 graph, I updated the graph to use a csi camera source and an rtsp sink for output rather than the OSD renderer. The issues i encountered are:

  • the stream is either very slow or cannot be viewed externally on VLC. I have set my jetson device to the highest power setting
  • the docker service cannot be initiated via ssh. The service can only be initiated on the host device after running xhost +. Otherwise i run into errors like below. I have setup and confirmed X11 forwarding works.
deepstream-1  | 2024-11-15 00:11:51.295 ERROR extensions/nvdsbase/nvds_scheduler.cpp@312: Error from /GstPipeline:NvDsScheduler-Pipeline/GstDsNvInferBin:vehicle_type_classifier/vehicle_type_classifier/GstNvInfer:nvinfer_bin_nvinfer: Failed to set buffer pool to active
deepstream-1  | 2024-11-15 00:11:51.295 ERROR extensions/nvdsbase/nvds_scheduler.cpp@314: Debug info:
deepstream-1  | /dvs/git/dirty/git-master_linux/deepstream/sdk/src/gst-plugins/gst-nvinfer/gstnvinfer.cpp(994): gst_nvinfer_start (): /GstPipeline:NvDsScheduler-Pipeline/GstDsNvInferBin:vehicle_type_classifier/vehicle_type_classifier/GstNvInfer:nvinfer_bin_nvinfer
deepstream-1  | 2024-11-15 00:11:51.304 ERROR extensions/nvdsbase/nvds_scheduler.cpp@184: Failed to set GStreamer pipeline to PLAYING
deepstream-1  | Returned, stopping playback
15

DeepStream does not support X11 forwarding.

You can only run with the host machine.

16

Do you mean i can only launch Deepstream on the host machine even if I am using an RTSP output sink?

17

You can launch DeepStream graph with RTSP output.

18

Yes that is what I have done. I am able to play the slow stream on VLC Player for a few minutes before the stream closes. Any attempts after that doesn’t work. The stream buffers and eventually closes. I thought it could be due to connecting via wifi to my router, but the result is the same with an ethernet connection.

The container still logs information about the stream and published frames to the console. I have attached my deepstream-test2 camera source → rtsp out graph as a .txt file below.

deepstream-test2-rtsp.txt (7.3 KB)

19

What do you mean by “the stream closes”?

20

I am able to view the stream on VLC for about 2 minutes until the ends abruptly. The deepstream container does not log any errors to the console during this event.

21

There is no clue in your description for the sudden “ends”. Did you run the vlc player in the same device as where you ran the graph? Can you check the rtsp output with simple GStreamer pipeline such as “gst-launch-1.0 rtspsrc location=rtsp://xxxx ! rtph264depay ! fakesink”?