Cross-compile Deepstream Docker Image for Jetson on x86_64

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

• dGPU
• Deepstream 6.4
• 552.74
• Questions
• Why do we need to do this? We would like to automate the the build our of Deepstream application for both x86_64 DGPU and ARM64 Jetson platforms on one workstation. We have a custom gst plugin that we based off of gst-dsexample that creates a PIP(Picture-in-Picture) with a scaled close up of the detected object in the bottom right of the frame.
• How to reproduce the issue ?

We are looking to automate the build of our Jetson Deepstream Docker Image on an x86_64 platform using QEMU. First thing that stands out is that when I run the following image on my x86_64 device is I have broken links for libnvbufsurface and libnvbufsurftransform:
docker run --platform linux/arm64 -it --runtime=nvidia nvcr.io/nvidia/deepstream:6.4-triton-multiarch /bin/bash
cd /opt/nvidia/deepstream/deepstream-6.4/lib
ls -l # Show’s the broken linkage because the /usr/lib/aarch64-linux-gnu/tegra is broken or empty

Looking around in this docker image the /usr/lib/aarch64-linux-gnu/tegra folder seems to be missing or broken as it has a “red” look.

A simple reproduce is to try and build gst-example plugin like so:
This assumes you are inside of the above container.

cd /opt/nvidia/deepstream/deepstream-6.4/sources/gst-plugins/gst-dsexample
# Install dependencies for plugin
apt-get update
apt-get install libgstreamer-plugins-base1.0-dev libgstreamer1.0-dev libopencv-dev
export CUDA_VER=12.2
make -j

You will see that it fails to build because it can’t find libnvbufsurface and libnvbufsurftransform. Is it possible to cross-compile like this? Where does the /usr/lib/aarch64-linux-gnu/tegra folder come from exactly?

Thanks

3

libnvbufsurftransform/libnvbufsurface are user space drivers.

They are part of Jetpack and are only mounted from the host to the container at runtime.

You can find the /etc/nvidia-container-runtime/host-files-for-container.d/drivers.csv on Jetson.

If you want to cross-compile on x86, copy libnvbufsurftransform/libnvbufsurface to a new image base on 6.4-triton-multiarch and then use that image.

4

Interesting, thanks for the information. So where exactly should I copy them into? The same location the drivers.csv specifies which is here?
sym, /usr/lib/aarch64-linux-gnu/nvidia/libnvbufsurface.so
sym, /usr/lib/aarch64-linux-gnu/nvidia/libnvbufsurftransform.so

5

Copy the shared library to the corresponding location of docker, and then save the image
Such as add the following entries in the dockerfile, then build and export the custom image.

mkdir /usr/lib/aarch64-linux-gnu/nvidia/
cp /usr/lib/aarch64-linux-gnu/nvidia/libnvbufsurface.so /usr/lib/aarch64-linux-gnu/nvidia/
6

There were a lot of dependencies that I also had to pull in because I was getting build errors when only copying in libnvbufsurftransform/libnvbufsurface, in my Dockerfile I had to add the following:

# Copy all required library files to both directories
COPY tegra/libnvbufsurface.so.1.0.0 /usr/lib/aarch64-linux-gnu/nvidia/
COPY tegra/libnvbufsurftransform.so.1.0.0 /usr/lib/aarch64-linux-gnu/nvidia/
COPY tegra/libnvrm_mem.so* /usr/lib/aarch64-linux-gnu/nvidia/
COPY tegra/libnvrm_surface.so* /usr/lib/aarch64-linux-gnu/nvidia/
COPY tegra/libnvrm_chip.so* /usr/lib/aarch64-linux-gnu/nvidia/
COPY tegra/libnvos.so* /usr/lib/aarch64-linux-gnu/nvidia/
COPY tegra/libnvbuf_fdmap.so.1.0.0 /usr/lib/aarch64-linux-gnu/nvidia/
COPY tegra/libnvrm_gpu.so* /usr/lib/aarch64-linux-gnu/nvidia/
COPY tegra/libnvrm_host1x.so* /usr/lib/aarch64-linux-gnu/nvidia/
COPY tegra/libnvvic.so* /usr/lib/aarch64-linux-gnu/nvidia/
COPY tegra/libnvdla_compiler.so* /usr/lib/aarch64-linux-gnu/nvidia/
COPY tegra/libnvcudla.so* /usr/lib/aarch64-linux-gnu/nvidia/
# Additional dependencies
COPY tegra/libnvsciipc.so* /usr/lib/aarch64-linux-gnu/nvidia/
COPY tegra/libnvsocsys.so* /usr/lib/aarch64-linux-gnu/nvidia/
COPY tegra/libnvrm_sync.so* /usr/lib/aarch64-linux-gnu/nvidia/
COPY tegra/libnvtegrahv.so* /usr/lib/aarch64-linux-gnu/nvidia/
COPY tegra/libnvrm_stream.so* /usr/lib/aarch64-linux-gnu/nvidia/
COPY tegra/libnvcolorutil.so* /usr/lib/aarch64-linux-gnu/nvidia/
COPY tegra/libcuda.so* /usr/lib/aarch64-linux-gnu/nvidia/
COPY tegra/libnvdla_runtime.so* /usr/lib/aarch64-linux-gnu/nvidia/

# Copy the same files to tegra directory
COPY tegra/libnvbufsurface.so.1.0.0 /usr/lib/aarch64-linux-gnu/tegra/
COPY tegra/libnvbufsurftransform.so.1.0.0 /usr/lib/aarch64-linux-gnu/tegra/
COPY tegra/libnvrm_mem.so* /usr/lib/aarch64-linux-gnu/tegra/
COPY tegra/libnvrm_surface.so* /usr/lib/aarch64-linux-gnu/tegra/
COPY tegra/libnvrm_chip.so* /usr/lib/aarch64-linux-gnu/tegra/
COPY tegra/libnvos.so* /usr/lib/aarch64-linux-gnu/tegra/
COPY tegra/libnvbuf_fdmap.so.1.0.0 /usr/lib/aarch64-linux-gnu/tegra/
COPY tegra/libnvrm_gpu.so* /usr/lib/aarch64-linux-gnu/tegra/
COPY tegra/libnvrm_host1x.so* /usr/lib/aarch64-linux-gnu/tegra/
COPY tegra/libnvvic.so* /usr/lib/aarch64-linux-gnu/tegra/
COPY tegra/libnvdla_compiler.so* /usr/lib/aarch64-linux-gnu/tegra/
COPY tegra/libnvcudla.so* /usr/lib/aarch64-linux-gnu/tegra/
# Additional dependencies in tegra
COPY tegra/libnvsciipc.so* /usr/lib/aarch64-linux-gnu/tegra/
COPY tegra/libnvsocsys.so* /usr/lib/aarch64-linux-gnu/tegra/
COPY tegra/libnvrm_sync.so* /usr/lib/aarch64-linux-gnu/tegra/
COPY tegra/libnvtegrahv.so* /usr/lib/aarch64-linux-gnu/tegra/
COPY tegra/libnvrm_stream.so* /usr/lib/aarch64-linux-gnu/tegra/
COPY tegra/libnvcolorutil.so* /usr/lib/aarch64-linux-gnu/tegra/
COPY tegra/libcuda.so* /usr/lib/aarch64-linux-gnu/tegra/
COPY tegra/libnvdla_runtime.so* /usr/lib/aarch64-linux-gnu/tegra/

# Create symbolic links for all libraries in nvidia directory
RUN cd /usr/lib/aarch64-linux-gnu/nvidia && \
    for f in *.so.*; do \
    base=$(echo $f | sed 's/\([^.]*\).so.*/\1.so/'); \
    ln -sf $f $base; \
    done

# Create symbolic links for all libraries in tegra directory
RUN cd /usr/lib/aarch64-linux-gnu/tegra && \
    for f in *.so.*; do \
    base=$(echo $f | sed 's/\([^.]*\).so.*/\1.so/'); \
    ln -sf $f $base; \
    done

RUN ldconfig

Is there a better way then this? I got all of these from the Jetson devices /usr/lib/aarch64-linux-gnu/tegra/ folder.

Running the Docker image I cross-compiled I am seeing a couple warnings that I don’t see when building the image on the Jetson device:

xod-optical-deepstream-onvif  | /bin/bash: line 1: lsmod: command not found
xod-optical-deepstream-onvif  | /bin/bash: line 1: modprobe: command not found

Are these of any concern?

7

Yeah it looks like not having lsmod and modprobe cause the gst-dsexample to fail at runtime because it’s unable to determine that the GPU is integrated instead of dGPU. I am getting this runtime error:

xod-optical-deepstream-onvif  | /dvs/git/dirty/git-master_linux/nvutils/nvbufsurftransform/nvbufsurftransform.cpp:4550: => Surface type not supported for transformation NVBUF_MEM_CUDA_PINNED
xod-optical-deepstream-onvif  | 
xod-optical-deepstream-onvif  | 0:00:13.736806334     1 0xffff1c001d20 ERROR         nvvideoconvert gstnvvideoconvert.c:4208:gst_nvvideoconvert_transform: buffer transform failed

I have narrowed it down to the gst-dsexample plugin(we call pip(Picture-in-Picture)). Because if I disable the plugin everything runs as expected. I think because we are missing modprobe and lsmod it’s not able to determine the GPU type here:

cudaDeviceGetAttribute(&val, cudaDevAttrIntegrated, pip->gpu_id);
    pip->is_integrated = val;

Which in turn causes this portion of the code to select NVBUF_MEM_CUDA_PINNED instead of correctly selecting NVBUF_MEM_DEFAULT:

if (pip->is_integrated) {
        create_params.memType = NVBUF_MEM_DEFAULT;
    } else {
        create_params.memType = NVBUF_MEM_CUDA_PINNED;
    }

Any ideas?

8

I guess you might be building this docker image on an x86 platform?

In fact, when executing docker pull nvcr.io/nvidia/deepstream:xxx-triton-multiarch, different layers will be pulled from ngc depending on the platform.

On Jetson
Save the following content as cross.dockerfile.

FROM nvcr.io/nvidia/deepstream:7.1-triton-multiarch

RUN mkdir /usr/lib/aarch64-linux-gnu/nvidia/

COPY libnvbufsurface.so /usr/lib/aarch64-linux-gnu/nvidia/
COPY libnvbufsurface.so.1.0.0 /usr/lib/aarch64-linux-gnu/nvidia/
COPY libnvbufsurftransform.so /usr/lib/aarch64-linux-gnu/nvidia/
COPY libnvbufsurftransform.so.1.0.0 /usr/lib/aarch64-linux-gnu/nvidia/

Build docker image

docker build -f cross.dockerfile -t jetson-cross /usr/lib/aarch64-linux-gnu/nvidia/ --no-cache

Export docker image

docker save jetson-cross | gzip > jetson-cross.tar.gz

On X86
Load docker image

docker load < jetson-cross.tar.gz

Run docker image just for cross-compile

docker run --platform linux/arm64 -it jetson-cross:latest  /bin/bash

In container

cd /opt/nvidia/deepstream/deepstream-7.1/sources/gst-plugins/gst-dsexample
make CUDA_VER=12.6

I can compile libnvdsgst_dsexample.so correctly (although not run), but it can be cross compiled, right?

9

I am getting a bunch of these errors:

403.4 /usr/bin/ld: warning: libnvrm_mem.so, needed by /opt/nvidia/deepstream/deepstream-6.4/lib/libnvbufsurface.so, not found (try using -rpath or -rpath-link)        
403.4 /usr/bin/ld: warning: libnvrm_surface.so, needed by /opt/nvidia/deepstream/deepstream-6.4/lib/libnvbufsurface.so, not found (try using -rpath or -rpath-link)    
403.4 /usr/bin/ld: warning: libnvrm_chip.so, needed by /opt/nvidia/deepstream/deepstream-6.4/lib/libnvbufsurface.so, not found (try using -rpath or -rpath-link)       
403.4 /usr/bin/ld: warning: libnvos.so, needed by /opt/nvidia/deepstream/deepstream-6.4/lib/libnvbufsurface.so, not found (try using -rpath or -rpath-link)
403.4 /usr/bin/ld: warning: libnvbuf_fdmap.so.1.0.0, needed by /opt/nvidia/deepstream/deepstream-6.4/lib/libnvbufsurface.so, not found (try using -rpath or -rpath-link)
403.4 /usr/bin/ld: warning: libnvrm_gpu.so, needed by /opt/nvidia/deepstream/deepstream-6.4/lib/libnvbufsurface.so, not found (try using -rpath or -rpath-link)        
403.4 /usr/bin/ld: warning: libnvrm_host1x.so, needed by /opt/nvidia/deepstream/deepstream-6.4/lib/libnvbufsurftransform.so, not found (try using -rpath or -rpath-link)
403.4 /usr/bin/ld: warning: libnvvic.so, needed by /opt/nvidia/deepstream/deepstream-6.4/lib/libnvbufsurftransform.so, not found (try using -rpath or -rpath-link)     
403.4 /usr/bin/ld: warning: libnvdla_compiler.so, needed by /lib/aarch64-linux-gnu/libnvinfer.so.8, not found (try using -rpath or -rpath-link)
403.5 /usr/bin/ld: warning: libnvcudla.so, needed by /usr/local/cuda/lib64/libcudla.so.1, not found (try using -rpath or -rpath-link)
10

These are just warnings, which should be related to the DS version. I did not find these problems on DS-7.1.

If the binary is deployed to Jetson, it can run normally and you can ignore these warnings

11

I am able to get it to compile but I am getting the following runtime error:

xod-optical-deepstream-onvif  | /dvs/git/dirty/git-master_linux/nvutils/nvbufsurftransform/nvbufsurftransform.cpp:4550: => Surface type not supported for transformation NVBUF_MEM_CUDA_UNIFIED
xod-optical-deepstream-onvif  | 
xod-optical-deepstream-onvif  | 0:00:12.454844444     1 0xffff1c001d20 ERROR         nvvideoconvert gstnvvideoconvert.c:4208:gst_nvvideoconvert_transform: buffer transform failed

Additionally, the only difference is that I am building the gst-dsexample(which I call pip as I have modified it somewhat), using CMakeLists.txt:

cmake_minimum_required(VERSION 3.10)

find_package(PkgConfig)
find_package(spdlog REQUIRED)
# Find CUDA
find_package(CUDA REQUIRED)

# Display the CUDA version found
message(STATUS "Found CUDA ${CUDA_VERSION_STRING} at ${CUDA_TOOLKIT_ROOT_DIR}")
pkg_check_modules(GST REQUIRED gstreamer-1.0 gstreamer-video-1.0)

# Creating library target for PIP Plugin
add_library(pip SHARED)

# Specify CUDA and C++ compiler options
set_target_properties(pip PROPERTIES
    CUDA_STANDARD 11
    CXX_STANDARD 11
    CXX_STANDARD_REQUIRED ON
    POSITION_INDEPENDENT_CODE ON
    OUTPUT_NAME "nvdsgst_pip"
)

# Add any definitions needed, these replace the COMMON variables in your Makefile
target_compile_definitions(pip PRIVATE -DOPENCV -DGRAPH DS_VERSION="${NVDS_VERSION}")

file(GLOB PLUGIN_SRCS pip.cpp pip.h pip_lib/pip_lib.c pip_lib/pip_lib.h)

# Associate the source files with the target
target_sources(pip PRIVATE ${PLUGIN_SRCS})

# Include directories
target_include_directories(pip PRIVATE 
    /opt/nvidia/deepstream/deepstream/sources/includes
    /usr/local/cuda-${CUDA_VER}/include
    /usr/include/opencv4
    plugin/pip
    plugin/pip/pip_lib
)

# Link libraries
target_link_libraries(pip PRIVATE
    ${CUDA_LIBRARIES}
    ${OpenCV_LIBS}
    ${GST_LIBRARIES}
    spdlog::spdlog
    dl nppc nppig npps nppicc nppidei
    nvdsgst_helper nvdsgst_meta nvds_meta
    nvbufsurface nvbufsurftransform
    nvinfer_plugin nvinfer nvparsers nvonnxparser
    cuda cudart cublas stdc++fs
)

set_target_properties(pip PROPERTIES INSTALL_RPATH "$ORIGIN:/opt/nvidia/deepstream/deepstream-${NVDS_VERSION}/lib")

message("GST_INSTALL_DIR: ${GST_INSTALL_DIR}")
# Installing the library
install(TARGETS pip 
        LIBRARY DESTINATION ${GST_INSTALL_DIR}
)

Any ideas?

12

Turns out it was the nvbuf-memory-type parameter not being set correctly to 0:

[pip]
enable=1
processing-width=640
processing-height=480
#batch-size for batch supported optimized plugin
#batch-size=1
unique-id=15
gpu-id=0
# Supported memory types are 1 and 3
nvbuf-memory-type=0

However, one thing I am noticing is that I am getting runtime errors when a bounding box is drawn on the screen. It runs and then I get an exited with code 139. I have made sure that all of my nvbuf-memory-type for each plugin is of type 0. Any ideas? Have you tested your cross compile gst-dsexample from above on an actual Jetson device?

13

Did you modify the code?

I didn’t modify anything, and the cross-compiled libnvdsgst_dsexample.so works fine on my Jetson Orin.

Add ds-example node to source2_1080p_dec_infer-resnet_demux_int8.txt

[ds-example]
enable=1
processing-width=640
processing-height=480
full-frame=0
#batch-size for batch supported optimized plugin
batch-size=1
unique-id=15
gpu-id=0
blur-objects=1
# Supported memory types for blur-objects:
# For x86: 1 and 3
# For Jetson: 0
nvbuf-memory-type=0

Run deepstream-app

deepstream-app -c /opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/source2_1080p_dec_infer-resnet_demux_int8.txt
14

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks

15

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.