Failure in reproducing DeepStream-3D Multi-Modal BEVFusion

jaydenst0307 · November 25, 2024, 10:23am

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) NVIDIA GeForce RTX 4070
• DeepStream Version 7.1-gc-triton-devel
• JetPack Version (valid for Jetson only) NA
• TensorRT Version NA
• NVIDIA GPU Driver Version (valid for GPU only) 566.14
• Issue Type( questions, new requirements, bugs)
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
When I run this command

bevfusion/docker_run_generate_trt_engine_models.sh bevfusion/model_root

I got this error

docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-3d-lidar-sensor-fusion/bevfusion/build_bevfusion_model.sh" to rootfs at "/opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-3d-lidar-sensor-fusion/bevfusion/build_bevfusion_model.sh": create mount destination for /opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-3d-lidar-sensor-fusion/bevfusion/build_bevfusion_model.sh mount: cannot mkdir in /var/lib/docker/overlay2/2868b762a09cfb04be9975cc11d1834202a02f2c2bc28c68c139176b226c3cb6/merged/opt/nvidia/deepstream/deepstream-7.1/sources/apps/sample_apps/deepstream-3d-lidar-sensor-fusion/bevfusion/build_bevfusion_model.sh: not a directory: unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type.
failed to build models into local folder: bevfusion/model_root

This is how I launch the docker image

docker run --gpus all -it --rm --net=host --privileged \
     -v /usr/bin/docker:/usr/bin/docker \
     -v /usr/libexec/docker:/usr/libexec/docker \
     -v /var/run/docker.sock:/var/run/docker.sock \
     -v /tmp/.X11-unix:/tmp/.X11-unix -e DISPLAY=$DISPLAY nvcr.io/nvidia/deepstream:7.1-gc-triton-devel

• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)
https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_3D_MultiModal_Lidar_Camera_BEVFusion.html

junshengy · November 26, 2024, 1:49am

Please follow the steps in the document. Before building the model, you need to build the bevfusion docker image.

bevfusion/docker_build_bevfusion_image.sh nvcr.io/nvidia/deepstream:7.1-gc-triton-devel

jaydenst0307 · November 26, 2024, 2:36am

I have built that image too! Could you point out how would it help to solve the error?

jaydenst0307 · November 26, 2024, 2:41am

I met with the same error, be it inside nvcr.io/nvidia/deepstream:7.1-gc-triton-devel or the newly built image deepstream-triton-bevfusion 7.1.

docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/opt/nvidia/deepstream/deepstream-7.1/sources/apps/sample_apps/deepstream-3d-lidar-sensor-fusion/bevfusion/build_bevfusion_model.sh" to rootfs at "/opt/nvidia/deepstream/deepstream-7.1/sources/apps/sample_apps/deepstream-3d-lidar-sensor-fusion/bevfusion/build_bevfusion_model.sh": create mount destination for /opt/nvidia/deepstream/deepstream-7.1/sources/apps/sample_apps/deepstream-3d-lidar-sensor-fusion/bevfusion/build_bevfusion_model.sh mount: cannot mkdir in /var/lib/docker/overlay2/eb8896fad311ef42fc67ea3c5161e019e62b4c38ac12ffa057c52bab1f169b74/merged/opt/nvidia/deepstream/deepstream-7.1/sources/apps/sample_apps/deepstream-3d-lidar-sensor-fusion/bevfusion/build_bevfusion_model.sh: not a directory: unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type.
failed to build models into local folder: bevfusion/model_root

junshengy · November 26, 2024, 4:00am

This problem may be caused by Docker not being installed correctly. Try the following command line. The steps provided in the document can work properly. I have tried it.

curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh
sudo reboot

jaydenst0307 · November 26, 2024, 4:51am

Can I check with you what’s the version of your linux and your gpu driver version?

junshengy · November 26, 2024, 4:59am

Ubuntu 22.04 / GPU driver 550.127.05

I have also tried other OS versions. If you use docker, usually as long as the GPU driver is higher than 550, it should work.

jaydenst0307 · November 26, 2024, 6:04am

I have purge docker and reinstall docker and nvidia container toolkit. Same error.

junshengy · November 26, 2024, 9:07am

A tip is ignored in the document

The bevfusion uses the docker-in-docker feature, but Docker containers within Docker containers use the parent host’s Docker daemon, so any volumes mounted in the “docker-in-docker” situation are still referenced from the host, not from the container.

So you need to install deepstream sdk in the host first (no dependencies need to be installed, the runtime is completely in the container)

sudo rm -rf /opt/nvidia/deepstream/

wget --content-disposition 'https://api.ngc.nvidia.com/v2/resources/org/nvidia/deepstream/7.1/files?redirect=true&path=deepstream-7.1_7.1.0-1_amd64.deb' -O deepstream-7.1_7.1.0-1_amd64.deb

# Please ignore warnings or errors
sudo dpkg -i deepstream-7.1_7.1.0-1_amd64.deb 

# start docker
docker run --gpus all -it --rm --net=host --privileged \
     -v /usr/bin/docker:/usr/bin/docker \
     -v /usr/libexec/docker:/usr/libexec/docker \
     -v /var/run/docker.sock:/var/run/docker.sock \
     -v /tmp/.X11-unix:/tmp/.X11-unix -e DISPLAY=$DISPLAY nvcr.io/nvidia/deepstream:7.1-triton-multiarch

cd /opt/nvidia/deepstream/deepstream-7.1/sources/apps/sample_apps/deepstream-3d-lidar-sensor-fusion

# build bevfusion docker image
bevfusion/docker_build_bevfusion_image.sh nvcr.io/nvidia/deepstream:7.1-triton-multiarch

# generate the trt model
mkdir bevfusion/model_root
bevfusion/docker_run_generate_trt_engine_models.sh bevfusion/model_root

# run triton server
bevfusion/docker_run_triton_server_bevfusion.sh bevfusion/model_root/

jaydenst0307 · November 27, 2024, 5:52am

Thanks junshengy. It makes sense now. I can launch triton server successfully.

However, when I run

bevfusion/docker_run_ds3d_sensor_fusion_bevfusion_pipeline.sh \
  ds3d_lidar_plus_multi_cam_bev_fusion.yaml

I encoutered this error:

lidarinference_filter_impl.cpp:204, INFO: lidarinference datafilter is starting to filter datamap
gl_cuda_utils.cpp:43, ERROR: register gl texture into cuda failed, cuda err_no:999, err_str:cudaErrorUnknown
gl_texture_render.cpp:177, ERROR: Failed to register texture to cuda, check failure
gl_texture_render.cpp:203, ERROR: allocate texture failed for texture3d render, check failure
gl_texture_render.cpp:310, ERROR: Failed to upload image into texture3d., check failure
gles_3d_ensemble_render_impl.cpp:125, ERROR: upload datamap into Gl failed., check failure
glfw_window.cpp:347, ERROR: user GL render failed, check failure
gles_3d_ensemble_render_impl.cpp:63, ERROR: glwinow run failed.
nvds3d_gst_plugin.cpp:167, ERROR: datarender error: GLwindow run failed
nvds3d_gst_plugin.cpp:167, ERROR: datarender error: render callback report failure
/dvs/git/dirty/git-master_linux/deepstream/sdk/src/utils/ds3d/gst/nvds3d_pipeline_context.h:329, ERROR: ERROR from element ds3d_sensor_fusion_render: appsink Failed to stream due to ds3d errors, details: nvds3d_gst_plugin.cpp(168): NotifySinkError (): /GstPipeline:DeepStreamSensorFusionApp/GstBin:ds3d_sensor_fusion_render/GstAppSink:ds3d_sensor_fusion_render:
appsink error: GLwindow run failed

Detection output is shown in the terminal. I think there is some issue with openGL.

If I launch with this command instead of using the shell file, I am able to see the visualization:

__NV_PRIME_RENDER_OFFLOAD=1 __GLX_VENDOR_LIBRARY_NAME=nvidia /opt/nvidia/deepstream/deepstream/bin/deepstream-3d-lidar-sensor-fusion -c ds3d_lidar_plus_multi_cam_bev_fusion.yaml

Could you please advise on this?

junshengy · November 27, 2024, 7:31am

Are you running the bevfusion over ssh? x11-forword is not supported.

What is the output of the below command line ?

cat /proc/*/environ 2>/dev/null | tr '\0' '\n' | grep '^DISPLAY=:' | uniq

Try

export DISPLAY=:x (x is the value output of the above command)

Or use fakesink, remove the original render node in the configuration file and add the following content

# lidar data rendering
---
name: ds3d_sensor_fusion_render
type: ds3d::gstparsebin
in_caps: ds3d/datamap
with_queue: sink
config_body:
  parse_bin: fakesink

jaydenst0307 · November 27, 2024, 10:08am

This is the output

DISPLAY=:0
DISPLAY=:0.0
DISPLAY=:0

gdhtcmail · February 11, 2025, 7:36pm

I was getting the exact same error. It turns out that after a fresh reinstall of my graphics drivers my processor’s GPU was set as prime, so doing the following and a reboot fixed it:

sudo prime-select nvidia

junshengy · February 12, 2025, 7:11am

Thanks for your sharing, If you have more issues, please open a new topic.thanks.

system · March 10, 2025, 10:23am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
DeepStream-3D Sensor Fusion sample gl texture cuda register failed DeepStream SDK	3	203	June 25, 2024
Problems with deepstream docker on Jetson nano DeepStream SDK	13	1035	October 12, 2021
Cross-compile Deepstream Docker Image for Jetson on x86_64 DeepStream SDK deepstream	13	141	December 30, 2024
Running deepstream sample apps in docker container requires host networking and root user DeepStream SDK	12	1884	October 26, 2022
Containerization using docker of deepstream python script DeepStream SDK deepstream	14	65	December 18, 2024
Deepstream 6.3-triton-multiarch in Docker cannot use Tracker DeepStream SDK	6	731	August 28, 2023
Cannot run example in deepstream docker container DeepStream SDK test	8	994	September 26, 2022
Running a DeepStream pipeline in Nvidia DGX station A100 (With Docker) DeepStream SDK a100	3	1913	December 28, 2021
New installation Multiple Failues DeepStream SDK	18	1130	June 28, 2022
Deepstream6.4 docker run test1.py issue DeepStream SDK	5	325	April 23, 2024

Failure in reproducing DeepStream-3D Multi-Modal BEVFusion

Related topics