Issues Running DeepStream on WSL2 (Docker Container): usr/lib/x86_64-linux-gnu/libcuda.so.1: file exists\\\\n\\\"\"": unknown

Accelerated Computing CUDA CUDA on Windows Subsystem for Linux
#1

So, i was try to run the deepstream container, having this error

docker run --gpus all -it --rm -v /tmp/.X11-unix:/tmp/.X11-unix -e DISPLAY=$DISPLAY -w /opt/nvidia/deepst
ream/deepstream-5.0 nvcr.io/nvidia/deepstream:5.0-dp-20.04-triton

docker: Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container pr
ocess caused “process_linux.go:449: container init caused “process_linux.go:432: running prestart hook 0 ca
used \“error running hook: exit status 1, stdout: , stderr: nvidia-container-cli: mount error: file creati
on failed: /var/lib/docker/overlay2/35b2ed80985f81e2e2a6a0703b0a3b4c687b1e77fccba7d6605f77f402556078/merged/
usr/lib/x86_64-linux-gnu/libcuda.so.1: file exists\\n\”””: unknown.

I also tried to install the necessary drivers and tried to run a sample deepstream app, most of the time resulted in a segmentation fault, or this error:

/deepstream-image-decode-app …/…/…/…/samples/streams/sample_720p.mjpeg ─╯
WARNING: Overriding infer-config batch-size (2) with number of sources (1)
Now playing: …/…/…/…/samples/streams/sample_720p.mjpeg,
Running…
ERROR from element nvv4l2-decoder: Could not open device ‘/dev/nvhost-nvdec’ for reading and writing.
Error details: v4l2_calls.c(656): gst_v4l2_open (): /GstPipeline:dstest-image-decode-pipeline/GstBin:source-bin-00/nvv4l2decoder:nvv4l2-decoder:
system error: No such file or directory
Returned, stopping playback
Deleting pipeline

here is the driver and runtime details
╰─ deepstream-app --version-all ─╯
deepstream-app version 5.0.0
DeepStreamSDK 5.0.0
CUDA Driver Version: 11.1
CUDA Runtime Version: 10.2
TensorRT Version: 7.1
cuDNN Version: 8.0
libNVWarp360 Version: 2.0.1d3

and yes the sample docker container for tensorflow works as followed from this tutorial: https://docs.nvidia.com/cuda/wsl-user-guide/index.html

#2

It is a bit hard to understand the exact setup and workloads you have run. Could you outline all the steps you performed in your WSL2 container so that we could try to repro it locally?

Also, since you added this:
-e DISPLAY=$DISPLAY

I would like to point out that there is currently no GUI support for WSL2.
Are you running XRDP session to the container?

#3

I followed the exact guide from https://docs.nvidia.com/cuda/wsl-user-guide/index.html
i’ve installed the cuda toolkit apt-get install -y cuda-toolkit-11-0 and setup docker as per the instructions, i am able to run the example docker given in the tutorial, so seems like everything is working

GUI works fine, something i tried using XServer on Windows
No there is no XRDP session, its just plain container from nvcr.io/nvidia/deepstream:5.0-dp-20.04-triton

Windows 10 20152
Ubuntu 18.04 on WSL2 Kernel 4.19.121
(and specifically did not use the docker desktop for windows, rather i used the Docker-CE for Linux distributions as suggested in the tutorial)

but i feel like something is wrong with running deepstream container that causes this,

nvidia-container-cli: mount error: file creation failed: /var/lib/docker/overlay2/35b2ed80985f81e2e2a6a0703b0a3b4c687b1e77fccba7d6605f77f402556078/merged/
usr/lib/x86_64-linux-gnu/libcuda.so.1: file exists\\n\”””: unknown.

what could be the reason ?

#4

I’m having an identical issue to the original poster. Is there a resolution?

I’m running Ubuntu 20.04.1 on WSL 2, Windows build 20190.rs_prerelease.200807-1609.

srm@McWindows:~$ docker run --gpus all -it --rm -v /tmp/.X11-unix:/tmp/.X11-unix -e DISPLAY=$DISPLAY -w /opt/nvidia/deepstream/deepstream-5.0 nvcr.io/nvidia/deepstream:5.0-20.07-triton

docker: Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused “process_linux.go:449: container init caused “process_linux.go:432: running prestart hook 0 caused \“error running hook: exit status 1, stdout: , stderr: nvidia-container-cli: mount error: file creation failed: /var/lib/docker/overlay2/cc9f2e9d433e58ca24a7672b59d50a4d5fef5aaf6085bd2d293fbc2dbd629bd4/merged/usr/lib/x86_64-linux-gnu/libcuda.so.1: file exists\\n\”””: unknown.

#5

Hello. I got exactly the same issue and I could solve it as follows:

  1. run the container WITHOUT the option “–gpus all”
  2. attach the container and just remove the libcuda.so.1 like “rm /usr/lib/x86_64-linux-gnu/libcuda.so.1”
  3. commit the container
  4. run the new container WITH the option “–gpus all”

I hope it works for you.

2 Likes
#6

Hello, @monotaro_ex

Thank you for detailed steps…
I have followed your instructions and it worked for me.
In my case, I had to remove two files
rm /usr/lib/x86_64-linux-gnu/libcuda.so.1
rm /usr/lib/x86_64-linux-gnu/libnvidia-ml.so.1

Need your help with this situation:
Same docker image works fine in Linux docker environment but it is not working with WSL. We use auto updater like watchtower to push latest updates to docker container from docker hub. So when we push an update to Dockerfile created for Linux systems, the changes we did (removing two files) might get reverted in WSL containers. Can you please help with a solution that will not create these two files within image itself?

Appreciate your help in this! Thank You!

1 Like
#7

Hello,

Yes we are aware that this is a big source of problems: native drivers shadowing the WSL ones (both in containers with the case you just described, but as well outside of the container if people accidentally install the native driver on their WSL distro).
We are looking at various fixes and improvement we can do in this area, and that particular container case is one of them.

Stay tuned for some update on this situation,
Thanks for your patience,

1 Like
#8

Hi @vamsicr1nk and @monotaro_ex
Thank you for your workaround tip.
I successfully started my Deepstream contanier after deleting libcuda.so.1 and libnvidia-ml-so. 1.

But when I compiled and executed the deepstream-Test1 example, I got the following error message:
Error: Could not get cuda device count (cudaErrorNoDevice)
Failed to parse group property
** ERROR: <gst_nvinfer_parse_config_file:1158>: failed
Now playing: /home/developer//alpr-docker_share/test8/1.avi
Unable to set device in gst_nvstreammux_change_state
Unable to set device in gst_nvstreammux_change_state

Do you have any solutions?
Thanks :)

Running Intro to Clara Train SDK fails in WSL 2
#9

Thank you.
Same error, different lib.
Same solution worked, actually i just renamed older lib.
Installing nvidia learning AI on nano docker

#10

Thank you, this workaround worked in my case

#11

Hi,

Thanks but did any of you were able to run GPU based inference? I think, by deleting these files, the container is able to run but WITHOUT GPU support. Please correct me if I am wrong.

1 Like
#12

Hi @rboissel , any news on this?

It’s a big problem that we’re ALMOST there with running inference in WSL and thus avoiding having to spread the different tasks over multiple machines.
Just missing this last piece of the puzzle…

/Jesper

#13

I ran into a similar issue. Should I follow @monotaro_ex’s solution and remove the related files??? Are there detailed steps for attaching and saving the changes to the docker container?

kaisark@nano:~$ sudo docker run -it --rm --runtime nvidia --network host -e DISPLAY=$DISPLAY nvcr.io/nvidia/jetson-pose:r32.4.2
docker: Error response from daemon: OCI runtime create failed: container_linux.go:367: starting container process caused: process_linux.go:495: container init caused: Running hook #0:: error running hook: exit status 1, stdout: , stderr: exec command: [/usr/bin/nvidia-container-cli --load-kmods --debug=/var/log/nvidia-container-toolkit.log configure --ldconfig=@/sbin/ldconfig.real --device=all --compute --compat32 --graphics --utility --video --display --pid=5684 /var/lib/docker/overlay2/6df5916510d0691201dbd3ba93e2ec30cf69a2fe6abd320360734e8796382b63/merged]
nvidia-container-cli: mount error: file creation failed: /var/lib/docker/overlay2/6df5916510d0691201dbd3ba93e2ec30cf69a2fe6abd320360734e8796382b63/merged/usr/lib/aarch64-linux-gnu/libnvidia-fatbinaryloader.so.440.18: file exists: unknown.

#14

Hi, we are still having similar issues on nvidia-wsl driver 471.
I’m curious, why does this work on cuda-samples docker image and not with other images with cuda/cudnn on it?

#15

Well, the impact is pretty big.