Accessing the GPU from docker on L4T R32.1

On the previous L4T R28.2 I could access the GPU from docker containers following the approach outlined in:

Unfortunately this doesn’t seem to work for the latest L4T R32.1 from Jetpack 4.2 I haven’t been able to make it work e.g. if a I create a simple container with all the samples included using:

FROM arm64v8/ubuntu:16.04
ENV LD_LIBRARY_PATH=/usr/local/cuda-10.0/lib64
COPY NVIDIA_CUDA-10.0_Samples NVIDIA_CUDA-10.0_Samples
CMD /bin/bash

… and then run it mapping all the /dev/nv* devices:

$ docker run -it --device=/dev/nvhost-as-gpu --device=/dev/nvhost-ctrl --device=/dev/nvhost-ctrl-gpu --device=/dev/nvhost-ctrl-isp --device=/dev/nvhost-ctrl-nvcsi --device=/dev/nvhost-ctrl-nvdec --device=/dev/nvhost-ctxsw-gpu --device=/dev/nvhost-dbg-gpu --device=/dev/nvhost-gpu --device=/dev/nvhost-isp --device=/dev/nvhost-msenc --device=/dev/nvhost-nvcsi --device=/dev/nvhost-nvdec --device=/dev/nvhost-nvjpg --device=/dev/nvhost-prof-gpu --device=/dev/nvhost-sched-gpu --device=/dev/nvhost-tsec --device=/dev/nvhost-tsecb --device=/dev/nvhost-tsg-gpu --device=/dev/nvhost-vi --device=/dev/nvhost-vic --device=/dev/nvmap -v /usr/local/cuda-10.0:/usr/local/cuda-10.0 cuda_container

root@e06082804232:/# ./NVIDIA_CUDA-10.0_Samples/bin/aarch64/linux/release/deviceQuery
./NVIDIA_CUDA-10.0_Samples/bin/aarch64/linux/release/deviceQuery Starting…

CUDA Device Query (Runtime API) version (CUDART static linking)

cudaGetDeviceCount returned 35
→ CUDA driver version is insufficient for CUDA runtime version
Result = FAIL

… which suggests some part of the driver isn’t properly punched through?

I am doing something obviously wrong or is there a real issue here?

Thanks!,

Geoff

PS. f I run the deviceQuery directly in L4T it works fine obviously.

Hi,

JetPack4.2 is Ubuntu18.04.
Could you update your dockerfile with ubuntu 18.04 and try it again?

FROM arm64v8/ubuntu:18.04

Thanks.

That makes no difference unfortunately - same error.

Regards,

Geoff

I see over in the ticket asking about nvidia-docker we are told:

The nvidia-docker will be included with JetPack 4.2.1, to be released first week of July.

Does that imply that there is no way of getting CUDA and docker to play nicely before that with Jetpack 4.2. I have a system that is scheduled to be mounted onto a vehicle (and thus become very expensive and disruptive to access for flashing - though no problem with ssh access) in the last week of June.

Any suggestions?

Thanks,

Geoff

wow, that is GREAT news

Hi,

I can run the deviceQuery sample without issue on the JetPack4.2.
Here are our steps for your reference:

1. Build deviceQuery sample

/usr/local/cuda-10.0/bin/cuda-install-samples-10.0.sh .
cd NVIDIA_CUDA-10.0_Samples/1_Utilities/
make
cp deviceQuery ~

2. Dockerfile

FROM arm64v8/ubuntu:18.04
  
ENV LD_LIBRARY_PATH=/usr/lib/aarch64-linux-gnu/tegra
RUN mkdir /cudaSamples
COPY deviceQuery /cudaSamples/

CMD /cudaSamples/deviceQuery

3. Build and execute

cd ~
sudo docker build -t device_query .
sudo docker run --device=/dev/nvhost-ctrl --device=/dev/nvhost-ctrl-gpu --device=/dev/nvhost-prof-gpu --device=/dev/nvmap --device=/dev/nvhost-gpu --device=/dev/nvhost-as-gpu -v /usr/lib/aarch64-linux-gnu/tegra:/usr/lib/aarch64-linux-gnu/tegra device_query

Thanks.

Almost - that works for deviceQuery but fails for some of the other examples which requre more of the libraries.

If I tweak it slightly e.g.:

FROM arm64v8/ubuntu:18.04

ENV LD_LIBRARY_PATH=/usr/lib/aarch64-linux-gnu/tegra:/usr/local/cuda-10.0/lib64

COPY NVIDIA_CUDA-10.0_Samples/bin/aarch64/linux/release/matrixMulCUBLAS .

CMD ./matrixMulCUBLAS

… and run with:

docker run --device=/dev/nvhost-ctrl --device=/dev/nvhost-ctrl-gpu --device=/dev/nvhost-prof-gpu --device=/dev/nvmap --device=/dev/nvhost-gpu --device=/dev/nvhost-as-gpu -v /usr/lib/aarch64-linux-gnu/tegra:/usr/lib/aarch64-linux-gnu/tegra -v /usr/local/cuda-10.0/lib64:/usr/local/cuda-10.0/lib64 cuda_container

Then it works.

Thanks for your help with this!