Test nvidia-smi by nvidia docker

Robotics & Edge Computing Jetson & Embedded Systems Jetson TX2
1

After flashing TX2, I wanna do some basic tests with nvidia docker.
I don’t know why I usually get this kind of error messages.
One of error messages is like this below. I searched some discussions which it seems like cuda version? I used the Jetpack version 4.3 and I checked the cuda version which is 10.0.
Is there any idea about this? How can I solve it?

nvidia@nvidia-desktop:~/Desktop/github/tensorrt-utils/OSS$ docker run --runtime=nvidia --rm nvidia/cuda:10.0-base nvidia-smi
Unable to find image 'nvidia/cuda:10.0-base' locally
10.0-base: Pulling from nvidia/cuda
7ddbc47eeb70: Pull complete 
c1bbdc448b72: Pull complete 
8c3b70e39044: Pull complete 
45d437916d57: Pull complete 
d8f1569ddae6: Pull complete 
de5a2c57c41d: Pull complete 
ea6f04a00543: Pull complete 
Digest: sha256:e6e1001f286d084f8a3aea991afbcfe92cd389ad1f4883491d43631f152f175e
Status: Downloaded newer image for nvidia/cuda:10.0-base
docker: Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "process_linux.go:430: container init caused \"process_linux.go:413: running prestart hook 1 caused \\\"error running hook: exit status 1, stdout: src: /etc/vulkan/icd.d/nvidia_icd.json, src_lnk: /usr/lib/aarch64-linux-gnu/tegra/nvidia_icd.json, dst: /var/lib/docker/overlay2/087a00e36b3c875103eab9b2adda2cef67e2006de4055974723c81d0e72666a8/merged/etc/vulkan/icd.d/nvidia_icd.json, dst_lnk: /usr/lib/aarch64-linux-gnu/tegra/nvidia_icd.json\\\\nsrc: /usr/lib/aarch64-linux-gnu/libcuda.so, src_lnk: tegra/libcuda.so, dst: /var/lib/docker/overlay2/087a00e36b3c875103eab9b2adda2cef67e2006de4055974723c81d0e72666a8/merged/usr/lib/aarch64-linux-gnu/libcuda.so, dst_lnk: tegra/libcuda.so\\\\nsrc: /usr/lib/aarch64-linux-gnu/libdrm_nvdc.so, src_lnk: tegra/libdrm.so.2, dst: /var/lib/docker/overlay2/087a00e36b3c875103eab9b2adda2cef67e2006de4055974723c81d0e72666a8/merged/usr/lib/aarch64-linux-gnu/libdrm_nvdc.so, dst_lnk: tegra/libdrm.so.2\\\\nsrc: /usr/lib/aarch64-linux-gnu/libv4l2.so.0.0.999999, src_lnk: tegra/libnvv4l2.so, dst: /var/lib/docker/overlay2/087a00e36b3c875103eab9b2adda2cef67e2006de4055974723c81d0e72666a8/merged/usr/lib/aarch64-linux-gnu/libv4l2.so.0.0.999999, dst_lnk: tegra/libnvv4l2.so\\\\nsrc: /usr/lib/aarch64-linux-gnu/libv4lconvert.so.0.0.999999, src_lnk: tegra/libnvv4lconvert.so, dst: /var/lib/docker/overlay2/087a00e36b3c875103eab9b2adda2cef67e2006de4055974723c81d0e72666a8/merged/usr/lib/aarch64-linux-gnu/libv4lconvert.so.0.0.999999, dst_lnk: tegra/libnvv4lconvert.so\\\\nsrc: /usr/lib/aarch64-linux-gnu/libv4l/plugins/nv/libv4l2_nvvidconv.so, src_lnk: ../../../tegra/libv4l2_nvvidconv.so, dst: /var/lib/docker/overlay2/087a00e36b3c875103eab9b2adda2cef67e2006de4055974723c81d0e72666a8/merged/usr/lib/aarch64-linux-gnu/libv4l/plugins/nv/libv4l2_nvvidconv.so, dst_lnk: ../../../tegra/libv4l2_nvvidconv.so\\\\nsrc: /usr/lib/aarch64-linux-gnu/libv4l/plugins/nv/libv4l2_nvvideocodec.so, src_lnk: ../../../tegra/libv4l2_nvvideocodec.so, dst: /var/lib/docker/overlay2/087a00e36b3c875103eab9b2adda2cef67e2006de4055974723c81d0e72666a8/merged/usr/lib/aarch64-linux-gnu/libv4l/plugins/nv/libv4l2_nvvideocodec.so, dst_lnk: ../../../tegra/libv4l2_nvvideocodec.so\\\\nsrc: /usr/lib/aarch64-linux-gnu/tegra/libcuda.so, src_lnk: libcuda.so.1.1, dst: /var/lib/docker/overlay2/087a00e36b3c875103eab9b2adda2cef67e2006de4055974723c81d0e72666a8/merged/usr/lib/aarch64-linux-gnu/tegra/libcuda.so, dst_lnk: libcuda.so.1.1\\\\nsrc: /usr/lib/aarch64-linux-gnu/tegra/libnvbufsurface.so, src_lnk: libnvbufsurface.so.1.0.0, dst: /var/lib/docker/overlay2/087a00e36b3c875103eab9b2adda2cef67e2006de4055974723c81d0e72666a8/merged/usr/lib/aarch64-linux-gnu/tegra/libnvbufsurface.so, dst_lnk: libnvbufsurface.so.1.0.0\\\\nsrc: /usr/lib/aarch64-linux-gnu/tegra/libnvbufsurftransform.so, src_lnk: libnvbufsurftransform.so.1.0.0, dst: /var/lib/docker/overlay2/087a00e36b3c875103eab9b2adda2cef67e2006de4055974723c81d0e72666a8/merged/usr/lib/aarch64-linux-gnu/tegra/libnvbufsurftransform.so, dst_lnk: libnvbufsurftransform.so.1.0.0\\\\nsrc: /usr/lib/aarch64-linux-gnu/tegra/libnvbuf_utils.so, src_lnk: libnvbuf_utils.so.1.0.0, dst: /var/lib/docker/overlay2/087a00e36b3c875103eab9b2adda2cef67e2006de4055974723c81d0e72666a8/merged/usr/lib/aarch64-linux-gnu/tegra/libnvbuf_utils.so, dst_lnk: libnvbuf_utils.so.1.0.0\\\\nsrc: /usr/lib/aarch64-linux-gnu/tegra/libnvid_mapper.so, src_lnk: libnvid_mapper.so.1.0.0, dst: /var/lib/docker/overlay2/087a00e36b3c875103eab9b2adda2cef67e2006de4055974723c81d0e72666a8/merged/usr/lib/aarch64-linux-gnu/tegra/libnvid_mapper.so, dst_lnk: libnvid_mapper.so.1.0.0\\\\nsrc: /usr/share/glvnd/egl_vendor.d/10_nvidia.json, src_lnk: ../../../lib/aarch64-linux-gnu/tegra-egl/nvidia.json, dst: /var/lib/docker/overlay2/087a00e36b3c875103eab9b2adda2cef67e2006de4055974723c81d0e72666a8/merged/usr/share/glvnd/egl_vendor.d/10_nvidia.json, dst_lnk: ../../../lib/aarch64-linux-gnu/tegra-egl/nvidia.json\\\\nsrc: /usr/lib/aarch64-linux-gnu/libnvcaffe_parser.so.6, src_lnk: libnvparsers.so.6.0.1, dst: /var/lib/docker/overlay2/087a00e36b3c875103eab9b2adda2cef67e2006de4055974723c81d0e72666a8/merged/usr/lib/aarch64-linux-gnu/libnvcaffe_parser.so.6, dst_lnk: libnvparsers.so.6.0.1\\\\nsrc: /usr/lib/aarch64-linux-gnu/libnvcaffe_parser.so.6.0.1, src_lnk: libnvparsers.so.6.0.1, dst: /var/lib/docker/overlay2/087a00e36b3c875103eab9b2adda2cef67e2006de4055974723c81d0e72666a8/merged/usr/lib/aarch64-linux-gnu/libnvcaffe_parser.so.6.0.1, dst_lnk: libnvparsers.so.6.0.1\\\\nsrc: /usr/lib/aarch64-linux-gnu/libnvinfer.so.6, src_lnk: libnvinfer.so.6.0.1, dst: /var/lib/docker/overlay2/087a00e36b3c875103eab9b2adda2cef67e2006de4055974723c81d0e72666a8/merged/usr/lib/aarch64-linux-gnu/libnvinfer.so.6, dst_lnk: libnvinfer.so.6.0.1\\\\nsrc: /usr/lib/aarch64-linux-gnu/libnvinfer_plugin.so.6, src_lnk: libnvinfer_plugin.so.6.0.1, dst: /var/lib/docker/overlay2/087a00e36b3c875103eab9b2adda2cef67e2006de4055974723c81d0e72666a8/merged/usr/lib/aarch64-linux-gnu/libnvinfer_plugin.so.6, dst_lnk: libnvinfer_plugin.so.6.0.1\\\\nsrc: /usr/lib/aarch64-linux-gnu/libnvonnxparser.so.6, src_lnk: libnvonnxparser.so.6.0.1, dst: /var/lib/docker/overlay2/087a00e36b3c875103eab9b2adda2cef67e2006de4055974723c81d0e72666a8/merged/usr/lib/aarch64-linux-gnu/libnvonnxparser.so.6, dst_lnk: libnvonnxparser.so.6.0.1\\\\nsrc: /usr/lib/aarch64-linux-gnu/libnvonnxparser_runtime.so.6, src_lnk: libnvonnxparser_runtime.so.6.0.1, dst: /var/lib/docker/overlay2/087a00e36b3c875103eab9b2adda2cef67e2006de4055974723c81d0e72666a8/merged/usr/lib/aarch64-linux-gnu/libnvonnxparser_runtime.so.6, dst_lnk: libnvonnxparser_runtime.so.6.0.1\\\\nsrc: /usr/lib/aarch64-linux-gnu/libnvparsers.so.6, src_lnk: libnvparsers.so.6.0.1, dst: /var/lib/docker/overlay2/087a00e36b3c875103eab9b2adda2cef67e2006de4055974723c81d0e72666a8/merged/usr/lib/aarch64-linux-gnu/libnvparsers.so.6, dst_lnk: libnvparsers.so.6.0.1\\\\nsrc: /usr/lib/libvisionworks_sfm.so, src_lnk: libvisionworks_sfm.so.0.90, dst: /var/lib/docker/overlay2/087a00e36b3c875103eab9b2adda2cef67e2006de4055974723c81d0e72666a8/merged/usr/lib/libvisionworks_sfm.so, dst_lnk: libvisionworks_sfm.so.0.90\\\\nsrc: /usr/lib/libvisionworks_sfm.so.0.90, src_lnk: libvisionworks_sfm.so.0.90.4, dst: /var/lib/docker/overlay2/087a00e36b3c875103eab9b2adda2cef67e2006de4055974723c81d0e72666a8/merged/usr/lib/libvisionworks_sfm.so.0.90, dst_lnk: libvisionworks_sfm.so.0.90.4\\\\nsrc: /usr/lib/libvisionworks_tracking.so, src_lnk: libvisionworks_tracking.so.0.88, dst: /var/lib/docker/overlay2/087a00e36b3c875103eab9b2adda2cef67e2006de4055974723c81d0e72666a8/merged/usr/lib/libvisionworks_tracking.so, dst_lnk: libvisionworks_tracking.so.0.88\\\\nsrc: /usr/lib/libvisionworks_tracking.so.0.88, src_lnk: libvisionworks_tracking.so.0.88.2, dst: /var/lib/docker/overlay2/087a00e36b3c875103eab9b2adda2cef67e2006de4055974723c81d0e72666a8/merged/usr/lib/libvisionworks_tracking.so.0.88, dst_lnk: libvisionworks_tracking.so.0.88.2\\\\nsrc: /usr/lib/aarch64-linux-gnu/libcudnn.so.7, src_lnk: libcudnn.so.7.6.3, dst: /var/lib/docker/overlay2/087a00e36b3c875103eab9b2adda2cef67e2006de4055974723c81d0e72666a8/merged/usr/lib/aarch64-linux-gnu/libcudnn.so.7, dst_lnk: libcudnn.so.7.6.3\\\\n, stderr: exec command: [/usr/bin/nvidia-container-cli --load-kmods configure --ldconfig=@/sbin/ldconfig.real --device=all --compute --utility --require=cuda>=10.0 brand=tesla,driver>=384,driver<385 brand=tesla,driver>=410,driver<411 --pid=1594 /var/lib/docker/overlay2/087a00e36b3c875103eab9b2adda2cef67e2006de4055974723c81d0e72666a8/merged]\\\\nnvidia-container-cli: mount error: stat failed: /var/lib/docker/overlay2/087a00e36b3c875103eab9b2adda2cef67e2006de4055974723c81d0e72666a8/merged/usr/local/cuda-10.0/compat/libcuda.so.410.129: no such file or directory\\\\n\\\"\"": unknown.

Besides above problems, I was very confused about that can I run the cuda version of nvidia docker which is different with local cuda version? Suppose my jetson cuda version is 10.0, could I run a container of cuda 10.2 version by nvidia docker?

[TX2 detail version information]

JetPack version: 4.3
ubuntu version: 18.04
python3 version: 3.6.9
Tensorflow version: 1.15
TensorRT version: 6.0.1.10
CUDA version: 10.0.326

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Mon_Mar_11_22:13:24_CDT_2019
Cuda compilation tools, release 10.0, V10.0.326

cuDNN version: 7.6.3
docker version: Docker version 18.09.7, build 2d0083d

Thanks in Advance.

Chieh

2

Hi Chieh, the container you tried above is for PC/server systems. On Jetson, you should use the nvidia-l4t-base container found here:

https://ngc.nvidia.com/catalog/containers/nvidia:l4t-base
docker pull nvcr.io/nvidia/l4t-base:r32.3.1

Also, nvidia-smi isn’t supported on Jetson, so it isn’t available regardless of container. Instead it’s recommended to try tegrastats or deviceQuery tool.

That isn’t possible, because the CUDA version has dependencies on the underlying L4T version on your Jetson. The nvidia-l4t-base container pulls your CUDA version from your Jetson, so they are always the same.