CUDA driver issues DGX-1 ??

Hi, I’m new to using NGC so please forgive me if I’ve missed something obvious…

I’ve pulled CUDA using: sudo docker pull nvcr.io/nvidia/cuda:latest
I then run it with: sudo nvidia-docker run -i -t f072016d63a4
or with sudo docker run -i -t f072016d63a4

and it all seems fine, nvcc -V returns…
nvcc: NVIDIA ® Cuda compiler driver
Copyright © 2005-2018 NVIDIA Corporation
Built on Sat_Aug_25_21:08:01_CDT_2018
Cuda compilation tools, release 10.0, V10.0.130

However when I try to run any CUDA code (code that works fine elsewhere) it either simply doesn’t run the CUDA parts, or returns this error…
ERROR: Kernel SOM_OnDevice Failed!: CUDA driver version is insufficient for CUDA runtime version

I could update the driver, but given that this is a container/image I shouldn’t have this problem right???

Also is there any way to install an editor (nano would do) just once so that its there each time I fire up the container rather than having to reinstall it each time.

Our NVIDIA GPU Cloud DL Frameworks images versioned 18.09 and later are built on CUDA 10, but they also include the (new) compatibility layer that allow them to work on R384 drivers that previously shipped on DGX-1 and DGX-Station. You don’t have to wait for a new DGX OS to use them.

But here the specific question is about nvcr.io/nvidia/cuda:latest, which is a CUDA 10 image without the compatibility layer. You can either install cuda-compat-10-0 into that image (but beware the image will then not work with other R410 drivers, only with the compatibility case – a circumstance we went to some lengths to work around for you in the DL Frameworks images) or you can upgrade your driver or you can use a nvidia/cuda image based on CUDA 9 instead of :latest.

I’m still stuck with this. I’ve tried pulling and running CUDA 9-devel and I have exactly the same issue there. I’ve updated the system to the latest base and still no GPU’s detected… what have I missed?

Hi Anthony,

You can send detail to enterprisesupport@nvidia.com go get NVIDIA Enterprise Support if your DGX-1 have valid service entitlement.

I need to use CUDA 10.0.
I understand that the 384.145 driver can work with the aforementioned version. However, there seems to be a lack of resources on documenting how a user can add the new toolkit. I installed https://developer.nvidia.com/cuda-10.0-download-archive but when using a framework (darknet) I am greeted with CUDA Error: CUDA driver version is insufficient for CUDA runtime version.

btw:

> nvcc --version
nvcc: NVIDIA ® Cuda compiler driver
Copyright © 2005-2018 NVIDIA Corporation
Built on Sat_Aug_25_21:08:01_CDT_2018
Cuda compilation tools, release 10.0, V10.0.130