Cuda Insufficient Drivers within Docker Container

Hey, I’m getting the error

cudaErrorInsufficientDriver(CUDA driver version is insufficient for CUDA runtime version)

For my Docker containers running a basic cuda sample. This was working up until about a week ago, and with no changes to our build system, it started happening. Is there a known fix for this?

We’ve tried building directly off of the Nvidia ubuntu/Cuda and Centos/cuda binary packages that are here: https://hub.docker.com/r/nvidia/cuda/

We are building the Centos7/Cuda8.0 package, and the Ubuntu16.04/Cuda8.0 Packages

But they are running into the same error, or, when doing it locally within the test suite dockerfile that is failing, locally we can run it and get passing marks, but when run through our Jenkins build system, it still gets insufficient driver errors.

Any help would be appreciated, thank you.

This may mean that the driver installed in the base machine does not match the driver installed in the container. Perhaps a week ago your base machines got updated.

If that is the case, then nvidia-docker was created to help with this case.