What ? ( CUDA appears to have spontaneously stopped working )

Accelerated Computing CUDA CUDA Programming and Performance
1

Everything still compiles, but no longer runs. lspci shows that the cards are present :

$ ../NVIDIA_GPU_Computing_SDK/C/bin/linux/release/deviceQuery 

../NVIDIA_GPU_Computing_SDK/C/bin/linux/release/deviceQuery Starting...

CUDA Device Query (Runtime API) version (CUDART static linking)

cudaGetDeviceCount FAILED CUDA Driver and Runtime version may be mismatched.

FAILED

Press <Enter> to Quit...

-----------------------------------------------------------

It was working last month. Someone else may have changed the configuration ? I’m not sure. I am sure that the amount of time lost trying to get CUDA to run cancels out the runtime benefits.

2

check version of cudart (windows name) which is used by cuda. maybe path is somehow changed to wrong library. And maybe driver was rolled back by somebody. I am not a linux expert, just little ideas.

3

so, I realize the course of action is clear : re load the drivers and toolkit

4

I am not sure about the toolkit. Interesting, do other cuda programs work?

5

I followed through and downloaded the latest driver and runtime yet the error persists. I’m thinking I’ve wasted enough time trying to fix broken CUDA configurations, I’m going to do this one in plain old C. \

6

Hey guys, sorry I wasn’t refreshing the forum page and didn’t see your helpful comments. I guess I thought the toolkit was the runtime … I’ll check out your suggestions now, sorry.

7

sorry to be ignorant about this, but how do I check the version of cudart ?

8

this is what I have

$ ls libcuda*

libcuda.so  libcuda.so.1  libcuda.so.260.19.26

/usr/local/cuda/lib$ ls

libcublas.so    libcublas.so.3.2.16  libcudart.so.3       libcufft.so    libcufft.so.3.2.16  libcurand.so.3       libcusparse.so    libcusparse.so.3.2.16

libcublas.so.3  libcudart.so         libcudart.so.3.2.16  libcufft.so.3  libcurand.so        libcurand.so.3.2.16  libcusparse.so.3

/usr/local/cuda/lib64$ ls

libcublas.so         libcublas.so.3.2.16  libcudart.so.3       libcufft.so         libcufft.so.3.2.16  libcurand.so.3.2.12  libcusparse.so.3

libcublas.so.3       libcudart.so         libcudart.so.3.2.12  libcufft.so.3       libcurand.so        libcurand.so.3.2.16  libcusparse.so.3.2.12

libcublas.so.3.2.12  libcudart.so.2       libcudart.so.3.2.16  libcufft.so.3.2.12  libcurand.so.3      libcusparse.so       libcusparse.so.3.2.16
9

also this bug appears to affect all CUDA programs, whether they were built today or last month.

10

cuda program loads additonal (on windows it is cudart.dll), its version should be the same as toolkit version and driver should support it. If you have last driver, I assume that cuda programs should work, if they found proper cudart library. I suppose on weekdays there will more linux experts. I suppose that program path was comehow changed so programs could not locate proper cuda lib. Maybe it leads now to older version of cuda library.

11

That sounds likely, but I don’t really know how to fix it. Maybe someone else will know how.