CUDA runtime version 0.0

I am setting up CUDA to run on a cluster. All hosts have exactly the same hardware and should also have the same software but I’ve got a problem I can’t seem to find a solution for. All hosts are running Ubuntu 10.10 (uname -a: Linux … 2.6.35-25-generic #44-Ubuntu SMP Fri Jan 21 17:40:44 UTC 2011 x86_64 GNU/Linux) fully updated.

I have succesfully installed CUDA on the first cluster host (master). Running deviceQuery gives this output:

[./deviceQuery] starting…

./deviceQuery Starting…

CUDA Device Query (Runtime API) version (CUDART static linking)

There are 2 devices supporting CUDA

Device 0: “GeForce GTX 460”

CUDA Driver Version / Runtime Version 4.0 / 4.0

CUDA Capability Major/Minor version number: 2.1

Total amount of global memory: 1024 MBytes (1073414144 bytes)

( 7) Multiprocessors x (48) CUDA Cores/MP: 336 CUDA Cores

GPU Clock Speed: 1.35 GHz

Memory Clock rate: 1800.00 Mhz

Memory Bus Width: 256-bit

L2 Cache Size: 524288 bytes

Max Texture Dimension Size (x,y,z) 1D=(65536), 2D=(65536,65535), 3D=(2048,2048,2048)

Max Layered Texture Size (dim) x layers 1D=(16384) x 2048, 2D=(16384,16384) x 2048

Total amount of constant memory: 65536 bytes

Total amount of shared memory per block: 49152 bytes

Total number of registers available per block: 32768

Warp size: 32

Maximum number of threads per block: 1024

Maximum sizes of each dimension of a block: 1024 x 1024 x 64

Maximum sizes of each dimension of a grid: 65535 x 65535 x 65535

Maximum memory pitch: 2147483647 bytes

Texture alignment: 512 bytes

Concurrent copy and execution: Yes with 1 copy engine(s)

Run time limit on kernels: No

Integrated GPU sharing Host Memory: No

Support host page-locked memory mapping: Yes

Concurrent kernel execution: Yes

Alignment requirement for Surfaces: Yes

Device has ECC support enabled: No

Device is using TCC driver mode: No

Device supports Unified Addressing (UVA): Yes

Device PCI Bus ID / PCI location ID: 4 / 0

Compute Mode:

 < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >

Device 1: “GeForce GTX 460”

CUDA Driver Version / Runtime Version 4.0 / 4.0

CUDA Capability Major/Minor version number: 2.1

Total amount of global memory: 1023 MBytes (1072889856 bytes)

( 7) Multiprocessors x (48) CUDA Cores/MP: 336 CUDA Cores

GPU Clock Speed: 1.35 GHz

Memory Clock rate: 1800.00 Mhz

Memory Bus Width: 256-bit

L2 Cache Size: 524288 bytes

Max Texture Dimension Size (x,y,z) 1D=(65536), 2D=(65536,65535), 3D=(2048,2048,2048)

Max Layered Texture Size (dim) x layers 1D=(16384) x 2048, 2D=(16384,16384) x 2048

Total amount of constant memory: 65536 bytes

Total amount of shared memory per block: 49152 bytes

Total number of registers available per block: 32768

Warp size: 32

Maximum number of threads per block: 1024

Maximum sizes of each dimension of a block: 1024 x 1024 x 64

Maximum sizes of each dimension of a grid: 65535 x 65535 x 65535

Maximum memory pitch: 2147483647 bytes

Texture alignment: 512 bytes

Concurrent copy and execution: Yes with 1 copy engine(s)

Run time limit on kernels: No

Integrated GPU sharing Host Memory: No

Support host page-locked memory mapping: Yes

Concurrent kernel execution: Yes

Alignment requirement for Surfaces: Yes

Device has ECC support enabled: No

Device is using TCC driver mode: No

Device supports Unified Addressing (UVA): Yes

Device PCI Bus ID / PCI location ID: 6 / 0

Compute Mode:

 < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >

deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 4.0, CUDA Runtime Version = 4.0, NumDevs = 2, Device = GeForce GTX 460, Device = GeForce GTX 460

[./deviceQuery] test results…

PASSED

Press ENTER to exit…

On the other host running deviceQuery, I get (the CUDA Driver Version/Runtime version is due to my modification of the program):

[./deviceQuery] starting…

./deviceQuery Starting…

CUDA Device Query (Runtime API) version (CUDART static linking)

cudaGetDeviceCount FAILED CUDA Driver and Runtime version may be mismatched.

CUDA Driver Version / Runtime Version 0.0 / 0.0

[deviceQuery] test results…

FAILED

I admit the installataion procedure for the first host was messy so I might have forgotten something. Anyway, my installation procedure for the new host is:

  1. sudo sh devdriver_4.0_linux_64_270.40.run (yes to all)

  2. sudo reboot

  3. cat /proc/driver/nvidia/version:

    NVRM version: NVIDIA UNIX x86_64 Kernel Module 270.40 Sat Mar 26 13:00:34 PDT 2011

    GCC version: gcc version 4.4.5 (Ubuntu/Linaro 4.4.4-14ubuntu5)

  4. sudo sh cudatoolkit_4.0.13_linux_64_ubuntu10.10.run

  5. sudo echo /usr/local/cuda/lib64 > /etc/ld.so.conf.d/cuda.conf

  6. run deviceQuery (from a NFS shared with the master)

Any ideas what I could be missing? I have tried compiling deviceQuery on the new host but that doesn’t change anything…

Thanks in advance

If your cluster nodes don’t start an X server, you will need to modify the startup scripts to create the /dev/nvidia* device files after a reboot. See the release notes:

http://developer.download.nvidia.com/compute/cuda/4_0_rc2/toolkit/docs/CUDA_Toolkit_Release_Notes.txt

Search for “modprobe” to find an example shell script to do this.

You’re right my node does not start an X server. I’ve put the script creating the /dev/nvidia* device files as a startup script. Now I should be ready to roll the installation out to the rest of the cluster nodes.

Thanks