I am setting up CUDA to run on a cluster. All hosts have exactly the same hardware and should also have the same software but I’ve got a problem I can’t seem to find a solution for. All hosts are running Ubuntu 10.10 (uname -a: Linux … 2.6.35-25-generic #44-Ubuntu SMP Fri Jan 21 17:40:44 UTC 2011 x86_64 GNU/Linux) fully updated.
I have succesfully installed CUDA on the first cluster host (master). Running deviceQuery gives this output:
[./deviceQuery] starting…
./deviceQuery Starting…
CUDA Device Query (Runtime API) version (CUDART static linking)
There are 2 devices supporting CUDA
Device 0: “GeForce GTX 460”
CUDA Driver Version / Runtime Version 4.0 / 4.0
CUDA Capability Major/Minor version number: 2.1
Total amount of global memory: 1024 MBytes (1073414144 bytes)
( 7) Multiprocessors x (48) CUDA Cores/MP: 336 CUDA Cores
GPU Clock Speed: 1.35 GHz
Memory Clock rate: 1800.00 Mhz
Memory Bus Width: 256-bit
L2 Cache Size: 524288 bytes
Max Texture Dimension Size (x,y,z) 1D=(65536), 2D=(65536,65535), 3D=(2048,2048,2048)
Max Layered Texture Size (dim) x layers 1D=(16384) x 2048, 2D=(16384,16384) x 2048
Total amount of constant memory: 65536 bytes
Total amount of shared memory per block: 49152 bytes
Total number of registers available per block: 32768
Warp size: 32
Maximum number of threads per block: 1024
Maximum sizes of each dimension of a block: 1024 x 1024 x 64
Maximum sizes of each dimension of a grid: 65535 x 65535 x 65535
Maximum memory pitch: 2147483647 bytes
Texture alignment: 512 bytes
Concurrent copy and execution: Yes with 1 copy engine(s)
Run time limit on kernels: No
Integrated GPU sharing Host Memory: No
Support host page-locked memory mapping: Yes
Concurrent kernel execution: Yes
Alignment requirement for Surfaces: Yes
Device has ECC support enabled: No
Device is using TCC driver mode: No
Device supports Unified Addressing (UVA): Yes
Device PCI Bus ID / PCI location ID: 4 / 0
Compute Mode:
< Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >
Device 1: “GeForce GTX 460”
CUDA Driver Version / Runtime Version 4.0 / 4.0
CUDA Capability Major/Minor version number: 2.1
Total amount of global memory: 1023 MBytes (1072889856 bytes)
( 7) Multiprocessors x (48) CUDA Cores/MP: 336 CUDA Cores
GPU Clock Speed: 1.35 GHz
Memory Clock rate: 1800.00 Mhz
Memory Bus Width: 256-bit
L2 Cache Size: 524288 bytes
Max Texture Dimension Size (x,y,z) 1D=(65536), 2D=(65536,65535), 3D=(2048,2048,2048)
Max Layered Texture Size (dim) x layers 1D=(16384) x 2048, 2D=(16384,16384) x 2048
Total amount of constant memory: 65536 bytes
Total amount of shared memory per block: 49152 bytes
Total number of registers available per block: 32768
Warp size: 32
Maximum number of threads per block: 1024
Maximum sizes of each dimension of a block: 1024 x 1024 x 64
Maximum sizes of each dimension of a grid: 65535 x 65535 x 65535
Maximum memory pitch: 2147483647 bytes
Texture alignment: 512 bytes
Concurrent copy and execution: Yes with 1 copy engine(s)
Run time limit on kernels: No
Integrated GPU sharing Host Memory: No
Support host page-locked memory mapping: Yes
Concurrent kernel execution: Yes
Alignment requirement for Surfaces: Yes
Device has ECC support enabled: No
Device is using TCC driver mode: No
Device supports Unified Addressing (UVA): Yes
Device PCI Bus ID / PCI location ID: 6 / 0
Compute Mode:
< Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >
deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 4.0, CUDA Runtime Version = 4.0, NumDevs = 2, Device = GeForce GTX 460, Device = GeForce GTX 460
[./deviceQuery] test results…
PASSED
Press ENTER to exit…
On the other host running deviceQuery, I get (the CUDA Driver Version/Runtime version is due to my modification of the program):
[./deviceQuery] starting…
./deviceQuery Starting…
CUDA Device Query (Runtime API) version (CUDART static linking)
cudaGetDeviceCount FAILED CUDA Driver and Runtime version may be mismatched.
CUDA Driver Version / Runtime Version 0.0 / 0.0
[deviceQuery] test results…
FAILED
I admit the installataion procedure for the first host was messy so I might have forgotten something. Anyway, my installation procedure for the new host is:
-
sudo sh devdriver_4.0_linux_64_270.40.run (yes to all)
-
sudo reboot
-
cat /proc/driver/nvidia/version:
NVRM version: NVIDIA UNIX x86_64 Kernel Module 270.40 Sat Mar 26 13:00:34 PDT 2011
GCC version: gcc version 4.4.5 (Ubuntu/Linaro 4.4.4-14ubuntu5)
-
sudo sh cudatoolkit_4.0.13_linux_64_ubuntu10.10.run
-
sudo echo /usr/local/cuda/lib64 > /etc/ld.so.conf.d/cuda.conf
-
run deviceQuery (from a NFS shared with the master)
Any ideas what I could be missing? I have tried compiling deviceQuery on the new host but that doesn’t change anything…
Thanks in advance