Hello.
I have installed CUDA 7.5 on an Ubuntu 14.04 server. The machine has 4 graphics cards in it. My ultimate goal is to install and run Caffe for a machine learning application I am working with, and although I can compile Caffe just fine I fail the runtests every time I attempt them. I believe I have done something incorrectly while installing CUDA despite attempting to follow the official install instructions precisely. I have noticed that when I reboot the system I am able to run ./deviceQuery and it passes:
user@cuda:~/NVIDIA_CUDA-7.5_Samples/bin/x86_64/linux/release$ ./deviceQuery
./deviceQuery Starting...
CUDA Device Query (Runtime API) version (CUDART static linking)
Detected 4 CUDA Capable device(s)
Device 0: "Tesla C2050"
CUDA Driver Version / Runtime Version 7.5 / 7.5
CUDA Capability Major/Minor version number: 2.0
Total amount of global memory: 2687 MBytes (2817982464 bytes)
(14) Multiprocessors, ( 32) CUDA Cores/MP: 448 CUDA Cores
GPU Max Clock rate: 1147 MHz (1.15 GHz)
Memory Clock rate: 1500 Mhz
Memory Bus Width: 384-bit
L2 Cache Size: 786432 bytes
Maximum Texture Dimension Size (x,y,z) 1D=(65536), 2D=(65536, 65535), 3D=(2048, 2048, 2048)
Maximum Layered 1D Texture Size, (num) layers 1D=(16384), 2048 layers
Maximum Layered 2D Texture Size, (num) layers 2D=(16384, 16384), 2048 layers
Total amount of constant memory: 65536 bytes
Total amount of shared memory per block: 49152 bytes
Total number of registers available per block: 32768
Warp size: 32
Maximum number of threads per multiprocessor: 1536
Maximum number of threads per block: 1024
Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
Max dimension size of a grid size (x,y,z): (65535, 65535, 65535)
Maximum memory pitch: 2147483647 bytes
Texture alignment: 512 bytes
Concurrent copy and kernel execution: Yes with 2 copy engine(s)
Run time limit on kernels: No
Integrated GPU sharing Host Memory: No
Support host page-locked memory mapping: Yes
Alignment requirement for Surfaces: Yes
Device has ECC support: Enabled
Device supports Unified Addressing (UVA): Yes
Device PCI Domain ID / Bus ID / location ID: 0 / 1 / 0
Compute Mode:
< Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >
Device 1: "GeForce GT 430"
CUDA Driver Version / Runtime Version 7.5 / 7.5
CUDA Capability Major/Minor version number: 2.1
Total amount of global memory: 1023 MBytes (1072185344 bytes)
( 2) Multiprocessors, ( 48) CUDA Cores/MP: 96 CUDA Cores
GPU Max Clock rate: 1400 MHz (1.40 GHz)
Memory Clock rate: 900 Mhz
Memory Bus Width: 128-bit
L2 Cache Size: 131072 bytes
Maximum Texture Dimension Size (x,y,z) 1D=(65536), 2D=(65536, 65535), 3D=(2048, 2048, 2048)
Maximum Layered 1D Texture Size, (num) layers 1D=(16384), 2048 layers
Maximum Layered 2D Texture Size, (num) layers 2D=(16384, 16384), 2048 layers
Total amount of constant memory: 65536 bytes
Total amount of shared memory per block: 49152 bytes
Total number of registers available per block: 32768
Warp size: 32
Maximum number of threads per multiprocessor: 1536
Maximum number of threads per block: 1024
Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
Max dimension size of a grid size (x,y,z): (65535, 65535, 65535)
Maximum memory pitch: 2147483647 bytes
Texture alignment: 512 bytes
Concurrent copy and kernel execution: Yes with 1 copy engine(s)
Run time limit on kernels: No
Integrated GPU sharing Host Memory: No
Support host page-locked memory mapping: Yes
Alignment requirement for Surfaces: Yes
Device has ECC support: Disabled
Device supports Unified Addressing (UVA): Yes
Device PCI Domain ID / Bus ID / location ID: 0 / 2 / 0
Compute Mode:
< Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >
Device 2: "GeForce GTX 470"
CUDA Driver Version / Runtime Version 7.5 / 7.5
CUDA Capability Major/Minor version number: 2.0
Total amount of global memory: 1280 MBytes (1341849600 bytes)
(14) Multiprocessors, ( 32) CUDA Cores/MP: 448 CUDA Cores
GPU Max Clock rate: 1215 MHz (1.22 GHz)
Memory Clock rate: 1674 Mhz
Memory Bus Width: 320-bit
L2 Cache Size: 655360 bytes
Maximum Texture Dimension Size (x,y,z) 1D=(65536), 2D=(65536, 65535), 3D=(2048, 2048, 2048)
Maximum Layered 1D Texture Size, (num) layers 1D=(16384), 2048 layers
Maximum Layered 2D Texture Size, (num) layers 2D=(16384, 16384), 2048 layers
Total amount of constant memory: 65536 bytes
Total amount of shared memory per block: 49152 bytes
Total number of registers available per block: 32768
Warp size: 32
Maximum number of threads per multiprocessor: 1536
Maximum number of threads per block: 1024
Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
Max dimension size of a grid size (x,y,z): (65535, 65535, 65535)
Maximum memory pitch: 2147483647 bytes
Texture alignment: 512 bytes
Concurrent copy and kernel execution: Yes with 1 copy engine(s)
Run time limit on kernels: No
Integrated GPU sharing Host Memory: No
Support host page-locked memory mapping: Yes
Alignment requirement for Surfaces: Yes
Device has ECC support: Disabled
Device supports Unified Addressing (UVA): Yes
Device PCI Domain ID / Bus ID / location ID: 0 / 5 / 0
Compute Mode:
< Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >
Device 3: "GeForce GTX 470"
CUDA Driver Version / Runtime Version 7.5 / 7.5
CUDA Capability Major/Minor version number: 2.0
Total amount of global memory: 1280 MBytes (1341849600 bytes)
(14) Multiprocessors, ( 32) CUDA Cores/MP: 448 CUDA Cores
GPU Max Clock rate: 1215 MHz (1.22 GHz)
Memory Clock rate: 1674 Mhz
Memory Bus Width: 320-bit
L2 Cache Size: 655360 bytes
Maximum Texture Dimension Size (x,y,z) 1D=(65536), 2D=(65536, 65535), 3D=(2048, 2048, 2048)
Maximum Layered 1D Texture Size, (num) layers 1D=(16384), 2048 layers
Maximum Layered 2D Texture Size, (num) layers 2D=(16384, 16384), 2048 layers
Total amount of constant memory: 65536 bytes
Total amount of shared memory per block: 49152 bytes
Total number of registers available per block: 32768
Warp size: 32
Maximum number of threads per multiprocessor: 1536
Maximum number of threads per block: 1024
Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
Max dimension size of a grid size (x,y,z): (65535, 65535, 65535)
Maximum memory pitch: 2147483647 bytes
Texture alignment: 512 bytes
Concurrent copy and kernel execution: Yes with 1 copy engine(s)
Run time limit on kernels: No
Integrated GPU sharing Host Memory: No
Support host page-locked memory mapping: Yes
Alignment requirement for Surfaces: Yes
Device has ECC support: Disabled
Device supports Unified Addressing (UVA): Yes
Device PCI Domain ID / Bus ID / location ID: 0 / 6 / 0
Compute Mode:
< Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >
> Peer access from Tesla C2050 (GPU0) -> GeForce GT 430 (GPU1) : No
> Peer access from Tesla C2050 (GPU0) -> GeForce GTX 470 (GPU2) : Yes
> Peer access from Tesla C2050 (GPU0) -> GeForce GTX 470 (GPU3) : Yes
> Peer access from GeForce GT 430 (GPU1) -> Tesla C2050 (GPU0) : No
> Peer access from GeForce GT 430 (GPU1) -> GeForce GTX 470 (GPU2) : No
> Peer access from GeForce GT 430 (GPU1) -> GeForce GTX 470 (GPU3) : No
> Peer access from GeForce GTX 470 (GPU2) -> Tesla C2050 (GPU0) : Yes
> Peer access from GeForce GTX 470 (GPU2) -> GeForce GT 430 (GPU1) : No
> Peer access from GeForce GTX 470 (GPU2) -> GeForce GTX 470 (GPU3) : Yes
> Peer access from GeForce GTX 470 (GPU3) -> Tesla C2050 (GPU0) : Yes
> Peer access from GeForce GTX 470 (GPU3) -> GeForce GT 430 (GPU1) : No
> Peer access from GeForce GTX 470 (GPU3) -> GeForce GTX 470 (GPU2) : Yes
deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 7.5, CUDA Runtime Version = 7.5, NumDevs = 4, Device0 = Tesla C2050, Device1 = GeForce GT 430, Device2 = GeForce GTX 470, Device3 = GeForce GTX 470
Result = PASS
I do not have the experience with CUDA to be able to tell if there are significant problems with the above output, I have only been going off of the “PASS” result so far.
Here is where my troubles start. After getting this initial PASS, I move on to test other example programs. They seem to work fine initially. I attempted the following commands in order, leading up to the generation of the errors. The first two tests with matrixMul and matrixMulCUBLAS work fine, and then I begin having issues, culminating in the machine becoming unable to pass deviceQuery a second time:
user@cuda:~/NVIDIA_CUDA-7.5_Samples/bin/x86_64/linux/release$ ./matrixMul
[Matrix Multiply Using CUDA] - Starting...
GPU Device 0: "Tesla C2050" with compute capability 2.0
MatrixA(320,320), MatrixB(640,320)
Computing result using CUDA Kernel...
done
Performance= 173.56 GFlop/s, Time= 0.755 msec, Size= 131072000 Ops, WorkgroupSize= 1024 threads/block
Checking computed result for correctness: Result = PASS
NOTE: The CUDA Samples are not meant for performance measurements. Results may vary when GPU Boost is enabled.
user@cuda:~/NVIDIA_CUDA-7.5_Samples/bin/x86_64/linux/release$ ./matrixMulCUBLAS
[Matrix Multiply CUBLAS] - Starting...
GPU Device 0: "Tesla C2050" with compute capability 2.0
MatrixA(640,480), MatrixB(480,320), MatrixC(640,320)
Computing result using CUBLAS...done.
Performance= 516.29 GFlop/s, Time= 0.381 msec, Size= 196608000 Ops
Computing result using host CPU...done.
Comparing CUBLAS Matrix Multiply with CPU results: PASS
NOTE: The CUDA Samples are not meant for performance measurements. Results may vary when GPU Boost is enabled.
Here specifically is where things take a negative turn. It should be noted that I chose to run matrixMul → matrixMulCUBLAS → simpleP2P in that order simply on a whim to try out test programs and see if they work correctly since I have isolated my CUDA install as the likely source of my issues with Caffe.
user@cuda:~/NVIDIA_CUDA-7.5_Samples/bin/x86_64/linux/release$ ./simpleP2P
[./simpleP2P] - Starting...
Checking for multiple GPUs...
CUDA error at simpleP2P.cu:63 code=10(cudaErrorInvalidDevice) "cudaGetDeviceCount(&gpu_n)"
Since the last test of simpleP2P didn’t go well I attempted to backtrack and run matrixMul again:
user@cuda:~/NVIDIA_CUDA-7.5_Samples/bin/x86_64/linux/release$ ./matrixMul
[Matrix Multiply Using CUDA] - Starting...
cudaGetDevice returned error invalid device ordinal (code 10), line(396)
cudaGetDeviceProperties returned error invalid device ordinal (code 10), line(409)
MatrixA(160,160), MatrixB(320,160)
cudaMalloc d_A returned error invalid device ordinal (code 10), line(164)
And finally, running deviceQuery a second time:
user@cuda:~/NVIDIA_CUDA-7.5_Samples/bin/x86_64/linux/release$ ./deviceQuery
./deviceQuery Starting...
CUDA Device Query (Runtime API) version (CUDART static linking)
cudaGetDeviceCount returned 10
-> invalid device ordinal
Result = FAIL
If anyone could shed some light on what is going on here I would be forever grateful. I have been working to get this running for some time with no luck.