CUDA 3.2 Driver BROKE ? Oops....

I have a Tesla S2050 with driver version 260.19.21 installed. Each of the GPUs is supposed to have 3GB of global memory and that’s what deviceQuery tells me too.

However, when I run a simple program that calls cudaMemGetInfo (see code snippet below), I find that only 307757056 (~300 MB) of the 2817982464 bytes are “free”. I get the same result for all 4 GPUs. No other jobs are running on my GPUs.

I bump into this memory issue also when I try running the SDK’s FDTD3d example:


fdtdGPU…
cudaGetDeviceCount
cudaSetDevice (device 0)
cudaMalloc bufferOut
cudaMalloc bufferIn

!!! Error # 0 at cudaMalloc ‘out of memory’.
GPU FDTD loop

fdtdGPU complete

FAILED

!!! Error # 0 at line 44 , in file src/FDTD3d.cpp !!!

Exiting…

However when I run the corresponding example in OpenCL, the code runs just fine. So is this a driver problem?

Thanks in advance,
Mark

==============NVSMI LOG==============
Timestamp : Wed Dec 22 09:40:18 2010
Driver Version : 260.19.21
Unit 0:
Product Name : NVIDIA Tesla S2050
Product ID : 920-20804-0020
Serial Number : 0383110000030
Firmware Ver : 6.2
Intake Temperature : 28 C
GPU 0:
Product Name : Tesla S2050
Serial : 0322810019679
PCI Device/Vendor ID : 6de10de
PCI Location ID : 1:7:0
Bridge Port : 0
Temperature : 58 C
Utilization :
GPU : 0%
Memory : 0%
Volatile ECC errors:
Single Bit :
FB : 0
RF : 0
L1 : 0
L2 : 0
Total : 0
Double Bit :
FB : 0
RF : 0
L1 : 0
L2 : 0
Total : 0
Aggregate ECC errors:
Single Bit :
Total : 0
Double Bit :
Total : 510203

----- Mem check code -----

#include
#include <cuda_runtime.h>

int main(int argc, char** argv)
{
cudaSetDevice(0);
size_t freeMem;
size_t totalMem;
cudaMemGetInfo(&freeMem, &totalMem);
printf(“Free = %lu, Total = %lu\n”, (long)freeMem, (long)totalMem);
return 0;
}