Tagged with tesla - NVIDIA Developer Forums

Tesla vs GTX560M this is wierd!

Bleiner — Wed, 16 May 2012 16:32:03 -0400

Hello Everyone,

So I have been working on porting over a lot of the more computationally intensive portions of my code from matlab to .ptx gpu code. I have been doing the development on my laptop where I have made HUGE increases in speed. Achieving at least a 10X speed up in a lot of areas. Now all my work is double precision, and involves millions of objects. My laptop with a gaming GPU (that by design has hampered double precision performance, and fewer processing cores) can complete a task in about 46 seconds.

I have a desktop with a Tesla C2075, that typically outperforms my laptop by a factor of two. When I bring this code over to the Tesla machine it is running anywhere from 42-48 seconds to complete the same task. Does anyone have any idea why this would be?

The only thing that comes to mind on this is that I upgraded my laptop to the 301.40 driver version to use Nsight visual studio, while the Tesla machine is still on 301.32 (or someething like that. When I attempted to upgrade the tesla machine it appears that the 301.40 drivers have been removed from the website. When I upgraded to the Beta Cuda 5 version drivers 301.53 (i think) Matlab no longer recognizes that there is a GPU attached to the system.

Could the issues be the driver? Did it improve that much from 301.27 to 301.4? Does it have anything to do with the GPU on the laptop being compute level 2.1 while tesla is 2.0? Is there a memory manager issue that 2.1 does a lot better?

The strange thing is that TESLA USED to perform at twice the speed of the laptop (and that includes all the overhead in matlab that is taking place on the CPU. So Tesla must have been significantly more than 2x faster.

Any thoughts?

Thanks
Ben

As an after thought here are the specs of the machines in question..
Tesla Work station
Dell Precision T5500
Xeon E5620 2.4ghz
12gb ram
Telsa C2075 with 6gb ram driver 301.32

Laptop
Asus G74S
Core I7 2670QM 2.2ghz
12gb Ram
GTX 560M 3gb ram Driver 301.40

Cuda, PTX and Debugging symbols

Bleiner — Wed, 28 Mar 2012 13:08:21 -0400

Hello everyone,

I have a question about .ptx files and debugging in Visual studio 10 professional. I am attempting to write some ptx code to integrate into Matlab. However I am ONLY writing the __global__ functions and I have not written a main function or any host code. The functions are fairly simple, but what I want to do is be able to debug my code when matlab is running it. I am compiling my code using nvcc -ptx 'functionname.cu'. and when I try to use nvcc -G -ptx 'functionname.cu' to get debugging information nothing else is returned except the .ptx file. I should mention that I am relatively new to visual studio and that is why I am compiling using the command line.

I believe that if I had the symbols generated I could attach the process to Matlab and debug it. Since .ptx is supposed to be just in time compiled code does it not allow me to have debugging symbols?

Any help would be greatly appreciated
Ben

OpenCL callbacks scheduling

rjmarques — Thu, 10 May 2012 09:59:48 -0400

Greatings,

I am having a huge overhead when using callbacks on linux. After I enqueue the necessary read operation, I set a callback for apropriate threatment. The read takes less then a milisecond to complete, however the callback is only issued after, about, 19 miliseconds. Is this a driver issue?

The graphics card is a Tesla C2050.
The driver version is 295.41.
And the GCC version is 4.4.3.

Thanks,
Ricardo Marques

i write this program but its not work, please can you tell me what wrong with it , im new programmer

analdo — Thu, 10 May 2012 04:42:48 -0400

#include
#include
#define N 16

__global__ void matAdd(float* A, float* B, float* C)
{
int i= threadIdx.x + threadIdx.y*blockDim.x;
C[i]= A[i] + B[i];
}
int main()
{
int i;
int numBlocks = 1;
size_t size = N* sizeof(float);
dim3 threadsPerBlock (4, 4);

// Initialisation des vecteurs
float A[]= {1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16};
float B[]= {1.0,1.1,1.2,1.3,1.4,1.5,1.6,1.7,1.8,1.9,2.0,2.1,2.2,2.3,2.4,2.5};
float C[N];

// Allocation des vecteurs dans la mémoire du GPU
float* d_A;
cudaMalloc(&d_A, size);
float* d_B;
cudaMalloc(&d_B, size);
float* d_C;
cudaMalloc(&d_C, size);

// Copie des vecteurs de la mémoire du CPU vers la mémoire du GPU
cudaMemcpy(d_A, A, size, cudaMemcpyHostToDevice);
cudaMemcpy(d_B, B, size, cudaMemcpyHostToDevice);

// Execution du kernel
matAdd<<>>(A, B, C);

cudaMemcpy(C,d_C,size,cudaMemcpyDeviceToHost);

printf("C= {");
for(i=0;i {
printf("%2.2f ", C);
}
printf("}\n");
cudaFree(d_A), cudaFree(d_B), cudaFree(d_C);
}

Streaming Multiprocessors

Saouli — Sat, 28 Apr 2012 16:08:59 -0400

Hello
How can we know the Number of Streaming Multiprocessors in Nvdia devices and how much threads can take like the Nvidia G80 have i guess 16 SMP each one can take 8 blocks of threads and max thread shoud be 768 thread

cudaEvent timers vs. Host timers

dlowell — Wed, 25 Apr 2012 16:35:14 -0400

In doing performance testing we are trying two methods of timing.
Cuda event based timing and system timer.
We are running on a Fermi 2070 sm_20, with CUDA SDK 4.2
I haven't seen anything the internet that makes it clear whether one is superior over the other in terms of timing. I've seen the cudaDeviceSynchronize() used for this purpose,...any insight would be valuable.

Thanks ahead of time!

The first is the built in event based:

cudaEventRecord(start, 0);
kernel<<>>(devy,devx,alpha,length);
cudaEventRecord(stop, 0);
cudaEventSynchronize(stop);

The second is a system time based timer using a barrier.

  start = getclock();
  kernel<<>>(devy,devx,alpha,length);
  cudaDeviceSynchronize();
  finish = getclock();

where getclock() is defined as:

#include 
double getclock(){
  struct timezone tzp;
  struct timeval tp;
  gettimeofday (&tp, &tzp);
  return (tp.tv_sec + tp.tv_usec*1.0e-6);
}

The kernel we are running is:

__global__ void  kernel(double* devY,double* devX, double alpha, int length){
 /* w <- y + alpha*x */
  int tid = blockIdx.x*blockDim.x+threadIdx.x;
  if(tid    devY[tid]=alpha*devX[tid];
  }
}

uncertain results in CUDA program

tanjun2525 — Tue, 01 May 2012 05:07:32 -0400

Hi,
I'm trying to write a matrix iteration program by CUDA. But I get some problems.
The whole program procedures are: CPU assign the matrix to GPU; GPU iterate the matrix following a fomular; CPU get back the new matrix; CPU judge whether the matrix is convergent, if it is, stop iteration, otherwise, continue.
Matrices are transformed to 1-D arrays while delivering between CPU and GPU.
Now I cannot get right results. Because the iteration process is uncertain. Different results would be obtained while excuting the program several times.
Each element of the matrix is assigned one thread to iterate. So, I defined n1*n2 threads to iterate an n1*n2 matrix.
I cannot figure out why the result is uncertain.
Maybe some "volatile" keywords are needed. But I dont know how to define a volatile variable in global memory. I tried, but "error: argument type 'volatile double *' is imconpatible with parameter of type 'void *'" was reported while assigning values to them using cudaMemcpy.
The codes are shown following. I'm sorry for my poor English.
Any suggestions would be highly appreciated.

CPU_function{
const int num_threads = n1 * n2;
const int threadsPerBlock = 16 * 16;
const int blocksPerGrid = (num_threads + threadsPerBlock - 1) / threadsPerBlock;

...//variables defining and values assigning

do
{
if(times > 1000)
break;

// assign the new result to the next iteration step
CUDA_SAFE_CALL( cudaMemcpy( d_Sk2, h_Sk2, n1*n2*sizeof(double), cudaMemcpyHostToDevice) );

// core function
ComputeSim_Kernel<<>>(d_Sim,
d_Sk2,
d_Sk1,
d_adjMatrix_Yeast,
d_adjMatrix_Fly,
d_yeastIndex,
d_flyIndex,
n1, n2
);
cudaThreadSynchronize();

// get the new matrix from GPU
CUDA_SAFE_CALL( cudaMemcpy( h_Sk1, d_Sk1, n1*n2*sizeof(double), cudaMemcpyDeviceToHost) );

// get the maxum element of the matrix
maxw = h_Sk1[0];
for(int i=1; i {
if(h_Sk1[i] > maxw)
maxw = h_Sk1[i];
}

minSk = 1;
maxDeltaSk01 = 0;
maxDeltaSk02 = 0;
deltaSk01 = 0;
deltaSk02 = 0;

...// some compute for judging convergency

tmpsk = h_Sk2;
h_Sk2 = h_Sk1;
h_Sk1 = h_Sk0;
h_Sk0 = tmpsk;

++times;

}while((maxDeltaSk01 > 0.01) && (maxDeltaSk02 > 0.01));
}

Codes on device:

#ifndef __COMPUTESIM_KERNEL_H__
#define __COMPUTESIM_KERNEL_H__

#include "cuda.h"
#include "cutil.h"
#include "computeSim_kernel.h"

const int threadsPerBlock = 256;

inline __global__ void
ComputeSim_Kernel(double *Sim, double *Sk2, double *Sk1, int *adjMatrix_Yeast,
int *adjMatrix_Fly, int *yeastIndex, int *flyIndex, int n1, int n2)
{
for(int i=0; i yeastIndex[i] = 0;
for(int j=0; j flyIndex[j] = 0;

unsigned int tid = threadIdx.x + blockIdx.x * blockDim.x;

const unsigned int index = threadIdx.x;
const unsigned int stride = blockDim.x * gridDim.x;
int iIndex, jIndex;
int iDegree, jDegree;
double N1, N2;

N1 = 0.0;
N2 = 0.0;
iDegree = 0;
jDegree = 0;

// iterate the n1*n2 matrix
while(tid < (n1*n2))
{
Sk1[tid] = 0;// for storing the new value
if(Sim[tid] == 0)// if the element is 0, no need to iterate
{
tid += stride;
continue;
}

iIndex = tid / n2; // get the row index in matrix
jIndex = tid % n2; // get the column index in matrix

// some data structure for iteration
for(int i=0; i yeastIndex[i] = adjMatrix_Yeast[iIndex * (n1+1) + i];
iDegree = adjMatrix_Yeast[iIndex * (n1+1) + n1];

// some data structure for iteration
for(int j=0; j flyIndex[j] = adjMatrix_Fly[jIndex * (n2+1) + j];
jDegree = adjMatrix_Fly[jIndex * (n2+1) + n2];

// compute N1 for iteration
if((iDegree != 0) && (jDegree != 0))
{
for(int i=0; i {
if(yeastIndex[i] == 1)
for(int j=0; j {
// a2<->a, b2<->b
if(flyIndex[j] == 1)
N1 += Sk2[i * n2 + j];
}
}
N1 /= (iDegree * jDegree);
}
else if((iDegree == 0) && (jDegree == 0))
{
for(int i=0; i {
for(int j=0; j N1 += Sk2[i * n2 + j];
}
N1 /= (n1 * n2);
}
else
N1 = 0;

// compute N2 for iteration
if((iDegree != n1) && (jDegree != n2))
{
for(int i=0; i {
if(yeastIndex[i] == 0)
for(int j=0; j {
// a2 !<->! a, b2 !<->! b
if(flyIndex[j] == 0)
N2 += Sk2[i * n2 + j];
}
}
N2 /= ((n1 - iDegree) * (n2 - jDegree));
}
else if((iDegree == n1) && (jDegree == n2))
{
for(int i=0; i {
for(int j=0; j N2 += Sk2[i * n2 + j];
}
N2 /= (n1 * n2);
}
else
N2 = 0;

// update the matrix using N1 and N2
Sk1[tid] = (N1 + N2)/2 * Sim[tid];

tid += stride;

} // while(tid}

#endif // __COMPUTESIM_KERNEL_H__

Level of detail in Nsight profiling?

obuzko — Thu, 26 Apr 2012 20:00:54 -0400

Hi all,
I'm trying to optimize a fairly complex kernel that includes quite a few functions. Is Parallel Nsight capable of determining function-level performance bottlenecks within a kernel?
I'm using version 2.0 with VS2010 and a Tesla2050, and it appears to be limited to the kernel as a unit with no further breakdown. Can the latest version provide function-level profiling? If not, could someone suggest an alternative (if available)?
Thanks in advance

Sasha

M2050 performance and bandwidth test

ddjbrown — Fri, 20 Apr 2012 14:11:27 -0400

We have 44 IBM iDataPlex dx360-M3 machines with two M2050 GPUs attached via PCIe. The results of running bandwidthTest vary by up to 25% on various machines, but in general seem to be about half of what we would expect from PCIe Gen2. Here's the output from the best-performing node:

Device 0: Tesla M2050
Quick Mode

Host to Device Bandwidth, 1 Device(s), Paged memory
Transfer Size (Bytes) Bandwidth(MB/s)
33554432 3972.9

Device to Host Bandwidth, 1 Device(s), Paged memory
Transfer Size (Bytes) Bandwidth(MB/s)
33554432 3339.3

Device to Device Bandwidth, 1 Device(s)
Transfer Size (Bytes) Bandwidth(MB/s)
33554432 123992.6

[bandwidthTest] - Test results:
PASSED

Also, nvidia-smi on two nodes shows the "SM" clock rate as half of MAX, and one node shows PCIe Gen 1 rather than 2. Finally, there seems to be no temperature and fan info, and a lot of other stuff shows up as N/A. WTF???

Here is the nvidia-smi excerpt:
GPU Link Info
PCIe Generation
Max : 2
Current : 1
Link Width
Max : 16x
Current : 16x
Fan Speed : N/A
.
.
.
Temperature
Gpu : N/A
Power Readings
Power Management : N/A
Power Draw : N/A
Power Limit : N/A
Clocks
Graphics : 270 MHz <<<<<
SM : 540 MHz <<<<<
Memory : 1546 MHz
Max Clocks
Graphics : 573 MHz
SM : 1147 MHz
Memory : 1546 MHz

Compute-modify with __threadfence()

chrism0dwk — Wed, 18 Apr 2012 11:44:28 -0400

Hi All,

I have an algorithm in which requires a summary measure of a dataset to be computed before modifying that dataset (necessarily in that order). The modification is only small -- in fact, only one element of a large array is changed. My current code does the following:

1. myKernel<<>>(...) ;
2. (thrust::device_vector) theDataset[modIdx] = modVal ; // Memcpy implemented as a thrust::device_vector

According to the profiler, I get a long latency associated with the cudaMemcpy() call. I wondered (since I'm actually passing the modified value to the kernel anyway) if there was a sensible way of getting the kernel to do the update, but only after all threads have done their bit of the calculation?

I wondered about the following kernel definition:

__global__
void
myKernel(float modifiedVal, int modifiedIdx, float* dataset,...)
{
  int tid = threadIdx.x + blockIdx.x*blockDim.x;

  // Calculations here...

  __threadfence(); // threads in all blocks must have 
                   // read from dataset before modification

  if(tid == 0) dataset[modifiedIdx] = modifiedVal;
}

might be a good idea? The bit that concerns me is the "if(tid==0)" line -- does this mean that the first block will be sat idle, consuming resources, whilst other blocks are executed around it? Is there a better way to achieve my aim?

Thanks,

Chris

Remote Debugging with EC2 GPU Cluster Instance

psstatuser — Wed, 11 Apr 2012 20:35:29 -0400

Hi,

I am trying to set up a parallel nsight remote debugging infrastructure with one of Amazon's EC2 GPU instances. My setup is as follows:

1. A local windows 7 x64 machine that has visual studio 2008 installed, all configured for CUDA development. I am able to compile and run CUDA programs locally without issue.

2. A remote EC2 GPU instance with 2 telsa GPU's on it and all nvidia drivers and nsight monitor installed.

I have an ssh tunnel between the two machines, forwarding port 8000 from the local machine to the remote machine running the nsight monitor. When I start nsight debugging, I am able to connect to the remote machine, however, I receive an error complaining that the remote machine is connected via remote desktop. After doing some research in the various forums, I noticed that the session 0 isolation in windows server 2008 may be the culprit and that many people have succssfully debugged running a remote session via the console using a remote software like VNC. See here:

http://forum-archive.developer.nvidia.com/index.php?showtopic=5371&st=20&p=18346&#entry18346

However, this requires physical access to the remote machine, which I do not have, in order to start the VNC server and connect to the remote machine in a non session 0 environment. As this is an Amazon EC2 instance, I cannot do this.

I was under the impression that the TCC (telsa compute cluster) drivers solved this problem.

Any ideas on how to make this work?

Thanks.

tesla card issue

mushymac2000 — Thu, 23 Feb 2012 23:16:28 -0500

Hi,

As you know, I'm trying to get the CUDA Toolkit up and running on the SUSE Linux box for the job.

I managed to download and install the latest Nvidia Driver (ver. 285.05.33), CUDA Toolkit (4.1) and GPU Computing SDK.

When I attempted to programmers included in the SDK, I got the following error.

CUDA Runtime API error 39: uncorrectable ECC error encountered.

When I disabled the ECC feature on the card, I was able to run the programs

In the attachment I've included the output I got with ECC enabled and ECC disabled in two separate folders. The programs I tested with were
• deviceQuery
• bandwidthTest
• matrixMul
• oceanFFT

The details on the environment is as follows (refer attachment for more info);
• SUSE Linux Enterprise Server 11 - x86_64 (kernel 2.6.32.12-0.7)
• gcc 4.3.4
• Nvidia driver 285.05.33
• Cuda Toolkit 4.1

The details on the Tesla Card is as follows;
• Tesla C2075 (S/N: 0323111025730)
• Bios version 70.10.46.00.05

The Motherboard is an Asus Maximus IV Extreme-Z.

PhysX with CUDA on Linux

Bonaducci — Tue, 03 Apr 2012 23:12:10 -0400

Actually, I'm working now on physics for MMO RPG game with big ability of modyfying scenery. Basicly I'd like to use PhysX to perform most of expected actions, but I have linux server running on tesla. I've been using this mainly to calculate sparse matrix-vector multiplications and other things like this. Of cource CUDA with linux works fine, but I'm new with PhysX and I'm wondering, if it's possible to use CUDA for PhysX on linux OS?

device query

vishalthelegend — Mon, 19 Mar 2012 12:49:11 -0400

Can anybody let me know how many grids, blockspergrid and threadperblocks are present in
geforce 8200m G
guys help me out

how to use multiple blocks??

vishalthelegend — Fri, 23 Mar 2012 12:15:53 -0400

how do i make use of multiple blocks in cuda
i am using dim3 gridsize(n,n)
but it is not wrking unless n=1...that is i am able to use only one block
help me out..
i have geforce 8200 m gpu

unexpected situation when using nvidia-smi command

AlanKao2012 — Fri, 23 Mar 2012 03:44:26 -0400

Hi there:

If anything goes fine, it should be like this When I keyed in "nvidia-smi":
+------------------------------------------------------+
| NVIDIA-SMI 2.285.05 Driver Version: 285.05.33 |
|-------------------------------+----------------------+----------------------+
| Nb. Name | Bus Id Disp. | Volatile ECC SB / DB |
| Fan Temp Power Usage /Cap | Memory Usage | GPU Util. Compute M. |
|===============================+======================+======================|
| 0. Tesla C2070 | 0000:02:00.0 On | Off |
| 38% 80 C P8 Off / Off | 1% 54MB / 6143MB | 0% Default |
|-------------------------------+----------------------+----------------------|
| 1. Tesla C2070 | 0000:03:00.0 Off | Off |
| 30% 57 C P8 Off / Off | 0% 10MB / 6143MB | 0% Default |
|-------------------------------+----------------------+----------------------|
| 2. Tesla C2070 | 0000:83:00.0 Off | Off |
| 30% 66 C P8 Off / Off | 0% 10MB / 6143MB | 0% Default |
|-------------------------------+----------------------+----------------------|
| 3. Tesla C2070 | 0000:84:00.0 Off | Off |
| 30% 67 C P8 Off / Off | 0% 10MB / 6143MB | 0% Default |
|-------------------------------+----------------------+----------------------|
| Compute processes: GPU Memory |
| GPU PID Process name Usage |
|=============================================================================|
| No running compute processes found |
+-----------------------------------------------------------------------------+

But sometimes, it halts for a while, and:
+------------------------------------------------------+
| NVIDIA-SMI 2.285.05 Driver Version: 285.05.33 |
|-------------------------------+----------------------+----------------------+
| Nb. Name | Bus Id Disp. | Volatile ECC SB / DB |
| Fan Temp Power Usage /Cap | Memory Usage | GPU Util. Compute M. |
|===============================+======================+======================|
| 0. Tesla C2070 | 0000:02:00.0 On | Off |
| 38% ERR! P8 Off / Off | 1% 54MB / 6143MB | 99% Default |
|-------------------------------+----------------------+----------------------|
| 1. Tesla C2070 | 0000:03:00.0 Off | Off |
| 30% 57 C P8 Off / Off | 0% 10MB / 6143MB | 0% Default |
|-------------------------------+----------------------+----------------------|
| 2. Tesla C2070 | 0000:83:00.0 Off | Off |
| 30% 66 C P8 Off / Off | 0% 10MB / 6143MB | 0% Default |
|-------------------------------+----------------------+----------------------|
| 3. Tesla C2070 | 0000:84:00.0 Off | Off |
| 30% 67 C P8 Off / Off | 0% 10MB / 6143MB | 0% Default |
|-------------------------------+----------------------+----------------------|
| Compute processes: GPU Memory |
| GPU PID Process name Usage |
|=============================================================================|
| No running compute processes found |
+-----------------------------------------------------------------------------+

It happens to a supermicro 7046GT server with 4 tesla C2070. Each time this error occurs, I always reboot it, but soon I would find that it would not successfully boot up and need to be rebooted again.
It is so disturbing and it DOES cause some errors when trying to do computing on GPU.

Is there anyone who has encountered this situation?
Is it more likely to be a hardware problem(motherboard, GPU, ...)? or software problem(bad usage of program, driver, OS)?

Any reply will be appreciated. Thank you very much.

Enabling Double Precision on Tesla C2050 Using arch=sm_20

waltee1000 — Mon, 19 Mar 2012 08:53:09 -0400

Hello All,
I tried to declare double in my .cu file and compile the code using -arch=sm_20, the code compiles without any error or warning but when I run the code I get a following message and the run aborts.
*** glibc detected *** ./a.out: double free or corruption (out): 0x0000000000a83da0 ***
======= Backtrace: =========
/lib/libc.so.6[0x7f38fb293928]
/lib/libc.so.6(cfree+0x76)[0x7f38fb295a36]
./a.out[0x41b744]
/lib/libc.so.6(__libc_start_main+0xe6)[0x7f38fb23e1a6]
./a.out(__gxx_personality_v0+0xa1)[0x401449]
======= Memory map: ========

If I replace the double with float everything works fine. the code complies, runs to completion with correct answers.

I saw on the internet that the only change needed to make a double data type work is to compile the code with the -arch=sm_20 argument. why is it not working even after doing setting the argument?

Regards,

Walter

Microbenchmarking using clock()

fwende — Thu, 15 Mar 2012 14:40:04 -0400

hi,

i'am going to do some microbenchmarking on a tesla m2090. for that purpose i decided to use the clock() function within the respective kernels. unfortunately, the difference of timestamps returned from successive calls of the clock() function seems to depend on the number of threadblocks used and the number of threads within the thread blocks. consider the following kernel:


__global__ void kernel( uint *time ) {

  uint
    t1,t2;

  t1 = clock();
  t2 = clock();

  time[blockIdx.x*blockDim.x+threadIdx.x] = t2-t1;

}

if i run the kernel with less than 17 thread blocks and less than 385 threads per block, 't2-t1' is equal to 24. if the number of thread blocks is less than 17 but the number of threads per block is between 385 and 416 't2-t1' is 24 for threads within half the number of warps, and 28 for threads within the other warps. if the number of threads is between 417 and 448 't2-t1' is 28 for all threads. with increasing number of threads per block the value of 't2-t1' also increases.

the same happens if the number of thread blocks becomes larger than 16.

for me it is not obvious what causes this behavior.

does someone has any idea.

thanks

CUDA Runtime Error 39

dyanwithu — Wed, 29 Feb 2012 04:44:46 -0500

Hi everybody,
I meet a Cuda Runtime Error 39
and my device is tesla c2070 with ECC enabled

does anyone know the details of this error code? I just can't find it.

thx

System with 8 GPUs and 1 IOH chip

Voronyuk Mikhail — Wed, 26 Oct 2011 04:18:00 -0400

Hello!

In webinar "Multi-GPU and Host Multi-Threading Considerations" by Dr Paulius Micikevicius (http://developer.download.nvidia.com/CUDA/training/cuda_webinars_multi_gpu.pdf) was example with 8-GPU-system which has only 1 IOH chip (p.19) (so it is possible to use GPUDirect technology for peer copy between each couple of devices).

Could you please tell what exact this system is.

Thanks in advance

Overclocking/underclock

jbraswe3 — Tue, 06 Mar 2012 10:26:29 -0500

Is it possible to overclock/underclock the Tesla GPU programmatically? I have read and seen things about using MSI Afterburner and other such programs to adjust the clock speeds but for my proposes I would need my program to be able to dynamically adjust the speeds.

Any info would be greatly approciated.

shall I install the developer drivers?

martinaG — Fri, 24 Feb 2012 05:18:18 -0500

HI all,

I need to set up my workstation with CUDA. It already has the nVidia runtime installed


NVRM version: NVIDIA UNIX x86_64 Kernel Module  280.13  Wed Jul 27 16:53:56 PDT 2011

I installed the toolkit 4.1, then I saw the Developer Drivers Downloads section.
What are them? shall i install these drivers too?
In the installation guide, they suggest to install the drivers before installing the toolkit.

So, in the case I need the dev drivers too, shall i re-install the toolkit?

Thank you in advance for any help you would give me.

Bests,

Martina

My OS: Ubuntu 11.10
My GPUs: Quadro FX 4800
Tesla C1060

NVIDIA Parallel Nsight 2.1 Release Candidate 2 now available!

Sebastien Domine — Mon, 05 Dec 2011 19:16:15 -0500

NVIDIA Parallel Nsight 2.1 Release Candidate 2 now available!

Dear Parallel Nsight User,

Building on the NVIDIA Parallel Nsight™ 2.1 Release Candidate 1 release with multiple bug fixes and stability improvements, we are proud to announce the release of NVIDIA Parallel Nsight™ 2.1 Release Candidate 2. This release
brings support for the new CUDA Toolkit 4.1 Release Candidate 2, which can be downloaded under the CUDA Registered Developer Program (www.developer.nvidia.com/join). Parallel Nsight 2.1 adds a number of new features to enhance debugging and profiling capabilities.

This release requires NVIDIA Display Driver Release 285.86, available on the same download site.

Traced workloads can now navigate the dependencies and call stack to allow the developer to follow through GPU workloads, corresponding API calls and host code that was the cause of the activity.
CUDA warp watch visualizes variables and expressions across an entire CUDA warp.

Typecasting to custom struct in kernel

avion85 — Sun, 26 Feb 2012 17:17:30 -0500

Since this is my first post, greetings to everyone.

I've encountered a problem regarding casting to a custom struct in a kernel, hopefully someone else was in the same situation.

I'm passing as a parameter into a kernel from a .cu file a large array which I would like to cast into a struct and access as an array of structures.

pseudo-code:

kernels.cu (with nvcc)


struct myMatrix
{
	float e[6];
};
__global__ myKernel(float *raw, myMatrix *p){
 myID = int me_idx = blockIdx.x * blockDim.x + threadIdx.x;

 myMatrix m = p[myID];	  //does not work - "???" in nsight for all values 

 myMatrix n =((myMatrix *)raw)[myID];     //does not work also - "???"

 float a = raw[0];    //works and I get correct single float values, but unstructured

 float 4 b = ((float4*)raw)[0];  //works and I get correct tuples

//what I want:
Matrix m = p[myID];
float something = m.e[3];
}

main.cu (with microsoft c compiler)


float *p = [large array];
myKernel<<>>(p,(myMatrix*)p);

I am using Parallel Nsight to inspect the values and what I get is "???" while stepping through the program. I have never had problems if I use the built-in types like float4. However, I would of course, like to have my own structures working properly.
Maybe the problem is in the alignment? If so, to which value to I align?

Appreciate the help.
Avion

PS.Working with Visual Studio, everything is 64bit.

EDIT: added another example that works.

GPUs stopped working after being left idling

Gary Chandler — Thu, 20 Oct 2011 13:03:01 -0400

Hello,

My university department bought a system with 2 Tesla 2050 GPUs about 1 year ago. They were working when new, but after leaving them idle for a few months they stopped working. A few days after I first tried to access them after leaving them idle, they "woke up" and worked OK again. After leaving them idle again for a few months they have stopped working again.

When they stop working it is like they do not exist.
lspci | grep -i nvidia returns nothing.

nvidia-smi returns:
NVIDIA: could not open the device file /dev/nvidiactl (No such file or directory).
Failed to allocate an RM client
Could not allocate resources!

From a CUDA code that calls CUFFT:
Cuda error in file 'cuda.cu' in line 11 : unspecified driver error.

Does anyone know what this problem might be? and is there any way to "wake" them up straight away? Instead of having to wait a few days.

My operating system is openSUSE 11.2

Thanks

Gary

Degradion Performance 4.1 over 4.0

kalman — Wed, 08 Feb 2012 09:29:40 -0500

Hi all,
due the fact our application has to not be simply fast but it should perform
some operations with fixed deadlines (we analyze a continuous radio signal)
we perform several time per day benchmarks of all our algorithm.

We are experiencing a clear degradation adopting CUDA 4.1 over the old CUDA 4.0.

I have attached 4 images showing the historical performance data of 4 algorithms
(they are not all the affected ones, but the simplest to show you the kernel code).

For all graphs the reported time is in milliseconds (y-axis).

All kernels are launched in this way:

#define BLOCK_SIZE (1<<9)

dim3 myThreads(BLOCK_SIZE);
dim3 myGrid( (aSize + BLOCK_SIZE - 1) / BLOCK_SIZE);

Kernel<<< myGrid, myThreads>>>(.....);

We have the C2050 cards with ECC off.

============================================================================
Sum of two complex vectors (2^20 complex): add_cc.png

__global__ void
VectorVectorSumKernelCC_O(const float2* aIn1,
const float2* aIn2,
float2* aOut,
const unsigned int aSize) {
const unsigned int myPos = blockIdx.x * blockDim.x + threadIdx.x;
if (myPos < aSize) {
aOut[myPos].x = aIn1[myPos].x + aIn2[myPos].x;
aOut[myPos].y = aIn1[myPos].y + aIn2[myPos].y;
}
}
============================================================================
Product of two complex vectors (2^20 complex): mul_cc.png

__global__ void
MulKernel_cv_cv_o(const float2* aIn1,
const float2* aIn2,
float2* aOut,
const unsigned int aSize) {
const unsigned int myPos = blockIdx.x * blockDim.x + threadIdx.x;
if (myPos < aSize) {
const float myReal1 = aIn1[myPos].x;
const float myReal2 = aIn2[myPos].x;
const float myImag1 = aIn1[myPos].y;
const float myImag2 = aIn2[myPos].y;
aOut[myPos].x = myReal1 * myReal2 - myImag1 * myImag2;
aOut[myPos].y = myReal1 * myImag2 + myImag1 * myReal2;
}
}
============================================================================
Product of two complex vectors (2^20 complex), in place: mul_cc_i.png

__global__ void
MulKernel_cv_cv_i(const float2* aIn,
float2* aInOut,
const unsigned int aSize) {
const unsigned int myPos = blockIdx.x * blockDim.x + threadIdx.x;
if (myPos < aSize) {
const float myTmp = aInOut[myPos].x;
const float myInR = aIn[myPos].x;
const float myInI = aIn[myPos].y;
aInOut[myPos].x = myInR * aInOut[myPos].x - myInI * aInOut[myPos].y;
aInOut[myPos].y = myInR * aInOut[myPos].y + myInI * myTmp;
}
}
============================================================================
Tone generation (2^20 vector long): tone.png

__global__ void
ComplexExpKernel(float2* aInOut,
const unsigned int aSize,
const float aMagnitude,
const float aNormalizedFrequency,
const float aInverseFrequency,
const float aPhase) {

const unsigned int myPos = blockIdx.x * blockDim.x + threadIdx.x;
if (myPos < aSize) {
const float myArgument = aNormalizedFrequency * fmodf((float)myPos, aInverseFrequency) + aPhase;
aInOut[myPos].x = aMagnitude * __cosf(myArgument);
aInOut[myPos].y = aMagnitude * __sinf(myArgument);
}
}
============================================================================

kepler device memory architecture

iourikarpov — Wed, 22 Feb 2012 21:18:04 -0500

Hello, with plenty of rumours swirling around kepler number of cores, processors, etc., there has beed very little info (that I could find) on the L1 memory size, shared, constant memory sizes, and other key metrics that are extremely important to developers. Does anyone have any links, leaks, or ideas on what the kepler architecture will bring on this front?

Thanks in advance, Joe

Out of memory on first CUDA runtime call

chrism0dwk — Thu, 16 Feb 2012 06:51:05 -0500

Hi all,

I've hit a strange bug while porting my CUDA code from my MacBook Pro (NVIDIA GeForce GT 330M) to my university's HPC gpu node with its two Tesla M2050s. My code is as follows: I have 2 classes defined as follows:


class MatLikelihood
{
 // Class to manage maths environment
 MatLikelihood(DataFromMain& data)
  {
    gpu_ = new GpuLikelihood(data.data, data.number, data.moredata, data.etc);
    ...
  }
private:
  GpuLikelihood* gpu_;
}

class GpuLikelihood
{
  // Class to manage GPU computation
  GpuLikelihood(float* data, int number, float* moredata, int etc)
  {
    int numDevices; cudaError_t err;
    err = cudaGetDeviceCount(&numDevices);
    if (err != cudaSuccess) throw AnException("cudaGetDeviceCount bombs out",err);

    
  }
}

These classes are in separate files (MatLikelihood.hpp/cpp and GpuLikelihood.hpp/cu), and I compile .cpp files with the system g++ and .cu files with nvcc, before linking with the system g++. GpuLikelihood makes use of both cudablas_v2 and cusparse (v1) libraries.

On my MacBook Pro (CUDA 4.1, GCC 4.2.1) all works well. However, I have a problem on the HPC node (CUDA 4.0.17, GCC 4.3.4). Every single call I make to the CUDA runtime, even cudaGetDeviceCount, fails with error 2 "out of memory"! This also occurs if I try to explicitly set the device id using cudaSetDevice. cuda-memcheck doesn't tell me anything. cuda-gdb gives me an error "The CUDA driver failed initialization. (error code = 20)".

The interesting thing is that if I write a standalone CUDA test code (eg. call cudaGetDeviceCount, do a cudaMalloc or two, and end with cudaFrees) in one .cu file, all works fine. I'm wondering if my problem is anything to do with my class hierarchy, or the separate gcc linking?

Any ideas?

Thanks,

Chris

Developing Windows applications for the Tesla GPU

paysonwelch — Fri, 10 Feb 2012 06:51:21 -0500

Greetings I have a very quick question. I am a Windows application developer who needs to crunch lots of data. Can anyone tell me if it is relatively easy to write code that can be offloaded to a Tesla GPU using Visual Studio?

Specifically my application has a GUI front-end, at the very least is it possible to compile a C++ library that can be included without much hassle?

Also any tips or suggestions for research would be greatly appreciated. Thanks in advance.

Upgrading an S870

paysonwelch — Sat, 11 Feb 2012 18:50:29 -0500

I'm looking at an S870 1U rackmount enclosure with 4xC870 GPU cards. I am wondering if it is possible to swap out the C870 cards for something with a little more punch. Does anyone know if this is possible? E.g. if I replace the C870 cards will the S870 accept new ones and continue to function properly?

Concurrent Kernel Execution - Scheduling Mechanism

fwende — Mon, 06 Feb 2012 16:51:07 -0500

currently, i'am doing some experiments on how to efficiently adapt concurrent kernel execution to my projects. to get a deeper insight into how kernels are scheduled on the gpu/device, i played around with a small kernel doing some simple math on a vector of O(10^6) floats. the kernel itself is designed such that it uses just one threadblock consisting of 256 threads, so that each kernel can be mapped onto one multiprocessor.

if i now run n=32 of these kernels (each of them manipulating its own vector) using the same stream, i get a total runtime for all kernels which is t_0=432ms. running all 32 kernels using 32 different streams gives a total runtime t_1=28ms, which is approximately t_0/15, which is close to 16 (i run the program on a tesla m2090, which is capable of running up to 16 kernels concurrently). using just 16 different streams also results in t_1=28ms, which is clear, as the device allows for at most 16 concurrent kernels at the same time.


#define NUM_KERNELS 32
#define NUM_STREAMS 16
#define NUM_VEC_ELEMENTS ( 1024*1024 )

static float *d_array[NUM_KERNELS];
static float *h_array[NUM_KERNELS];

static __global__ void kernel( float *ptr ) {
  // do some math
}

int main() {

  cudaStream_t 
    streamPool[NUM_KERNELS],
    *stream[NUM_KERNELS];

  // create arrays on device & host.
  for( int i=0; i    cudaMalloc( (void **) &d_array[i], NUM_VEC_ELEMENTS*sizeof( float ) );
    h_array[i] = new float[NUM_VEC_ELEMENTS];
  }

  // fill 'h_array' at random using drand48(),
  // and copy data to device ('d_array').

  // create 'NUM_KERNELS' streams.
  for( int i=0; i    cudaStreamCreate( &streamPool[i] );

  // map streams onto 'streamPool' using this pattern:
  // 0 1 2 3 ... 15 0 1 2 3 ... 15
  for( int i=0; i    stream[i] = &streamPool[i%NUM_STREAMS];

  // start timer, and then run kernels.
  for( int i=0; i    kernel<<< 1, 256, 0, ( *stream[i] ) >>>( d_array[i] );

  // cudaDeviceSynchronize & stop timer.

  // copy data from device in order to check results.

  // free memory.

  return 0;

}

up to this point, all is well. if i now switch to the following stream mapping


// code as above

int main() {
  ...
  // map streams onto 'streamPool' using this pattern.
  // 0 0 1 1 2 2 3 3 ... 15 15
  for( int i=0; i    stream[i] = &streamPool[i/( NUM_KERNELS/NUM_STREAMS )];
  ...
}

the total runtime is t_2=230ms, which gives a speedup of almost a factor 2 compared with serial kernel execution. my understanding of this is as follows:

kernels sent to gpu are placed into some kind of gpu-internal queue (i have a dequeue in mind). say we have some kind of scheduler on the gpu, then this scheduler takes kernels from the queue's front and makes them run on any available multiprocessor. if successive kernels in the queue should run on different streams, the scheduler is free to run them concurrently on the device, but at most 16 of them. if two (or more) successive kernels should run on the same stream, the scheduler takes the first of them and then stops dequeuing kernels from the queue, since they may be not independent (it also stops dequeuing even if after the stop-dequeuing-kernel there is a kernel that should run on a different stream). it restarts dequeuing kernels if the previous kernel (which makes the scheduler stop dequeuing) is finished.
with respect to my second code-sample (which gives a speedup of almost 2, although 16 different streams are used), this would explain the bad performance. instead of 16 kernels, there are just 2 kernels that can run concurrently on the device. all in all, this happens 15 times, so that the speedup over serial execution is 32/(32-32/2+1)=1.88 -> execution time t_2=(432/1.88)ms=230ms.

my question now is: are my considerations right? is there a scheduler on the device, which acts similar to what i described above.

if yes: the speedup that can be achieved using concurrent kernel execution significantly depends on the order in which kernels are send to gpu/device.

Tesla multi copy not as fast as expected

paulvisschers — Wed, 08 Feb 2012 04:37:54 -0500

When I run the simpleMultiCopy in the SDK (4.0) on the Tesla C2050 I get the following results:

[simpleMultiCopy] starting...
[Tesla C2050] has 14 MP(s) x 32 (Cores/MP) = 448 (Cores)
> Device name: Tesla C2050
> CUDA Capability 2.0 hardware with 14 multi-processors
> scale_factor = 1.00
> array_size   = 4194304


Relevant properties of this CUDA device
(X) Can overlap one CPU<>GPU data transfer with GPU kernel execution (device property "deviceOverlap")
(X) Can overlap two CPU<>GPU data transfers with GPU kernel execution
    (compute capability >= 2.0 AND (Tesla product OR Quadro 4000/5000)

Measured timings (throughput):
 Memcpy host to device	: 2.725792 ms (6.154988 GB/s)
 Memcpy device to host	: 2.723360 ms (6.160484 GB/s)
 Kernel			: 0.611264 ms (274.467599 GB/s)

Theoretical limits for speedup gained from overlapped data transfers:
No overlap at all (transfer-kernel-transfer): 6.060416 ms 
Compute can overlap with one transfer: 5.449152 ms
Compute can overlap with both data transfers: 2.725792 ms

Average measured timings over 10 repetitions:
 Avg. time when execution fully serialized	: 6.113555 ms
 Avg. time when overlapped using 4 streams	: 4.308822 ms
 Avg. speedup gained (serialized - overlapped)	: 1.804733 ms

Measured throughput:
 Fully serialized execution		: 5.488530 GB/s
 Overlapped using 4 streams		: 7.787379 GB/s
[simpleMultiCopy] test results...
PASSED

This shows that the expected runtime is 2.7 ms, while it actually takes 4.3. What is it exactly that causes this discrepancy?

On streams and asynchronous execution

lucana — Mon, 16 Jan 2012 03:32:28 -0500

This question is just to make sure I'm understading well how CUDA streams work.

Imagine I have a for loop like this. I am using only one stream.

for (i=0; i < N; i++)
{
run operations on CPU
copy results of CPU operations to CUDA kernel with cudaMemcpyAsync
call kernel <<< , >>>
}

My understanding is that the kernel for i and the CPU operations for i+1 at the begining of the loop will execute concurrently, but the kernel won't start for i+1 until the CPU has finished computing results for i+1.

Is this right? Or will the operations on CPU and GPU never overlap? Will the kernel start before have the proper results computed from the CPU? Is it necessary to put some control flags to make sure the operations on the CPU have finished before the kernel starts?

This diagram shows what I want to do. In fact it is a pipeline, but I'm still unsure if it is possible with CUDA.

----------i = 0 -------------------- i = 1 --------------------------- i = 2
(t0) compute results on CPU
(t1) copy results to CUDA kernel -- compute results on CPU
(t2) execute kernel --------------- copy results to CUDA kernel -- compute results on CPU
(t3) ------------------------------ execute kernel --------------- copy results to CUDA kernel
(t4)-------------------------------------------------------------- execute kernel

Finally, I would like to ask if it makes sense to use CUDA streams when there is data dependacy between streams, with a pipeline like the one showed before.

Can we change the number of running cores in Tesla Card?

hugepuff — Thu, 26 Jan 2012 14:14:30 -0500

I want to know how to change the number for the running cores in Tesla card.
When you call and initial a kernel, you have pointed out how many thread and stream processor you are going to use.
However if you have 480 core gpu card,and you are running a program with only 120 thread. What the status of the other 360 stream processor?
Have they automatically closed by Nivdia power saving strategy for saving power?

Kernel panics with 285.5.32 driver

jlong — Tue, 31 Jan 2012 12:12:07 -0500

I installed the 285.5.32 driver pointed to by the Cuda 4.1
download page on to my Dell C6145 compute node. It is connected
to eight M2070 GPUs through the Dell C410X chassis. The driver
installed without error, loads without issues, and created the
/dev/nvidia* files as expected. When I run something like
nvidia-smi -L, the system immediately kernel panics.

kernel: BUG: unable to handle kernel NULL pointer dereference at 0000000000000048

Has anyone else experienced this issue?

Nvidia X11 driver does not allow stereo with Tesla M2090.

tripiana — Wed, 25 Jan 2012 08:57:38 -0500

Hi all,

I'm trying to enable the stereo 3D using a Tesla M2090 but the Nvidia X11 driver doesn't allow me to do that.

Here are some parts of the Xorg log:

[...]
(**) NVIDIA(0): Option "NoLogo" "true"
(**) NVIDIA(0): Option "Stereo" "3"
(**) NVIDIA(0): Option "NvAGP" "0"
(**) NVIDIA(0): Option "UseEDID" "false"
(**) NVIDIA(0): Option "MetaModes" "1920x1200_60 +0+0"
(==) Jan 25 13:55:41 NVIDIA(0): Using HW cursor
(**) Jan 25 13:55:41 NVIDIA(0): Onboard stereo requested (DIN connector)
(==) Jan 25 13:55:41 NVIDIA(0): Video key set to default value of 0x101fe
(==) Jan 25 13:55:41 NVIDIA(0): Not mapping the primary surface by default.
(**) Jan 25 13:55:41 NVIDIA(0): Use of AGP disabled per request
(**) Jan 25 13:55:41 NVIDIA(0): Ignoring EDIDs
(II) Jan 25 13:55:42 NVIDIA(0): Implicitly enabling NoScanout
(II) Jan 25 13:55:42 NVIDIA(0): NVIDIA GPU Tesla M2090 (GF110) at PCI:2:0:0 (GPU-0)
[...]
(--) Jan 25 13:55:42 NVIDIA(0): Connected display device(s) on Tesla M2090 at PCI:2:0:0
(--) Jan 25 13:55:42 NVIDIA(0):     none
(II) Jan 25 13:55:42 NVIDIA(0): NoScanout X screen configured with resolution 1920x1200 (from
(II) Jan 25 13:55:42 NVIDIA(0):     Virtual X configuration option)
(II) Jan 25 13:55:42 NVIDIA(0): Validated modes:
(II) Jan 25 13:55:42 NVIDIA(0): MetaMode "nvidia-auto-select":
(II) Jan 25 13:55:42 NVIDIA(0):     Bounding Box: [0, 0, 1920, 1200]
(**) Jan 25 13:55:42 NVIDIA(0): Virtual screen size configured to be 1920 x 1200
(==) Jan 25 13:55:42 NVIDIA(0): DPI set to (75, 75); computed from built-in default
(WW) Jan 25 13:55:42 NVIDIA(0): Stereo not supported with NoScanout; disabling Stereo.
[...]
(**) NVIDIA(1): Option "NoLogo" "true"
(**) NVIDIA(1): Option "Stereo" "3"
(**) NVIDIA(1): Option "NvAGP" "0"
(**) NVIDIA(1): Option "UseEDID" "false"
(**) NVIDIA(1): Option "MetaModes" "1920x1200_60 +0+0"
(==) Jan 25 13:55:42 NVIDIA(1): Using HW cursor
(**) Jan 25 13:55:42 NVIDIA(1): Onboard stereo requested (DIN connector)
(==) Jan 25 13:55:42 NVIDIA(1): Video key set to default value of 0x101fe
(==) Jan 25 13:55:42 NVIDIA(1): Not mapping the primary surface by default.
(**) Jan 25 13:55:42 NVIDIA(1): Use of AGP disabled per request
(**) Jan 25 13:55:42 NVIDIA(1): Ignoring EDIDs
(II) Jan 25 13:55:43 NVIDIA(0): Implicitly enabling NoScanout
(II) Jan 25 13:55:43 NVIDIA(1): NVIDIA GPU Tesla M2090 (GF110) at PCI:131:0:0 (GPU-1)
[...]
(--) Jan 25 13:55:43 NVIDIA(1): Connected display device(s) on Tesla M2090 at PCI:131:0:0
(--) Jan 25 13:55:43 NVIDIA(1):     none
(II) Jan 25 13:55:43 NVIDIA(1): NoScanout X screen configured with resolution 1920x1200 (from
(II) Jan 25 13:55:43 NVIDIA(1):     Virtual X configuration option)
(II) Jan 25 13:55:43 NVIDIA(1): Validated modes:
(II) Jan 25 13:55:43 NVIDIA(1): MetaMode "nvidia-auto-select":
(II) Jan 25 13:55:43 NVIDIA(1):     Bounding Box: [0, 0, 1920, 1200]
(**) Jan 25 13:55:43 NVIDIA(1): Virtual screen size configured to be 1920 x 1200
(==) Jan 25 13:55:43 NVIDIA(1): DPI set to (75, 75); computed from built-in default
(WW) Jan 25 13:55:43 NVIDIA(1): Stereo not supported with NoScanout; disabling Stereo.
[...]

"Stereo not supported with NoScanout; disabling Stereo". That's the problem. "NoScanout" is enabled by default. "NoScanout" is also possible to be enabled using "UseDisplayDevice" "none". For this reason I tried "UseDisplayDevice" "DFP-0" with no result.

After that, I looked with nvidia-settings for more info and I found this:

Attribute 'ConnectedDisplays' (nvb119:0.0): 0x00000000.
    'ConnectedDisplays' is a bitmask attribute.
    'ConnectedDisplays' is a read-only attribute.
    'ConnectedDisplays' can use the following target types: X Screen, GPU.

I suppose the problem is that physically there are no screen connectors, and the driver is not able to set stereo on for virtual displays.

I think it would be possible but I don't know if I have missed something or if I'm dealing with an unimplemented feature. Can anyone help me please? It is very important for my company, we are trying to enable stereo on a remote cluster".

Thanks in advance!

Need your expertise?

mabouali — Tue, 24 Jan 2012 15:43:49 -0500

Hi,
I have a CUDA code (a __global__ function which give calls to couple of __device__ function). Every variable is float, all the constants has f at the end (let say like 0.4f), and I have replaced all the sin, exp, ... function with sinf,expf, ...
Yet when I compile it with -arch=sm_11 it tells me double not supported demoting to float, and the provided line number is in the .ptx file, (not the .cu file). Well, I tried to have a look at ptx file, (no luck there).
I am using nvcc 4.0 along with gcc 4.2.1 on a MacOS 10.6.8 system. But I get the same warning on Linux CentOS.
The code is doing fine, (amazing speedup, more than 86 times faster). It does what it should do and the output is correct. My concern is that I have to compile it with sm_11 even if I am running it on Tesla M1060. If I compile it with sm_13 (of course I don't get the warning) but the code runs slower (obviously something is using double there).
Any clues how I can pinpoint where the double operation is performed, considering I have changed everything to float). I appreciate your help.

NSIGHT doesn't let me choose threads with id greater than 15.

lucana — Thu, 19 Jan 2012 10:41:02 -0500

I have managed to stop CUDA debugging at breakpoints. I'm working with VS2010. I can use the Debug Focus to select threads and blocks to follow. But I can't select any of the threads/blocks defined. The dimensions of grid and block written there are wrong. For example, I launched 1024 (kernel<<<1, 1024>>>)threads, but it only lets me choose up to thread number 15. Is it normal? I'm I doing something wrong?

Different results between debug and release version when working in VS2010

lucana — Thu, 19 Jan 2012 10:32:58 -0500

I have a CUDA application which runs on a TESLA C2075. I'm using CUDA toolkit 4.0 with 285.86 drivers.

I'm experiencing strange problems. When I run my application in debug mode (no CUDA debugging, only host debugging) things work. When I try to run the release version, strange results arise. Has anyone experienced something similar? Does any of you know what could be the problem?

Anyway, has anyone had good results working with CUDA in the VS2010 environment? By good I mean accurate from the scientific point of view. Would you recommend to switch to a Linux environment with gcc compiler?

Thanks for your help.

Hardware H264 encoding/decoding

jankowalski8234 — Wed, 18 Jan 2012 05:39:53 -0500

Hello,

I am developer of an app which allows live video streaming. I plan to spend some amount of money on NVidia Tesla GPUs to do the video encoding but really I do not think it is a good choice:

Here are the details. I have source stream frames already decoded and I am encoding these again using open source H264 codec. It is really time consuming for CPU to do this work, and CPU time is relative expensive. This is why I want to move the hard work to hi-end GPU and I belive it is really possible with a little amount of work to replace the open source codec to 'H264 GPU api' of some sort.

I know there is Windows API for H264 hardware codec and a example. It is working fine however... I can not run secondary instance of this application. Is this a hardware limitation? - Also, I know I can run up to 4 NVidia CUDA kernels but is this problem related to these limitations ? The Windows platform is also not a great choice for me and there is no such video encidng API on the Linux platform. How much I must pay to obtain such API for Linux ?

How I should accomplish this? Real time H264 encoding on the GPU is the primary goal but the problem is - in my opionion - I can not have more video streams than physical GPU cards.

If there is no way NVidia could license this API for Linux then which commercial GPU H264 codec will you recommend for such a solution?

I hope someone will clear my mind. Help!

driver for GTX 295 + tesla?

haynor — Sat, 03 Dec 2011 14:54:30 -0500

hi gang,

i have a 64-bit Win7 system with GTX 295 graphics card and 2 Tesla (C1060) cards. if i uninstall the nvidia driver, i see all the cards. if i install the current 270.81 driver, though, the Teslas disappear from Device Manager and aren't accessible in CUDA. any advice? anyone with a similar system -- if so, which drivers do you use?

thanks in advance.

Bad performance with peer to peer memory transfer

Pyt — Tue, 22 Nov 2011 18:04:14 -0500

Hello everyone,

I am currently using a multi-GPU cluster equipped with 3 Tesla M2090 cards.
I wanted to monitor the speed at which data is transferred from one GPU to another.
I used the "simpleP2P" code from the SDK and modified it to obtain the transfer rate for all GPUs (see code attached - you'll need to rename it to .cu). The memory allocated is still 64 MB, though I tried with bigger arrays (128,1024,2048 and 4096 MB - the results are still the same).
The arrays are transferred a hundred times. I tried to transfer memory with cudaMemcpy() and cudaMemcpyPeer() but obviously both give the same results.

I get the following speeds, which are not that good for a GPU to GPU transfer rate:

cudaMemcpy between GPU0 and GPU1: 316.54MB/s
cudaMemcpy between GPU0 and GPU2: 316.55MB/s
cudaMemcpy between GPU1 and GPU0: 316.86MB/s
cudaMemcpy between GPU2 and GPU0: 316.93MB/s
cudaMemcpy between GPU1 and GPU2: 3699.74MB/s
cudaMemcpy between GPU2 and GPU1: 3669.23MB/s

According to Dr. Micikevicius in the multi-GPU webinar, speeds of roughly 6.6 GB/s can be achieved with transfers through PCI-e, so the results here are quite puzzling.

Additional details:
I compiled my code with the flags -O3 and -m 64.
All the drivers are up to date (version 4.0 for CUDA).
As for the operating system, uname -a gives me the following:
2.6.32-131.17.1.el6.x86_64 #1 SMP Thu Sep 29 10:24:25 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux

The deviceQry program from the SDK gives the following output:

Found 3 CUDA Capable device(s)

Device 0: "Tesla M2090"
  CUDA Driver Version / Runtime Version          4.0 / 4.0
  CUDA Capability Major/Minor version number:    2.0
  Integrated GPU sharing Host Memory:            No
  Device has ECC support enabled:                Yes
  Device is using TCC driver mode:               No
  Device supports Unified Addressing (UVA):      Yes
  Device PCI Bus ID / PCI location ID:           8 / 0

Device 1: "Tesla M2090"
  CUDA Driver Version / Runtime Version          4.0 / 4.0
  CUDA Capability Major/Minor version number:    2.0
  Integrated GPU sharing Host Memory:            No
  Device has ECC support enabled:                Yes
  Device is using TCC driver mode:               No
  Device supports Unified Addressing (UVA):      Yes
  Device PCI Bus ID / PCI location ID:           27 / 0

Device 2: "Tesla M2090"
  CUDA Driver Version / Runtime Version          4.0 / 4.0
  CUDA Capability Major/Minor version number:    2.0
  Integrated GPU sharing Host Memory:            No
  Device has ECC support enabled:                Yes
  Device is using TCC driver mode:               No
  Device supports Unified Addressing (UVA):      Yes
  Device PCI Bus ID / PCI location ID:           21 / 0

The output of the command lspci -tvv can be found attached. It seems that two cards are on the same IOH chip but not the last one, which would explain a transfer rate of 315 MB/s.
However, it does not explain a transfer rate of 3.7 GB/s between the two other ones. Maybe the quality of the PCI-e connection is not really good ?

What would be a possible explanation of such a low speed ?

Thank you for your answers.

OpenMPI + CUDA usage

ph0enix — Wed, 23 Nov 2011 09:00:59 -0500

Hello!

I want to try GPUDirect with openmpi with cuda extension. I'm a little confused. In the FAQ it says that cuInit() and cuCtxCreate() should be called prior to MPI_Init().

But how can I do this if I need to bind a GPU in each process after I call MPI_Init()? And before this call other processes aren't available.

NVIDIA Parallel Nsight 2.1 Release Candidate 1 now available for download!

Sebastien Domine — Thu, 10 Nov 2011 20:50:50 -0500

Dear Parallel Nsight Users,

NVIDIA is proud to announce the new release of NVIDIA Parallel Nsight™ 2.1 Release Candidate 1. This new release brings support for the new CUDA Toolkit 4.1 Release Candidate 1, which can be downloaded under the CUDA Registered Developer Program (www.developer.nvidia.com/join). Parallel Nsight 2.1 also adds new features for an enhanced CUDA debugging and profiling experience, and new features for DirectX graphics developers such as the ability to edit shaders, while the application is running, and measure drawcall timings for more advanced profiling analysis.

Parallel Nsight 2.1 RC1 can be downloaded from http://parallelnsight.nvidia.com/content/parallel-nsight-early-access, and requires Driver Release 285.67, available on the same download site.

For a complete list of the new exciting 2.1 features, go to http://parallelnsight.nvidia.com/content/parallel-nsight-early-access.

We recommend developers to give this new release an early trial and send us feedback via the developer tools forums or report issues by emailing ParallelNsight-Support@nvidia.com.

The NVIDIA Developer Tools Team

How do I see/control which programs use which GPUs?

Gary Chandler — Thu, 20 Oct 2011 13:21:38 -0400

Hello,

I have a system with 2 tesla 2050 GPUs and Matrox G200eW onboard graphics. Obviously when I run codes in CUDA I can use cudaSetDevice() to specify which GPU to use for computations, but how do I know what GPU is being used for normal graphics? Will running computations on the tesla GPUs affect the running of other programs that put graphics on the screen? I also use visualization software to render images from my data, will running computations on the tesla GPUs affect rendering? Does rendering automatically use one of the tesla GPUs?

Is it possible to see and control which programs use which GPUs?

I access the system with the tesla cards remotely from my desktop with ssh -X. does this change the graphics behaviour in any way?

Thanks

Gary

Question about race condition

salahsaleh — Thu, 20 Oct 2011 21:04:08 -0400

Hello!,
In the situations were different threads are accessing the same location of shared memory(just reading), how much overhead will be notice if compared to the overhead of read/write situation?

GPU doesn't work in exclusive compute mode

Gary Chandler — Thu, 20 Oct 2011 13:08:40 -0400

Hello,

I have a system with 2 tesla 2050 GPUs. They both work fine in Shared compute mode (mode 0) but after I switched them both to exclusive mode (mode 1) only GPU 1 works. The call to cudaSetDevice(0) works, but when i make a call to a CUFFT routine I get:
CUFFT error in file 'cuda.cu' in line 17

Does anyone know why my GPU 0 stops working in compute mode 1? Does anyone know how to fix this?

Thanks

Gary

CUDA 4.0 with AS 5.6

jackson312 — Thu, 13 Oct 2011 21:12:34 -0400

I just installed CUDA 4.0 on a Dell R5500 with a C2050 card in it. I can see the card using lspci, but I am not able to query the card using the deviceQuery which is built in the SDK.

I have used CUDA 4.0 successfully on a Dell Optiplex 740 with a 9800GT card on AS 5.5. Are there any issues with CUDA 4.0 and AS 5.6?

I did not have any problems getting the development driver to build or load. I got the SDK to compile fine.

Thanks,

Jackson

Problems getting CUDA samples running on Red Hat 5.5 (64-bit)

brucep — Wed, 12 Oct 2011 13:48:05 -0400

We have a research cluster and just added a new compute node into it that has a GPU so we can start providing GPU capabilities to our users. The server is an IBM iDataPlex running Red Hat linux 5.5 (64-bit). When I run lspci it shows:

[root@node46 ~]# lspci | grep -i nvidia
19:00.0 3D controller: nVidia Corporation GF100 [Tesla S2050] (rev a3)
19:00.1 Audio device: nVidia Corporation GF100 High Definition Audio Controller (rev a1)
1a:00.0 3D controller: nVidia Corporation GF100 [Tesla S2050] (rev a3)
1a:00.1 Audio device: nVidia Corporation GF100 High Definition Audio Controller (rev a1)

So I downloaded and installed NVIDIA-Linux-x86_64-270.41.34.run, cudatoolkit_4.0.17_linux_64_rhel5.5.run, and gpucomputingsdk_4.0.17_linux.run. After installing each of those and adding the appropriate paths to my PATH & LD_LIBRARY_PATH I rebooted the server to ensure everything is kosher. lsmod shows that the nvidia driver is installed:

[root@node46 ~]# lsmod | grep -i nvidia
nvidia 10765936 0
i2c_core 57537 3 i2c_ec,i2c_i801,nvidia

At this point I went into the NVIDIA_GPU_Computing_SDK/C directory and did a "make x86_64=1" to build all the samples. I then went into /NVIDIA_GPU_Computing_SDK/C/bin/linux/release and tried to run matrixMul as the SDK documentation suggested. However when I try running it I get:

./matrixMul
[matrixMul] starting...
[ matrixMul ]
./matrixMul Starting (CUDA and CUBLAS tests)...

matrixMul.cu(83) : cudaSafeCall() Runtime API error 38: no CUDA-capable device is detected.

So what am I missing here?

Thanks,

-Bruce

GPU computing with National Instruments LabWindows

LarryM — Mon, 12 Sep 2011 16:58:56 -0400

I'm developing a numerical application using National Instruments LabWindows development environment. We believe it can be speeded up considerably by using a GPU. I've never used one before. Has anyone ever used CUDA or OpenCl with LabWindows?