Help! First cudaMalloc takes 10 seconds!

Accelerated Computing CUDA CUDA Programming and Performance
1

Hi,

I have heard some discussions about cuda function startup cost, but I just found a strange 10 seconds of first cudaMalloc call.

Even in a simplest CUDA program, the first call of “cudaMalloc” takes almost 10 seconds. The server is Linux-x86-64 Intel Xeon system with 4 Tesla C2070 GPUs. The CUDA is release 4.0, V0.2.1221. From some posts here, I have tried to add -code=sm_13 or sm_20 in nvcc command lines, but none of them could reduce this overhead.

I have tried the same code in the second Linux x86_64 server, with 4 Tesla C2050 GPUs. It takes 3-4 seconds for the first cudaMalloc call.
But in the third Linux i386 server, which is a little bit old, with one Quadro 600 GPU. It only takes around 0.06 seconds for the first cudaMalloc call.

All the above testing are carried out more than 5 times. Any suggestions?

Thanks.

Lei

2

It would be interesting to repeat these experiments with CUDA Driver API. Please also post code you are testing for us to suggest something more.

3

Thanks for reply.

The code is the simplest as follows:

#include <stdio.h>

#include <ctime>

int main(void) {

        double * a;

        clock_t t11 = clock();

        cudaMalloc((void**)&a, 2*sizeof(double));

        clock_t t12 = clock();

        printf(" cudaMalloc time: %f s\n", ((double)t12 - (double)t11)/CLOCKS_PER_SEC );

        return 0;

}

The results shown on three platforms:

Platform1(Linux-x86_64,Tesla C2070): 9.47 s

Platform2(Linux-x86_64,Tesla C2050): 1.16 s

Platform3(Linux-i386, Quadro 600): 0.07 s

4

Try upgrading/downgrading the driver and see if that changes the startup time.

5

FYI: In CUDA code events are usually used for any kind of performance testing (CUDA C Programmin Guide, p. 35, chapter 3.2.5.6).

6

10 seconds sounds insane…It should not be that bad…

7

That is because the driver is not loaded and that time is required by the OS to load it.

To test it out write a programm doing a cudaMalloc and then spleep for hours, while that is sleeping (it keeps

the driver loaded) try out now how much a cudaMalloc takes.

8

Probably some other processes were running in system when you measured the time. For me the first run of your cudaMalloc code on Linux86x64+TeslaC2070 gives 0.05sec

9

Hi, I have similar problem. I want to some real time image processing about 30 frame/second.

But only copy 0.5 MBytes between CPU and GPU take me 0.06s. Is not that too slow. I am using Quadro 420.