hi,

i guess the problem here is, that (see the source code) the kernel overlaps with just one copy process (the one which brings new data to the device). the second copy process which moves the computed data from the device to the host runs on the same stream like the kernel, so that both the kernel and the (async-)memcpy execute serial. the latter is necessary since data is valid only if the kernel has finished execution. so you cannot compare the 2.7ms with the 4.3ms, as the former one considers data-transfers in both directions (as supported by your device) concurrently to kernel execution. but why is it not (2.7+0.69)ms=3.39ms? the kernel and the memcpy processes run inside a loop which for each iteration forces a stream synchronisation, and this is expensive. at the end we get 4.3ms.

the output of the sample is somewhat missleading, as it suggest that 4 streams are used in order to get high performance.

the 2.7ms are computed as the max( memcpy-host-to-device, memcpy-device-to-host, kernel-execution ). if you would incorporate some kind of double buffering then it might be possible to achieve the 2.7ms.

hi again,

at first, you're right: almost all kernel calls overlap with at least two memcompies (i read over the last line in the loop where streams are switched round robin). that is, the problem is not the concurrent kernel-memcpy scheme. but the point is, that between lines 157 and 182 in the code, runtimes for kernel execution and memcopies are measured with no concurrency. the result is what you see in the program output under 'measured timings (throughput)'.

if within the loop kernels overlap with memcopies, the situation is somewhat different, as now both memcpy-host-to-device and memcopy-device-to-host share the same pcie-bus which allows for at most 8gb/s in both directions (pcie-2). i measured on two different up-to-date systems (with tesla m2090 cards installed) that even in the case that one kernel and one memcpy-host-to-device and one memcpy-device-to-host run on 3 different streams, the pcie-bus does not give 6gb/s in both directions, but something which is significantly below this value. if i run the kernel concurrently to just one memcpy, then i get the 6gb/s. so the values given by the sdk-sample assume that your system is capabale of writing data to the device with 6gb/s and at the same time to read data from device also with 6gb/s. but this does not seem to be the case. so what is given are theoretical values. what is actually measured differs due to limited pcie-bandwidth (or limited memory bandwidth of your host system). if you draw the gpu-queue/pipeline filled with kernels and memcopies, you will also see that there are 4 memcopies which at the same time try to move data over the same pcie-bus. it should be clear than none of them gets 6gb/s up-/downstream. since after the 3rd iteration stream-synchronizations become necessary, overall timings slow down.

to summarize: the time for concurrent memcpy-host-to-device and memcpy-device-to-host is not max( t1, t2 ) if t1 and t2 are the respective times given in the 'measured timings (throughput)' section (here the 2.72..ms). but this is what the sdk-sample assumes as theoretical value.

i hope that now my post does not suffer from 'misunderstandings' :)

@paulvisschers: 8gb/s is the theoretical limit, but most likely you will never reach that limit. there is a couple of limiting factors which make your program run slower than theoretically expected. as you can see from the program output, your system is not even capable of copying data from device to host (or vice versa) with the theoretical 8gb/s, with no kernel and no other memcpy interfering. maybe its your gpu that limits the bandwidth from or to device, or it is your host's memory that limits. so why should your system then give you 8gb/s (or 6gb/s) in both directions when performing two simultaneous memcopies.

if i run the simpleMultiCopy program on a tesla m2090 cluster node, the program output is almost the same as yours, and also on gtx590/580 workstations bandwidths are below 7gb/s.

if you have a multi-socket system (maybe a nehalem dual xeon system), it might be possible that the calling host-thread runs on a cpu that is not directly connected to the gpu your calculations run onto. you can try to pin your host program to one of the sockets and then run your kernels on a gpu that is connected to the respective socket. for instance you can use libnuma in your code or you can run your program with 'numactl -N xxx -l ./program.x' (where xxx=1,2,3,...) or something like that.

runtime predictions: according to the program output, your system allows for up to 7.8gb/s in the asynchronous case, and up to 5.5gb/s for serial execution (for the scenario considered). so you have upper and lower bounds that can be used for runtime 'predictions'. if you are to use your gpu in multi-threaded applications with high cpu-load, then it would become hard to give a prediction as multiple threads (if you run more threads than your system has cores) may interfer eachother and barriers are not reached at expected times.