cudaEventElapsedTime with multiple devices

Accelerated Computing CUDA CUDA Programming and Performance
1

cudaEventElapsedTime doesn’t work when the events are from different devices. This is probably not a fundamental limitation, because nSight can create a timeline of the whole application, with all of the devices it used. So is there a good way of comparing the time difference between two events on different devices programmatically (not in nSight)?

(I tried some hackish solutions like having one device wait for an event on another, and then registering its own event, so that I’d have two events occurring in close proximity, but the results seem poor)

2

Indeed, events are specific to devices:

[url]http://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#device-selection[/url]

"cudaEventElapsedTime() will fail if the two input events are associated to different devices. "

You could try inserting a callback to a routine containing a host-based time-stamping function (e.g. gettimeofday):

[url]http://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#stream-callbacks[/url]

at each point where you want timing measurements. Later, you can subtract the two timestamps.

Note that the callback is executed at the point when all previous cuda activity in that stream has executed, so it behaves similarly to event (completion). Separate devices (should) have separate streams:

[url]Programming Guide :: CUDA Toolkit Documentation

Note that callbacks should not make use of any CUDA API functionality.

3

Thanks, Txbob!

(BTW, when I wrote that one approach I tried seemed to produce poor results, I didn’t realize that there was a bug in my code stemming from misunderstanding the semantics of cudaStreamWaitEvent)