Troubleshooting errors when using the driver API

[jack]

I’ve begun a new CUDA project recently, and I’ve run into an error that’s got me stumped. I’m building the project on top of the driver API because I don’t want to have to include the runtime API DLL in my releases (for various reasons). Unfortunately, the driver API doesn’t give much information in the way of errors (other than the error code and a relative idea of where the error is occurring).

Are there any methods for troubleshooting errors that occur when working with the driver API? It would be helpful if the driver could write information out to a log file (like when kernels were launched, a printout of the parameters that were passed (in raw hex format), why an error occurred, etc.).

In any case, the error I’m getting is a CUDA_ERROR_NOT_READY value when I make a call to cuEventElapsedTime(); prior to that, I’m simply uploading some data, executing a small kernel (which I know works fine, since I compiled it with the <<< >>> syntax in a previous project), and reading the data back. I recorded an event at the start and finish of all that, and none of it returns an error until the cuEventElapsedTime() call (and I’m not using any async functions).

[Nico]

Did you synchronize with the stop event before calling cuEventElapsedTime?

CUresult cuEventElapsedTime (float ∗ pMilliseconds, CUevent hStart, CUevent hEnd)

Computes the elapsed time between two events (in milliseconds with a resolution of around 0.5 microseconds). If

either event has not been recorded yet, this function returns CUDA_ERROR_NOT_READY. If either event has been

recorded with a non-zero stream, the result is undefined.

N.

[jack]

Did you synchronize with the stop event before calling cuEventElapsedTime?

N.

Yep. I record the end event, synchronize with cuEventSynchronize(), then call cuEventElapsedTime() which throws the error.

If it helps, this is the general flow of the program right now (after getting the device handle):

cuCtxCreate

cuMemAlloc

cuMemcpyHtoD

cuModuleLoadData

cuModuleGetFunction

cuParamSetv

cuParamSetSize

cuEventCreate (the start event)

cuEventCreate (the end event)

cuEventRecord (the start event)

cuFuncSetBlockShape

cuLaunch

cuEventRecord (the end event)

cuEventSynchronize (on the end event)

cuEventElapsedTime <-- Error is returned here

(then copy back from the device and cleaning up the context and such)

[Nico]

Could you check if cuEventQuery(the end event) returns CUDA_SUCCESS, after you call cuEventSynchronize (the end event)?

PS. Maybe it’s just me, but the cuEventSynchronize documentation does not make sense to me:

CUresult cuEventSynchronize (CUevent hEvent)

Waits until the event has actually been recorded. If cuEventRecord() has been called on this event, the function returns

CUDA_ERROR_INVALID_VALUE.

If cuEventRecord() has previously been called and the event has not been recorded yet, this function returns CUDA_-

ERROR_INVALID_VALUE.

N.

[jack]

Could you check if cuEventQuery(the end event) returns CUDA_SUCCESS, after you call cuEventSynchronize (the end event)?

PS. Maybe it’s just me, but the cuEventSynchronize documentation does not make sense to me:

N.

I put a call to cuEventQuery after cuEventSynchronize, and now I’m actually getting an Unknown error from the cuEventSynchronize (it’s not even getting to cuEventQuery now).

And it’s not just you…I thought that documentation was quite confusing as well. There are also some other oddities I noticed in the API itself, like the fact that cuFuncSetBlockShape takes parameters of type int, when the block size given by blockDim is a dim3 value (which contains uint types).

[Nico]

What happens if you pass the CU_EVENT_BLOCKING_SYNC flag to cuEventCreate?

N.

[jack]

What happens if you pass the CU_EVENT_BLOCKING_SYNC flag to cuEventCreate?

N.

Wow, good call Nico. I added the flag to both cuEventCreate calls and now it works! Thanks for your help :)

[Nico]

Wow, good call Nico. I added the flag to both cuEventCreate calls and now it works! Thanks for your help :)

Glad to hear it’s working now :)

N.