cudaStreamSynchronize error

Autonomous Machines Jetson & Embedded Systems Jetson TX2
1

Hey all,

I have a long-running process that uses the GPU for various things including using OpenCV’s HAAR GPU implementation… It works great most of the time but rarely, and unpredictably, I get the following GPU error:

OpenCV(3.4.1) Error: Gpu API call
NCV Assertion Failed:
cudaError_t=4,
file=.../NCVHaarObjectDetection.cu, line=1213 in NCVDebugOutputHandler,
file .../cascadeclassifier.cpp, line 156

It looks like this is triggered by a call to cudaStreamSynchronize in the file referenced above, but I can’t figure out why or what triggers this scenario.

Any ideas on how to debug this or what it could mean?

Thanks!

2

Hi,

CUDA error 4 is cudaErrorLaunchFailure:
-------------------------------------------------
An exception occurred on the device while executing a kernel. Common causes include dereferencing an invalid device pointer and accessing out of bounds shared memory. The device cannot be used until cudaThreadExit() is called. All existing device memory allocations are invalid and must be
reconstructed if the program is to continue using CUDA.
-------------------------------------------------

Could you check if there is any possible invalid memory in your code?
Thanks.

3

Hi billn9ek5,

Have you found the root cause?
Please update the status then can give further suggestions.

Thanks

4

Unfortunately I have not been able to isolate the cause, but I will keep watching. As I say, it happens somewhat rarely, so it is difficult to diagnose.

Of course, as you suggest, it could be invalid memory access or similar, but it is surprising to me that the code can run (aggressively) for (sometimes) a few days before it hits this error…

5

Hi,

Maybe you can check your code with cuda-memcheck to see if any invalid access.

Thanks.