cuda-memcheck causes a failure when calling curandGenerateUniform in the code

Developer Tools CUDA Developer Tools CUDA-MEMCHECK
1

Calling the function curandGenerateUniform from the cuRand library fails with the error CURAND_STATUS_LAUNCH_FAILURE only when running the compiled code from the command line with cuda-memcheck

The problem can be generated with a standard example from https://docs.nvidia.com/cuda/curand/host-api-overview.html#host-api-example. (see report below).

I could generate the problem on 2 PCs:

  • windows 7 with CUDA SDK v9.0
  • windows 10 with CUDA SDK v9.1

Any ideas?
Thanks!

========= CUDA-MEMCHECK
Error at c:/Users/alkamu1/documents/visual studio 2017/Projects/curandTest/curandTest/kernel.cu:39
========= Program hit cudaErrorLaunchFailure (error 4) due to "unspecified launch failure" on CUDA API call to cudaFree.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:C:\Windows\system32\nvcuda.dll (cuTexRefSetAddress + 0x2cae4e) [0x2d849b]
=========     Host Frame:C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.1\bin\curand64_91.dll (curandGeneratePoissonMethod + 0x83f2) [0x3ee92]
=========     Host Frame:C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.1\bin\curand64_91.dll (curandGenerateSeeds + 0x28f) [0x2172f]
=========     Host Frame:C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.1\bin\curand64_91.dll (curandGenerateSeeds + 0x3b01) [0x24fa1]
=========     Host Frame:C:\Users\alkamu1\Documents\Visual Studio 2017\Projects\curandTest\x64\Debug\curandTest.exe (main + 0x111) [0x1831]
=========     Host Frame:C:\Users\alkamu1\Documents\Visual Studio 2017\Projects\curandTest\x64\Debug\curandTest.exe (invoke_main + 0x34) [0x3f24]
=========     Host Frame:C:\Users\alkamu1\Documents\Visual Studio 2017\Projects\curandTest\x64\Debug\curandTest.exe (__scrt_common_main_seh + 0x127) [0x3de7]
=========     Host Frame:C:\Users\alkamu1\Documents\Visual Studio 2017\Projects\curandTest\x64\Debug\curandTest.exe (__scrt_common_main + 0xe) [0x3cae]
=========     Host Frame:C:\Users\alkamu1\Documents\Visual Studio 2017\Projects\curandTest\x64\Debug\curandTest.exe (mainCRTStartup + 0x9) [0x3f49]
=========     Host Frame:C:\Windows\System32\KERNEL32.DLL (BaseThreadInitThunk + 0x14) [0x12784]
=========     Host Frame:C:\Windows\SYSTEM32\ntdll.dll (RtlUserThreadStart + 0x21) [0x70d51]
=========
========= Program hit cudaErrorLaunchFailure (error 4) due to "unspecified launch failure" on CUDA API call to cudaThreadSynchronize.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:C:\Windows\system32\nvcuda.dll (cuTexRefSetAddress + 0x2cae4e) [0x2d849b]
=========     Host Frame:C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.1\bin\curand64_91.dll (curandGeneratePoissonMethod + 0x7c3a) [0x3e6da]
=========     Host Frame:C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.1\bin\curand64_91.dll (curandGenerateSeeds + 0x4b8) [0x21958]
=========     Host Frame:C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.1\bin\curand64_91.dll (curandGenerateSeeds + 0x3b01) [0x24fa1]
=========     Host Frame:C:\Users\alkamu1\Documents\Visual Studio 2017\Projects\curandTest\x64\Debug\curandTest.exe (main + 0x111) [0x1831]
=========     Host Frame:C:\Users\alkamu1\Documents\Visual Studio 2017\Projects\curandTest\x64\Debug\curandTest.exe (invoke_main + 0x34) [0x3f24]
=========     Host Frame:C:\Users\alkamu1\Documents\Visual Studio 2017\Projects\curandTest\x64\Debug\curandTest.exe (__scrt_common_main_seh + 0x127) [0x3de7]
=========     Host Frame:C:\Users\alkamu1\Documents\Visual Studio 2017\Projects\curandTest\x64\Debug\curandTest.exe (__scrt_common_main + 0xe) [0x3cae]
=========     Host Frame:C:\Users\alkamu1\Documents\Visual Studio 2017\Projects\curandTest\x64\Debug\curandTest.exe (mainCRTStartup + 0x9) [0x3f49]
=========     Host Frame:C:\Windows\System32\KERNEL32.DLL (BaseThreadInitThunk + 0x14) [0x12784]
=========     Host Frame:C:\Windows\SYSTEM32\ntdll.dll (RtlUserThreadStart + 0x21) [0x70d51]
=========
========= Program hit cudaErrorLaunchFailure (error 4) due to "unspecified launch failure" on CUDA API call to cudaGetLastError.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:C:\Windows\system32\nvcuda.dll (cuTexRefSetAddress + 0x2cae4e) [0x2d849b]
=========     Host Frame:C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.1\bin\curand64_91.dll (curandGeneratePoissonMethod + 0x7d67) [0x3e807]
=========     Host Frame:C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.1\bin\curand64_91.dll (curandGenerateSeeds + 0x4bd) [0x2195d]
=========     Host Frame:C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.1\bin\curand64_91.dll (curandGenerateSeeds + 0x3b01) [0x24fa1]
=========     Host Frame:C:\Users\alkamu1\Documents\Visual Studio 2017\Projects\curandTest\x64\Debug\curandTest.exe (main + 0x111) [0x1831]
=========     Host Frame:C:\Users\alkamu1\Documents\Visual Studio 2017\Projects\curandTest\x64\Debug\curandTest.exe (invoke_main + 0x34) [0x3f24]
=========     Host Frame:C:\Users\alkamu1\Documents\Visual Studio 2017\Projects\curandTest\x64\Debug\curandTest.exe (__scrt_common_main_seh + 0x127) [0x3de7]
=========     Host Frame:C:\Users\alkamu1\Documents\Visual Studio 2017\Projects\curandTest\x64\Debug\curandTest.exe (__scrt_common_main + 0xe) [0x3cae]
=========     Host Frame:C:\Users\alkamu1\Documents\Visual Studio 2017\Projects\curandTest\x64\Debug\curandTest.exe (mainCRTStartup + 0x9) [0x3f49]
=========     Host Frame:C:\Windows\System32\KERNEL32.DLL (BaseThreadInitThunk + 0x14) [0x12784]
=========     Host Frame:C:\Windows\SYSTEM32\ntdll.dll (RtlUserThreadStart + 0x21) [0x70d51]
=========
========= ERROR SUMMARY: 3 errors
2

Hi User muhammedsoubhi.alk,

First sorry for the inconvenience that has taken to you. Could you please provide the following information for us to reproduce and locate the issue

  1. The full CUDA SDK version you used. eg:v9.0.178 or something else
  2. The Driver version that you used
  3. Can you run the sample curandTest.exe successfully?

Thanks & Best Wishes
Amy Liu

3

One more question, have you enabled the TDR? You can check this by launch “Nsight monitor”, right click Nisght Monitor->Options->General->WDDM TDR Enabled section. If “Yes”, please set as “False” and then restart your computer, then check whether this issue still happen.

Thanks.
Amy Liu

4

The problem is solved by disabling TDR!

Thanks Amy!