Unable to debug CUDA samples in GTX1080

Hello,

I have some trouble debugging some CUDA code using Nsight.
so I’ve tried to debug one of the CUDA sample projects (sortingNetworks) using Nsight as well.
I reach the desired breakpoint, but when I continue execution I have an exception thrown:

CUDA module loaded: 03045ec0 oddEvenMergeSort.cu
CUDA grid launch failed: CUcontext: 40233744 CUmodule: 50617792 Function: Z18bitonicSortShared1PjS_S_S

and the program halts.

under normal visual studio debugger the code executes well.

I did not tinker at all with the project’s/files’ properties.

some background:
Nsight version: 5.2.0.16321
CUDA driver: 378.49
CUDA toolkit: 8.0.44
Visual studio 2013, under update 5

GPU: GTX-1080

Hi matanw,

This issue always caused by the driver, could you please try the driver 376.33, I’m aware that 378.49 doesn’t work well with cuda.

Best Regards
Harry

Hi Harry,

is this a known issue? I’ve looked it up and didn’t find anything about it…

is this a driver issue with gtx-1080, or the pascal architecture?
I have the same code that runs well on previous generation Titan-X?

thanks,
Matan

just installed driver 376.33, got the same exception…
tried to revert to each version from current, down to 375.95, but got the same exception every time…

tried to install CUDA Toolkit 8.0.61, got the same exception…

any thoughts?

TIA
Matan

Hi matanw,

I met the same issue before, reinstalling the driver solved it, so it’s hard to say why.

Could you tell me which os you use? Do you enable the memcheck in nsight?

Best Regards
Harry

I’m on Maxwell (980 Ti), and have the same issue. Also tried to revert driver versions and CUDA versions to no avail. The lack of “debugability” of this problem is very frustrating :(

Hi Harry,

My OS is windows 7.
I’ve tried debugging both with enabled memcheck and disabled, to similar effect.

I have a separate GTX-1080, tried to install it as well on a another machine with similar environment,
got the same result.

moreover, the second GTX-1080 I’ve just specified was installed onto a machine that
ran and debugged the sample with a previous generation TITAN-X successfully, and now fails to debug with the GTX-1080.

Is there a way to collect valuable error data (a log file, a dump file, whatever)
that will assist in detecting the error and fixing it in the upcoming drivers / cuda toolkit?

Matan

Hi all,

I’m experiencing the same issues as well.

Recalling debugging issues I had with Nsight 5.2 RC2,
https://devtalk.nvidia.com/default/topic/975742/nsight-5-2-rc2-debugging-issues/#5026457
I’ve reverted to that version and voila, problem (temporarily) solved!
By the way, I’m still using 375.95 that driver.

HTH,
Avishay

My bad.
Seems that if I use no breakpoints at all, running under the debugger eventually ends up with grid launch failed.

  • Avishay

But it also seems a problem, there should be no grid launch failure, please update the state if you meet it again.

Hello Harry,

I get grid launch failure repeatedly, each time I try to debug using Nsight.

I ran my code, and some of the CUDA sample code, mostly sortingNetworks.

  1. I am able to hit a breakpoint if I set it early enough, but when I continue to the next kernel I get grid launch failure exception

  2. If I try to debug using Nsight without any breakpoints, I also get the grid launch failure (at the same point in the same kernel as thrown in (1)) and the program halts.

  3. all samples and my code work well if not trying to debug the kernels using Nsight.

this was tested on two separate PCs with two different GTX-1080s, running windows 7 with VS2013-update 5.
different drivers were tested from 375.95 up to the recent one.

all of the above works and debugs well on TITAN-X (Maxwell).

Matan

I’ll try to repo it today.

Hi matanw,

Bad news, I cannot repo it on both win7 and win10 with the same configuration, I’ve also tried the 1070 and 1060 and they both work well, maybe I should involve the dev to see what they can find out.

BTY I just see the latest driver is 378.66, I think you should have a try, maybe Custom(Advanced) -> “perform clean installation” can beat this issue.

If clean installation and 378.66 still cannot solve it, I will raise a bug to track it.

Hi Harry,

Thanks for addressing the issue.

I have tried a clean installation for 375.95, 376.33, 378.49 in the past.
Now I’ve tried a clean installation for 378.66 (which is from yesterday) and got the same error.

Is there any additional information you require, regarding the GPU, PC or environment settings?

TIA,
Matan

Thank you for trying this, yeah, I need your system information including GPU info, windows and CUDA verions.

BTW, did you try to debug the x86 or x64 app?

Hi Harry,

tried to debug the sortingNetworks example as-is, meaning x64.

system:
cpu: i7-6700K
ram: 16 gb
hd: 256 gb ssd + 1 tb hdd

windows:
windows 7 professional, service pack 1 build 7601 (6.1.7601)

gpu:
geforce gtx-1080, GP104 revision A1
BIOS version 86.04.17.00.11
subvendor: asus
device id: 10DE 1B80 - 1043 859B
driver version: 21.21.13.7866 (ForceWare 378.66) WHQL / Win7 64

versions:
cuda toolkit 8.0.61
nsight version 5.2.0.16321, standard
visual studio version - 12.0 (vs2013 with update 5)
geforce experience 3.3.0.95

please note that I have LogMeIn installed, and under display adapters, I have a LogMeIn mirror driver
in addition to the GTX-1080.
btw, the GTX-1080 is the only gpu installed on the PC, and the monitors are connected to it.

please let me know if you require any more information.

Matan

Thank you for your info, I’ve tried LogMeIn but still no luck, I’ll raise a bug for dev to check what more I can do.

Hi Harry,

is there any news about this bug?
do you know if someone successfully reproduced it?
do you require additional data from me?

TIA
Matan

Nope, the task has been assigned to the dev, could you please give me you email, I can add it into the bug, then you will be notified when it’s updated

Opps, I don’t have authorization to add your email into the bug system, so I will still track this topic.

Actually some other customers got the similar issue, fail to launch kernel while debugging, could you please have a check if the nsight memcheck is off, also disable the TDR in nsight monitor might help.