Still cannot properly watch variables in CUDA kernels

I have cut out a huge chunk of our project to try to reproduce this problem, but it doesn’t.
On a full project it reproduces every time.

Like I’ve written before - it started happening after introducing a small change that uses cub.h.
Code works, our unit tests pass, cuda_memcheck does not report any problems. But debugger is showing “Could not resolve name” for all variables.
After reverting the change - debugger starts to work again.

After many days of work I have managed to create a minimal (still big!) reproduction case for this problem.
It is available in this archive:

Please see README.md inside the archive for details.
Bug reported as: https://developer.nvidia.com/nvidia_bug/2677977

I can repro your issue on my local with your app attached ,thanks zbychs, we will look into this.

Hi cybernoid and zbychs ,

We tried both your project for repro . We can see the "Could not resolve name… "issue in Legacy debug. But we tried that they all should be working in Next-gen debug. And we fixed the demangled types of variables to make them more readable. Thus it can be a quick workaround for both of you.

for cybernoid ,
Win7 64bit, GTX1080Ti & GTX1080, VS2017 15.8.1 (And earlier), NSight 5.6, Driver 398.82
Suggestion is update to R400+ driver (if can be Latest , better), Nsight 2019.3 , win7+ pascal need to turn your pascal into TCC to run Next-gen debug. If you don’t have a spare dispaly for it , I am afraid you have to get a win10 RS4+ that supports both pascal wddm and Tcc for Next-gen debug.

for zbychs ,
Legacy debugger on GeForce 940MX, CUDA 10.1 Update 1, driver 425.25, VS 2017 and 2019.
I am afraid a maxwell can’t do nextgen debug .But I tried your mini projcet (thanks for that) , I can see the varibles you care in Watch (maxPValu/matchingYsForValu/shared_storage_union) display the values except the types are not readable which we will fix in next version. I also tried modify the source/ add another kernel/remane kernel …they can be read anyway .

see the shots what it will look like in next version. Hope this can help you ,thanks.

Hello yni. Thanks for the info.

However you have written that: “I can see the varibles […] display the values except the types are not readable”.
This is not the behavior I see on my side.

The values of the variables are not displayed. Instead of their values I get one of the messages:

  • Condition(false) in method: Void TypeCheckObjectName(Nvda.CppExpressions.FrontEnd.CppParseArguments)
  • Could not resolve name 'matchingYsForValue'

See this picture:


(or here: https://drive.google.com/open?id=1Xn5ue1K0TrtGLNLS3HjWDCgHcUIFadYB)

If the values were displayed it wouldn’t be a big problem. But since there are no values it makes debugging impossible.

If you don’t reproduce this problem on your side you can either try the exact version of tools I used (I have listed them in README.md), or you can give me versions of tools I should prepare reproduction for. I will try to do this.

Sorry zbychs, your app on my side even in Legacy can show the variables, I tried same config as you on 2019.2.
So I really don’t know how to help you then since you are with Maxwell + legacy can’t do next-gen debug.
My config is
2019.3(2019.2) + Latest Driver + pascal
maybe you can try our latest version 2019.3 + R435 .Good luck to you.

I have updated to newest versions of tools, and the reproduction case doesn’t reproduce any more.
But the bug still does reproduce on the original, non-minified project.
This is not surprising, considering that this bug is very sensitive to practically everything in the universe. It appears and disappears with tiniest changes.

Last time I’ve spent around 30-40 hours trying to get minimal reproduction case for the then-newest tools. That’s almost a week of work. I’m not sure I can afford to do that again.

HELLO,
I have similar anomalous Next Gen debugger behavior when trying to watch local and parameter variables. As I execute the code through breakpoints in VS 2019 sometimes some variables show expected values, sometimes not at all, or sometimes the wrong values, even when they are in scope. Sometimes breakpoints are hit twice when the next one should be hit? Even some if statement comparisons are failing even when I verify the values being tested, but I am unsure because of the debug misbehavior. Just wanted to see if anyone else has found other things to try…

Here are the GPU specs on the WIN 10 box…

 Name: NVIDIA GeForce RTX 2080 Ti

Device ID: PCI\VEN_10DE&DEV_1E04&SUBSYS_12AF196E&REV_A1\4&F1F8FEB&0&0018
Driver: C:\Windows\system32\DRIVERS\NVIDIA Corporation\Drs\dbInstaller.exe, 26.21.0014.4141 (English), 11/21/2019 11:42:34, 733144 bytes
Driver: C:\Windows\system32\DRIVERS\NVIDIA Corporation\Drs\nvdrsdb.bin, 11/20/2019 20:56:06, 1629108 bytes
Driver: C:\Windows\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_3334892267d8a3d1\NvContainerSetup.exe, 1.00.0010.0000 (English), 11/21/2019 11:43:00, 4701416 bytes
Driver: C:\Windows\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_3334892267d8a3d1\NvCplSetupInt.exe, 1.00.0010.0000 (English), 11/21/2019 11:43:00, 104153248 bytes
Driver: C:\Program Files\NVIDIA Corporation\license.txt, 11/20/2019 20:56:06, 27216 bytes
Driver: C:\Program Files\NVIDIA Corporation\NVSMI\MCU.exe, 1.01.5204.20580 (English), 11/21/2019 11:42:42, 859888 bytes
Driver: C:\Program Files\NVIDIA Corporation\NVSMI\nvdebugdump.exe, 6.14.0014.4141 (English), 11/21/2019 11:43:12, 451528 bytes
Driver: C:\Program Files\NVIDIA Corporation\NVSMI\nvidia-smi.1.pdf, 11/20/2019 20:56:06, 81581 bytes
Driver: C:\Program Files\NVIDIA Corporation\NVSMI\nvidia-smi.exe, 8.17.0014.4141 (English), 11/21/2019 11:43:26, 573584 bytes
Driver: C:\Program Files\NVIDIA Corporation\NVSMI\nvml.dll, 8.17.0014.4141 (English), 11/21/2019 11:43:38, 1000864 bytes
Driver: C:\Windows\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_3334892267d8a3d1\nvopencl32.dll, 26.21.0014.4141 (English), 11/21/2019 11:43:56, 25236552 bytes
Driver: C:\Windows\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_3334892267d8a3d1\nvopencl64.dll, 26.21.0014.4141 (English), 11/21/2019 11:44:00, 29925336 bytes
Driver: C:\Windows\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_3334892267d8a3d1\nvlddmkm.sys, 26.21.0014.4141 (English), 11/21/2019 11:43:34, 22744688 bytes
Driver: C:\Windows\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_3334892267d8a3d1\nv-vk64.json, 11/20/2019 20:56:06, 671 bytes
Driver: C:\Windows\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_3334892267d8a3d1\nvDecMFTMjpeg.dll, 26.21.0014.4141 (English), 11/21/2019 11:43:12, 611928 bytes
Driver: C:\Windows\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_3334892267d8a3d1\nvDecMFTMjpegx.dll, 26.21.0014.4141 (English), 11/21/2019 11:43:14, 755440 bytes
Driver: C:\Windows\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_3334892267d8a3d1\nvEncMFTH264.dll, 26.21.0014.4141 (English), 11/21/2019 11:43:18, 1141896 bytes
Driver: C:\Windows\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_3334892267d8a3d1\nvEncMFTH264x.dll, 26.21.0014.4141 (English), 11/21/2019 11:43:18, 1452472 bytes
Driver: C:\Windows\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_3334892267d8a3d1\nvEncMFThevc.dll, 26.21.0014.4141 (English), 11/21/2019 11:43:20, 1147848 bytes
Driver: C:\Windows\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_3334892267d8a3d1\nvEncMFThevcx.dll, 26.21.0014.4141 (English), 11/21/2019 11:43:20, 1463504 bytes
Driver: C:\Windows\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_3334892267d8a3d1\nvcbl64.dll, 26.21.0014.4141 (English), 11/21/2019 11:42:46, 526792 bytes
Driver: C:\Windows\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_3334892267d8a3d1\nvd3dumx.dll, 26.21.0014.4141 (English), 11/21/2019 15:42:12, 22298920 bytes
Driver: C:\Windows\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_3334892267d8a3d1\nvd3dumx_cfg.dll, 26.21.0014.4141 (English), 11/21/2019 15:42:16, 22499680 bytes
Driver: C:\Windows\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_3334892267d8a3d1\nvdlistx.dll, 26.21.0014.4141 (English), 11/21/2019 11:43:16, 199256 bytes
Driver: C:\Windows\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_3334892267d8a3d1\nvinitx.dll, 26.21.0014.4141 (English), 11/21/2019 11:43:32, 217808 bytes
Driver: C:\Windows\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_3334892267d8a3d1\nvldumdx.dll, 26.21.0014.4141 (English), 11/21/2019 15:42:26, 961984 bytes
Driver: C:\Windows\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_3334892267d8a3d1\nvoglshim64.dll, 26.21.0014.4141 (English), 11/21/2019 11:43:42, 193136 bytes
Driver: C:\Windows\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_3334892267d8a3d1\nvoglv64.dll, 26.21.0014.4141 (English), 11/21/2019 11:43:48, 40446608 bytes
Driver: C:\Windows\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_3334892267d8a3d1\nvoptix.dll, 7.01.0000.0000 (English), 11/21/2019 11:44:08, 126657680 bytes
Driver: C:\Windows\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_3334892267d8a3d1\nvrtum64.dll, 26.21.0014.4141 (English), 11/21/2019 11:44:32, 27527680 bytes
Driver: C:\Windows\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_3334892267d8a3d1\nvumdshimx.dll, 26.21.0014.4141 (English), 11/21/2019 11:44:40, 674208 bytes
Driver: C:\Windows\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_3334892267d8a3d1\nvvkwddc64.dll, 11/21/2019 11:44:44, 564984 bytes
Driver: C:\Windows\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_3334892267d8a3d1\nvwgf2umx.dll, 26.21.0014.4141 (English), 11/21/2019 15:42:34, 40510448 bytes
Driver: C:\Windows\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_3334892267d8a3d1\nvwgf2umx_cfg.dll, 26.21.0014.4141 (English), 11/21/2019 15:42:42, 41010768 bytes
Driver: C:\Windows\system32\NvFBC64.dll, 6.14.0014.4141 (English), 11/21/2019 11:43:24, 2074712 bytes
Driver: C:\Windows\system32\NvIFR64.dll, 6.14.0014.4141 (English), 11/21/2019 11:43:30, 1482376 bytes
Driver: C:\Windows\system32\NvIFROpenGL.dll, 26.21.0014.4141 (English), 11/21/2019 11:43:32, 685800 bytes
Driver: C:\Windows\system32\OpenCL.dll, 2.02.0002.0000 (English), 11/21/2019 11:45:16, 450976 bytes
Driver: C:\Windows\system32\nvEncodeAPI64.dll, 26.21.0014.4141 (English), 11/21/2019 11:43:20, 813656 bytes
Driver: C:\Windows\system32\nvapi64.dll, 26.21.0014.4141 (English), 11/21/2019 15:42:06, 4937584 bytes
Driver: C:\Windows\system32\nvcompiler.dll, 26.21.0014.4141 (English), 11/21/2019 11:42:52, 40511064 bytes
Driver: C:\Windows\system32\nvcuda.dll, 26.21.0014.4141 (English), 11/21/2019 11:43:06, 17458840 bytes
Driver: C:\Windows\system32\nvcuvid.dll, 7.17.0014.4141 (English), 11/21/2019 11:43:10, 5380736 bytes
Driver: C:\Windows\system32\nvfatbinaryLoader.dll, 26.21.0014.4141 (English), 11/21/2019 11:43:22, 1370256 bytes
Driver: C:\Windows\system32\nvinfo.pb, 11/20/2019 20:56:06, 56258 bytes
Driver: C:\Windows\system32\nvmcumd.dll, 26.21.0014.4141 (English), 11/21/2019 11:43:38, 825928 bytes
Driver: C:\Windows\system32\nvofapi64.dll, 11/21/2019 11:43:40, 676824 bytes
Driver: C:\Windows\system32\nvptxJitCompiler.dll, 26.21.0014.4141 (English), 11/21/2019 11:44:28, 11843184 bytes
Driver: C:\Windows\system32\vulkan-1-999-0-0-0.dll, 1.01.0114.0000 (English), 11/21/2019 11:45:20, 1073872 bytes
Driver: C:\Windows\system32\vulkan-1.dll, 1.01.0114.0000 (English), 11/21/2019 11:45:20, 1073872 bytes
Driver: C:\Windows\system32\vulkaninfo-1-999-0-0-0.exe, 1.01.0114.0000 (English), 11/21/2019 11:45:22, 848592 bytes
Driver: C:\Windows\system32\vulkaninfo.exe, 1.01.0114.0000 (English), 11/21/2019 11:45:22, 848592 bytes
Driver: C:\Windows\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_3334892267d8a3d1\DisplayDriverRAS.dll, 1.10.0000.0000 (English), 11/21/2019 11:42:34, 2582488 bytes
Driver: C:\Windows\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_3334892267d8a3d1\nv-vk32.json, 11/20/2019 20:56:06, 671 bytes
Driver: C:\Windows\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_3334892267d8a3d1\nv_Icc_AdvancedColor_Identity.icm, 11/20/2019 20:56:06, 3288 bytes
Driver: C:\Windows\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_3334892267d8a3d1\nvd3dum.dll, 26.21.0014.4141 (English), 11/21/2019 15:42:08, 18583560 bytes
Driver: C:\Windows\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_3334892267d8a3d1\nvd3dum_cfg.dll, 26.21.0014.4141 (English), 11/21/2019 15:42:22, 18942008 bytes
Driver: C:\Windows\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_3334892267d8a3d1\nvdlist.dll, 26.21.0014.4141 (English), 11/21/2019 11:43:16, 173984 bytes
Driver: C:\Windows\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_3334892267d8a3d1\nvinit.dll, 26.21.0014.4141 (English), 11/21/2019 11:43:32, 189160 bytes
Driver: C:\Windows\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_3334892267d8a3d1\nvldumd.dll, 26.21.0014.4141 (English), 11/21/2019 15:42:24, 803160 bytes
Driver: C:\Windows\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_3334892267d8a3d1\nvoglshim32.dll, 26.21.0014.4141 (English), 11/21/2019 11:43:40, 166112 bytes
Driver: C:\Windows\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_3334892267d8a3d1\nvoglv32.dll, 26.21.0014.4141 (English), 11/21/2019 11:43:42, 29901952 bytes
Driver: C:\Windows\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_3334892267d8a3d1\nvumdshim.dll, 26.21.0014.4141 (English), 11/21/2019 11:44:40, 547760 bytes
Driver: C:\Windows\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_3334892267d8a3d1\nvvkwddc32.dll, 11/21/2019 11:44:44, 431360 bytes
Driver: C:\Windows\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_3334892267d8a3d1\nvwgf2um.dll, 26.21.0014.4141 (English), 11/21/2019 15:42:26, 34870360 bytes
Driver: C:\Windows\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_3334892267d8a3d1\nvwgf2um_cfg.dll, 26.21.0014.4141 (English), 11/21/2019 15:42:52, 35626256 bytes
Driver: C:\Windows\SysWow64\NvFBC.dll, 6.14.0014.4141 (English), 11/21/2019 11:43:22, 1567688 bytes
Driver: C:\Windows\SysWow64\NvIFR.dll, 6.14.0014.4141 (English), 11/21/2019 11:43:30, 1144920 bytes
Driver: C:\Windows\SysWow64\NvIFROpenGL.dll, 26.21.0014.4141 (English), 11/21/2019 11:43:30, 556672 bytes
Driver: C:\Windows\SysWow64\OpenCL.dll, 2.02.0002.0000 (English), 11/21/2019 11:45:14, 352512 bytes
Driver: C:\Windows\SysWow64\nvEncodeAPI.dll, 26.21.0014.4141 (English), 11/21/2019 11:43:20, 659088 bytes
Driver: C:\Windows\SysWow64\nvapi.dll, 26.21.0014.4141 (English), 11/21/2019 15:42:04, 4205568 bytes
Driver: C:\Windows\SysWow64\nvcompiler.dll, 26.21.0014.4141 (English), 11/21/2019 11:42:46, 35380336 bytes
Driver: C:\Windows\SysWow64\nvcuda.dll, 26.21.0014.4141 (English), 11/21/2019 11:43:02, 15027776 bytes
Driver: C:\Windows\SysWow64\nvcuvid.dll, 7.17.0014.4141 (English), 11/21/2019 11:43:08, 4716176 bytes
Driver: C:\Windows\SysWow64\nvfatbinaryLoader.dll, 26.21.0014.4141 (English), 11/21/2019 11:43:22, 1064408 bytes
Driver: C:\Windows\SysWow64\nvofapi.dll, 11/21/2019 11:43:40, 544728 bytes
Driver: C:\Windows\SysWow64\nvptxJitCompiler.dll, 26.21.0014.4141 (English), 11/21/2019 11:44:26, 10167432 bytes
Driver: C:\Windows\SysWow64\vulkan-1-999-0-0-0.dll, 1.01.0114.0000 (English), 11/21/2019 11:45:22, 931536 bytes
Driver: C:\Windows\SysWow64\vulkan-1.dll, 1.01.0114.0000 (English), 11/21/2019 11:45:22, 931536 bytes
Driver: C:\Windows\SysWow64\vulkaninfo-1-999-0-0-0.exe, 1.01.0114.0000 (English), 11/21/2019 11:45:24, 706256 bytes
Driver: C:\Windows\SysWow64\vulkaninfo.exe, 1.01.0114.0000 (English), 11/21/2019 11:45:24, 706256 bytes
Driver: C:\Windows\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_3334892267d8a3d1\NvTelemetry64.dll, 14.03.0015.0000 (English), 11/21/2019 11:44:38, 4417928 bytes
Driver: C:\Windows\system32\nvdispco6444141.dll, 2.00.0057.0000 (English), 11/21/2019 11:43:14, 1733264 bytes
Driver: C:\Windows\system32\nvdispgenco6444141.dll, 2.00.0028.0002 (English), 11/21/2019 11:43:16, 1491568 bytes

- prop {name=0x000000c1498f5660 “GeForce RTX 2080 Ti” uuid={bytes=0x000000c1498f5760 "´›Y—Òú\x5Nb7Tò \f³\x13… } …} cudaDeviceProp
+ name 0x000000c1498f5660 “GeForce RTX 2080 Ti” char[256]
+ uuid {bytes=0x000000c1498f5760 "´›Y—Òú\x5Nb7Tò \f³\x13… } CUuuid_st
+ luid 0x000000c1498f5770 “Î^\x1” char[8]
luidDeviceNodeMask 1 unsigned int
totalGlobalMem 11811160064 unsigned __int64
sharedMemPerBlock 49152 unsigned __int64
regsPerBlock 65536 int
warpSize 32 int
memPitch 2147483647 unsigned __int64
maxThreadsPerBlock 1024 int
+ maxThreadsDim 0x000000c1498f57a4 {1024, 1024, 64} int[3]
+ maxGridSize 0x000000c1498f57b0 {2147483647, 65535, 65535} int[3]
clockRate 1545000 int
totalConstMem 65536 unsigned __int64
major 7 int
minor 5 int
textureAlignment 512 unsigned __int64
texturePitchAlignment 32 unsigned __int64
deviceOverlap 1 int
multiProcessorCount 68 int
kernelExecTimeoutEnabled 1 int
integrated 0 int
canMapHostMemory 1 int
computeMode 0 int
maxTexture1D 131072 int
maxTexture1DMipmap 32768 int
maxTexture1DLinear 268435456 int
+ maxTexture2D 0x000000c1498f5804 {131072, 65536} int[2]
+ maxTexture2DMipmap 0x000000c1498f580c {32768, 32768} int[2]
+ maxTexture2DLinear 0x000000c1498f5814 {131072, 65000, 2097120} int[3]
+ maxTexture2DGather 0x000000c1498f5820 {32768, 32768} int[2]
+ maxTexture3D 0x000000c1498f5828 {16384, 16384, 16384} int[3]
+ maxTexture3DAlt 0x000000c1498f5834 {8192, 8192, 32768} int[3]
maxTextureCubemap 32768 int
+ maxTexture1DLayered 0x000000c1498f5844 {32768, 2048} int[2]
+ maxTexture2DLayered 0x000000c1498f584c {32768, 32768, 2048} int[3]
+ maxTextureCubemapLayered 0x000000c1498f5858 {32768, 2046} int[2]
maxSurface1D 32768 int
+ maxSurface2D 0x000000c1498f5864 {131072, 65536} int[2]
+ maxSurface3D 0x000000c1498f586c {16384, 16384, 16384} int[3]
+ maxSurface1DLayered 0x000000c1498f5878 {32768, 2048} int[2]
+ maxSurface2DLayered 0x000000c1498f5880 {32768, 32768, 2048} int[3]
maxSurfaceCubemap 32768 int
+ maxSurfaceCubemapLayered 0x000000c1498f5890 {32768, 2046} int[2]
surfaceAlignment 512 unsigned __int64
concurrentKernels 1 int
ECCEnabled 0 int
pciBusID 2 int
pciDeviceID 0 int
pciDomainID 0 int
tccDriver 0 int
asyncEngineCount 3 int
unifiedAddressing 1 int
memoryClockRate 7000000 int
memoryBusWidth 352 int
l2CacheSize 5767168 int
maxThreadsPerMultiProcessor 1024 int
streamPrioritiesSupported 1 int
globalL1CacheSupported 1 int
localL1CacheSupported 1 int
sharedMemPerMultiprocessor 65536 unsigned __int64
regsPerMultiprocessor 65536 int
managedMemory 1 int
isMultiGpuBoard 0 int
multiGpuBoardGroupID 0 int
hostNativeAtomicSupported 0 int
singleToDoublePrecisionPerfRatio 32 int
pageableMemoryAccess 0 int
concurrentManagedAccess 0 int
computePreemptionSupported 1 int
canUseHostPointerForRegisteredMem 0 int
cooperativeLaunch 0 int
cooperativeMultiDeviceLaunch 0 int
sharedMemPerBlockOptin 65536 unsigned __int64
pageableMemoryAccessUsesHostPageTables 0 int
directManagedMemAccessFromHost 0 int