Unable to use cuda-memcheck --tool racecheck

Hello all,

I am trying to use cuda-memcheck --tool racecheck, but I get the following errors:

Internal Memcheck Error: Device not supported. Please refer to the "Supported Devices" section of the cuda-memcheck documentation

Internal Memcheck Error: Initialization failed

Checking the documentation, cuda-memcheck apparently works on all devices of Compute Capability 3.5 or higher. I am using a GTX 1050 on a laptop, which is of Compute Capability 6.1. I am using CUDA Toolkit 11.2 and my driver version is 461.40. I am using Nsight Visual Studio version 2020.3.0.20315.

Additionally, running nvprof --profile-all-processes gives the following error:

cupti64_2020.3.0.dll was not found. Reinstalling the program may fix this problem.

I reinstalled but get the same errors. For reference, here are the full error messages when trying to run cuda-memcheck --tool racecheck:

========= Internal Memcheck Error: Device not supported. Please refer to the “Supported Devices” section of the cuda-memcheck documentation
========= Saved host backtrace up to driver entry point at error
========= Host Frame:C:\WINDOWS\system32\DriverStore\FileRepository\nvami.inf_amd64_c4c88e9630d3f61b\nvcuda64.dll (cuProfilerStop + 0x97ba8) [0x2b8928]
========= Host Frame:C:\WINDOWS\system32\DriverStore\FileRepository\nvami.inf_amd64_c4c88e9630d3f61b\nvcuda64.dll (cuProfilerStop + 0x994dc) [0x2ba25c]
========= Host Frame:C:\WINDOWS\system32\DriverStore\FileRepository\nvami.inf_amd64_c4c88e9630d3f61b\nvcuda64.dll [0x8b33b]
========= Host Frame:C:\WINDOWS\system32\DriverStore\FileRepository\nvami.inf_amd64_c4c88e9630d3f61b\nvcuda64.dll [0x7a7c2]
========= Host Frame:C:\WINDOWS\system32\DriverStore\FileRepository\nvami.inf_amd64_c4c88e9630d3f61b\nvcuda64.dll (cuProfilerStop + 0x11cc6a) [0x33d9ea]
========= Host Frame:C:\WINDOWS\system32\DriverStore\FileRepository\nvami.inf_amd64_c4c88e9630d3f61b\nvcuda64.dll [0x134f74]
========= Host Frame:C:\WINDOWS\system32\DriverStore\FileRepository\nvami.inf_amd64_c4c88e9630d3f61b\nvcuda64.dll [0x1314e3]
========= Host Frame:C:\WINDOWS\system32\DriverStore\FileRepository\nvami.inf_amd64_c4c88e9630d3f61b\nvcuda64.dll [0x136e53]
========= Host Frame:C:\WINDOWS\system32\DriverStore\FileRepository\nvami.inf_amd64_c4c88e9630d3f61b\nvcuda64.dll [0x27d37]
========= Host Frame:C:\WINDOWS\system32\DriverStore\FileRepository\nvami.inf_amd64_c4c88e9630d3f61b\nvcuda64.dll (cuDevicePrimaryCtxRetain + 0x19a) [0x1cdb6a]
========= Host Frame:D:\phd_projects\HFV1_GPU_2D\x64\Release\HFV1_GPU_2D.exe (cudart::contextStateManager::initPrimaryContext + 0x62) [0xc702]
========= Host Frame:D:\phd_projects\HFV1_GPU_2D\x64\Release\HFV1_GPU_2D.exe (cudart::contextStateManager::tryInitPrimaryContext + 0x32) [0xd332]
========= Host Frame:D:\phd_projects\HFV1_GPU_2D\x64\Release\HFV1_GPU_2D.exe (cudart::contextStateManager::initDriverContext + 0x134) [0xc644]
========= Host Frame:D:\phd_projects\HFV1_GPU_2D\x64\Release\HFV1_GPU_2D.exe (cudart::contextStateManager::getRuntimeContextState + 0x66) [0xc4b6]
========= Host Frame:D:\phd_projects\HFV1_GPU_2D\x64\Release\HFV1_GPU_2D.exe (cudart::doLazyInitContextState + 0x23) [0xd373]
========= Host Frame:D:\phd_projects\HFV1_GPU_2D\x64\Release\HFV1_GPU_2D.exe (cudart::cudaApiMalloc + 0x2d) [0x2bed]
========= Host Frame:D:\phd_projects\HFV1_GPU_2D\x64\Release\HFV1_GPU_2D.exe (cudaMalloc + 0xdd) [0x1dcd]
========= Host Frame:D:\phd_projects\HFV1_GPU_2D\x64\Release\HFV1_GPU_2D.exe (malloc_device + 0x13) [0x14903]
========= Host Frame:D:\phd_projects\HFV1_GPU_2D\x64\Release\HFV1_GPU_2D.exe (main + 0x28a) [0x2b4ba]
========= Host Frame:D:\phd_projects\HFV1_GPU_2D\x64\Release\HFV1_GPU_2D.exe (__scrt_common_main_seh + 0x10c) [0x32cc8]
========= Host Frame:C:\WINDOWS\System32\KERNEL32.DLL (BaseThreadInitThunk + 0x1e) [0x14d2e]
========= Host Frame:C:\WINDOWS\SYSTEM32\ntdll.dll (RtlUserThreadStart + 0x2b) [0x65a7b]
=========
========= Internal Memcheck Error: Initialization failed
========= Saved host backtrace up to driver entry point at error
========= Host Frame:C:\WINDOWS\system32\DriverStore\FileRepository\nvami.inf_amd64_c4c88e9630d3f61b\nvcuda64.dll (cuProfilerStop + 0x97ba8) [0x2b8928]
========= Host Frame:C:\WINDOWS\system32\DriverStore\FileRepository\nvami.inf_amd64_c4c88e9630d3f61b\nvcuda64.dll (cuProfilerStop + 0x994dc) [0x2ba25c]
========= Host Frame:C:\WINDOWS\system32\DriverStore\FileRepository\nvami.inf_amd64_c4c88e9630d3f61b\nvcuda64.dll [0x7a7da]
========= Host Frame:C:\WINDOWS\system32\DriverStore\FileRepository\nvami.inf_amd64_c4c88e9630d3f61b\nvcuda64.dll (cuProfilerStop + 0x11cc6a) [0x33d9ea]
========= Host Frame:C:\WINDOWS\system32\DriverStore\FileRepository\nvami.inf_amd64_c4c88e9630d3f61b\nvcuda64.dll [0x134f74]
========= Host Frame:C:\WINDOWS\system32\DriverStore\FileRepository\nvami.inf_amd64_c4c88e9630d3f61b\nvcuda64.dll [0x1314e3]
========= Host Frame:C:\WINDOWS\system32\DriverStore\FileRepository\nvami.inf_amd64_c4c88e9630d3f61b\nvcuda64.dll [0x136e53]
========= Host Frame:C:\WINDOWS\system32\DriverStore\FileRepository\nvami.inf_amd64_c4c88e9630d3f61b\nvcuda64.dll [0x27d37]
========= Host Frame:C:\WINDOWS\system32\DriverStore\FileRepository\nvami.inf_amd64_c4c88e9630d3f61b\nvcuda64.dll (cuDevicePrimaryCtxRetain + 0x19a) [0x1cdb6a]
========= Host Frame:D:\phd_projects\HFV1_GPU_2D\x64\Release\HFV1_GPU_2D.exe (cudart::contextStateManager::initPrimaryContext + 0x62) [0xc702]
========= Host Frame:D:\phd_projects\HFV1_GPU_2D\x64\Release\HFV1_GPU_2D.exe (cudart::contextStateManager::tryInitPrimaryContext + 0x32) [0xd332]
========= Host Frame:D:\phd_projects\HFV1_GPU_2D\x64\Release\HFV1_GPU_2D.exe (cudart::contextStateManager::initDriverContext + 0x134) [0xc644]
========= Host Frame:D:\phd_projects\HFV1_GPU_2D\x64\Release\HFV1_GPU_2D.exe (cudart::contextStateManager::getRuntimeContextState + 0x66) [0xc4b6]
========= Host Frame:D:\phd_projects\HFV1_GPU_2D\x64\Release\HFV1_GPU_2D.exe (cudart::doLazyInitContextState + 0x23) [0xd373]
========= Host Frame:D:\phd_projects\HFV1_GPU_2D\x64\Release\HFV1_GPU_2D.exe (cudart::cudaApiMalloc + 0x2d) [0x2bed]
========= Host Frame:D:\phd_projects\HFV1_GPU_2D\x64\Release\HFV1_GPU_2D.exe (cudaMalloc + 0xdd) [0x1dcd]
========= Host Frame:D:\phd_projects\HFV1_GPU_2D\x64\Release\HFV1_GPU_2D.exe (malloc_device + 0x13) [0x14903]
========= Host Frame:D:\phd_projects\HFV1_GPU_2D\x64\Release\HFV1_GPU_2D.exe (main + 0x28a) [0x2b4ba]
========= Host Frame:D:\phd_projects\HFV1_GPU_2D\x64\Release\HFV1_GPU_2D.exe (__scrt_common_main_seh + 0x10c) [0x32cc8]
========= Host Frame:C:\WINDOWS\System32\KERNEL32.DLL (BaseThreadInitThunk + 0x1e) [0x14d2e]
========= Host Frame:C:\WINDOWS\SYSTEM32\ntdll.dll (RtlUserThreadStart + 0x2b) [0x65a7b]
=========
========= RACECHECK SUMMARY: 2 hazards displayed (0 errors, 0 warnings)

Cheers.

for the cupti dll, you have to provide the path to the cupti dll in your PATH environment variable(s)

In case anyone is facing something similar, I solved my problem by running compute-sanitizer --tool racecheck instead, as per this forum post.

I also resolved the cupti64_2020.3.0.dll was not found error by following the advice from this answer (same as above).