Nvcc error : 'cicc' died with status 0xC0000005 - Only in DEBUG mode

ForceflowBE · June 8, 2023, 11:43am

I’m trying to compile the latest dev version of my cuda_voxelizer project: GitHub - Forceflow/cuda_voxelizer at dev

In Release mode (x64), the compilation succeeds, and my compiled result behaves like I expect it to, getting correct results from my tests (voxelizing polygon structures).
In Debug mode (x64), the compilation fails because NVCC crashes.

Build started...
1>------ Build started: Project: cuda_voxelizer, Configuration: Debug x64 ------
1>Compiling CUDA source file ..\..\src/voxelize.cu...
1>Compiling CUDA source file ..\..\src\voxelize_solid.cu...
1>
1>D:\dev\cuda_voxelizer\msvc\vs2022>"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.1\bin\nvcc.exe" -gencode=arch=compute_52,code=\"sm_52,compute_52\" -gencode=arch=compute_60,code=\"sm_60,compute_60\" -gencode=arch=compute_61,code=\"sm_61,compute_61\" -gencode=arch=compute_70,code=\"sm_70,compute_70\" -gencode=arch=compute_75,code=\"sm_75,compute_75\" -gencode=arch=compute_80,code=\"sm_80,compute_80\" --use-local-env -ccbin "C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.36.32532\bin\HostX64\x64" -x cu   -I"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.1\include" -I"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.1\include"  -G   --keep-dir x64\Debug  -maxrregcount=0   --machine 64 --compile -cudart static --source-in-ptx -g  -DWIN32 -DWIN64 -D_DEBUG -D_CONSOLE -D_MBCS -Xcompiler "/EHsc /W3 /nologo /Od /FS /Zi /RTC1 /MDd " -Xcompiler "/Fdx64\Debug\vc143.pdb" -o D:\dev\cuda_voxelizer\msvc\vs2022\x64\Debug\voxelize.cu.obj "D:\dev\cuda_voxelizer\src\voxelize.cu"
1>
1>D:\dev\cuda_voxelizer\msvc\vs2022>"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.1\bin\nvcc.exe" -gencode=arch=compute_52,code=\"sm_52,compute_52\" -gencode=arch=compute_60,code=\"sm_60,compute_60\" -gencode=arch=compute_61,code=\"sm_61,compute_61\" -gencode=arch=compute_70,code=\"sm_70,compute_70\" -gencode=arch=compute_75,code=\"sm_75,compute_75\" -gencode=arch=compute_80,code=\"sm_80,compute_80\" --use-local-env -ccbin "C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.36.32532\bin\HostX64\x64" -x cu   -I"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.1\include" -I"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.1\include"  -G   --keep-dir x64\Debug  -maxrregcount=0   --machine 64 --compile -cudart static --source-in-ptx -g  -DWIN32 -DWIN64 -D_DEBUG -D_CONSOLE -D_MBCS -Xcompiler "/EHsc /W3 /nologo /Od /FS /Zi /RTC1 /MDd " -Xcompiler "/Fdx64\Debug\vc143.pdb" -o D:\dev\cuda_voxelizer\msvc\vs2022\x64\Debug\voxelize_solid.cu.obj "D:\dev\cuda_voxelizer\src\voxelize_solid.cu"
1>voxelize.cu
1>CUDACOMPILE : nvcc error : 'cicc' died with status 0xC0000005 (ACCESS_VIOLATION)
1>C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\BuildCustomizations\CUDA 12.1.targets(799,9): error MSB3722: The command ""C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.1\bin\nvcc.exe" -gencode=arch=compute_52,code=\"sm_52,compute_52\" -gencode=arch=compute_60,code=\"sm_60,compute_60\" -gencode=arch=compute_61,code=\"sm_61,compute_61\" -gencode=arch=compute_70,code=\"sm_70,compute_70\" -gencode=arch=compute_75,code=\"sm_75,compute_75\" -gencode=arch=compute_80,code=\"sm_80,compute_80\" --use-local-env -ccbin "C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.36.32532\bin\HostX64\x64" -x cu   -I"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.1\include" -I"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.1\include"  -G   --keep-dir x64\Debug  -maxrregcount=0   --machine 64 --compile -cudart static --source-in-ptx -g  -DWIN32 -DWIN64 -D_DEBUG -D_CONSOLE -D_MBCS -Xcompiler "/EHsc /W3 /nologo /Od /FS /Zi /RTC1 /MDd " -Xcompiler "/Fdx64\Debug\vc143.pdb" -o D:\dev\cuda_voxelizer\msvc\vs2022\x64\Debug\voxelize.cu.obj "D:\dev\cuda_voxelizer\src\voxelize.cu"" exited with code 5. Please verify that you have sufficient rights to run this command.
1>Done building project "cuda_voxelizer.vcxproj" -- FAILED.
1>voxelize_solid.cu
1>CUDACOMPILE : nvcc error : 'cicc' died with status 0xC0000005 (ACCESS_VIOLATION)
1>C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\BuildCustomizations\CUDA 12.1.targets(799,9): error MSB3722: The command ""C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.1\bin\nvcc.exe" -gencode=arch=compute_52,code=\"sm_52,compute_52\" -gencode=arch=compute_60,code=\"sm_60,compute_60\" -gencode=arch=compute_61,code=\"sm_61,compute_61\" -gencode=arch=compute_70,code=\"sm_70,compute_70\" -gencode=arch=compute_75,code=\"sm_75,compute_75\" -gencode=arch=compute_80,code=\"sm_80,compute_80\" --use-local-env -ccbin "C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.36.32532\bin\HostX64\x64" -x cu   -I"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.1\include" -I"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.1\include"  -G   --keep-dir x64\Debug  -maxrregcount=0   --machine 64 --compile -cudart static --source-in-ptx -g  -DWIN32 -DWIN64 -D_DEBUG -D_CONSOLE -D_MBCS -Xcompiler "/EHsc /W3 /nologo /Od /FS /Zi /RTC1 /MDd " -Xcompiler "/Fdx64\Debug\vc143.pdb" -o D:\dev\cuda_voxelizer\msvc\vs2022\x64\Debug\voxelize_solid.cu.obj "D:\dev\cuda_voxelizer\src\voxelize_solid.cu"" exited with code 5. Please verify that you have sufficient rights to run this command.
1>Done building project "cuda_voxelizer.vcxproj" -- FAILED.
========== Build: 0 succeeded, 1 failed, 0 up-to-date, 0 skipped ==========
========== Build started at 1:39 PM and took 03,031 seconds ==========

As far as I can tell, my code doesn’t do anything out of the ordinary. The only external library I’m using in the kernels is the helper_cuda_math files from the official CUDA Samples. This is an example of a .cu file that fails to compile in DEBUG.

Update: Removing the –source-in-ptx option from the debug options resolves the issue, but makes my debug builds useless, afaik.

It’s hard to narrow down what the offending line is, because I’m getting no feedback. When I comment out nearly everything in the .cu files, the project continues, but uncommenting doesn’t narrow it down, seems random.

This has been happening on every release of the 12.x CUDA branch.

I can compile the CUDA Samples in Release and Debug Mode just fine, so I doubt anything is wrong with my CUDA install. I’ve got a relatively new card (4070 Ti). Can’t remember this happening on my older card (RTX 2070 Super).

Already tried:

Starting over with a totally new Visual Studio CUDA project, and adding my sources again, but I bump into the same problem.
Clean install of graphics drivers
New CUDA install

My environment:

Windows 10
Cuda 12.1
RTX 4070 TI, latest drivers
VS2022, latest updates

Any ideas?

kenneth.hoste · June 8, 2023, 5:56pm

I can reproduce this problem, or at least a very similar one, on Linux (RHEL8, x86_64) with CUDA 12.1

$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Tue_Feb__7_19:32:13_PST_2023
Cuda compilation tools, release 12.1, V12.1.66
Build cuda_12.1.r12.1/compiler.32415258_0

cmake -DCMAKE_CUDA_FLAGS="--source-in-ptx -G" -DCMAKE_BUILD_TYPE=Debug .. -DTrimesh2_INCLUDE_DIR=../../trimesh2/include -DTrimesh2_LINK_DIR=../../trimesh2/lib.Linux64

followed by

make VERBOSE=1

results in:

[ 75%] Building CUDA object CMakeFiles/cuda_voxelizer.dir/src/voxelize.cu.o
/software/CUDA/12.1.0/bin/nvcc -forward-unknown-to-host-compiler  -I/tmp/trimesh2/include -isystem=/software/CUDA/12.1.0/include --source-in-ptx -G -g --generate-code=arch=compute_52,code=[compute_52,sm_52] -std=c++17 -MD -MT CMakeFiles/cuda_voxelizer.dir/src/voxelize.cu.o -MF CMakeFiles/cuda_voxelizer.dir/src/voxelize.cu.o.d -x cu -c /tmp/cuda_voxelizer/src/voxelize.cu -o CMakeFiles/cuda_voxelizer.dir/src/voxelize.cu.o
nvcc error   : 'cicc' died due to signal 11 (Invalid memory reference)
nvcc error   : 'cicc' core dumped
make[2]: *** [CMakeFiles/cuda_voxelizer.dir/build.make:146: CMakeFiles/cuda_voxelizer.dir/src/voxelize.cu.o] Error 139

This smells like a bug in the NVCC compiler to me, or perhaps it’s a problem with excessive memory usage (but not likely I think).

kenneth.hoste · June 8, 2023, 6:05pm

Similar problem with totally different code reported recently in Cicc crash when compiling pytorch

kenneth.hoste · June 8, 2023, 6:09pm

Looks like a good candidate for reporting a bug, see How to report a bug

ForceflowBE · June 10, 2023, 11:24am

@kenneth.hoste Thanks for the repro. The problem is, I think I’ve got to narrow it down to a minimal working example to demonstrate the bug.

Yuki_Ni · June 28, 2023, 5:52am

Thanks for reporting this to NVBUG 4160838 . We confirmed this is a known issue to us and you are correct that it will be triggered with '–source-in-ptx -G ’ in debug mode . A fix is ready in house and it will target next CUDA release soon .

ForceflowBE · July 5, 2023, 11:31am

Update: Cuda 12.2 doesn’t fix it, but they’re targeting the fix for next release.

yukini20091 · April 30, 2024, 11:15am

CUDA 12.2 Update1 and later contain this fix .

Topic		Replies	Views
'cicc' compilation error and debug flag CUDA Programming and Performance	25	13908	May 23, 2023
Nvcc error ACCESS_VIOLATION CUDA NVCC Compiler	0	814	January 4, 2023
Nvcc 12.3 with gcc 13.2 not working CUDA NVCC Compiler	11	8026	March 12, 2024
CUDACOMPILE : nvcc error : 'cudafe++' died with status 0xC0000409 CUDA NVCC Compiler	16	4518	June 3, 2024
nvcc can't compile anything (Windows 10) CUDA Setup and Installation	0	2665	October 28, 2019
Problem with nvcc and ptxas when cross compiling tensorflow CUDA NVCC Compiler	1	1168	July 28, 2022
Can't build CUDA Separable Compilation sample application with CMake CUDA NVCC Compiler	1	1436	October 22, 2021
Cuda building error CUDA Setup and Installation	8	13661	September 12, 2021
Error in my code... CUDA Programming and Performance	11	2531	December 19, 2014
CUDA compile trouble CUDA Programming and Performance	47	5099	November 8, 2010

Nvcc error : 'cicc' died with status 0xC0000005 - Only in DEBUG mode

Related topics