CUDACOMPILE : nvcc error : 'cudafe++' died with status 0xC0000409

[vaa]

Windows 11 (10.0.22621.1992/22H2)
CUDA 12.1

After the Microsoft Visual Studio upgrade from version 17.4 to 17.6.2 I started to get the above error for every .cu file in the project. I end up with two versions of Visual Studio installed on my computer, 17.4 LTS and 17.6.5. 17.4 LTS compiles the code without any issues, but 17.6.5 always returns the following error for every CUDA file in the project:

1>CUDACOMPILE : nvcc error : 'cudafe++' died with status 0xC0000409
1>C:\Program Files\Microsoft Visual Studio\2022\Professional\MSBuild\Microsoft\VC\v170\BuildCustomizations\CUDA 12.1.targets(799,9): 
error MSB3721: The command ""C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.1\bin\nvcc.exe" 
 -gencode=arch=compute_61,code=\"sm_61,compute_61\" 
 -gencode=arch=compute_70,code=\"sm_70,compute_70\" 
 -gencode=arch=compute_75,code=\"sm_75,compute_75\" 
 -gencode=arch=compute_80,code=\"sm_80,compute_80\" 
 -gencode=arch=compute_86,code=\"sm_86,compute_86\" 
 -gencode=arch=compute_89,code=\"sm_89,compute_89\" 
 -gencode=arch=compute_90,code=\"sm_90,compute_90\" 
 --use-local-env -ccbin "C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.36.32532\bin\HostX64\x64" -x cu 
-I"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.1\include"
-I"C:\projects\vcpkg\installed\x64-windows\include" 
-I"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.1\include"  
-G   --keep-dir x64\Debug  -maxrregcount=0   --machine 64 --compile -cudart static 
-std=c++20 -Xcompiler "/std:c++20" -g  -DWIN32 -DWIN64 -D_DEBUG -D_CONSOLE 
-D_CRT_SECURE_NO_WARNINGS -D_WINDLL -D_MBCS -Xcompiler "/EHsc /W3 /nologo /Od /FS /Zi /RTC1 
/MDd " -Xcompiler "/Fdx64\Debug\vc143.pdb" 
-o C:\projects\MyProject\x64\Debug\Search.cu.obj "C:\projects\MyProject\Search.cu"" exited with code 9.
1>Done building project "MyProject.vcxproj" -- FAILED.
1>Compilation aborted.

Haven’t found any information what this error could mean.

Would appreciate any help.

[vaa]

I identified the issue. This block of code causes the above crush with the Visual Studio 17.6 on Windows, but works with VS <=17.5:

#include <source_location>

This piece of code is used in some common file in my project which is responsible for error handling, which in turn included by other files.

Is this a nvcc or Microsoft bug?

[vaa]

Reported bug #4240353.
It could be somehow related to this issue.

[dejvbayer]

Hi, it is solved, just upgrade to CUDA 12.2.

Dave

[vaa]

Hi @dejvbayer,

No, it isn’t.
Having the same issue with this compiler version:

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Tue_Jul_11_03:10:21_Pacific_Daylight_Time_2023
Cuda compilation tools, release 12.2, V12.2.128
Build cuda_12.2.r12.2/compiler.33053471_0

Just create an empty C++ file, put #include <source_location> there and use these compiler options: -std=c++20 -Xcompiler "/std:c++20"

[dejvbayer]

Aaah, okey, I did not try it out using MSVC. Maybe you should submit a bug and describe your problems.

Dave

[chojnace]

Hi, it seems to be an issue with how the MSVC implements std::source_location using the new c++20 type of constant evaluation that NVCC doesn’t support yet.

I posted a code snippet in the other thread about how to work around it for anyone that might be having the same issue.

[shocker.0x15]

Not sure if the issue has been once resolved at some point, but now I see the same error with CUDA 12.5 + VS 2022 17.10.0.

I made the minimal reproducer:
repro_cpp20_cudafepp.zip
(I have already reported the bug #4664399.)

CUDA 12.5
Visual Studio 2022 17.10.0
NVIDIA Driver: 555.85
Win 11 Pro 23H2
RTX 4080

[sergeyn]

I’ve just hit same problem . Any workarounds ?

[MaWy]

Same problem here after upgrading Visual Studio 2022 to version 17.10
CUDA 12.4.1 and CUDA 12.5; Win10; RTX 4080

[Yuki_Ni]

This maps to NVBUG 4666709 4664399
[Public] Hi Mark ,

Thanks for filing this ticket . We were aware of this issue yesterday , VS 2022 v17.10 just released almost on same day to our 12.5 . Our compiler engineering team is eagerly working on this problem . We will keep users informed once it is resolved . Please kindly try an older MSVC version other than the latest 14.40.33807 as a workaround if this is being any real blocker to you . Thanks !

Best,
Yuki

[shocker.0x15]

I did rollback to VS 2022 17.9.7 for now.
Fortunately rollback itself is quite easy because VS has implemented the rollback feature in VS installer.

[sergeyn]

Can we get notified here when the fix is ready ?

[patrikhuber]

You don’t need to rollback the VS IDE version to 17.9.7.
You can keep 17.10.0, you just need to go to the VS Installer, and under Individual Components, select the MSVC Build Tools x64/x86 14.39 - 17.9. This will give you the new IDE but with the old compiler & build tools.

Still, this issue is a big bummer - it will probably mean that if you need an older CUDA version, you’ll be stuck on the MSVC14.39 build tools version, so we need to wait using the latest MSVC compiler until the very last part of the dependency chain has updated CUDA to 12.5 / 12.6 or whatever version will have this issue fixed.
I hope that perhaps NVIDIA can release maintenance patches for older CUDA versions as a solution to this @Yuki_Ni?

[rainer.bitschi]

I can confirm the issue with VS 17.10.1 and also the workaround by @patrikhuber

[Yuki_Ni]

[Public] Hi All ,

We are glad to let you know the issue is fixed and verified in house . This fix will be part of next Update release of CUDA 12.5 soon . We don’t back port fixes to older CUDA releases and there is no patch version to older CUDA releases . To make previous version work on c++20 (the issue is specific to c++20) , please use MSVC < 14.40.33807 . Thanks again for reporting this .

Best,
Yuki