Sporadic build errors in visual studio (MSB4175)

Accelerated Computing CUDA CUDA Programming and Performance
1

I’m getting sporadic build errors with no discernable cause.

C:\Program Files\[...]\CUDA 12.1.targets(799,9): error MSB4175: The task factory "XamlTaskFactory" could not be loaded from the assembly "[...]". Could not find file 'C:\Users\[...]\AppData\Local\Temp\w50zprkx.dll'.

CUDA 12.1, VS 2022, Windows 11.

This instability is new - I’ve built this same code hundreds of times. Hardware doesn’t seem to be an issue as the system is stable. I did just get an update (KB5032874) pushed, though.

The error occurs in random locations (not consistently sourced from compiling one module), sometimes up to four DLL’s missing.

Any thoughts? TIA

  Using "SetEnv" task from assembly "C:\Program Files\Microsoft Visual Studio\2022\Professional\MSBuild\Microsoft\VC\v170\Microsoft.Build.CppTasks.Common.dll".
  Task "SetEnv"
	PATH=C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.37.32822\bin\HostX64\x64;C:\Program Files (x86)\Windows Kits\10\bin\10.0.22000.0\x64;;C:\Program Files (x86)\Microsoft SDKs\Windows\v10.0A\bin\NETFX 4.8.1 Tools\x64;C:\Program Files (x86)\Windows Kits\10\bin\10.0.22000.0\x86;;C:\Program Files (x86)\Microsoft SDKs\Windows\v10.0A\bin\NETFX 4.8.1 Tools;C:\Program Files\Microsoft Visual Studio\2022\Professional\Common7\tools;C:\Program Files\Microsoft Visual Studio\2022\Professional\Common7\ide;C:\Program Files (x86)\HTML Help Workshop;;C:\Program Files\Microsoft Visual Studio\2022\Professional\MSBuild\Current\Bin\amd64;C:\WINDOWS\Microsoft.NET\Framework64\v4.0.30319\;C:\WINDOWS\Microsoft.NET\Framework\v4.0.30319\;C:\Program Files\Microsoft Visual Studio\2022\Professional\MSBuild\Current\Bin;C:\WINDOWS\Microsoft.NET\Framework\v4.0.30319\;;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.3\bin;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.3\libnvvp;C:\Python311\Scripts\;C:\Python311\;C:\Program Files (x86)\Razer Chroma SDK\bin;C:\Program Files\Razer Chroma SDK\bin;C:\Program Files (x86)\Razer\ChromaBroadcast\bin;C:\Program Files\Razer\ChromaBroadcast\bin;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.1\bin;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.1\libnvvp;C:\Program Files\Microsoft\jdk-11.0.16.101-hotspot\bin;C:\Program Files (x86)\Common Files\Oracle\Java\javapath;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;C:\Program Files\Microsoft SQL Server\150\Tools\Binn\;C:\Program Files\Microsoft SQL Server\Client SDK\ODBC\170\Tools\Binn\;C:\Program Files\dotnet\;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\Program Files\Git\cmd;C:\Program Files\Amazon\AWSCLIV2\;C:\WINDOWS\system32\config\systemprofile\AppData\Local\Microsoft\WindowsApps;C:\Program Files (x86)\Microsoft SQL Server\160\Tools\Binn\;C:\Program Files\Microsoft SQL Server\160\Tools\Binn\;C:\Program Files\Microsoft SQL Server\160\DTS\Binn\;C:\Program Files (x86)\Microsoft SQL Server\160\DTS\Binn\;C:\Program Files\Azure Data Studio\bin;C:\Program Files (x86)\Common Files\Acronis\SnapAPI\;C:\Program Files (x86)\Common Files\Acronis\VirtualFile\;C:\Program Files (x86)\Common Files\Acronis\VirtualFile64\;C:\Program Files (x86)\Common Files\Acronis\FileProtector\;C:\Program Files (x86)\Common Files\Acronis\FileProtector64\;C:\Program Files\nodejs\;C:\ProgramData\chocolatey\bin;C:\Program Files\NVIDIA Corporation\Nsight Compute 2023.3.0\;C:\Users\Robert\AppData\Local\Microsoft\WindowsApps;C:\Program Files\Azure Data Studio\bin;C:\Users\Robert\AppData\Local\Programs\Microsoft VS Code\bin;C:\Users\Robert\AppData\Roaming\npm;C:\Program Files\JetBrains\PyCharm 2023.2.2\bin;;C:\Users\Robert\.dotnet\tools;
  Done executing task "SetEnv".
  Task "SetEnv"
	LIB=C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.37.32822\lib\x64;;C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.37.32822\atlmfc\lib\x64;;C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Auxiliary\VS\lib\x64;;C:\Program Files (x86)\Windows Kits\10\lib\10.0.22000.0\ucrt\x64;;;C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Auxiliary\VS\UnitTest\lib;C:\Program Files (x86)\Windows Kits\10\lib\10.0.22000.0\um\x64;C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8.1\lib\um\x64;
  Done executing task "SetEnv".
  Task "SetEnv"
	LIBPATH=C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.37.32822\atlmfc\lib\x64;;C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.37.32822\lib\x64;
  Done executing task "SetEnv".
  Task "SetEnv"
	INCLUDE=C:\Work\Repos\Icarus-Newer\CudaCommon\;C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.37.32822\include;;C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.37.32822\atlmfc\include;;C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Auxiliary\VS\include;;C:\Program Files (x86)\Windows Kits\10\Include\10.0.22000.0\ucrt;;;C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Auxiliary\VS\UnitTest\include;C:\Program Files (x86)\Windows Kits\10\Include\10.0.22000.0\um;C:\Program Files (x86)\Windows Kits\10\Include\10.0.22000.0\shared;C:\Program Files (x86)\Windows Kits\10\Include\10.0.22000.0\winrt;C:\Program Files (x86)\Windows Kits\10\Include\10.0.22000.0\cppwinrt;C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8.1\Include\um;
  Done executing task "SetEnv".
  Task "SetEnv"
	EXTERNAL_INCLUDE=C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.37.32822\include;;C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.37.32822\atlmfc\include;;C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Auxiliary\VS\include;;C:\Program Files (x86)\Windows Kits\10\Include\10.0.22000.0\ucrt;;;C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Auxiliary\VS\UnitTest\include;C:\Program Files (x86)\Windows Kits\10\Include\10.0.22000.0\um;C:\Program Files (x86)\Windows Kits\10\Include\10.0.22000.0\shared;C:\Program Files (x86)\Windows Kits\10\Include\10.0.22000.0\winrt;C:\Program Files (x86)\Windows Kits\10\Include\10.0.22000.0\cppwinrt;C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8.1\Include\um;
  Done executing task "SetEnv".
Target "SetUserMacroEnvironmentVariables" skipped, due to false condition; ('@(BuildMacro)' != '' and '$(DesignTimeBuild)' != 'true') was evaluated as ('' != '' and '' != 'true').
Target PrepareForCudaBuild:
  Using "SanitizePaths" task from assembly "C:\Program Files\Microsoft Visual Studio\2022\Professional\MSBuild\Microsoft\VC\v170\BuildCustomizations\Nvda.Build.CudaTasks.v12.1.dll".
  Task "SanitizePaths"
  Done executing task "SanitizePaths".
  Using "CudaSetEnvironmentVariable" task from assembly "C:\Program Files\Microsoft Visual Studio\2022\Professional\MSBuild\Microsoft\VC\v170\BuildCustomizations\Nvda.Build.CudaTasks.v12.1.dll".
  Task "CudaSetEnvironmentVariable"
	Nvda.Build.CudaTasks.CudaSetEnvironmentVariable: Setting 'CUDAFE_FLAGS' to '--sdk_dir "C:\Program Files (x86)\Windows Kits\10"'
  Done executing task "CudaSetEnvironmentVariable".
Target CudaBuildCore:
  Building target "CudaBuildCore" completely.
  Output file "C:\Work\Repos\Icarus-Newer\CudaRuntimeCompiler\x64\Release\gen_98.cu.obj" does not exist.
  Task "Message"
	Compiling CUDA source file gen_98.cu...
  Done executing task "Message".
  Task "Message" skipped, due to false condition; ('$(__ExcludedFromBuild)' == 'true') was evaluated as ('' == 'true').
  Task "MakeDir" skipped, due to false condition; (!Exists('$(CompileOutDir)')) was evaluated as (!Exists('C:\Work\Repos\Icarus-Newer\CudaRuntimeCompiler\x64\Release')).
  Task "MakeDir" skipped, due to false condition; ('$(__Keep)' == 'true' AND !Exists('$(__KeepDir)')) was evaluated as ('false' == 'true' AND !Exists('x64\Release')).
  Initializing task factory "XamlTaskFactory" from assembly "Microsoft.Build.Tasks.v4.0, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a".
  C:\Program Files\Microsoft Visual Studio\2022\Professional\MSBuild\Microsoft\VC\v170\BuildCustomizations\CUDA 12.1.targets(799,9): error MSB4175: The task factory "XamlTaskFactory" could not be loaded from the assembly "Microsoft.Build.Tasks.v4.0, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a". Could not find file 'C:\Users\Robert\AppData\Local\Temp\dqfytygx.dll'.
Done building target "CudaBuildCore" in project "CudaRuntimeCompiler.vcxproj" -- FAILED.
2

Is this a parallelized build? If so, there could be a race condition somewhere in the build process. Are you building using MSVS’s native build process, or some other build tool?

I recall seeing infrequent reports in this sub-forum of race conditions when building CUDA-enabled apps with MSVS. I seem to recall the most recent one had to do with separately generated .pdb files. Is there anything in the recent installed update that could lead to race conditions?

[Later:] Just looked at the description of KB5032874 and it’s a “balance of security updates” kind of update. Past experience tells me that MS has a habit of mixing unrelated updates into their “security updates” blobs.

3

Nope, no parallelization.

I’m building in Visual Studio and my automation calls devenv /build for consistency. I just tried it in msbuild and it worked in the command line, but my automated builds failed with some new errors.

fatal error C1083: Cannot open compiler generated file: 'C:\[...]\CudaRuntimeCompiler\x64\Release\gen_88.cu.obj': Permission denied [C:\[...]\CudaRuntimeCompiler\CudaRuntimeCompiler.vcxproj]

I’m starting to think it’s Windows Defender or something. I have it disabled but it’s still running (and consuming 20% of my 7950X). I’m starting to see now why there are so many people running on Linux.

4

Sorry, I don’t have any ideas what could be going wrong. I don’t use the MSVS IDE and build tools.

I am pretty OS agnostic having spent roughly half my career working on Windows and the other half on Linux/Unix. My (usually unpopular) advice for development under Windows is to use the Microsoft command line tools only and build with gmake. In other words, do not use MSVS as a build environment. Plus use all Unix-style helper tools: through the commercial MKS Toolkit in the olden days; then Cygwin for the past 20 years.

I have been using Microsoft toolchains since the first version (a rebadged Lattice C, ca. 1983) and while they have never been top-notch at any particular time, they are perfectly fine for day-to-day work. For performance work I have been partial to Intel C/C++ (and Fortran) compilers since around 1995, but their early “Proton” compilers were full of bugs and became generally useful about 20 years ago. But of course they just integrate into whatever the native development environment on a particular platform is.

Is Linux a bit less tedious as a development platform compared to Windows? In my experience, yes. I would put what I call the “Windows tax” at around 20% additional effort. But of course then there are developers who insist on using a widely popular idiosyncratic Linux distro and cmake, and that levels the playing field :-)

5

That was an awesome story :) Thanks for sharing.

I started in on a C64 in basic, moved to QBasic in DOS, VB on Windows, then moved into C and ASM. I didn’t touch Linux until much later in my career, and even then, those were just the prod environments. Anyone who tried to lure me away was using emacs or VIM, to which I would often retort, “Ok sure, but if we’re going to take a step back, why not punch cards? =P”. (tongue-in-cheek: I do respect those editors). I’ve never met anyone who runs a real GUI on Linux - all the Linux junkies I know who use a GUI use Apple.

VB made coding easy, and taught me good debugger habits. MS brought a lot of that ease into Visual Studio, and that has always felt like driving a Lamborghini, especially in lower-level languages like C. VS still feels like a lambo - a 20 year old lambo rife with maintenance issues. I haven’t brought myself to switch to something else like Rider.

I still love Windows - it’s really an awesome OS, but the user experience is increasingly annoying. Notification popups that block my screen, taskbar hangs, unwanted advertisements, and settings sprawled across 3 generations of attempts to unify it (Control Panel, MMC, Metro Settings). I yearn for better.

But yeah back on topic: msbuild issue was resolved by disabling defender, but it’s like 20 times slower for some reason (which just makes me think it’s still a race condition mitigated by slower performance). Using devenv still breaks.

I rolled back to a much earlier commit and the issue went away, which is progress. I just have to chase down exactly what changed. The biggest change was that I changed to relocatable code for a custom 128-bit int library I wrote for use with MSVC (I can use ___int128 on Linux, but no support on Windows AFAIK). I inlined it at-first, but some requirements changed and I needed to compile it. Gonna start there I guess.

6

Also, thanks for the advice on gmake. I’ll look into it.

7

The new clang-based Intel compiler icx supports __int128 on Windows, but at least with the compiler I downloaded at the start of this year that seems to be somewhat buggy. I got link errors due to some missing helper routines with some of the code I tried. I don’t think that is my fault, but I have not spent enough time to track that down. This new Intel compiler integrated just fine into the MSVS 2019 infrastructure I use here. After spending thousands of dollars on multiple generations of Intel compilers in the past, I was happy that (likely due to competition in the HPC field provided by NVIDIA :-), this compiler was offered free of charge.

My biggest gripe about Windows right now is that one can defer updates only for a limited amount of time. I sometimes run long-running brute-force optimization programs, and more than once a system update with reboot was enforced before such a multi-hundred-hour run was complete. I have an old Windows 7 machine which no longer receives updates that I can use to run such code forever (or the next power outage at least), but obviously that’s not as zippy as my newer Windows10 system, and it lacks certain CPU features like AVX-512.

8

I wish NVIDIA officially supported something other than MSVC on Windows. Some years ago I invested a week setting up clang for this project, and everything worked great except for one case with functors and anonymous methods. I had to get into the PTX to verify the issue. I wanted to report the bug, but I stumbled across another thread complaining about some idiosyncrasy with CUDA on clang and the devs were like essentially like “CUDA is outside of the scope of what we support.” Not wanting to learn how to debug clang, I mothballed it and settled on using MSVC, hoping the official support model would change by now. Surprised it hasn’t.

/edit: My software compiles kernels JIT so i was looking for something faster and redistributable on windows as a stop-gap until I could implement a proper LLVM front-end. clang was much faster, but I couldn’t use it :(

I have a game server running Windows and I have the same issue - it always restarts itself no matter how I reconfigure it. Allow me to do something or don’t, but like, be clear about it. It’s like they don’t want you to disable updates or anti-virus, but they’ll let you jump through hoops thinking you can, but you really can’t. I understand why to a degree, but at least let the power users configure the OS the way they need.

Any advice on inlining my lib? I switched to linking because when I inlined my Int128_t class in multiple modules I kept getting duplicate symbol errors because __device__ was exporting the same Int128_t methods used in different modules with the same name. Linking solved the issue since the Int128_t methods only got exported once, but maybe there is something I don’t understand well enough to make inlining work here?

9

CUDA integrates tightly with the host compiler. That is why adjustments from the CUDA side are necessary for major host compiler releases. When we started CUDA, we debated whether the integration with the host compiler should be tight, or loose, as was later chosen by OpenCL. We opted for tight integration as this made the use of CUDA as seamless as possible and allowed for __host__ __device__ code. In hindsight this was the right move for getting CUDA off the ground quickly and achieving a steep adoption curve.

Unfortuntately this approach also comes with a good chunk of ongoing maintenance work which strongly encourages keeping the number of supported host compilers as small as possible. On Linux, it used to be gcc. I assume through pressure from the HPC community eventually support for the Intel compiler was added to the Linux platform. And in recent years clang is taking over from gcc (for good reasons, judging by the generated code). On Windows, MSVC dominates and toolchains other than MSVC are a tiny niche, and so I understand NVIDIA’s reluctance to take on more support burden for little gain.

I do not have any good advice on how to go about your library. I would think that a header-file only library with __host__ __device__ functions should inline just fine? As a point of reference, until CUDA version 6.5 or so the entire CUDA standard math library (that is, what gets expose via math.h) was simply a header-file only implementation, and the code inlined just fine. For various reasons we then moved that functionality to the LLVM level.

I do not need 128-bit integers often, and have never looked into providing a general purpose library for them as I am comfortable dropping down to a bit on inline assembly should that be needed locally.

10

Thanks so much for the advice. I’ve identified the root cause of my issue. Using devenv /build causes the build to fail ~50% of the time. I have no problems if I use msbuild or build the project manually within Visual Studio. I’ll open a ticket with the VS team. Hopefully this helps someone else: it seems at least one other person hit this same issue in 2021.

I was using devenv originally under the presumption it would more accurately replicate the build conditions under which the application was developed. msbuild is definitely the more popular automation tool, though, and it seems like a poor judgement call in hindsight.

The justification for using native toolsets makes complete sense - I appreciate the insight.

11

VS ticket opened here.

This could also be a bug in the CUDA build customizations that’s only reproducible when invoked this specific way, so I also submitted a bug with NVIDIA.