Visual Studio 2017 - it takes a few minutes to start compilation of a sample program

Hello there

Every time I compile a Cuda file I can see Visual Studio hangs for a few minutes at a certain build step - the IDE is still responsive, but no output is generated from the build process. After waiting a few minutes compilation goes further and finishes in a few seconds.

Currently I am using Cuda Tookit 9.2 with Visual Studio 15.6, but I observed the same behaviour with Cuda 10.0 and the newset version of Visual Studio.

I set MSBuild output level to ‘detailed’ and pasted output from the compilation here: https://justpaste.it/7l3ck

I marked the place where compilation stops for a few minutes with a line:

A HANG TAKES PLACE HERE <<<

I will appreciate every help and suggestion to solve this issue.

The log shows that the slow part of the build is “generating dependencies”. Googling shows that slowness in this portion of a build is a frequent issue with MSVC independent of CUDA. Dependency building involves the host compiler pre-processing and parsing the code to extract everything that looks like a file reference, then retrieving the OS status of those files.

As is typical on the internet, tons of people are reporting this problem, through multiple versions of MSVC, but not much info on a root cause or a solution can be found. One person seemed to think that it has to do with a file belonging to the project being located in a directory with numerous (literally: thousands) of files, all of which appeared to get indexed by the OS as part of the file status determination.

If that were true, turning off file indexing in the OS configuration settings could be the solution. I know I turned that off years ago on my Windows 7 developer system, not sure whether Windows 10 still allows this.

Thank you for your response.

I disabled file indexing on the entire drive, unfortunately it didn’t help.

Are 100% sure that this issue cannot be dependent of CUDA? I have a single project containing usual cpp files and cu files. Every time I compile a cpp file then building is fast, only if I build a cu file I experience hangs.

Also note that I do not see any specific disk activity during hang, in the task manager the disk activity is 0%.

I don’t think I am 100% sure about much of anything :-) As I said, I Googled the problem description and found multiple reports of slowdown without CUDA involvement on the first page of results. It is certainly possible that the use of CUDA toolchain exacerbates an existing issue. But as you can see from the log the “hang” happens when the host compiler is invoked on a .cu file, i.e. the CUDA toolchain is not in the picture at that stage:

I never use any IDE (or cmake, for that matter) to build CUDA projects. I am a dinosaur who is still familiar buiding with makefiles. No mysterious slowdowns there. The “building dependencies” is something the MSVC IDE does, and I am not aware that Microsoft has documented what exactly happens in that step. As a sanity check: Do you have any files that belong to this project that are located in huge directories? Any files that reside on (slow) network storage rather than fast local storage? A highly fragmented disk drive?

The only build “hangs” I have seen with the CUDA toolchain is when an optimization pass exhibits exponential growth in resource usage, e.g. a build chews through 8GB of memory and takes 20 minutes to compile one file. Those are basically bugs in the compiler, but I haven’t encountered any such issues in years. Since CUDA makes extensive use of function inlining and loop unrolling, sometimes a small-looking source file can balloon to several tens of thousands of instructions, causing longer compiles. But all those issues impact compilation proper, not “building dependencies”.