Slow compilation C++/Cuda GenerateDeps

Hello,
I use Microsoft Visual Studio Professional 2017 15.8.2. My project is configurated with “Plateform Toolset” : Visual studio 2015 (v140) and “Target Plateform Version” : 10.016299.0. I use Window 7 and CUDA 9.1.
When I build my project (C++/CUDA code), I get a slow compiltation issue in debug and realease mode .
When I launch the build of this same project on another computer with quite simular configuration (same memory/Visual Studio version/CUDA version …), I have big compiltation time difference (~6min VS ~18s).
With the “diagnostic” compilation ouput, I can see that the compilation is very slow in the “GenerateDeps” step on my computer. I have tried to compare the configuration of this two computers but I don’t see link between the configuration difference and slow compiltation.
This issue have been reported by others people on the web, but no solution have been given (https://stackoverflow.com/questions/35776620/why-cuda-7-5-visual-studio-2013-extremely-slow-generating-dependencies, https://developercommunity.visualstudio.com/content/problem/236387/slow-performance-after-update-kb4093112.html, https://devtalk.nvidia.com/default/topic/820722/cuda-setup-and-installation/tremendously-slow-build-with-visual-studio-2013-msbuild-/ ).
Have you some lead to explore?
Thanks for your help,
Regards,
Pierre.

Disk I/O is often a significant performance factor in compilation activities. Is it possible one machine has a HDD, the other an SSD? The working hypothesis being that the machine with SSD is faster due to much higher sustained I/O per second (e.g. 30 vs 3000).

Other factors in compilation speed are typically CPU performance and system memory throughput. Those would not explain the factor 20 in compile time you observe, however.

There could be issues triggered by environment variables, but I don’t use MSVS project files or the GUI (I use command-line tools and makefiles) so wouldn’t know what they might be. Maybe some extensive logging is enabled on the slow machine?

Thanks for your answer.

I have compared the disk between the two computers, it is the same type of disk.
I have done the compilation of my project with using of command-line and the slow compilation issue disappears. The issue comes from MSVS and my environment. I have set the same log level on the two MSVS and I get always the same issue of compiltation time difference. It is the “GenerateDeps” step which take time, but I don’t know why.

The only other generic reason I can think of is fragmentation of the disk storage slowing things down. But that wouldn’t amount to a factor of twenty.

Since the slowdown appears to be specific to the MSVS GUI in combination with your environment settings (search paths?), I would suggest asking in a relevant Microsoft forum.

I have cleaned my environment settings, install and reinstall MSVS but this issue remains. I will format and re-install Windows to start from a clear environment. I have informed Microsoft forum about that.
Thanks for your suggestion. I will keep you informed.

I have done a disk formatting and I have reinstalled Windows and MSVS2015. I have done the compilation and I didn’t get the slow compiltation issue. The issue should be came from my environment settings by I don’t know where was the issue.

Hello. Not here to bring up a solution, but to mention that I just started experiencing this with a new computer that the IT installed. It did not come from a clean install.
Using CUDA 10.2 and MS Visual Studio 2019. My home computer with the same hardware and software config doesn’t show this problem, so it’s clearly related to the environment.

Next step is formatting and trying again, but I thought that perhaps this issue (which is clearly a bug IMHO, 30 seconds to build up dependencies ?) might be relevant to the nvcc team. (Note that a C/C++ project builds fast)

1>Task Performance Summary:
1> 0 ms MakeDir 3 calls
1> 0 ms SanitizePaths 3 calls
1> 0 ms SplitToItemMetadata 2 calls
1> 0 ms CountItems 1 calls
1> 0 ms CudaSetEnvironmentVariable 2 calls
1> 1 ms Message 3 calls
1> 1 ms Delete 1 calls
1> 1 ms CheckVCToolsetVersion 1 calls
1> 1 ms ReadLinesFromFile 2 calls
1> 1 ms Touch 1 calls
1> 2 ms ResolvePaths 1 calls
1> 3 ms SetEnv 9 calls
1> 9 ms WriteLinesToFile 2 calls
1> 3900 ms CudaCompile 1 calls
1> 4196 ms MSBuild 1 calls
1> 31228 ms GenerateDeps 1 calls

Best,
Norman

EDIT: Found a duplicate: Tremendously slow build with Visual Studio 2013 (MSBuild)

EDIT 2: Reinstalling Windows 10 from scratch didn’t help.

EDIT 3: As found online, the flag /E used in cl.exe for generating dependencies is detrimental to performance (in my case 90% are white lines). Running with /P is a lot faster.