NVCC intermediate files; trying to diagnose intermittent build fails

Hi,

I am looking at an issue where NVCC only sometimes fails to compile some files while building large Visual Studio solutions. (The common denominator is that the files are using one of our headers that contain some not very complicated template functions in combination with macros. Several of us have looked at them and believe they are correct. And they work fine with CL.)

I cannot reliably reproduce the problem even in our own environments and with the full code base, so rather than a proper bug report I am instead going for some questions related to a vague suspicion to try to narrow down the issue:

  1. Is the NVCC use of intermediate files safe from race conditions when building a Visual Studio solution in parallel? (Is parallelization supported? I think I remember that the building of CUDA-files was serial before we switched from CUDA 9.1 and Visual Studio 2017 (using the 2015 C++ compiler).)

  2. Is the NVCC use of intermediate files safe from race conditions when concurrently building several solutions containing CUDA files with the same name? (E.g., different repository clones.)

  3. How can I best control where the intermediate files are stored? If I understand https://docs.nvidia.com/cuda/cuda-compiler-driver-nvcc/index.html#miscellaneous-nvcc-usage correctly the location can only be controlled via command line options if the files are to be kept (not desirable for us) or via the TEMP environment variable (which is a bit hard to get right for all entry points to our builds).

Environment: Visual Studio 2019 16.3.1/16.3.2, MSBuild, CUDA 10.1.243, Runtime API.

Thanks,
T