I have a problem with CUDA 10.0. We are building our solution file with VS2017 + Incredibuild. Incredibuild doesn’t support the /Zi flag (pdb error occurs randomly). so the IB support suggested that we should use /Z7 instead. We changed our compiler flags according to this, but after the cmake generation I can see in the logs that all of the nvcc parts are using the old flag:
How many people use Incredibuild? I hadn’t even heard of it until I read this thread. It seems to me you would want to tell IB to get a move on and go fix their bug (“doesn’t support the /Zi flag”).
About 20 IB users. And it’s not a bug. IB builds the projects parallel and sometimes they try to write the same .pdb file. The solution is: change /Zi to /Z7, and every other projects are fine with that. Only the CUDA related compiles are failing, randomly. The main problem is that nvcc forces /Zi when cmake generates the files even if I strictly use /Z7 in our compiler flags.
So who constructs the commandline with the offending flag now? Incredibuild, MSVS, cmake, or actually nvcc? I have been a user of Microsoft C/C++ compilers since they ran on 16-bit DOS. I don’t recall problems with /Zi, it’s a perfectly legitimate flag to specify and use. IMHO, if IB has trouble parallelizing builds that use /Zi, that is IB’s problem to solve, worst case by forcing serialization. This, to me, is a bug (namely, invoking undefined behavior): “IB builds the projects parallel and sometimes they try to write the same .pdb file.”
Anyhow, the answer to your question seems to be: the intersection of the three sets “IB user”, “CUDA user”, “NVIDIA forum participant”, possibly has cardinality 1 at this time.
If you would like to see changes in how nvcc behaves, you can always file a feature request via the bug reporting form.
/Zi is actually a big problem with highly parallel builds. See this thread about “fatal error C1090: PDB API call failed”: Visual Studio Feedback
This sounds a lot like what the OP is seeing. nvcc unilaterally adding /Zi prevents working around the problem. The issue isn’t with Incredibuild-- its a bug with cl that nvcc is exercising.
If you believe this “bug with cl” is something that should be addressed by nvcc, you are free to file a bug with NVIDIA. There should be a sticky note at the top of this forum on how to do that.
If the root cause of the issue is something that cl does incorrectly, the correct course of action seems to be to file a bug with Microsoft, as it is always preferable to fix root causes, not add layers of workarounds. More people filing bug reports on the same issue could move the bug up in the priority list.
Looking more closely at nvcc builds with MSVC, it does seem to add a -Zi to the response file it passes to cl.exe whenever nvcc is invoked with -g, indicating a debug build. And if one manually adds an -Xcompiler /Z7, this is inserted in the response file before the -Zi, which means the latter overrides the former and one winds up with -Zi again.
The main difference between /Zi and /Z7 is that the former dumps the debug information into a separate .pdb file, while the latter dumps it directly into the .obj object file. The linker can later collect the debug information from the object files into a .pdb file when it creates the executable. Therefore, specifying /Z7 could have an impact on build times and debugger functionality. Whether it has a negative impact on the CUDA toolchain, I cannot say.
If I recall correctly, the distinction between /Zi and /Z7 was introduced with Microsoft’s first 32-bit toolchain (Microsoft C/C++ Optimizing Compiler 8) in 1993, where /Z7 was for backwards compatibility with version 7 of that compiler. Which leads me to believe that /Z7 invokes a less capable legacy mode.
I would suggest filing an enhancement request with NVIDIA (using the bug reporting mechanism) to let a user-specified /Z7 override /Zi (maybe by simply changing the order in which compiler flags are added to the response file) and see what they come back with.
Incorrect. In terms of capabilities, both formats are identical. What /Zi (as a compiler flag) implies, is a shared caching service (mspdbsrv.exe) which runs in parallel to the compiler (cl.exe) and attempts to pull ahead the workload of merging debug symbols from multiple translation units into a single file from link-time to compilation time.
That’s also part of the reason what renders it so highly problematic with any distributed and/or caching build systems. The other half of the issue is long-standing problem where this service is ABI-incompatible between different compiler versions and build configurations, but yet fails to properly isolate/instance itself by default.
The cross-project isolation issues can be hacked around (_MSPDBSRV_ENDPOINT_), and the translation unit isolation can be hacked around (/Fd compiler flag, forcing an individual PDB per translation unit).
Except when you reach that point, literally all benefits /Zi had provided are eliminated. Same link times, same file size. In return you are merely left with some additional IPC overhead for communication with the shared service. Or non-shared actually, so it even slows down compilation as additional processes need to be spun up in a distributed environment…
Supporting /Z7 in nvcc is by no means a “nice to have”. In fact, trying to enforce /Zi is probably one of the many details which the compiler team at NVidia should be regretting whenever MS publishes a new compiler version. Remember what I just wrote, the ABI behind /Zi tends to break on a regular base.
It’s neither a bug on IB nor on MSVC side - the /Zi-feature is behaving as designed. And it’s also why the /Z7 format has never been declared deprecated. It’s what you need to keep the compiler / linker compatible with any distributed compilation scheme. It’s even the recommended way to distribute debug symbols with static libraries.
Thanks for relating interesting some technical details. In terms of a practical resolution I would refer back to the first and last paragraphs of my previous post.