Nvcc overrides /Z7 with /Zi

Accelerated Computing CUDA CUDA Programming and Performance
#1

Hi,

I have a problem with CUDA 10.0. We are building our solution file with VS2017 + Incredibuild. Incredibuild doesn’t support the /Zi flag (pdb error occurs randomly). so the IB support suggested that we should use /Z7 instead. We changed our compiler flags according to this, but after the cmake generation I can see in the logs that all of the nvcc parts are using the old flag:

“C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0\bin\nvcc.exe” … -Xcompiler "/EHsc /W0 /nologo /Od /FdDebug\vc141.pdb /FS /Zi /RTC1 …

How should I change that behaviour?

Thanks in advance,
Gergely Pilisi

#2

Strange, 0 answer. I am the only one who run into this?

#3

How many people use Incredibuild? I hadn’t even heard of it until I read this thread. It seems to me you would want to tell IB to get a move on and go fix their bug (“doesn’t support the /Zi flag”).

#4

About 20 IB users. And it’s not a bug. IB builds the projects parallel and sometimes they try to write the same .pdb file. The solution is: change /Zi to /Z7, and every other projects are fine with that. Only the CUDA related compiles are failing, randomly. The main problem is that nvcc forces /Zi when cmake generates the files even if I strictly use /Z7 in our compiler flags.

#5

So who constructs the commandline with the offending flag now? Incredibuild, MSVS, cmake, or actually nvcc? I have been a user of Microsoft C/C++ compilers since they ran on 16-bit DOS. I don’t recall problems with /Zi, it’s a perfectly legitimate flag to specify and use. IMHO, if IB has trouble parallelizing builds that use /Zi, that is IB’s problem to solve, worst case by forcing serialization. This, to me, is a bug (namely, invoking undefined behavior): “IB builds the projects parallel and sometimes they try to write the same .pdb file.”

Anyhow, the answer to your question seems to be: the intersection of the three sets “IB user”, “CUDA user”, “NVIDIA forum participant”, possibly has cardinality 1 at this time.

If you would like to see changes in how nvcc behaves, you can always file a feature request via the bug reporting form.

#6

/Zi is actually a big problem with highly parallel builds. See this thread about “fatal error C1090: PDB API call failed”:
https://developercommunity.visualstudio.com/content/problem/48897/c1090-pdb-api-call-failed-error-code-23.html

This sounds a lot like what the OP is seeing. nvcc unilaterally adding /Zi prevents working around the problem. The issue isn’t with Incredibuild-- its a bug with cl that nvcc is exercising.

#7

If you believe this "bug with cl" is something that should be addressed by nvcc, you are free to file a bug with NVIDIA. There should be a sticky note at the top of this forum on how to do that.

If the root cause of the issue is something that cl does incorrectly, the correct course of action seems to be to file a bug with Microsoft, as it is always preferable to fix root causes, not add layers of workarounds. More people filing bug reports on the same issue could move the bug up in the priority list.

Looking more closely at nvcc builds with MSVC, it does seem to add a -Zi to the response file it passes to cl.exe whenever nvcc is invoked with -g, indicating a debug build. And if one manually adds an -Xcompiler /Z7, this is inserted in the response file before the -Zi, which means the latter overrides the former and one winds up with -Zi again.

The main difference between /Zi and /Z7 is that the former dumps the debug information into a separate .pdb file, while the latter dumps it directly into the .obj object file. The linker can later collect the debug information from the object files into a .pdb file when it creates the executable. Therefore, specifying /Z7 could have an impact on build times and debugger functionality. Whether it has a negative impact on the CUDA toolchain, I cannot say.

If I recall correctly, the distinction between /Zi and /Z7 was introduced with Microsoft’s first 32-bit toolchain (Microsoft C/C++ Optimizing Compiler 8) in 1993, where /Z7 was for backwards compatibility with version 7 of that compiler. Which leads me to believe that /Z7 invokes a less capable legacy mode.

I would suggest filing an enhancement request with NVIDIA (using the bug reporting mechanism) to let a user-specified /Z7 override /Zi (maybe by simply changing the order in which compiler flags are added to the response file) and see what they come back with.