--keep deletes sources

My Makefile rules typically expand to something like

nvcc -DHAS_CUDA -I/sfw/cuda/4.0/include -O0 -g -G -DENABLE_PARAMETER_CHECK --ptxas-options=-v  -arch sm_13 -DHAS_CUDADOUBLEPREC -DNUM_MEM_BANK=16 -c -o /home/user/goeddeke/nobackup/feastobj/temp/1/object/pc-coreduo-linux64-intel-goto-optNO/coproc_axpy_cuda.o coproc_axpy_cuda.cu

which is not too funky except that I’m using the Intel 11.1 suite for all non-cuda stuff in my build process. I switched to CUDA 4.0 today, added --keep to compare PTX output, and guess what happened: The $%&/$$% new CUDA compiler deleted my .cu source files!!! Luckily, it didn’t remove the backup files my editor tends to generate ;)

Honestly, guys. As much as I like debugging by redoing things over again properly, I don’t want a compiler to optimise all my efforts away on the file system level :)

This is on an Ubuntu 10.04 LTS box (which is for some funky reason unsupported although I tend to expect LTS from CUDA by now, but glibc and gcc and everything are very close to the supported 10.10, check http://distrowatch.com/table.php?distribution=ubuntu ), and I am 100% sure that nvcc never sees the Intel (or PSC, or PGI, or Sun or whatever) compilers I use under the hood.

Mildly annoyed, and confirmed again in never using .0 software :)

Dominik

My Makefile rules typically expand to something like

nvcc -DHAS_CUDA -I/sfw/cuda/4.0/include -O0 -g -G -DENABLE_PARAMETER_CHECK --ptxas-options=-v  -arch sm_13 -DHAS_CUDADOUBLEPREC -DNUM_MEM_BANK=16 -c -o /home/user/goeddeke/nobackup/feastobj/temp/1/object/pc-coreduo-linux64-intel-goto-optNO/coproc_axpy_cuda.o coproc_axpy_cuda.cu

which is not too funky except that I’m using the Intel 11.1 suite for all non-cuda stuff in my build process. I switched to CUDA 4.0 today, added --keep to compare PTX output, and guess what happened: The $%&/$$% new CUDA compiler deleted my .cu source files!!! Luckily, it didn’t remove the backup files my editor tends to generate ;)

Honestly, guys. As much as I like debugging by redoing things over again properly, I don’t want a compiler to optimise all my efforts away on the file system level :)

This is on an Ubuntu 10.04 LTS box (which is for some funky reason unsupported although I tend to expect LTS from CUDA by now, but glibc and gcc and everything are very close to the supported 10.10, check http://distrowatch.com/table.php?distribution=ubuntu ), and I am 100% sure that nvcc never sees the Intel (or PSC, or PGI, or Sun or whatever) compilers I use under the hood.

Mildly annoyed, and confirmed again in never using .0 software :)

Dominik

When you use [font=“Courier New”]–keep[/font], [font=“Courier New”]nvcc[/font] creates intermediate files of the form [font=“Courier New”]x.cu.cpp[/font] (for [font=“Courier New”]x.cu[/font]). I believe the problem is that [font=“Courier New”]make[/font] sees your rule:

x: x.cu

	...

and sees that there exists a file named [font=“Courier New”]x.cu.cpp[/font] which is newer than [font=“Courier New”]x.cu[/font]. Therefore it uses an implicit rule and first deletes [font=“Courier New”]x.cu[/font], then tries to make it (as an executable) from [font=“Courier New”]x.cu.cpp[/font] (which fails). You can check that this is the case by disabling implicit rules using [font=“Courier New”]make -r[/font] or [font=“Courier New”]make --no-builtin-rules[/font]. You can also create a rule to override the implicit rule:

x.cu:

#empty

Incidentally, I would have expected the same behaviour with older versions of the CUDA Toolkit since it is actually [font=“Courier New”]make[/font] that is deleting the files. Did you use [font=“Courier New”]–keep[/font] before? Or did you explicitly cleanup the intermediate files between builds?

Also, if you are a registered developer I encourage you to submit a bug about this to have NVIDIA look into a (different) workaround (and if you’re not, then I encourage you to become one!) ;)

When you use [font=“Courier New”]–keep[/font], [font=“Courier New”]nvcc[/font] creates intermediate files of the form [font=“Courier New”]x.cu.cpp[/font] (for [font=“Courier New”]x.cu[/font]). I believe the problem is that [font=“Courier New”]make[/font] sees your rule:

x: x.cu

	...

and sees that there exists a file named [font=“Courier New”]x.cu.cpp[/font] which is newer than [font=“Courier New”]x.cu[/font]. Therefore it uses an implicit rule and first deletes [font=“Courier New”]x.cu[/font], then tries to make it (as an executable) from [font=“Courier New”]x.cu.cpp[/font] (which fails). You can check that this is the case by disabling implicit rules using [font=“Courier New”]make -r[/font] or [font=“Courier New”]make --no-builtin-rules[/font]. You can also create a rule to override the implicit rule:

x.cu:

#empty

Incidentally, I would have expected the same behaviour with older versions of the CUDA Toolkit since it is actually [font=“Courier New”]make[/font] that is deleting the files. Did you use [font=“Courier New”]–keep[/font] before? Or did you explicitly cleanup the intermediate files between builds?

Also, if you are a registered developer I encourage you to submit a bug about this to have NVIDIA look into a (different) workaround (and if you’re not, then I encourage you to become one!) ;)

Hi Tom,

thanks for the update (and also thanks to Mark who pointed me into the same direction). It seems I never triggered this “bug/annoying interference” before, because I only compiled single source files on the command line with the extra --keep and not the whole thing. I’ll go back to this approach because it’s much easier than changing the build infrastructure for a huge software base :)

Here’s an analogy back from my Fortran/GNU cpp preprocessor days: If nvcc could dump intermediate files to the directory specified by -o instead of the directory where the sources are, then make should not complain, right? This seems like an implementable fix to me, though I am not sure if this breaks common practice with how other compilers treat intermediates. We actually did something like that manually to be able to use cpp for Fortran, see the attached script.

Next time, I’ll use my nvdeveloper account, but generally, response time on the forums seems to be quicker due to more people reading posts here.

Cheers,

Dominik
f90cpp.txt (14.9 KB)

Hi Tom,

thanks for the update (and also thanks to Mark who pointed me into the same direction). It seems I never triggered this “bug/annoying interference” before, because I only compiled single source files on the command line with the extra --keep and not the whole thing. I’ll go back to this approach because it’s much easier than changing the build infrastructure for a huge software base :)

Here’s an analogy back from my Fortran/GNU cpp preprocessor days: If nvcc could dump intermediate files to the directory specified by -o instead of the directory where the sources are, then make should not complain, right? This seems like an implementable fix to me, though I am not sure if this breaks common practice with how other compilers treat intermediates. We actually did something like that manually to be able to use cpp for Fortran, see the attached script.

Next time, I’ll use my nvdeveloper account, but generally, response time on the forums seems to be quicker due to more people reading posts here.

Cheers,

Dominik

just for the record: Bug ID 841010, repro at http://www.mathematik.tu-dortmund.de/~goeddeke/temp/make-keep.tgz