Avoid generating temp files in /tmp while nvcc compiling

Accelerated Computing CUDA CUDA NVCC Compiler
,
#1

We have a customized compiling environment that /tmp directory may not work well. When compiling cuda kernels with nvcc + gcc, temparory files like tmpxft_[digits]_[digits]-digit_[kernel name].ptx format will be saved in /tmp directory, for example: tmpxft_0000000e_00000017-6_matmul.ptx.

If we add --keep option in nvcc commands, the temporary files will be saved in current workdir as this post. And this make our compiling work well in the most time. Because /tmp will not be read/write any files theoretically.

However, we still found a small number of temporary file like tmpxft_00000004_00000000-1.cpp appearing in the /tmp directory. The content of this temp file looks like:

#if defined(__cplusplus)                                            
__NV_CPLUSPLUS __cplusplus                                          
#endif /* defined(__cplusplus) */                                   
#if defined(__GNUC__)                                               
__NV_GNUC__ __GNUC__                                                
#if defined(__GNUC_MINOR__)                                         
__NV_GNUC_MINOR__ __GNUC_MINOR__                                    
#endif /* defined(__GNUC_MINOR__) */                                
#if defined(__GNUC_PATCHLEVEL__)                                    
__NV_GNUC_PATCHLEVEL__ __GNUC_PATCHLEVEL__                          
#endif /* defined(__GNUC_PATCHLEVEL__) */                           

...
                               
__NV_HORIZON__                                                      
#endif /* defined(__HORIZON__) */                                   
#if defined(_MSC_VER)                                               
__NV_MSC_VER _MSC_VER                                               
#endif /* defined(_MS_VER) */                                       
#if defined(_MSC_FULL_VER)                                          
__NV_MSC_FULL_VER _MSC_FULL_VER                                     
#endif /* defined(_MSC_FULL_VER) */                                 
#if defined(__CHAR_UNSIGNED__) || defined(_CHAR_UNSIGNED)           
__NV_CHAR_UNSIGNED__                                                
#endif /* defined(__CHAR_UNSIGNED__) || defined(_CHAR_UNSIGNED) */

It seems that --keep doesn’t work all time? Is there any method to avoid any usage of /tmp directory and nvcc only take current workdir for using?

#2

Does using the “–objdir-as-tempdir” option help?

https://docs.nvidia.com/cuda/cuda-compiler-driver-nvcc/index.html#file-and-path-specifications-objdir-as-tempdir

#3

Sorry, I have lost environment message. We are working on CUDA 10.2. And the –objdir-as-tempdir option is not found. Is there any option in CUDA 10.2 like –objdir-as-tempdir?

#4

Maybe try using the “–keep-dir” option as well.

https://docs.nvidia.com/cuda/archive/10.2/cuda-compiler-driver-nvcc/index.html#options-for-guiding-compiler-driver-keep-dir

#5

We have tried this option. However, the --keep-dir works like --keep option, cannot avoid all temporary file generated in /tmp directory. It seems that --keep-dir have the same effects with --keep option except the specific directory.

#6

Maybe then, the “–objdir-as-tempdir” option was added to get around the problems you are seeing.

#7

Unfortunately, our CUDA version is limited to 10.2. Maybe we have to fix the /tmp directory in compiling.

#8

Yes. Is there a reason why you need to stay on 10.2?

#9

We are work on DRIVE os. And the DRIVEOS 5.2.6 limit to CUDA 10.2

#10

It seems that nvcc generates unique temporary files in /tmp by pid?

Our problem have been fixed. It is caused by conflict temporary files name in /tmp. We integrate nvcc and bazel as cross compile tools. And the bazel sandbox is enabled. However, the /proc is isolated in bazel sandbox, so pid will start from zero in each sandbox (like starting in docker container). But the /tmp directory is mounted from host filesystem. So the temparory files maybe rewrite by other nvcc process in else sandboxes occasionally.

2 Likes