Avoid generating temp files in /tmp while nvcc compiling

We have a customized compiling environment that /tmp directory may not work well. When compiling cuda kernels with nvcc + gcc, temparory files like tmpxft_[digits]_[digits]-digit_[kernel name].ptx format will be saved in /tmp directory, for example: tmpxft_0000000e_00000017-6_matmul.ptx.

If we add --keep option in nvcc commands, the temporary files will be saved in current workdir as this post. And this make our compiling work well in the most time. Because /tmp will not be read/write any files theoretically.

However, we still found a small number of temporary file like tmpxft_00000004_00000000-1.cpp appearing in the /tmp directory. The content of this temp file looks like:

#if defined(__cplusplus)                                            
__NV_CPLUSPLUS __cplusplus                                          
#endif /* defined(__cplusplus) */                                   
#if defined(__GNUC__)                                               
__NV_GNUC__ __GNUC__                                                
#if defined(__GNUC_MINOR__)                                         
__NV_GNUC_MINOR__ __GNUC_MINOR__                                    
#endif /* defined(__GNUC_MINOR__) */                                
#if defined(__GNUC_PATCHLEVEL__)                                    
__NV_GNUC_PATCHLEVEL__ __GNUC_PATCHLEVEL__                          
#endif /* defined(__GNUC_PATCHLEVEL__) */                           

...
                               
__NV_HORIZON__                                                      
#endif /* defined(__HORIZON__) */                                   
#if defined(_MSC_VER)                                               
__NV_MSC_VER _MSC_VER                                               
#endif /* defined(_MS_VER) */                                       
#if defined(_MSC_FULL_VER)                                          
__NV_MSC_FULL_VER _MSC_FULL_VER                                     
#endif /* defined(_MSC_FULL_VER) */                                 
#if defined(__CHAR_UNSIGNED__) || defined(_CHAR_UNSIGNED)           
__NV_CHAR_UNSIGNED__                                                
#endif /* defined(__CHAR_UNSIGNED__) || defined(_CHAR_UNSIGNED) */

It seems that --keep doesn’t work all time? Is there any method to avoid any usage of /tmp directory and nvcc only take current workdir for using?

Does using the “–objdir-as-tempdir” option help?

https://docs.nvidia.com/cuda/cuda-compiler-driver-nvcc/index.html#file-and-path-specifications-objdir-as-tempdir

Sorry, I have lost environment message. We are working on CUDA 10.2. And the –objdir-as-tempdir option is not found. Is there any option in CUDA 10.2 like –objdir-as-tempdir?

Maybe try using the “–keep-dir” option as well.

https://docs.nvidia.com/cuda/archive/10.2/cuda-compiler-driver-nvcc/index.html#options-for-guiding-compiler-driver-keep-dir

We have tried this option. However, the --keep-dir works like --keep option, cannot avoid all temporary file generated in /tmp directory. It seems that --keep-dir have the same effects with --keep option except the specific directory.

Maybe then, the “–objdir-as-tempdir” option was added to get around the problems you are seeing.

Unfortunately, our CUDA version is limited to 10.2. Maybe we have to fix the /tmp directory in compiling.

Yes. Is there a reason why you need to stay on 10.2?

We are work on DRIVE os. And the DRIVEOS 5.2.6 limit to CUDA 10.2

It seems that nvcc generates unique temporary files in /tmp by pid?

Our problem have been fixed. It is caused by conflict temporary files name in /tmp. We integrate nvcc and bazel as cross compile tools. And the bazel sandbox is enabled. However, the /proc is isolated in bazel sandbox, so pid will start from zero in each sandbox (like starting in docker container). But the /tmp directory is mounted from host filesystem. So the temparory files maybe rewrite by other nvcc process in else sandboxes occasionally.

2 Likes