1.) The *.cu files with the OptiX device programs only need to be translated from CUDA to PTX source.
That can either be done with NVCC at compile time or with NVRTC at runtime.
(Note that your application could also contain native CUDA kernels (apart from OptiX device programs) which you can handle as you need.
That might actually not be too seldom with OptiX 7 applications since these are using CUDA runtime or driver host API calls to manage most resources used in OptiX 7.)
The resulting *.ptx source files are neither linked, nor automatically found.
Again, it’s the developer’s responsibility to provide that PTX input source to OptiX at application runtime.
In OptiX 7 that’s done with optixModuleCreateFromPTX().
(In OptiX 1 to 6 that is done with rtProgramCreateFromPTXString() of rtProgramCreateFromPTXFile())
Please read the OptiX Programming Guide(s): https://raytracing-docs.nvidia.com/
2.) All host code only needs to be translated by the host compiler, like for example GCC under Linux or Visual Studio under Windows.
The CUDA toolkits set the main requirements for this. Check the CUDA_Installation_Guide_Linux.pdf resp. CUDA_Installation_Guide_Windows.pdf files in your CUDA toolkit installation’s documentation folder which contains a table with the compatible host compiler versions at the beginning.
3.) In my OptiX 7 examples that works like this:
The NVCUDA_COMPILE_PTX script (EDIT: Now replaced with the nvcuda_compile_module.cmake inside 3rdparty/CMake) called here:
https://github.com/NVIDIA/OptiX_Apps/blob/master/apps/intro_runtime/CMakeLists.txt#L122
generates one custom build rule per *.cu files I listed in SHADERS and uses the list of files in SHADER_HEADERS as dependencies so that when I change a file of either list, the project recompiles the affected *.cu files.
If you want to see the individual NVCC command lines which are used per *.cu file, you would only need to add this message command into line 42 of my nvcuda_compile_ptx.cmake script.
message("${CUDA_NVCC_EXECUTABLE} " "--machine=64;" "--ptx;" "${NVCUDA_COMPILE_PTX_NVCC_OPTIONS} " "${input} " "-o " "${output}")
These get printed when configuring the CMake project.
Means you could even put these individual commands in a script and translate your CUDA files to PTX outside the project.
The resulting *.ptx filenames land in PTX_SOURCES and that is added to the add_executable() list to make sure the custom build rules are called on these when building the executable. They aren’t linked or added to the executable binary in any way.
I also put them into the files of the solution (I’m working under Windows) in source_group( "ptx" FILES ${PTX_SOURCES}) to be able to open them quickly in case I want to see the effect of a change in the *.cu files in source_group( "shaders" FILES ${SHADERS} )
The OptiX SDK 7 examples do that similarly but it’s harder to see. Follow the OPTIX_add_sample_executable definition and look for the CUDA_WRAP_SRCS function in FindCUDA.cmake.