So you’re saying you want to implement an application framework which compiles some CUDA *.cu files with OptiX device code to PTX or OptiX IR targets and other CUDA *.cu files with native CUDA code to native cubin object code?
That is exactly what the optixRaycasting CMakeLists.txt is doing.
If you look at the first instruction inside its CMakeLists.txt:
set_source_files_properties(${CMAKE_CURRENT_SOURCE_DIR}/optixRaycastingKernels.cu
PROPERTIES CUDA_SOURCE_PROPERTY_FORMAT OBJ
)
that sets the source properties to OBJ for only the optixRaycastingKernels.cu source.
All other *.cu files get translated to the default targets set by the OptiX FindCUDA.cmake implementation
If you look at the implementation of the OPTIX_add_sample_executable functions inside the OptiX SDK root CMakeLists.txt, that partitions the given *.cu files into cu_obj_source_files and cu_optix_source_files which are then compiled differently by the following CUDA_WRAP_SRCS macros.
Now, that whole OptiX SDK CUDA handling is still using its own FindCUDA.cmake implementation.
Means that is all only working inside the OptiX SDK example application framework you probably shouldn’t be using for your own standalone applications but develop your own.
Note that the native FindCUDA.cmake implementation shipping with CMake is deprecated since CMake 3.10 and and replaced with FindCUDAToolkit.cmake which works similarly. (Means the whole OptiX SDK FindCUDA folder is obsolete in the meantime.)
https://cmake.org/cmake/help/latest/module/FindCUDA.html
CMake added native support for the CUDA language for some time. This is especially simple when writing native CUDA applications.
Unfortunately I don’t have an example which does both, native CUDA and OptiX device source with the CMake CUDA language feature. It’s on my list of things to try and my own OptiX application framework uses a different method with custom build rules and also doesn’t mix native CUDA object files. I use native PTX kernels and JIT-compile them.
If you’re not familiar with CMake, the native CUDA examples at https://github.com/nvidia/cuda-samples are using Visual Studio projects and solutions directly.
It should be possible to define individual build rules for the *.cu files with OptiX device code to compile them to *.ptx or *.optixir instead, but I haven’t tried that either.
I’d need to do some more experiments to be able to provide additional answers.
Related topic: https://forums.developer.nvidia.com/t/cmake-dont-compile-cuda-kernels/140937