You did not at all try the second method, because then your CMakeLists.txt wouldn’t have any remainders of the OptiX SDK CMakeLists.txt. (e.g. the OPTIX_add_sample_executable macro)
I repeat:
If you’re planning to add an example program inside the OptiX SDK application framework which should use CUDA native kernels and OptiX device code, look at the OptiX SDK 8.0.0/SDK/optixRaycasting/CMakeLists.txt file.
Specifically take a good look at this statement which changes only the optixRaycastingKernels.cu translation to CUDA object code.
set_source_files_properties(${CMAKE_CURRENT_SOURCE_DIR}/optixRaycastingKernels.cu
PROPERTIES CUDA_SOURCE_PROPERTY_FORMAT OBJ
)
I would not recommend that if you plan to ever give that application to somebody else as binary, because OptiX SDK example executables only work in the location where they have been built due to some hardcoded paths inside the executable for the module input code and possible data (textures, etc). which requires two environment variables to be set when running these executables in another location. Deploying OptiX SDK example programs as standalone programs is not what that application framework is meant for.
Now, if you want to develop a standalone CUDA Runtime API application framework with the CMake native support for the CUDA language and also translate OptiX device code to PTX or OptiX-IR code in the same project, then follow the instructions I posted in the linked thread above by the letter. That is, the part after “If you want to build a CMake project for OptiX from scratch, I would recommend the following:” in that post to get the necessary dependencies (*.cmake scripts and root CMakeLists.txt).
You must use the whole CMakeLists.txt I posted there and add the necessary parts from the root CMakeLists.txt in my example framework into it, esp. the statement setting the CMAKE_MODULE_PATH and the find_package(OptiX80)
Then exchange the list of files in all these set(NAME files) sections with the names HEADERS_HOST, SOURCES_HOST, HEADERS_CUDA, SOURCES_CUDA, HEADERS_OPTIX, SOURCES_OPTIX with the files you want.
For that you better copy all necessary files your application consists of into a local folder where you want your project to live (not inside the OptiX SDK).
Then try getting that CMakeLists.txt to work and produce a solution which builds.
If there is any issue reported by CMake which is not about missing files, read the CMake documentation.
I don’t want to use the find_package (optix) framework at the moment, because changing the current project may have a bigger problem.
All that find_package does, is just finding the desired OptiX SDK version’s include directory.
Inside the OptiX SDK examples that is a relative location. Look for OptiX_INCLUDE inside the OptiX SDK root CMakeLists.txt:
# Search for the OptiX libraries and include files.
find_package(OptiX REQUIRED)
# Add the path to the OptiX headers to our include paths.
include_directories(
"${OptiX_INCLUDE}"
"${CMAKE_CURRENT_SOURCE_DIR}/cuda"
)