And what was the compilation error you got?
If that is a direct copy of the CMakeLists.txt, you’re using the incorrect quotation marks!
set(CUDA_NVCC_FLAGS “${CUDA_NVCC_FLAGS} -rdc=true”)
# See the difference?
set(CUDA_NVCC_FLAGS "-rdc=true;${CUDA_NVCC_FLAGS}")
# Add this message after you changed your CUDA_NVCC_FLAGS and see if there are non-ASCII characters printed.
message("CUDA_NVCC_FLAGS = " "${CUDA_NVCC_FLAGS}")
Also CUDA_NVCC_FLAGS is a semicolon separated list.
I’m not sure that will help though. You’re enabling separable compilation and relocatable-device-code for the whole project but only want to do that for a single file and it’s unclear what’s included in there. That might run into linking issues with CUDA runtime calls. I tried this with the optixRaycasting example and it is not working like this.
I would recommend to not use the OptiX SDK application framework for such things but migrate to your own standalone OptiX application framework as soon as possible if you ever want to ship your programs.
Please read these two posts and all links inside the whole threads about options for such OptiX application frameworks, especially since you seem to be trying to build an application with CUDA native kernels and OptiX device code.
There are better ways to set that up with the native CMake LANGUAGE CUDA feature described here.
That CMake example project has the separable compilation and relocatable device code set for the OptiX device code and it should be possible to do the same for the native CUDA code.
https://forums.developer.nvidia.com/t/converting-vs-property-sheet-into-cmake-settings/287159/5
https://forums.developer.nvidia.com/t/why-am-i-getting-optix-dir-notfound/279085/4