Damn, that’s due to the CUDA_HOST_COMPILER setting which is not finding the matching cl.exe location, because MSVS 2017 changed the installation folder hierarchy and FindCUDA.cmake is probably not aware of that.
The solution is simple though and part of this thread:
[url]https://devtalk.nvidia.com/default/topic/1036401/optix/optix-installation-windows-cuda-9-2/post/5265364/#5265364[/url]
To be able to compile *.cu files which are not used in OptiX device code, but as native kernels (all OptiX Prime examples and the optixRaycasting example and for unknown reasons sutil) with MSVS 2017, I also needed to set the CUDA_HOST_COMPILER in the CMake GUI explicitly to the 64-bit cl.exe.
In my installation that was here:
“C:/Program Files (x86)/Microsoft Visual Studio/2017/Professional/VC/Tools/MSVC/14.14.26428/bin/Hostx64/x64/cl.exe”