From the error messages, you’re probably missing some OptiX include directories in the build setting of your *.cu to *.ptx translation step.
Hmm, or you’re actually not only translating to PTX, but try to build CUBINs and those do not know about the OptiX device code intrinsics because those are translated by OptiX from the input PTX code.
Means you’re compiling the *.cu code to the wrong target. It needs to compile to the --ptx target only.
How do I add OptiX Support to existing VS projects?
Please have a look into the OptiX SDK 7.6.0 example optixConsole.
As mentioned inside the OptiX SDK Release Notes:
New sample optixConsole
demonstrates how to build OptiX applications in Microsoft Visual Studio without the use of CMake
Prints to stdout an OptiX generated image converted to ASCII art
It contains a *.vcxproj and *.sln file.
Please look through the individual settings done for all *.cu files inside the CUDA Visual Studio Integration.
Both build environments (CMake custom build rules and the CUDA Visual Studio Integration) are using the NVCC compiler in the end.
You basically need to set all CUDA compiler settings to the values mentioned in this section of the OptiX Programming Guide:
https://raytracing-docs.nvidia.com/optix7/guide/index.html#program_pipeline_creation#program-input