Your NVCC command line tries to compile shader.cu to object code. (See your --compile and -o render.dir\Debug\shaders.obj options.)
If that shaders.cu file contains OptiX device code, that is the wrong NVCC command line.
The OptiX module input is either PTX source code or OptiX-IR binary code.
To translate a *.cu file to a *.ptx or *.optixir file, you use the NVCC command line options --ptx and --optix-ir.
https://raytracing-docs.nvidia.com/optix8/guide/index.html#program_pipeline_creation#program-input
I explained two ways how to setup standalone OptiX application frameworks which generate NVCC command lines for OptiX device code translation with CMake in this recent post. That had exactly the same incorrect NVCC options.
https://forums.developer.nvidia.com/t/converting-vs-property-sheet-into-cmake-settings/287159/2
The cuda/LocalGeometry.h header is from the OptiX SDK examples and you seem to be missing the proper include directory.
You have -I"C:\ProgramData\NVIDIA Corporation\OptiX SDK 8.0.0\include" which contains all headers for the OptiX API. That is usually all you need when writing your own applications.
The OptiX SDK examples are separate and their headers are inside C:\ProgramData\NVIDIA Corporation\OptiX SDK 8.0.0\SDK\cuda then.
Some helpful links about why it’s a good idea to implement an own OptiX application framework outside the OptiX SDK examples:
https://forums.developer.nvidia.com/t/optixhello-embeded-in-new-application-run-in-release-but-not-in-debug-mode/253037/4
https://forums.developer.nvidia.com/t/question-about-add-a-cu-in-my-project/284761/2
https://forums.developer.nvidia.com/t/a-problem-when-i-want-to-createmodule/276228
I’m not sure what you’re intending to do with tiny-cuda-nn but note that OptiX device code has some restrictions on what CUDA features can be used, most notably “applications cannot use shared memory, synchronization, barriers, or other SM-thread-specific programming constructs in their programs supplied to OptiX.”
https://raytracing-docs.nvidia.com/optix8/guide/index.html#introduction#overview
If you need that, you could implement them inside native CUDA kernels and use OptiX for intersection testing only, explained here:
https://forums.developer.nvidia.com/t/lack-of-support-for-threadfence-in-optix-ir/269353/4