First of all, if you’re starting new with OptiX and you can run the OptiX SDK 7.2.0 examples, there is little reason to look at earlier versions.
OptiX 6.5.0 and OptiX 7 have a completely different API and OptiX 7 is the future.
Please read this post: https://forums.developer.nvidia.com/t/optix-7-breaking-changes/156801
For more resources about OptiX 7, follow the links in the sticky posts of this sub-forum.
https://forums.developer.nvidia.com/t/optix-7-2-release/156619
About your issues:
1.) OptiX SDK 6.5.0 is not officially supporting CUDA 11 versions.
Please always refer to the OptiX Release Notes before setting up a development environment. (Link directly beneath the individual OptiX version’s download buttons.)
I would recommend using CUDA 10.1 for OptiX 6.5.0, because that’s the version with which it was build.
2.) Seems only the Linux makefile does anything.
I compile my main.cpp file with nvcc by following command:
This is not necessary.
In an OptiX program you would normally only compile the CUDA files which contain the device code with the OptiX programs from *.cu to *.ptx source.
The OptiX SDK examples use a custom build rule for each *.cu file to accomplish that.
For a different version of such custom build rule generation with CMake look into my OptiX 7 applications:
https://github.com/NVIDIA/OptiX_Apps/blob/master/3rdparty/CMake/nvcuda_compile_module.cmake
https://github.com/NVIDIA/OptiX_Apps/blob/master/apps/rtigo10/CMakeLists.txt#L195
These PTX source files get loaded via the rtProgramCreateFromPTXString() of rtProgramCreateFromPTXFile() functions inside OptiX 1 to 6 versions, resp. with optixModuleCreateFromPTX() in OptiX 7 versions.
3.) There is no need to compile to the latest/current GPU’s streaming multi-processor version.
There actually have been cases where the OptiX PTX parser was not supporting all latest PTX instructions, yet.
If you want to support all GPU architectures with only a single version of the PTX files, you should compile against the lowest supported streaming multiprocessor version which would be SM 5.0 (Maxwell) for both OptiX 6 and 7 versions.
See these posts and the links in them: https://forums.developer.nvidia.com/t/support-multiple-compute-capabilities/126076/2
Note that CUDA 11 removed support for SM 3.0 and 3.2 and deprecated support for SM 3.5, 3.7 (Kepler), and 5.0 (Maxwell).
That’s why there is a -Wno-deprecated-gpu-targets in my NVCC_OPTIONS.
The OptiX SDK 7 examples build against SM 6.0 (Pascal) by default to avoid the deprecation warnings.
https://forums.developer.nvidia.com/t/optix-7-1-issue-with-running-samples-on-a-maxwell-card/140118
4.) CUDA 11 changed some defines for the 64-bit detection in NVRTC. It requries a check or #if defined(__x86_64) instead of #if defined(_WIN64). The shipping OptiX SDK 6.5.0 is not doing that. As said, it’s not officially supporting CUDA 11 versions.
Inside the OptiX SDK CMake build, you can disable the use of NVRTC by disabling the CMake variable CUDA_NVRTC_ENABLED. The *.ptx files are then compiled during build-time with the custom rules using NVCC.