In the following steps replace the OptiX SDK version number with what you’re using.
With a clean OptiX SDK 7.7.0 installation:
Do you build the OptiX SDK example from the top-level CMakeLists.txt file inside the SDK folder?
Did you pick a separate build folder inside the CMake GUI? Something like <your_path>/OptiX SDK 7.7.0/SDK/build
That should generate a MSVS solution inside that build folder named OptiX-Samples.sln.
If you open that inside your MSVS version, there should be all OptiX SDK examples listed, including optixHello.
If you build them all as Release x64 target, their executables should land in <your_path>\OptiX SDK 7.7.0\SDK\build\bin\Release.
Open a command prompt and change directory to that folder.
Issue the command optixHello.exe. Ther should open a wiondow with a green background.
Now, inside your MSVS version, open the optixHello draw_solid_color.cu file.
Change it from
make_color( make_float3( rtData->r, rtData->g, rtData->b ) );
to
make_color( make_float3( 1.0f, 0.0f, 0.0f ) );
and build the solution.
Go to the command prompt again and start optixHello.exe again.
Now the color of the client window area should be pure red.
Please try to get that working inside a clean OptiX SDK installation.
After you built all SDK examples and got them running from the command prompt, changing the optixHello draw_solid_color.cu file inside the MSVS IDE will generate an updated optixHello_generated_draw_solid_color.cu.optixir (or *.ptx file when you disabled the SAMPLES_INPUT_GENERATE_OPTIXIR variable inside the CMake GUI).
That should land in <your_path>\OptiX SDK 7.7.0\SDK\build\lib\ptx\Release
Make sure that destination file is actually updated after you changed and rebuild the draw_solid_color.cu file.
This must work first. Don’t touch anything inside the OptiX SDK until you get this right.
OptiX device code debugging might not work like that depending on the display driver you’re using. It’s unfinished and partly broken in R530 drivers.
Read this OptiX Programming Guide chapter 6.1 for the prerequisites and this thread for current issues and workarounds: https://forums.developer.nvidia.com/t/debugging-is-broken-after-updating-to-cuda-12-1/245616