I am new to optix. The lab uses a lower version of optix, so optix6.5 is used, but the official example cannot be single-step debugged by VSE after being processed by cmake. I would like to ask how to deal with single-step debugging. VS2015, CUDA10.1, optix6.5, Nsight Minitor2019.1
Do you mean single stepping through the OptiX device code?
I’m not sure that ever fully worked with OptiX 6 versions and VSE. Maybe also try cuda-gdb.
Source code debugging would require the necessary debug information to be present inside the input PTX source code. Meaning you would need to translate the *.cu files to *.ptx files with the --generate-line-info (-lineinfo) and --device_debug (-G) and no optimizations NVCC options.
That’s not even recommended inside the OptiX 6.5.0 Programming Guide:
https://raytracing-docs.nvidia.com/optix6/guide_6_5/index.html#building#ptx-generation
After that OptiX would be required to retain that information when rewriting the PTX input source code to the kernels and there have been many occasions where that was not working in the past.
Even with the newest OptiX SDK 7.7.0 and display driver releases the device code debugging feature is still in a preview phase and not fully working, yet. Have a look through the other forum threads which discuss device code debugging.
It’s possible to use the rtPrintf or the native CUDA printf function (recommended) inside OptiX 6 device programs though, to get information about what’s happening without any need for NVCC debug options.
Mind that many threads run in parallel and you would need to limit the output to a specific launch index for reasonably precise information
If you have any chance of using OptiX 7.x versions, that would be highly recommended over the legacy API used in earlier versions.