For debugging host code, use whatever development environment you are comfortable with.
cuda-gdb would be the Linux debugger for the CUDA kernel side, but be aware that debugging OptiX device code might not function as expected and is an ongoing development in the OptiX team.
What will work is Nsight Systems for application profiling and Nsight Compute for CUDA/OptiX kernel profiling.
Depending on the GPU you’re using, you might need specific versions of these, e.g. Nsight Compute 2020 dropped support for Pascal GPUs.
Check the resp. release notes before setting up a development system.
You can always use CUDA’s printf() inside OptiX kernels. It’s useful to limit output to a single launch index. Careful when using multi-GPUs and workload distribution, window coordinates are not launch coordinates then. In single GPU mind that the launch index [0, 0] is normally the left bottom corner of the image.
Please see these threads for more information on that. YMMV.
The first link is one of the three sticky posts in this sub-forum and already a little outdated.
OptiX debugging and profiling tools
https://forums.developer.nvidia.com/t/how-can-i-debug-the-cu-shaders-of-optix-in-linux/123514/3
https://forums.developer.nvidia.com/t/compile-error-in-debug/158255/2
https://forums.developer.nvidia.com/t/optix-compile-debug-level-full-and-optix-compile-debug-level-full-incompatible-with-new-drivers/156467