Thanks for the report.
There is at least one known problem inside the currently released drivers which could result in that behavior with debug device code generation (nvcc option -G) and OptiX-IR module input.
That should not happen without the -G option, so only using -lineinfo should work, but then the debugLevel might complain when using OPTIX_COMPILE_DEBUG_LEVEL_FULL.
It should be solved inside one of the upcoming R560 driver releases. (I don’t know the exact version number containing the fix at this time.)
I cannot easily try a Debug build with the Optimization settings (i.e. C++ Debug, but OptiX optimized) due to our cmake setup.
Depending on how you’re setting the nvcc command line options for your OptiX device code translation, that shouldn’t be impossible.
I’m doing that all the time inside my OptiX examples.
I either use custom build rules for each OptiX *.cu file where I can hardcode the command line parameters to be the same for all build targets, where I usually never use debug device code option -G because that is super slow.
https://github.com/NVIDIA/OptiX_Apps/blob/master/apps/rtigo12/CMakeLists.txt#L210
Using this CMake macro:
https://github.com/NVIDIA/OptiX_Apps/blob/master/3rdparty/CMake/nvcuda_compile_module.cmake
Or I use standalone CMake projects with the CMake native LANGUAGES CUDA feature where the OptiX device code is built as a CMake “Object Library” which is just translating *.cu files to PTX or OptiX-IR and can have its own nvcc command line options, even per build target in a multi-target build system (MSVS), shown here:
https://github.com/NVIDIA/OptiX_Apps/blob/master/apps/GLTF_renderer/CMakeLists.txt#L269
Also see links in this post for these CMake setup explanations for OptiX application frameworks.