I’m unable to reproduce this when changing all my direct callables to continuation callables in one of my OptiX 7 examples (intro_driver) from the above links.
Works without issues either way, just slower with continuation callables.
My system is Windows 10 20H2, RTX A6000, 471.41 display drivers, OptiX 7.3.0, CUDA 11.1, MSVS 2019, PTX code target SM 5.0 (Maxwell, is deprecated in CUDA 11) or SM 6.0 (Pascal) both worked.
I’m explicitly not using CUDA 11.3 because OptiX 7.3.0 is built with CUDA 11.1.
Means you could try the following steps to match that:
1.) Build your application with CUDA 11.1.
You can install as many CUDA toolkits side-by-side on a system as you want. I usually use the web-based installer and disable everything related to the display driver in there because that’s always out-of-date as soon as there are official display driver releases which support that CUDA version.
Make sure to set the CUDA_PATH to the version you want to use. That’s usually used to find the current CUDA toolkit version inside CMake scripts. If you rely on CUDA binaries to be in the PATH environment variable, make sure the order is the one you want.
2.) Update the display driver to 471.41 if you’re not running that already.
3.) If that still doesn’t work, please provide a complete and minimal reproducer project in failing state.
If you can make one of the existing OptiX 7.3.0 SDK examples fail the same way, that would be the easiest to analyze.
PS: If your optixTrace call above is not for a visibility ray but should find the closest hit, then do not set OPTIX_RAY_FLAG_TERMINATE_ON_FIRST_HIT. That will return the first hit during traversal, not the closest!