OptiX 5.1 does not support rtTrace in callable programs.
That is only available since OptiX 6.0.0, as said at the beginning of the OptiX 6.5.0 programming guide chapter I cited first.
This should work in OptiX 6.5.0 as well when done right. That’s the latest available OptiX SDK for the old API.
This problem would not appear in OptiX 7 because the host and device API are completely different.
There you would be using the so called continuation callables which have been added for that purpose.
Still, it’s always better for performance if you can avoid using trace calls in callables. If you can calculate the necessary data you require for the next trace call in a direct callable program and then trace the ray afterwards, outside the callable, the performance is expected to be better. Or if you can inline everything and call the trace in the outermost scope, that would be even better.
Porting to the OptiX 7 API requires a rewrite of the host application and some systematic adjustments to the device programs.
I’m not using any OptiX version before 7.0.0 anymore and for many reasons. The major two are API performance and flexibility.
Follow the links in these threads for some explanations of the differences in the new OptiX 7 API:
https://forums.developer.nvidia.com/t/transitioning-from-optix-6-5-to-optix-7/147116/2
https://forums.developer.nvidia.com/t/optix-7-breaking-changes/156801/2
Just for reference, this is the OptiX 7 version of the same direct callables for the lens shaders and the call to it.
All callables are stored inside the Shader Binding Table (SBT) record in OptiX 7, so there is no buffer of bindless callable program IDs anymore.