OptiX doesn’t support virtual function pointers, just as the error message says.
See these related threads.
https://forums.developer.nvidia.com/t/virtual-methods-problem/32386
https://forums.developer.nvidia.com/t/failed-to-translate-ptx-input-to-llvm/71806
In OptiX 6 and before, function tables would need to be implemented manually with buffers of bindless callable program IDs
In OptiX 7 that would need to be done with direct callables or continuation callables inside the shader binding table (SBT).
Examples for OptiX 7 direct callable programs can be found here:
https://forums.developer.nvidia.com/t/optix-advanced-samples-on-github/48410