Yes, this not going to work like that without additional changes and there is no example inside the OptiX SDK, yet.
OptiX cannot always detect automatically if bindless callable programs along a hierarchy of calls contain an rtTrace call which would need additional internal instrumentation to be able to call rtTrace.
For that OptiX added a call site instrumentation which allows to tell OptiX which bindless callable program IDs are potentially calling which others.
This works automatically when holding a bindless callable program ID directly in an rtDeclareVariable.
It also works automatically when using buffers of bindless callable program IDs.
All other cases need additional call site instrumentation on device side and some host side configuration.
First, you need to use rtMarkedCallableProgramId instead of rtCallableProgramId when calling a bindless callable program ID with an rtTrace inside.
Please look into the optix_device.h headers for more information on rtMarkedCallableProgramId.
That rtMarkedCallableProgramId allows to define a call site via a constant string which can be used on the host side inside the newly added function rtProgramCallsiteSetPotentialCallees which allows to specify which bindless callable program IDs are potentially being called from specific rtMarkedCallableProgramId locations inside the device code.
This allows OptiX to instrument the hierarchy of calls with the necessary information to be able to call an rtTrace.
So your code should look something like this:
RT_PROGRAM void closest_hit_generic()
{
if (mat_id >= material_data.size()) return;
MaterialData mat_data = material_data[mat_id];
prd.result = ((rtMarkedCallableProgramId<float3()>)mat_data.radiance_program_id, "my_call_site")();
}
RT_CALLABLE_PROGRAM float3 closest_hit_radiance()
{
PerRayData_radiance new_prd;
new_prd.result = make_float3(0);
Ray refl_ray = make_Ray(make_float3(1, 0, 10), make_float3(0, 0, 1), RADIANCE_RAY, 1e-4, RT_DEFAULT_MAX);
rtTrace(top_object, refl_ray, new_prd);
return new_prd.result;
}
// On the host:
Program ch_generic = context->createProgramFromPTXString(ptx, "closest_hit_generic");
Program cp_radiance = context->createProgramFromPTXString(ptx, "closest_hit_radiance");
// Gather all bindless callable program IDs which can be called from "my_call_site":
std::vector<int> callees;
callees.push_back(cp_radiance->getId());
// Let OptiX know that these bindless callable program IDs can potentially be called from "my_call_site" inside the closest_hit_generic program object:
ch_generic->setCallsitePotentialCallees("my_call_site", callees);
That said, I would not use that mechanism when I can avoid it.
If you can make the bindless callable programs only calculate information which can be used after the return inside the closest hit program to do the necessary rtTrace with these information, that would speed up the bindless callable programs. My OptiX introduction examples do it this way.