Unknown Value when trying to figure out pointer space...

I’m getting an error I can’t understand:

rtProgramCreateFromPTXString: Unknown error (Details: Function "RTresult _rtProgramCreateFromPTXString(RTcontext, const char*, const char*, RTprogram_api**)" caught exception: Compile Error: Unknown Value when trying to figure out pointer space for ray payload argument to rt_trace at: [ i64 %2 ])

The code in this case is my main integration function, which I’m trying to move into a separate RT_CALLABLE_PROGRAM to give me switchable integrators. The same code works fine when called as a regular function from my ray generation program, but moving it to a separate file and prefixing with RT_CALLABLE_PROGRAM gives me this error when compiling the PTX.

What exactly is the error message trying to tell me?

I have not seen that error message before. Looking at the code, the error message just describes the problem. The OptiX compiler could not figure out to what memory space (local, global) the per ray data argument belongs to. I can’t say why it happens with the given information.
How does the code look like which does the rtTrace()?

Are you trying to access the rtPayload variable inside a bindless callable program directly?
That wouldn’t work because bindless callable programs have only themselves and the context as variable scope.

You’re planning to use rtTrace() inside bindless callable programs?
While that feature was added in OptiX 6.0.0, it should really only be used if there is no other way. The additional resource management for these “continuation callables” are going to be more expensive than when using bindless callable programs to just calculate data and afterwards do the rtTrace() in the caller depending on the outcome.
Also depending on the way you manage your bindless callable program ID variables, the setup for these can get a little involved.
https://raytracing-docs.nvidia.com/optix_6_0/guide_6_0/index.html#programs#4635

Another way to switch between integrators would be multiple ray generation entry points. The first argument to rtContextLaunch() would switch between them instantly. Application startup behaviour will be different because each of the entry points will be compiled once on first usage. The OptiX shader cache will alleviate that after the first run.

Thanks for the reply Detlef. I’d missed that part of the docs (hadn’t realised this was only enabled in 6.0). The callable program takes the payload as an argument like so:

rtDeclareVariable(rtObject, scene_root, , );

RT_CALLABLE_PROGRAM void integrator(Ray& ray, PerRayData_radiance& prd, float time, float4& L) {
    float4 throughput = make_float4(1);
    L = make_float4(0);
    for (prd.depth = 0; prd.depth < 32; ++prd.depth) {
        prd.flags = RF_NONE;
        prd.radiance = make_float4(0);
        // trace the new ray
        rtTrace(scene_root, ray, time, prd);

        // accumulate any emissive radiance
        L += prd.radiance * throughput;
        //
        // etc...
        //

I’d wanted to do it like this so that I could separate the mechanics of the image sampling, AOV management etc from the integrator. What I’ve done previously is conditionaly compile in different integrators from headers using the preprocessor so I’ll probably just fall back on that if the performance of trying to trace from a callable program is poor.

How do you store the variable with the bindless callable program ID and how do you call it?

The OptiX host side cannot always determine that it’s a continuation call automatically.
The listings 4.43 and 4.44 in the link above show which cases work automatically, resp. which need manual instrumentation with
rtMarkedCallableProgramId on device side and rtProgramCallsiteSetPotentialCallees on host side.

did you succeed to solve this problem in your code?
if yes, can you explain?