Error "Invalid Traversable", even with pipeline and stack options

I want to mix primitives of differents types, so I created two GAS of custom primitives first to test and I merged them into an IAS.


  • pipelineCompileOptions.traversableGraphFlags = OPTIX_TRAVERSABLE_GRAPH_FLAG_ALLOW_ANY (also tried with single instance)
  • optixPipelineSetStackSize with maxTraversableDepth = 2

What other thing can causes this error ? I have no problem with the examples of OptiX, like hairs.

The code that merged the two GAS is that :

    OptixAccelBuildOptions getDefaultBuildOptions()
        OptixAccelBuildOptions options = {};
        options.operation = OPTIX_BUILD_OPERATION_BUILD; // Créer la structure et pas l'actualiser
        options.motionOptions.numKeys = 0; // Pas de motion blur
        return options;

    OptixInstance getDefaultInstance(OptixTraversableHandle traversableHandle)
        OptixInstance instance = {};
        instance.instanceId = 0;
        instance.visibilityMask = 0xff;
        instance.sbtOffset = 0;
        instance.flags = OPTIX_INSTANCE_FLAG_NONE;
        instance.traversableHandle = traversableHandle;
        return instance;

    OptixBuildInput createBuildInputIAS(const managed_device_ptr& d_instance, unsigned int numInstances)
        OptixBuildInput buildInput = {};

        buildInput.type = OPTIX_BUILD_INPUT_TYPE_INSTANCES;
        buildInput.instanceArray.instances = d_instance;
        buildInput.instanceArray.numInstances = numInstances;

        return buildInput;

    struct AccelBuildBuffers {
        managed_device_ptr outputBuffer;
        managed_device_ptr tempBuffer;

    AccelBuildBuffers accelComputeMemoryUsage(
        OptixDeviceContext context, const OptixAccelBuildOptions* options, const OptixBuildInput* input, unsigned int numInputs)
        OptixAccelBufferSizes bufferSizes = {};


        AccelBuildBuffers buffers;
        buffers.tempBuffer = managed_device_ptr(bufferSizes.tempSizeInBytes);
        buffers.outputBuffer = managed_device_ptr(bufferSizes.outputSizeInBytes);
        return buffers;

    void accelBuildInstanceArray(
        OptixDeviceContext context, CUstream stream, const OptixAccelBuildOptions* accelOptions,
        const OptixBuildInput* buildInputs, unsigned int numBuildInputs,
        AccelBuildBuffers& buffers, OptixTraversableHandle* outputHandle)
            context, stream,
            buildInputs, numBuildInputs,
            buffers.tempBuffer, buffers.tempBuffer.size(),
            buffers.outputBuffer, buffers.outputBuffer.size(),
            nullptr, 0 // Emitted properties, not used

TraversableHandleStorage mergeGASIntoIAS(OptixDeviceContext context, CUstream stream,
    OptixTraversableHandle geometry1, OptixTraversableHandle geometry2)
    const OptixAccelBuildOptions buildOptions = getDefaultBuildOptions();

    const unsigned int numGeometries = 2;
    const unsigned int numBuildInputs = 1;
    const OptixInstance instances[numGeometries] = {

    const managed_device_ptr d_instances(instances, sizeof(OptixInstance) * numGeometries);
    OptixBuildInput buildInput = createBuildInputIAS(d_instances, numGeometries);
    AccelBuildBuffers buffers = accelComputeMemoryUsage(context, &buildOptions, &buildInput, numBuildInputs);

    TraversableHandleStorage accelerationStructure;
    accelBuildInstanceArray(context, stream, &buildOptions, &buildInput, numBuildInputs, buffers, &accelerationStructure.handle);

    return accelerationStructure;

Each of the GAS (geometry1 and geometry2) work.

Hard to say without the complete source code, for example, which traversable handle you used inside your launch parameters, etc.

Please also have a look at the other OptiX 7 SDK examples which use different geometric primitive types in two GAS and put them under an IAS.
You’ll find them by searching the source code for examples which use build input types OPTIX_BUILD_INPUT_TYPE_INSTANCES and OPTIX_BUILD_INPUT_TYPE_CUSTOM_PRIMITIVES and either OPTIX_BUILD_INPUT_TYPE_TRIANGLES or OPTIX_BUILD_INPUT_TYPE_CURVES.
That should be the examples: optixCutouts, optixHair, optixSimpleMotionBlur, optixVolumeViewer.

Since these work, you should check each and every OptiX structure parameter for differences in your complete code.

One incorrect thing would be that your OptixAccelBuildOptions buildFlags contains OPTIX_BUILD_FLAG_ALLOW_RANDOM_VERTEX_ACCESS which is only used for built-in triangles and curve primitives, not for custom primitives or instances.

Your getDefaultInstance() function is always setting the instance.sbtOffset to zero.
That will not work when the GAS contain different geometric primitives because the hit record assigned to that SBT entry consists of intersection, closesthit and anyhit programs, and different geometric primitives normally use different intersection programs, so they cannot use the same SBT hit record (unless the intersection program is handling multiple custom primitive types which is not recommended for performance reasons).

If your render graph structure consists of only the top-level IAS with multiple GAS children, I would recommend setting the OptixPipelineCompileOptions traversableGraphFlags to OPTIX_TRAVERSABLE_GRAPH_FLAG_ALLOW_SINGLE_LEVEL_INSTANCING.

I would also explicitly set the OptixPipelineCompileOptions usesPrimitiveTypeFlags to the primitives actually used inside the scene. While the default 0 means triangles and custom primtives, curves must be explicitly enabled.

To get more information, set a logger callback with the highest level 4 inside the OptixDeviceContextOptions and enable the validation mode in there as well in debug targets.

Ok, nice I’ve found it.
Was not related to OptiX but to C++ stuff.

If you look at the code I have sent, buffers.outputBuffer was destroyed at the end of the function mergeGASIntoIAS because not saved anywhere. So the output buffer was empty.

I just added some accelerationStructure.d_output = std::move(buffers.outputBuffer).
The managed_device_ptr is a class I created to facilite the use initially, not to search errors during hours, but now its ok, thank you.