I don’t know of an example which does this and I consider it a bad idea.
Why exactly would you need that?
The error in your code was that you used the wrong traversable handle argument in optixGetInstanceChildFromHandle:
OptixTraversableHandle child1 = optixGetInstanceTraversableFromIAS(self.world, 1);
OptixTraversableHandle child = optixGetInstanceChildFromHandle(self.world); // BUG: This will return 0, and should have been child1 instead of self.world.
An optixTrace() call with traversableHandle == 0 argument will immediately call the miss program.
Again, it makes absolutely no sense inside an AS structure with only IAS->GAS for performance reasons because tracing directly against the GAS will be slower.
Here’s some example code how to get the necessary traversable handles in your case and comments about it:
{
// DEBUG
uint3 theLaunchIndex = optixGetLaunchIndex();
if (theLaunchIndex.x == 256 && theLaunchIndex.y == 256)
{
// 1.) This is a bad idea.
// Query which instance index we hit in some IAS.
// This is the index inside the bottom-most(!) IAS over the GAS.
// Means you wouldn't actually know what IAS you're at inside a deeper hierarchy, unless you stored the instance traversable handle for some unique path to a GAS to be able to look up its children.
// Actually the optixGetTransformListHandle() and optixGetTransformListSize() minus 1 and 2 can be used to query instances and IAS handles upwards.
// Another issue is that you could reuse GAS under different instances and with different materials.
// So if this is, for example, required to do sub-surface scattering only inside the GAS, then the material assignment cannot be at instance level.
// Also when adding scene lighting into the mix, you would need the current matrices at the GAS to be able to transform the ray back into the light geometry's world space again,
// but only if you're in that local GAS system, so that condition also needs to be tracked inside the per ray payload.
// In a IAS->GAS scene structure the IAS is always the topObject and the children are the GAS traversables.
// This is a value in the range [0, m_instances.size()-1] in this example.
const unsigned int index = optixGetInstanceIndex();
OptixTraversableHandle instance = optixGetInstanceTraversableFromIAS(sysParameter.topObject, index);
OptixTraversableHandle child = optixGetInstanceChildFromHandle(instance);
// 2.) What you actually want is probably just the GAS.
// This is identical to "child" if the structure is IAS->GAS, but then this whole trace against a lower level AS doesn't make any sense for performance reasons.
OptixTraversableHandle gas = optixGetGASTraversableHandle();
printf("index %d: instance = %llu, child = %llu, gas = %llu\n", index, instance, child, gas);
}
}
Then you would also need to store the two transform matrices onto the per ray payload to be able to transform the ray into the world space of the GAS inside the raygeneration program and then track that you’re not tracing against the top-level traversable handle, or use a different ray type to make sure you’re not recursing into that case again.
If this is about volume effects, you would need to be careful about scaling inside the instance matrices because that would change the world space inside which volume scattering and absorption coefficients are usually defined.
// Put this into the PRD somehow.
// You wouldn't need this if the instances all use the identity transform, which would mean the GAS data is actually in "world space".
float4 objectToWorld[3];
float4 worldToObject[3];
// These functions getting the current transform matrix and its inverse work for any transform hierarchy.
optix_impl::optixGetObjectToWorldTransformMatrix(objectToWorld[0], objectToWorld[1], objectToWorld[2]);
optix_impl::optixGetWorldToObjectTransformMatrix(worldToObject[0], worldToObject[1], worldToObject[2]);