We’re trying to use the new visibility mask API to selectively rtTrace specific scene groups but it does not seem to have an effect.
To reproduce the issue, we modified the optixSphere project from the Optix 6 samples:
optixSphere.cpp line 243:
+ RT_CHECK_ERROR( rtGeometryGroupSetVisibilityMask(geometrygroup, 2) );
pinhole_camera.cu line 72:
~ rtTrace(top_object, ray, prd, 4);
So the group that contains the sphere instance has visibility mask 2 and we trace visibility mask 4.
With this change, I expected the sphere not to be intersected since there’s no overlap between the two bitmasks, but it’s still rendered.
Is there something wrong with the way we use visibility masks?