Am I correct in understanding that for custom primitives constructed through AABBs, an IS program will be called if a ray is originated within the AABB, no matter what the RayMax value passed through optixTrace is and no matter what tMax value is passed through optixReportIntersection?
an IS program will be called if a ray is originated within the AABB, no matter what the RayMax value passed through optixTrace
Not quite. The special case is t_min == t_max. That is an instant miss.
Otherwise, yes, any AABB the start point of the ray interval (ray.origin + ray.direction * t_min) lies in must be considered or raytracing wouldn’t work. That is not limited to custom primitives.
But mind that visiting AABBs along a ray interval is not happening in ray direction order but in BVH traversal order. That affects intersection and anyhit programs.
yeah i can see why this makes sense from a rendering perspective.