The above discussion describes that the __intersection__ is called every time a ray hits one of the boxes (AABB) we provided enclosing custom primitive. When is the __intersection__ called for triangles and spheres? Is the time a ray reaches a leaf AABB node?
In addition, I know a leaf AABB node can contain multiple triangles for high performance. As for spheres, does a leaf AABB node contain multiple spheres?
When is the __intersection__ called for triangles and spheres? Is the time a ray reaches a leaf AABB node?
Yes, the intersection program is called whenever an AABB around a geometric primitive is hit within the [tmin, tmax] interval of the current ray during the BVH traversal.
Because the intersection program is usually calculating the closest hit intersection which is shortening that interval by changing the ray tmax value, that will automatically result in finding the closest intersection.
In addition, I know a leaf AABB node can contain multiple triangles for high performance. As for spheres, does a leaf AABB node contain multiple spheres?
You shouldn’t be concerned about implementation dependent information like that. The implementation of the BVH is intentionally completely abstracted from the user to be able to have different implementations without affecting the API. You cannot rely on any of that information because that is GPU specific and could change among display driver versions.
All methods you have to influence size and speed of the acceleration structures themselves are the OptixGeometryFlags. Please read this thread: https://forums.developer.nvidia.com/t/some-questions-about-ray/250279/2