When is the
__intersection__called for triangles and spheres? Is the time a ray reaches a leaf AABB node?
Yes, the intersection program is called whenever an AABB around a geometric primitive is hit within the [tmin, tmax] interval of the current ray during the BVH traversal.
Because the intersection program is usually calculating the closest hit intersection which is shortening that interval by changing the ray tmax value, that will automatically result in finding the closest intersection.
Please read this thread about when which program domain is called:
https://forums.developer.nvidia.com/t/relationship-of-nvidia-optix-7-programs/251690/2
Some related topic about intersections with spheres:
https://forums.developer.nvidia.com/t/optix-7-spheres/259057
In addition, I know a leaf AABB node can contain multiple triangles for high performance. As for spheres, does a leaf AABB node contain multiple spheres?
You shouldn’t be concerned about implementation dependent information like that. The implementation of the BVH is intentionally completely abstracted from the user to be able to have different implementations without affecting the API. You cannot rely on any of that information because that is GPU specific and could change among display driver versions.
All methods you have to influence size and speed of the acceleration structures themselves are the OptixGeometryFlags. Please read this thread: https://forums.developer.nvidia.com/t/some-questions-about-ray/250279/2