The OptiX anyhit programs are invoked after the intersection program to determine if a potential intersection is accepted or ignored.
Intersection and anyhit programs are not called in ray direction order but in acceleration traversal order.
Each time an intersection is accepted which has a ray tmax value (intersection distance) which is less than the current ray tmax value, the smaller (closer) tmax is used for further intersection tests.
Means the ray intersection test interval [tmin, tmax] gets smaller with each accepted potential intersection until the closest intersection is found (or the anyhit program calls optixTerminateRay which ends the traversal and calls into the closesthit program (when present) with the last current tmax value.)
Now in your geometry setup, all three spheres cover the origin at (0, 0, 0) and shooting a ray from that origin into the direction (0, -1, 0) with a tmin == 0.0f and tmax big enough to leave all three spheres will find the closest intersection with any of the three spheres.
But depending on what sphere is tested first, which again depends on the internal traversal order through the bounding volume hierarchy inside the acceleration structure, the first hit sphere would shrink the tmax and the smaller intersection test interval [tmin, tmax] wouldn’t reach some sphere surfaces anymore which are farther away than that first or second hit.
Please read the OptiX Programming Guide here:
https://raytracing-docs.nvidia.com/
and also look for optixIgnoreIntersection and optixTerminateRay device functions which are available in anyhit programs.
and these related topics on OptiX’ built-in sphere primitives and intersections in general:
https://forums.developer.nvidia.com/t/sphere-intersection/272447
https://forums.developer.nvidia.com/t/does-there-update-ray-information-at-next-intersection-point-after-call-optixignoreintersection-in-any-hit/120040/2
https://forums.developer.nvidia.com/t/relationship-of-nvidia-optix-7-programs/251690
https://forums.developer.nvidia.com/t/when-is-the-intersection-shader-called-for-various-primitive-types/260029/2
https://forums.developer.nvidia.com/t/some-questions-about-ray/250279/2
If you need to find all intersections along a ray, please look though these forum search results:
https://forums.developer.nvidia.com/search?q=intersection%20%23visualization%3Aoptix