Hello ,
I just started trying to learn optix programming.
I ran into a problem where some of the any hits that I thought were supposed to happen were not being triggered.
For example, I constructed three spheres with radius 2.f, and the centers are: {0 0 0}, {0 1 0}, {1 0 0}. ray_origin={0,0,0}, ray_direction={0. f,-1.f,0.f}. According to my understanding, the light should intersect three spheres and trigger any hit, but it only triggers any that intersects {0 1 0}, {1 0 0} hit, I want to know why any hit with {0 0 0} did not occur?

here is my code for raytrace:
optixTrace(
handle,
ray_origin,
ray_direction,
tmin,
tmax,
0.0f, // rayTime
OptixVisibilityMask(1),
OPTIX_RAY_FLAG_ENFORCE_ANYHIT,
0, // SBT offset
0, // SBT stride
0, // missSBTIndex
p0, p1, p2);

The OptiX anyhit programs are invoked after the intersection program to determine if a potential intersection is accepted or ignored.

Intersection and anyhit programs are not called in ray direction order but in acceleration traversal order.

Each time an intersection is accepted which has a ray tmax value (intersection distance) which is less than the current ray tmax value, the smaller (closer) tmax is used for further intersection tests.
Means the ray intersection test interval [tmin, tmax] gets smaller with each accepted potential intersection until the closest intersection is found (or the anyhit program calls optixTerminateRay which ends the traversal and calls into the closesthit program (when present) with the last current tmax value.)

Now in your geometry setup, all three spheres cover the origin at (0, 0, 0) and shooting a ray from that origin into the direction (0, -1, 0) with a tmin == 0.0f and tmax big enough to leave all three spheres will find the closest intersection with any of the three spheres.
But depending on what sphere is tested first, which again depends on the internal traversal order through the bounding volume hierarchy inside the acceleration structure, the first hit sphere would shrink the tmax and the smaller intersection test interval [tmin, tmax] wouldn’t reach some sphere surfaces anymore which are farther away than that first or second hit.

Please read the OptiX Programming Guide here: https://raytracing-docs.nvidia.com/
and also look for optixIgnoreIntersection and optixTerminateRay device functions which are available in anyhit programs.

Thank you very much for your prompt response.
I’m wondering if there’s a way to trigger any hit for all intersections of my rays with the three spheres mentioned above. I hope to be able to obtain the indices of all spheres intersected by the ray in a single ray launch.
My any hit program does not include optixIgnoreIntersection or optixTerminateRay. I just want to know which spheres are intersected by the ray.
here is my any hit code:
extern “C” global void anyhit()
{

Methods how to gather all hits along a ray have been discussed multiple times on this forum and the links I provided point to some of these discussions already.

If you want to gather all hits along a ray inside anyhit program invocations, you would need to call optixIgnoreIntersection inside it to keep traversing the acceleration structure until there is no more intersection test and the miss program is reached.
Note that optixIgnoreIntersection immediately ends the anyhit program, so all per ray payload values must be set before that. (Similar for optixTerminateRay.)

There are multiple issues with that method:

If you implement any counting algorithm inside anyhit programs, you must make sure that each geometric primitive invokes the anyhit program only once, which is not the default because OptiX is splitting geometric primitives when that results in a better AS and that needs to be explicitly disabled by setting the OPTIX_GEOMETRY_FLAG_REQUIRE_SINGLE_ANYHIT_CALL flag.

Another issue with that is that you get a varying number of anyhit invocations depending on what the ray hits and the hits are not ordered along the ray direction. Means you would need to have some per ray memory available which is big enough to hold all possible results.
OptiX doesn’t support dynamic memory allocations inside device code. You could allocate some local buffer inside the ray generation program which is big enough to hold the maximum number of possible anyhit invocations and store some sphere ID inside that.
The worst case would be all spheres in your scene, so that might get unmanageable with a huge number of primitives.

Another method which is iterative and orders the hits along the ray automatically would be using closest hit programs and continuation rays is described here, (also reached when reading the links I posted above): https://forums.developer.nvidia.com/t/radiation-physics-problems/42281/2
The problem there is to make sure to not miss any coplanar hits by using some robust self-intersection avoidance mechanism.