Methods how to gather all hits along a ray have been discussed multiple times on this forum and the links I provided point to some of these discussions already.
If you want to gather all hits along a ray inside anyhit program invocations, you would need to call optixIgnoreIntersection inside it to keep traversing the acceleration structure until there is no more intersection test and the miss program is reached.
Note that optixIgnoreIntersection immediately ends the anyhit program, so all per ray payload values must be set before that. (Similar for optixTerminateRay.)
There are multiple issues with that method:
If you implement any counting algorithm inside anyhit programs, you must make sure that each geometric primitive invokes the anyhit program only once, which is not the default because OptiX is splitting geometric primitives when that results in a better AS and that needs to be explicitly disabled by setting the OPTIX_GEOMETRY_FLAG_REQUIRE_SINGLE_ANYHIT_CALL flag.
Another issue with that is that you get a varying number of anyhit invocations depending on what the ray hits and the hits are not ordered along the ray direction. Means you would need to have some per ray memory available which is big enough to hold all possible results.
OptiX doesn’t support dynamic memory allocations inside device code. You could allocate some local buffer inside the ray generation program which is big enough to hold the maximum number of possible anyhit invocations and store some sphere ID inside that.
The worst case would be all spheres in your scene, so that might get unmanageable with a huge number of primitives.
Another method which is iterative and orders the hits along the ray automatically would be using closest hit programs and continuation rays is described here, (also reached when reading the links I posted above):
https://forums.developer.nvidia.com/t/radiation-physics-problems/42281/2
The problem there is to make sure to not miss any coplanar hits by using some robust self-intersection avoidance mechanism.
If you only need to flag which spheres are hit at all, you could also write some value to a per-sphere primitive sized global buffer, but that wouldn’t be per ray.
If you need that per ray, that would need more data and methods for that have also been described here:
https://forums.developer.nvidia.com/t/whats-your-solution-to-get-all-hit-primitives-of-multiple-rays/239528
Maybe describe exactly what algorithm you need to implement instead of the details.