Is your anyhit program calling optixIgnoreIntersection()?
If not, the observed behavior is reasonable.
The ray intersects with one of the spheres, changed the ray tmax value and that’s it. There is no closer intersection possible. Which AABB it picked would be depending on the BVH traversal.
A single ray cannot distinguish between two coplanar surfaces.
If you want to count something with anyhit programs, please have a look through the threads linked in this post:
https://forums.developer.nvidia.com/t/distinguish-objects-when-closest-hit-program-occured/73988/5
Esp. look for the OPTIX_GEOMETRY_FLAG_REQUIRE_SINGLE_ANYHIT_CALL
The OptiX built-in sphere intersection program can intersect front and back faces.
That is reported via the ray tmax and an additional intersection attribute register (see the unused optixGetAttribute_0 inside the optixSphere example code).
https://raytracing-docs.nvidia.com/optix7/guide/index.html#curves#spheres-and-the-hit-program
My second question is regarding the hitkind on spheres. It looks like the spheres’ hitkind is different depending on whether the ray hit from the inside (hitkind=139) or the outside (hitkind=138). I couldn’t find any documentation about this in the api and wanted to confirm here.
It also sets the hit kind to be able to distinguish which kind of primitive has been hit and on which side.
Use the OptiX device functions listed here to determine the hit primitive type and the face:
https://raytracing-docs.nvidia.com/optix7/guide/index.html#device_side_functions#intersection-information
Lastly, I imagine the intersection test for spheres (unlike triangles) is happening on the SMs rather than on the RTCores?
Yes, the built-in intersection programs you get via optixBuiltinISModuleGet are running on the SMs .