OPTIX_RAY_FLAG_TERMINATE_ON_FIRST_HIT is set in optixTrace function
For this reason, I call the optixTrace function recursively by terminating the ray at the first hit and updating the ray origin and direction.
That won’t work that way with OPTIX_RAY_FLAG_TERMINATE_ON_FIRST_HIT set.
Instead that calls into the closest hit program with the first hit, not necessarily the closest hit!
https://raytracing-docs.nvidia.com/optix7/guide/index.html#device_side_functions#12136
Which one that is, depends on the BVH traversal which depends on the ray origin and direction. (Means it’s view dependent for camera rays.)
That flag is normally used for a faster visibility tests than with anyhit programs and optixTerminateRay.
For such a visibility test you would not need an anyhit or a closesthit program at all. That can be done with just a miss program which indicates nothing was in in the way of the visibility ray [t_min, t_max] range.
If you always want the closest hit inside the closesthit program, please try removing the OPTIX_RAY_FLAG_TERMINATE_ON_FIRST_HIT from the optixTrace call.
When possible it’s also recommended to use iterative over recursive ray tracing. That will save device side stack space and perform better.
Reaching the closesthit program is the end of the current ray anyway. If you want to continue a single path with another single ray from there, you can also return to the ray generation program and set up the next path segment ray there.
Follow this link and the links in there for further threads about processing all intersections along a ray:
https://forums.developer.nvidia.com/t/processing-intersections-in-order/74859/2