The order in which the acceleration structure BVH is traversed is not along the ray direction which makes it view dependent.
If you use OPTIX_RAY_FLAG_TERMINATE_ON_FIRST_HIT it will report the very first hit, which is not necessarily the closest hit.
The same is happening for anyhit program invocations which are also not in ray direction order.
Means OPTIX_RAY_FLAG_TERMINATE_ON_FIRST_HIT should not be used if you require the closest hit!
It is meant to be used for a binary visibility decision, for example, as a faster alternative to an anyhit program which unconditionally calls optixTerminateRay as used for shadow rays on opaque objects.
Read these two threads explaining that as well:
https://forums.developer.nvidia.com/t/optix-7-2-ray-triangle-intersection-failure/178611/4
https://forums.developer.nvidia.com/t/anyhit-program-as-shadow-ray-with-optix-7-2/181312/2