Does Optix update ray information like tmax from ray position after calling optixIgnoreInsections?
No, optixIgnoreIntersection() will immediately end the anyhit program and the tmax of that potential intersection will be ignored.
https://raytracing-docs.nvidia.com/optix7/guide/index.html#device_side_functions#terminating-or-ignoring-traversal
For example, in 1D, ray position is 0, and ray will intersect something at 2 and 4, so I will get 2 after calling optixGetRayTmax at first intersection in any hit program ?
No, the anyhit program invocation order on a ray [tmin, tmax] interval depends on the acceleration structure traversal order. The order is NOT along the intersection distance.
Means for your example you can reach the anyhit program for the object at 4 before the object at 2, or vice versa, depending on how the ray traverses through the BVH. That’s not under your control.
If you ignore one of the potential intersections, the other one will become the potential intersection if it lies inside the ray interval to be tested.
Seems like the OptiX 7 programming manual doesn’t describe this, but it behaves like this in OptiX 6 terms:
https://forums.developer.nvidia.com/t/when-the-rtpotentialintersection-t-return-true-is-t-the-any-hit-or-must-closest-hit/73862/4
Note that OptiX 7 allows to control the invocation of the anyhit programs with some of the OptixInstanceFlags
https://raytracing-docs.nvidia.com/optix7/api/html/group__optix__types.html#gadefef6077edef8bb8f4008d27d784745
and the OptixRayFlags
https://raytracing-docs.nvidia.com/optix7/api/html/group__optix__types.html#ga50e4054620bfddb9b3e282d1a53e211b
That was not possible in OptiX 6.
And can I change the ray position and ray direction before I call the optixIgnoreInsections so that I can get 2 when the answer of last question is 4;
Not at all. You’re still inside a single optixTrace() call doing the traversal through the BVH.
The ray origin and direction are immutable during that.
(Only the tmax (and the current intersection attributes) can be changed inside a custom intersection program with optixReportIntersection(). That’s also not possible when using the hardware triangle intersection on RTX boards.)
In your example, if you ignored the potential intersection at 4 inside the anyhit program, the one at 2 will be the only remaining one if you don’t ignore that as well. Otherwise you’ll get a miss program invocation.
If you always need the closest intersection and there is no cutout opacity involved, you don’t need an anyhit program at all on that ray type.
If you need to find all intersections along a ray in order please read this post and follow all links in them:
https://forums.developer.nvidia.com/t/distinguish-objects-when-closest-hit-program-occured/73988/5