First of all, you’re not implementing what I described.
1.) Your tmax calculation is incorrect and not consistent with what you wrote:
You printed Direction vector length(= tmax) : 1613.840332
But in your code length is always 1.0f.
const float3 rayDirection = normalize(GetPos(target) - GetPos(origin));
const float tmax = length(rayDirection);
Compare that to my example code.
float3 vector = ray_end_point - ray_origin; // Unnormalized Vector from start to end sample point.
float ray_tmax = length(vector); // Distance between the two sample points in world coordinates. This will define ray.tmax.
// TODO: Add a check here to make sure that ray_tmax is greater than some epsilon to not divide by zero in the following normalization.
// This could actually happen for adjacent triangles!
float3 ray_direction = vector / ray_tmax; // Normalized direction vector from start to end sample point.
Maybe that was already the whole problem, but there are other things I would change:
2.) You wanted to offset the origin along the face normals of the two triangles, but you offset it along the ray direction, which should also work, but is less robust. When doing that you can also simply offset the tmin which would have been faster.
That epsilon offset is is dependent on the scene size. Depending on your scene extents your epsilon might not be big enough.
And again, if you used a visibility ray implementation like I explained multiple times now, it would be required to offset the ray at both ends of its test interval, so the same calculation would need to happen at the target triangle and only then the ray direction would need to be calculated between the offset points.
Or you do it along the ray direction by changing the tmin and tmax values given to optixTrace like in my example code.
3.) No closest hit is required if you followed my example code. That was disabling anyhit and closesthit program altogether because the visibility ray can be implemented with just a miss program as shown inside in my very first posted reply of this thread.
https://forums.developer.nvidia.com/t/anyhit-program-as-shadow-ray-with-optix-7-2/181312/2
extern "C" __global__ void __miss__shadow()
{
optixSetPayload_0(1); // isVisible != 0
}
The OPTIX_RAY_FLAG_TERMINATE_ON_FIRST_HIT is not going to result in the closest hit anyway. Instead that is terminating the ray traversal on any hit inside the ray’s tmin, tmax interval. If there is no hit, the miss program will be reached and that is where you set the isVisible flag.
The construction of the visibility ray already knows which two triangles are tested. There is no need to test against either of the two involved triangles inside some closest hit program.
(If you wanted to implement a more robust self intersection test by testing the hit primitives, that would have required an anyhit program to be able to ignore the two involved triangles whoch would slow down the algorithm considerably and shoudn’t be required. Don’t do that Try to get this to work as I explained first.)
4.) I’m not sure what your Figure 3 should be showing but although you’re saying that your geometry triangles are wound consistently (counter-clockwise is front face) to result in front faces being always outside, have you verified that, like by visualizing the face normals?
Because none of the ray offseting or face normal comparisons will work if the triangle winding is inconsistent.
5.) I would also try without the OPTIX_RAY_FLAG_CULL_BACK_FACING_TRIANGLES ray flag.
That wasn’t part of my example code either.
Please just implement the visibility ray exactly as I described and do the above consistency checks of your geometry. That should all work and is really rather simple if you just follow my advice.