The normal vector of the triangle is normalized (unit vector).
But what does it mean to normalize based on triangle size?
That was just describing the case what would happen when the face normal was not normalized inadvertently.
Then the ray origin offsets along the unnormalized face normal would have been differently far away from the triangle surface depending on the triangle size. If your face normals are normalized, all is fine.
I have a triangle mesh whose offset is not behaving as expected (causing self-intersection).
Again, these offsets used for self-intersection avoidance are scene size dependent and might need to be tuned to your scene coordinate range. If they are not behaving as you would expect, then they are either not the appropriate value for your scene, or there is something incorrect inside your algorithm.
As I pointed out before, there is a more robust implementation for that inside the OptiX Toolkit:
https://forums.developer.nvidia.com/t/solving-self-intersection-artifacts-in-optix/267805/2
If you’re confused about what happens when inside the OptiX program domains, please read these:
https://forums.developer.nvidia.com/t/relationship-of-nvidia-optix-7-programs/251690
https://forums.developer.nvidia.com/t/some-questions-about-ray/250279
Finally, please just implement the visibility ray method I described multiple times. It’s exactly the algorithm you need and will be faster than any of your closest hit experiments so far.
Mind that the visibility test between the offset ray origin and end points along the face normal direction above the two tested triangle sample points would actually return true for two adjacent coplanar triangles!
Not sure if you want that. If not, you would need to filter out these results by checking if the two triangles are coplanar when the visibility test succeeded.