I’ve been thinking about this a little. As a vague generalization, it’s always true that when precision is an issue, squares and square roots should be considered carefully. They certainly can affect precision. On the other hand, for very short rays, I would guess those things shouldn’t be compromising your precision. It is possible that your setup is just hitting the limits of fp32 in general, but I maybe don’t fully understand your plots.
One thing I did in a capsule intersector of my own to analyze precision is to build a complete path through the intersector using double precision, and use the exact same set of intermediate calculations. I added a pixel-clicking debug mode that turns on printf for a single ray, based on the pixel I click on, that will print the intermediate results of every calculation in single precision, double precision, and then also print out the relative error of the fp32 calculation as well as count the number of mantissa bits that match when I convert the double precision result to single precision (assuming the exponent matches). This is a heavy hammer, but shines a lot of light onto the precision of every line of code, and allows you to do A/B testing of different setups. In my case, I found that I can’t really blame the squares or square roots per se, the precision simply degrades when the determinant of the calculation (the part inside the square root) approaches zero. For a capsule, this can happen not just near the surface boundary, but other places in space too, far away from the surface.
There might be some scale-specific effects in your case, but as long as you’re not hitting any overflow or underflow conditions, in general it’s usually fair to scale the entire scene and expect the same precision behavior; only the exponents will change. So, here’s a datapoint you can compare to. If I set up a sphere with a radius of 6.4e6 (radius of the earth in meters) that touches the origin at (0,0,0) and place a camera at position (0,2,0) (approximate height of a tall adult human in meters) looking down at the origin, then I don’t see any ray misses. When I start lowering the camera toward the origin, I start seeing misses at around 1.6m, and then they increase dramatically around 0.6m, and anything less than that becomes extremely noisy - equal mix of hits and misses or worse. Things don’t change after that until the camera distance is less than 1e-19 and I start getting underflow in some of the terms. Anyway, this ratio of radius to camera distance where things begin to fail is suspiciously close to 2^23 (~8.3e6), and suggests the primary precision factor is the radius/rayLength ratio. I remember there being a pretty common rule of thumb in Z-buffer raster graphics to avoid having your scene size ever exceed 8 million of the smallest unit you care about, for similar reasons.
So worth noting that the above is basically saying that ray-sphere intersection precision can be thought of (to a first approximation) in terms of a fraction of the sphere’s radius, where you want the ray to originate at least 1/(2^23) * radius away from the sphere surface. Your plots don’t fit into this category at all, which makes me assume something else might be going on, or that I am making incorrect assumptions about what plots mean. It seems like your plots are showing errors that are in the neighborhood of 0.2 * radius, is that accurate to say? The reasons that could happen might be related to accumulated numerical errors along a multi-segment path, or maybe to accidental noise introduced in the scattering step? Another thing to maybe try is translating your ray & sphere with an offset inside your intersector to place the sphere or the ray or the approximate hit point at the origin. One of those might improve precision on top of the 3 methods I first mentioned.
That was a bit of a ramble, I hope you find something valuable in there. ;) Let me know if you want to dive a little deeper into the specifics of your setup.
–
David.