Yes, tricky problems.
If you have coplanar faces shared by two geometries, the closest hit cannot be properly determined by the intersection and can change depending on the BVH traversal order.
Using a scene epsilon to continue the ray after each intersection will also skip one of the faces.
Thinking about possible solutions results in scary and complex ideas. To be tried another time.
I have a solution for nested materials and volume scattering in my MDL capable path tracer, but I do not handle the case of coplanar surfaces or partially intersecting objects.
It’s already a little involved to implement all tiny details of nested materials and maybe I’ll put out an OptiX advanced sample showing that in the future.
What you need is a small “material stack” which stores the volumetric parameters: absorption coefficients, IOR, volume scattering coefficients, phase function value, and the material index of the material the ray is currently in.
Being able to actually index materials and their programs is a feature which requires a whole specific rendering architecture I developed over time. The material is needed during volume in-scattering lighting calculation to determine if a hit is the same material the ray is currently in, and because the surface of the volume tints the incoming light if there is a transmission of the light ray at all.
Then you would need the IOR of the volume the ray is currently in, the IOR of the surrounding volume, and the IOR of the material surface you’re currently hitting.
Then a transparent material would need to determine the effective IOR from current volume IOR to surface IOR when entering the volume (hitting a frontface) or the surface IOR and surrounding IOR when leaving a volume (when hitting a backface, surface IOR and current IOR should be identical)
Absorption calculations need to be done with the last ray segment’s distance in the current volume material. That also handles the case of different nested materials, for example entering the water in a pool and then hitting the floor if the water surface is not modeled as closed volume.
Now if there was a transmission and you entered a volume you need to put the new volumetric parameters onto the material stack.
If you left the current volume you need to pop the current volume parameters from the material stack.
Volume scattering is another level of complexity.
If you have a path tracer, that can be done with a brute force random walk through the volume, with some limit on the number of steps. Each vertex of that path which missed some geometry, means which did a random step inside the volume, would need to calculate in-scattering light.
That needs even more attention to detail if you want to handle the case of lights and other materials inside the volume (fog with lights and volumetric shadows).
Topics to research that are the Henyey-Greenstein phase function, in-scattering, source term, Fresnel, and then there are other methods to approximate volume scattering, like dipoles, some of which require pre-processing of incoming light on the volume’s surface.
To find the linked GTC presentation use this portal:
In the search dialog select Year 2016 and search the results for the given topic.
Searching for it directly somehow broke.