To iterate on this a little more in case others are interested:
Another more elegant solution to handle coplanar faces without the need to change the original scene would be to use two different ray types. One ray type would only handle front faces, the other ray type only back faces.
That needs to be detected inside the anyhit programs to be able to ignore the other condition, and the closest hit program would just return the hit distance, material properties, and maybe some attributes inside the per ray payload.
Then shooting two rays with these two ray types inside the ray generation program would be enough to detect and handle coplanar faces and nested materials.
Four resulting cases to be handled inside the ray generation program:
1.) If the two nearest hits are in an epsilon environment around the resulting intersection distance, then that would be the coplanar case and could be handled as single boundary between volumes. In case of a transmission to the adjacent volume, the material information on the top of the material stack would need to be exchanged against the new material.
Note that this also means that the simple scene epsilon method to prevent self-intersections would be applicable.
2.) If the nearest hit is on a front face, handle that as usual and if there is a transmission, push the entered volume’s material information onto the material stack.
3.) If the nearest hit is a on a back face, that must be from the current volume. In case of a transmission event, pop the current material from the material stack.
4.) Nothing got hit. If that was while shooting a volume scattering ray with limited distance, progress to the next volume sample point and calculate the next volume scattering direction and distance. The materials stack remains unchanged.
If that was while outside the volume, no object was hit. End of path.
Depending on additional information known about the scene setup (e.g. no nested materials), the above can also be simplified a little.