I calculate Fresnel’s probability(for refraction or reflection) using the refractive indices of the components on both sides of the intersection surface. I can query the refractive index of one side from my mesh details. How do I keep track of the refractive index for the next component?
Please have a look at the material system and IOR handling inside the OptiX Introduction examples.
They also support nested materials with a small stack tracking the absorption and IOR of the volumes.
You need to keep track of the surrounding volume properties at the per ray payload.
Read this description of what the individual examples add. The images show the individual rendering features added.
At example #7 it matches the block diagram of the renderer architecture at the bottom of that README.md page.
In there check the raygeneration.cu for the MATERIAL_STACK and FLAG_VOLUME code blocks:
and the specular_reflection_transmission BSDF which calculates the effective eta depending on the surrounding volume proplerties: