how can optix solve the problem of z-fighting?


my quesiton:

how can I solve the z-fighting with optix?


This is mainly a problem of the model data. Means if your geometry is that bad, there is no solution to this when rendering individual fragments in a single pass.

This could be solved per primitive or per object using multiple passes with partitioned geometry.
You don’t really want that in a ray tracer though.

In a rasterizer you could sort your geometry up front and decide which whole primitive renders in front by blocking the fragments of those primitives inside the stencil buffer with some object ID. That allows to check both the stencil and depth test results in a second render pass rendering the other set of primitives to decide if a fragment with similar depth should be rendered or not.

This is not generally applicable. It won’t work for transparency for example.

In a ray tracer which has the whole scene geometry to render from inside the acceleration structures at the same time, this wouldn’t work. You’re not rendering whole primitives there, but individual intersections. Presorting and splitting base and decal polygons into different passes would break reflections for example.

You could try to make your intersection routine as precise as possible to get more depth resp. intersection distance precision, but if your geometry has overlapping geometry at the same coordinates, that is a problem of your model data.

The best solution would be to fix the geometry to not have coplanar primitives with different materials.

And now for something completely different, just in case you’re mixing the artifacts.
(None of the below is solving depth fighting due to bad geometry.)

In a ray tracer which renders shadows there can be similar effects like depth-fighting due to self intersections, means rays hit the same (planar) primitive they started from and assume they are in shadow although they shouldn’t.
The “shadow acne” artifact normally looks like moire rings on planar surfaces. You can force that artifact easily when setting the scene_epsilon in some of the OptiX SDK examples to zero.
(This can also happen on low tessellated geometry at grazing angles to the light, but that can only really be fixed with better geometry.)

What you would need to do there is implement some self-intersection avoidance mechanism.
The simple (faster) but not very robust way is to use a small epsilon to start your ray t_min value with so that there is a small offset for the ray’s intersection test from the starting position on the primitive. Such an epsilon value is very scene size dependent and needs to be manually adjusted!

Another way would be to offset the ray origin a little from the primitive surface in orthogonal direction to the surface on the hemisphere the ray direction points to.

Both these methods have the potential to miss geometry, esp. in the coplanar geometry case responsible for depth fighting.

A robust method to avoid self intersection is to use unique primitive IDs and make sure that the primitive you start your ray from is not the same one the ray hits. This requires to have a user defined unique ID per triangle for example. (I’m using indexed triangles with uint4 indices and the .w component is the triangle ID.)

Distinguishing unique primitive IDs gets a little more complicated when using instances (identical geometry under different transforms) since the primitive IDs are per geometry not per instance and there is no way in OptiX to identify an instance easily. But for self intersection avoidance it’s not actually required to identify an instance, it’s enough to figure out if it’s a different one than applied to the primitive you started from.
This “is different” condition can be determined with rtTransformPoint() calls on some default ortho-onormal basis. If the result of this check done in the anyhit program is different, it’s a different transform and even if the primitive ID matches you would handle the hit.

For better performance this can be combined with the scene epsilon method to avoid hitting the starting primitive which normally happens with t_min == 0.0f.

I posted this in the wrong forum yesterday… Posting the link here as an example of me stumbling upon what appears to be z fighting according to the 2 posts above.

(EDIT: Moved to OptiX forum and updated link.)

I’ll move your thread to this OptiX forum and adjust the link afterwards.

That is not depth fighting though, that effect is from self-intersections of the shadow ray hitting the primitive it started from.

That is simply related to the floating point precision. The bigger your scene the more inaccurate get the intersection distances. The simple scene epsilon method to avoid self-intersections is scene size dependent as the name implies.

There are more robust methods to avoid self-intersections and to increase the accuracy of hit point position, which are all more expensive.
E.g. maintaining a unique primitive ID which can be ignored when it’s the same as the one the ray started from.
The Raytracing Gems Book contains an article about self-intersection avoidance.

For far away hit positions it can be more precise to interpolate the vertex positions and transform to the world position instead of the usual ray.origin + ray.direction * intersectionDistance calculation.

None of this will solve real “z-bleeding” effects from coplanar faces with a single ray though.
See this thread: