Auto intersection: is there a clean way to avoid them

I made an optix prime project based on primeSimple, where I:

  1. I create a bunch of rays: laying on the centroid of triangle:
//Identify vertexs of the triangle
                        int3 tri = mesh.getVertexIndices()[i];
                        float3 v0 = mesh.getVertexData()[ tri.x ];
                        float3 v1 = mesh.getVertexData()[ tri.y ];
                        float3 v2 = mesh.getVertexData()[ tri.z ];
                        float3 centroid;
                        centroid.x = (v0.x+v1.x+v2.x)/3.0 ;
                        centroid.y = (v0.y+v1.y+v2.y)/3.0 ;
                        centroid.z = (v0.z+v1.z+v2.z)/3.0 ;

                        Ray r = { centroid, tolerance, rayDir, 1e34f };
  1. launch the hit process.
RTPquery query;
  CHK_PRIME( rtpQueryCreate( model, RTP_QUERY_TYPE_ANY, &query ) );
  CHK_PRIME( rtpQuerySetRays( query, raysDesc ) );
  CHK_PRIME( rtpQuerySetHits( query, hitsDesc ) );  
  CHK_PRIME( rtpQueryExecute( query, 0 /* hints */ ) );


tolerance = 0

meaning, the initial ray origin is actually on the triangle, I get about half my ray intersecting with the initial triangle. I want to avoid that

So in order to avoid that, I need to increase the tolerance to a value.
This does the trick for now.


n1) it will not work when mixing triangle of different size whose order of magnitude differs from about the floating precision (I don’t want to use double for performance purpose).

n2) It will also fail if I have triangle that are very close from each other. That makes sense.

n3) It will also be an issue if I use uniform sampling distribution on the triangle and that the angle between 2 adjacent triangles is >90 deg (or close)

So I was thinking that maybe there is a way to exclude a triangle from the intersection process.

I could consider doing that as an iterative post-processing, but I would need to move the origin of the ray anyway so that the ray does not perpetually stays there. This brings me back to this being an issue for n3) anyway

So my question is: Is there a way to exclude a triangle from the process when CHK_PRIME( rtpQueryExecute( query, 0 /* hints */ ) is called so “auto intersection” is excluded ?

As a side question, and if that is not possible in optiX prime, I assume that I can implement that in a void mesh_intersect( int primIdx ) program, for example in sphereTessellate/ . Please correct me if I am wrong. (or if there is a more straight forward way )

I don’t use OptiX Prime, but regarding your last question:

You could theoretically decide in the intersection program not to call rtPotentialIntersection() and thus ignore an intersection. However, you need to know the id of the triangle you want to exclude from the intersection process. If this is available via an rtVariable or a rtBuffer, it will work.

Alternatively, you could also call rtIgnoreIntersection() in the any hit program.


I also notice that the example primeMasking could actually be what I want. I will look into detail.

I have tried to use the masking to avoid auto-intersection. What I wanted to do is to create the triangles with a unique mask (triangle id starting from 1). However, from the documentation I understand that intersection test will be skipped if the AND operation between the ray mask and the triangle mask is non-zero. In case we use the triangle id as the triangle mask, I don’t see how we can create a ray mask that will work for a large number of triangles (more than the number of bits in the mask). If other operators would be available (XOR, EQ etc) it would have been possible in my opinion, but unfortunately there is no control on the choice of the operator in Optix Prime. It would be nice to be able to specify the operator (NAND, XOR etc) to the mask in order to have more flexibility.
Or maybe there is another way around this problem…

I’m curious to know what NVIDIA thinks of allowing the developer to set the choice of the mask operator.
It would give more flexibility, and should not affect performance.

I understand that with the current test (triangle masked if bitwise AND operator is non-zero) we can only create sizeof(int) triangle to be addressed uniquely (ray mask has one bit set, triangle mask has same bit set)

The advantage of the AND operator is that for a ray we can select multiple triangles to be masked (for example ray mask 0011
will mask triangles with masks 0001, 0010 and 0011).

But if we could also use the test “triangle masked if XOR non zero" we can address 2^sizeof(int) uniquely

I’m aware I could switch from Optix Prime to Optix to have more flexibility allowing to solve this issue, but would like to stick to Optix Prime.

Maybe such a functionality could be implemented in a future version?