Is it possible to choose some triangles in a BVH to test the intesection while ignore other triangles?

Hello everyone, here’s detailed description of my problem: Assuming I have RayGroup_1, RayGroup_2, and an acceleration structure with N triangles {T1, T2, ..., TN} (BVH Tree). I want to test the intesection of RayGroup_1 with {T1, T3, T5, ...}, and RayGroup_2 with {T2, T4, T6, ...}. Is there a way to do that? (Such as setting some tag and let the ray only hit the triangles with tag matched and ignore the others)


Yes, this can be done multiple ways.

It depends a little on how flexible you are with the construction of your acceleration structures.
Also I’m assuming a ray group is either a separate optixLaunch call or an optixTrace call inside the ray generation which has different arguments, either for the SBT offset or payloads.

Let’s go about this from fast to slow:

1.) If you know which triangles need to be intersected by each of your ray groups, then the fastest method to ignore the other ones would be to not actually put them into the geometry acceleration structure (GAS).
Means you could build two GAS, one with your triangles T1, T3, T5, … and the other GAS with triangles T2 ,T4 ,T6, … (See OPTIX_TRAVERSABLE_GRAPH_FLAG_ALLOW_SINGLE_GAS inside the OptiX Programming Guide.)
That would provide you with two traversable handles, let’s call them A and B, and each group of rays would use either the traversable handle A or B as argument in optixTrace to start the traversable
Means the “ignored” triangles are not even in the scene for that group of rays.
That should be the fastest possible implementation.

2.) If you need all triangles in the scene for other use cases, you could also use the same partitioning of triangles into GAS A and B, but put them under two instances of a top-level instance acceleration structure (IAS). (See OPTIX_TRAVERSABLE_GRAPH_FLAG_ALLOW_SINGLE_LEVEL_INSTANCING inside the OptiX Programming Guide).
Each instance and the optixTrace call have an 8 bit visibilityMask which is tested with the logic operation AND and if it’s not zero, the ray will traverse the instance. Means if you put a visibilityMask 1 to the instance holding the GAS A and a visibilityMask 2 to the instance holding the GAS B, you can select with the same visibilityMask value inside the optixTrace, using the IAS traversable handle as start, which instance to traverse, which is again ignoring the other triangles.
(Also in case you need to shoot other type of rays which can intersect both sets of triangles, you can simply set the optixTrace visibilityMask to 3 which covers both bits.)
This would be almost as fast as 1. on RTX GPUs because the single level instancing (IAS->GAS scene hierarchy) and the visibilityMask are all evaluated in hardware.

3.) Slow method:
You put all triangles inside a single GAS. Then each triangle is uniquely identified by its primitive index inside the GAS which is available via the optixGetPrimitiveIndex device function (See
With that you can assign any additional data to a triangle by accessing some array with that primitive index.
In this case for example a byte per triangle which tells if the triangle belongs to one or the other group of rays.
(Actually if your triangles are really partitioned into even and odd triangles then the primitive index would be enough to distinguish these.)
Now to actually ignore a triangle you would need to implement an anyhit program which compares some information on the per ray payload (optixGetPayload_*() functions) with the per triangle information and call optixIgnoreIntersection when the triangle should be ignored, which lets the ray traversal continue.
This is slower because the hardware BVH traversal on RTX boards will be interrupted for each intersected triangle and the anyhit program will be called to determine if the hit is valid or not.

4.) A variation of 3. would be to use two SBT entries and the sbtIndexOffsetBuffer to define two different SBT hit records (basically two different “materials” assigned to different triangles in a single GAS) which would then have an anyhit program which can reject rays of the other group. The difference here is that the two anyhit programs would know (hardcoded) which ray group they need to ignore instead of reading the additional per triangle data.

Thanks! My case needs all triangles in the scene, so I will first try the second method, that helps me a lot!

Hello, happy new year! I have an extra question: if I use method 2, is that mean I can only setup 8 different group given the mask is 8 bit? If I have say 256 (2^8) groups, is that means I can only use method 3 or 4?

Happy New Year!

Yes, the AND operation between the visibility masks of the instance and ray is only allowing to implement a binary condition like in a DIP switch, not a counter.

Just give the anyhit program method a try. Maybe it’s not as bad as expected.

If you need to identify one of many triangle partitions of your scene geometry, you could still use method 1 and traverse individual GAS with their own traversable handle you would store in buffer which pointer you put into the launch parameters. Then you would have a vector of GAS traversable handles you could index with whatever method you need. That would not be limited to 256 partitions either.
In case rays need to be able to intersect all primitives as well, you could put the GAS under an instance with identity transform (primitives would be in world space) into a top level IAS and traverse that.
So it would be possible to intersect one of many partitions of triangles and all primitives in the scene.
If you need subsets, you could use different IAS which reference only the necessary GAS, or set visibility masks again. That requires an update of the IAS with optixAccelBuild.