Yes, this can be done multiple ways.
It depends a little on how flexible you are with the construction of your acceleration structures.
Also I’m assuming a ray group is either a separate optixLaunch call or an optixTrace call inside the ray generation which has different arguments, either for the SBT offset or payloads.
Let’s go about this from fast to slow:
1.) If you know which triangles need to be intersected by each of your ray groups, then the fastest method to ignore the other ones would be to not actually put them into the geometry acceleration structure (GAS).
Means you could build two GAS, one with your triangles T1, T3, T5, … and the other GAS with triangles T2 ,T4 ,T6, … (See
OPTIX_TRAVERSABLE_GRAPH_FLAG_ALLOW_SINGLE_GAS inside the OptiX Programming Guide.)
That would provide you with two traversable handles, let’s call them A and B, and each group of rays would use either the traversable handle A or B as argument in optixTrace to start the traversable
Means the “ignored” triangles are not even in the scene for that group of rays.
That should be the fastest possible implementation.
2.) If you need all triangles in the scene for other use cases, you could also use the same partitioning of triangles into GAS A and B, but put them under two instances of a top-level instance acceleration structure (IAS). (See
OPTIX_TRAVERSABLE_GRAPH_FLAG_ALLOW_SINGLE_LEVEL_INSTANCING inside the OptiX Programming Guide).
Each instance and the optixTrace call have an 8 bit visibilityMask which is tested with the logic operation AND and if it’s not zero, the ray will traverse the instance. Means if you put a visibilityMask 1 to the instance holding the GAS A and a visibilityMask 2 to the instance holding the GAS B, you can select with the same visibilityMask value inside the optixTrace, using the IAS traversable handle as start, which instance to traverse, which is again ignoring the other triangles.
(Also in case you need to shoot other type of rays which can intersect both sets of triangles, you can simply set the optixTrace visibilityMask to 3 which covers both bits.)
This would be almost as fast as 1. on RTX GPUs because the single level instancing (IAS->GAS scene hierarchy) and the visibilityMask are all evaluated in hardware.
3.) Slow method:
You put all triangles inside a single GAS. Then each triangle is uniquely identified by its primitive index inside the GAS which is available via the optixGetPrimitiveIndex device function (See https://raytracing-docs.nvidia.com/optix7/guide/index.html#device_side_functions#device-side-functions)
With that you can assign any additional data to a triangle by accessing some array with that primitive index.
In this case for example a byte per triangle which tells if the triangle belongs to one or the other group of rays.
(Actually if your triangles are really partitioned into even and odd triangles then the primitive index would be enough to distinguish these.)
Now to actually ignore a triangle you would need to implement an anyhit program which compares some information on the per ray payload (optixGetPayload_*() functions) with the per triangle information and call optixIgnoreIntersection when the triangle should be ignored, which lets the ray traversal continue.
This is slower because the hardware BVH traversal on RTX boards will be interrupted for each intersected triangle and the anyhit program will be called to determine if the hit is valid or not.
4.) A variation of 3. would be to use two SBT entries and the sbtIndexOffsetBuffer to define two different SBT hit records (basically two different “materials” assigned to different triangles in a single GAS) which would then have an anyhit program which can reject rays of the other group. The difference here is that the two anyhit programs would know (hardcoded) which ray group they need to ignore instead of reading the additional per triangle data.