Labeled objects in GAS

I consider using Optix for non-rendering applications. Is there a way to add a label or flag to each object in GAS? So when each ray hits the object, the call back function is able to know which object it hit and return the label/flag in the HIT call back function.

Sure. You could even return each individual primitive you hit in a GAS.

There are multiple ways to design that.

If you use a GAS with multiple shader binding table (SBT) entries, you could store user defined data on the SBT data which gets accessed via optixGetSbtDataPointer() and written to your per ray payload inside the closest hit program.
The OptiX SDK optixPathTracer example uses that method to assign different diffuse and emission colors to the parts in a single GAS.

You could also store an additional per primitive attribute which indicates the index you want to return. That needs more memory, but could have different IDs per primitives in one GAS irrespective of the assigned SBT hit record.

If you use a render graph hierarchy with a single top level instance acceleration structure (IAS) where each OptixInstance is referencing an individual GAS with one SBT record per GAS, things get more flexible because there is the instance index (the child number), the user defined instance ID (and the instance’s SBT offset) with which you could uniquely identify your GAS.
Look at the different OptiX device functions here to find how to get the instance index and ID values.

Sorry, I am still not quite sure how to do it. Still use the triangle example. Follow are two triangles If I want to know all the rays that hit triangle A. How do I label { -1.0f, 1.0f, 0.0f },{ -1.0f, -1.0f, 0.0f },{ 1.0f, 1.0f, 0.0f }, and what to add in the hit program to achieve this label, and then I will output it with payload, so I know which ray hit which triangle.
Can you provide me more instruction on how to achieve this?

        const std::array<float3,6> vertices =
        { {
              { -1.0f, 1.0f, 0.0f },
              { -1.0f, -1.0f, 0.0f },
              {  1.0f,  1.0f, 0.0f },
              { -1.0f, -1.0f, 0.0f },
              {  1.0f,  1.0f, 0.0f },
              {  1.0f,  -1.0f, 0.0f }
        } };

In that simple example all triangles are in a single geometry acceleration structure (GAS) and the triangles in that are already uniquely identified by their primitive index. There is no need for an additional label in that case.

To identify which primitive index in a GAS you hit, you call the OptiX device function optixGetPrimitiveIndex() inside the closest hit program. (That link is the same as above.)
Search for that inside the OptiX SDK example code to see how it’s used.

That unsigned int primitive index is the zero-based index of your primitives in the same order in which you stored them inside the respective OptiX acceleration structure build input using inside the optixAccelBuild() call for this GAS.

When defining triangles you want to make sure that they are all having the same winding. The default winding is counter-clockwise for front faces. Your second triangle is defined clockwise.
For that unit square [-1, 1] I would normally define the two triangles in the following order by walking on the outside edges of the quad in counter-clockwise order starting in the lower left.

(-1, -1), ( 1, -1), ( 1,  1),
( 1,  1), (-1,  1), (-1, -1)

Thanks, say if I have 4 triangles, and they are belong to two groups, and I only want to know which group the ray hit. In this way, I will need to add a label to the triangle right? How can I do that? can you explain more about this “You could also store an additional per primitive attribute which indicates the index you want to return.” Is there a example for this?

Again, there are multiple ways to handle that, which I already explained in the first answer.

1.) If you put the triangles into a separate GAS per group, then you would need an instance acceleration structure (IAS) on top of them and there are OptiX device functions in that same Programming Guide chapter linked twice now, which allow querying the child index inside that IAS or the user defined instance ID field at each OptixInstance.
Means that instance index or ID would be your unique “group” identifier.

(That instance index or ID plus the primitive index would uniquely identify each triangle in the scene, even when instancing the same geometry multiple times.)

There are many OptiX SDK examples which use OptixInstance structures.

2.) If you want to store all triangles inside a single GAS, you would need to store a unique per triangle attribute containing your “group” identifier with each triangle.
I would usually define indexed triangles, means there is an additional uint3 array with the indices into the vertices pool which define which vertices build one triangle primitive. Means if you wish to assing each triangle to a specific “group” there would need to be another unsigned int array with the same number of elements (triangle count) as the uint3 triangle indices in which you’d store your group identifier.
(More advanced: That could also be defined interleaved, like using an uint4 and the .w component is the unique triangle index, which can then be read along with the triangle vertex indices in the hit shaders. Reading an uint4 is faster than an uint3 because there is a specialize vectorized load instruction for that.)

This is working the same way as providing any other per triangle data to the OptiX device code. This is not different than accessing triangle indices. Search the OptiX SDK examples for optixGetPrimitiveIndex.
It just needs some device side memory buffer containing your group indices per triangle. You provide that as CUdeviceptr pointer either on the launch parameters or the SBT hit record data and index into that with the primitive index.

3.) If you’re using more than one shader binding table hit record for that single GAS where each “group” of triangles has its own SBT hit record entry, you could store the “group” identifier along with the SBT hit record data.
(The optixPathTracer example is using this to assign material and light colors.)

I would recommend the first method because that has more flexibility for more complex scenes in the future.

Please work through OptiX Programming Guide and the other OptiX SDK examples to see how instance acceleration structures with OptiX instances are used.