I understand rectangle is the built-in shape for RT core, but what if it is for a GPU without RT core?
No! The only built-in geometric primitives in OptiX are triangles and curves (linear, quadratic and cubic B-splines).
Please read the OptiX Programming Guide again. This is said in the second sentence of the first chapter Overview
Will there be a performance difference using triangle vs rectangle build with OptixAabb?
For the acceleration structure build itself? No, not really.
But for the runtime performance the performance difference between built-in triangle and custom primitives will be dramatic on RTX devices because these have hardware support for ray-triangle intersections,
For devices without RT cores there are highly optimized intersection routines for triangles (and curves on all devices) inside OptiX which are hard to beat in performance at the implemented precision.
To represent a rectangle with two triangles, I will need 6 vect3, and for a rectangle, I only need 4. The memory benefits are straightforward.
That is actually the smaller part of possible memory savings.
First, you don’t need to specify the triangles as independent triangle array with three vertices per triangle. You would normally define indexed triangles, which means you have the same four vertices and use six indices to build two triangles, sharing two of the vertices.
This is especially efficient in fully connected triangle meshes where the internal vertices are reused for six triangles!
This is the standard method for mesh definitions in rasterizer and raytracer APIs.
The more important memory saving between these two geometric primitives is actually that a custom rectangle primitive needs only one axis aligned bounding box (AABB), and two triangles need two AABB. That can make quite a difference for the acceleration structure (AS) size and if the only concern is memory usage, it’s a valid method to use custom rectangle primitives with your own intersection program at the cost of a considerable runtime performance hit from not using hardware ray-triangle intersections.
But also note that geometry AS can be compacted and esp. RTX devices can compress them very efficiently.
Also, I went through the optix example code, The only example of using OptixAabb is to build a sphere. I still do not quite know how to build a rectangle using optixAabb.
All custom geometric primitives are defined by an AABB per primitive you calculate and the intersection program you implement for them.
Means you need to have a function which calculates the AABB over your four vertices defining your rectangle primitive and give that resulting array of AABBs per primitive as OptixBuildInputCustomPrimitiveArray to the optixAccelBuild() function as you already found inside the programming guide below.
The primitive index you get inside your OptiX device program when hitting any of the AABBs is the same as the index of the AABB inside that build-input array.
Follow are from the programming guide, but I still don’t know how to define rectangles for d_aabbBuffer. Can you provide an example?
The calculation of the AABB for for point based primitives is super easy. You only need to find the minimum and maximum x-, y-, z -components of all your positions per primitive. That’s all.
The OptiX SDK example optixVolumeViewer builds AABBs for boxes. It’s always the same method.
You can implement that on the host or with a native CUDA kernel on the device if it needs to be much faster.
In the end, the d_aabbBuffer must be on the device, so if you calculated it on the host you need to copy it from host to device with a cudaMemCpy() or cuMemcpyHtoD() depending on which CUDA API you use (runtime or driver API).
In the old OptiX API before version 7.0.0 you needed to specify a “bound box” program for that and before there where build-in triangles (since OptiX 6.0.0) you needed that for triangles as well. Here’s an example in one of my old OptiX 5.1. based examples:
https://github.com/nvpro-samples/optix_advanced_samples/blob/master/src/optixIntroduction/optixIntro_03/shaders/boundingbox_triangle_indexed.cu
For indexed rectangles that would obviously be the same routine just with four vertices. Port that code to the host and adjust it to your rectangle primitive definition.
The more complicated problem is how to implement an intersection program for rectangles.
If they can be arbitrary and not even planar, that is going to become interesting. If they are special cases like a parallelogram, there are examples for that inside the older OptiX SDKs.
With all that said, I seriously recommend using indexed triangles to represent your rectangles, simply for performance reasons and because you don’t need to implement a ray-rectangle intersection routine.
(The OptiX 7 SDK’s optixPathTracer example defines the Cornell Box with rectangles but then uses two independent triangles (not indexed) to build the acceleration structures.
The one in the older OptiX SDKs used a parallelogram primitive for the area light, but that is defined by an anchor point and two edge vectors.)