The OptiX interface for custom primitives always needs to have exactly 1 AABB per “primitive”. What you do inside that AABB is up to you. You have complete and total control over what a “primitive” means when it comes to custom primitives.
You want to create a bounding box for each and every “primitive”, and then pass those bounding boxes to
optixAccelBuild() in order to create your acceleration structure. When you trace, OptiX is going to call
__intersection__ every time a ray hits one of those boxes you provided, and in the intersection program, OptiX will tell you that the primitive ID is the index of the corresponding box you created.
This means you almost certainly want to give 1 box per quadrilateral, which implies that your intersection program will only lookup the data for a single quadrilateral, and the calculate the ray intersection for only that quad.
This also means that you could, if you wanted, define a primitive to be a group of quads. Suppose you want to save memory by grouping 4 quads at a time, and suppose it’s acceptable to you if intersection goes slower. Then you could provide a bounding box for every group of 4 quads. You would have four times as many quads as AABBs. Your intersection program would need to iterate through all four quads in your 4-quad “primitive” and calculate the intersection correctly. You would have to make sure that in your intersection program, you can map the OptiX primitive ID to the 4 quads that were used to create the original AABB. (The old OptiX 6 SDK sampled called “optixMDLDisplacement” is an example of doing exactly this, of bounding and intersecting multiple sub-primitives within a single “primitive”. The same could be done with OptiX 7.)