Actually no. The correct prefix for intersection programs in OptiX 7 is __intersection__. See this OptiX 7 Programming Guide chapter: 6.1 Program Input
When you’ve compiled that into an OptixModule you would then need to assign it to all OptixProgramDescriptions which get assigned to such primitive.
Fill in the fields hitgroup.moduleIS and hitgroup.entryFunctionNameIS with the module and intersection program name.
The anyhit and closesthit programs which should be called for these custom primitives will otherwise not be invoked.
The rest is the same as in all OptiX 7 programs: create the OptixPipeline, calculate and set the pipeline’s stack space, call optixSbtRecordPackHeader to get each program’s SBT record information, put everything together inside the SBT.
Read these threads to see what else should be done inside custom intersection programs to be able to determine what primitive type actually has been hit. That allows reusing anyhit and closesthit programs for multiple geometric primitive types and calculation of the respective final vertex attributes of the hit surface accordingly.
https://forums.developer.nvidia.com/t/how-to-get-the-transform-matrix-of-an-instance/157125/2
https://forums.developer.nvidia.com/t/optix-triangle-hit-face/83511/2
https://forums.developer.nvidia.com/t/objects-appearing-in-the-wrong-order-after-scaling/83884/7
For runtime performance reasons it doesn’t make sense to implement a custom quad primitive.
Since only BVH traversal and ray-triangle intersections are fully hardware accelerated on RTX boards, implementing a custom geometric primitive which can be represented with two triangles is definitely going to render slower.
The potential benefit would be less data for the acceleration structure, but that will also only become relevant for a huge number of quads. The AS compaction is pretty effective.
Also is there any reason to use the OptiX 7.0.0 version instead of the current OptiX 7.2.0?