We do not use GeometryTriangles in our current implementation, but triangle meshes with intersection and BB programs.
Interesting, then you’re actually not using the RT core hardware triangle intersection on the RTX boards but only the hardware BVH traversal. Means there a huge potential to get more performance on the RTX boards.
The OptiX Advanced Samples on https://forums.developer.nvidia.com/t/optix-advanced-samples-on-github/48410 were written against OptiX 5.1.0 and would run similarly with OptiX 6.5.0 on RTX boards.
But if you say that all other SDK examples work, there is little to do about this from remote.
In any case, this shouldn’t happen when everything else is correct.
Long term I would recommend to port to OptiX 7 before changing the existing renderer more.