Not sure why that is the error message.
You have the single rtReportIntersection(0) inside the if (rtPotentialIntersection(t)) block.
https://forums.developer.nvidia.com/t/setacceleration-fails-after-upgrading-from-3-9-1-to-4-0-0/156842/10
But what is definitely incorrect inside your mesh_triangle.cu code is the use of rtIgnoreIntersection() inside an intersection program.
That function is only allowed inside anyhit programs.
This table describes which OptiX device functions are available in which program domains:
https://raytracing-docs.nvidia.com/optix6/guide_6_5/index.html#programs#the-program-scope-of-api-function-calls
You need to remove that function and either simply return instead or restructure the conditions to not reach the rtReportIntersection call when you want to ignore it.