I want to do intersection test with optix prime. I have different types of rays and a triangle set. After the intersection test, besides the intersected triangles, I also want to know which type of ray the triangle is intersected with, how can I do this with optix?
There are no different types of rays in OptiX Prime, only different types of ray intersection queries.
Please read through Chapter 9. OptiX Prime: Low-Level Ray Tracing API inside the OptiX Programming Manual and work through the OptiX examples prefixed with simplePrime.