Which OptiX 6 version (major.minor.micro) are you using?
How can I be sure that I have the video card’s RTX mode turned on and used correctly?
There is no need to enable the RTX execution strategy at all. That is the default since over a year.
Please do not use the old execution strategy anymore.
The performance of a ray tracer depends on many things, and it’s more dependent on the number of rays than on the number of triangles.
It’s not possible to say what the limiting factors are in your case with the given information.
Maybe you’re shading bound, or maybe you had vertical sync enabled during the benchmark, etc.
Please have a look into some related topics about what RT cores accelerate:
https://forums.developer.nvidia.com/t/optix-6-0-rtx-acceleration-is-supported-on-maxwell-and-newer-gpus/70206
https://forums.developer.nvidia.com/t/api-related-to-triangle-mesh/82909
https://forums.developer.nvidia.com/t/leveraging-rtx-hardware-capabilities-with-optix-7-0/107733