Is it possible to make the RTX 5000 perform the OptiX intersection calculations without using the RT cores?
That is not possible. Please read these explanations of a similar question and follow the links in them:
https://forums.developer.nvidia.com/t/optix-6-5-wireframe-rendering/119619/10
https://forums.developer.nvidia.com/t/enable-rtx-mode-in-optix-7-0/122620
If you’re comparing the Quadro RTX 5000 and Quadro M4000 with respect to the intersection precision of the built-in triangle primitives, they should be very similar, but you cannot expect a perfect match between different hardware.
A single difference like using a fused-multiply-add and a code path with separate multiply and add can already result in different rounding.
From a performance standpoint your RTX board should always be faster than a three generations older board of a lesser level.
You could implement your own triangle intersection program, but then you lose all performance improvements from the hardware triangle intersection and need to call back to the streaming multiprocessors during BVH traversal. That’s considerably slower.
Also note that the built-in triangle intersection routine is watertight. The triangle intersection code provided in earlier OptiX SDK API versions is not.
Maybe explain why your rendering algorithm is so sensitive to the floating point accuracy.
Is you scene huge? Are nearby intersections very close to each other? Etc.