Yes. Many of the OpenCV CUDA tests fail if you run them. OpenCV isn’t a bad library, but it isn’t optimized for Tegra either, where Nvidia’s solutions will give you the best performance on x86/Nvidia and Tegra. It’s not really worth it to use OpenCV unless you have a very specific reason.
Besides, even if you build OpenCV with CUDA support, you’ll still have to rewrite any CPU based OpenCV code to use the OpenCV CUDA equivalents.