For the precompiled Isaac ROS release-3.2 packages, I would stay with the OpenCV version used by the official release-3.2 environment.
The official isaac_ros_common release-3.2 Dockerfile pins:
libopencv-dev=4.5.4+dfsg-9ubuntu4
python3-opencv=4.5.4+dfsg-9ubuntu4
So if you are using precompiled isaac_mapping_ros / isaac_ros_visual_global_localization binaries from release 3.2, the safe path is OpenCV 4.5.4. Other version of OpenCV should only be used if all dependent packages are rebuilt consistently.
About CUDA support:
Isaac ROS Visual Global Localization’s intended GPU acceleration path is cuVGL / CV-CUDA / CUDA, not OpenCV CUDA modules from libopencv-dev. In release 3.2, the Dockerfile installs Ubuntu OpenCV 4.5.4 packages and installs CV-CUDA separately via nvcv-lib/nvcv-dev. The VGL ROS wrapper uses OpenCV/cv_bridge mainly for image conversion into cv::Mat, then calls cuvgl_->Localize(...).
So using libopencv-dev=4.5.4+dfsg-9ubuntu4 should not remove cuVGL/CV-CUDA acceleration. It only means you are using the OpenCV ABI expected by the release-3.2 environment. A custom CUDA-enabled OpenCV build is a separate thing and should not be mixed into the same process unless all dependent packages are rebuilt consistently against it.
And your platform is generally in the Isaac ROS 3.2 supported range.