It is possible. Configure your opencv-3.2.0 build with CMAKE_INSTALL_PATH=/usr/local/opencv-3.2.0, build and install.
Be aware that opencv4tegra is in /usr, so its include and libs path will be found by default by gcc, so for opencv-3.2.0 be sure to give explicit paths to your opencv-3.2.0 with -I and -L. You may also have to set LD_LIBRARY_PATH accordingly.
One extra thing to know is that pkg-config is sometimes used to provide flags or libs in toolchains. Check also if
pkg-config --modversion opencv
pkg-config --cflags --libs opencv
returns the expected version and paths.
If not, you might rename file /usr/lib/pkgconfig/opencv.pc and make there instead a symlink to /usr/local/opencv-3.2.0/lib/pkgconfig/opencv.pc.
Opencv3 has a slighlty different API from opencv2. Mainly cv::gpu changed to cv::cuda, and you may have to adapt includes and libraries.
For comparison, especially if using gpu, be aware that first execution may take some time to bring up everything.