OpenCV expects BGR or GRAY8 format as input (although last versions can support more formats).
What format does your camera provide ?
sudo apt-get install v4l-utils
v4l2-ctl -d /dev/video0 --list-formats
If it cannot provide BGR, you may use gstreamer for converting, but opencv4tegra doesn’t have gstreamer support.
Building opencv from source is not so complicated (just need a disk with a several GB available and about one hour for building).
For building opencv, you may create a build directory (separate from your source directory) and configure using (not verified):
cmake -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=/usr/local/<your_opencv> \
-D WITH_CUDA=ON -D CUDA_ARCH_BIN="3.2" -D CUDA_ARCH_PTX="" \
-D ENABLE_FAST_MATH=ON -D CUDA_FAST_MATH=ON -D ENABLE_NEON=ON \
-D WITH_LIBV4L=ON -D WITH_GSTREAMER=ON -D BUILD_TESTS=OFF \
-D BUILD_PERF_TESTS=OFF -D BUILD_EXAMPLES=OFF <your_opencv_src_directory>
sudo make install
Note that using parallel make, compilers may run out of memory. In such case, just relaunch make.
Once installed, as opencv4tegra is installed in path /usr, be sure to specify <your_opencv> path for compiling/linking your opencv applications and set it in LD_LIBRARY_PATH before running, otherwise opencv4tegra may be used instead.