Capturing from nvarguscamerasrc with gstreamer, you would have the buffers into NVMM memory ready for GPU processing. If your processing doesn’t change resolution, you may use nvivafilter that can perform CUDA processing on NVMM buffers (RGBA or NV12, the latter may have stride constraint).
As an example of using opencv/cuda from nvivafilter, see: