If the operations you intend to perform on frames are supported as GpuMats with opencv_cuda* libs, then you may try nvivafilter. Here is an example for fisheye undistort, you would adapt to your case (searching this forum you may find other examples but for older L4T and opencv versions).
Note that you can use nvivafilter without opencv, and directly use CUDA.