First try boosting with jetson_clocks script.
If not enough, it would be possible to do that with PVA/VPI or with OpenCV/CUDA using GPU such as here :
Note that the linked sample is from 2019, not tried it with recent JetPack releases. If the channels are not in the expected order, you may see this sample using nvivafilter also mapping RGBA frame from NVMM memory into a GpuMat, that has been tried in recent versions.