Hi,
Running the two pipelines takes CPU resources and CPU capability dominates performance. You may run $ sudo tegrastats to check whether CPU cores are at 100% loading.
One thing you may try is to use nvv4l2camerasrc to capture YUY2 data. By default it supports UYVY and you can apply this patch to enable YUY2:
Macrosilicon USB - #5 by DaneLLL
So that certain loading is shifted to hardware converter. Should have some improvement.
Source code of nvv4l2camerasrc is in
Jetson Linux | NVIDIA Developer
Driver Package (BSP) Sources