The first thing you can do is using BGRx instead of I420 as output of nvvidconv, and use videoconvert for BGRx → BGR. It may be a bit faster than converting in opencv. Reading I420 frames in opencv would only be better if you process your frames in Y(UV) format. If you need BGR (as most opencv algorithm expect), this would be better IMHO.
nvarguscamerasrc ! video/x-raw(memory:NVMM), width=1920, height=1080,format=NV12, framerate=30/1 ! nvvidconv ! video/x-raw,format=BGRx ! videoconvert ! video/x-raw, format=BGR ! appsink
A better alternative would be using jetson-utils for getting frames RGB frames at high rate. See this example.