Extremely high latency when feeding camera to cv2

I have an imx447 camera and in most test programs the latency is only a few hundred milliseconds, however using python and opencv anything over a resolution of 400 pixels or so has a delay of over 3 seconds sometimes. Is this an expected behavior?

For using OpenCV there is a memory copy and may be slow in high resolution. Please refer to discussion in
[Gstreamer] nvvidconv, BGR as INPUT - #2 by DaneLLL

Please run sudo tegrastats to check if CPU usage is full in the condition. If yes, it is the system limit.