We can assume that right now I don’t need to do any operations on images beyond resize and crop - both I can do in gstreamer. I also need OpenCV to dynamically stream cropped video from the camera (it will be digital zoom feature of part of the image).
Resuming, my main goal is to:
a) Capture 4k resolution screen
b) Resize it to e.g. 1600 x 1200
c) Resize it to 320 x 240
d) Dynamically crop (from 4K) part of the video in resolution 640x480.
I have a 4k camera (Arducam on IMX477 chip). When I use the following code in bash:
gst-launch-1.0 nvarguscamerasrc ! 'video/x-raw(memory:NVMM), format=NV12, width=4032, height=3040' ! tee name=t ! queue ! nvv4l2h264enc insert-sps-pps=true ! h264parse ! rtph264pay pt=96 ! udpsink host=127.0.0.1 port=5004 sync=false t. ! queue ! nvvidconv ! 'video/x-raw(memory:NVMM), width=640, height=480' ! nvv4l2h264enc insert-sps-pps=true ! h264parse ! rtph264pay pt=96 ! udpsink host=127.0.0.1 port=5005 sync=false
I got two streams, the 4k stream has a latency of about 3-4 seconds (!), the smaller one has a minimum latency (<150ms).
- Do you have an idea why the latency of the 4k stream is that big?
When I use the following code from Python
import cv2
cap = cv2.VideoCapture("nvarguscamerasrc ! video/x-raw(memory:NVMM), width=(int)4032, height=(int)3040,format=(string)NV12, framerate=(fraction)30/1 ! nvvidconv ! video/x-raw, format=(string)BGRx ! videoconvert ! video/x-raw, format=(string)BGR ! queue ! appsink drop=1", cv2.CAP_GSTREAMER)
print(cap)
if not cap.isOpened():
print('Failed to open camera')
exit
rtpudp1013 = cv2.VideoWriter("appsrc ! queue ! videoconvert ! video/x-raw,format=BGRx ! nvvidconv ! nvv4l2h264enc insert-sps-pps=1 insert-vui=1 ! rtph264pay ! udpsink host=127.0.0.1 port=5004", cv2.CAP_GSTREAMER, 0, 30.0, (1440, 1080))
if not rtpudp1013.isOpened():
print('Failed to open rtpudp1013')
cap.release()
exit
rtpudp82 = cv2.VideoWriter("appsrc ! queue ! videoconvert ! video/x-raw,format=BGRx ! nvvidconv ! video/x-raw(memory:NVMM),width=320,height=240 ! nvv4l2h264enc insert-sps-pps=1 insert-vui=1 ! rtph264pay ! udpsink host=127.0.0.1 port=5005", cv2.CAP_GSTREAMER, 0, 30.0, (109,82))
if not rtpudp82.isOpened():
print('Failed to open rtpudp82')
rtpudp1013.release()
cap.release()
exit
while True:
ret_val, img = cap.read()
if not ret_val:
break
rtpudp1013.write(img)
rtpudp82.write(img)
cv2.waitKey(1)
rtpudp1013.release()
rtpudp82.release()
cap.release()
Both streams look like “color horizontal stripes”. When I
a) change the video resolution in the resized pipeline (rtpudp1013) to 1920x1080 there is no change in the stream (color horizontal stripes)
b) change video resolutions both in captured and resized (rtpudp1013) pipeline to 1920x1080 it works well.
- Why I can’t capture 4k resolution?
- Are there any rules on what I can capture and how can I resize it? If yes. How Can I find them?
There is also one important thing to mention, each time when I use gstreamer from python I can see:
[ WARN:0] global /home/stageeye/Projects/stream/workspace/opencv-4.5.0/modules/videoio/src/cap_gstreamer.cpp (935) open OpenCV | GStreamer warning: Cannot query video position: status=0, value=-1, duration=-1
It is only a warning, but I’m not sure if it is an issue or not.