Saving video and snapshot-preview images with gst

Hello.

I need to save a high fps video, and get access to frames with python in real time for showing a preview with pygame or something similar.

I´m trying something like this:

cap = cv2.VideoCapture('v4l2src device=/dev/video0  ! image/jpeg, width=1920, height=1080, framerate=60/1 ! nvv4l2decoder mjpeg=1 ! nvvidconv ! tee name=t t. ! omxh264enc control-rate=2 bitrate=20000000 ! queue ! mux. alsasrc ! audio/x-raw,width=16,depth=16,rate=32000,channel=1 ! queue ! audioconvert ! audioresample ! voaacenc ! aacparse ! qtmux name=mux ! filesink location=/home/ances/Desktop/videos/v0.mp4 sync=True t. ! appsink ', cv2.CAP_GSTREAMER)

But I have errors, like this:

NvMMLiteOpen : Block : BlockType = 277
NVMEDIA: Reading vendor.tegra.display-size : status: 6
NvMMLiteBlockCreate : Block : BlockType = 277
[ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (1757) handleMessage OpenCV | GStreamer warning: Embedded video playback halted; module v4l2src0 reported: Internal data stream error.
[ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (886) open OpenCV | GStreamer warning: unable to start pipeline
[ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (480) isPipelinePlaying OpenCV | GStreamer warning: GStreamer: pipeline have not been created

I tried saving images and video simultaneously, hoping for reading the images with python:

gst-launch-1.0 v4l2src device=/dev/video0  ! image/jpeg, width=1920, height=1080, framerate=60/1 ! nvv4l2decoder mjpeg=1 ! nvvidconv ! tee name=t t. ! omxh264enc control-rate=2 bitrate=20000000 ! queue ! mux. alsasrc ! audio/x-raw,width=16,depth=16,rate=32000,channel=1 ! queue ! audioconvert ! audioresample ! voaacenc ! aacparse ! qtmux name=mux ! filesink location=/home/ances/Desktop/videos/v0.mp4 sync=True t. !  queue ! jpegenc ! multifilesink location="/home/ances/Desktop/videos/frame%06d.jpg"  max-files=300  -e 

But the video fps decreases dramatically.

I don´t need the highest fps preview, but it is important to save the video with the highest fps.

Could anyone suggest any ideas?

Thanks.

I found this, that preserve the original fps and reduce it for saving images, and work almost fine (I can’t reach the máximum fps in video).

gst-launch-1.0 v4l2src device=/dev/video0  ! image/jpeg, width=1920, height=1080, framerate=60/1 ! nvv4l2decoder mjpeg=1 ! nvvidconv ! tee name=t t. ! omxh264enc control-rate=2 bitrate=20000000 ! queue ! mux. alsasrc ! audio/x-raw,width=16,depth=16,rate=32000,channel=2 ! queue ! audioconvert ! audioresample ! voaacenc ! aacparse ! qtmux name=mux ! filesink location=/home/ances/Desktop/videos/v0.mp4 sync=True t. !  queue ! videoconvert ! videoscale ! videorate  max-rate=3 !  jpegenc ! multifilesink location="/home/ances/Desktop/videos/frame%06d.jpg"  max-files=10  -e 

But is there a elegant way of getting frames with python-opencv?

Hi,
For this use-case, you would need to have cv2.VideoCapture() and cv2.VideoWriter(). Please refer to
Displaying to the screen with OpenCV and GStreamer - #9 by DaneLLL

Thanks!

I had found this, that save video without VideoWriter:

import time, cv2, threading


def capturing(x):
    x=str(x)
    cap = cv2.VideoCapture("v4l2src device=/dev/video"+x+" io-mode=2  ! image/jpeg, width=1920, height=1080, framerate=60/1  ! nvv4l2decoder mjpeg=1 ! tee name=t t. !  nvvidconv ! nvv4l2h264enc control-rate=1 bitrate=20000000 ! h264parse ! matroskamux  ! filesink location=/home/ances/Desktop/videos/"+x+".mp4 t. ! queue ! nvvidconv ! video/x-raw, format=BGRx ! videoconvert ! video/x-raw,format=BGR ! appsink ")
    start_time = time.time()
    for a in range(600):
        f,ret=cap.read()
    print("video%s: %s seconds" % (x,time.time() - start_time))
    cap.release()
    

x0 = threading.Thread(target=capturing, args=(0,))
x0.start()

With 1920x1080, the maximum fps reached was 45, but with 960x540, it reach 60 fps.

Is it possible to achieve 60fps @ 1920x1080?

Could anyone suggest any idea?

Thanks.

Hi,
Looks like the performance is capped by CPU capability. You can run sudo tegrastats to get system status.

Sine CPU capability limits performance performance it may not achieve 1080p60 while using OpenCV. We would suggest check if you can run standalone gstreamer command or use jetson_multimedia_api.

1 Like

This is tegrastats:

RAM 1197/3964MB (lfb 452x4MB) SWAP 0/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [0%@921,0%@921,off,off] EMC_FREQ 0%@1600 GR3D_FREQ 0%@76 APE 25 PLL@34.5C CPU@35.5C PMIC@50C GPU@36.5C AO@43C thermal@35.75C POM_5V_IN 1607/1648 POM_5V_GPU 0/0 POM_5V_CPU 164/185

And when I run this:

import time, cv2, threading


def capturing(x):
    x=str(x)
    cap = cv2.VideoCapture("v4l2src device=/dev/video"+x+" io-mode=2  ! image/jpeg, width=1920, height=1080, framerate=60/1  ! nvv4l2decoder mjpeg=1 ! tee name=t t. !  nvvidconv ! nvv4l2h264enc control-rate=1 bitrate=20000000 ! h264parse ! matroskamux  ! filesink location=/home/ances/Desktop/videos/"+x+".mp4 t. ! queue ! nvvidconv ! video/x-raw, format=BGRx ! videoconvert ! video/x-raw,format=BGR ! appsink ")
    start_time = time.time()
    for a in range(600):
        f,ret=cap.read()
    print("video%s: %s seconds" % (x,time.time() - start_time))
    cap.release()
    
    

x0 = threading.Thread(target=capturing, args=(0,))
x0.start()
x1 = threading.Thread(target=capturing, args=(1,))
x1.start()
x2 = threading.Thread(target=capturing, args=(2,))
x2.start()
x3 = threading.Thread(target=capturing, args=(3,))
x3.start()

I get this:

RAM 1754/3964MB (lfb 341x4MB) SWAP 0/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [100%@921,100%@921,off,off] EMC_FREQ 24%@1600 GR3D_FREQ 0%@76 NVENC 268 NVJPG 486 APE 25 PLL@35C CPU@37C PMIC@50C GPU@37C AO@44C thermal@37.25C POM_5V_IN 3721/3309 POM_5V_GPU 0/0 POM_5V_CPU 970/815

The videos reach to 13 fps.

Doing something like this:

gst-launch-1.0 v4l2src device=/dev/video0 io-mode=2  ! image/jpeg, width=1920, height=1080, framerate=60/1  ! nvv4l2decoder mjpeg=1  ! nvvidconv ! tee name=t t.  ! omxh264enc control-rate=2 bitrate=20000000  ! queue  ! mux. alsasrc  ! audio/x-raw,width=16,depth=16,rate=32000,channel=1  ! queue  ! audioconvert  ! audioresample  ! voaacenc  ! aacparse  ! qtmux name=mux  ! filesink location=/home/ances/Desktop/videos/3.mp4 sync=True t. !  videoscale ! videorate  max-rate=1 ! jpegenc ! multifilesink location="/home/ances/Desktop/videos/video0%06d.jpg"  max-files=60 -e

I get this:

RAM 1434/3964MB (lfb 345x4MB) SWAP 0/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [8%@921,20%@921,off,off] EMC_FREQ 12%@1600 GR3D_FREQ 0%@76 APE 25 PLL@35C CPU@36.5C PMIC@50C GPU@37C AO@44.5C thermal@35.75C POM_5V_IN 2490/3507 POM_5V_GPU 0/0 POM_5V_CPU 244/687

And video reach 58 fps.

I had enabled the four processors with:

sudo nvpmodel -m 0
sudo jetson_clocks

With this, trying to record 4 cameras 1080p60 with this:

import time, cv2, threading


def capturing(x):
    x=str(x)
    cap = cv2.VideoCapture("v4l2src device=/dev/video"+x+" io-mode=2  ! image/jpeg, width=1920, height=1080, framerate=60/1  ! nvv4l2decoder mjpeg=1 ! tee name=t t. !  nvvidconv ! nvv4l2h264enc control-rate=1 bitrate=20000000 ! h264parse ! matroskamux  ! filesink location=/home/ances/Desktop/videos/"+x+".mp4 t. ! queue ! nvvidconv ! video/x-raw, format=BGRx ! videoconvert ! video/x-raw,format=BGR ! appsink ")
    start_time = time.time()
    for a in range(600):
        f,ret=cap.read()
    print("video%s: %s seconds" % (x,time.time() - start_time))
    cap.release()
    
    

x0 = threading.Thread(target=capturing, args=(0,))
x0.start()
x1 = threading.Thread(target=capturing, args=(1,))
x1.start()
x2 = threading.Thread(target=capturing, args=(2,))
x2.start()
x3 = threading.Thread(target=capturing, args=(3,))
x3.start()

It can reach about 30 fps, although with1280x720, it can reach 60fps.

I would like to reach 1080p60, but if it is impossible, I think 720p60 would be fine.

If no one has a suggestion, I could close this post.

Thanks.

1 Like

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.