Open V4L2SRC YV12 60FPS camera on TX2 with opencv

Autonomous Machines Jetson & Embedded Systems Jetson TX2
1

Hey all,

I’m trying to open up in the fastest and efficient way a v4l2src YV12 60FPS camera using JETSON TX2.

Right now i’m using:

cap.open("v4l2src device=/dev/video0 ! video/x-raw, width=1280, height=720, framerate=
(fraction)60/1, format=(string)YV12 ! videoconvert ! video/x-raw, width=1280, height=720, framerate=
(fraction)60/1, format=(string)I420 ! appsink sync=false async=false emit_signals=true drop=true
 max-buffers=1");

and then:

cvtColor(img, img, CV_YUV2BGR_I420);

This thing gives me 42 FPS top and I’m looking a way to run faster if possible.

That way i’m getting 30 FPS:

cap.open("v4l2src device=/dev/video0 ! video/x-raw, width=1280, height=720, framerate=
(fraction)60/1, format=(string)YV12 ! videoconvert ! video/x-raw, width=1280, height=720, framerate=
(fraction)60/1, format=(string)BGR ! appsink sync=false async=false emit_signals=true drop=true max-
buffers=1");

And i’m trying to do:

cap.open("v4l2src device=/dev/video0 ! video/x-raw, width=1280, height=720, framerate=
(fraction)60/1, format=(string)YV12 ! videoconvert ! video/x-raw, width=1280, height=720, framerate=
(fraction)60/1, format=(string)I420 ! nvvidconv ! video/x-raw, width=1280, height=720, framerate=
(fraction)60/1, format=(string)BGR ! appsink sync=false async=false emit_signals=true drop=true max-
buffers=1");

but it failed.
what am I doing wrong?

2

nvvidconv expects at least one of its input (sink) or output (src) to be in NVMM memory.

You may use instead:

cap.open("v4l2src device=/dev/video0 ! video/x-raw, width=1280, height=720, framerate=
(fraction)60/1, format=(string)YV12 ! videoconvert ! video/x-raw, width=1280, height=720, framerate=
(fraction)60/1, format=(string)I420 ! nvvidconv ! video/x-raw(memory:NVMM), width=1280, height=720, framerate=
(fraction)60/1, format=(string)BGRx ! videoconvert ! video/x-raw, format=BGR ! appsink sync=false async=false emit_signals=true drop=true max-buffers=1")

but you may also bench direct conversion:

cap.open("v4l2src device=/dev/video0 ! video/x-raw, width=1280, height=720, framerate=(fraction)60/1, format=(string)YV12 ! videoconvert ! video/x-raw, width=1280, height=720, framerate=(fraction)60/1, format=(string)BGR ! appsink sync=false async=false emit_signals=true drop=true max-buffers=1")
3

For some reason, that:

cap.open("v4l2src device=/dev/video0 ! video/x-raw, width=1280, height=720, framerate=
(fraction)60/1, format=(string)YV12 ! videoconvert ! video/x-raw, width=1280, height=720, framerate=
(fraction)60/1, format=(string)I420 ! nvvidconv ! video/x-raw(memory:NVMM), width=1280, height=720, framerate=
(fraction)60/1, format=(string)BGRx ! videoconvert ! video/x-raw, format=BGR ! appsink sync=false async=false emit_signals=true drop=true max-buffers=1")

Does not work, even if I use just:

cap.open("v4l2src device=/dev/video0 ! video/x-raw, width=1280, height=720, framerate=
(fraction)60/1, format=(string)YV12 ! videoconvert ! video/x-raw, width=1280, height=720, framerate=
(fraction)60/1, format=(string)I420 ! nvvidconv ! video/x-raw(memory:NVMM), width=1280, height=720, framerate=
(fraction)60/1, format=(string)I420 ! appsink sync=false async=false emit_signals=true drop=true max-buffers=1")

and then:

cvtColor(img, img, CV_YUV2BGR_I420);

It’s got stuck on:

cap.read(img);

The second one works but as I mentioned in the original message it’s very slow.

4

Sorry, my proposal was incomplete. videoconvert only expects CPU memory, so it fails to accept NVMM memory.
You may just use a second nvvidconv for copying from NVMM to CPU memory:

v4l2src device=/dev/video0 ! video/x-raw, width=1280, height=720, framerate=60/1, format=YV12 ! videoconvert ! video/x-raw, width=1280, height=720, framerate=60/1, format=I420 ! nvvidconv ! video/x-raw(memory:NVMM), width=1280, height=720, framerate=60/1, format=BGRx ! nvvidconv ! video/x-raw, width=1280, height=720, framerate=60/1, format=(string)BGRx ! videoconvert ! video/x-raw, format=BGR ! appsink ...

For almost the same reason your second trial fails because appsink expects its input in CPU memory, not NVMM memory.

For performance, be sure to boost your TX2 with

sudo nvpmodel -m0  # All cores up
sudo /home/nvidia/jetson_clocks.sh    # boost clocks

You may also read this article from eCon Systems explaining how to use v4l2 userptr for reading frames into opencv. Checkout their helper lib, and you would just use cv::cvtColor with COLOR_YUV2BGR_YV12.

5

Thanks @Honey_Patouceul,

The “All cores up” relly boost my performance, I thought jetson_clocks.sh does that when it need to but I guess not.

With your capture suggestion and the “All cores up” I get around 47 FPS top, but with my first capture:

cap.open("v4l2src device=/dev/video0 ! video/x-raw, width=1280, height=720, framerate=
(fraction)60/1, format=(string)YV12 ! videoconvert ! video/x-raw, width=1280, height=720, framerate=
(fraction)60/1, format=(string)I420 ! appsink sync=false async=false emit_signals=true drop=true
 max-buffers=1");

and then:

cvtColor(img, img, CV_YUV2BGR_I420);

I get around 56 FPS top which is great.

Could you please take a look on my h264 pipeline?
What’s your opinion? Can it be improved?

pipline = "appsrc ! videoconvert ! nvvidconv ! video/x-raw(memory:NVMM), format=I420 ! queue !
 omxh264enc control-rate=4 bitrate=2000000 ! video/x-h264, stream-format=(string)byte-stream !
 h264parse ! rtph264pay mtu=1514 ! udpsink host="+Host+" port="+Port+" async=false sync=false";
6

Your pipeline will probably add extra load on CPUs and your framerate may decrease. If this comes to be a problem, you may check reading frame from V4L and using only one cvtColor in opencv.

About your pipeline itself, I cannot comment a lot, I’m not so familiar with H264 encoding and UDP streaming. Seems you have a very low bitrate, but this may be deliberate looking at your contol-rate setting.