Avoid use of videoconvert

Hello,

i’m using mediapipe on the Jetson NX, and it needs to be fed images this way:

auto input_frame = absl::make_uniquemediapipe::ImageFrame(
mediapipe::ImageFormat::SRGBA, camera_frame_raw.cols, camera_frame_raw.rows,
mediapipe::ImageFrame::kGlDefaultAlignmentBoundary);
cv::Mat input_frame_mat = mediapipe::formats::MatView(input_frame.get());
cv::cvtColor(camera_frame_raw, input_frame_mat, cv::COLOR_BGR2RGBA);

I’m using this gstreamer pipeline:

nvarguscamerasrc ! video/x-raw(memory:NVMM), width=(int)4032, height=(int)3040, format=(string)NV12, framerate=(fraction)30/1 ! nvvidconv ! video/x-raw,format=(string)BGRx ! videoconvert ! video/x-raw, format=(string)BGR ! appsink

And of course i can capture >> camera_frame_raw, visualize it and send it in mediapipe after cvtColor to RGBA

Now, to increase framerate and efficiency, i would like to get rid of ! videoconvert ! because it’s a big frame and the operation is done on the CPU .

This is the pipeline i’m trying to use:

nvarguscamerasrc ! video/x-raw(memory:NVMM), width=(int)4032, height=(int)3040, format=(string)NV12, framerate=(fraction)30/1 ! nvvidconv ! video/x-raw,format=(string)I420 ! appsink

I also tried using nvvideoconvert ( RGBA / I420 ) with the right cvtColor conversion applied before feeding it into mediapipe, but, although i can visualize it , it seems mediapipe would not accept it even tough the final color space is the same ( using cv::COLOR_YUV2BGRA_I420 ) ( i also tried YUV->BGR->RGBA )

What am i doing wrong?

Thanks

Hi,
It is limitation of Jetson platforms. Need to take certain CPU usage in using OpenCV. Please check

By default the system runs in dynamic frequency scaling. Please execute sudo jetson_clocks to run hardware engines in maximum clocks.

Thank you, i saw that Deepstream has the plugin nvvideoconvert that allows you to convert to format=(string)RGBA, but that doesn’t work either

However, if i use something like:

nvarguscamerasrc ! video/x-raw(memory:NVMM), width=(int)4032, height=(int)3040, framerate=(fraction)30/1 ! nvvidconv ! nvvideoconvert ! video/x-raw,format=RGBA ! appsink

I get a

[ WARN:0] global /home/nvidia/opencv_build/opencv-4.4.0/modules/videoio/src/cap_gstreamer.cpp (1761) handleMessage OpenCV | GStreamer warning: Embedded video playback halted; module nvarguscamerasrc0 reported: Internal data stream error.

Hi,
RGBA is not supported in appsink implementation of OpenCV. Please check the source code:
https://github.com/opencv/opencv/blob/master/modules/videoio/src/cap_gstreamer.cpp

There are 11 data types and RGBA is not included. You have to use I420 or BGR.