Avoid use of videoconvert

Robotics & Edge Computing Jetson & Embedded Systems Jetson Xavier NX
, ,
1

Hello,

i’m using mediapipe on the Jetson NX, and it needs to be fed images this way:

auto input_frame = absl::make_uniquemediapipe::ImageFrame(
mediapipe::ImageFormat::SRGBA, camera_frame_raw.cols, camera_frame_raw.rows,
mediapipe::ImageFrame::kGlDefaultAlignmentBoundary);
cv::Mat input_frame_mat = mediapipe::formats::MatView(input_frame.get());
cv::cvtColor(camera_frame_raw, input_frame_mat, cv::COLOR_BGR2RGBA);

I’m using this gstreamer pipeline:

nvarguscamerasrc ! video/x-raw(memory:NVMM), width=(int)4032, height=(int)3040, format=(string)NV12, framerate=(fraction)30/1 ! nvvidconv ! video/x-raw,format=(string)BGRx ! videoconvert ! video/x-raw, format=(string)BGR ! appsink

And of course i can capture >> camera_frame_raw, visualize it and send it in mediapipe after cvtColor to RGBA

Now, to increase framerate and efficiency, i would like to get rid of ! videoconvert ! because it’s a big frame and the operation is done on the CPU .

This is the pipeline i’m trying to use:

nvarguscamerasrc ! video/x-raw(memory:NVMM), width=(int)4032, height=(int)3040, format=(string)NV12, framerate=(fraction)30/1 ! nvvidconv ! video/x-raw,format=(string)I420 ! appsink

I also tried using nvvideoconvert ( RGBA / I420 ) with the right cvtColor conversion applied before feeding it into mediapipe, but, although i can visualize it , it seems mediapipe would not accept it even tough the final color space is the same ( using cv::COLOR_YUV2BGRA_I420 ) ( i also tried YUV->BGR->RGBA )

What am i doing wrong?

Thanks

4

Hi,
It is limitation of Jetson platforms. Need to take certain CPU usage in using OpenCV. Please check

By default the system runs in dynamic frequency scaling. Please execute sudo jetson_clocks to run hardware engines in maximum clocks.

5

Thank you, i saw that Deepstream has the plugin nvvideoconvert that allows you to convert to format=(string)RGBA, but that doesn’t work either

6

However, if i use something like:

nvarguscamerasrc ! video/x-raw(memory:NVMM), width=(int)4032, height=(int)3040, framerate=(fraction)30/1 ! nvvidconv ! nvvideoconvert ! video/x-raw,format=RGBA ! appsink

I get a

[ WARN:0] global /home/nvidia/opencv_build/opencv-4.4.0/modules/videoio/src/cap_gstreamer.cpp (1761) handleMessage OpenCV | GStreamer warning: Embedded video playback halted; module nvarguscamerasrc0 reported: Internal data stream error.

7

Hi,
RGBA is not supported in appsink implementation of OpenCV. Please check the source code:
opencv/cap_gstreamer.cpp at master · opencv/opencv · GitHub

There are 11 data types and RGBA is not included. You have to use I420 or BGR.