The image of AR0233 have lag

[jason.tseng914]

Hi all,

I use Xavier, JetPack 4.4 and AR0233.
The raw format of AR0233 is NV12.
Because BGR need by OpenCV.
I want convert format from NV12 to BGR.
Following is my test code.

#include <stdio.h>
#include <iostream>

#include <opencv4/opencv2/opencv.hpp>
#include <opencv4/opencv2/videoio.hpp>
#include <opencv4/opencv2/highgui.hpp>

using namespace cv;
using namespace std;

int main(int argc, char** argv)
{
  Mat bgr;
  
  VideoCapture cap("nvarguscamerasrc sensor-id=0  ! video/x-raw(memory:NVMM), width=2048, height=1280, framerate=30/1, format=NV12 ! nvvidconv ! video/x-raw, width=2048, height=1280, format=BGRx ! videoconvert ! video/x-raw, width=2048, height=1280, format=BGR ! appsink");
    
  if (!cap.isOpened())
    {
      cout << "Failed to open camera." << endl;
      return -1;
    }

  for(;;)
    {    
      cap >> bgr;
      imshow("original", bgr);
      waitKey(1);
      
    }

  cap.release();
}

But the image have lag.
How can I fix the problem?
Appreciate your help.

-Jason

[JerryChang]

hello jason.tseng914,

it’s due to format conversion,
raw formats are not supported in OpenCV.
if you would like to get raw frames and do processing, please check v4l2cuda for capturing frames through v4l2, you may save to CUDA buffers and implement processing through CUDA.
thanks

[jason.tseng914]

Hi Jerry,

Thank you about your reply.
I have following question about camera_v4l2_cuda.

Q1:
I used following camera_v4l2_cuda command. But the screen is black.

nvidia@nvidia-desktop:~/jetson_multimedia_api/jetson_multimedia_api/samples/12_camera_v4l2_cuda$ ./camera_v4l2_cuda -d /dev/video0 -s 2048x1280 -f YUYV -n 30 -c 
[INFO] (NvEglRenderer.cpp:110) <renderer0> Setting Screen width 2048 height 1280
WARN: request_camera_buff(): (line:380) Camera v4l2 buf length is not expected
WARN: request_camera_buff(): (line:380) Camera v4l2 buf length is not expected
WARN: request_camera_buff(): (line:380) Camera v4l2 buf length is not expected
WARN: request_camera_buff(): (line:380) Camera v4l2 buf length is not expected
----------- Element = renderer0 -----------
Total Profiling time = 0
Average FPS = 0
Total units processed = 0
Num. of late units = 0
-------------------------------------
App run was successful

I display the format information.

nvidia@nvidia-desktop:~$ v4l2-ctl --list-formats-ext
ioctl: VIDIOC_ENUM_FMT
	Index       : 0
	Type        : Video Capture
	Pixel Format: 'RG12'
	Name        : 12-bit Bayer RGRG/GBGB
		Size: Discrete 2048x1280
			Interval: Discrete 0.033s (30.000 fps)

Why the screen is black when run camera_v4l2_cuda?
Can I use camera_v4l2_cuda on my camera?

Q2:
How can I convert NV12 to BGR by CUDA?
Is there sample code?

Best regards.

-Jason

[DaneLLL]

Hi,
Since hardware engines in Jetson series do not support BGR, so you would need to use software converter to convert to the format. Please check discussion in
[Gstreamer] nvvidconv, BGR as INPUT

For better solution in using OpenCV, please execute sudo nvpmodel -m 0 and sudo jetson_clocks. The commands can enable CPU cores at maximum clocks so that software converter runs at max throughput.

For using jetson_multimedia_api + OpenCV, you may refer to this patch:
NVBuffer (FD) to opencv Mat - #6 by DaneLLL