OpenCV Ffmpg recording using HWenc?

Autonomous Machines Jetson & Embedded Systems Jetson Nano
, ,
1

Hello,
I’m working with OpenCV 4.5 on Jetson Nano 2GB.
I need to record video from my webcam and save it to .mov or .mp4 ( I use cv::VideoWriter), the problem is that I do not know which encoding to use, while I tried a variety of fourcc-s ( like “mp4v”, “h264”, “avc1”) all of them generate a drop of fps (from 25 to 8 ) with resolution 1280x720. If I resize to a smaller size, it’s work’s better but I can’t go any lower.

So I thought about two solutions:

  • Saving raw frames and encoding them only when a user would like to download recordings.
  • Use some faster encoding

But I have no idea how to do that, I could only try different combinations for the whole day and I found nearly nothing!

At first, I thought it’s an OpenCV problem so askt on their forum but was redirected here :P

Thay suggested to me there that as OpenCV uses FFmpeg in the background it’s maybe a problem with my installation of it - when I set encoding to H264 (which should be supported by Jetson) it doesn’t use HW encoders or GPU at all - I used jtop to check that. It seems CPU only. Is there a way to install FFmpeg in a way that uses build-in encoding via VideoWriter?

I found a NvAPI example that does that using HW encoders, but that API is not really a thing that I would like to work with unless there’s no other choice :( , I tried to execute that example on some files saved from OpenCV (that go to my first idea) that maybe I could save my video somehow raw and then encode using that example - but I don’t know how to save the video without encoding at all. I tried anyway with a few fourcc-s, but it’s either not allowed in the example or I get shifted video :

image
image933×519 71.9 KB

I had issues with video reading speed so I’m not using VideoCapture but NvAPI and I get cv::Mat from it. [Check Here]

3

Hi,
So your current implementation is based on 12_camera_v4l2_cuda? We have a patch to demonstrate 12_camera_v4l2_cuda + NvVideoEncoder. It may be referred and applied in this usecase.

4

Yes.
I do not know about that, where can I find it?

5

Hi,
Please check this patch:

6

Ok, I implemented that patch (I also needed to add NvVideoEncoder *enc into centext_t - .h) but I’m not sure that I understand what it does…
It’s just encode frame after frame inside program memory? No recording?

So now if I want to save it to file what should I do?

Can I now just convert that to cv::Mat the same way as here and then use cv::VideoWriter on it? If so then what parameters I need to set (fourcc) for it to be playable or maybe just throw it into the file binarly like that

ofstream outfile("image.mp4", ios::out | ios::binary); //not sure about extension ?
outfile.write(image.ptr(),image.rows*image.cols*image.channels());
outfile.close();

Do I need to convert it additionally somehow? Cause if I understand correctly to encode I need YUV format, so I can’t just change input_params to BGR like before?

7

Probably the size is wrong here, as this would mean no compression. @DaneLLL may help you with that. Once you’ll get the correct h264 file, you may use a container such as mp4/qt mov or MKV for storing it. You may do that with ffmpeg as in this example.

The simplest way to do what you want to achieve doesn’t answer your initial question, but probably you may try a videoWriter with gstreamer pipeline such as:

  cv::VideoWriter gst_omxh264_writer ("appsrc ! queue ! videoconvert ! video/x-raw,format=I420 ! queue ! omxh264enc ! video/x-h264,format=byte-stream ! matroskamux ! filesink location=test-omxh264-writer.mkv ", 
                                      cv::CAP_GSTREAMER, 
                                      0, fps, 
                                      cv::Size (width, height));
  if (!gst_omxh264_writer.isOpened ()) {
    std::cout << "Failed to open gst-omxh264 writer." << std::endl;
    return (-1);
  }

For answering to your question, you would have to build jocover’s nvmpi lib, apply the patch to ffmpeg sources, set:

export PKG_CONFIG_PATH=/usr/local/share/pkgconfig:$PKG_CONFIG_PATH

Configure this ffmpeg version with pkg-config support and install in your own location (not tested, but such as):

./configure  --extra-libs='-L/usr/lib/aarch64-linux-gnu/tegra -lnvbuf_utils' --extra-cflags='-I /usr/src/jetson_multimedia_api/include/' --enable-nvmpi --enable-libx264 --prefix=/usr/local/ffmpeg --pkgconfigdir=/usr/local/ffmpeg/lib/pkgconfig

When built and installed, adjust PKG_CONFIG_PATH and LD_LIBRARY_PATH in your environment so that your new ffmpeg version can be found:

export LD_LIBRARY_PATH=/usr/local/ffmpeg/lib:$LD_LIBRARY_PATH
export PATH=/usr/local/ffmpeg/bin:$PATH
export PKG_CONFIG_PATH=/usr/local/ffmpeg/lib/pkgconfig:$PKG_CONFIG_PATH

and reconfigure opencv with -D WITH_FFMPEG=ON -DFFMPEG_PREFIX=/usr/local/ffmpeg and check with cmake-gui or looking at CMakeCache.txt that variables starting with pkgcfg_lib_FFMPEG_ are set to correct location. Rebuild and install opencv. Then you should be able to use a videoWriter with HW enc using ffmpeg backend:

  cv::VideoWriter ff_h264_writer ("test-ff_h264-writer.mkv",
                                   cv::CAP_FFMPEG,
                                   cv::VideoWriter::fourcc ('X', '2', '6', '4'), fps,
                                   cv::Size (width, height)); 
  if (!ff_h264_writer.isOpened ()) {
    std::cout << "Failed to open ff_h264-writer." << std::endl;
    return (-2);
  }
8

Hi,
In the patch, it demonstrates how to get h264 stream but the stream is not saved. You may open a file to save the stream. For saving to mp4(or mkv), you would need to check if there is 3rdparty solution to mux h264 stream into mp4.