OpenCV 4.1.1 and writing Mat (or GpuMat) to video

I have a Ximea xiC USB3 camera that I want to use to record high speed (150 fps) videos with my Jetson TX2. I have managed with the help of Ximeas examples to make a code that reads the camera and sends images to zerocopy memory. From there the GpuMat is created and rendered to imshow with OpenGL. Works beautifully!

The problem is that I want to write the stream to a video file. OpenCV built-in videoWriter does not work directly with GpuMat in linux. So the only way to do this is, if I have understood correctly, is to use Gstreamer inside videoWriter with Mat (so I convert the GpuMat to Mat before making video). Problem is that when running my program I get the following errors:

OpenCV: FFMPEG: tag 0x00000000/'????' is not supported with codec id 14 and format 'mp4 / MP4 (MPEG-4 Part 14)'

(openCVDemosaic:6610): GStreamer-CRITICAL **: 22:39:53.697: gst_element_make_from_uri: assertion 'gst_uri_is_valid (uri)' failed

(openCVDemosaic:6610): GStreamer-CRITICAL **: 22:39:53.783: gst_element_make_from_uri: assertion 'gst_uri_is_valid (uri)' failed
[ WARN:0] global /home/nvidia/dan/installOCV411/opencv-4.1.1/modules/videoio/src/cap_gstreamer.cpp (1422) open OpenCV | GStreamer warning: error opening writer pipeline: syntax error

My videoWriter code:

char* outputPipe = "appsrc !  'video/x-raw, format=(string)GRAY8, width=(int)1936, height=(int)1216' ! omxh264enc ! 'video/x-h264, stream-format=(string)byte-stream' ! h264parse ! qtmux ! filesink location=test.mp4";
    cv::VideoWriter writer(outputPipe, 0, (float)FPS, cv::Size(width,height));

I suppose there is something wrong with my Gstreamer pipeline. What could that be?

My system is Jetson TX2 with the latest Jetpack. OpenCV 4.1.1 with QT4

Maybe you could start with resolving below error first.

OpenCV: FFMPEG: tag 0x00000000/'????' is not supported with codec id 14 and format 'mp4 / MP4 (MPEG-4 Part 14)'

https://stackoverflow.com/questions/46477123/writing-video-in-mp4-format-with-opencv-python

Thanks. I changed the output file to .avi. This removed the error. What is left are these:

(openCVDemosaic:24962): GStreamer-CRITICAL **: 10:49:12.507: gst_element_make_from_uri: assertion 'gst_uri_is_valid (uri)' failed

(openCVDemosaic:24962): GStreamer-CRITICAL **: 10:49:12.596: gst_element_make_from_uri: assertion 'gst_uri_is_valid (uri)' failed
[ WARN:0] global /home/nvidia/dan/installOCV411/opencv-4.1.1/modules/videoio/src/cap_gstreamer.cpp (1422) open OpenCV | GStreamer warning: error opening writer pipeline: syntax error

These same error appear if I use the example given in “ACCELERATED GSTREAMER USER GUIDE 32.1”:

gst-launch-1.0 videotestsrc ! \
 'video/x-raw, format=(string)I420, width=(int)640, \
 height=(int)480' ! omxh264enc ! \
 'video/x-h264, stream-format=(string)byte-stream' ! h264parse ! \
 qtmux ! filesink location=test.mp4

In my cpp code it looks like this:

char* outputPipe = "videotestsrc ! 'video/x-raw, format=(string)I420, width=(int)640, height=(int)480' ! omxh264enc ! 'video/x-h264, stream-format=(string)byte-stream' ! h264parse ! qtmux ! filesink location=test.avi";

The mp4 error disappears also when changing to .avi.

I got it working!! Sort of…

First I needed to convert my 8UC1 GpuMat to 8UC3 GpuMat with this command:

cuda::cvtColor(gpu_mat_raw, gpu_mat_color, cv::COLOR_GRAY2BGR);

Then I found this thread that had a suitable gstreamer + OCV command line:

const char* outputPipe = " appsrc num-buffers=750 ! video/x-raw, framerate=(fraction)150/1, format=(string)BGR ! \
			      videoconvert ! video/x-raw, format=(string)I420 ! \
 		              nvvidconv ! video/x-raw(memory:NVMM) ! \
			      omxh264enc ! h264parse ! matroskamux ! filesink location=test.mkv ";

Now my program starts to make a video as it should. But the outputted .mkv video file is corrupt. The length is correct but videoplayer shows only about 2 frames during 5 seconds and the video is supposed to be 150 fps = 750 frames. My guess is that the video never receives an EOS and therefore the .mkv file never gets wrapped up correctly. What do you think?

So the question is: Is there some good way to tell the video when to stop? I get a terminal error if I give -e flag to gstreamer pipe. In the end of my code I have “writer.release()” to release the OCV VideWriter. But this does not seem to give an EOS to GStreamer. I have also included “num-buffers = 750” in the Gstreamer command but it does not help.

Hi,
We have OpenCV + gstreamer samples in source code package:
https://developer.nvidia.com/embedded/dlc/r32-3-1_Release_v1.0/Sources/T186/public_sources.tbz2

Please download the package and take a look. Thanks.