Gstreamer opencv videowriter

Robotics & Edge Computing Jetson & Embedded Systems Jetson AGX Xavier
1

Hello,

I am trying to construct a video pipeline with opencv-python and gstreamer, however I cannot correctly display the output, while in opencv the output looks fine with imshow…
my pipeline is with a usb camera:

gst_in= ‘v4l2src device=/dev/video1 ! video/x-raw,format=(string)YUY2,width=640,height=480,framerate=30/1 ! videorate ! video/x-raw,framerate=30/1 ! nvvidconv ! video/x-raw(memory:NVMM) ! nvvidconv ! video/x-raw,format=BGRx ! videoconvert ! video/x-raw, format=BGR ! queue silent=1 max-size-buffers=30 ! appsink’
cam= cv2.VideoCapture(gst_in, cv2.CAP_GSTREAMER)
###process frames
gst_out= 'appsrc ! videoconvert ! xvimagesink ’
out = cv2.VideoWriter(gst_out, cv2.CAP_GSTREAMER, 0, 30, (416,416))

I get the same result using nveglglessink

imagen
imagen998×796 121 KB

you can actually kind of see the background of the camera in the output, I am pretty sure it is a timing issue, so far I have not been able to solve it, the only thing that works is writing to a file with out=cv2.VideoWriter(“test.avi”,cv2.CAP_GSTREAMER, 0, 30, (416,416)). If I just launch the command from the terminal removing appsrc and appsink it works as well. I have tried to write frames both in BGR and RGB with the same result.

any idea why I’m getting this display with both xvimagesink and nveglglessink?
Thanks

2

No idea how this works… with GSTREAMER backend, a pipeline string is expected, but just a file is not appropriate for that… probably it switched to another backend, presumably FFMPEG.

The issue may be the size of the frames you’re pushing. I don’t know your processing, but you’re reading 640x480 frames, and unless your processing resizes it, it doesn’t match the VideoWriter expecting 416x416.

You may try this pipeline, using HW for converting BGR into YUY2 while rescaling:

gst_out= 'appsrc ! video/x-raw,format=BGR ! queue ! videoconvert ! video/x-raw,format=BGRx ! nvvidconv ! video/x-raw(memory:NVMM),width=416,height=416 ! nvvidconv ! video/x-raw,format=YUY2 ! xvimagesink’
out = cv2.VideoWriter(gst_out, cv2.CAP_GSTREAMER, 0, float(30), (640,480))
3

Hello,
thank you for your answer, basically i resize the frames detect objects in them and write them out with videowriter, so thats why they are in that resolution. I have tried your suggestion and again it works fine launched from the terminal with no appsrc or appsink.

Ive also realized that I get a warning from the opencv wrapper when I start to read frames, I can read them show them and process them with opencv nonetheless, the warning is coming from here:

status_ = gst_element_query_position(sink, CV_GST_FORMAT(format_), &value_);
if (!status_ || value_ != 0 || duration < 0)
{
CV_WARN(“Cannot query video position: status=” << status_ << “, value=” << value_ << “, duration=” << duration);
isPosFramesSupported = false;
isPosFramesEmulated = true;
emulatedFrameNumber = 0;
}
else
isPosFramesSupported = true;

status=0, value=duration=-1, so it pops that warning. It is the only warning I get, no other errors or warnings that print anything at least. Could this be the cause?
thanks!

4

This warning is normal and harmless, a live source has no duration.

It is unclear what works or not.
Be sure that your opencv version has gstreamer support. You can use function cv::getBuildInformation() that returns a string and check for the line with GSTREAMER…should be YES.

[EDIT: If gstreamer support from OpenCV is ok, be also sure that frames you’re pushing are in BGR format (BGRx or RGBA may give the pattern as in your original post). Also be sure to push one frame for each 33ms period. You would have a main loop after VideoCapture and VideoWriter are opened with:

  1. Read frame from capture
  2. Process 640x480 frame into a single 416x416 frame
  3. Push it into VideoWriter
  4. Optionally call cv::waitKey(1) This is from opencv highgui module.

Where this loop takes not more than 33ms.

Better share some simple code snippet for reproducing your issue easily.
]

5

Hi,

I think I solved it, the problem is I was being a dumbass and exiting immediately after seeing that output, however after 4 or 5 seconds it outputs the correct image, although it does go a bit slow (Because I am taking more than 33 ms to detect things…?).

The whole point of this was trying to determine what is most efficient thing in image processing pipeline with object detection, because as I understand a lot of cv2 operations happen in cpu and I would like to distribute, so for example in this code I use cuda wrapper to resize instead of cpu, but I am not sure if it is faster with the jetson HW or upload and download from gpu. Also I am using jetson_clocks command

I am resizing in cuda like this after reading frame:

gpu_frame = cv2.cuda_GpuMat()
gpu_frame.upload(frame)
resized_net = cv2.cuda.resize(gpu_frame, (416,416), interpolation=cv2.INTER_CUBIC)
resized = resized_net.download()

after this detect objects in resized frame, I am using darknet with yolo but you can use tensorrt pytorch or whatever, then write them to videowriter. Since it work after a few seconds I am not sure if this is issue anymore…
thank you for your help!

6

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.