Something wrong when using opencv with gstream to display video in hdmi because set_mode=1

I want to get the stream frame from one usb camera, and use the gstreamer framework to display the stream into the hdmi in real time and also save the stream into files.

So I write the code bellow, and it run well in the phase before stream.release() is called(I call the function to generate one size-limited file, and set the max frame count to 1200 for testing), but after , the process is blocked by the stream.write function when the cnt is 1203, and the hdmi screen become nothing displayed.
Cloud anybody help me to troubleshotting it?

import cv2

g_test = 1
print(cv2.__version__)
cap = cv2.VideoCapture('/dev/video8', cv2.CAP_V4L)

cap.set(cv2.CAP_PROP_FRAME_WIDTH, 640)
cap.set(cv2.CAP_PROP_FRAME_HEIGHT, 360)
cap.set(cv2.CAP_PROP_FPS, 20)           

cnt = 0
gst_video_writer = " t. ! queue ! videoconvert ! video/x-raw,format=BGRx ! nvvidconv ! nvv4l2h264enc profile=4 bitrate=16384000 ! h264parse ! matroskamux ! filesink location=./test_"+str(cnt)+".avi"
gst_out_hdmi = " t. ! timeoverlay ! textoverlay name=txt0 shaded-background=false valignment=top halignment=right text="+str("Road") +" ! videoconvert ! nvvidconv ! nvdrmvideosink max-bitrate=16384000 set_mode=1 sync=false"
gst_str= "appsrc ! tee name=t "+gst_video_writer+gst_out_hdmi
print(gst_str)
stream = cv2.VideoWriter(gst_str, cv2.CAP_GSTREAMER, 0, 20, (640,360))
stream.set(cv2.VIDEOWRITER_PROP_QUALITY,100)

max_cnt_per_one_video = 20*60

while True:
    ret, frame_read = cap.read()
    cnt = cnt +1 
    print('get frame',frame_read.shape,cnt)
    stream.write(frame_read)
    print('writen frame',frame_read.shape,cnt)
    if (cnt % (max_cnt_per_one_video) ==0):
        stream.release()
        gst_str= "appsrc ! tee name=t "+gst_video_writer+gst_out_hdmi
        print("get new str",gst_str)
        stream = cv2.VideoWriter(gst_str, cv2.CAP_GSTREAMER, 0, 20, (640,360))

The output as bellow:

get frame (360, 640, 3) 1196

writen frame (360, 640, 3) 1196

get frame (360, 640, 3) 1197

writen frame (360, 640, 3) 1197

get frame (360, 640, 3) 1198

writen frame (360, 640, 3) 1198

get frame (360, 640, 3) 1199

writen frame (360, 640, 3) 1199

get frame (360, 640, 3) 1200

writen frame (360, 640, 3) 1200

get new str appsrc ! tee name=t t. ! queue ! videoconvert ! video/x-raw,format=BGRx ! nvvidconv ! nvv4l2h264enc profile=4 bitrate=16384000 ! h264parse ! matroskamux ! filesink location=./test_0.avi t. ! timeoverlay ! textoverlay name=txt0 shaded-background=false valignment=top halignment=right text=Road ! videoconvert ! nvvidconv ! nvdrmvideosink max-bitrate=16384000 set_mode=1 sync=false

Opening in BLOCKING MODE

get frame (360, 640, 3) 1201

writen frame (360, 640, 3) 1201

NvMMLiteOpen : Block : BlockType = 4

===== NVMEDIA: NVENC =====

get frame (360, 640, 3) 1202

NvMMLiteBlockCreate : Block : BlockType = 4

(python3:154730): GStreamer-**CRITICAL** **: 15:47:41.208: gst_mini_object_copy: assertion 'mini_object != NULL' failed

(python3:154730): GStreamer-**CRITICAL** **: 15:47:41.208: gst_caps_get_structure: assertion 'GST_IS_CAPS (caps)' failed

(python3:154730): GStreamer-**CRITICAL** **: 15:47:41.208: gst_structure_copy: assertion 'structure != NULL' failed

(python3:154730): GStreamer-**CRITICAL** **: 15:47:41.208: gst_caps_append_structure_full: assertion 'GST_IS_CAPS (caps)' failed

(python3:154730): GStreamer-**CRITICAL** **: 15:47:41.208: gst_caps_get_structure: assertion 'GST_IS_CAPS (caps)' failed

(python3:154730): GStreamer-**CRITICAL** **: 15:47:41.208: gst_structure_copy: assertion 'structure != NULL' failed

(python3:154730): GStreamer-**CRITICAL** **: 15:47:41.208: gst_caps_append_structure_full: assertion 'GST_IS_CAPS (caps)' failed

(python3:154730): GStreamer-**CRITICAL** **: 15:47:41.208: gst_mini_object_copy: assertion 'mini_object != NULL' failed

(python3:154730): GStreamer-**CRITICAL** **: 15:47:41.208: gst_caps_get_structure: assertion 'GST_IS_CAPS (caps)' failed

(python3:154730): GStreamer-**CRITICAL** **: 15:47:41.208: gst_structure_copy: assertion 'structure != NULL' failed

(python3:154730): GStreamer-**CRITICAL** **: 15:47:41.208: gst_caps_append_structure_full: assertion 'GST_IS_CAPS (caps)' failed

(python3:154730): GStreamer-**CRITICAL** **: 15:47:41.208: gst_caps_get_structure: assertion 'GST_IS_CAPS (caps)' failed

(python3:154730): GStreamer-**CRITICAL** **: 15:47:41.208: gst_structure_copy: assertion 'structure != NULL' failed

(python3:154730): GStreamer-**CRITICAL** **: 15:47:41.208: gst_caps_append_structure_full: assertion 'GST_IS_CAPS (caps)' failed

color_range if supported, to be set to default by DRM

color_range if supported, to be set to default by DRM

writen frame (360, 640, 3) 1202

color_range if supported, to be set to default by DRM

get frame (360, 640, 3) 1203

H264: Profile = 100, Level = 0

NVMEDIA: Need to set EMC bandwidth : 84000

NVMEDIA_ENC: bBlitMode is set to TRUE

color_range if supported, to be set to default by DRM

color_range if supported, to be set to default by DRM

color_range if supported, to be set to default by DRM

color_range if supported, to be set to default by DRM

You may try creating 2 separate writers, so that one could still run when the other one is stopped.

But I only has one appsrc,if I want to create two writer , I must create two appsrc for example use another queue?

Try this code that would continuously display in an X window, and record the first 10s (300 frames at 30 fps) encoded as H264 into avi container :

import cv2

cap = cv2.VideoCapture("videotestsrc ! videoconvert ! video/x-raw,width=640,height=480,framerate=30/1,format=BGR ! appsink drop=1", cv2.CAP_GSTREAMER)

if not cap.isOpened():
   print('failed to open video capture')
   exit(-1)

xvImgWriter = cv2.VideoWriter("appsrc ! video/x-raw,format=BGR ! queue ! videoconvert ! xvimagesink", cv2.CAP_GSTREAMER, 0, 30.0, (640,480))
if not xvImgWriter.isOpened():
   print('failed to open xvimagesink writer')
   cap.release()
   exit(-2)

fileWriter = cv2.VideoWriter("appsrc ! queue ! videoconvert ! video/x-raw,format=BGRx ! nvvidconv ! nvv4l2h264enc profile=4 bitrate=16384000 ! h264parse ! avimux ! filesink location=./test.avi", cv2.CAP_GSTREAMER, 0, 30.0, (640,480))
if not fileWriter.isOpened():
   print('failed to open file writer')
   cap.release()
   xvImgWriter.release()
   exit(-3)

max_frame_cnt=300
frames=0
while True:
   ret_val, img = cap.read();
   if not ret_val:
      print('Failed to read, aborting')
      break
   frames = frames + 1

   xvImgWriter.write(img)
   
   if frames < max_frame_cnt :
      fileWriter.write(img)
   else :
      if frames == max_frame_cnt :
         print('Stop recording')
         fileWriter.release()
	     	
   if cv2.waitKey(1) == ord('q'):
      break
 
xvImgWriter.release()
if fileWriter.isOpened():
   fileWriter.release()
cap.release()

Thanks, I will have a try tomorrow, it is 11:00 PM in my timezone.

But why I could not just release the hdmi-file combined writer and construct a new one writer?

Re-looking at that, maybe the cause is a missing queue for the second subpipeline:

" t. ! queue ! timeoverlay ! ..

If change to " t. ! queue ! timeoverlay", the second round also failed. But the errors bellow are outputed.
[ WARN:0@61.726] global /usr/local/robu/source/opencv-4.5.5/modules/videoio/src/cap_gstreamer.cpp (2401) handleMessage OpenCV | GStreamer warning: Embedded video playback halted; module appsrc1 reported: Internal data stream error.

(python3:5317): GStreamer-**CRITICAL** **: 07:37:23.667: gst_caps_get_structure: assertion 'GST_IS_CAPS (caps)' failed

(python3:5317): GStreamer-**CRITICAL** **: 07:37:23.667: gst_structure_copy: assertion 'structure != NULL' failed

(python3:5317): GStreamer-**CRITICAL** **: 07:37:23.667: gst_caps_append_structure_full: assertion 'GST_IS_CAPS (caps)' failed

(python3:5317): GStreamer-**CRITICAL** **: 07:37:23.667: gst_caps_get_structure: assertion 'GST_IS_CAPS (caps)' failed

(python3:5317): GStreamer-**CRITICAL** **: 07:37:23.667: gst_structure_copy: assertion 'structure != NULL' failed

(python3:5317): GStreamer-**CRITICAL** **: 07:37:23.667: gst_caps_append_structure_full: assertion 'GST_IS_CAPS (caps)' failed

get frame (360, 640, 3) 1203

writen frame (360, 640, 3) 1203

color_range if supported, to be set to default by DRM

get frame (360, 640, 3) 1204

writen frame (360, 640, 3) 1204

color_range if supported, to be set to default by DRM

color_range if supported, to be set to default by DRM

H264: Profile = 100, Level = 0

NVMEDIA: Need to set EMC bandwidth : 84000

NVMEDIA_ENC: bBlitMode is set to TRUE

color_range if supported, to be set to default by DRM

get frame (360, 640, 3) 1205

writen frame (360, 640, 3) 1205

get frame (360, 640, 3) 1206

[ WARN:0@61.726] global /usr/local/robu/source/opencv-4.5.5/modules/videoio/src/cap_gstreamer.cpp (2401) handleMessage OpenCV | GStreamer warning: Embedded video playback halted; module appsrc1 reported: Internal data stream error.

[ WARN:0@61.738] global /usr/local/robu/source/opencv-4.5.5/modules/videoio/src/cap_gstreamer.cpp (2293) writeFrame OpenCV | GStreamer warning: Error pushing buffer to GStreamer pipeline

writen frame (360, 640, 3) 1206

get frame (360, 640, 3) 1207

[ WARN:0@61.778] global /usr/local/robu/source/opencv-4.5.5/modules/videoio/src/cap_gstreamer.cpp (2293) writeFrame OpenCV | GStreamer warning: Error pushing buffer to GStreamer pipeline

writen frame (360, 640, 3) 1207

get frame (360, 640, 3) 1208

[ WARN:0@61.826] global /usr/local/robu/source/opencv-4.5.5/modules/videoio/src/cap_gstreamer.cpp (2293) writeFrame OpenCV | GStreamer warning: Error pushing buffer to GStreamer pipeline

writen frame (360, 640, 3) 1208

get frame (360, 640, 3) 1209

[ WARN:0@61.878] global /usr/local/robu/source/opencv-4.5.5/modules/videoio/src/cap_gstreamer.cpp (2293) writeFrame OpenCV | GStreamer warning: Error pushing buffer to GStreamer pipeline

writen frame (360, 640, 3) 1209

I changed the set-mode to the default value 0, and the problem is sovled, the program can be executed normally, although I don’t know the real meaning of the parameter.
I just use the set-mode=1 to make the display of hdmi to become full-screen.
So if changed to the default value, I have to use the parameters bellow,to make it full-screen.
gst_out_hdmi = " t. ! timeoverlay ! textoverlay name=txt0 shaded-background=false valignment=top halignment=right text=“+str(“Road”) +” ! videoconvert ! nvvidconv ! video/x-raw(memory:NVMM), format=RGBA , width=1920, height=1080 ! nvdrmvideosink max-bitrate=16384000 set_mode=1 sync=false"

set-mode            : Selects whether user wants to choose the default mode which is 
                        already set by connector (set_mode = 0) or wants to select the mode 
                        of the video stream (set_mode = 1). In the latter case, error is 
                        thrown when the input stream resolution does not match with 
                        the supported modes of the connector.

But when I reboot the machine with no hdmi device connected , the progrom which has been changed to set_mode=0, goes to error again, this time, it shows “double free detected in tcache 2”.
If I reboot the machine with one hdmi device, it will ok, even if I unplug the hdmi device.

4.5.5

appsrc ! tee name=t t. ! queue ! videoconvert ! video/x-raw,format=BGRx ! nvvidconv ! nvv4l2h264enc profile=4 bitrate=16384000 ! h264parse ! matroskamux ! filesink location=./test_0.avi t. ! timeoverlay ! textoverlay name=txt0 shaded-background=false valignment=top halignment=right text=Road ! videoconvert ! nvvidconv ! nvdrmvideosink max-bitrate=16384000 set_mode=0 sync=false

Invalid connector id

double free or corruption (fasttop)

Aborted (core dumped)

The problem has been posted in When the hdmi not be connected, nvdrmvideosink double free happen.

But when I reboot the machine with no hdmi device connected , the progrom which has been changed to set_mode=0, goes to error again, this time, it shows “double free detected in tcache 2”.
If I reboot the machine with one hdmi device, it will ok, even if I unplug the hdmi device.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.