Cannot record video with IMX708 while simultaneously displaying the screen by using Gstreamer in OpenCV

Hi all.

I have some trouble to record a video with IMX708.
I successfully recored the video without displaying the screen by following pipeline.

Now I wanted to be able to see the screen while recording, so I added tee element to separate the tasks and utilized openCV for displaying the screen.
However, the pipeline didn’t work (I couldn’t see the screen, and the video wasn’t recorded either). Also the terminal didn’t show any errors. It just stopped.
Strangely, I’ve succeeded in displaying images and recording video simultaneously at IMX219 and IMX477 sensors. Could you give me some advice?

This is my pipeline.

def pip_video(exp,name):
    return "nvarguscamerasrc wbmode=0 aelock=true awblock=true sensor-id=0 ispdigitalgainrange=\"2 2\"  gainrange=\"16 16\" ! video/x-raw(memory:NVMM), width=(int)4608, height=(int)2592,format=(string)NV12, framerate=(fraction)14/1, exposurecompensation=0 ! tee name=t ! queue ! nvvidconv ! video/x-raw, width=1920,height=1080 ! x264enc ! qtmux ! filesink location=test.mp4 t.! queue ! nvvidconv ! video/x-raw, format=(string)I420, width=4608, height=2592 ! appsink sync=0 ";

This is my terminal

nvbuf_utils: Could not get EGL display connection
GST_ARGUS: NvArgusCameraSrc: Setting ISP Digital Gain Range : 2 2
GST_ARGUS: NvArgusCameraSrc: Setting Gain Range : 16 16
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected...
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 4608 x 2592 FR = 14.000001 fps Duration = 71428568 ; Analog Gain range min 1.000000, max 16.000000; Exposure Range min 500000, max 65487000;

GST_ARGUS: Running with following settings:
   Camera index = 0 
   Camera mode  = 0 
   Output Stream W = 4608 H = 2592 
   seconds to Run    = 0 
   Frame Rate = 14.000001 
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.

Is there anyone who can help to solve this issue?

hello riul5627k,

it looks like display issue according to the failure messages…

please try setup the environment variable, for instance, $ export DISPLAY=:0
you may also execute xrandr to dump all those display supported resolutions for double check.
for example,

$ xrandr
Screen 0: minimum 8 x 8, current 1920 x 1080, maximum 32767 x 32767
DP-0 disconnected (normal left inverted right x axis y axis)
DP-1 connected primary 1920x1080+0+0 (normal left inverted right x axis y axis) 477mm x 268mm
   1920x1080     60.00*+  59.94    50.00    29.97  

Hi JerryChang. I appreciate your advice.

However, I don’t believe that would be the issue. I can see the screen when I use the following pipeline.

def pip_video(exp):
    return "nvarguscamerasrc wbmode=0 aelock=true awblock=true sensor-id=0 exposuretimerange = \""+ str(exp)+" "+str(exp)+"\" ispdigitalgainrange=\"2 2\"  gainrange=\"16 16\" ! video/x-raw(memory:NVMM), width=(int)4608, height=(int)2592,format=(string)NV12, framerate=(fraction)14/1, exposurecompensation=0 ! nvvidconv ! videobalance brightness=0 saturation=1 contrast=1 ! video/x-raw, format=(string)I420, width=4608, height=2592 ! appsink sync=0 ";

The upper pipeline also displays the same EGL display connection error message. But it shows the screen well in real time.

Hi,
Please try the command like:
Problems using gstreamer pipeline in openCV for recording video - #5 by DaneLLL

It should be working in gst-launch-1.0 like:

$ gst-launch-1.0 nvarguscamerasrc ! "video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080,format=(string)NV12, framerate=(fraction)30/1" ! tee name=t t. ! queue ! nvv4l2h264enc ! h264parse ! matroskamux ! filesink location=a.mkv t. ! queue ! nvvidconv ! video/x-raw, format=(string)I420, width=640, height=480 ! xvimagesink sync=0

Hi DaneLLL.
I changed my code and got another error message.

"_ValidateEncodeParams: Invalid encode width "

python test.py
nvbuf_utils: Could not get EGL display connection
Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 4 
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 4 
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected...
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 4608 x 2592 FR = 14.000001 fps Duration = 71428568 ; Analog Gain range min 1.000000, max 16.000000; Exposure Range min 500000, max 65487000;

GST_ARGUS: Running with following settings:
   Camera index = 0 
   Camera mode  = 0 
   Output Stream W = 4608 H = 2592 
   seconds to Run    = 0 
   Frame Rate = 14.000001 
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.
H264: Profile = 66, Level = 0 
_ValidateEncodeParams: Invalid encode width 
tvmrVideoEncoderBitsAvailable_MSENC: ucode ERROR = 16 
NvVideoEncTransferOutputBufferToBlock: DoWork failed line# 674 
NvVideoEnc: NvVideoEncTransferOutputBufferToBlock TransferBufferToBlock failed Line=685

This is my full code.

def pip_video():
    return "nvarguscamerasrc sensor-id=0 ! video/x-raw(memory:NVMM), width=(int)4608, height=(int)2592,format=(string)NV12, framerate=(fraction)14/1 ! tee name=t t. ! queue ! nvv4l2h264enc ! h264parse ! matroskamux ! filesink location=test.mp4 t.! queue ! nvvidconv ! video/x-raw, format=(string)I420, width=640, height=480 ! appsink sync=0 ";

def read_cam():
    cap = cv2.VideoCapture(pip_video())

    if cap.isOpened():
        cv2.namedWindow("demo", cv2.WINDOW_AUTOSIZE)
        while True:
            ret_val, img = cap.read();
            img2 = cv2.cvtColor(img, cv2.COLOR_YUV2BGR_I420);
            img3 = cv2.resize(img2,(1200,900));
            cv2.imshow('demo',img3)
            keyCode = cv2.waitKey(10)
	    # Stop the program on the BS key
            if keyCode == 8:
                break
    else:
     print("camera open failed")

    cv2.destroyAllWindows()

I am uncertain as to why this error is occurring. This code functioned without issue with the IMX477. (The width and height were adjusted to 4032 and 3040, respectively, to accommodate the IMX477.)

Hi,
Please downscale the resolution to 4K and then send to encoder. Hardware encoder does not support resolution above 4K.

Hi DaneLLL

You mean I should change the code "video/x-raw(memory:NVMM), width=(int)4608, height=(int)2592~~ "
to " video/x-raw(memory:NVMM), width=(int)3840, height=(int)2160 "?

But as you know, I cannot use any other options except 4608 x 2592 because the driver doesn’t support other resolutions. In fact, even when I attempt to adjust the resolution to 4k, the camera still operates at 4608 x 2592.

GST_ARGUS: Available Sensor modes :
GST_ARGUS: 4608 x 2592 FR = 14.000001 fps Duration = 71428568 ; Analog Gain range min 1.000000, max 16.000000; Exposure Range min 500000, max 65487000;

Furthermore, the IMX477, which has a resolution of 4032 x 3040, provided me with a live screen while recording videos even beyond 4K resolution. Additionally, video recording operates smoothly with the IMX708, although it does not provide me with a live screen when using the pipeline below.

SENSOR_ID=0 
FRAMERATE=14
gst-launch-1.0 -e nvarguscamerasrc sensor_id=$SENSOR_ID ! 'video/x-raw(memory:NVMM),width=4608,height=2592,framerate=$FRAMERATE/1, format=NV12' ! nvvidconv ! "video/x-raw,width=1920,height=1080" ! x264enc ! qtmux ! filesink location=RidgeRun_out.mp4

This issue is of utmost importance to me. I sincerely hope to resolve it.

Hi,
Please use hardware converter nvvidconv plugin to downscale the source to 4K or 1080p. And then send downscaled frames to encoder.

Hi.

I already used the below pipeline with nvvidconv plugin.
Is it what you intended?

The pipeline don’t show me any error messages or screens. It just stopped.

Hi,
Please try

"nvarguscamerasrc sensor-id=0 ! video/x-raw(memory:NVMM), width=(int)4608, height=(int)2592,format=(string)NV12, framerate=(fraction)14/1 ! tee name=t t. ! queue ! nvvidconv ! video/x-raw(memory:NVMM),width=1920,height=1080 ! nvv4l2h264enc ! h264parse ! matroskamux ! filesink location=test.mkv t. ! queue ! nvvidconv ! video/x-raw, format=(string)I420, width=640, height=480 ! appsink sync=0 ";

Hi.

Oh my… now it works…
However, I need to record the video with a full frame.
Is there no way to record it with a full frame?

Hi,
It is constraint of hardware encoder, so please downscale to 3840x2160.

Hi, DaneLLL
Thank you for the advice.

I noticed that the video was recorded at 4608 x 2592 resolution when I utilized your recommended code.

I am currently perplexed about what occurred with the hardware encoder. Does the video, originally in NVMM memory at 4608 x 2592 resolution, undergo downscaling to 3840 x 2160 through the hardware encoder? Is there any data loss when going through the hardware encoder? If so, why was my video stored at 4608 x 2592 resolution?

Could you provide an explanation?
Thank you for your assistance.

Hi,
In the pipeline, the frames are downscaled to 1920x1080 and then send to encoder:

... ! nvvidconv ! video/x-raw(memory:NVMM),width=1920,height=1080 ! nvv4l2h264enc ! ...

So it doesn’t hit the constraint.

Hi, DaneLLL
Thank you for your answer.

I understand that downscaling frees us from the constraints of the encoder.
However, I am concerned about potential data loss during the downscaling process.

1. Can you explain the principle behind downscaling? Does it simply resize the image from 4608 x 2592 to 1920 x 1080, or does it crop the image to 1920 x 1080 from the original 4608 x 2592?

If the hardware encoder cannot handle 4608 x 2592 resolution, I would prefer to crop the image to 2592 x 2592 instead of downsizing it. As shown in the image below:

image

2. Is there a way to crop only the desired portion and input it to the hardware encoder?

Hi,
Please set the properties of nvvidconv plugin to crop the frame:

  left                : Pixels to crop at left
                        flags: readable, writable
                        Integer. Range: 0 - 2147483647 Default: 0
  right               : Pixels to crop at right
                        flags: readable, writable
                        Integer. Range: 0 - 2147483647 Default: 0
  top                 : Pixels to crop at top
                        flags: readable, writable
                        Integer. Range: 0 - 2147483647 Default: 0
  bottom              : Pixels to crop at bottom
                        flags: readable, writable
                        Integer. Range: 0 - 2147483647 Default: 0
1 Like

Hi DaneLLL.
Thank you so much.
Now it perfectly works. Thank you.