Deteriorating GStreamer pipeline performance

denscake · March 25, 2021, 12:22pm

GStreamer pipeline performs ok when just launched, but performance quickly deteriorates, after 10 min it freezes completely.

HW: IMX378 connected via 2-lane CSI.
Jetson Nano B01 dev kit.
Cooling fan ()

SW: JetPack 4.3? Ubuntu LTS 18.04
IMX378 driver, I compiled myself following the instruction provided by the manufacturer. (as far as I can tell there are no problems with that)
The relevant sensor profile is 1920*1080p@120
GStreamer 1.0

What this pipeline has to do: display 1080p@25 (initially @50) through HDMI (accomplished with nvoverlaysink) and push 1080p@100 through RTMP to another computer (initially to the Nginx RTMP on the Jetson itself) so that the video stream can be used by streaming software (vMix). VLC works well for tests.

The pipeline:

gst-launch-1.0 nvarguscamerasrc sensor-id=0 wbmode=9 awblock=false aelock=false \
! "video/x-raw(memory:NVMM), width=(int)1920, height=(int)900, framerate=(fraction)100/1" ! nvvidconv flip-method=2 \
! queue ! tee name=t ! queue ! videorate \
! "video/x-raw, framerate=(fraction)25/1" \
! nvvidconv ! nvoverlaysink sync=0 t. \
! queue ! omxh264enc bitrate=10000000 ! flvmux streamable=true \
! rtmpsink location='rtmp://192.168.88.190:1935/live/live live=1'

Commands performed before the pipeline is started:

sudo nvpmodel -m 0
sudo jetson_clocks
sudo xrandr --output HDMI-0 --mode 1920x1080 --rate 50

Video from pipeline start to the freeze: https://youtu.be/0DpZVCXbXvM

What I did wrong in this pipeline? How can I fix this?

Additional question: how to fix greenish image through nvoverlaysink?

DaneLLL · March 26, 2021, 2:14am

Hi,
Since we have deprecated omx plugins, please replace omxh264enc with nvv4l2h264enc and try again.

denscake · March 26, 2021, 10:37am

Hello,
Since you can’t connect nvv4l2h264enc to flvmux directly I modified the pipeline the following way (according to your recommendation in another thread)

gst-launch-1.0 nvarguscamerasrc sensor-id=0 wbmode=9 awblock=false aelock=false \
! "video/x-raw(memory:NVMM), width=(int)1920, height=(int)900, framerate=(fraction)100/1" ! nvvidconv flip-method=2 \
! queue ! tee name=t ! queue ! videorate \
! "video/x-raw, framerate=(fraction)25/1" \
! nvvidconv ! nvoverlaysink sync=0 t. \
! queue ! nvv4l2h264enc bitrate=10000000 ! h264parse ! flvmux streamable=true \
! rtmpsink location='rtmp://192.168.88.190:1935/live/live live=1'

But I’m getting the following output

Setting pipeline to PAUSED ...
Opening in BLOCKING MODE 
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected...
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 4032 x 3040 FR = 20,000000 fps Duration = 50000000 ; Analog Gain range min 1,000000, max 22,000000; Exposure Range min 1000, max 65515000;

GST_ARGUS: 3840 x 2160 FR = 29,999999 fps Duration = 33333334 ; Analog Gain range min 1,000000, max 22,000000; Exposure Range min 1000, max 65515000;

GST_ARGUS: 1920 x 1080 FR = 120,000005 fps Duration = 8333333 ; Analog Gain range min 1,000000, max 22,000000; Exposure Range min 1000, max 65515000;

GST_ARGUS: 640 x 480 FR = 240,000038 fps Duration = 4166666 ; Analog Gain range min 1,000000, max 22,000000; Exposure Range min 1000, max 65515000;

GST_ARGUS: Running with following settings:
   Camera index = 0 
   Camera mode  = 2 
   Output Stream W = 1920 H = 1080 
   seconds to Run    = 0 
   Frame Rate = 120,000005 
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.
ERROR: from element /GstPipeline:pipeline0/GstNvArgusCameraSrc:nvarguscamerasrc0: Internal data stream error.
Additional debug info:
gstbasesrc.c(3055): gst_base_src_loop (): /GstPipeline:pipeline0/GstNvArgusCameraSrc:nvarguscamerasrc0:
streaming stopped, reason not-negotiated (-4)
Execution ended after 0:00:00.639547698
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
CONSUMER: Done Success
GST_ARGUS: Cleaning up
GST_ARGUS: Done Success
Setting pipeline to NULL ...
Freeing pipeline ...

denscake · March 27, 2021, 1:48am

I changed properties in original pipeline for rtmpsink to constant bitrate and async false, nvoverlaysink to async false.
It now works between 1 and 80 minutes before freeze without deteriorating framerate, but I’d like it to run indefinitely.

Gst debug levels 1,2,3 and 4 don’t display anything, 5,6,7 make it work at “seconds per frame” performance levels.

The only clue is
nvoverlaysink0: A lot of buffers are being dropped.
There may be a timestamping problem, or this computer is too slow.

denscake · March 28, 2021, 5:44am

I made a slightly different pipeline. I’ll update after testing.

denscake · March 31, 2021, 9:19am

Tested for 18 hours, works “good enough” if not perfect. Can’t really check the fps on both HDMI and RTMP (RTMP reports 1000 FPS, checked with a metronome, it’s spot-on 120. And no real way to know what HDMI is showing)

sudo nvpmodel -m 0 #set the highest power setting
sudo jetson_clocks #overclock the thing
#sudo xrandr --output HDMI-0 --mode 1920x1080 --rate 50 #set HDMI to 1080p50



gst-launch-1.0 nvarguscamerasrc sensor-id=0 wbmode=9 awblock=false aelock=false \ #read from the IMX378
! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, framerate=(fraction)120/1, format=NV12' \ #120 fps FullHD stream
! nvvidconv flip-method=2 ! tee name=t ! queue \
\ #flip the image
! nvv4l2h264enc bitrate=1000000 ! h264parse ! flvmux streamable=true \ #encode for streaming
! queue max-size-buffers=5 max-size-time=10000000 leaky=upstream ! rtmpsink location='rtmp://192.168.88.190:1935/live/live live=1' \ #stream
\ #strange queue configuration but it works so far
t. ! videorate ! 'video/x-raw(memory:NVMM), framerate=50/1' \ #we need only 50 fps at HDMI
! queue max-size-buffers=5 max-size-time=10000000 leaky=upstream \ #another super strange queue 
! nvvidconv flip-method=2 ! nvvidconv flip-method=2 ! nvoverlaysink async=false
#you actually need to flip image at this point since it will screen tear so much it's unusabe
#flip back so it's the right way up
#TODO: change bitrate so it makes sense

That’s it! Thanks to everybody who didn’t help and special thanks to DaneLLL for not responding on how to retrofit nvv4l2h264enc because it’s not obvious.

DaneLLL · April 8, 2021, 6:44am

Hi,
For high fps case, we suggest run VIC engine at max clocks, please refer to
Camera's frame rate unstable - #24 by DaneLLL
And you can get maximum throughput of nvvidconv plugin. And set the proerty in nvv4l2h264enc:

  maxperf-enable      : Enable or Disable Max Performance mode
                        flags: readable, writable, changeable only in NULL or READY state
                        Boolean. Default: false

In the default mode, two CPU cores are online. Please execute sudo nvpmodel -m 0 and sudo tegrastats, to get all cores online and at max clocks.