Usage of tee to split encoder output

Hi Folks,

  1. I am looking to build this pipeline -

camera_rd --> encode --> tee t1 --> queue1 --> filesink t1. --> queue2 --> decode --> opencv processing

  1. First I am trying to accomplish -

camera_rd --> encode --> tee t1 --> queue1 --> filesink t1. --> queue2 --> playbin --> nveglglessink

  1. I have following pipeline that is not working -

ubuntu@tegra-ubuntu:~/tegra_multimedia_api/argus/samples$ gst-launch-1.0 nvcamerasrc fpsRange=“30.0 30.0” ! ‘video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)30/1’ ! omxh265enc bitrate=3000000 ! h265parse! queue name=queenc ! matroskamux name=mux ! tee name=t ! queue ! filesink location=test_file.mkv async=0 t. ! queue ! playbin

WARNING: erroneous pipeline: could not link queue2 to playbin0

Any idea how to connect (last queue) queue to playbin ?

Hi dumbgeorge,
I know you have asked this questions in multiple posts. It is more like an issue in gstreamer framewroks and right now I don’t have enough experience to give good answers. Could you try to get help from http://gstreamer-devel.966125.n4.nabble.com/ ?

You can replace nvcamerasrc with v4l2src, and omxh265enc with SW encoder, to have a pure SW pipeline. Once you get a pure SW pipeline working well, you can replace these back to HW components.

Hi DaneLLL,

My apologies, for bringing it up here. I am lucky to get quick help from Gstreamer community. I do have a command line that works - well almost. Per your suggestion, may be I can ask for help again here - since it is now coming to HW components.

The trailing part of command line - i.e. decoder is able to decode and show few frames on screen - but it seems encoder is getting choked with -

ERROR: from element /GstPipeline:pipeline0/GstOMXH265Enc-omxh265enc:omxh265enc-omxh265enc0: Internal data stream error.

Command line -

gst-launch-1.0 nvcamerasrc fpsRange="30.0 30.0" ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)30/1' ! omxh265enc bitrate=3000000 ! h265parse ! matroskamux name=mux ! tee name=t ! queue ! filesink location=test_file.mkv async=0 t. !  matroskademux name=demux.video_0  !  h265parse ! omxh265dec ! nveglglessink -e

Overall Log -

Setting pipeline to PAUSED ...

Available Sensor modes : 
3864 x 2174 FR=60.000000 CF=0x1009208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10
Pipeline is live and does not need PREROLL ...
Got context from element 'eglglessink0': gst.egl.EGLDisplay=context, display=(GstEGLDisplay)NULL;

NvCameraSrc: Trying To Set Default Camera Resolution. Selected sensorModeIndex = 0 WxH = 3864x2174 FrameRate = 60.000000 ...

Setting pipeline to PLAYING ...
New clock: GstSystemClock
Framerate set to : 30 at NvxVideoEncoderSetParameterNvMMLiteOpen : Block : BlockType = 8 
===== MSENC =====
NvMMLiteBlockCreate : Block : BlockType = 8 
===== NVENC blits (mode: 1) into block linear surfaces =====
NvMMLiteOpen : Block : BlockType = 279 
TVMR: NvMMLiteTVMRDecBlockOpen: 7907: NvMMLiteBlockOpen 
NvMMLiteBlockCreate : Block : BlockType = 279 
TVMR: cbBeginSequence: 1223: BeginSequence  1920x1088, bVPR = 0
TVMR: LowCorner Frequency = 180000 
TVMR: cbBeginSequence: 1622: DecodeBuffers = 3, pnvsi->eCodec = 10, codec = 9 
TVMR: cbBeginSequence: 1693: Display Resolution : (1920x1080) 
TVMR: cbBeginSequence: 1694: Display Aspect Ratio : (1920x1080) 
TVMR: cbBeginSequence: 1762: ColorFormat : 5 
TVMR: cbBeginSequence:1776 ColorSpace = NvColorSpace_YCbCr601
TVMR: cbBeginSequence: 1904: SurfaceLayout = 3
TVMR: cbBeginSequence: 2005: NumOfSurfaces = 10, InteraceStream = 0, InterlaceEnabled = 0, bSecure = 0, MVC = 0 Semiplanar = 1, bReinit = 1, BitDepthForSurface = 8 LumaBitDepth = 8, ChromaBitDepth = 8, ChromaFormat = 5
TVMR: cbBeginSequence: 2007: BeginSequence  ColorPrimaries = 2, TransferCharacteristics = 2, MatrixCoefficients = 2
Allocating new output: 1920x1088 (x 10), ThumbnailMode = 0
OPENMAX: HandleNewStreamFormat: 3464: Send OMX_EventPortSettingsChanged : nFrameWidth = 1920, nFrameHeight = 1088 
TVMR: FrameRate = 1000 
TVMR: NVDEC LowCorner Freq = (576000 * 1024) 
ERROR: from element /GstPipeline:pipeline0/GstOMXH265Enc-omxh265enc:omxh265enc-omxh265enc0: Internal data stream error.
Additional debug info:
/dvs/git/dirty/git-master_linux/external/gstreamer/gst-omx/omx/gstomxvideoenc.c(1351): gst_omx_video_enc_loop (): /GstPipeline:pipeline0/GstOMXH265Enc-omxh265enc:omxh265enc-omxh265enc0:
stream stopped, reason error
EOS on shutdown enabled -- waiting for EOS after Error
TVMR: NvMMLiteTVMRDecDoWork: 6768: NVMMLITE_TVMR: EOS detected
Waiting for EOS...
TVMR: TVMRBufferProcessing: 5723: Processing of EOS 
TVMR: TVMRBufferProcessing: 5800: Processing of EOS Done
Got EOS from element "pipeline0".
EOS received - stopping pipeline...
Execution ended after 0:00:01.870566352
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
TVMR: TVMRFrameStatusReporting: 6369: Closing TVMR Frame Status Thread -------------
TVMR: TVMRVPRFloorSizeSettingThread: 6179: Closing TVMRVPRFloorSizeSettingThread -------------
TVMR: TVMRFrameDelivery: 6219: Closing TVMR Frame Delivery Thread -------------
TVMR: NvMMLiteTVMRDecBlockClose: 8105: Done 
Setting pipeline to NULL ...
Freeing pipeline ...

Hi dumbogeorge,

The problem may not be really with tee. If you launch this pipeline

gst-launch-1.0 -e nvcamerasrc ! 'video/x-raw(memory:NVMM), format=I420, width=640, height=480, framerate=30/1' ! omxh265enc bitrate=3000000 ! h265parse ! h265parse ! omxh265dec ! nveglglessink

you should see it runs fine (same with 1280x720@120fps).

However, adding matroskamux/matroskademux will give many frames dropped:

gst-launch-1.0 -e nvcamerasrc ! 'video/x-raw(memory:NVMM), format=I420, width=640, height=480, framerate=30/1' ! omxh265enc bitrate=3000000 ! h265parse ! matroskamux name=mux.video_0 ! matroskademux name=demux.video_0   ! h265parse ! omxh265dec ! nveglglessink

Adding more load with file saving probably fools gstreamer or a plugin.

tegrastats doesn’t show any bottleneck… when I just play the recorded mkv file through matroskademux, I see much higher number in NVDEC than when both mux/demux are used in the same pipeline, but I am unsure how to interpret this.

This may not be releavant, but with GST_DEBUG=omxh265enc:2, there is a warning

Framerate set to : 30 at NvxVideoEncoderSetParameter0:00:00.226125114  4053       0x5cdd90 WARN              omxh265enc gstomxh265enc.c:146:gst_omx_h265_enc_set_format:<omxh265enc-omxh265enc0> Setting profile/level not supported by component

whatever the framerate or resolution is set to.

Please try
[nvoverlaysink]

gst-launch-1.0 nvcamerasrc fpsRange="30.0 30.0" num-buffers=300 ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)30/1' ! omxh265enc ! h265parse ! tee name=t t. ! queue ! matroskamux ! filesink location=test_file.mkv t. ! h265parse ! omxh265dec ! nvoverlaysink -e

[nveglglessink]

export DISPLAY=:0
gst-launch-1.0 nvcamerasrc fpsRange="30.0 30.0" num-buffers=300 ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)30/1' ! omxh265enc ! h265parse ! tee name=t t. ! queue ! matroskamux ! filesink location=test_file.mkv t. ! h265parse ! omxh265dec ! nveglglessink -e

Hi Honey_Patouceul,

Thanks for help and giving me better command lines. I do see frame drops.

Hi DaneLLL,

Thanks for giving options of different sinks. They both work. I am curious about the tee. The tee has two branches - one goes to file and another goes to decoder. I am wondering what will happen - If I remove display and attach an appsink at the end ? I will try this - but would like to ask a question in this context.

Lets say I attach an appsink at the end (instead of display) - and read frames out of that for opencv processing. If my opencv engine takes much more than 33 milli sec to process a frame (a duration which roughly corresponds to 30 fps rate, at which encoder is encoding) - will the pipeline be ‘backpressured’ to tee ? Will encoder still work at 30 fps and will ‘test_file.mkv’ still play at 30 fps - when played by another decoder ?

Thanks,

I do not study the source code of tee, but I think it works in the concept of reference count. If you have two branches, reference count=2. When the next element returns the buffer, it decreases the reference count. If reference count=0, it returns the buffer back to previous element. So if one branch cannot complete processing each buffer in 33ms, the whole pipeline is affected to run below 30fps.

But probably my understanding is wrong. Others may correct me and share correct flow.