Gstreamer fpsdisplaysink reports double framerate

I am debugging framerate issues the team and I recently noticed in our application. Part of the issue is that the reported framerate does not match in different places and the actual number of frames does not correspond with either of them. Our application uses a high speed Alvium CSI-2 camera attached to a jetson nano and is accessed using python gstreamer bindings. Normally the camera runs at 153.917 fps @1408x1088.

Encoded video shows a framerate of 153.917 fps as reported by vlc in the media information → coded tab and using an online tool such as https://www.metadata2go.com/. 1) The actual number of frames does not correspond with that. 2) Opening the video in matlab or windows explorer properties shows 76.96 fps, ie. exactly half.

To make debugging easier and I went back to cli gstreamer using gst-launch-1.0 and a videotestsrc. Using the pipeline below the fpsdisplay sink displays double the framerate ¯_(ツ)_/¯. Full output attached for reference.
gst-launch-1.0 videotestsrc ! 'video/x-raw, format=(string)GRAY8, framerate=(fraction)153917/1000' ! nvvidconv ! 'video/x-raw(memory:NVMM)' ! nvv4l2vp8enc ! webmmux ! fpsdisplaysink video-sink=fakesink --verbose

nvbuf_utils: Could not get EGL display connection
Setting pipeline to PAUSED ...
Opening in BLOCKING MODE
Opening in BLOCKING MODE 
Pipeline is PREROLLING ...
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstFakeSink:fakesink0: sync = true
/GstPipeline:pipeline0/GstVideoTestSrc:videotestsrc0.GstPad:src: caps = video/x-raw, format=(string)GRAY8, width=(int)320, height=(int)240, framerate=(fraction)153917/1000, multiview-mode=(string)mono, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = video/x-raw, format=(string)GRAY8, width=(int)320, height=(int)240, framerate=(fraction)153917/1000, multiview-mode=(string)mono, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive
/GstPipeline:pipeline0/Gstnvvconv:nvvconv0.GstPad:src: caps = video/x-raw(memory:NVMM), width=(int)320, height=(int)240, framerate=(fraction)153917/1000, multiview-mode=(string)mono, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, format=(string)I420
/GstPipeline:pipeline0/GstCapsFilter:capsfilter1.GstPad:src: caps = video/x-raw(memory:NVMM), width=(int)320, height=(int)240, framerate=(fraction)153917/1000, multiview-mode=(string)mono, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, format=(string)I420
/GstPipeline:pipeline0/nvv4l2vp8enc:nvv4l2vp8enc0.GstPad:src: caps = video/x-vp8, profile=(string)0, width=(int)320, height=(int)240, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)153917/1000, interlace-mode=(string)progressive, colorimetry=(string)bt601, chroma-site=(string)jpeg, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono
/GstPipeline:pipeline0/GstWebMMux:webmmux0.GstMatroskamuxPad:video_0: caps = video/x-vp8, profile=(string)0, width=(int)320, height=(int)240, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)153917/1000, interlace-mode=(string)progressive, colorimetry=(string)bt601, chroma-site=(string)jpeg, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono
Redistribute latency...
NvMMLiteOpen : Block : BlockType = 7 
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 7 
/GstPipeline:pipeline0/nvv4l2vp8enc:nvv4l2vp8enc0.GstPad:sink: caps = video/x-raw(memory:NVMM), width=(int)320, height=(int)240, framerate=(fraction)153917/1000, multiview-mode=(string)mono, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, format=(string)I420
/GstPipeline:pipeline0/GstCapsFilter:capsfilter1.GstPad:sink: caps = video/x-raw(memory:NVMM), width=(int)320, height=(int)240, framerate=(fraction)153917/1000, multiview-mode=(string)mono, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, format=(string)I420
/GstPipeline:pipeline0/Gstnvvconv:nvvconv0.GstPad:sink: caps = video/x-raw, format=(string)GRAY8, width=(int)320, height=(int)240, framerate=(fraction)153917/1000, multiview-mode=(string)mono, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = video/x-raw, format=(string)GRAY8, width=(int)320, height=(int)240, framerate=(fraction)153917/1000, multiview-mode=(string)mono, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive
NVMEDIA_ENC: bBlitMode is set to TRUE 
/GstPipeline:pipeline0/GstWebMMux:webmmux0: streamable = true
/GstPipeline:pipeline0/GstWebMMux:webmmux0.GstPad:src: caps = video/webm
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0.GstGhostPad:sink.GstProxyPad:proxypad0: caps = video/webm
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay.GstPad:src: caps = video/webm(memory:SystemMemory, meta:GstVideoOverlayComposition)
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstFakeSink:fakesink0.GstPad:sink: caps = video/webm(memory:SystemMemory, meta:GstVideoOverlayComposition)
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay.GstPad:video_sink: caps = video/webm
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0.GstGhostPad:sink: caps = video/webm
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstFakeSink:fakesink0: sync = true
/GstPipeline:pipeline0/GstWebMMux:webmmux0.GstPad:src: caps = video/webm, streamheader=(buffer)< 1a45dfa301000000000000104282857765626d0042878102428581021853806701ffffffffffffff1549a96601000000000000562ad7b1830f42404d80a54753747265616d6572206d6174726f736b616d75782076657273696f6e20312e31342e35005741994753747265616d6572204d6174726f736b61206d75786572004461880928b55d5c9a9c001654ae6b0100000000000045ae010000000000003cd7810183810173c5882c7102372337bf3323e383836322f0536e86566964656f00e0010000000000000ab0820140ba81f09a81028686565f56503800 >
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0.GstGhostPad:sink.GstProxyPad:proxypad0: caps = video/webm, streamheader=(buffer)< 1a45dfa301000000000000104282857765626d0042878102428581021853806701ffffffffffffff1549a96601000000000000562ad7b1830f42404d80a54753747265616d6572206d6174726f736b616d75782076657273696f6e20312e31342e35005741994753747265616d6572204d6174726f736b61206d75786572004461880928b55d5c9a9c001654ae6b0100000000000045ae010000000000003cd7810183810173c5882c7102372337bf3323e383836322f0536e86566964656f00e0010000000000000ab0820140ba81f09a81028686565f56503800 >
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay.GstPad:src: caps = video/webm(memory:SystemMemory, meta:GstVideoOverlayComposition), streamheader=(buffer)< 1a45dfa301000000000000104282857765626d0042878102428581021853806701ffffffffffffff1549a96601000000000000562ad7b1830f42404d80a54753747265616d6572206d6174726f736b616d75782076657273696f6e20312e31342e35005741994753747265616d6572204d6174726f736b61206d75786572004461880928b55d5c9a9c001654ae6b0100000000000045ae010000000000003cd7810183810173c5882c7102372337bf3323e383836322f0536e86566964656f00e0010000000000000ab0820140ba81f09a81028686565f56503800 >
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstFakeSink:fakesink0.GstPad:sink: caps = video/webm(memory:SystemMemory, meta:GstVideoOverlayComposition), streamheader=(buffer)< 1a45dfa301000000000000104282857765626d0042878102428581021853806701ffffffffffffff1549a96601000000000000562ad7b1830f42404d80a54753747265616d6572206d6174726f736b616d75782076657273696f6e20312e31342e35005741994753747265616d6572204d6174726f736b61206d75786572004461880928b55d5c9a9c001654ae6b0100000000000045ae010000000000003cd7810183810173c5882c7102372337bf3323e383836322f0536e86566964656f00e0010000000000000ab0820140ba81f09a81028686565f56503800 >
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay.GstPad:video_sink: caps = video/webm, streamheader=(buffer)< 1a45dfa301000000000000104282857765626d0042878102428581021853806701ffffffffffffff1549a96601000000000000562ad7b1830f42404d80a54753747265616d6572206d6174726f736b616d75782076657273696f6e20312e31342e35005741994753747265616d6572204d6174726f736b61206d75786572004461880928b55d5c9a9c001654ae6b0100000000000045ae010000000000003cd7810183810173c5882c7102372337bf3323e383836322f0536e86566964656f00e0010000000000000ab0820140ba81f09a81028686565f56503800 >
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0.GstGhostPad:sink: caps = video/webm, streamheader=(buffer)< 1a45dfa301000000000000104282857765626d0042878102428581021853806701ffffffffffffff1549a96601000000000000562ad7b1830f42404d80a54753747265616d6572206d6174726f736b616d75782076657273696f6e20312e31342e35005741994753747265616d6572204d6174726f736b61206d75786572004461880928b55d5c9a9c001654ae6b0100000000000045ae010000000000003cd7810183810173c5882c7102372337bf3323e383836322f0536e86566964656f00e0010000000000000ab0820140ba81f09a81028686565f56503800 >
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = rendered: 159, dropped: 0, current: 317,04, average: 317,04
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 159, dropped: 0, current: 317,04, average: 317,04
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = rendered: 313, dropped: 0, current: 307,84, average: 312,44
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 313, dropped: 0, current: 307,84, average: 312,44
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = rendered: 467, dropped: 0, current: 307,83, average: 310,91
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 467, dropped: 0, current: 307,83, average: 310,91
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = rendered: 621, dropped: 0, current: 307,84, average: 310,14
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 621, dropped: 0, current: 307,84, average: 310,14
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = rendered: 775, dropped: 0, current: 307,83, average: 309,68
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 775, dropped: 0, current: 307,83, average: 309,68
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = rendered: 929, dropped: 0, current: 307,83, average: 309,37
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 929, dropped: 0, current: 307,83, average: 309,37
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = rendered: 1083, dropped: 0, current: 307,84, average: 309,15
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 1083, dropped: 0, current: 307,84, average: 309,15

I can also add that the muxer seems to be causing the incorrect framerate displayed since running this pipe reports ~153.94 fps avg.
gst-launch-1.0 videotestsrc ! 'video/x-raw, format=(string)GRAY8, framerate=(fraction)153917/1000' ! nvvidconv ! 'video/x-raw(memory:NVMM)' ! nvv4l2vp8enc ! fpsdisplaysink video-sink=fakesink --verbose

Hi,
Looks like the issue is triggered by webmmux. The encoder should output buffers based on frames. Each frame is encoded and put in single buffer. For using videotestsrc, please set is-live=1 or it will generate frames continuously without interval.

Hi @DaneLLL thanks for your reply.

I adjusted the pipe but can only report the same double rate ouput.
gst-launch-1.0 videotestsrc is-live=1 ! 'video/x-raw, format=(string)GRAY8, framerate=(fraction)153917/1000' ! nvvidconv ! 'video/x-raw(memory:NVMM)' ! nvv4l2vp8enc ! webmmux ! fpsdisplaysink video-sink=fakesink --verbose

I also tested using a matroskamux with yet the same double rate.
gst-launch-1.0 videotestsrc is-live=1 ! 'video/x-raw, format=(string)GRAY8, framerate=(fraction)153917/1000' ! nvvidconv ! 'video/x-raw(memory:NVMM)' ! nvv4l2vp8enc ! matroskamux ! fpsdisplaysink video-sink=fakesink --verbose

Hi,
We see correct frmae rate in running:

$ gst-launch-1.0 videotestsrc is-live=1 ! 'video/x-raw, format=(string)GRAY8, framerate=(fraction)153917/1000' ! nvvidconv ! 'video/x-raw(memory:NVMM)' ! nvv4l2vp8enc ! fpsdisplaysink text-overlay=0 video-sink=fakesink sync=0 -v
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendere
d: 78, dropped: 0, current: 155.43, average: 155.43
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendere
d: 155, dropped: 0, current: 153.96, average: 154.70
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendere
d: 232, dropped: 0, current: 153.88, average: 154.42
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendere
d: 309, dropped: 0, current: 153.92, average: 154.30

So the framerate after muxer is not frame rate of the source. For getting correct frame rate please eliminate muxer from the command.

@DaneLLL This not really surprising as I reported that pipe working in my 2nd post. I must ask though, is the muxers doubling the reported framerate working as intended? I can find no mention of this behaviour anywhere.

EDIT: I still feel like this is not working as indended, or there is something else going on. For example, modifiying the last pipeline to output to a .webm file does something remarkable as well. The framerate as reported by windows explorer (or matlab’s videoreader for that matter) is 76.96 fps, which is exactly half. I would assume this is somehow related.
gst-launch-1.0 videotestsrc is-live=1 ! 'video/x-raw, format=(string)GRAY8, framerate=(fraction)153917/1000' ! nvvidconv ! 'video/x-raw(memory:NVMM)' ! nvv4l2vp8enc ! webmmux ! filesink location=video.webm

Hi,
Could you try h264 encoding + matroskamux(or qtmux)? The muxer plugins are gstreamer native plugins and probably webmmux doesn’t work properly. Please try and compare with other muxer plugins.

Generally we run h264/h265 encoding + matroskamux/qtmux. Don’t observe the issue. Please also try the cases for further comparison/information.

I tried both the matroska and qtmux in combination with the h264 encoder. The matroskamux file also results in the half framerate, the qtmux does not. Furtermore the qtmux pipe results in a file that contains the actual framerate, not just the negociated value. (ie. 135 instead of 153, though this more relevant for the other forum thread we a communicating in)

I think it is not far fetched to point the finger to matroskamux and webmmux. Athough it does still seem weird this was never before reported ¯_(ツ)_/¯. For now I will see about switching our application over to use the qtmux.

Matroskamux
gst-launch-1.0 videotestsrc is-live=1 ! 'video/x-raw, format=(string)GRAY8, framerate=(fraction)153917/1000' ! nvvidconv ! 'video/x-raw(memory:NVMM)' ! nvv4l2h264enc ! h264parse ! matroskamux ! filesink location=video.webm -e
Qtmux:
gst-launch-1.0 videotestsrc is-live=1 ! 'video/x-raw, format=(string)GRAY8, framerate=(fraction)153917/1000' ! nvvidconv ! 'video/x-raw(memory:NVMM)' ! nvv4l2h264enc ! h264parse ! qtmux ! filesink location=video.mp4 -e

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.