[TX1] 4k60 video EGLImage readback stuttering

hfrappier · February 29, 2020, 1:19am

There appears to be a performance bottleneck when hardware decoding a 4k60 video to an EGLImage target.

This happens in our application, which uses the Multimedia API, but can easily be reproduced with nvgstplayer-1.0, which uses OMX:

nvgstplayer-1.0 BBBUHD.mp4

outputs the following:

** Message: <main:3939> iterating...
** Message: <goto_next_track:869> uriCount: 0,  uriTotal: 1
** Message: <on_input:538> pausing
NvMMLiteOpen : Block : BlockType = 261 
TVMR: NvMMLiteTVMRDecBlockOpen: 7907: NvMMLiteBlockOpen 
NvMMLiteBlockCreate : Block : BlockType = 261 
TVMR: cbBeginSequence: 1223: BeginSequence  3840x2160, bVPR = 0
TVMR: LowCorner Frequency = 345000 
TVMR: cbBeginSequence: 1622: DecodeBuffers = 5, pnvsi->eCodec = 4, codec = 0 
TVMR: cbBeginSequence: 1693: Display Resolution : (3840x2160) 
TVMR: cbBeginSequence: 1694: Display Aspect Ratio : (3840x2160) 
TVMR: cbBeginSequence: 1762: ColorFormat : 5 
TVMR: cbBeginSequence:1776 ColorSpace = NvColorSpace_YCbCr601
TVMR: cbBeginSequence: 1904: SurfaceLayout = 3
TVMR: cbBeginSequence: 2005: NumOfSurfaces = 12, InteraceStream = 0, InterlaceEnabled = 0, bSecure = 0, MVC = 0 Semiplanar = 1, bReinit = 1, BitDepthForSurface = 8 LumaBitDepth = 8, ChromaBitDepth = 8, ChromaFormat = 5
TVMR: cbBeginSequence: 2007: BeginSequence  ColorPrimaries = 2, TransferCharacteristics = 2, MatrixCoefficients = 2
Allocating new output: 3840x2160 (x 12), ThumbnailMode = 0
OPENMAX: HandleNewStreamFormat: 3464: Send OMX_EventPortSettingsChanged : nFrameWidth = 3840, nFrameHeight = 2160 
** Message: <cb_newpad:2035> creating the pipe for "video/x-raw, width=(int)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ], framerate=(fraction)[ 0/1, 2147483647/1 ]"
** Message: <cb_newpad:2035> creating the pipe for "audio/x-raw, format=(string)S32LE, layout=(string)interleaved, rate=(int){ 8000, 11025, 12000, 16000, 22050, 24000, 32000, 44100, 48000 }, channels=(int)[ 1, 2 ]"
** Message: <cb_newpad:2035> creating the pipe for "audio/x-raw, format=(string)F32LE, layout=(string)interleaved, rate=(int)[ 4000, 96000 ], channels=(int)[ 1, 6 ]"

** (nvgstplayer-1.0:10771): WARNING **: <cb_newpad:2198> decoder pipeline for "audio/x-raw" not created <0>


Stream have 2 audio tracks and 1 video tracks. 
By default it is picking first track 
If want to switch track then give play a stream with an option 
--audio-track or --video-track track no.

** Message: <bus_call:1153> redistribute the latency...
TVMR: FrameRate = 60 
TVMR: NVDEC LowCorner Freq = (576000 * 1024) 

** (nvgstplayer-1.0:10771): WARNING **: <bus_call:1027> WARNING on bus from /GstPipeline:player/GstBin:video_bin/GstEglGlesSink:eglglessink0: A lot of buffers are being dropped.

** (nvgstplayer-1.0:10771): WARNING **: <bus_call:1029> debug info:
gstbasesink.c(2854): gst_base_sink_is_too_late (): /GstPipeline:player/GstBin:video_bin/GstEglGlesSink:eglglessink0:
There may be a timestamping problem, or this computer is too slow.


** (nvgstplayer-1.0:10771): WARNING **: <bus_call:1027> WARNING on bus from /GstPipeline:player/GstBin:video_bin/GstEglGlesSink:eglglessink0: A lot of buffers are being dropped.

** (nvgstplayer-1.0:10771): WARNING **: <bus_call:1029> debug info:
gstbasesink.c(2854): gst_base_sink_is_too_late (): /GstPipeline:player/GstBin:video_bin/GstEglGlesSink:eglglessink0:
There may be a timestamping problem, or this computer is too slow.

TVMR: FrameRate = 60.000240 
TVMR: FrameRate = 60.000240 
TVMR: FrameRate = 60.000240 

** (nvgstplayer-1.0:10771): WARNING **: <bus_call:1027> WARNING on bus from /GstPipeline:player/GstBin:video_bin/GstEglGlesSink:eglglessink0: A lot of buffers are being dropped.

** (nvgstplayer-1.0:10771): WARNING **: <bus_call:1029> debug info:
gstbasesink.c(2854): gst_base_sink_is_too_late (): /GstPipeline:player/GstBin:video_bin/GstEglGlesSink:eglglessink0:
There may be a timestamping problem, or this computer is too slow.


** (nvgstplayer-1.0:10771): WARNING **: <bus_call:1027> WARNING on bus from /GstPipeline:player/GstBin:video_bin/GstEglGlesSink:eglglessink0: A lot of buffers are being dropped.

** (nvgstplayer-1.0:10771): WARNING **: <bus_call:1029> debug info:
gstbasesink.c(2854): gst_base_sink_is_too_late (): /GstPipeline:player/GstBin:video_bin/GstEglGlesSink:eglglessink0:
There may be a timestamping problem, or this computer is too slow.

This does not occur when rendering to nvhdmioverlaysink

nvgstplayer-1.0 -i BBBUHD.mp4 --svs=nvhdmioverlaysink

** Message: <main:3939> iterating...
** Message: <goto_next_track:869> uriCount: 0,  uriTotal: 1
** Message: <on_input:538> pausing
NvMMLiteOpen : Block : BlockType = 261 
TVMR: NvMMLiteTVMRDecBlockOpen: 7907: NvMMLiteBlockOpen 
NvMMLiteBlockCreate : Block : BlockType = 261 
TVMR: cbBeginSequence: 1223: BeginSequence  3840x2160, bVPR = 0
TVMR: LowCorner Frequency = 345000 
TVMR: cbBeginSequence: 1622: DecodeBuffers = 5, pnvsi->eCodec = 4, codec = 0 
TVMR: cbBeginSequence: 1693: Display Resolution : (3840x2160) 
TVMR: cbBeginSequence: 1694: Display Aspect Ratio : (3840x2160) 
TVMR: cbBeginSequence: 1762: ColorFormat : 5 
TVMR: cbBeginSequence:1776 ColorSpace = NvColorSpace_YCbCr601
TVMR: cbBeginSequence: 1904: SurfaceLayout = 3
TVMR: cbBeginSequence: 2005: NumOfSurfaces = 12, InteraceStream = 0, InterlaceEnabled = 0, bSecure = 0, MVC = 0 Semiplanar = 1, bReinit = 1, BitDepthForSurface = 8 LumaBitDepth = 8, ChromaBitDepth = 8, ChromaFormat = 5
TVMR: cbBeginSequence: 2007: BeginSequence  ColorPrimaries = 2, TransferCharacteristics = 2, MatrixCoefficients = 2
Allocating new output: 3840x2160 (x 12), ThumbnailMode = 0
OPENMAX: HandleNewStreamFormat: 3464: Send OMX_EventPortSettingsChanged : nFrameWidth = 3840, nFrameHeight = 2160 
** Message: <cb_newpad:2035> creating the pipe for "video/x-raw, width=(int)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ], framerate=(fraction)[ 0/1, 2147483647/1 ]"
** Message: <cb_newpad:2035> creating the pipe for "audio/x-raw, format=(string)S32LE, layout=(string)interleaved, rate=(int){ 8000, 11025, 12000, 16000, 22050, 24000, 32000, 44100, 48000 }, channels=(int)[ 1, 2 ]"
** Message: <cb_newpad:2035> creating the pipe for "audio/x-raw, format=(string)F32LE, layout=(string)interleaved, rate=(int)[ 4000, 96000 ], channels=(int)[ 1, 6 ]"

** (nvgstplayer-1.0:10906): WARNING **: <cb_newpad:2198> decoder pipeline for "audio/x-raw" not created <0>


Stream have 2 audio tracks and 1 video tracks. 
By default it is picking first track 
If want to switch track then give play a stream with an option 
--audio-track or --video-track track no.

** Message: <bus_call:1153> redistribute the latency...
TVMR: FrameRate = 60 
TVMR: NVDEC LowCorner Freq = (576000 * 1024) 
TVMR: FrameRate = 60.000240 
TVMR: FrameRate = 60.000240 
TVMR: FrameRate = 60.000240 
TVMR: FrameRate = 60.000240 
TVMR: FrameRate = 60.000240 
TVMR: FrameRate = 60.000240 
TVMR: FrameRate = 60.000240 
TVMR: FrameRate = 60.000240 
TVMR: FrameRate = 60.000240 
TVMR: FrameRate = 60.000240 
TVMR: FrameRate = 60.000240 
TVMR: FrameRate = 60.000240 
TVMR: FrameRate = 60.000240 
TVMR: FrameRate = 60.000240 
TVMR: FrameRate = 60.000240 
TVMR: FrameRate = 60.000240 
TVMR: FrameRate = 60.000240 
TVMR: FrameRate = 60.000240 
TVMR: FrameRate = 60.000240 
TVMR: FrameRate = 60.000240 
TVMR: FrameRate = 60.000240 
TVMR: FrameRate = 60.000240 
TVMR: FrameRate = 60.000240 
TVMR: FrameRate = 60.000240 
TVMR: FrameRate = 60.000240 
TVMR: FrameRate = 60.000240

ffprobe of BBBUHD.mp4:

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'BBBUHD.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 1
    compatible_brands: isomavc1
    creation_time   : 2013-12-17 16:40:26
    title           : Big Buck Bunny, Sunflower version
    artist          : Blender Foundation 2008, Janus Bager Kristensen 2013
    comment         : Creative Commons Attribution 3.0 - http://bbb3d.renderfarming.net
    genre           : Animation
    composer        : Sacha Goedegebure
  Duration: 00:10:34.53, start: 0.000000, bitrate: 8487 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 3840x2160 [SAR 1:1 DAR 16:9], 8002 kb/s, 60 fps, 60 tbr, 60k tbn, 120 tbc (default)
    Metadata:
      creation_time   : 2013-12-17 16:40:26
      handler_name    : GPAC ISO Video Handler
    Stream #0:1(und): Audio: mp3 (mp4a / 0x6134706D), 48000 Hz, stereo, s16p, 160 kb/s (default)
    Metadata:
      creation_time   : 2013-12-17 16:40:28
      handler_name    : GPAC ISO Audio Handler
    Stream #0:2(und): Audio: ac3 (ac-3 / 0x332D6361), 48000 Hz, 5.1(side), fltp, 320 kb/s (default)
    Metadata:
      creation_time   : 2013-12-17 16:40:28
      handler_name    : GPAC ISO Audio Handler
    Side data:
      audio service type: main

DaneLLL · February 29, 2020, 1:19am

Hi hfrappier,
Please try max performance:

sudo ./jetson_clocks.sh

If the issue is still present in max performance, please share a sample based on Multimedpa API so that we can reproduce the issue. Are you on r28.1?

kayccc · February 1, 2020, 6:11am

Hi hfrappier,

Any update? Have you clarified the cause and resolved the problem?

Thanks