There appears to be a performance bottleneck when hardware decoding a 4k60 video to an EGLImage target.
This happens in our application, which uses the Multimedia API, but can easily be reproduced with nvgstplayer-1.0, which uses OMX:
nvgstplayer-1.0 BBBUHD.mp4
outputs the following:
** Message: <main:3939> iterating...
** Message: <goto_next_track:869> uriCount: 0, uriTotal: 1
** Message: <on_input:538> pausing
NvMMLiteOpen : Block : BlockType = 261
TVMR: NvMMLiteTVMRDecBlockOpen: 7907: NvMMLiteBlockOpen
NvMMLiteBlockCreate : Block : BlockType = 261
TVMR: cbBeginSequence: 1223: BeginSequence 3840x2160, bVPR = 0
TVMR: LowCorner Frequency = 345000
TVMR: cbBeginSequence: 1622: DecodeBuffers = 5, pnvsi->eCodec = 4, codec = 0
TVMR: cbBeginSequence: 1693: Display Resolution : (3840x2160)
TVMR: cbBeginSequence: 1694: Display Aspect Ratio : (3840x2160)
TVMR: cbBeginSequence: 1762: ColorFormat : 5
TVMR: cbBeginSequence:1776 ColorSpace = NvColorSpace_YCbCr601
TVMR: cbBeginSequence: 1904: SurfaceLayout = 3
TVMR: cbBeginSequence: 2005: NumOfSurfaces = 12, InteraceStream = 0, InterlaceEnabled = 0, bSecure = 0, MVC = 0 Semiplanar = 1, bReinit = 1, BitDepthForSurface = 8 LumaBitDepth = 8, ChromaBitDepth = 8, ChromaFormat = 5
TVMR: cbBeginSequence: 2007: BeginSequence ColorPrimaries = 2, TransferCharacteristics = 2, MatrixCoefficients = 2
Allocating new output: 3840x2160 (x 12), ThumbnailMode = 0
OPENMAX: HandleNewStreamFormat: 3464: Send OMX_EventPortSettingsChanged : nFrameWidth = 3840, nFrameHeight = 2160
** Message: <cb_newpad:2035> creating the pipe for "video/x-raw, width=(int)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ], framerate=(fraction)[ 0/1, 2147483647/1 ]"
** Message: <cb_newpad:2035> creating the pipe for "audio/x-raw, format=(string)S32LE, layout=(string)interleaved, rate=(int){ 8000, 11025, 12000, 16000, 22050, 24000, 32000, 44100, 48000 }, channels=(int)[ 1, 2 ]"
** Message: <cb_newpad:2035> creating the pipe for "audio/x-raw, format=(string)F32LE, layout=(string)interleaved, rate=(int)[ 4000, 96000 ], channels=(int)[ 1, 6 ]"
** (nvgstplayer-1.0:10771): WARNING **: <cb_newpad:2198> decoder pipeline for "audio/x-raw" not created <0>
Stream have 2 audio tracks and 1 video tracks.
By default it is picking first track
If want to switch track then give play a stream with an option
--audio-track or --video-track track no.
** Message: <bus_call:1153> redistribute the latency...
TVMR: FrameRate = 60
TVMR: NVDEC LowCorner Freq = (576000 * 1024)
** (nvgstplayer-1.0:10771): WARNING **: <bus_call:1027> WARNING on bus from /GstPipeline:player/GstBin:video_bin/GstEglGlesSink:eglglessink0: A lot of buffers are being dropped.
** (nvgstplayer-1.0:10771): WARNING **: <bus_call:1029> debug info:
gstbasesink.c(2854): gst_base_sink_is_too_late (): /GstPipeline:player/GstBin:video_bin/GstEglGlesSink:eglglessink0:
There may be a timestamping problem, or this computer is too slow.
** (nvgstplayer-1.0:10771): WARNING **: <bus_call:1027> WARNING on bus from /GstPipeline:player/GstBin:video_bin/GstEglGlesSink:eglglessink0: A lot of buffers are being dropped.
** (nvgstplayer-1.0:10771): WARNING **: <bus_call:1029> debug info:
gstbasesink.c(2854): gst_base_sink_is_too_late (): /GstPipeline:player/GstBin:video_bin/GstEglGlesSink:eglglessink0:
There may be a timestamping problem, or this computer is too slow.
TVMR: FrameRate = 60.000240
TVMR: FrameRate = 60.000240
TVMR: FrameRate = 60.000240
** (nvgstplayer-1.0:10771): WARNING **: <bus_call:1027> WARNING on bus from /GstPipeline:player/GstBin:video_bin/GstEglGlesSink:eglglessink0: A lot of buffers are being dropped.
** (nvgstplayer-1.0:10771): WARNING **: <bus_call:1029> debug info:
gstbasesink.c(2854): gst_base_sink_is_too_late (): /GstPipeline:player/GstBin:video_bin/GstEglGlesSink:eglglessink0:
There may be a timestamping problem, or this computer is too slow.
** (nvgstplayer-1.0:10771): WARNING **: <bus_call:1027> WARNING on bus from /GstPipeline:player/GstBin:video_bin/GstEglGlesSink:eglglessink0: A lot of buffers are being dropped.
** (nvgstplayer-1.0:10771): WARNING **: <bus_call:1029> debug info:
gstbasesink.c(2854): gst_base_sink_is_too_late (): /GstPipeline:player/GstBin:video_bin/GstEglGlesSink:eglglessink0:
There may be a timestamping problem, or this computer is too slow.
This does not occur when rendering to nvhdmioverlaysink
nvgstplayer-1.0 -i BBBUHD.mp4 --svs=nvhdmioverlaysink
** Message: <main:3939> iterating...
** Message: <goto_next_track:869> uriCount: 0, uriTotal: 1
** Message: <on_input:538> pausing
NvMMLiteOpen : Block : BlockType = 261
TVMR: NvMMLiteTVMRDecBlockOpen: 7907: NvMMLiteBlockOpen
NvMMLiteBlockCreate : Block : BlockType = 261
TVMR: cbBeginSequence: 1223: BeginSequence 3840x2160, bVPR = 0
TVMR: LowCorner Frequency = 345000
TVMR: cbBeginSequence: 1622: DecodeBuffers = 5, pnvsi->eCodec = 4, codec = 0
TVMR: cbBeginSequence: 1693: Display Resolution : (3840x2160)
TVMR: cbBeginSequence: 1694: Display Aspect Ratio : (3840x2160)
TVMR: cbBeginSequence: 1762: ColorFormat : 5
TVMR: cbBeginSequence:1776 ColorSpace = NvColorSpace_YCbCr601
TVMR: cbBeginSequence: 1904: SurfaceLayout = 3
TVMR: cbBeginSequence: 2005: NumOfSurfaces = 12, InteraceStream = 0, InterlaceEnabled = 0, bSecure = 0, MVC = 0 Semiplanar = 1, bReinit = 1, BitDepthForSurface = 8 LumaBitDepth = 8, ChromaBitDepth = 8, ChromaFormat = 5
TVMR: cbBeginSequence: 2007: BeginSequence ColorPrimaries = 2, TransferCharacteristics = 2, MatrixCoefficients = 2
Allocating new output: 3840x2160 (x 12), ThumbnailMode = 0
OPENMAX: HandleNewStreamFormat: 3464: Send OMX_EventPortSettingsChanged : nFrameWidth = 3840, nFrameHeight = 2160
** Message: <cb_newpad:2035> creating the pipe for "video/x-raw, width=(int)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ], framerate=(fraction)[ 0/1, 2147483647/1 ]"
** Message: <cb_newpad:2035> creating the pipe for "audio/x-raw, format=(string)S32LE, layout=(string)interleaved, rate=(int){ 8000, 11025, 12000, 16000, 22050, 24000, 32000, 44100, 48000 }, channels=(int)[ 1, 2 ]"
** Message: <cb_newpad:2035> creating the pipe for "audio/x-raw, format=(string)F32LE, layout=(string)interleaved, rate=(int)[ 4000, 96000 ], channels=(int)[ 1, 6 ]"
** (nvgstplayer-1.0:10906): WARNING **: <cb_newpad:2198> decoder pipeline for "audio/x-raw" not created <0>
Stream have 2 audio tracks and 1 video tracks.
By default it is picking first track
If want to switch track then give play a stream with an option
--audio-track or --video-track track no.
** Message: <bus_call:1153> redistribute the latency...
TVMR: FrameRate = 60
TVMR: NVDEC LowCorner Freq = (576000 * 1024)
TVMR: FrameRate = 60.000240
TVMR: FrameRate = 60.000240
TVMR: FrameRate = 60.000240
TVMR: FrameRate = 60.000240
TVMR: FrameRate = 60.000240
TVMR: FrameRate = 60.000240
TVMR: FrameRate = 60.000240
TVMR: FrameRate = 60.000240
TVMR: FrameRate = 60.000240
TVMR: FrameRate = 60.000240
TVMR: FrameRate = 60.000240
TVMR: FrameRate = 60.000240
TVMR: FrameRate = 60.000240
TVMR: FrameRate = 60.000240
TVMR: FrameRate = 60.000240
TVMR: FrameRate = 60.000240
TVMR: FrameRate = 60.000240
TVMR: FrameRate = 60.000240
TVMR: FrameRate = 60.000240
TVMR: FrameRate = 60.000240
TVMR: FrameRate = 60.000240
TVMR: FrameRate = 60.000240
TVMR: FrameRate = 60.000240
TVMR: FrameRate = 60.000240
TVMR: FrameRate = 60.000240
TVMR: FrameRate = 60.000240
ffprobe of BBBUHD.mp4:
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'BBBUHD.mp4':
Metadata:
major_brand : isom
minor_version : 1
compatible_brands: isomavc1
creation_time : 2013-12-17 16:40:26
title : Big Buck Bunny, Sunflower version
artist : Blender Foundation 2008, Janus Bager Kristensen 2013
comment : Creative Commons Attribution 3.0 - http://bbb3d.renderfarming.net
genre : Animation
composer : Sacha Goedegebure
Duration: 00:10:34.53, start: 0.000000, bitrate: 8487 kb/s
Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 3840x2160 [SAR 1:1 DAR 16:9], 8002 kb/s, 60 fps, 60 tbr, 60k tbn, 120 tbc (default)
Metadata:
creation_time : 2013-12-17 16:40:26
handler_name : GPAC ISO Video Handler
Stream #0:1(und): Audio: mp3 (mp4a / 0x6134706D), 48000 Hz, stereo, s16p, 160 kb/s (default)
Metadata:
creation_time : 2013-12-17 16:40:28
handler_name : GPAC ISO Audio Handler
Stream #0:2(und): Audio: ac3 (ac-3 / 0x332D6361), 48000 Hz, 5.1(side), fltp, 320 kb/s (default)
Metadata:
creation_time : 2013-12-17 16:40:28
handler_name : GPAC ISO Audio Handler
Side data:
audio service type: main