Hi,
The decoded frame data is in hardware DMA buffer and then copies to CPU buffer in ffmpeg stack. This impacts overall performance. For optimal throughput, we would suggest use jetson_multimedia_api. You can take a look at this sample and give it a try:
/usr/src/jetson_multimedia_api/samples/00_video_decode