Out of order frames from NVDEC

Gaming and Visualization Technologies Video Processing & Optical Flow
1

Hello everyone.

I’m developing hardware accelerated transcoding software, which handles multicast live streams. I’ve found during a test, that decoder produces out of order pictures on some streams. I’ve dump timestamps and raw output frames to check if this is my bug with timestamps or something.

Timestamps are correct but pics aren’t. The decoder produces a frame that should be shown 10 seconds ago or so, but with the current timestamp. You can see them in output.ts at 0:18, 0:29, 0:59, 1:30, 2:00, 2:10, 2:21, 2:37. For verification, I’ve ran this stream through the FFmpeg with the same result. You can find input and output streams here: https://yadi.sk/d/pKQ-waO2N-tNFw Configurations on which I’ve checked for this issue:

  • Ubuntu 18.04
  • 396.37 and 410.78 linux drivers
  • CUDA Toolkit 9.2 and 10
  • Video Codec SDK 8.1
  • FFmpeg release 3.4
  • NVIDIA Quadro P5000

FFmpeg sample command:

ffmpeg -hwaccel cuvid -c:v h264_cuvid -vsync 0 -deint 2 -drop_second_field 1 -i input.ts -c:v h264_nvenc -c:a copy -f mpegts output.ts

May be someone faced with the same issue and know how to handle it?

Appreciate any help. Thanks.

2

Hi Yar1k,

Not sure if this is related but it is worth a look.

[url]https://devtalk.nvidia.com/default/topic/1023481/jetson-tx2/frames-returned-from-nveglstreamsrc-via-egl-stream-out-of-order/2[/url]

3

Hello!

I was faced with the same problem.

I tried to understand the problem and I think I found a solution.

Have a look here [url]https://github.com/Svechnikov/ffmpeg-cuda-deinterlace-problems[/url]

The reason behind the problems is that sometimes CUVIDPARSERDISPINFO has property progressive_frame equal to 1 (which is wrong when videos are interlaced). In order to fix the problem we should check if the video is interlaced or progressive in the beginning of a video sequence (the best place for that is cuvid_handle_video_sequence function). And then we just use this information instead of the faulty progressive_frame in CUVIDPARSERDISPINFO.

Basically, you can just apply a patch and the problem will go away ([url]https://github.com/Svechnikov/ffmpeg-cuda-deinterlace-problems/blob/master/images/fix.patch[/url])

4

@svechnikov66, Thanks for sharing!

Cheers,
Tom

5

@TomK@Nvidia, I’ve spent so much time debugging this issue and making it work, hope people will use it :)

I have a technical question:

When decoding interlaced content, why does cuvid determine that some frames are progressive (when in fact they are not)?

In this callback (https://github.com/FFmpeg/FFmpeg/blob/master/libavcodec/cuviddec.c#L343) which is called by cuvid, we sometimes get CUVIDPARSERDISPINFO.progressive_frame = 1, which is wrong. Here’s a definition of CUVIDPARSERDISPINFO https://github.com/FFmpeg/nv-codec-headers/blob/master/include/ffnvcodec/dynlink_nvcuvid.h#L307

Is this forum the right place to ask such questions or I should submit it somewhere else?

@Yar1k, Why did you use the drop_second_field option? Your input is 50i, so you can get 50p as output. I suggest you to modify your ffmpeg command:

ffmpeg -hwaccel cuvid -c:v h264_cuvid -deint adaptive -copyts -i input.ts -c:v h264_nvenc -c:a copy -r 50 -f mpegts output.ts

This way the video will be much more smoother. Option copyts is needed because without it ffmpeg messes up pts’ for some reason.

Pay attention though on h264 profile level, as it will be higher, and some devices might not support it (especially, when you convert 1080i to 1080p).

6

@svechnikov66, thank you!

Your solution do the trick. Really appreciate your work.

About dropping of the second field:

It reduces the loading of NVENC and allows you to transcode more channels per one card. 50 fps are good for sports but aren’t always necessary for cinema.

This ffmpeg command is just a showcase, but thank you for your additions about the params. :)