DeckLink Mini Recorder 4K streaming via NVidia GeForce GTX 1660

Hello!

Ubuntu 18.04.3
ffmpeg 4.2
Driver Version: 435.21
CUDA Version: 10.1
Nvenc version 9.1

Capture Device: DeckLink Mini Recorder 4K
GPU: NVidia GeForce GTX 1660

i need stream input video to mpegts udp:

ffmpeg \
    -use_wallclock_as_timestamps 1 \
    -loglevel debug \
    -hwaccel nvdec \
    -threads 8 -thread_type frame \
    -f decklink \
    -video_pts wallclock \
    -audio_pts wallclock \
    -draw_bars true \
    -raw_format yuv422p10 \
    -channels 2 \
    -i 'DeckLink Mini Recorder 4K' \
\
    -c:v h264_nvenc \
    -vsync 0 \
    -preset:v slow \
    -profile:v high \
    -level:v 4.2 \
    -rc cbr_hq \
    -rc-lookahead 32 \
    -surfaces 64 \
    -weighted_pred 1 \
    -pix_fmt yuv420p \
    -b:v 4.8M \
    -cq 17 \
    -qmin 1 \
    -qmax 18 \
    -g 12 \
    -refs 2 \
\
    -c:a libfdk_aac \
    -profile:a aac_he_v2 \
    -ac 2 \
    -ar 48000 \
    -af 'aresample=async=1' \
\
    -f mpegts 'udp://@239.0.0.1:2266?pkt_size=1316&fifo_size=1000000&buffer_size=100000000&overrun_nonfatal=1'

i get valid and successfull output:

[decklink @ 0x564090345bc0] Autodetected the input mode
[decklink @ 0x564090345bc0] Found Decklink mode 1920 x 1080 with rate 25.00(i)
Guessed Channel Layout for Input Stream #0.0 : stereo
Input #0, decklink, from 'DeckLink Mini Recorder 4K':
  Duration: N/A, start: 1570604250.716705, bitrate: 1107456 kb/s
    Stream #0:0: Audio: pcm_s16le, 48000 Hz, stereo, s16, 1536 kb/s
    Stream #0:1: Video: v210 (V210 / 0x30313256), yuv422p10le(top first), 1920x1080, 1105920 kb/s, 25 tbr, 1000k tbn, 1000k tbc
Stream mapping:
  Stream #0:1 -> #0:0 (v210 (native) -> h264 (h264_nvenc))
  Stream #0:0 -> #0:1 (pcm_s16le (native) -> aac (libfdk_aac))
Press [q] to stop, [?] for help
Output #0, mpegts, to 'udp://@239.0.0.1:2266?pkt_size=1316&fifo_size=1000000&buffer_size=100000000&overrun_nonfatal=1':
  Metadata:
    encoder         : Lavf58.32.104
    Stream #0:0: Video: h264 (h264_nvenc) (High), yuv420p(top coded first (swapped)), 1920x1080, q=1-18, 4800 kb/s, 25 fps, 90k tbn, 25 tbc
    Metadata:
      encoder         : Lavc58.56.101 h264_nvenc
    Side data:
      cpb: bitrate max/min/avg: 0/0/4800000 buffer size: 9600000 vbv_delay: N/A
    Stream #0:1: Audio: aac (libfdk_aac) (HE-AACv2), 48000 Hz, stereo, s16, 52 kb/s
    Metadata:
      encoder         : Lavc58.56.101 libfdk_aac
frame=   26 fps=0.0 q=0.0 size=       0kB time=00:00:00.96 bitrate=   0.0kbits/s speed=1.89x
frame=   39 fps= 38 q=0.0 size=       0kB time=00:00:01.47 bitrate=   0.0kbits/s speed=1.43x
frame=   52 fps= 34 q=0.0 size=       0kB time=00:00:01.98 bitrate=   0.0kbits/s speed=1.28x
...

but some time later (2-25 min) i get crush:

[decklink @ 0x55dc42b4ac00] Frame received (#17175) - Valid (5529600B) - QSize 0.000000MB
frame=17186 fps= 25 q=26.0 size=  419306kB time=00:11:27.39 bitrate=4997.1kbits/s speed=   1x
frame=17199 fps= 25 q=25.0 size=  419614kB time=00:11:27.90 bitrate=4997.0kbits/s speed=   1x
[decklink @ 0x55dc42b4ac00] Frame received (#17200) - Valid (5529600B) - QSize 0.000000MB
frame=17212 fps= 25 q=25.0 size=  419923kB time=00:11:28.41 bitrate=4997.0kbits/s speed=   1x
[decklink @ 0x55dc42b4ac00] Frame received (#17225) - Valid (5529600B) - QSize 0.000000MB
frame=17225 fps= 25 q=26.0 size=  420227kB time=00:11:28.92 bitrate=4996.9kbits/s speed=   1x
frame=17238 fps= 25 q=26.0 size=  420537kB time=00:11:29.43 bitrate=4996.9kbits/s speed=   1x
[decklink @ 0x55dc42b4ac00] Frame received (#17250) - Valid (5529600B) - QSize 0.000000MB
frame=17251 fps= 25 q=26.0 size=  420858kB time=00:11:29.95 bitrate=4997.0kbits/s speed=   1x
frame=17264 fps= 25 q=26.0 size=  421170kB time=00:11:30.50 bitrate=4996.7kbits/s speed=   1x
[decklink @ 0x55dc42b4ac00] Frame received (#17275) - Valid (5529600B) - QSize 0.000000MB
frame=17277 fps= 25 q=26.0 size=  421484kB time=00:11:31.01 bitrate=4996.7kbits/s speed=   1x
[decklink @ 0x55dc42b4ac00] Frame received (#17300) - Valid (5529600B) - QSize 63.378754MB
[decklink @ 0x55dc42b4ac00] Frame received (#17325) - Valid (5529600B) - QSize 195.402374MB
[decklink @ 0x55dc42b4ac00] Frame received (#17350) - Valid (5529600B) - QSize 327.425995MB
[decklink @ 0x55dc42b4ac00] Frame received (#17375) - Valid (5529600B) - QSize 459.449615MB
[decklink @ 0x55dc42b4ac00] Frame received (#17400) - Valid (5529600B) - QSize 591.473236MB
[decklink @ 0x55dc42b4ac00] Frame received (#17425) - Valid (5529600B) - QSize 723.496857MB
[decklink @ 0x55dc42b4ac00] Frame received (#17450) - Valid (5529600B) - QSize 855.520477MB
[decklink @ 0x55dc42b4ac00] Frame received (#17475) - Valid (5529600B) - QSize 987.544098MB
[decklink @ 0x55dc42b4ac00] Decklink input buffer overrun!
    Last message repeated 36 times
[decklink @ 0x55dc42b4ac00] Frame received (#17500) - Valid (5529600B) - QSize 1024.503296MB
[decklink @ 0x55dc42b4ac00] Decklink input buffer overrun!
    Last message repeated 49 times
[decklink @ 0x55dc42b4ac00] Frame received (#17525) - Valid (5529600B) - QSize 1024.503296MB
[decklink @ 0x55dc42b4ac00] Decklink input buffer overrun!
    Last message repeated 49 times
[decklink @ 0x55dc42b4ac00] Frame received (#17550) - Valid (5529600B) - QSize 1024.503296MB
[decklink @ 0x55dc42b4ac00] Decklink input buffer overrun!
    Last message repeated 49 times

if i use CPU encodind like libx264, all good, over9000 hours encode stream.
Why GPU crushes?
Perhaps the reason is out of sync equipment?

hmm… it`s a driver bug.
when i do pipeline

decklink > libx246(q=1-3) -> local rtmp ( rtmp://localhost/feed/app ; 150Mbit/s) -> h264_nvenc(mpegts udp)

, it`s working fine!

syslog when i try do

ffmpeg -f decklink -i 'DeclLink Mini Recorder 4K' -c:v h264_nvenc -c:a libfdk_aac -f mpegts udp://xxx.xxx.xxx.xxx:foo

syslog:

kernel: [ 1240.080358] NVRM: GPU at PCI:0000:01:00: GPU-6caba61c-b997-efe4-1f42-012b974eadb5
kernel: [ 1240.080362] NVRM: GPU Board Serial Number:
kernel: [ 1240.080369] NVRM: Xid (PCI:0000:01:00): 31, pid=2196, Ch 00000038, intr 00000000. MMU Fault: ENGINE NVENC0 HUBCLIENT_NVENC0 faulted @ 0x0_00001000. Fault is of type FAULT_PDE ACCESS_TYPE_VIRT_READ

up
nvidia-bug-report.log.gz (1.01 MB)

Some other users reported a similar bug with ffmpeg/nvenc running into out-of-memory conditions with video memory.
You could try the just released beta driver:
https://devtalk.nvidia.com/default/topic/1065039

Though it only refers to Vulkan maybe this helps in your situation, too.

Yeah! NVIDIA-Linux-x86_64-440.26.run is amazing!
19 hours of streaming in 2 streams

ffmpeg -i decklink -vf split=2[out1][out2] -map [out1] -c:v h264_nvenc output1 -map [out2] -c:v h264_nvenc output2

Thanks!

Good to know this is fixing it for you.
TBH though, I think this is just part of the truth. When encoding just two streams, why the heck is ffmpeg eating up all the vmem so there’s nothing left for driver internal tasks in the first place? Since you showed that piping didn’t exhibit this, I strongly suspect there’s also an underlying bug in ffmpeg’s nvenc implementation regarding vmem allocations. This should also be looked at since that possibly would lead to other problems, as well.