Video Codec SDK 9.1.23 crash with RC lookahead

I don’t see anywhere else to report this bug, so here it is. I spent quite a few days tracking this down. I’m testing on Linux, ubuntu 16.04 with the Video Codec SDK 9.1.23 and the 440.82 release of the drivers. I’m integrating the NVENC h264 encoder with our broadcast server product. The acute symptom is that the encoder crashes in certain situations after calling the nvEncEncodePicture() function. By examining the program state with GDB, I can see that the has allocated a very large buffer (in some cases about 275MB, in other cases over 3GB) and is in the middle of a memcpy operation when it crashes. I also captured the crash with valgrind, and here are some interesting results:

==23882== Invalid read of size 8
==23882== at 0x5BE34B0: memcpy@GLIBC_2.2.5 (vg_replace_strmem.c:1032)
==23882== by 0x84EA38F: ??? (in /usr/lib/x86_64-linux-gnu/
==23882== by 0x84EB1C6: ??? (in /usr/lib/x86_64-linux-gnu/
==23882== by 0x84E77B7: ??? (in /usr/lib/x86_64-linux-gnu/
==23882== by 0x84E788D: ??? (in /usr/lib/x86_64-linux-gnu/
==23882== by 0x84E7FEC: ??? (in /usr/lib/x86_64-linux-gnu/
==23882== by 0x84E5406: ??? (in /usr/lib/x86_64-linux-gnu/
==23882== by 0x84F309A: ??? (in /usr/lib/x86_64-linux-gnu/
==23882== by 0x4FBE39: nvenc_264_video_encode_thread_func (encode_h264.c:1658)
==23882== by 0x6AAA6B9: start_thread (pthread_create.c:333)
==23882== by 0x8D9941C: clone (clone.S:109)
==23882== Address 0x1173c0038 is 8 bytes before a block of size 2,477,955,468 alloc’d
==23882== at 0x5BDEF66: malloc (vg_replace_malloc.c:307)
==23882== by 0x84EA2EF: ??? (in /usr/lib/x86_64-linux-gnu/
==23882== by 0x84EB1C6: ??? (in /usr/lib/x86_64-linux-gnu/
==23882== by 0x84E77B7: ??? (in /usr/lib/x86_64-linux-gnu/
==23882== by 0x84E788D: ??? (in /usr/lib/x86_64-linux-gnu/
==23882== by 0x84E7FEC: ??? (in /usr/lib/x86_64-linux-gnu/
==23882== by 0x84E5406: ??? (in /usr/lib/x86_64-linux-gnu/
==23882== by 0x84F309A: ??? (in /usr/lib/x86_64-linux-gnu/
==23882== by 0x4FBE39: nvenc_264_video_encode_thread_func (encode_h264.c:1658)
==23882== by 0x6AAA6B9: start_thread (pthread_create.c:333)
==23882== by 0x8D9941C: clone (clone.S:109)

In this case it actually allocated a 2.75GB buffer, and read the memory locations 8 bytes (and later 16 bytes) prior to the start of the buffer.

Here are the other big clues which should lead your driver developers to the source of the bug. The first is that it only crashes when the rate control lookahead is enabled. I used the RC_CBR rate control method. It doesn’t matter how many frames are used for the lookahead depth (the crash will happen with a depth of 1 or 32), or whether or not the adaptive I/B frame insertion is enabled.

The second big clue is that this only happens when I submit all of my beginning frames with codecPicParams.h264PicParams.seiPayloadArrayCnt set to 1, and provide an auxiliary SEI message with all my submitted frames, and then submit a frame with seiPayloadArrayCnt set to 0. The nvEncEncodePicture() function call will crash when I give it the frame with no extra SEI message. Note that the encoder will not crash if, instead of including my usual ITU T35 SEI message with the frame which doesn’t have one, I include a filler message (type 3). There’s something about reducing the number of SEI messages included with the input frames, when rate control lookahead is enabled, which is the root cause of this crash.

Hi richard.goedeken.

Do you have a reproducer that you can share along with steps to use, to internally reproduce this issue. That will help further expedite debugging.
Along with the backtrace in gdb, can you provide output of ‘info proc mappings’ from gdb command line as well?


Hello, I was able to make a small modification to ffmpeg which will demonstrate the crash. Start with the source code from ffmpeg tag name “n4.2.3”, and apply this diff to libavcodec/nvenc.c:

--- ./nvenc_orig.c	2020-07-07 08:55:32.105268303 -0700
+++ nvenc.c	2020-07-07 09:04:24.441360966 -0700
@@ -2110,6 +2110,7 @@
         pic_params.inputTimeStamp = frame->pts;
+        /*
         if (ctx->a53_cc && av_frame_get_side_data(frame, AV_FRAME_DATA_A53_CC)) {
             if (ff_alloc_a53_sei(frame, sizeof(NV_ENC_SEI_PAYLOAD), (void**)&sei_data, &sei_size) < 0) {
                 av_log(ctx, AV_LOG_ERROR, "Not enough memory for closed captions, skipping\n");
@@ -2121,6 +2122,18 @@
                 sei_data->payload = (uint8_t*)(sei_data + 1);
+        */
+        // attach a Filler SEI message, to only the first 100 frames
+        static int frame_count = 0;
+        if (frame_count++ < 100)
+        {
+            sei_size = 16;
+            sei_data = av_mallocz(sei_size + sizeof(NV_ENC_SEI_PAYLOAD));
+            sei_data->payloadSize = (uint32_t)sei_size;
+            sei_data->payloadType = 3;
+            sei_data->payload = (uint8_t*)(sei_data + 1);
+            memset(sei_data->payload, 0xff, sei_size);
+        }
         nvenc_codec_specific_pic_params(avctx, &pic_params, sei_data);
     } else {

I configured the source code with the following options:

./configure --enable-version3 --enable-cuda --enable-cuvid --enable-nvenc --enable-nonfree --enable-libnpp --enable-libvmaf --extra-cflags=-I/usr/local/cuda/include --extra-ldflags=-L/usr/local/cuda/lib64 --nvcc=/usr/local/cuda/bin/nvcc --enable-cuda-nvcc

I ran the transcoder with the following command:

./ffmpeg_g -y -loglevel verbose -f rawvideo -s:v 1920x1080 -pix_fmt yuv420p -r 30 -i /media/source_yuv/test_nvidia_rc_lookahead_crash_1920x1080_420.yuv -c:v h264_nvenc -preset medium -rc cbr -profile:v high -coder cabac -no-scenecut 1 -b_ref_mode disabled -bf 3 -spatial-aq 1 -b:v 4M -g 30 -rc-lookahead 15 /home/richard/

In this case I was using uncompressed YUV planar as an input format and quicktime as the output, but I’m sure the crash will be reproducible regardless of the input and output formats. The important settings are to use the h264_nvenc codec, CBR rate control, and enable rc-lookahead.

Here is the back trace from the crash:

Thread 1 "ffmpeg_g" received signal SIGSEGV, Segmentation fault.
__memmove_avx_unaligned () at ../sysdeps/x86_64/multiarch/memcpy-avx-unaligned.S:244
244	../sysdeps/x86_64/multiarch/memcpy-avx-unaligned.S: No such file or directory.
(gdb) bt
#0  __memmove_avx_unaligned () at ../sysdeps/x86_64/multiarch/memcpy-avx-unaligned.S:244
#1  0x00007fffe91ed390 in ?? () from /usr/lib/x86_64-linux-gnu/
#2  0x00007fffe91ee1c7 in ?? () from /usr/lib/x86_64-linux-gnu/
#3  0x00007fffe91ea7b8 in ?? () from /usr/lib/x86_64-linux-gnu/
#4  0x00007fffe91ea88e in ?? () from /usr/lib/x86_64-linux-gnu/
#5  0x00007fffe91eafed in ?? () from /usr/lib/x86_64-linux-gnu/
#6  0x00007fffe91e8407 in ?? () from /usr/lib/x86_64-linux-gnu/
#7  0x00007fffe91f609b in ?? () from /usr/lib/x86_64-linux-gnu/
#8  0x0000000000fe0745 in ff_nvenc_send_frame (avctx=0x2126280, frame=0x2943d40) at libavcodec/nvenc.c:2148
#9  0x0000000000877c82 in avcodec_send_frame (avctx=avctx@entry=0x2126280, frame=frame@entry=0x2943d40)
    at libavcodec/encode.c:403
#10 0x00000000004a6b2b in do_video_out (of=of@entry=0x2123840, ost=ost@entry=0x2126040, 
    next_picture=next_picture@entry=0x2943d40, sync_ipts=<optimized out>, sync_ipts@entry=15.000007629394531)
    at fftools/ffmpeg.c:1288
#11 0x00000000004a7eb6 in reap_filters (flush=flush@entry=0) at fftools/ffmpeg.c:1505
#12 0x00000000004abfe9 in transcode_step () at fftools/ffmpeg.c:4650
#13 transcode () at fftools/ffmpeg.c:4694
#14 0x00000000004896b2 in main (argc=<optimized out>, argv=0x7fffffffd148) at fftools/ffmpeg.c:4896

And the process memory mappings:

(gdb) info proc mappings
process 32659
Mapped address spaces:

          Start Addr           End Addr       Size     Offset objfile
            0x400000          0x167e000  0x127e000        0x0 /home/richard/Software/ffmpeg/ffmpeg_g
           0x187d000          0x187e000     0x1000  0x127d000 /home/richard/Software/ffmpeg/ffmpeg_g
           0x187e000          0x18cc000    0x4e000  0x127e000 /home/richard/Software/ffmpeg/ffmpeg_g
           0x18cc000          0x324b000  0x197f000        0x0 [heap]
         0x200000000        0x200200000   0x200000        0x0 
         0x200200000        0x200400000   0x200000        0x0 /dev/nvidiactl
         0x200400000        0x200600000   0x200000        0x0 /dev/nvidia0
         0x200600000        0x203e00000  0x3800000        0x0 /dev/nvidiactl
         0x203e00000        0x204000000   0x200000        0x0 /dev/nvidia0
         0x204000000        0x205000000  0x1000000        0x0 
         0x205000000        0x205200000   0x200000        0x0 /dev/nvidiactl
         0x205200000        0x205400000   0x200000        0x0 /dev/nvidiactl
         0x205400000        0x205600000   0x200000 0x205400000 /dev/nvidia-uvm
         0x205600000        0x205800000   0x200000        0x0 /dev/nvidiactl
         0x205800000        0x205a00000   0x200000        0x0 
         0x205a00000        0x205c00000   0x200000        0x0 /dev/nvidiactl
         0x205c00000        0x300200000 0xfa600000        0x0 
       0x10000000000      0x10004000000  0x4000000        0x0 
      0x7fffb30a3000     0x7fffb59e8000  0x2945000        0x0 
      0x7fffb59e8000     0x7fffb5cf4000   0x30c000        0x0 /dev/nvidiactl
      0x7fffb5cf4000     0x7fffb6000000   0x30c000        0x0 /dev/nvidiactl
      0x7fffb6000000     0x7fffb632a000   0x32a000        0x0 /dev/zero (deleted)
      0x7fffb632a000     0x7fffb6400000    0xd6000        0x0 
      0x7fffb6400000     0x7fffb672a000   0x32a000        0x0 /dev/zero (deleted)
      0x7fffb672a000     0x7fffb6800000    0xd6000        0x0 
      0x7fffb6800000     0x7fffb6b2a000   0x32a000        0x0 /dev/zero (deleted)
      0x7fffb6b2a000     0x7fffb6c00000    0xd6000        0x0 
      0x7fffb6c00000     0x7fffb6f2a000   0x32a000        0x0 /dev/zero (deleted)
      0x7fffb6f2a000     0x7fffb7000000    0xd6000        0x0 
      0x7fffb7000000     0x7fffb732a000   0x32a000        0x0 /dev/zero (deleted)
      0x7fffb732a000     0x7fffb7400000    0xd6000        0x0 
      0x7fffb7400000     0x7fffb772a000   0x32a000        0x0 /dev/zero (deleted)
      0x7fffb772a000     0x7fffb8000000   0x8d6000        0x0 
      0x7fffb8004000     0x7fffb8310000   0x30c000        0x0 /dev/nvidiactl
      0x7fffb8310000     0x7fffb861c000   0x30c000        0x0 /dev/nvidiactl
      0x7fffb861c000     0x7fffb8928000   0x30c000        0x0 /dev/nvidiactl
      0x7fffb8928000     0x7fffb8c34000   0x30c000        0x0 /dev/nvidiactl
      0x7fffb8c34000     0x7fffb8f40000   0x30c000        0x0 /dev/nvidiactl
      0x7fffb8f40000     0x7fffb924c000   0x30c000        0x0 /dev/nvidiactl
      0x7fffb924c000     0x7fffb9558000   0x30c000        0x0 /dev/nvidiactl
      0x7fffb9558000     0x7fffb9864000   0x30c000        0x0 /dev/nvidiactl
      0x7fffb9864000     0x7fffb9b70000   0x30c000        0x0 /dev/nvidiactl
      0x7fffb9b70000     0x7fffb9e7c000   0x30c000        0x0 /dev/nvidiactl
      0x7fffb9e7c000     0x7fffba188000   0x30c000        0x0 /dev/nvidiactl
      0x7fffba188000     0x7fffba494000   0x30c000        0x0 /dev/nvidiactl
      0x7fffba494000     0x7fffba7a0000   0x30c000        0x0 /dev/nvidiactl
      0x7fffba7a0000     0x7fffbaaac000   0x30c000        0x0 /dev/nvidiactl
      0x7fffbaaac000     0x7fffbadb8000   0x30c000        0x0 /dev/nvidiactl
      0x7fffbadb8000     0x7fffbb0c4000   0x30c000        0x0 /dev/nvidiactl
      0x7fffbb0c4000     0x7fffbb3d0000   0x30c000        0x0 /dev/nvidiactl
      0x7fffbb3d0000     0x7fffbb6dc000   0x30c000        0x0 /dev/nvidiactl
      0x7fffbb6dc000     0x7fffbb9e8000   0x30c000        0x0 /dev/nvidiactl
      0x7fffbb9e8000     0x7fffbbcf4000   0x30c000        0x0 /dev/nvidiactl
      0x7fffbbcf4000     0x7fffbc000000   0x30c000        0x0 /dev/nvidiactl
      0x7fffbc000000     0x7fffbc32a000   0x32a000        0x0 /dev/zero (deleted)
      0x7fffbc32a000     0x7fffbc400000    0xd6000        0x0 
      0x7fffbc400000     0x7fffbc72a000   0x32a000        0x0 /dev/zero (deleted)
      0x7fffbc72a000     0x7fffbc800000    0xd6000        0x0 
      0x7fffbc800000     0x7fffbcb2a000   0x32a000        0x0 /dev/zero (deleted)
      0x7fffbcb2a000     0x7fffbcc00000    0xd6000        0x0 
      0x7fffbcc00000     0x7fffbcf2a000   0x32a000        0x0 /dev/zero (deleted)
      0x7fffbcf2a000     0x7fffbd000000    0xd6000        0x0 
      0x7fffbd000000     0x7fffbd32a000   0x32a000        0x0 /dev/zero (deleted)
      0x7fffbd32a000     0x7fffbd400000    0xd6000        0x0 
      0x7fffbd400000     0x7fffbd72a000   0x32a000        0x0 /dev/zero (deleted)
      0x7fffbd72a000     0x7fffbd800000    0xd6000        0x0 
      0x7fffbd800000     0x7fffbdb2a000   0x32a000        0x0 /dev/zero (deleted)
      0x7fffbdb2a000     0x7fffbdc00000    0xd6000        0x0 
      0x7fffbdc00000     0x7fffbdf2a000   0x32a000        0x0 /dev/zero (deleted)
      0x7fffbdf2a000     0x7fffbe000000    0xd6000        0x0 
      0x7fffbe188000     0x7fffbe494000   0x30c000        0x0 /dev/nvidiactl
      0x7fffbe494000     0x7fffbe7a0000   0x30c000        0x0 /dev/nvidiactl
      0x7fffbe7a0000     0x7fffbeaac000   0x30c000        0x0 /dev/nvidiactl
      0x7fffbeaac000     0x7fffbedb8000   0x30c000        0x0 /dev/nvidiactl
      0x7fffbedb8000     0x7fffbf0c4000   0x30c000        0x0 /dev/nvidiactl
      0x7fffbf0c4000     0x7fffbf3d0000   0x30c000        0x0 /dev/nvidiactl
      0x7fffbf3d0000     0x7fffbf6dc000   0x30c000        0x0 /dev/nvidiactl
      0x7fffbf6dc000     0x7fffbf9e8000   0x30c000        0x0 /dev/nvidiactl
      0x7fffbf9e8000     0x7fffbfcf4000   0x30c000        0x0 /dev/nvidiactl
      0x7fffbfcf4000     0x7fffc0000000   0x30c000        0x0 /dev/nvidiactl
      0x7fffc0000000     0x7fffc4c00000  0x4c00000        0x0 
      0x7fffc4c00000     0x7fffc4f2a000   0x32a000        0x0 /dev/zero (deleted)
      0x7fffc4f2a000     0x7fffc5000000    0xd6000        0x0 
      0x7fffc5000000     0x7fffc532a000   0x32a000        0x0 /dev/zero (deleted)
      0x7fffc532a000     0x7fffc5400000    0xd6000        0x0 
      0x7fffc5400000     0x7fffc572a000   0x32a000        0x0 /dev/zero (deleted)
      0x7fffc572a000     0x7fffc5800000    0xd6000        0x0 
      0x7fffc5800000     0x7fffc5b2a000   0x32a000        0x0 /dev/zero (deleted)
      0x7fffc5b2a000     0x7fffc5c00000    0xd6000        0x0 
      0x7fffc5c00000     0x7fffc5f2a000   0x32a000        0x0 /dev/zero (deleted)
      0x7fffc5f2a000     0x7fffc7200000  0x12d6000        0x0 
      0x7fffc7200000     0x7fffc752a000   0x32a000        0x0 /dev/zero (deleted)
      0x7fffc752a000     0x7fffc7600000    0xd6000        0x0 
      0x7fffc7600000     0x7fffc792a000   0x32a000        0x0 /dev/zero (deleted)
      0x7fffc792a000     0x7fffc7a00000    0xd6000        0x0 
      0x7fffc7a00000     0x7fffc7d2a000   0x32a000        0x0 /dev/zero (deleted)
      0x7fffc7d2a000     0x7fffc8000000   0x2d6000        0x0 
      0x7fffc8000000     0x7fffc8021000    0x21000        0x0 
      0x7fffc8021000     0x7fffcc000000  0x3fdf000        0x0 
      0x7fffcc188000     0x7fffcc494000   0x30c000        0x0 /dev/nvidiactl
      0x7fffcc494000     0x7fffcc7a0000   0x30c000        0x0 /dev/nvidiactl
      0x7fffcc7a0000     0x7fffccaac000   0x30c000        0x0 /dev/nvidiactl
      0x7fffccaac000     0x7fffccdb8000   0x30c000        0x0 /dev/nvidiactl
      0x7fffccdb8000     0x7fffcd0c4000   0x30c000        0x0 /dev/nvidiactl
      0x7fffcd0c4000     0x7fffcd3d0000   0x30c000        0x0 /dev/nvidiactl
      0x7fffcd3d0000     0x7fffcd6dc000   0x30c000        0x0 /dev/nvidiactl
      0x7fffcd6dc000     0x7fffcd9e8000   0x30c000        0x0 /dev/nvidiactl
      0x7fffcd9e8000     0x7fffcdcf4000   0x30c000        0x0 /dev/nvidiactl
      0x7fffcdcf4000     0x7fffce000000   0x30c000        0x0 /dev/nvidiactl
      0x7fffce000000     0x7fffce400000   0x400000        0x0 
      0x7fffce400000     0x7fffce600000   0x200000        0x0 /dev/zero (deleted)
      0x7fffce600000     0x7fffce800000   0x200000        0x0 /dev/nvidiactl
      0x7fffce800000     0x7fffcea00000   0x200000        0x0 /dev/zero (deleted)
      0x7fffcea00000     0x7fffcee00000   0x400000        0x0 
      0x7fffcee00000     0x7fffcf000000   0x200000        0x0 /dev/nvidiactl
      0x7fffcf000000     0x7fffcf200000   0x200000        0x0 
      0x7fffcf200000     0x7fffcf400000   0x200000        0x0 /dev/zero (deleted)
      0x7fffcf400000     0x7fffcf4e7000    0xe7000        0x0 /dev/nvidiactl
      0x7fffcf4e7000     0x7fffcf600000   0x119000        0x0 
      0x7fffcf600000     0x7fffcf92a000   0x32a000        0x0 /dev/zero (deleted)
      0x7fffcf92a000     0x7fffcfa00000    0xd6000        0x0 
      0x7fffcfa00000     0x7fffcfd2a000   0x32a000        0x0 /dev/zero (deleted)
      0x7fffcfd2a000     0x7fffe0000000 0x102d6000        0x0 
      0x7fffe00ce000     0x7fffe00d3000     0x5000        0x0 /dev/nvidiactl
      0x7fffe00d3000     0x7fffe00d4000     0x1000        0x0 /dev/nvidiactl
      0x7fffe00d4000     0x7fffe00d9000     0x5000        0x0 /dev/nvidiactl
      0x7fffe00d9000     0x7fffe00da000     0x1000        0x0 /dev/nvidiactl
      0x7fffe00da000     0x7fffe00df000     0x5000        0x0 /dev/nvidiactl
      0x7fffe00df000     0x7fffe00e0000     0x1000        0x0 /dev/nvidiactl
      0x7fffe00e0000     0x7fffe00e5000     0x5000        0x0 /dev/nvidiactl
      0x7fffe00e5000     0x7fffe00e6000     0x1000        0x0 /dev/nvidiactl
      0x7fffe00e6000     0x7fffe00eb000     0x5000        0x0 /dev/nvidiactl
      0x7fffe00eb000     0x7fffe00ec000     0x1000        0x0 /dev/nvidiactl
      0x7fffe00ec000     0x7fffe00f1000     0x5000        0x0 /dev/nvidiactl
      0x7fffe00f1000     0x7fffe00f2000     0x1000        0x0 /dev/nvidiactl
      0x7fffe00f2000     0x7fffe00f7000     0x5000        0x0 /dev/nvidiactl
      0x7fffe00f7000     0x7fffe00f8000     0x1000        0x0 /dev/nvidiactl
      0x7fffe00f8000     0x7fffe00fd000     0x5000        0x0 /dev/nvidiactl
      0x7fffe00fd000     0x7fffe00fe000     0x1000        0x0 /dev/nvidiactl
      0x7fffe00fe000     0x7fffe0103000     0x5000        0x0 /dev/nvidiactl
      0x7fffe0103000     0x7fffe0104000     0x1000        0x0 /dev/nvidiactl
      0x7fffe0104000     0x7fffe0109000     0x5000        0x0 /dev/nvidiactl
      0x7fffe0109000     0x7fffe010a000     0x1000        0x0 /dev/nvidiactl
      0x7fffe010a000     0x7fffe010f000     0x5000        0x0 /dev/nvidiactl
      0x7fffe010f000     0x7fffe0110000     0x1000        0x0 /dev/nvidiactl
      0x7fffe0110000     0x7fffe0115000     0x5000        0x0 /dev/nvidiactl
      0x7fffe0115000     0x7fffe0116000     0x1000        0x0 /dev/nvidiactl
      0x7fffe0116000     0x7fffe011b000     0x5000        0x0 /dev/nvidiactl
      0x7fffe011b000     0x7fffe011c000     0x1000        0x0 /dev/nvidiactl
      0x7fffe011c000     0x7fffe0121000     0x5000        0x0 /dev/nvidiactl
      0x7fffe0121000     0x7fffe0122000     0x1000        0x0 /dev/nvidiactl
      0x7fffe0122000     0x7fffe0127000     0x5000        0x0 /dev/nvidiactl
      0x7fffe0127000     0x7fffe0128000     0x1000        0x0 /dev/nvidiactl
      0x7fffe0128000     0x7fffe012d000     0x5000        0x0 /dev/nvidiactl
      0x7fffe012d000     0x7fffe012e000     0x1000        0x0 /dev/nvidiactl
      0x7fffe012e000     0x7fffe0133000     0x5000        0x0 /dev/nvidiactl
      0x7fffe0133000     0x7fffe0134000     0x1000        0x0 /dev/nvidiactl
      0x7fffe0134000     0x7fffe0139000     0x5000        0x0 /dev/nvidiactl
      0x7fffe0139000     0x7fffe013a000     0x1000        0x0 /dev/nvidiactl
      0x7fffe013a000     0x7fffe013f000     0x5000        0x0 /dev/nvidiactl
      0x7fffe013f000     0x7fffe0140000     0x1000        0x0 /dev/nvidiactl
      0x7fffe0140000     0x7fffe0145000     0x5000        0x0 /dev/nvidiactl
      0x7fffe0145000     0x7fffe0146000     0x1000        0x0 /dev/nvidiactl
      0x7fffe0146000     0x7fffe014b000     0x5000        0x0 /dev/nvidiactl
      0x7fffe014b000     0x7fffe014c000     0x1000        0x0 /dev/nvidiactl
      0x7fffe014c000     0x7fffe0151000     0x5000        0x0 /dev/nvidiactl
      0x7fffe0151000     0x7fffe0152000     0x1000        0x0 /dev/nvidiactl
      0x7fffe0152000     0x7fffe045e000   0x30c000        0x0 /dev/nvidiactl
      0x7fffe045e000     0x7fffe0463000     0x5000        0x0 /dev/nvidiactl
      0x7fffe0463000     0x7fffe0464000     0x1000        0x0 /dev/nvidiactl
      0x7fffe0464000     0x7fffe0770000   0x30c000        0x0 /dev/nvidiactl
      0x7fffe0770000     0x7fffe0a7c000   0x30c000        0x0 /dev/nvidiactl
      0x7fffe0a7c000     0x7fffe0a81000     0x5000        0x0 /dev/nvidiactl
      0x7fffe0a81000     0x7fffe0a82000     0x1000        0x0 /dev/nvidiactl
      0x7fffe0a82000     0x7fffe0d8e000   0x30c000        0x0 /dev/nvidiactl
      0x7fffe0d8e000     0x7fffe109a000   0x30c000        0x0 /dev/nvidiactl
      0x7fffe109a000     0x7fffe109b000     0x1000        0x0 /dev/nvidiactl
      0x7fffe109b000     0x7fffe109c000     0x1000        0x0 /dev/nvidiactl
      0x7fffe109c000     0x7fffe109d000     0x1000        0x0 /dev/nvidiactl
      0x7fffe109d000     0x7fffe109e000     0x1000        0x0 /dev/nvidiactl
      0x7fffe109e000     0x7fffe109f000     0x1000        0x0 /dev/nvidiactl
      0x7fffe109f000     0x7fffe10a0000     0x1000        0x0 /dev/nvidiactl
      0x7fffe10a0000     0x7fffe10a1000     0x1000        0x0 /dev/nvidiactl
      0x7fffe10a1000     0x7fffe10a2000     0x1000        0x0 /dev/nvidiactl
      0x7fffe10a2000     0x7fffe10a3000     0x1000        0x0 /dev/nvidiactl
      0x7fffe10a3000     0x7fffe10a4000     0x1000        0x0 /dev/nvidiactl
      0x7fffe10a4000     0x7fffe10a5000     0x1000        0x0 /dev/nvidiactl
      0x7fffe10a5000     0x7fffe10a6000     0x1000        0x0 /dev/nvidiactl
      0x7fffe10a6000     0x7fffe10a7000     0x1000        0x0 /dev/nvidiactl
      0x7fffe10a7000     0x7fffe10a8000     0x1000        0x0 /dev/nvidiactl
      0x7fffe10a8000     0x7fffe10a9000     0x1000        0x0 /dev/nvidiactl
      0x7fffe10a9000     0x7fffe10aa000     0x1000        0x0 /dev/nvidiactl
      0x7fffe10aa000     0x7fffe10ad000     0x3000        0x0 /dev/nvidiactl
      0x7fffe10ad000     0x7fffe10b0000     0x3000        0x0 /dev/nvidiactl
      0x7fffe10b0000     0x7fffe10b3000     0x3000        0x0 /dev/nvidiactl
      0x7fffe10b3000     0x7fffe10b6000     0x3000        0x0 /dev/nvidiactl
      0x7fffe10b6000     0x7fffe10b9000     0x3000        0x0 /dev/nvidiactl
      0x7fffe10b9000     0x7fffe10bc000     0x3000        0x0 /dev/nvidiactl
      0x7fffe10bc000     0x7fffe10bf000     0x3000        0x0 /dev/nvidiactl
      0x7fffe10bf000     0x7fffe10c2000     0x3000        0x0 /dev/nvidiactl
      0x7fffe10c2000     0x7fffe10c5000     0x3000        0x0 /dev/nvidiactl
      0x7fffe10c5000     0x7fffe10c8000     0x3000        0x0 /dev/nvidiactl
      0x7fffe10c8000     0x7fffe10cb000     0x3000        0x0 /dev/nvidiactl
      0x7fffe10cb000     0x7fffe10ce000     0x3000        0x0 /dev/nvidiactl
      0x7fffe10ce000     0x7fffe10d1000     0x3000        0x0 /dev/nvidiactl
      0x7fffe10d1000     0x7fffe10d4000     0x3000        0x0 /dev/nvidiactl
      0x7fffe10d4000     0x7fffe10d7000     0x3000        0x0 /dev/nvidiactl
      0x7fffe10d7000     0x7fffe10da000     0x3000        0x0 /dev/nvidiactl
      0x7fffe10da000     0x7fffe1103000    0x29000        0x0 /dev/nvidiactl
      0x7fffe1103000     0x7fffe112c000    0x29000        0x0 /dev/nvidiactl
      0x7fffe112c000     0x7fffe1154000    0x28000        0x0 /dev/nvidiactl
      0x7fffe1154000     0x7fffe124c000    0xf8000        0x0 
      0x7fffe124c000     0x7fffe144c000   0x200000        0x0 /dev/zero (deleted)
      0x7fffe144c000     0x7fffe144d000     0x1000        0x0 
      0x7fffe144d000     0x7fffe1cce000   0x881000        0x0 
      0x7fffe1cce000     0x7fffe1ccf000     0x1000        0x0 
      0x7fffe1ccf000     0x7fffe24cf000   0x800000        0x0 
      0x7fffe24cf000     0x7fffe24d0000     0x1000        0x0 
      0x7fffe24d0000     0x7fffe2cd0000   0x800000        0x0 
      0x7fffe2cd0000     0x7fffe8cd0000  0x6000000        0x0 
      0x7fffe8cd0000     0x7fffe8fc8000   0x2f8000        0x0 /usr/lib/x86_64-linux-gnu/
      0x7fffe8fc8000     0x7fffe91c7000   0x1ff000   0x2f8000 /usr/lib/x86_64-linux-gnu/
      0x7fffe91c7000     0x7fffe91e2000    0x1b000   0x2f7000 /usr/lib/x86_64-linux-gnu/
      0x7fffe91e2000     0x7fffe91e4000     0x2000        0x0 
      0x7fffe91e4000     0x7fffe91fa000    0x16000        0x0 /usr/lib/x86_64-linux-gnu/
      0x7fffe91fa000     0x7fffe93fa000   0x200000    0x16000 /usr/lib/x86_64-linux-gnu/
      0x7fffe93fa000     0x7fffe93fd000     0x3000    0x16000 /usr/lib/x86_64-linux-gnu/
      0x7fffe93fd000     0x7fffea2d7000   0xeda000        0x0 /usr/lib/x86_64-linux-gnu/
      0x7fffea2d7000     0x7fffea4d6000   0x1ff000   0xeda000 /usr/lib/x86_64-linux-gnu/
      0x7fffea4d6000     0x7fffea5f1000   0x11b000   0xed9000 /usr/lib/x86_64-linux-gnu/
      0x7fffea5f1000     0x7fffea607000    0x16000        0x0 
      0x7fffea607000     0x7fffea608000     0x1000        0x0 
      0x7fffea608000     0x7fffeae08000   0x800000        0x0 
      0x7fffeae08000     0x7fffeae09000     0x1000        0x0 
      0x7fffeae09000     0x7fffeb609000   0x800000        0x0 
      0x7fffeb609000     0x7fffeb60a000     0x1000        0x0 
      0x7fffeb60a000     0x7fffebe0a000   0x800000        0x0 
      0x7fffebe0a000     0x7fffebe0b000     0x1000        0x0 
      0x7fffebe0b000     0x7fffec60b000   0x800000        0x0 
      0x7fffec60b000     0x7fffec60c000     0x1000        0x0 
      0x7fffec60c000     0x7fffece0c000   0x800000        0x0 
      0x7fffece0c000     0x7fffece0d000     0x1000        0x0 
      0x7fffece0d000     0x7fffed60d000   0x800000        0x0 
      0x7fffed60d000     0x7fffed60e000     0x1000        0x0 
      0x7fffed60e000     0x7fffede0e000   0x800000        0x0 
      0x7fffede0e000     0x7fffede0f000     0x1000        0x0 
      0x7fffede0f000     0x7fffee60f000   0x800000        0x0 
      0x7fffee60f000     0x7fffee610000     0x1000        0x0 
      0x7fffee610000     0x7fffeee10000   0x800000        0x0 
      0x7fffeee10000     0x7fffeee11000     0x1000        0x0 
      0x7fffeee11000     0x7fffef611000   0x800000        0x0 
      0x7fffef611000     0x7fffef612000     0x1000        0x0 
      0x7fffef612000     0x7fffefe12000   0x800000        0x0 
      0x7fffefe12000     0x7fffefe13000     0x1000        0x0 
      0x7fffefe13000     0x7ffff0613000   0x800000        0x0 
      0x7ffff0613000     0x7ffff0614000     0x1000        0x0 
      0x7ffff0614000     0x7ffff0e14000   0x800000        0x0 
      0x7ffff0e14000     0x7ffff0e15000     0x1000        0x0 
      0x7ffff0e15000     0x7ffff1615000   0x800000        0x0 
      0x7ffff1615000     0x7ffff1616000     0x1000        0x0 
      0x7ffff1616000     0x7ffff1e16000   0x800000        0x0 
      0x7ffff1e16000     0x7ffff1e17000     0x1000        0x0 
      0x7ffff1e17000     0x7ffff2617000   0x800000        0x0 
      0x7ffff290f000     0x7ffff2985000    0x76000        0x0 /usr/local/cuda-10.1/targets/x86_64-linux/lib/
      0x7ffff2985000     0x7ffff2b84000   0x1ff000    0x76000 /usr/local/cuda-10.1/targets/x86_64-linux/lib/
      0x7ffff2b84000     0x7ffff2b88000     0x4000    0x75000 /usr/local/cuda-10.1/targets/x86_64-linux/lib/
      0x7ffff2b88000     0x7ffff2b8a000     0x2000        0x0 
      0x7ffff2b8a000     0x7ffff2ba0000    0x16000        0x0 /lib/x86_64-linux-gnu/
      0x7ffff2ba0000     0x7ffff2d9f000   0x1ff000    0x16000 /lib/x86_64-linux-gnu/
      0x7ffff2d9f000     0x7ffff2da0000     0x1000    0x15000 /lib/x86_64-linux-gnu/
      0x7ffff2da0000     0x7ffff2f12000   0x172000        0x0 /usr/lib/x86_64-linux-gnu/
      0x7ffff2f12000     0x7ffff3112000   0x200000   0x172000 /usr/lib/x86_64-linux-gnu/
      0x7ffff3112000     0x7ffff311c000     0xa000   0x172000 /usr/lib/x86_64-linux-gnu/
      0x7ffff311c000     0x7ffff311e000     0x2000   0x17c000 /usr/lib/x86_64-linux-gnu/
      0x7ffff311e000     0x7ffff3122000     0x4000        0x0 
      0x7ffff3122000     0x7ffff3129000     0x7000        0x0 /lib/x86_64-linux-gnu/
      0x7ffff3129000     0x7ffff3328000   0x1ff000     0x7000 /lib/x86_64-linux-gnu/
      0x7ffff3328000     0x7ffff3329000     0x1000     0x6000 /lib/x86_64-linux-gnu/
      0x7ffff3329000     0x7ffff332a000     0x1000     0x7000 /lib/x86_64-linux-gnu/
      0x7ffff332a000     0x7ffff332f000     0x5000        0x0 /usr/lib/x86_64-linux-gnu/
      0x7ffff332f000     0x7ffff352e000   0x1ff000     0x5000 /usr/lib/x86_64-linux-gnu/
      0x7ffff352e000     0x7ffff352f000     0x1000     0x4000 /usr/lib/x86_64-linux-gnu/
      0x7ffff352f000     0x7ffff3530000     0x1000     0x5000 /usr/lib/x86_64-linux-gnu/
      0x7ffff3530000     0x7ffff3532000     0x2000        0x0 /usr/lib/x86_64-linux-gnu/
      0x7ffff3532000     0x7ffff3732000   0x200000     0x2000 /usr/lib/x86_64-linux-gnu/
      0x7ffff3732000     0x7ffff3733000     0x1000     0x2000 /usr/lib/x86_64-linux-gnu/
      0x7ffff3733000     0x7ffff3734000     0x1000     0x3000 /usr/lib/x86_64-linux-gnu/
      0x7ffff3734000     0x7ffff38f4000   0x1c0000        0x0 /lib/x86_64-linux-gnu/
      0x7ffff38f4000     0x7ffff3af4000   0x200000   0x1c0000 /lib/x86_64-linux-gnu/
      0x7ffff3af4000     0x7ffff3af8000     0x4000   0x1c0000 /lib/x86_64-linux-gnu/
      0x7ffff3af8000     0x7ffff3afa000     0x2000   0x1c4000 /lib/x86_64-linux-gnu/
      0x7ffff3afa000     0x7ffff3afe000     0x4000        0x0 
      0x7ffff3afe000     0x7ffff3b16000    0x18000        0x0 /lib/x86_64-linux-gnu/
      0x7ffff3b16000     0x7ffff3d15000   0x1ff000    0x18000 /lib/x86_64-linux-gnu/
      0x7ffff3d15000     0x7ffff3d16000     0x1000    0x17000 /lib/x86_64-linux-gnu/
      0x7ffff3d16000     0x7ffff3d17000     0x1000    0x18000 /lib/x86_64-linux-gnu/
      0x7ffff3d17000     0x7ffff3d1b000     0x4000        0x0 
      0x7ffff3d1b000     0x7ffff3d34000    0x19000        0x0 /lib/x86_64-linux-gnu/
      0x7ffff3d34000     0x7ffff3f33000   0x1ff000    0x19000 /lib/x86_64-linux-gnu/
      0x7ffff3f33000     0x7ffff3f34000     0x1000    0x18000 /lib/x86_64-linux-gnu/
      0x7ffff3f34000     0x7ffff3f35000     0x1000    0x19000 /lib/x86_64-linux-gnu/
      0x7ffff3f35000     0x7ffff3f38000     0x3000        0x0 /lib/x86_64-linux-gnu/
      0x7ffff3f38000     0x7ffff4137000   0x1ff000     0x3000 /lib/x86_64-linux-gnu/
      0x7ffff4137000     0x7ffff4138000     0x1000     0x2000 /lib/x86_64-linux-gnu/
      0x7ffff4138000     0x7ffff4139000     0x1000     0x3000 /lib/x86_64-linux-gnu/
      0x7ffff4139000     0x7ffff4902000   0x7c9000        0x0 /usr/local/cuda-10.1/targets/x86_64-linux/lib/
      0x7ffff4902000     0x7ffff4b01000   0x1ff000   0x7c9000 /usr/local/cuda-10.1/targets/x86_64-linux/lib/
      0x7ffff4b01000     0x7ffff4b06000     0x5000   0x7c8000 /usr/local/cuda-10.1/targets/x86_64-linux/lib/
      0x7ffff4b06000     0x7ffff4b29000    0x23000        0x0 
      0x7ffff4b29000     0x7ffff4f5d000   0x434000        0x0 /usr/local/cuda-10.1/targets/x86_64-linux/lib/
      0x7ffff4f5d000     0x7ffff515d000   0x200000   0x434000 /usr/local/cuda-10.1/targets/x86_64-linux/lib/
      0x7ffff515d000     0x7ffff5162000     0x5000   0x434000 /usr/local/cuda-10.1/targets/x86_64-linux/lib/
      0x7ffff5162000     0x7ffff5166000     0x4000        0x0 
      0x7ffff5166000     0x7ffff6a2b000  0x18c5000        0x0 /usr/local/cuda-10.1/targets/x86_64-linux/lib/
      0x7ffff6a2b000     0x7ffff6c2a000   0x1ff000  0x18c5000 /usr/local/cuda-10.1/targets/x86_64-linux/lib/
      0x7ffff6c2a000     0x7ffff6c30000     0x6000  0x18c4000 /usr/local/cuda-10.1/targets/x86_64-linux/lib/
      0x7ffff6c30000     0x7ffff6c38000     0x8000        0x0 
      0x7ffff6c38000     0x7ffff6d98000   0x160000        0x0 /usr/local/lib/x86_64-linux-gnu/
      0x7ffff6d98000     0x7ffff6f98000   0x200000   0x160000 /usr/local/lib/x86_64-linux-gnu/
      0x7ffff6f98000     0x7ffff6f99000     0x1000   0x160000 /usr/local/lib/x86_64-linux-gnu/
      0x7ffff6f99000     0x7ffff6f9c000     0x3000   0x161000 /usr/local/lib/x86_64-linux-gnu/
      0x7ffff6f9c000     0x7ffff7095000    0xf9000        0x0 /usr/lib/x86_64-linux-gnu/
      0x7ffff7095000     0x7ffff7294000   0x1ff000    0xf9000 /usr/lib/x86_64-linux-gnu/
      0x7ffff7294000     0x7ffff729b000     0x7000    0xf8000 /usr/lib/x86_64-linux-gnu/
      0x7ffff729b000     0x7ffff729c000     0x1000    0xff000 /usr/lib/x86_64-linux-gnu/
      0x7ffff729c000     0x7ffff72a2000     0x6000        0x0 /usr/lib/x86_64-linux-gnu/
      0x7ffff72a2000     0x7ffff74a2000   0x200000     0x6000 /usr/lib/x86_64-linux-gnu/
      0x7ffff74a2000     0x7ffff74a3000     0x1000     0x6000 /usr/lib/x86_64-linux-gnu/
      0x7ffff74a3000     0x7ffff74a4000     0x1000     0x7000 /usr/lib/x86_64-linux-gnu/
      0x7ffff74a4000     0x7ffff74a7000     0x3000        0x0 /usr/lib/x86_64-linux-gnu/
      0x7ffff74a7000     0x7ffff76a6000   0x1ff000     0x3000 /usr/lib/x86_64-linux-gnu/
      0x7ffff76a6000     0x7ffff76a7000     0x1000     0x2000 /usr/lib/x86_64-linux-gnu/
      0x7ffff76a7000     0x7ffff76a8000     0x1000     0x3000 /usr/lib/x86_64-linux-gnu/
      0x7ffff76a8000     0x7ffff76aa000     0x2000        0x0 /usr/lib/x86_64-linux-gnu/
      0x7ffff76aa000     0x7ffff78aa000   0x200000     0x2000 /usr/lib/x86_64-linux-gnu/
      0x7ffff78aa000     0x7ffff78ab000     0x1000     0x2000 /usr/lib/x86_64-linux-gnu/
      0x7ffff78ab000     0x7ffff78ac000     0x1000     0x3000 /usr/lib/x86_64-linux-gnu/
      0x7ffff78ac000     0x7ffff78cd000    0x21000        0x0 /usr/lib/x86_64-linux-gnu/
      0x7ffff78cd000     0x7ffff7acc000   0x1ff000    0x21000 /usr/lib/x86_64-linux-gnu/
      0x7ffff7acc000     0x7ffff7acd000     0x1000    0x20000 /usr/lib/x86_64-linux-gnu/
      0x7ffff7acd000     0x7ffff7ace000     0x1000    0x21000 /usr/lib/x86_64-linux-gnu/
      0x7ffff7ace000     0x7ffff7bd6000   0x108000        0x0 /lib/x86_64-linux-gnu/
      0x7ffff7bd6000     0x7ffff7dd5000   0x1ff000   0x108000 /lib/x86_64-linux-gnu/
      0x7ffff7dd5000     0x7ffff7dd6000     0x1000   0x107000 /lib/x86_64-linux-gnu/
      0x7ffff7dd6000     0x7ffff7dd7000     0x1000   0x108000 /lib/x86_64-linux-gnu/
      0x7ffff7dd7000     0x7ffff7dfd000    0x26000        0x0 /lib/x86_64-linux-gnu/
      0x7ffff7dfd000     0x7ffff7dfe000     0x1000        0x0 /dev/nvidiactl
      0x7ffff7dfe000     0x7ffff7dff000     0x1000        0x0 /dev/nvidiactl
      0x7ffff7dff000     0x7ffff7e00000     0x1000        0x0 /dev/nvidiactl
      0x7ffff7e00000     0x7ffff7e01000     0x1000        0x0 /dev/nvidia0
      0x7ffff7e01000     0x7ffff7e02000     0x1000        0x0 /dev/nvidiactl
      0x7ffff7e02000     0x7ffff7e03000     0x1000        0x0 /dev/nvidiactl
      0x7ffff7e03000     0x7ffff7e04000     0x1000        0x0 /dev/nvidia0
      0x7ffff7e04000     0x7ffff7e05000     0x1000        0x0 /dev/nvidiactl
      0x7ffff7e05000     0x7ffff7e15000    0x10000        0x0 /dev/nvidia0
      0x7ffff7e15000     0x7ffff7fd2000   0x1bd000        0x0 
      0x7ffff7fd2000     0x7ffff7fd3000     0x1000        0x0 /dev/nvidiactl
      0x7ffff7fd3000     0x7ffff7fd4000     0x1000        0x0 /dev/nvidiactl
      0x7ffff7fd4000     0x7ffff7fd5000     0x1000        0x0 /dev/nvidiactl
      0x7ffff7fd5000     0x7ffff7fd6000     0x1000        0x0 /dev/nvidiactl
      0x7ffff7fd6000     0x7ffff7fd7000     0x1000        0x0 /dev/nvidiactl
      0x7ffff7fd7000     0x7ffff7fd8000     0x1000        0x0 /dev/nvidiactl
      0x7ffff7fd8000     0x7ffff7fd9000     0x1000        0x0 /dev/nvidiactl
      0x7ffff7fd9000     0x7ffff7fda000     0x1000        0x0 /dev/nvidiactl
      0x7ffff7fda000     0x7ffff7fdb000     0x1000        0x0 /dev/nvidiactl
      0x7ffff7fdb000     0x7ffff7fdc000     0x1000        0x0 /dev/nvidiactl
      0x7ffff7fdc000     0x7ffff7fdd000     0x1000        0x0 /dev/nvidiactl
      0x7ffff7fdd000     0x7ffff7fde000     0x1000        0x0 /dev/nvidiactl
      0x7ffff7fde000     0x7ffff7fdf000     0x1000        0x0 /dev/nvidiactl
      0x7ffff7fdf000     0x7ffff7fe0000     0x1000        0x0 /dev/nvidiactl
      0x7ffff7fe0000     0x7ffff7fe1000     0x1000        0x0 /dev/nvidiactl
      0x7ffff7fe1000     0x7ffff7fe2000     0x1000        0x0 /dev/nvidiactl
      0x7ffff7fe2000     0x7ffff7fe3000     0x1000        0x0 /dev/nvidiactl
      0x7ffff7fe3000     0x7ffff7fe4000     0x1000        0x0 /dev/nvidiactl
      0x7ffff7fe4000     0x7ffff7fe5000     0x1000        0x0 /dev/nvidiactl
      0x7ffff7fe5000     0x7ffff7fe6000     0x1000        0x0 /dev/nvidiactl
      0x7ffff7fe6000     0x7ffff7ff6000    0x10000        0x0 /dev/nvidia0
      0x7ffff7ff6000     0x7ffff7ff7000     0x1000        0x0 
      0x7ffff7ff7000     0x7ffff7ffa000     0x3000        0x0 [vvar]
      0x7ffff7ffa000     0x7ffff7ffc000     0x2000        0x0 [vdso]
      0x7ffff7ffc000     0x7ffff7ffd000     0x1000    0x25000 /lib/x86_64-linux-gnu/
      0x7ffff7ffd000     0x7ffff7ffe000     0x1000    0x26000 /lib/x86_64-linux-gnu/
      0x7ffff7ffe000     0x7ffff7fff000     0x1000        0x0 
      0x7ffffffdd000     0x7ffffffff000    0x22000        0x0 [stack]
  0xffffffffff600000 0xffffffffff601000     0x1000        0x0 [vsyscall]


Note that I have integrated the NVENC H.265/HEVC encoder with our broadcast transcoder, and it does not have this problem with the rate-control lookahead. This issue only applies to the H.264 encoder.