Lock bitstream function failed on Tesla V100 using NvHWEncoder and FFmpeg

Hi,

I am using Tesla V100 on x64 Linux Ubuntu 16.04. Installed driver 384.9. When I try to run NvEncoder sample application, I am getting below error.

…/common/src/NvHWEncoder.cpp line 1068: lock bitstream function failed

I get the same error when trying to use FFmpeg also. Please find FFmpeg CLI and error message below:

ffmpeg -i big_buck_bunny_1080p_h264.mov -c:v h264_nvenc bbb.mp4

[h264_nvenc @ 0x16321c0] Failed locking bitstream buffer: invalid param (8)

nvdia-smi output is below:

Thu Dec 7 10:35:56 2017
±----------------------------------------------------------------------------+
| NVIDIA-SMI 384.90 Driver Version: 384.90 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla V100-SXM2… Off | 00000000:00:1E.0 Off | 0 |
| N/A 37C P0 44W / 300W | 512MiB / 16152MiB | 0% Default |
±------------------------------±---------------------±---------------------+

±----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 1662 C ./NvEncoder 501MiB |
±----------------------------------------------------------------------------+

Can you please help?

Best Regards,
Harinandan

Your FFmpeg command is working fine when we tried it. In order to troubleshoot further, can you please provide the following information?

  1. Exact FFmpeg version and how you obtained it (did you compile it or are you building it yourself?)
  2. FFmpeg output with loglevel trace
  3. Video SDK version which gives the same error with NvEncoder application

Hi Abhijit,

I am using Amazon AWS p3.2x instance which has 1 nVIDIA TESLA V 100 GPU installed.

Starting from a fresh instance of Ubuntu 16.04, following were the steps. Referred to https://developer.nvidia.com/ffmpeg getting started section for instructions.

  1. Downloaded and installed driver - nvidia-diag-driver-local-repo-ubuntu1404-384.81_1.0-1_amd64.deb

  2. Downloaded and installed cuda toolkit - cuda-repo-ubuntu1604-9-1-local_9.1.85-1_amd64.deb

  3. Reboot machine.

  4. Download latest Video Codec SDK and compile NvEncoder. Video_Codec_SDK_8.0.14 was the version.

  5. Latest FFmpeg was downloaded from git repository and compiled from source on the machine. Configure command is below:

export PREFIX=/usr/local
export BINDIR=/usr/local/bin

./configure --prefix="$PREFIX" --extra-cflags=-I/usr/local/cuda/include --extra-cflags="-I$PREFIX/include" --extra-ldflags=-L/usr/local/cuda/lib64 --extra-ldflags="-L$PREFIX/lib" --bindir="$BINDIR" --pkg-config-flags="–static" --enable-static --enable-shared --enable-cuda --enable-cuvid --enable-nvenc --enable-nonfree --enable-libnpp --enable-libfdk-aac --enable-libfreetype --enable-libx264 --enable-openssl

FFmpeg version is version N-89525-gb178278

Partial output with loglevel trace is below. It has some lines checking for timestamps at the start

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from ‘big_buck_bunny_1080p_h264.mov’:
Metadata:
major_brand : qt
minor_version : 537199360
compatible_brands: qt
creation_time : 2008-05-27T18:40:35.000000Z
com.apple.quicktime.player.movie.audio.gain: 1.000000
com.apple.quicktime.player.movie.audio.treble: 0.000000
com.apple.quicktime.player.movie.audio.bass: 0.000000
com.apple.quicktime.player.movie.audio.balance: 0.000000
com.apple.quicktime.player.movie.audio.pitchshift: 0.000000
com.apple.quicktime.player.movie.audio.mute:
com.apple.quicktime.player.movie.visual.brightness: 0.000000
com.apple.quicktime.player.movie.visual.color: 1.000000
com.apple.quicktime.player.movie.visual.tint: 0.000000
com.apple.quicktime.player.movie.visual.contrast: 1.000000
com.apple.quicktime.player.version: 7.4.1 (14)
com.apple.quicktime.version: 7.4.1 (14) 0x7418000 (Mac OS X, 10.5.2, 9C31)
timecode : 00:00:00:00
Duration: 00:09:56.46, start: 0.000000, bitrate: 9725 kb/s
Stream #0:0(eng), 2, 1/2400: Video: h264 (Main), 1 reference frame (avc1 / 0x31637661), yuv420p(tv, bt709, topleft), 1920x1080 (1920x1088), 0/1, 9282 kb/s, 24 fps, 24 tbr, 2400 tbn, 4800 tbc (default)
Metadata:
rotate : 0
creation_time : 2008-05-27T18:40:35.000000Z
handler_name : Apple Alias Data Handler
encoder : H.264
Side data:
displaymatrix: rotation of -0.00 degrees
Stream #0:1(eng), 0, 1/2400: Data: none (tmcd / 0x64636D74), 0/1 (default)
Metadata:
rotate : 0
creation_time : 2008-05-27T18:40:35.000000Z
handler_name : Apple Alias Data Handler
timecode : 00:00:00:00
Stream #0:2(eng), 47, 1/48000: Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, 5.1, fltp, 437 kb/s (default)
Metadata:
rotate : 0
creation_time : 2008-05-27T18:40:35.000000Z
handler_name : Apple Alias Data Handler
Successfully opened the file.
Parsing a group of options: output url bbb.mp4.
Applying option c:v (codec name) with argument h264_nvenc.
Successfully parsed a group of options.
Opening an output file: bbb.mp4.
File ‘bbb.mp4’ already exists. Overwrite ? [y/N] y
[file @ 0x1e23260] Setting default whitelist ‘file,crypto’
Successfully opened the file.
detected 8 logical cores
[h264 @ 0x1e2d380] nal_unit_type: 7, nal_ref_idc: 1
[h264 @ 0x1e2d380] nal_unit_type: 8, nal_ref_idc: 1
Stream mapping:
Stream #0:0 -> #0:0 (h264 (native) -> h264 (h264_nvenc))
Stream #0:2 -> #0:1 (aac (native) -> aac (native))
Press [q] to stop, [?] for help
cur_dts is invalid (this is harmless if it occurs once at the start per stream)
[graph_1_in_0_2 @ 0x227aaa0] Setting ‘time_base’ to value ‘1/48000’
[graph_1_in_0_2 @ 0x227aaa0] Setting ‘sample_rate’ to value ‘48000’
[graph_1_in_0_2 @ 0x227aaa0] Setting ‘sample_fmt’ to value ‘fltp’
[graph_1_in_0_2 @ 0x227aaa0] Setting ‘channel_layout’ to value ‘0x3f’
[graph_1_in_0_2 @ 0x227aaa0] tb:1/48000 samplefmt:fltp samplerate:48000 chlayout:0x3f
[format_out_0_1 @ 0x20fedc0] Setting ‘sample_fmts’ to value ‘fltp’
[format_out_0_1 @ 0x20fedc0] Setting ‘sample_rates’ to value ‘96000|88200|64000|48000|44100|32000|24000|22050|16000|12000|11025|8000|7350’
[AVFilterGraph @ 0x1e1c0a0] query_formats: 4 queried, 9 merged, 0 already done, 0 delayed
cur_dts is invalid (this is harmless if it occurs once at the start per stream)
Last message repeated 45 times
[h264 @ 0x1e2d380] nal_unit_type: 6, nal_ref_idc: 0
Last message repeated 1 times
[h264 @ 0x1e2d380] nal_unit_type: 5, nal_ref_idc: 1
Last message repeated 7 times
[h264 @ 0x1e2d380] Format yuv420p chosen by get_format().
[h264 @ 0x1e2d380] Reinit context to 1920x1088, pix_fmt: yuv420p
cur_dts is invalid (this is harmless if it occurs once at the start per stream)
Last message repeated 1 times
[h264 @ 0x1e2d380] no picture
cur_dts is invalid (this is harmless if it occurs once at the start per stream)
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x1e1c1e0] stream 0, sample 2, dts 41667
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x1e1c1e0] stream 1, sample 0, dts 0
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x1e1c1e0] stream 2, sample 47, dts 1002667
[h264 @ 0x1e53f00] nal_unit_type: 6, nal_ref_idc: 0
[h264 @ 0x1e53f00] nal_unit_type: 1, nal_ref_idc: 1
Last message repeated 7 times
cur_dts is invalid (this is harmless if it occurs once at the start per stream)
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x1e1c1e0] stream 0, sample 3, dts 83333
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x1e1c1e0] stream 1, sample 0, dts 0
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x1e1c1e0] stream 2, sample 47, dts 1002667
[h264 @ 0x1e590c0] nal_unit_type: 6, nal_ref_idc: 0
[h264 @ 0x1e590c0] nal_unit_type: 1, nal_ref_idc: 0
Last message repeated 7 times
cur_dts is invalid (this is harmless if it occurs once at the start per stream)
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x1e1c1e0] stream 0, sample 4, dts 125000
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x1e1c1e0] stream 1, sample 0, dts 0
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x1e1c1e0] stream 2, sample 47, dts 1002667
[h264 @ 0x212ae00] nal_unit_type: 6, nal_ref_idc: 0
[h264 @ 0x212ae00] nal_unit_type: 1, nal_ref_idc: 1
Last message repeated 7 times
cur_dts is invalid (this is harmless if it occurs once at the start per stream)
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x1e1c1e0] stream 0, sample 5, dts 166667
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x1e1c1e0] stream 1, sample 0, dts 0
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x1e1c1e0] stream 2, sample 47, dts 1002667
[h264 @ 0x2147040] nal_unit_type: 6, nal_ref_idc: 0
[h264 @ 0x2147040] nal_unit_type: 1, nal_ref_idc: 0
Last message repeated 7 times
cur_dts is invalid (this is harmless if it occurs once at the start per stream)
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x1e1c1e0] stream 0, sample 6, dts 208333
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x1e1c1e0] stream 1, sample 0, dts 0
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x1e1c1e0] stream 2, sample 47, dts 1002667
[h264 @ 0x1f6d520] nal_unit_type: 6, nal_ref_idc: 0
[h264 @ 0x1f6d520] nal_unit_type: 1, nal_ref_idc: 1
Last message repeated 7 times
cur_dts is invalid (this is harmless if it occurs once at the start per stream)
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x1e1c1e0] stream 0, sample 7, dts 250000
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x1e1c1e0] stream 1, sample 0, dts 0
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x1e1c1e0] stream 2, sample 47, dts 1002667
[h264 @ 0x1f893c0] nal_unit_type: 6, nal_ref_idc: 0
[h264 @ 0x1f893c0] nal_unit_type: 1, nal_ref_idc: 0
Last message repeated 7 times
cur_dts is invalid (this is harmless if it occurs once at the start per stream)
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x1e1c1e0] stream 0, sample 8, dts 291667
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x1e1c1e0] stream 1, sample 0, dts 0
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x1e1c1e0] stream 2, sample 47, dts 1002667
[h264 @ 0x1fa5600] nal_unit_type: 6, nal_ref_idc: 0
[h264 @ 0x1fa5600] nal_unit_type: 1, nal_ref_idc: 1
Last message repeated 7 times
[h264 @ 0x1fc1840] nal_unit_type: 6, nal_ref_idc: 0
[h264 @ 0x1fc1840] nal_unit_type: 1, nal_ref_idc: 0
Last message repeated 7 times
cur_dts is invalid (this is harmless if it occurs once at the start per stream)
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x1e1c1e0] stream 0, sample 9, dts 333333
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x1e1c1e0] stream 1, sample 0, dts 0
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x1e1c1e0] stream 2, sample 47, dts 1002667
[h264 @ 0x1e2d380] nal_unit_type: 6, nal_ref_idc: 0
[h264 @ 0x1e2d380] nal_unit_type: 1, nal_ref_idc: 1
Last message repeated 7 times
[graph 0 input from stream 0:0 @ 0x210ec00] Setting ‘video_size’ to value ‘1920x1080’
[graph 0 input from stream 0:0 @ 0x210ec00] Setting ‘pix_fmt’ to value ‘0’
[graph 0 input from stream 0:0 @ 0x210ec00] Setting ‘time_base’ to value ‘1/2400’
[graph 0 input from stream 0:0 @ 0x210ec00] Setting ‘pixel_aspect’ to value ‘0/1’
[graph 0 input from stream 0:0 @ 0x210ec00] Setting ‘sws_param’ to value ‘flags=2’
[graph 0 input from stream 0:0 @ 0x210ec00] Setting ‘frame_rate’ to value ‘24/1’
[graph 0 input from stream 0:0 @ 0x210ec00] w:1920 h:1080 pixfmt:yuv420p tb:1/2400 fr:24/1 sar:0/1 sws_param:flags=2
[format @ 0x2c1c200] Setting ‘pix_fmts’ to value ‘yuv420p|nv12|p010le|yuv444p|yuv444p16le|bgr0|rgb0|cuda’
[AVFilterGraph @ 0x210d600] query_formats: 4 queried, 3 merged, 0 already done, 0 delayed
[h264_nvenc @ 0x1e24600] Loaded lib: libcuda.so.1
[h264_nvenc @ 0x1e24600] Loaded sym: cuInit
[h264_nvenc @ 0x1e24600] Loaded sym: cuDeviceGetCount
[h264_nvenc @ 0x1e24600] Loaded sym: cuDeviceGet
[h264_nvenc @ 0x1e24600] Loaded sym: cuDeviceGetName
[h264_nvenc @ 0x1e24600] Loaded sym: cuDeviceComputeCapability
[h264_nvenc @ 0x1e24600] Loaded sym: cuCtxCreate_v2
[h264_nvenc @ 0x1e24600] Loaded sym: cuCtxPushCurrent_v2
[h264_nvenc @ 0x1e24600] Loaded sym: cuCtxPopCurrent_v2
[h264_nvenc @ 0x1e24600] Loaded sym: cuCtxDestroy_v2
[h264_nvenc @ 0x1e24600] Loaded sym: cuMemAlloc_v2
[h264_nvenc @ 0x1e24600] Loaded sym: cuMemFree_v2
[h264_nvenc @ 0x1e24600] Loaded sym: cuMemcpy2D_v2
[h264_nvenc @ 0x1e24600] Loaded sym: cuGetErrorName
[h264_nvenc @ 0x1e24600] Loaded sym: cuGetErrorString
[h264_nvenc @ 0x1e24600] Loaded lib: libnvidia-encode.so.1
[h264_nvenc @ 0x1e24600] Loaded sym: NvEncodeAPICreateInstance
[h264_nvenc @ 0x1e24600] Loaded sym: NvEncodeAPIGetMaxSupportedVersion
[h264_nvenc @ 0x1e24600] Loaded Nvenc version 8.1
[h264_nvenc @ 0x1e24600] Nvenc initialized successfully
[h264_nvenc @ 0x1e24600] 1 CUDA capable devices found
[h264_nvenc @ 0x1e24600] [ GPU #0 - < Tesla V100-SXM2-16GB > has Compute SM 7.0 ]
[h264_nvenc @ 0x1e24600] supports NVENC
Output #0, mp4, to ‘bbb.mp4’:
Metadata:
major_brand : qt
minor_version : 537199360
compatible_brands: qt
timecode : 00:00:00:00
com.apple.quicktime.player.movie.audio.gain: 1.000000
com.apple.quicktime.player.movie.audio.treble: 0.000000
com.apple.quicktime.player.movie.audio.bass: 0.000000
com.apple.quicktime.player.movie.audio.balance: 0.000000
com.apple.quicktime.player.movie.audio.pitchshift: 0.000000
com.apple.quicktime.player.movie.audio.mute:
com.apple.quicktime.player.movie.visual.brightness: 0.000000
com.apple.quicktime.player.movie.visual.color: 1.000000
com.apple.quicktime.player.movie.visual.tint: 0.000000
com.apple.quicktime.player.movie.visual.contrast: 1.000000
com.apple.quicktime.player.version: 7.4.1 (14)
com.apple.quicktime.version: 7.4.1 (14) 0x7418000 (Mac OS X, 10.5.2, 9C31)
encoder : Lavf58.3.100
Stream #0:0(eng), 0, 1/12288: Video: h264 (h264_nvenc) (Main), 1 reference frame (avc1 / 0x31637661), yuv420p(topleft), 1920x1080, 0/1, q=-1–1, 2000 kb/s, 0.04 fps, 12288 tbn, 24 tbc (default)
Metadata:
encoder : Lavc58.8.100 h264_nvenc
creation_time : 2008-05-27T18:40:35.000000Z
handler_name : Apple Alias Data Handler
Side data:
cpb: bitrate max/min/avg: 0/0/2000000 buffer size: 4000000 vbv_delay: -1
displaymatrix: rotation of -0.00 degrees
Stream #0:1(eng), 0, 1/48000: Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, 5.1, fltp, delay 1024, 341 kb/s (default)
Metadata:
encoder : Lavc58.8.100 aac
creation_time : 2008-05-27T18:40:35.000000Z
handler_name : Apple Alias Data Handler
Clipping frame in rate conversion by 0.000008
cur_dts is invalid (this is harmless if it occurs once at the start per stream)
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x1e1c1e0] stream 0, sample 10, dts 375000
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x1e1c1e0] stream 1, sample 0, dts 0
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x1e1c1e0] stream 2, sample 47, dts 1002667
[h264 @ 0x1e53f00] nal_unit_type: 6, nal_ref_idc: 0
[h264 @ 0x1e53f00] nal_unit_type: 1, nal_ref_idc: 0
Last message repeated 7 times
cur_dts is invalid (this is harmless if it occurs once at the start per stream)s speed=0.853x
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x1e1c1e0] stream 0, sample 11, dts 416667
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x1e1c1e0] stream 1, sample 0, dts 0
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x1e1c1e0] stream 2, sample 47, dts 1002667
[h264 @ 0x1e590c0] nal_unit_type: 6, nal_ref_idc: 0
[h264 @ 0x1e590c0] nal_unit_type: 1, nal_ref_idc: 1
Last message repeated 7 times
[h264_nvenc @ 0x1e24600] Failed locking bitstream buffer: invalid param (8)
Video encoding failed
[AVIOContext @ 0x1e23300] Statistics: 0 seeks, 1 writeouts

Can you please check what could be the issue? It can be reproduced in AWS p3.2x instance. I tried once earlier and once again today. Had same issues both time.

Best Regards,
Harinandan

Thank you. We are able to reproduce this issue on AWS and are looking into it. Will update this thread when more information is available.

We have root-caused the issue running FFmpeg on AWS P3 (Volta) which requires AWS to implement a fix which we have provided to them. While we don’t have a concrete date for the fix to be implemented, we expect AWS to roll out the update in next couple of weeks.