I have an application that runs NVDecoder on Ubuntu 22.04 based on the VideoProcessingFramework sample code.
It runs properly on the machine but fails when running inside Docker Container running on the same machine.
I get an error:
“NvDecoder : cuvidCreateVideoParser(&_parser, &videoParserParameters) returned error -97731136”.
I need help to understand why cuvidCreateVideoParser fails.
nvidia-docker --version
Docker version 20.10.23, build 7155243
Video Codec SDK 12.0.16
Docker & nvidia-smi output from within the Docker:
==========
== CUDA ==
==========
CUDA Version 11.7.1
Container image Copyright (c) 2016-2022, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:
https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license
A copy of this license is made available in this container at /NGC-DL-CONTAINER-LICENSE for your convenience.
mkdir: created directory '/root/.cache'
mkdir: created directory '/root/.cache/xdgr'
Tue Jun 13 12:07:23 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.105.17 Driver Version: 525.105.17 CUDA Version: 12.0 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA GeForce ... Off | 00000000:01:00.0 On | N/A |
| 0% 40C P8 18W / 250W | 926MiB / 11264MiB | 16% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
+-----------------------------------------------------------------------------+
Hello @rotem1 and welcome to the NVIDIA developer forums.
I took the liberty of enabling pretty-print for the console output above.
if you check that output you will see the initial CUDA version of the docker image, which is CUDA Version 11.7.1, while the nvidia-smicommand shows CUDA Version: 12.0.
You should make sure to have the same environments installed since the docker image will use the system installed GPU driver.
Indeed CUDA version is 11.7.
However this combination works fine when running on the host machine and fails when running inside the Docker container. Any reason for this?
==========
== CUDA ==
==========
CUDA Version 12.0.0
Container image Copyright (c) 2016-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:
https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license
A copy of this license is made available in this container at /NGC-DL-CONTAINER-LICENSE for your convenience.
*************************
** DEPRECATION NOTICE! **
*************************
THIS IMAGE IS DEPRECATED and is scheduled for DELETION.
https://gitlab.com/nvidia/container-images/cuda/blob/master/doc/support-policy.md
mkdir: created directory '/root/.cache'
mkdir: created directory '/root/.cache/xdgr'
Thu Jun 15 14:32:59 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.105.17 Driver Version: 525.105.17 CUDA Version: 12.0 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA GeForce ... Off | 00000000:01:00.0 On | N/A |
| 19% 47C P8 19W / 250W | 1019MiB / 11264MiB | 14% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
GPU in use: NVIDIA GeForce GTX 1080 Ti
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x55a947607c00] st: 0 edit list: 1 Missing key frame while searching for timestamp: 1000
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x55a947607c00] st: 0 edit list 1 Cannot find an index entry before timestamp: 1000.
[INFO ][14:33:01] Media format: QuickTime / MOV (mov,mp4,m4a,3gp,3g2,mj2)
terminate called after throwing an instance of 'NVDECException'
what(): NvDecoder : cuvidCreateVideoParser(&_parser, &videoParserParameters) returned error -1953180208 at /app/modules/videocodec/nvcodec/nvdecoder.cpp:646
Is this now the exact same error as before (same code line etc.)?
My suggestion would be to try and catch the Exception and find a more verbose Error output. Maybe it would be enough to run your app in Debug because NVDECException is supposed to give you the complete Error String as a result.
What docker container image are you actually using? You are not loading directly from NGC (nvcr.io), or are you?
I was able to solve it. The libcuvid libs had to be properly registered.
In my case I copy the cuvid libs from the build docker and had to copy them to the proper location and run ldconfig.