Hi all,
I’ve configured the GPU operator in my K8s cluster (Overview — NVIDIA Cloud Native Technologies documentation)
K8s version v1.23.7, T400 GPU.
I have the following manifest:
apiVersion: v1
kind: Pod
metadata:
name: nvidia-ubuntu-gpu
spec:
restartPolicy: OnFailure
runtimeClassName: nvidia
containers:
- name: nvidia-ubuntu-gpu
image: "ubuntu:22.04"
command:
- "sleep"
- "604800"
resources:
limits:
nvidia.com/gpu: 1
nvidia-smi from within the container:
Sun Nov 27 13:28:15 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 520.61.05 Driver Version: 520.61.05 CUDA Version: 11.8 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA T400 4GB On | 00000000:13:00.0 Off | N/A |
| 38% 30C P8 N/A / 31W | 1MiB / 4096MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| No running processes found |
deviceQuery
from within the container:
root@nvidia-ubuntu-gpu:/tmp/cuda-samples-11.6/Samples/1_Utilities/deviceQuery# deviceQuery
deviceQuery Starting...
CUDA Device Query (Runtime API) version (CUDART static linking)
Detected 1 CUDA Capable device(s)
Device 0: "NVIDIA T400 4GB"
CUDA Driver Version / Runtime Version 11.8 / 11.8
CUDA Capability Major/Minor version number: 7.5
Total amount of global memory: 3912 MBytes (4102029312 bytes)
(006) Multiprocessors, (064) CUDA Cores/MP: 384 CUDA Cores
GPU Max Clock rate: 1425 MHz (1.42 GHz)
Memory Clock rate: 5001 Mhz
Memory Bus Width: 64-bit
L2 Cache Size: 524288 bytes
Maximum Texture Dimension Size (x,y,z) 1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384)
Maximum Layered 1D Texture Size, (num) layers 1D=(32768), 2048 layers
Maximum Layered 2D Texture Size, (num) layers 2D=(32768, 32768), 2048 layers
Total amount of constant memory: 65536 bytes
Total amount of shared memory per block: 49152 bytes
Total shared memory per multiprocessor: 65536 bytes
Total number of registers available per block: 65536
Warp size: 32
Maximum number of threads per multiprocessor: 1024
Maximum number of threads per block: 1024
Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
Max dimension size of a grid size (x,y,z): (2147483647, 65535, 65535)
Maximum memory pitch: 2147483647 bytes
Texture alignment: 512 bytes
Concurrent copy and kernel execution: Yes with 3 copy engine(s)
Run time limit on kernels: No
Integrated GPU sharing Host Memory: No
Support host page-locked memory mapping: Yes
Alignment requirement for Surfaces: Yes
Device has ECC support: Disabled
Device supports Unified Addressing (UVA): Yes
Device supports Managed Memory: Yes
Device supports Compute Preemption: Yes
Supports Cooperative Kernel Launch: Yes
Supports MultiDevice Co-op Kernel Launch: Yes
Device PCI Domain ID / Bus ID / location ID: 0 / 19 / 0
Compute Mode:
< Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >
deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 11.8, CUDA Runtime Version = 11.8, NumDevs = 1
Result = PASS
I have compiled ffmpeg as such:
fmpeg version 5.1.2 Copyright (c) 2000-2022 the FFmpeg developers
built with gcc 11 (Ubuntu 11.3.0-1ubuntu1~22.04)
configuration: --enable-nonfree --enable-gpl --enable-openssl --enable-libsvtav1 --enable-libx264 --enable-libx265 --enable-libvpx --enable-libxvid --enable-libvidstab --enable-libaom --enable-libzimg --enable-lv2 --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libmp3lame --enable-libopus --enable-libvorbis --enable-libtheora --enable-libfdk-aac --enable-libwebp --enable-libsrt --enable-cuda-nvcc --enable-cuvid --enable-nvenc --enable-cuda-llvm --enable-libnpp --nvccflags='-gencode arch=compute_52,code=sm_52' --enable-amf --disable-debug --disable-doc --disable-shared --enable-pthreads --enable-static --enable-small --enable-version3 --extra-cflags='-I/home/packerbuilt/test/ffmpeg-build/workspace/include -I/home/packerbuilt/test/ffmpeg-build/workspace/include/lilv-0 -I/usr/local/cuda/include' --extra-ldexeflags= --extra-ldflags='-L/home/packerbuilt/test/ffmpeg-build/workspace/lib -L/usr/local/cuda/lib64' --extra-libs='-ldl -lpthread -lm -lz' --pkgconfigdir=/home/packerbuilt/test/ffmpeg-build/workspace/lib/pkgconfig --pkg-config-flags=--static --prefix=/home/packerbuilt/test/ffmpeg-build/workspace --extra-version=
libavutil 57. 28.100 / 57. 28.100
libavcodec 59. 37.100 / 59. 37.100
libavformat 59. 27.100 / 59. 27.100
libavdevice 59. 7.100 / 59. 7.100
libavfilter 8. 44.100 / 8. 44.100
libswscale 6. 7.100 / 6. 7.100
libswresample 4. 7.100 / 4. 7.100
libpostproc 56. 6.100 / 56. 6.100
Hyper fast Audio and Video encoder
usage: ffmpeg [options] [[infile options] -i infile]... {[outfile options] outfile}...
However when I try and convert a file:
root@nvidia-ubuntu-gpu:/video# ffmpeg -y -vsync 0 -hwaccel cuda -hwaccel_output_format cuda -i 'Gravity - 2K Trailer.mp4' -c:a copy -c:v hevc_nvenc -b:v 5M 'Gravity - 2K Trailer-hevc.mp4'
ffmpeg version 5.1.2 Copyright (c) 2000-2022 the FFmpeg developers
built with gcc 11 (Ubuntu 11.3.0-1ubuntu1~22.04)
configuration: --enable-nonfree --enable-gpl --enable-openssl --enable-libsvtav1 --enable-libx264 --enable-libx265 --enable-libvpx --enable-libxvid --enable-libvidstab --enable-libaom --enable-libzimg --enable-lv2 --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libmp3lame --enable-libopus --enable-libvorbis --enable-libtheora --enable-libfdk-aac --enable-libwebp --enable-libsrt --enable-cuda-nvcc --enable-cuvid --enable-nvenc --enable-cuda-llvm --enable-libnpp --nvccflags='-gencode arch=compute_52,code=sm_52' --enable-amf --disable-debug --disable-doc --disable-shared --enable-pthreads --enable-static --enable-small --enable-version3 --extra-cflags='-I/home/packerbuilt/test/ffmpeg-build/workspace/include -I/home/packerbuilt/test/ffmpeg-build/workspace/include/lilv-0 -I/usr/local/cuda/include' --extra-ldexeflags= --extra-ldflags='-L/home/packerbuilt/test/ffmpeg-build/workspace/lib -L/usr/local/cuda/lib64' --extra-libs='-ldl -lpthread -lm -lz' --pkgconfigdir=/home/packerbuilt/test/ffmpeg-build/workspace/lib/pkgconfig --pkg-config-flags=--static --prefix=/home/packerbuilt/test/ffmpeg-build/workspace --extra-version=
libavutil 57. 28.100 / 57. 28.100
libavcodec 59. 37.100 / 59. 37.100
libavformat 59. 27.100 / 59. 27.100
libavdevice 59. 7.100 / 59. 7.100
libavfilter 8. 44.100 / 8. 44.100
libswscale 6. 7.100 / 6. 7.100
libswresample 4. 7.100 / 4. 7.100
libpostproc 56. 6.100 / 56. 6.100
-vsync is deprecated. Use -fps_mode
Passing a number to -vsync is deprecated, use a string argument as described in the manual.
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x55749e9aeec0] stream 0, timescale not set
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'Gravity - 2K Trailer.mp4':
Metadata:
major_brand : mp42
minor_version : 0
compatible_brands: mp42isomavc1
creation_time : 2013-12-01T03:59:56.000000Z
genre : Trailer
artist : Warner Bros.
title : Gravity - 2K Trailer
encoder : HandBrake 0.9.9 2013051800
date : 2013
Duration: 00:02:27.07, start: 0.000000, bitrate: 20296 kb/s
Stream #0:0[0x1](und): Video: h264 (avc1 / 0x31637661), yuv420p(tv, bt709, progressive), 2048x858, 20149 kb/s, 23.98 fps, 23.98 tbr, 90k tbn (default)
Metadata:
creation_time : 2013-12-01T03:59:56.000000Z
vendor_id : [0][0][0][0]
encoder : JVT/AVC Coding
Stream #0:1[0x2](eng): Audio: aac (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 153 kb/s (default)
Metadata:
creation_time : 2013-12-01T03:59:56.000000Z
vendor_id : [0][0][0][0]
Stream #0:2[0x0]: Video: mjpeg, yuvj420p(pc, bt470bg/unknown/unknown), 102x150 [SAR 72:72 DAR 17:25], 90k tbr, 90k tbn (attached pic)
Stream mapping:
Stream #0:0 -> #0:0 (h264 (native) -> hevc (hevc_nvenc))
Stream #0:1 -> #0:1 (copy)
Press [q] to stop, [?] for help
[h264 @ 0x55749f3f9700] decoder->cvdl->cuvidGetDecoderCaps(&caps) failed -> CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
[h264 @ 0x55749f3f9700] Failed setup for format cuda: hwaccel initialisation returned error.
[hevc_nvenc @ 0x55749e9c2c80] OpenEncodeSessionEx failed: unsupported device (2): (no details)
Error initializing output stream 0:0 -- Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height
Outputs from the previous commands seem to indicate a CUDA capable card exists but FFMPEG can’t use it. I have checked FFMPEG doesn’t detect CUDA-capable device [CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected] but I’ve not been able to resolve this.
Any pointers are appreciated.