NVENC nvEncInitializeEncoder returns NV_ENC_ERR_OUT_OF_MEMORY (10) on RTX PRO 6000 Blackwell Server Edition in SPT CC; NVDEC works;

Accelerated Computing Confidential Computing
1

Environment

  • GPU: 1× NVIDIA RTX PRO 6000 Blackwell Server Edition (96 GB), passthrough to one CVM
  • GPU CC: SPT (CC State = ON). Not Hopper PPCIe. Not Workstation / Max-Q SKU.
  • CPU TEE: Intel TDX guest
  • OS: Ubuntu 24.04 (kernel: 6.8.0-90-generic)
  • nvidia-smi: 595.84
  • Driver Version: 595.84
  • CUDA Version (nvidia-smi): 13.2
  • Video Codec SDK: 13.0.37 (official AppDec / AppEncCuda)
  • FFmpeg: libavcodec 60.31.102 (Ubuntu package; h264_nvenc present)

What works

  • AppDec (NVDEC) H.264 → NV12 succeeds (example: 300 frames, coded 1920×1088). Session init ~2.1 s.
  • nvEncOpenEncodeSessionEx succeeds. The sample prints the GPU as RTX PRO 6000 Blackwell Server Edition.

What fails (first encode session; card mostly idle)

  1. Official sample AppEncCuda, after a successful OpenEncodeSessionEx:
    nvEncInitializeEncoder → NV_ENC_ERR_OUT_OF_MEMORY (10)
    Retry with -preset p1 -tuninginfo lowlatency -bf 0 -if nv12 -s 1920x1088: still error 10.
    Failure is before reading YUV / encoding any frame.

  2. Independent client (no SDK sample, no YUV file):
    ffmpeg -hide_banner -y -f lavfi -i testsrc=size=1920x1080:rate=30 -t 2
    -c:v h264_nvenc -preset p1 -tune ll /tmp/nvenc.mp4
    [h264_nvenc] InitializeEncoder failed: out of memory (10)
    Follow-on messages (“maybe incorrect parameters”, “Cannot allocate memory”,
    “Conversion failed!”) are cascading; the root is error 10 at InitializeEncoder.

root@tdx-guest:~/Video_Codec_SDK_13.0.37/Samples/build/AppEncode/AppEncCuda#ec SDK 13.0.37/Samples/build/AppDecode/AppDec/dec.yuv-Snfo lowlatency -bf 0-o /tmp/out.h264 -gpu 0GPU in use: NVIDIA RTX PRO 6000 Blackwell Server Edition[INF0 ][08:53:10] Encoding Parameters:
1920x1088 -if nv12
codec
h264
preset
p1
tuningInfo
lowlatency(default)
profilechroma
yuv420
bitdepth
8
/AppEncCuda
-i /root/Video_Cod
-codec h264
-preset p1 -tuningu
rc
fps
gop
bf
multipass
sizebttrate
maxbitratevbvbufsizevbvinit
cbr
30/1
INF
0
0
1920x1088
0
0
0
0
disableddisableddisabled
temporalag
lookahead
cq
0
P,B,I=0,0,0P,B,I=0,0,0P,B,I=0,0,0CreateEncoder : m nvenc.nvEncInitializeEncoder(m hEncoder, &m_ initializeParamsideo_ Codec SDK_ 13.0.37/Samples/NvCodec/NvEncoder/NvEncoder.cpp:330
qmin
qmaxinitqp
) returned error 10 at /root/V

Questions (product statements, please)

  1. Is NVENC supported on RTX PRO 6000 Blackwell Server Edition in SPT CC + Intel TDX
    with driver 595.84 / CUDA 13.2?
  2. If supported: is error 10 at nvEncInitializeEncoder a known bug? Fixed in which
    driver / VBIOS?
  3. If not supported: which official document states that? Should playback encode
    be done outside the TEE (CPU encode or non-CC GPU)?
  4. On this SKU, is a CC-off (plaintext) control expected to succeed?

Already ruled out

  • No NVENC silicon. That is H200: OpenEncodeSessionEx error 2 = UNSUPPORTED_DEVICE.
    Here OpenEncodeSessionEx succeeds; only InitializeEncoder fails.
  • AppEncCuda-only CUDA buffer registration (ffmpeg fails at the same API).
  • Input YUV / resolution (lavfi testsrc 1920×1080; also retried 1920×1088 NV12).
  • Typical GeForce session-cap or 1–2 GB vGPU OOM: this is the first session on 96 GB.