NVDEC/cuvid VP8: segment quantizer state leaks across key-frame decodes → silent color corruption when decoding lossy WebP stills (L4, 565.77)

Gaming and Visualization Technologies Video Codec, PyNv & OFA
1

Problem :

A WebP whose VP8 key frame enables segmentation but doesn’t re-transmit all per-segment quantizers decodes differently depending on the previously decoded WebP — so an independent still image is not decoded independently. Result: visible color corruption (RMSE ≈ 0.155). Minimal standalone repro attached (links only libcuda+libnvcuvid; feeds raw VP8 chunks, no demuxer).

▎ victim md5 (fresh decoder) : 823d23329eecc38f4a6f83c7a9ac5027 (correct)
▎ victim md5 (same decoder, after A) : d0940b0df153749ce7278cd22a0391fa << changed by previous image

EXPECTED_corruption
EXPECTED_corruption820×296 84.1 KB

Suspected cause :

Segment feature data (per-segment quantizers) is not reset at the key-frame boundary, so un-transmitted segments inherit the previous decode’s values instead of the spec default (RFC 6386 §9.3). Proof: decoding a tiny segment-reset key frame (seg enabled, all deltas = 0) between the two images makes the victim byte-identical to the fresh decode.

Context:

We decode lossy WebP still images on GPU by extracting the VP8 key-frame bitstream (the VP8 chunk of the WebP) and feeding it to the NVDECODE API (cuvidCreateVideoParser(cudaVideoCodec_VP8) → cuvidParseVideoData → cuvidDecodePicture → cuvidMapVideoFrame). Each WebP is an independent single still image.

Reproduce :

Environment

  • GPU: NVIDIA L4
  • Driver: 565.77, CUDA (driver): 12.7, CUDA Toolkit: 12.3 (V12.3.107)
  • Video Codec SDK: nvcuvid.h / cuviddec.h (2024), libnvcuvid.so.565.77
  • OS: Rocky Linux 9.3, kernel 5.14.0; Compiler: GCC 11.5.0
  • All NVCUVID calls return CUDA_SUCCESS; output size is unchanged, only bytes differ (silent).

minimal standalone repro attached (cuvid_vp8dec_color_corruption.zip; links only libcuda+libnvcuvid, feeds raw VP8 chunks, no demuxer).

1. build (replace the path with your Video Codec SDK Interface/ dir that has nvcuvid.h, cuviddec.h)

NVCUVID_INC=/path/to/VideoCodecSDK/Interface ./build.sh # → ./vp8_repro

2. the bug: decode polluter, then the victim, on ONE decoder → victim comes out corrupted

./vp8_repro samples/polluter.vp8 samples/victim_eiffel.vp8

3. the fix: decode the segment-reset key frame (decode-only) in between → victim correct again

./vp8_repro samples/polluter.vp8 samples/victim_eiffel.vp8 samples/reset_segment_init.vp8

cuvid_vp8dec_color_corruption.zip (4.5 MB)