Measured 2026-10-09 (tables in results.md inside the attached nvenc-sei-repro.zip); re-measured 2026-10-13 on driver 615.78.08.
Summary
With -udu_sei 1 (FFmpeg) the NVENC SDK writes a user-data-unregistered SEI message (payloadType 5) that is cut short, without error or warning, when the message needs many emulation-prevention bytes (EPB) relative to its size. The written NAL has a payloadSize that promises more bytes than are present, is missing the rbsp_trailing_bits byte 0x80, and has fewer EPB than the H.264/H.265 byte-stream rules require. Decoders reject or drop the message (FFmpeg: “Skipping invalid undecodable NALU: 39” for HEVC, “SEI type 5 size 116 truncated at 113” for H.264). The encoder call succeeds and the encoded picture is otherwise valid.
The effect is reproducible on h264_nvenc and hevc_nvenc, on IDR and P pictures, at every payload size tested (payloadSize 17 B to 60 kB).
Environment
| item | value |
|---|---|
| GPU | NVIDIA GeForce RTX 4090 (single GPU, PCI 0000:06:00.0) |
| Driver | 615.78.08 |
| OS / kernel | CachyOS Linux (rolling), kernel 7.2.9-2-cachyos |
| NVENC API | 13.1 (log: “Loaded Nvenc version 13.1”), nv-codec-headers 13.1.15.0.1 |
| FFmpeg | stock n9.0.2 (git describe n9.0.2, no local modifications), configure --disable-doc --disable-ffplay --disable-ffprobe |
| Measured | 2026-10-09; repeated 2026-10-13 from a fresh n9.0.2 build on the same driver: identical output bytes for the reproduction below, and the boundary cases in “Pinned trigger” gave the same pass/fail result in all 48 runs |
FFmpeg hands the SEI to NVENC in NV_ENC_SEI_PAYLOAD entries (payloadSize = 16-byte UUID + user data, payload = those bytes, unescaped; seiPayloadArray in NV_ENC_PIC_PARAMS_H264 / _HEVC). It does not escape or write the NAL itself; the NAL bytes below are what NVENC returned in the bitstream buffer.
Minimal reproduction
Needs: an ffmpeg with h264_nvenc / hevc_nvenc and the -udu_sei option (n9.0.2 used), Python 3, one NVIDIA GPU. Script: repro/nvenc_sei_repro.py (standard library only, in the attached zip).
C=h264 # or hevc
Z=100 # zero bytes of user data after the 16-byte UUID
ffmpeg -f lavfi -i color=gray:s=640x360:r=25 -frames:v 3 -c:v ${C}_nvenc -f $C base.bin
python3 nvenc_sei_repro.py inject $C base.bin in.bin $Z # put one correct SEI NAL in front of the first slice
ffmpeg -i in.bin -c:v ${C}_nvenc -udu_sei 1 -frames:v 1 -f $C out.bin # decoder exports the SEI as frame side data, NVENC re-writes it
python3 nvenc_sei_repro.py check $C out.bin $Z # exit 0 = complete, 1 = truncated
Z=40 gives “complete”. Z=100 gives “TRUNCATED” for both codecs. Equivalent programmatic reproduction: one frame with AV_FRAME_DATA_SEI_UNREGISTERED side data of 16 bytes UUID + 100 zero bytes (tools/seihx.c in the attached zip).
Expected versus actual (user data = UUID e1d7c4a2-6b93-4f08-9a55-0f3c2db17e64 + 100 zero bytes, payloadSize = 116)
| expected (spec) | NVENC output | |
|---|---|---|
| EPB in the NAL | 49 | 47 (both codecs) |
| user-data bytes present | 100 of 100 | 94 of 100 (NAL ends after the 47th 00 00 03) |
rbsp_trailing_bits 0x80 |
present | missing |
| H.264 NAL size (header + body) | 169 bytes | 160 bytes |
| HEVC NAL size (header + body) | 170 bytes | 161 bytes |
| decoder | accepts | FFmpeg hevc: “Skipping invalid undecodable NALU: 39”; h264: “SEI type 5 size 116 truncated at 113” |
H.264, SEI NAL bytes (start code omitted):
as written (160 bytes): 060574e1d7c4a26b934f089a550f3c2db17e64000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003
per spec (169 bytes): 060574e1d7c4a26b934f089a550f3c2db17e64000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000080
HEVC, SEI NAL bytes (start code omitted):
as written (161 bytes): 4e010574e1d7c4a26b934f089a550f3c2db17e64000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003
per spec (170 bytes): 4e010574e1d7c4a26b934f089a550f3c2db17e64000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000003000080
The written NAL is a strict prefix of the correct NAL.
Pinned trigger
Let P = payloadSize (UUID plus user data) and epb = the number of emulation-prevention bytes the spec requires for the RBSP 05 <size> <payload> 80 (a 0x03 after every 00 00 that is followed by 00, 01, 02 or 03). One SEI per picture:
- Complete when
epb <= ceil(P/3) + 3. - Truncated when
epb >= ceil(P/3) + 4.
Equivalently, the escaped message body (P + 3 + epb bytes) has to fit in floor(4 * (P + 5) / 3) bytes; the escaped size is sized with a 4/3 worst-case factor, while the worst case is 3/2 (a run of zero bytes needs one EPB per two bytes). With several SEI payloads on one picture, the limit for every message is floor(4 * (Pmax + 5) / 3) with Pmax the largest payloadSize of that picture.
Edge cases, run length alone does not decide (all measured, NAL bytes checked against the spec encoding):
| payload | result |
|---|---|
| P=116, 40 zero bytes after UUID (epb 19, limit 42) | complete |
| P=116, 100 zero bytes (epb 49, limit 42) | truncated |
| P=100, zero run with epb 37 (limit 37) / epb 38 | complete / truncated |
| P=300, epb 103 / 104 | complete / truncated |
| P=700, epb 237 / 238 | complete / truncated |
| P=2866 (H.264), epb 959 / 960 | complete / truncated |
00 00 01 repeated for the whole payload (epb = (P-16)/3, density 1/3) |
complete at every size tested, because 1/3 stays below the limit |
The boundary is identical for H.264 and HEVC, IDR and P pictures, resolutions 320x180 to 1920x1080, and does not depend on where the zero bytes sit, how many runs they form or what separates them (tested: run offsets 0 to payload end, 1 to 25 runs, separators 01, 02, 03, 04, 61, ff of 1, 2 and 4 bytes, pre-escaped 00 00 03 bytes in the payload). Only the total number of required EPB against P counts. About 77,000 payloads (random mixtures included) matched this rule with no exception.
Prior reports
- A 2021 FFmpeg patch review for the
udu_seioption ( [FFmpeg-devel] avcodec/nvenc: add udu_sei option to import user data unregistered SEIs - Patchwork ) noted that the NVENC SDK of that time (11.0) truncated user-data-unregistered SEI messages larger than 503 bytes. That was a size limit. With API 13.1,h264_nvencwrites low-EPB payloads in full up to the largest size we tested (payloadSize 58,948 bytes). The truncation described here depends on the EPB count relative to the payload size, not on size alone. - We found no thread about this on the NVIDIA Developer Forums.
Impact
The SEI is silently lost for any consumer that parses it: decoders drop the NAL (FFmpeg) or stop parsing the SEI (H.264). No error code is returned from nvEncEncodePicture or nvEncLockBitstream, so applications cannot detect it except by parsing the output. Zero-heavy payloads are common (padding, reserved fields, sparse maps, flat-content statistics). In our case, per-frame analysis blobs of 0.2 to 50 kB with long zero runs were lost whenever the content was flat.
Request
Size the SEI escape buffer for the true worst case (1.5 x the unescaped size) or escape into a buffer that cannot run out, and return NV_ENC_ERR_* instead of a partial NAL when the message does not fit.
Attachments
One file, nvenc-sei-repro.zip (the forum accepts no .py, .c or .md uploads): repro/nvenc_sei_repro.py, tools/seihx.c (programmatic repro), results.md (all tables), predicate.c (C form of the rule).