• Hardware Platform (Jetson / GPU) dGPU, NVIDIA GeForce RTX 5060 Ti 16 GB
• DeepStream Version 9.1.0 (nvcr.io/nvidia/deepstream:9.1-triton-multiarch, unmodified)
• JetPack Version (valid for Jetson only) N/A
• TensorRT Version 10.16.0.72 (not involved)
• NVIDIA GPU Driver Version (valid for GPU only) 610.57.04
• Issue Type( questions, new requirements, bugs) Bug, or a question about the intended usage
• How to reproduce the issue ? Below, with a self-contained script for the unmodified container
• Requirement details N/A
Summary
On dGPU, when the first buffer that nvv4l2decoder receives without the DELTA_UNIT flag cannot be decoded (for example, a slice damaged by packet loss on an RTSP/UDP link), the decoder:
- never outputs a frame again, even though every following access unit is intact and there is an IDR every 12 frames;
- never forwards EOS;
- blocks
set_state(GST_STATE_NULL)forever.
Only the first non-delta buffer matters: once the decoder has initialized, damaged buffers are handled fine.
We hit this in production with RTSP cameras. The affected source can no longer be removed, and with nvmultiurisrcbin that also hangs the REST server and freezes the other cameras (a separate topic about nvmultiurisrcbin, linked in a reply below).
Reproduction (unmodified container, about 1 minute)
Save repro_decoder_hang.py (at the end of this post) in the current directory, then:
docker run --rm -it --gpus all -e NVIDIA_DRIVER_CAPABILITIES=compute,utility,video \
-v "$PWD":/repro nvcr.io/nvidia/deepstream:9.1-triton-multiarch bash
apt-get update && apt-get install -y python3-gi
python3 /repro/repro_decoder_hang.py
The script (below) encodes a 3 s H.264 clip with NVENC (videotestsrc, GOP 12, IDR at frames 0, 12 and 24) and feeds it through the same chain nvurisrcbin builds for an RTSP H.264 source:
appsrc (avc + codec_data, as rtph264depay outputs) ! h264parse config-interval=-1 ! nvv4l2decoder ! fakesink
Each case runs in its own process. “Damaged” keeps the NAL header byte and overwrites the rest of that one slice NAL; every other access unit is intact.
clip: /tmp/tmpuqvir5ku/clip.mkv access units=36 keyframes at [0, 12, 24]
DeepStream 9.1.0 GStreamer 1.24.2
case A start at the IDR, intact (control)
pushed=36 decoded=36 expected=36 end=EOS set_state(NULL)=returned
case B join mid-GOP, intact (control)
pushed=35 decoded=24 expected=24 (from the next IDR) end=EOS set_state(NULL)=returned
case C join mid-GOP, first P slice damaged
pushed=35 decoded=0 expected=24 (from the next IDR) end=no EOS after 5 s set_state(NULL)=BLOCKED (> 5 s)
case D start at the IDR, IDR slice damaged, SPS/PPS only in codec_data
pushed=36 decoded=0 expected=24 (from the next IDR) end=no EOS after 5 s set_state(NULL)=BLOCKED (> 5 s)
- A and B are controls. Joining in the middle of a GOP (B), the decoder skips what it cannot use and starts at the next IDR, as expected.
- C and D differ from the controls in a single damaged slice, in the first non-delta buffer. Expected: the same as B (24 frames from the next IDR), or an error. Actual: 0 frames, no EOS, and
set_state(NULL)does not return.
How such a first buffer reaches the decoder with a real camera
- C: joining an RTSP stream in the middle of a GOP,
h264parse(config-interval=-1) inserts SPS/PPS before the first P frame and outputs that buffer withoutDELTA_UNIT. If that P slice is damaged, the decoder hangs. - D:
rtph264depaycarries SPS/PPS only incodec_data.h264parseinserts them before the IDR, and the buffer is non-delta even when the IDR slice is unreadable.
What we see inside the decoder
With gdb, the decoder’s streaming thread sits in usleep() inside libgstnvvideo4linux2.so (handle_frame, around offset +0x310fa), in a VIDIOC_DQEVENT polling loop that waits for the resolution-change event. For an undecodable first frame that event never comes, and the loop does not check for flushing or for the PAUSED→READY transition. set_state(NULL) then blocks in gst_pad_set_active(), waiting for the stream lock held by that thread.
Workaround we use
A pad probe on the decoder sink pad drops everything until the first non-delta buffer with SPS/PPS and an IDR whose slice header is valid. It costs up to one GOP when a source starts.
Questions
- Is this the expected behavior of
nvv4l2decoderon dGPU? - Is there a property or a recommended configuration that makes it drop the frame or post an error instead of waiting forever?
- Could the wait loop honor flushing and state changes, so the element can at least be stopped?
repro_decoder_hang.py
#!/usr/bin/env python3
"""nvv4l2decoder (dGPU) never recovers when its first keyframe-flagged buffer is undecodable.
Pipeline under test (same chain nvurisrcbin builds for an RTSP H.264 source):
appsrc (H.264 avc + codec_data, as rtph264depay outputs)
! h264parse config-interval=-1 ! nvv4l2decoder ! fakesink
Each case runs in its own process, because a hung case leaves stuck threads behind.
Cases:
A start at the IDR, intact (control)
B join in the middle of a GOP, intact (control)
C join in the middle of a GOP, first P slice damaged (bug)
D start at the IDR, IDR slice damaged, SPS/PPS only in codec_data (bug)
"Damaged" simulates what packet loss / bit errors do to a slice on an RTSP/UDP link: the NAL
header is kept and the slice payload after it is overwritten. Every following access unit is
intact, and there is a clean IDR every 12 frames, so a decoder is expected to recover at the
next IDR at the latest.
Usage:
python3 repro_decoder_hang.py [clip.mkv]
The clip is any short H.264 Matroska file with a GOP shorter than the clip. Without an
argument, one is generated with NVENC:
videotestsrc pattern=ball num-buffers=36 ! 1280x720 12 fps ! nvvideoconvert
! nvv4l2h264enc iframeinterval=12 idrinterval=12 insert-sps-pps=true profile=0
! h264parse ! matroskamux ! filesink
"""
import os
import struct
import subprocess
import sys
import tempfile
import threading
import gi
gi.require_version("Gst", "1.0")
from gi.repository import Gst # noqa: E402
FPS = 12
WAIT_EOS_S = 5
WAIT_NULL_S = 5
CASES = {
"A": "start at the IDR, intact (control)",
"B": "join mid-GOP, intact (control)",
"C": "join mid-GOP, first P slice damaged",
"D": "start at the IDR, IDR slice damaged, SPS/PPS only in codec_data",
}
def generate_clip(path):
pipeline = Gst.parse_launch(
"videotestsrc pattern=ball num-buffers=36 "
f"! video/x-raw,width=1280,height=720,framerate={FPS}/1 ! nvvideoconvert "
"! nvv4l2h264enc iframeinterval=12 idrinterval=12 insert-sps-pps=true profile=0 "
f"! h264parse ! matroskamux ! filesink location={path}")
pipeline.set_state(Gst.State.PLAYING)
msg = pipeline.get_bus().timed_pop_filtered(30 * Gst.SECOND, Gst.MessageType.EOS | Gst.MessageType.ERROR)
pipeline.set_state(Gst.State.NULL)
if msg is None or msg.type != Gst.MessageType.EOS:
sys.exit("could not generate the clip with NVENC; pass an H.264 .mkv as argument")
def load_access_units(path):
"""Access units in avc format (4-byte lengths), with their caps (codec_data)."""
pipeline = Gst.parse_launch(
f"filesrc location={path} ! matroskademux "
"! video/x-h264,stream-format=avc,alignment=au ! appsink name=sink sync=false")
sink = pipeline.get_by_name("sink")
pipeline.set_state(Gst.State.PLAYING)
units, caps = [], None
try:
while (sample := sink.emit("try-pull-sample", Gst.SECOND)) is not None:
caps = caps or sample.get_caps()
buf = sample.get_buffer()
ok, info = buf.map(Gst.MapFlags.READ)
units.append((bytes(info.data), not buf.has_flags(Gst.BufferFlags.DELTA_UNIT)))
buf.unmap(info)
finally:
pipeline.set_state(Gst.State.NULL)
return caps, units
def split_nals(unit):
nals, i = [], 0
while i + 4 <= len(unit):
size = struct.unpack(">I", unit[i:i + 4])[0]
nals.append(unit[i + 4:i + 4 + size])
i += 4 + size
return nals
def join_nals(nals):
return b"".join(struct.pack(">I", len(n)) + n for n in nals)
def nal_type(nal):
return nal[0] & 0x1F
def damage_slice(nal):
"""Keep the NAL header byte, overwrite the slice (header and data) with a fixed pattern."""
return nal[:1] + bytes((i * 37 + 11) & 0xFF for i in range(len(nal) - 1))
def damage_units(unit, types):
return join_nals([damage_slice(n) if nal_type(n) in types else n for n in split_nals(unit)])
def without_parameter_sets(unit):
return join_nals([n for n in split_nals(unit) if nal_type(n) not in (7, 8)])
def build_case(case, units):
keyframes = [i for i, (_, key) in enumerate(units) if key]
first_p = keyframes[0] + 1
data = [u for u, _ in units]
if case == "A":
return data
if case == "B":
return data[first_p:]
if case == "C":
return [damage_units(data[first_p], {1})] + data[first_p + 1:]
if case == "D":
stripped = [without_parameter_sets(u) for u in data]
return [damage_units(stripped[0], {5})] + stripped[1:]
raise ValueError(case)
def run_case(case, clip):
Gst.init(None)
caps, units = load_access_units(clip)
queue = build_case(case, units)
pipeline = Gst.parse_launch(
f'appsrc name=src format=time caps="{caps.to_string()}" '
"! h264parse config-interval=-1 ! nvv4l2decoder name=dec "
"! fakesink name=sink sync=false signal-handoffs=true")
decoded = []
pipeline.get_by_name("sink").connect("handoff", lambda *_: decoded.append(1))
src = pipeline.get_by_name("src")
pipeline.set_state(Gst.State.PLAYING)
for i, unit in enumerate(queue):
buf = Gst.Buffer.new_wrapped(unit)
buf.pts = i * Gst.SECOND // FPS
buf.duration = Gst.SECOND // FPS
src.emit("push-buffer", buf)
src.emit("end-of-stream")
msg = pipeline.get_bus().timed_pop_filtered(WAIT_EOS_S * Gst.SECOND, Gst.MessageType.EOS | Gst.MessageType.ERROR)
if msg is None:
end = f"no EOS after {WAIT_EOS_S} s"
elif msg.type == Gst.MessageType.EOS:
end = "EOS"
else:
end = "ERROR: " + msg.parse_error()[0].message
stopped = threading.Event()
threading.Thread(target=lambda: (pipeline.set_state(Gst.State.NULL), stopped.set()), daemon=True).start()
null = "returned" if stopped.wait(WAIT_NULL_S) else f"BLOCKED (> {WAIT_NULL_S} s)"
keyframes = [i for i, (_, key) in enumerate(units) if key]
expected = len(units) if case == "A" else f"{len(units) - keyframes[1]} (from the next IDR)"
print(f"case {case} {CASES[case]}\n"
f" pushed={len(queue)} decoded={len(decoded)} expected={expected} end={end} set_state(NULL)={null}",
flush=True)
os._exit(0)
def main():
if len(sys.argv) > 2 and sys.argv[1] == "--case":
run_case(sys.argv[2], sys.argv[3])
Gst.init(None)
clip = sys.argv[1] if len(sys.argv) > 1 and sys.argv[1] else None
if clip is None:
clip = os.path.join(tempfile.mkdtemp(), "clip.mkv")
generate_clip(clip)
_, units = load_access_units(clip)
print(f"clip: {clip} access units={len(units)} keyframes at {[i for i, (_, k) in enumerate(units) if k]}")
print(f"DeepStream {open('/opt/nvidia/deepstream/deepstream/version').readline().split()[-1]} "
f"{Gst.version_string()}\n", flush=True)
for case in CASES:
try:
subprocess.run([sys.executable, __file__, "--case", case, clip], timeout=60, check=False)
except subprocess.TimeoutExpired:
print(f"case {case} process did not exit in 60 s (killed)", flush=True)
if __name__ == "__main__":
main()