Please provide complete information as applicable to your setup.
-
Hardware Platform: dGPU, 3× NVIDIA A2 (one pipeline process per GPU)
-
DeepStream Version: 8.0 (container
nvcr.io/nvidia/deepstream:8.0-gc-triton-devel, x86_64) -
TensorRT Version: 10.9 (as shipped in the container)
-
NVIDIA GPU Driver Version: 595.71.05
-
Host: Ubuntu 24.04, kernel 6.8.0-124-generic, Docker
-
Issue Type: bug
fatal-rebuild-src3-20260830-154550-121-1.txt (96.6 KB)
fatal-freeze-20260906-053519-119-1.txt (86.8 KB)
fatal-freeze-20260906-053224-120-1.txt (86.8 KB)
fatal-freeze-20260903-145853-120-1.txt (86.8 KB)
Pipeline (Python app, pyds 1.2.2, legacy nvstreammux i.e. USE_NEW_NVSTREAMMUX=no):
4× (rtspsrc TCP → depay → parse → nvv4l2decoder) → input-selector per source → nvstreammux (batch-size 4, batched-push-timeout 40 ms) → nvinfer (pgie, YOLO) → nvtracker → 2× nvinfer (sgie) → probes → fakesink (sync=false, headless).
nvtracker config: stock libnvds_nvmultiobjecttracker.so with the IOU tracker (config_tracker_IOU.yml as shipped, no threading/sub-batch options set).
Symptom
After hours to days of normal operation, the entire pipeline freezes spontaneously: buffers stop flowing on all 4 sources within the same second, no more batches reach anything downstream of nvstreammux, GPU utilization / NVDEC / power drop to idle, and there is no bus error, no GStreamer warning, no dmesg/Xid — the process just stops flowing. Any subsequent set_state() on the source elements blocks forever (streaming threads hold their stream locks while stuck in downstream pushes), so the only recovery is killing the process. Frequency is load-correlated: 1–4 times/day, and only on our busiest pipeline (the A2 at ~80–90% utilization); the two lighter pipelines with the identical build have never frozen.
Native stacks at freeze time (the deadlock cycle)
We attached a watchdog that captures eu-stack -p (all OS threads, native frames) ~48 s into the freeze. Four independent captures (2026-08-30, 2026-09-03, and a back-to-back pair on 2026-09-06 where the second freeze hit ~2 minutes after a clean process start) show identical signatures. All tracker threads are parked in pthread_cond_wait — nobody is running, nobody wakes anybody:
# dispatch thread (gst task) — waits forever for the in-flight batch:
TID 310:
#1 pthread_cond_wait
#2 NvTrackerProc::waitForCompletion(InputParams&) (libnvds_nvmultiobjecttracker.so)
# batch processing thread — idle, waiting for work:
TID 309:
#1 pthread_cond_wait
#2 NvTrackerProc::processBatch()
# all 4 association workers — idle, waiting for work:
TID 368–371:
#1 pthread_cond_wait
#2 NvTrackerMultiThreadWorker<assocDataIn_t>::threadLoopFunc()
i.e. waitForCompletion believes a batch is still in flight, while processBatch and every worker sit idle — a lost wakeup / missed completion inside libnvds_nvmultiobjecttracker.so. The stuck batch’s buffers are never released, so the whole upstream chain starves on empty pools (these are the consequences, not causes):
nvinfer output thread: gst_buffer_pool_acquire_buffer → gst_pad_push (into nvtracker)
nvstreammux src thread: gst_buffer_pool_acquire_buffer (batch pool)
4× out-queue threads: gst_buffer_pool_acquire_buffer → gst_pad_push
4× decoder threads: CuvidV4L2_Poll_OPlane / gst_v4l2_allocator_dqbuf (output buffers never return)
4× jitterbuffer threads: blocked in gst_pad_push chains
Full dumps attached (faulthandler + py-spy + /proc thread inventory + eu-stack).
Questions
- Is this a known issue in DS 8.0’s nvmultiobjecttracker? It resembles the family of nvtracker sync/hang fixes mentioned for earlier releases (e.g. the DS 7.0 → 7.1 uninitialized-hang fix), but this one occurs mid-run under load with the plain IOU tracker.
- Is there a fix in a newer DeepStream release, or a patched
libnvds_nvmultiobjecttracker.sowe can drop in? - Any recommended workaround in the meantime — e.g. a config that forces single-threaded association (avoiding NvTrackerMultiThreadWorker), or sub-batching settings that change the completion-signaling path?
We can reproduce within ~a day on production load and can capture any additional diagnostics (GST_DEBUG categories, core dump of the frozen process, etc.) on request.