DeepStream 8.0 nvtracker (IOU) deadlocks mid-run: NvTrackerProc::waitForCompletion never wakes, whole pipeline freezes

Please provide complete information as applicable to your setup.

Pipeline (Python app, pyds 1.2.2, legacy nvstreammux i.e. USE_NEW_NVSTREAMMUX=no):

4× (rtspsrc TCP → depay → parse → nvv4l2decoder) → input-selector per source → nvstreammux (batch-size 4, batched-push-timeout 40 ms) → nvinfer (pgie, YOLO) → nvtracker → 2× nvinfer (sgie) → probes → fakesink (sync=false, headless).

nvtracker config: stock libnvds_nvmultiobjecttracker.so with the IOU tracker (config_tracker_IOU.yml as shipped, no threading/sub-batch options set).

Symptom

After hours to days of normal operation, the entire pipeline freezes spontaneously: buffers stop flowing on all 4 sources within the same second, no more batches reach anything downstream of nvstreammux, GPU utilization / NVDEC / power drop to idle, and there is no bus error, no GStreamer warning, no dmesg/Xid — the process just stops flowing. Any subsequent set_state() on the source elements blocks forever (streaming threads hold their stream locks while stuck in downstream pushes), so the only recovery is killing the process. Frequency is load-correlated: 1–4 times/day, and only on our busiest pipeline (the A2 at ~80–90% utilization); the two lighter pipelines with the identical build have never frozen.

Native stacks at freeze time (the deadlock cycle)

We attached a watchdog that captures eu-stack -p (all OS threads, native frames) ~48 s into the freeze. Four independent captures (2026-08-30, 2026-09-03, and a back-to-back pair on 2026-09-06 where the second freeze hit ~2 minutes after a clean process start) show identical signatures. All tracker threads are parked in pthread_cond_wait — nobody is running, nobody wakes anybody:

# dispatch thread (gst task) — waits forever for the in-flight batch:
TID 310:
#1  pthread_cond_wait
#2  NvTrackerProc::waitForCompletion(InputParams&)   (libnvds_nvmultiobjecttracker.so)

# batch processing thread — idle, waiting for work:
TID 309:
#1  pthread_cond_wait
#2  NvTrackerProc::processBatch()

# all 4 association workers — idle, waiting for work:
TID 368–371:
#1  pthread_cond_wait
#2  NvTrackerMultiThreadWorker<assocDataIn_t>::threadLoopFunc()

i.e. waitForCompletion believes a batch is still in flight, while processBatch and every worker sit idle — a lost wakeup / missed completion inside libnvds_nvmultiobjecttracker.so. The stuck batch’s buffers are never released, so the whole upstream chain starves on empty pools (these are the consequences, not causes):

nvinfer output thread:   gst_buffer_pool_acquire_buffer → gst_pad_push (into nvtracker)
nvstreammux src thread:  gst_buffer_pool_acquire_buffer (batch pool)
4× out-queue threads:    gst_buffer_pool_acquire_buffer → gst_pad_push
4× decoder threads:      CuvidV4L2_Poll_OPlane / gst_v4l2_allocator_dqbuf (output buffers never return)
4× jitterbuffer threads: blocked in gst_pad_push chains

Full dumps attached (faulthandler + py-spy + /proc thread inventory + eu-stack).

Questions

  1. Is this a known issue in DS 8.0’s nvmultiobjecttracker? It resembles the family of nvtracker sync/hang fixes mentioned for earlier releases (e.g. the DS 7.0 → 7.1 uninitialized-hang fix), but this one occurs mid-run under load with the plain IOU tracker.
  2. Is there a fix in a newer DeepStream release, or a patched libnvds_nvmultiobjecttracker.so we can drop in?
  3. Any recommended workaround in the meantime — e.g. a config that forces single-threaded association (avoiding NvTrackerMultiThreadWorker), or sub-batching settings that change the completion-signaling path?

We can reproduce within ~a day on production load and can capture any additional diagnostics (GST_DEBUG categories, core dump of the frozen process, etc.) on request.

Please upgrade to the latest DeepStream 9.1. And have a test if the issue happen on DeepStream 9.1.