DeepStream nvmultiurisrcbin remove_source timeout with RTSP sources after dynamic re-add

[pradeep_prabhu]

DeepStream 8.0 nvmultiurisrcbin remove_source timeout/hang with RTSP sources after dynamic re-add

• Hardware Platform (Jetson / GPU)
GPU / x86_64 + dGPU

GPU model: <fill exact GPU model from nvidia-smi>

• DeepStream Version
DeepStream 8.0

Container image: nvcr.io/nvidia/deepstream:8.0-samples-multiarch

• JetPack Version (valid for Jetson only)
Not applicable. This is x86_64 + dGPU deployment, not Jetson.

• TensorRT Version
TensorRT 10.9.0.34-1+cuda12.8

• NVIDIA GPU Driver Version (valid for GPU only)
Driver version: 595.97

• CUDA Version
CUDA 12.8.1

• OS / Container Environment
Host: Windows with Docker Desktop / WSL2 backend
Container OS: Ubuntu 24.04.1 LTS
GStreamer version: 1.20.3

• Issue Type
Question / possible bug

• Components involved
nvmultiurisrcbin, nvstreammux, RTSP dynamic source add/remove, native DeepStream REST API, nvinfer, metadata forwarding.

---

Problem Summary

I am using DeepStream 8.0 with nvmultiurisrcbin + nvstreammux for dynamic RTSP camera add/remove.

Native dynamic add_source works reliably. I can start one RTSP camera, add a second RTSP camera at runtime, and both streams produce FPS and metadata.

The issue is with native dynamic remove_source.

Sometimes remove_source works correctly:

• REST remove returns success.

• Removed camera disappears.

• Remaining camera continues.

• DeepStream Active sources changes from 2 to 1.

• pipeline_running=True.

But after repeated add/remove/re-add cycles, or under higher GPU/runtime load, native remove_source can hang or timeout. When this happens:

• remove_source does not complete.

• Backend times out after 45 seconds.

• A later get_runtime_state call also times out.

• Remaining stream may freeze or be affected.

• DeepStream/container runtime has to be restarted.

---

Runtime Flow

The application controls DeepStream through a Python JSON controller.

High-level flow:

Backend → JSON controller → native DeepStream REST API → nvmultiurisrcbin / nvstreammux

Runtime sequence:

1. Start DeepStream runtime with one RTSP camera.

2. Dynamically add a second RTSP camera using native REST add flow.

3. Both streams run and metadata/FPS are produced.

4. Remove one camera using native REST remove flow.

5. In some runs, remove works.

6. After repeated add/remove/re-add cycles, remove can hang.

Input sources:

• Two different physical RTSP cameras.

• Two different RTSP URLs.

• Two different application camera IDs.

---

Successful Native Remove Example

A successful remove looks like this from the backend/controller side:

remove_source sent for camera_id=matrix_cam_02
STREAM_REMOVE_SUCCESS
cleanup confirmed
still_present=False
remaining_missing=[]
runtime_started=True
pipeline_running=True

DeepStream side after successful remove:

Active sources : 1
matrix_cam_01[...] continues producing FPS

This confirms that native remove can work correctly in some cases.

---

Failed Native Remove Example

A failed remove looks like this:

native_remove_debug_start removed=['matrix_cam_02'] remaining=['matrix_cam_01']
json_cmd_send command=remove_source camera_ids=['matrix_cam_02']

native_remove_command_failed_no_fallback camera_id=matrix_cam_02
error=json_cmd controller command 'remove_source' timed out after 45s
state_after_error={'state_probe_error': "json_cmd controller command 'get_runtime_state' timed out after 10s"}

After this, the runtime/controller becomes unresponsive and the remaining camera can be affected.

---

Important Observations

1. Native add_source is stable.

2. Native remove_source can work once or several times.

3. Later, after repeated add/remove/re-add cycles, remove_source can hang.

4. The same camera can appear with a different metadata source_id after re-add. Example:

   • initially: camera_id=matrix_cam_01, source_id=0

   • after remove/re-add: camera_id=matrix_cam_01, source_id=2

5. The issue appears more likely under higher runtime load:

   • 1080p processing

   • higher GPU utilization

   • higher inference frequency

6. At lower load, such as 640x360 processing and lower inference frequency, native remove appears more stable.

7. I added backend-side guards to prevent overlapping add/remove/start/stop operations while native remove is running, but remove_source can still hang internally.

---

Configuration / Setup Details

Container image:

nvcr.io/nvidia/deepstream:8.0-samples-multiarch

Runtime:

Docker on Windows with WSL2 backend

DeepStream configuration:

use-nvmultiurisrcbin=1
RTSP inputs
nvmultiurisrcbin + nvstreammux
nvinfer
metadata forwarding
native REST add/remove API

Inference:

YOLOX model through nvinfer / TensorRT engine

Processing/load conditions where failure is more likely:

1080p processing
high GPU utilization
repeated RTSP add/remove/re-add

Processing/load conditions where remove is more stable:

640x360 processing
lower inference frequency
lower GPU utilization

---

Questions

1. For nvmultiurisrcbin REST remove, what is the correct identity to remove by:

   • sensorId

   • URI

   • metadata source_id

   • source index

   • application camera ID?

2. Is it expected that metadata source_id changes after removing and re-adding the same camera?

3. If source_id changes after re-add, how should an application map its stable camera ID to the correct native DeepStream source identity for later remove?

4. What is the exact expected REST payload for /api/v1/stream/remove for RTSP sources?

5. Are repeated RTSP add/remove/re-add cycles officially supported with nvmultiurisrcbin + nvstreammux in DeepStream 8.0?

6. Are there known issues where RTSP remove_source hangs during source-bin / decoder / nvstreammux cleanup?

7. How can I confirm that nvstreammux has released the removed source pad/bin?

8. What GST_DEBUG categories should I enable to debug nvmultiurisrcbin dynamic source removal?

9. Should the app wait for a specific DeepStream/GStreamer callback/event before re-adding the same camera after removal?

10. Is there a recommended sample/reference implementation for robust repeated RTSP add/remove using nvmultiurisrcbin?

11. Under high GPU load, can remove_source block because of in-flight decode/infer/mux buffers? If yes, is there a recommended drain/pause procedure before calling remove?

---

Expected Behavior

Removing one RTSP source should detach only that source and keep the remaining sources running without full pipeline restart.

Expected result:

Before remove:
Active sources = 2

After remove:
Active sources = 1
Removed camera gone
Remaining camera continues FPS/metadata
pipeline_running=True

---

Actual Behavior

Native remove_source sometimes succeeds, but after repeated dynamic operations or under higher load it can timeout/hang and make the runtime/controller unresponsive.

Observed failure:

remove_source timed out after 45s
get_runtime_state also timed out after 10s
remaining stream affected
runtime/controller needs restart

---

Requirement Details

This is not a new feature request. This is a question / possible bug about reliable dynamic RTSP source removal using nvmultiurisrcbin and nvstreammux.

The product requirement is to remove one camera without restarting the full DeepStream pipeline, while keeping remaining cameras running.he function description)

[fanzh]

I am checking. are you using legacy nvstreammux or new nvstreammux? what is your GPU model? could you share a simplified code if possbile and test steps? Thanks!

[pradeep_prabhu]

Thanks for checking.

I am using DeepStream 8.0 with the container:

nvcr.io/nvidia/deepstream:8.0-samples-multiarch

Hardware / driver:

Platform: x86_64 + dGPU
GPU model: NVIDIA GeForce RTX 2050 Laptop GPU, 4 GB dedicated VRAM
Driver version: 595.97
CUDA: 12.8.1
TensorRT: 10.9.0.34-1+cuda12.8
GStreamer: 1.20.3
Host environment: Windows + Docker Desktop / WSL2 backend

About nvstreammux:

I am using nvmultiurisrcbin with the native REST API. My generated DeepStream config enables:

use-nvmultiurisrcbin=1

and the pipeline path is effectively:

nvmultiurisrcbin → nvstreammux → nvinfer → metadata

I have not manually created/released nvstreammux pads in my application. The dynamic add/remove is handled through the nvmultiurisrcbin REST API.

I need to confirm from runtime logs/config whether this is using legacy nvstreammux or new nvstreammux. Could you please tell me the exact way to confirm this in DeepStream 8.0? I can share the config/log line after checking.

Current app path:

run_registry_rtsp_nvmultiurisrcbin.sh
→ run_tracksafe_native_deepstream_app.sh
→ deepstream-test5-tracksafe-native

The issue happens with two different physical RTSP cameras and two different RTSP URLs.

Simplified test steps:

1. Start DeepStream runtime with camera A.
2. Confirm camera A is running and metadata/FPS is produced.
3. Dynamically add camera B using /api/v1/stream/add.
4. Confirm DeepStream logs:
   Successfully added sensor id=[matrix_cam_02]
   new stream added [1:matrix_cam_02:matrix_cam_02]
   Active sources : 2
5. Manually remove camera B using /api/v1/stream/remove.
6. In the failed case, DeepStream receives:
   uri:/api/v1/stream/remove
   method:POST
7. After that, it does not print:
   STREAM_REMOVE_SUCCESS
   Successfully removed sensor
   new stream removed [...]
   Active sources : 1
8. The REST remove call hangs/times out, and state query also stops responding.

One important log seen before the failed remove:

GStreamer-WARNING ... Got data flow before segment event
... rtpsession ...
... rtpjitterbuffer ...

This makes me suspect the hang is around live RTSP/RTP/jitterbuffer/source cleanup inside nvmultiurisrcbin remove.

[fanzh]

Thanks for the details. If you run “export USE_NEW_NVSTREAMMUX=yes” or USE_NEW_NVSTREAMMUX is “yes”, the new streammux will be used, As shown in the doc.
could you share a complete running log including removing failure logs(GST_DEBUG=3)? Thanks!

Answers to your questions in the issue description.

1. as shown in the nvmultiurisrcbin doc, For REST remove, use the same camera_id and camera_url used when adding the stream. Do not use metadata source_id as the stable application identity.
2. Yes, source_id can change after remove/re-add. It is internally related to the nvstreammux sink pad/source index and should not be treated as a persistent camera ID.
3. Keep your own stable mapping by camera_id. For remove, send the stable camera_id plus exact RTSP URI. Internally nvmultiurisrcbin resolves that to the current source ID.
4. Expected remove payload:

{

"key": "sensor",

"value": {

"camera_id": "matrix_cam_02",

"camera_name": "matrix_cam_02",

"camera_url": "rtsp://...",

"change": "camera_remove"

},

"headers": {

"source": "vst",

"created_at": "2026-05-26T00:00:00.000Z"

}

}

5&6, yes, there is no known issues. currently the hang issue still can’t be reproduced.
7. To confirm nvstreammux released the pad, enable nvstreammux debug and check for:
gst_nvstreammux_release_pad:<src_bin_muxer> Pad deleted <pad_id>

After that, you should also see Successfully removed sensor / STREAM_REMOVE_SUCCESS.

8. Please collect logs with:
   GST_DEBUG=“nvmultiurisrcbin:5,nvstreammux:5,nvurisrcbin:4,rtspsrc:4,rtpjitterbuffer:4,rtpsession:4”
9. Before re-adding the same camera, wait until remove completes and you see stream-remove success / pad deleted. Do not re-add based only on your backend request being sent.
10. Please first compare against the nvmultiurisrcbin REST add/remove sample/test flow. If your app uses additional controller logic, try to reduce it to a gst-launch or small script reproducer.
11. Under high load, remove may take longer because RTSP/decode/mux buffers/events may still be in flight. However, in our stress test remove still completed in 0-1s. Please confirm whether the failed log contains Pad deleted. If not, the block is likely before/during mux pad release; if yes, it is likely after mux release in later cleanup/state synchronization.

[pradeep_prabhu]

NVIDIA Forum Reply: nvmultiurisrcbin RTSP Remove Hang

Thanks for the clarification. I confirmed that USE_NEW_NVSTREAMMUX=yes is active in this run.

The run command also printed this debug setting:

GST_DEBUG=nvmultiurisrcbin:5,nvstreammux:5,nvurisrcbin:4,rtspsrc:4,rtpjitterbuffer:4,rtpsession:4

I can rerun with plain GST_DEBUG=3 if you prefer that exact format, but the current attached logs include the requested remove failure window plus nvstreammux/nvmultiurisrcbin/RTSP details.

REST Remove Payload

For remove I am using the stable application camera_id and the same RTSP URL used for add. I am not using metadata source_id as the persistent identity.

Payload shape:

{
  "key": "sensor",
  "value": {
    "camera_id": "matrix_cam_02",
    "camera_name": "matrix_cam_02",
    "camera_url": "rtsp://<redacted>",
    "change": "camera_remove"
  },
  "headers": {
    "source": "vst",
    "created_at": "2026-05-26T00:00:00.000Z"
  }
}

Summary Of What I Observed

I tested the same application path, same two RTSP cameras, same stable camera_id + camera_url REST remove payload, and USE_NEW_NVSTREAMMUX=yes.

The behavior is nondeterministic:

• Sometimes REST remove completes and nvstreammux releases the pad.
• Sometimes the REST remove request is received, but the expected mux pad release and remove-success markers never appear before timeout.

This does not look like a backend state mismatch. In the failed case, the missing marker is before/during mux pad release.

Successful Remove Case

In the successful case, DeepStream reached the expected markers:

uri:/api/v1/stream/remove
gst_nvstreammux_release_pad:<src_bin_muxer> Pad deleted <pad_id>
Successfully removed sensor
STREAM_REMOVE_SUCCESS

Controller diagnosis for successful remove:

native_remove_diagnosis ... failure_bucket=successful_remove

Selected successful remove markers:

373479: uri:/api/v1/stream/remove
373554: 0:04:15.307154243 e[36m 2387e[00m 0x731a30006720 e[37mDEBUG  e[00m e[00m         nvstreammux gstnvstreammux.cpp:1332:gst_nvstreammux_release_pad:<src_bin_muxer>e[00m Pad deleted 0
373702: 0:04:15.433531664 e[36m 2387e[00m 0x731a30006720 e[37mDEBUG  e[00m e[00m    nvmultiurisrcbin gstdsnvmultiurisrcbin.cpp:1873:s_stream_api_impl:<src_nvmultiurisrcbin>e[00m Successfully removed sensor

Failed Remove Case

In the failed case, DeepStream received the REST request:

uri:/api/v1/stream/remove
method:POST

But after that remove request, I did not see:

gst_nvstreammux_release_pad:<src_bin_muxer> Pad deleted <pad_id>
Successfully removed sensor
STREAM_REMOVE_SUCCESS

The controller waited 30 seconds for cleanup confirmation and timed out:

Native DeepStream did not confirm source cleanup for camera_id=matrix_cam_02 within 30.0s;
last_log=uri:/api/v1/stream/remove

The controller diagnosis classified this as before/during mux pad release:

native_remove_diagnosis
rest_remove_returned=false
stream_remove_success_seen=false
mux_pad_deleted_seen=false
marker_stream_remove_api_seen=true
marker_successfully_removed_sensor_seen=false
marker_stream_remove_success_seen=false
marker_gst_nvstreammux_release_pad_seen=false
marker_pad_deleted_seen=false
failure_bucket=before_or_during_mux_pad_release

Selected failed remove block:

575616: 0:01:25.619894923 e[31m 2664e[00m 0x7d7218006720 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [remove_pad 464]App sequence error release pad called before push events
575617: 0:01:25.621099273 e[31m 2664e[00m 0x7d7218006720 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [remove_pad 464]App sequence error release pad called before push events
575618: 0:01:25.622433076 e[31m 2664e[00m 0x7d7218006720 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [remove_pad 464]App sequence error release pad called before push events
575619: 0:01:25.623695473 e[31m 2664e[00m 0x7d7218006720 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [remove_pad 464]App sequence error release pad called before push events
575620: 0:01:25.624912678 e[31m 2664e[00m 0x7d7218006720 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [remove_pad 464]App sequence error release pad called before push events
575621: 0:01:25.626198148 e[31m 2664e[00m 0x7d7218006720 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [remove_pad 464]App sequence error release pad called before push events
575622: 0:01:25.627450026 e[31m 2664e[00m 0x7d7218006720 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [remove_pad 464]App sequence error release pad called before push events
575623: 0:01:25.628844065 e[31m 2664e[00m 0x7d7218006720 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [remove_pad 464]App sequence error release pad called before push events
575624: 0:01:25.630150310 e[31m 2664e[00m 0x7d7218006720 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [remove_pad 464]App sequence error release pad called before push events
575625: 0:01:25.631402208 e[31m 2664e[00m 0x7d7218006720 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [remove_pad 464]App sequence error release pad called before push events
575626: 0:01:25.632668783 e[31m 2664e[00m 0x7d7218006720 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [remove_pad 464]App sequence error release pad called before push events
575627: 0:01:25.633911126 e[31m 2664e[00m 0x7d7218006720 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [remove_pad 464]App sequence error release pad called before push events
575628: 0:01:25.635170513 e[31m 2664e[00m 0x7d7218006720 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [remove_pad 473]100ms timeout exhausted. App sequence error release pad called before push events
575629: 0:01:25.635210007 e[31m 2664e[00m 0x7d7218006720 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [remove_pad 485]DEBUGME remove_pad 485 size=1
575630: 0:01:25.635323981 e[31m 2664e[00m 0x7d7218006720 e[37mDEBUG  e[00m e[00m    nvmultiurisrcbin gstdsnvmultiurisrcbin.cpp:1873:s_stream_api_impl:<src_nvmultiurisrcbin>e[00m Successfully removed sensor
575631: 0:01:25.635897569 e[31m 2664e[00m 0x7d726c003690 e[37mDEBUG  e[00m e[00m         nvstreammux gstnvstreammux.cpp:275:gst_nvstreammux_src_event:<src_bin_muxer>e[00m latency 0:00:00.000000000
575632: Active sources : 2
575633: Tue May 26 12:05:28 2026
575634: **PERF:  
575635: matrix_cam_01[testcam26] 14.00 (14.87)	matrix_cam_02[matrix_cam_02] 14.00 (16.03)	
575636: uri:/api/v1/stream/remove
575637: method:POST
575638: gstnvtracker: Successfully removed stream from low level tracker with Id 100000000.
575639: gstnvtracker: Successfully removed stream 1.
575640: new stream removed [1:matrix_cam_01]
575641: 0:01:25.761854080 e[31m 2664e[00m 0x7d726c0039d0 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [push_loop 363]stop_task=0 acc_batch=1 is_due=1
575642: 0:01:25.761903661 e[31m 2664e[00m 0x7d726c0039d0 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [push_loop 408]DEBUGME
575643: 0:01:25.761912148 e[31m 2664e[00m 0x7d726c0039d0 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [get_available 116]number of buffers available in queue of pad (0) = 4
575644: 0:01:25.761916235 e[31m 2664e[00m 0x7d726c0039d0 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [push_events 106][push_events] available buffers=4 q.size=4 event_indices.size=0
575645: 0:01:25.761920683 e[31m 2664e[00m 0x7d726c0039d0 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [push_events 85]sink=0 get_switched_to_idle()=0 queue.size()=4 get_eos()=0
575646: 0:01:25.761923975 e[31m 2664e[00m 0x7d726c0039d0 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [push_events 92]push_events num_sources_idle=0 pads_got_eos_and_empty_in_q=0(1)
575647: 0:01:25.761927148 e[31m 2664e[00m 0x7d726c0039d0 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [push_events 94]DEBUGME push_events 94 size=1 pads_got_eos_and_empty_in_q=0
575648: 0:01:25.761930406 e[31m 2664e[00m 0x7d726c0039d0 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [update_idle_sources 349]update_idle_sources 349 idle_sources=0
575649: 0:01:25.769607072 e[31m 2664e[00m 0x7d726c0039d0 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [push_loop 419]push_loop 419
575650: 0:01:25.769696148 e[31m 2664e[00m 0x7d726c0039d0 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [get_batch_size 320]get_batch_size inputs->size=1 num_sources_eos=-1 num_surfaces_per_frame=1 config=4 adaptive_batching=1, batch_size=2
575651: 0:01:25.769703473 e[31m 2664e[00m 0x7d726c0039d0 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [get_batch_size 320]get_batch_size inputs->size=1 num_sources_eos=-1 num_surfaces_per_frame=1 config=4 adaptive_batching=1, batch_size=2
575652: 0:01:25.769736961 e[31m 2664e[00m 0x7d726c0039d0 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [push_loop 313]push_loop 313 0x7d702800e550
575653: 0:01:25.769740358 e[31m 2664e[00m 0x7d726c0039d0 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [push_loop 325]DEBUGME what
575654: 0:01:25.769745952 e[31m 2664e[00m 0x7d726c0039d0 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [get_available 116]number of buffers available in queue of pad (0) = 4
575655: 0:01:25.769749609 e[31m 2664e[00m 0x7d726c0039d0 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [push_events 106][push_events] available buffers=4 q.size=4 event_indices.size=0
575656: 0:01:25.769753129 e[31m 2664e[00m 0x7d726c0039d0 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [push_events 85]sink=0 get_switched_to_idle()=0 queue.size()=4 get_eos()=0
575657: 0:01:25.769756220 e[31m 2664e[00m 0x7d726c0039d0 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [push_events 92]push_events num_sources_idle=0 pads_got_eos_and_empty_in_q=0(1)
575658: 0:01:25.769759307 e[31m 2664e[00m 0x7d726c0039d0 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [push_events 94]DEBUGME push_events 94 size=1 pads_got_eos_and_empty_in_q=0
575659: 0:01:25.769762351 e[31m 2664e[00m 0x7d726c0039d0 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [update_idle_sources 349]update_idle_sources 349 idle_sources=0
575660: 0:01:25.769767191 e[31m 2664e[00m 0x7d726c0039d0 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [get_batch_size 320]get_batch_size inputs->size=1 num_sources_eos=-1 num_surfaces_per_frame=1 config=4 adaptive_batching=1, batch_size=2
575661: 0:01:25.769772691 e[31m 2664e[00m 0x7d726c0039d0 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [form_batch 468]sources_to_process=2
575662: 0:01:25.769775375 e[31m 2664e[00m 0x7d726c0039d0 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [form_batch 485]update_with_source sid=0
575663: 0:01:25.769781411 e[31m 2664e[00m 0x7d726c0039d0 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [get_batch_size 320]get_batch_size inputs->size=1 num_sources_eos=-1 num_surfaces_per_frame=1 config=4 adaptive_batching=1, batch_size=2
575664: 0:01:25.769786435 e[31m 2664e[00m 0x7d726c0039d0 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [update_with_source 385]update_with_source sid=0
575665: 0:01:25.769791870 e[31m 2664e[00m 0x7d726c0039d0 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [get_available 116]number of buffers available in queue of pad (0) = 4
575666: 0:01:25.769795182 e[31m 2664e[00m 0x7d726c0039d0 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [update_with_source 390]num_avail=4 sid=0
575667: 0:01:25.769799821 e[31m 2664e[00m 0x7d726c0039d0 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [get_batch_size 320]get_batch_size inputs->size=1 num_sources_eos=-1 num_surfaces_per_frame=1 config=4 adaptive_batching=1, batch_size=2
575668: 0:01:25.769803756 e[31m 2664e[00m 0x7d726c0039d0 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [update_with_source 397]allowed_repeats=1 allowed_repeats_source=2 bs=2
575669: 0:01:25.769806817 e[31m 2664e[00m 0x7d726c0039d0 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [update_with_source 399]num_avail=1 sid=0
575670: 0:01:25.769810703 e[31m 2664e[00m 0x7d726c0039d0 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [update_with_source 405]synch_buffer=(nil) num_to_insert=1
575671: 0:01:25.769815176 e[31m 2664e[00m 0x7d726c0039d0 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [get_available 116]number of buffers available in queue of pad (0) = 4
575672: 0:01:25.769819030 e[31m 2664e[00m 0x7d726c0039d0 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [update_with_source 414]update_with_source insert 1 acc_batch 1 source_id 0 available 4 max repeats 1
575673: 0:01:25.769821789 e[31m 2664e[00m 0x7d726c0039d0 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [form_batch 485]update_with_source sid=1
575674: 0:01:25.769825787 e[31m 2664e[00m 0x7d726c0039d0 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [get_batch_size 320]get_batch_size inputs->size=1 num_sources_eos=-1 num_surfaces_per_frame=1 config=4 adaptive_batching=1, batch_size=2
575675: 0:01:25.769829353 e[31m 2664e[00m 0x7d726c0039d0 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [update_with_source 385]update_with_source sid=1
575676: 0:01:25.769832878 e[31m 2664e[00m 0x7d726c0039d0 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [form_batch 492]form_batch 492
575677: 0:01:25.769836190 e[31m 2664e[00m 0x7d726c0039d0 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [calculate_wait 176]we are running tight in muxer
575678: 0:01:25.769839280 e[31m 2664e[00m 0x7d726c0039d0 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [push_loop 346]wait_not_past_max=0 acc_batch=1
575679: 0:01:25.769841981 e[31m 2664e[00m 0x7d726c0039d0 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [push_loop 371]DEBUGME
575680: 0:01:25.769844893 e[31m 2664e[00m 0x7d726c0039d0 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [get_available 116]number of buffers available in queue of pad (0) = 4
575681: 0:01:25.769848541 e[31m 2664e[00m 0x7d726c0039d0 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [push_events 106][push_events] available buffers=4 q.size=4 event_indices.size=0
575682: 0:01:25.769851964 e[31m 2664e[00m 0x7d726c0039d0 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [push_events 85]sink=0 get_switched_to_idle()=0 queue.size()=4 get_eos()=0
575683: 0:01:25.769855244 e[31m 2664e[00m 0x7d726c0039d0 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [push_events 92]push_events num_sources_idle=0 pads_got_eos_and_empty_in_q=0(1)
575684: 0:01:25.769859377 e[31m 2664e[00m 0x7d726c0039d0 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [push_events 94]DEBUGME push_events 94 size=1 pads_got_eos_and_empty_in_q=0
575685: 0:01:25.769862475 e[31m 2664e[00m 0x7d726c0039d0 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [update_idle_sources 349]update_idle_sources 349 idle_sources=0
575686: 0:01:25.769865427 e[31m 2664e[00m 0x7d726c0039d0 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [copy_batch 147]DEBUGME
575687: 0:01:25.769867645 e[31m 2664e[00m 0x7d726c0039d0 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [copy_batch 150]DEBUGME
575688: 0:01:25.769872620 e[31m 2664e[00m 0x7d726c0039d0 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [copy_batch 163]DEBUGME; num_sources=1
575689: 0:01:25.769877794 e[31m 2664e[00m 0x7d726c0039d0 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [copy_batch 171]DEBUGME
575690: 0:01:25.769943512 e[31m 2664e[00m 0x7d726c0039d0 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [copy_batch 207]DEBUGME total_copied=1
575691: 0:01:25.769962977 e[31m 2664e[00m 0x7d726c0039d0 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [push_loop 385]DEBUGME
575692: 0:01:25.769987558 e[31m 2664e[00m 0x7d726c003350 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [add_buffer 514]id=0
575693: 0:01:25.770023901 e[31m 2664e[00m 0x7d726c0039d0 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [push_loop 408]DEBUGME
575694: 0:01:25.770028378 e[31m 2664e[00m 0x7d726c003350 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [add_buffer 555]add_buffer 555
575695: 0:01:25.770040382 e[31m 2664e[00m 0x7d726c0039d0 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [get_available 116]number of buffers available in queue of pad (0) = 4
575696: 0:01:25.770047204 e[31m 2664e[00m 0x7d726c0039d0 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [push_events 106][push_events] available buffers=4 q.size=4 event_indices.size=0
575697: 0:01:25.770052477 e[31m 2664e[00m 0x7d726c0039d0 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [push_events 85]sink=0 get_switched_to_idle()=0 queue.size()=4 get_eos()=0
575698: 0:01:25.770058428 e[31m 2664e[00m 0x7d726c0039d0 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [push_events 92]push_events num_sources_idle=0 pads_got_eos_and_empty_in_q=0(1)
575699: 0:01:25.770063142 e[31m 2664e[00m 0x7d726c0039d0 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [push_events 94]DEBUGME push_events 94 size=1 pads_got_eos_and_empty_in_q=0
575700: 0:01:25.770067154 e[31m 2664e[00m 0x7d726c0039d0 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [update_idle_sources 349]update_idle_sources 349 idle_sources=0
575701: 0:01:25.770071372 e[31m 2664e[00m 0x7d726c0039d0 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [push_loop 419]push_loop 419
575702: 0:01:25.770077799 e[31m 2664e[00m 0x7d726c0039d0 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [get_batch_size 320]get_batch_size inputs->size=1 num_sources_eos=-1 num_surfaces_per_frame=1 config=4 adaptive_batching=1, batch_size=2
575703: 0:01:25.770090819 e[31m 2664e[00m 0x7d726c0039d0 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [get_batch_size 320]get_batch_size inputs->size=1 num_sources_eos=-1 num_surfaces_per_frame=1 config=4 adaptive_batching=1, batch_size=2
575704: 0:01:25.770149694 e[31m 2664e[00m 0x7d726c0039d0 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [push_loop 313]push_loop 313 0x7d70280a0bf0
575705: 0:01:25.770192444 e[31m 2664e[00m 0x7d726c0039d0 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [push_loop 325]DEBUGME what
575706: 0:01:25.770199622 e[31m 2664e[00m 0x7d726c0039d0 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [get_available 116]number of buffers available in queue of pad (0) = 4
575707: 0:01:25.770206637 e[31m 2664e[00m 0x7d726c0039d0 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [push_events 106][push_events] available buffers=4 q.size=4 event_indices.size=0
575708: 0:01:25.770215811 e[31m 2664e[00m 0x7d726c0039d0 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [push_events 85]sink=0 get_switched_to_idle()=0 queue.size()=4 get_eos()=0
575709: 0:01:25.770155736 e[31m 2664e[00m 0x7d726c003350 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [get_pad_state 222] sinkPad->state=2
575710: 0:01:25.770274028 e[31m 2664e[00m 0x7d726c003350 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [add_buffer 505]id=0
575711: 0:01:25.770221846 e[31m 2664e[00m 0x7d726c0039d0 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [push_events 92]push_events num_sources_idle=0 pads_got_eos_and_empty_in_q=0(1)
575712: 0:01:25.770366385 e[31m 2664e[00m 0x7d726c0039d0 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [push_events 94]DEBUGME push_events 94 size=1 pads_got_eos_and_empty_in_q=0
575713: 0:01:25.770374117 e[31m 2664e[00m 0x7d726c0039d0 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [update_idle_sources 349]update_idle_sources 349 idle_sources=0
575714: 0:01:25.770404104 e[31m 2664e[00m 0x7d726c0039d0 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [get_batch_size 320]get_batch_size inputs->size=1 num_sources_eos=-1 num_surfaces_per_frame=1 config=4 adaptive_batching=1, batch_size=2
575715: 0:01:25.770411819 e[31m 2664e[00m 0x7d726c0039d0 e[37mDEBUG  e[00m e[00m         nvstreammux :0::<src_bin_muxer>e[00m [form_batch 468]sources_to_process=2
575716: 0:01:25.770417084 e[31m 2664e[0

Selected RTP/RTSP warnings after the failed remove request:

Diagnosis Markers

126: [deepstream-json-controller] event=native_rest_remove_response camera_id=matrix_cam_01 response={"reason": "STREAM_REMOVE_SUCCESS", "status": "HTTP/1.1 200 OK"}
132: [deepstream-json-controller] event=native_remove_diagnosis remove_camera_id=matrix_cam_01 rest_remove_returned=true stream_remove_success_seen=true mux_pad_deleted_seen=true marker_stream_remove_api_seen=true marker_successfully_removed_sensor_seen=true marker_stream_remove_success_seen=false marker_gst_nvstreammux_release_pad_seen=true marker_pad_deleted_seen=true failure_bucket=successful_remove
275: [deepstream-json-controller] event=native_rest_remove_response camera_id=matrix_cam_01 response={"reason": "STREAM_REMOVE_SUCCESS", "status": "HTTP/1.1 200 OK"}
281: [deepstream-json-controller] event=native_remove_diagnosis remove_camera_id=matrix_cam_01 rest_remove_returned=true stream_remove_success_seen=true mux_pad_deleted_seen=true marker_stream_remove_api_seen=true marker_successfully_removed_sensor_seen=true marker_stream_remove_success_seen=false marker_gst_nvstreammux_release_pad_seen=true marker_pad_deleted_seen=true failure_bucket=successful_remove
324: [deepstream-json-controller] event=native_rest_remove_response camera_id=matrix_cam_01 response={"reason": "STREAM_REMOVE_SUCCESS", "status": "HTTP/1.1 200 OK"}
330: [deepstream-json-controller] event=native_remove_diagnosis remove_camera_id=matrix_cam_01 rest_remove_returned=true stream_remove_success_seen=true mux_pad_deleted_seen=true marker_stream_remove_api_seen=true marker_successfully_removed_sensor_seen=true marker_stream_remove_success_seen=false marker_gst_nvstreammux_release_pad_seen=true marker_pad_deleted_seen=true failure_bucket=successful_remove
309: [DeepStreamRuntime] native_remove_debug_confirm camera_id=matrix_cam_01 still_present=False remaining_missing=[] controller_sources_after=['matrix_cam_02'] runtime_started=True pipeline_running=True
529: [DeepStreamRuntime] native_remove_debug_confirm camera_id=matrix_cam_01 still_present=False remaining_missing=[] controller_sources_after=['matrix_cam_02'] runtime_started=True pipeline_running=True
781: [DeepStreamRuntime] native_remove_debug_confirm camera_id=matrix_cam_01 still_present=False remaining_missing=[] controller_sources_after=['matrix_cam_02'] runtime_started=True pipeline_running=True
309: [DeepStreamRuntime] native_remove_debug_confirm camera_id=matrix_cam_01 still_present=False remaining_missing=[] controller_sources_after=['matrix_cam_02'] runtime_started=True pipeline_running=True
529: [DeepStreamRuntime] native_remove_debug_confirm camera_id=matrix_cam_01 still_present=False remaining_missing=[] controller_sources_after=['matrix_cam_02'] runtime_started=True pipeline_running=True
781: [DeepStreamRuntime] native_remove_debug_confirm camera_id=matrix_cam_01 still_present=False remaining_missing=[] controller_sources_after=['matrix_cam_02'] runtime_started=True pipeline_running=True
121: 	"reason" : "STREAM_REMOVE_SUCCESS",
124: [deepstream-json-controller] event=native_remove_phase_rest_call_done requested_camera_id=matrix_cam_01 remove_camera_id=matrix_cam_01 endpoint_url=http://127.0.0.1:9000/api/v1/stream/remove response={"reason": "STREAM_REMOVE_SUCCESS", "status": "HTTP/1.1 200 OK"} source_order_after_remove_response=['matrix_cam_01', 'matrix_cam_02']
125: [deepstream-json-controller] event=native_remove_rest_call_done requested_camera_id=matrix_cam_01 remove_camera_id=matrix_cam_01 status=HTTP/1.1 200 OK body={"reason": "STREAM_REMOVE_SUCCESS", "status": "HTTP/1.1 200 OK"}
126: [deepstream-json-controller] event=native_rest_remove_response camera_id=matrix_cam_01 response={"reason": "STREAM_REMOVE_SUCCESS", "status": "HTTP/1.1 200 OK"}
132: [deepstream-json-controller] event=native_remove_diagnosis remove_camera_id=matrix_cam_01 rest_remove_returned=true stream_remove_success_seen=true mux_pad_deleted_seen=true marker_stream_remove_api_seen=true marker_successfully_removed_sensor_seen=true marker_stream_remove_success_seen=false marker_gst_nvstreammux_release_pad_seen=true marker_pad_deleted_seen=true failure_bucket=successful_remove
270: 	"reason" : "STREAM_REMOVE_SUCCESS",
273: [deepstream-json-controller] event=native_remove_phase_rest_call_done requested_camera_id=matrix_cam_01 remove_camera_id=matrix_cam_01 endpoint_url=http://127.0.0.1:9000/api/v1/stream/remove response={"reason": "STREAM_REMOVE_SUCCESS", "status": "HTTP/1.1 200 OK"} source_order_after_remove_response=['matrix_cam_01', 'matrix_cam_02']
274: [deepstream-json-controller] event=native_remove_rest_call_done requested_camera_id=matrix_cam_01 remove_camera_id=matrix_cam_01 status=HTTP/1.1 200 OK body={"reason": "STREAM_REMOVE_SUCCESS", "status": "HTTP/1.1 200 OK"}
275: [deepstream-json-controller] event=native_rest_remove_response camera_id=matrix_cam_01 response={"reason": "STREAM_REMOVE_SUCCESS", "status": "HTTP/1.1 200 OK"}
281: [deepstream-json-controller] event=native_remove_diagnosis remove_camera_id=matrix_cam_01 rest_remove_returned=true stream_remove_success_seen=true mux_pad_deleted_seen=true marker_stream_remove_api_seen=true marker_successfully_removed_sensor_seen=true marker_stream_remove_success_seen=false marker_gst_nvstreammux_release_pad_seen=true marker_pad_deleted_seen=true failure_bucket=successful_remove
319: 	"reason" : "STREAM_REMOVE_SUCCESS",
322: [deepstream-json-controller] event=native_remove_phase_rest_call_done requested_camera_id=matrix_cam_01 remove_camera_id=matrix_cam_01 endpoint_url=http://127.0.0.1:9000/api/v1/stream/remove response={"reason": "STREAM_REMOVE_SUCCESS", "status": "HTTP/1.1 200 OK"} source_order_after_remove_response=['matrix_cam_01', 'matrix_cam_02']
323: [deepstream-json-controller] event=native_remove_rest_call_done requested_camera_id=matrix_cam_01 remove_camera_id=matrix_cam_01 status=HTTP/1.1 200 OK body={"reason": "STREAM_REMOVE_SUCCESS", "status": "HTTP/1.1 200 OK"}
324: [deepstream-json-controller] event=native_rest_remove_response camera_id=matrix_cam_01 response={"reason": "STREAM_REMOVE_SUCCESS", "status": "HTTP/1.1 200 OK"}
330: [deepstream-json-controller] event=native_remove_diagnosis remove_camera_id=matrix_cam_01 rest_remove_returned=true stream_remove_success_seen=true mux_pad_deleted_seen=true marker_stream_remove_api_seen=true marker_successfully_removed_sensor_seen=true marker_stream_remove_success_seen=false marker_gst_nvstreammux_release_pad_seen=true marker_pad_deleted_seen=true failure_bucket=successful_remove
3: [deepstream-native-runner] USE_NEW_NVSTREAMMUX=yes
50: [deepstream-native-runner] USE_NEW_NVSTREAMMUX=yes
51: [deepstream-native-runner] GST_DEBUG=nvmultiurisrcbin:5,nvstreammux:5,nvurisrcbin:4,rtspsrc:4,rtpjitterbuffer:4,rtpsession:4
373554: 0:04:15.307154243 e[36m 2387e[00m 0x731a30006720 e[37mDEBUG  e[00m e[00m         nvstreammux gstnvstreammux.cpp:1332:gst_nvstreammux_release_pad:<src_bin_muxer>e[00m Pad deleted 0
373702: 0:04:15.433531664 e[36m 2387e[00m 0x731a30006720 e[37mDEBUG  e[00m e[00m    nvmultiurisrcbin gstdsnvmultiurisrcbin.cpp:1873:s_stream_api_impl:<src_nvmultiurisrcbin>e[00m Successfully removed sensor
433505: [deepstream-native-runner] USE_NEW_NVSTREAMMUX=yes
433552: [deepstream-native-runner] USE_NEW_NVSTREAMMUX=yes
433553: [deepstream-native-runner] GST_DEBUG=nvmultiurisrcbin:5,nvstreammux:5,nvurisrcbin:4,rtspsrc:4,rtpjitterbuffer:4,rtpsession:4
575340: 0:01:25.511700098 e[31m 2664e[00m 0x7d7218006720 e[37mDEBUG  e[00m e[00m         nvstreammux gstnvstreammux.cpp:1332:gst_nvstreammux_release_pad:<src_bin_muxer>e[00m Pad deleted 1
575630: 0:01:25.635323981 e[31m 2664e[00m 0x7d7218006720 e[37mDEBUG  e[00m e[00m    nvmultiurisrcbin gstdsnvmultiurisrcbin.cpp:1873:s_stream_api_impl:<src_nvmultiurisrcbin>e[00m Successfully removed sensor
753299: 0:07:16.227293597 e[31m 2664e[00m 0x7d7214006720 e[37mDEBUG  e[00m e[00m         nvstreammux gstnvstreammux.cpp:1332:gst_nvstreammux_release_pad:<src_bin_muxer>e[00m Pad deleted 2
753461: 0:07:16.390595343 e[31m 2664e[00m 0x7d7214006720 e[37mDEBUG  e[00m e[00m    nvmultiurisrcbin gstdsnvmultiurisrcbin.cpp:1873:s_stream_api_impl:<src_nvmultiurisrcbin>e[00m Successfully removed sensor

Why I Think This Is Before/During Mux Pad Release

In the failed remove block:

• /api/v1/stream/remove is present.
• Pad deleted is not present after the failed remove request.
• Successfully removed sensor is not present after the failed remove request.
• STREAM_REMOVE_SUCCESS is not present after the failed remove request.
• RTP/RTSP warnings appear around the remove window.
• Backend/controller timeout happens after waiting for cleanup confirmation.

In the successful remove block, the same path later reaches Pad deleted and Successfully removed sensor.

So the same app path can complete correctly when nvstreammux releases the pad, but can also hang before/during that release.

Question

For live RTSP sources, is there a recommended application-side sequence before calling nvmultiurisrcbin REST remove to make nvstreammux pad release deterministic?

For example:

• wait/drain before remove?
• pause source before remove?
• send EOS/flush?
• change RTSP/jitterbuffer/source properties?
• avoid remove while mux has queued buffers for that pad?
• increase a specific DeepStream/nvmultiurisrcbin timeout?

Also, are the Got data flow before segment event warnings around rtpsession / rtpjitterbuffer relevant to the remove hang, or are they expected/noise during RTSP source teardown?

Attachments

I am attaching sanitized logs/files:

• nvidia_failed_remove_block_sanitized.txt
• nvidia_successful_remove_block_sanitized.txt
• nvidia_controller_diagnosis_sanitized.txt
• nvidia_backend_controller_runtime_markers_sanitized.txt

The RTSP credentials are redacted.

Created: 2026-05-26T18:32:05

[fanzh]

@pradeep_prabhu

100ms timeout exhausted. App sequence error release pad called before push events

Curently the hang issue still can’t be reproduced at my side. but from the log above, it is not expected and trigger a bug of new-streammux when removing pad. Since new-streammux is open-sourced in DS8.0, We will provide a fix.

1. cd /opt/nvidia/deepstream/deepstream/sources/libs/nvstreammux, replace nvstreammux.cpp with new nvstreammux.cpp (29.5 KB)。
2. run “export CUDA_VER=12.8 && make” to build new-streammux low-level lib libnvds_streammux.a, which is used by new-streammux plugin.
3. cd /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvmultistream2, run “make” to build libnvdsgst_multistream.so.
4. run

mv /opt/nvidia/deepstream/deepstream/lib/gst-plugins/libnvdsgst_multistream.so /opt/nvidia/deepstream/deepstream/lib/gst-plugins/libnvdsgst_multistream.so_ori   
cp  libnvdsgst_multistream.so /opt/nvidia/deepstream/deepstream/lib/gst-plugins

to make sure the fix takes effect, you can compare the md5 value of so, or add printing log in NvStreamMux::remove_pad.

[pradeep_prabhu]

Update after applying and testing the patched nvstreammux.cpp / libnvdsgst_multistream.so

I applied the NVIDIA-provided nvstreammux.cpp patch in a separate rollback-safe image and tested it with the same TrackSafe nvmultiurisrcbin native REST add/remove path.

Patched image/plugin proof:

Patched test image used during patch validation:
tracksafe-deepstream:nvstreammux-patch-test

Patched plugin md5 inside container:
da3182bd4f8922a951a7086e7284dd12  /opt/nvidia/deepstream/deepstream/lib/gst-plugins/libnvdsgst_multistream.so

Original plugin md5 before patch:
d88400f245148341d234ed470fe47d7d  /opt/nvidia/deepstream/deepstream/lib/gst-plugins/libnvdsgst_multistream.so_ori

Patched nvstreammux.cpp md5:
9764765f0754d078dc25e9cf7be84fe0

Result summary:

• The patched plugin was active during the patched test runs.
• Some remove cycles completed successfully after patch.
• However, the hard remove hang still reproduced intermittently.
• The failed patched run still got stuck before/during mux pad release.
• Because the hard hang still reproduces, I temporarily switched the product runtime back to the original DeepStream image for stability while waiting for further guidance.

Successful remove markers seen after patch:

deepstream_json_controller_control.log:166: "reason" : "STREAM_REMOVE_SUCCESS"
deepstream_json_controller_control.log:174: event=native_remove_confirmed_by_marker camera_id=matrix_cam_01 markers=['Successfully removed sensor', 'Pad deleted']
deepstream_json_controller_control.log:178: event=native_remove_diagnosis remove_camera_id=matrix_cam_01 rest_remove_returned=true stream_remove_success_seen=true mux_pad_deleted_seen=true marker_successfully_removed_sensor_seen=true marker_gst_nvstreammux_release_pad_seen=true marker_pad_deleted_seen=true failure_bucket=successful_remove

runtime log successful remove markers:
gst_nvstreammux_release_pad:<src_bin_muxer> Pad deleted 0
Successfully removed sensor

Failed patched remove block where pad deletion was not reached:

event=native_rest_remove_request camera_id=matrix_cam_02 camera_url=rtsp://***:***@192.168.0.37:554/unicaststream/1 endpoint=/api/v1/stream/remove payload={"headers":{"source":"vst"},"key":"sensor","value":{"camera_id":"matrix_cam_02","camera_name":"matrix_cam_02","camera_url":"rtsp://***:***@192.168.0.37:554/unicaststream/1","change":"camera_remove"}}

TimeoutError: timed out
TimeoutError: timed out
TimeoutError: timed out
TimeoutError: timed out
TimeoutError: timed out

event=native_rest_remove_cleanup_unconfirmed camera_id=matrix_cam_02 confirmation_error=Native DeepStream did not confirm source cleanup for camera_id=matrix_cam_02 within 30.0s

event=native_remove_failed_cleanup_unconfirmed requested_camera_id=matrix_cam_02 remove_camera_id=matrix_cam_02 source_order_after_cleanup_wait=['matrix_cam_01', 'matrix_cam_02'] error=Native DeepStream did not confirm source cleanup for camera_id=matrix_cam_02 within 30.0s

event=native_remove_diagnosis remove_camera_id=matrix_cam_02 rest_remove_returned=false stream_remove_success_seen=false mux_pad_deleted_seen=false marker_stream_remove_api_seen=false marker_successfully_removed_sensor_seen=false marker_stream_remove_success_seen=false marker_gst_nvstreammux_release_pad_seen=false marker_pad_deleted_seen=false failure_bucket=before_or_during_mux_pad_release

TimeoutError: timed out; confirmation_error=Native DeepStream did not confirm source cleanup for camera_id=matrix_cam_02 within 30.0s likely_bucket=native_remove_timeout_or_rtsp_cleanup_hang runner_running=True

What was not seen after the failed remove request:

gst_nvstreammux_release_pad:<src_bin_muxer> Pad deleted
Successfully removed sensor
STREAM_REMOVE_SUCCESS

Additional note:
During continued testing, Docker Desktop storage/VHDX was under disk pressure and Docker commands occasionally timed out. I am cleaning local Docker storage/logs before longer stress tests. I do not know whether this affects DeepStream runtime behavior directly, but it may affect container restart/recovery and log collection.

Questions:

1. Is the patched nvstreammux.cpp expected to fully eliminate this before_or_during_mux_pad_release hang, or could another nvmultiurisrcbin/new-streammux remove path still block before Pad deleted?
2. Are there additional debug prints or GST_DEBUG categories you want enabled in the patched run to identify the exact blocking point?
3. Should I test RTSP-specific settings such as TCP-only transport, adjusted latency/drop-on-latency, or a pre-remove drain/wait sequence, or avoid changing those until the mux remove issue is isolated?
4. During repeated add/remove validation on Windows + Docker Desktop + WSL2, Docker storage grows very quickly. In one short test run, C: free space dropped by about 2-3 GB, while the application log files were almost empty. The large growing item appears to be Docker Desktop’s WSL virtual disk on C:, not the application log directory.
   After a hard remove timeout, the DeepStream container can remain alive while the native app/REST path is unhealthy. Testing fallback restart/recreate flows then appears to grow Docker storage further.
   Do you recommend any DeepStream-specific Docker cleanup or storage-management practices for repeated dynamic RTSP add/remove validation? For example, should we regularly remove stopped containers, clear DeepStream/TensorRT/GStreamer caches, limit GST_DEBUG output, avoid rebuilding inside the runtime container, or move Docker’s WSL storage to a larger drive?

[fanzh]

1&2，
a. TX 2050 only has 4GB GPU memory. Which yolo model are you using? wondering if the the log in the fix is printed, could you share the running log with “GST_DEBUG=nvmultiurisrcbin:5,nvstreammux:5” log level? meanwhile please use “nvidia-smi dmon” to get the resource utilizaiton log. you can redirect the printing to local file, and please use ‘top’ to check if CPU utilization is high.
If the issue happened again, plese capture a stuck-process backtrace during the hang:

gdb -p <deepstream_pid>
thread apply all bt

This is probably the fastest way to identify the exact blocking call.
b. noticing you are using a custom code, can you reproduce the issue with the native sample /opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-server/? if yes, please share the running log, thanks!
3. what is the resolution of rtsp source? what do you mean by “1080p processing” and “640x360 processing” in the issue description?

[pradeep_prabhu]

Thanks. I can provide the setup details from the previous failed patched run, and I will collect the additional evidence you requested with a cleaner debug scope.

Current setup details from the failed patched remove test:

GPU: NVIDIA RTX 2050 Laptop GPU, 4 GB VRAM
DeepStream: 8.0 container
USE_NEW_NVSTREAMMUX=yes
Source type: live RTSP cameras through nvmultiurisrcbin REST add/remove
YOLO model/profile: YOLOX-M / medium profile
Primary config: /workspace/configs/custom/config_infer_primary_yolox_m.txt
TensorRT engine used in failed patched run: /workspace/artifacts/yolox_m.onnx_b4_gpu0_fp16.engine
Custom parser: /workspace/artifacts/libnvdsinfer_custom_impl_yolox.so
Runtime batch capacity: 4
Infer batch size: 4
Detector frame step: 4

Patch verification from the previous test:

Patched plugin md5:
da3182bd4f8922a951a7086e7284dd12

Original plugin md5:
d88400f245148341d234ed470fe47d7d

With the patched plugin, some remove cycles completed successfully with:

STREAM_REMOVE_SUCCESS
gst_nvstreammux_release_pad:<src_bin_muxer> Pad deleted <pad_id>
Successfully removed sensor

However, the hard remove hang still reproduced intermittently. In the failed patched case, REST remove was sent for matrix_cam_02, but the call timed out and I did not see:

gst_nvstreammux_release_pad:<src_bin_muxer> Pad deleted
Successfully removed sensor
STREAM_REMOVE_SUCCESS

The controller diagnosis remained:

failure_bucket=before_or_during_mux_pad_release

In one failed run, DeepStream metadata was still being produced before/around the remove timeout, for example:

[DSMETA][RAW] source_id=1 camera_id=matrix_cam_02 frame_num=88 objects=13 frame=1920x1080
[DSMETA][POST] camera_ids=['matrix_cam_02'] events=13 bbox_count=13

So the failed run does not look like “no inference running”; it still appears related to remove/cleanup.

Resolution clarification:

• The RTSP camera source is 1920x1080.
• When I said “1080p processing,” I meant the DeepStream/native pipeline receives the original RTSP camera frames at 1920x1080.
• When I said “640x360 processing,” I meant TrackSafe backend/UI preview, display, and bbox-normalization processing size. It is not the native RTSP camera resolution.
• The DeepStream metadata log also reported frame=1920x1080.

One note: I am currently doing a separate low-latency single-camera recovery test with:

/workspace/artifacts/yolox_m.onnx_b1_gpu0_fp16.engine

but that is not the engine used in the failed patched remove evidence above. The failed patched remove evidence used the b4 engine.

Also, my earlier storage-management question is still open. During repeated DeepStream Docker testing on Windows + Docker Desktop + WSL2, Docker’s WSL virtual disk grows quickly even when application logs are small. Could you confirm whether DeepStream creates any runtime caches/artifacts during repeated nvmultiurisrcbin add/remove tests, and whether there are recommended cleanup steps?

For example, should I regularly remove stopped containers, clear DeepStream/TensorRT/GStreamer caches, limit GST_DEBUG output, avoid rebuilding inside the runtime container, or move Docker’s WSL storage to a larger drive? Or should I treat this mainly as a Docker Desktop / WSL storage-management issue rather than a DeepStream-specific issue?

[fanzh]

regarding “Docker’s WSL virtual disk grows quickly”, the disk starts small and grows as Docker writes data into it. in your case, only nvinfer will save the generated engine to local file at the first run. Other plugins will not save data by default. It seems some logs are saved into the local file, such as nvidia_failed_remove_block_sanitized.
regarding “low-latency single-camera recovery”, do you mean two applications are startd? one is testing nvmultiurisrcbin with add/remove while the other is “single-camera recovery test”.
Importanly, to narrow down this isuse, please share the logs mentioned in my last comment.

[pradeep_prabhu]

Update after retesting with the patched image:

I recovered my application-side bbox/metadata path, so the bbox issue is now separate and fixed on my side. The remaining issue is still the dynamic RTSP remove path.

I tested again with the patched image. The behavior is still similar to the original image:

• Two streams can run with bbox/metadata working.
• matrix_cam_02 can be added successfully.
• The first remove completed successfully.
• After repeated add/remove operations, the crash/hang still reproduced around the second/third remove.
• During remove operations, including successful remove cases, GPU usage spikes heavily, roughly around 60–90%, and near the failed remove it reached close to 100%.

In the successful remove case I can see expected markers such as:

STREAM_REMOVE_SUCCESS
Successfully removed sensor
Pad deleted

But in the failed remove case, the remove request is sent and then the expected cleanup markers are missing:

Pad deleted
Successfully removed sensor
STREAM_REMOVE_SUCCESS

So the patched image did not fully resolve the repeated RTSP remove failure in my environment.

At this point I have two questions:

1. Is such a high GPU spike during nvmultiurisrcbin REST remove expected when using RTSP sources with nvstreammux, nvinfer, and tracker active?

2. If remove happens while decode/infer/mux buffers are still in flight, is there a recommended safe procedure before calling REST remove? For example:

   • pause/drop the source first
   • wait for a drain/EOS/idle marker
   • reduce inference load temporarily
   • wait for a specific mux/source-bin callback before remove
   • or use another recommended sequence

For product stability I am planning to keep native remove as the first path, but if remove times out or the native REST server becomes unresponsive, I will restart the DeepStream runtime/container and restore the remaining cameras. I still want to know whether there is a better recommended DeepStream-side remove sequence to avoid the failure instead of relying only on fallback restart.

[fanzh]

Thanks for the update!

1. after receiving “remove”, the server side will stop and remove the source and pad of nvstremmux. It should not cause a spike in GPU usage. Noticing some tracker log “575638: gstnvtracker:…”, Did you enable tracker plugin in your pipeline? which tracker cfg are you using?
2. nvmultiurisrcbin will perform the corresponding operation after receiving a remove message. Currently, There is no need to modify the nvmultiurisrcbin code.
3. regarding “Pad deleted” is missing after “remove request is sent”, coud you share the complete log? the nvidia_successful_remove_block_sanitized.txt above can’t be downoaded. What about CPU utilizaiton? To narrrow down this isuse, if removing nvinfer and tracker, will the hang issue persist? Thanks!
   Importantly, since nvmultiurisrcbin is open-sourced, you can add some logs to check which code the application is stuck in. please refer to gstdsnvmultiurisrcbin.zip (18.8 KB), where I have only added some printing. then rebuid and replace, then share the logs. You can only set “GST_DEBUG=nvmultiurisrcbin:5” to obtain the nvmultiurisrcbin log.

[pradeep_prabhu]

I reran the test with the NVIDIA debug gstdsnvmultiurisrcbin logging build.

Test setup:

Image: tracksafe-deepstream:nvmultiurisrcbin-debug
GST_DEBUG=nvmultiurisrcbin:5
Pipeline: RTSP -> nvmultiurisrcbin -> nvstreammux -> nvinfer -> NvDCF tracker
Tracker: libnvds_nvmultiobjecttracker.so
Tracker config: config_tracker_NvDCF_perf.yml

Result:

• Two streams ran with bbox/metadata working.
• matrix_cam_02 add succeeded.
• First two matrix_cam_02 remove cycles succeeded.
• On the next remove, the native DeepStream process stopped/died and REST remove never returned.
• Container stayed alive, but deepstream-test5-tracksafe-native was no longer running.
• Backend timed out after 30 seconds.

For the successful removes, the debug log showed:

in s_stream_api_impl /api/v1/stream/remove
before gst_nvmultiurisrcbincreator_remove_source
after gst_nvmultiurisrcbincreator_remove_source
new stream removed [...]
Successfully removed sensor
leaving process remove
gstnvtracker: Successfully removed stream ...

For the failed remove, the log reached:

uri:/api/v1/stream/remove
in s_stream_api_impl /api/v1/stream/remove
before gst_nvmultiurisrcbincreator_remove_source

After that, I did not see:

after gst_nvmultiurisrcbincreator_remove_source
Successfully removed sensor
new stream removed
leaving process remove
STREAM_REMOVE_SUCCESS

So this run appears to block/crash inside or immediately after:

gst_nvmultiurisrcbincreator_remove_source

Filtered log summary:

pad_deleted_seen: False
successfully_removed_sensor_seen: True
stream_remove_success_seen: False
timeout_error_seen: False
failure_bucket_seen: False

Note: pad_deleted_seen=False is expected for this specific log because this run used only:

GST_DEBUG=nvmultiurisrcbin:5

so it is focused on the nvmultiurisrcbin debug source, not nvstreammux pad logs.

Regarding GPU usage: during this run the pipeline had 2 RTSP streams with YOLO inference and NvDCF tracker. The second stream was running around 24 FPS, so GPU usage was high. However, the main failure evidence is not only GPU usage; it is that REST remove reached before gst_nvmultiurisrcbincreator_remove_source, then the native DeepStream process disappeared / REST remove never returned.

I am attaching the sanitized filtered log:

nvidia_nvmultiurisrcbin_filtered.log (10.5 KB)

[fanzh]

1. even though the GPU,CPU utilization is 99%, this issue still can’t be reproduced at my side.
   Let’s continue to check which code the application is stuck in. Since the related code is open-sourced. Please do the following test.
   1.1 cd /opt/nvidia/deepstream/deepstream-8.0/sources/libs/gstnvdscustomhelper, backup gst-nvmultiurisrcbincreator.cpp, then replace it with gst-nvmultiurisrcbincreator.cpp (58.2 KB). then run “export CUDA_VER=12.8 && make”, then run

mv /opt/nvidia/deepstream/deepstream/lib/libnvdsgst_customhelper.so /opt/nvidia/deepstream/deepstream/lib/libnvdsgst_customhelper.so_bk
cp libnvdsgst_customhelper.so /opt/nvidia/deepstream/deepstream/lib/

1.2 replace libnvdsgst_multistream.so with the new library built with the fix, wondering if the fix takes effect(the fix extents a timeout value).
1.3 run GST_DEBUG=nvmultiurisrcbin:5,nvstreammux:5 first to get nvmultiurisrcbin and nvstreammux’s logs.
1.4 run your server with “app >1.log 2>1.log”, after reproducing the hang issue , please the 1.log. Thanks!

2. to simplify your custom code and narrow down the issue, can you reroduce the issue with the following simple server? Or native sample deepstream-server mentioned above? Thanks!

  export USE_NEW_NVSTREAMMUX=yes && gst-launch-1.0 --gst-debug-level=3 --gst-debug=nvmultiurisrcbin:6   nvmultiurisrcbin name=src max-batch-size=16 port=9020 ip-address=0.0.0.0 live-source=1 width=1920 height=1080 drop-pipeline-eos=1 batched-push-timeout=33333 rtsp-reconnect-interval=0 uri-list=rtsp://127.0.0.1:8554/stream0 sensor-id-list=cam1 sensor-name-list=cam1 ! queue max-size-buffers=12 leaky=0 ! nvinfer config-file-path=/home/code/yolox/config_infer_yolox_m.txt ! nvtracker tracker-width=640 tracker-height=384 ll-lib-file=/opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so ll-config-file=/opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_tracker_NvDCF_perf.yml  ! fakesink sync=false async=false

[pradeep_prabhu]

I created a separate debug image for this test so the normal product image was not modified.

Applied:

• Replaced gst-nvmultiurisrcbincreator.cpp with the provided file.
• Rebuilt libnvdsgst_customhelper.so with CUDA_VER=12.8.
• Backed up the original libnvdsgst_customhelper.so.
• Installed the rebuilt libnvdsgst_customhelper.so.
• Ran with GST_DEBUG=nvmultiurisrcbin:5,nvstreammux:5.
• Captured stdout/stderr to 1.log using the requested style.

Not applied:

• Step 1.2 was not applied because I do not have the fixed replacement libnvdsgst_multistream.so.
• I found the existing libnvdsgst_multistream.so at:
  /opt/nvidia/deepstream/deepstream-8.0/lib/gst-plugins/libnvdsgst_multistream.so
  but I did not replace it because no fixed .so was provided.

Test result:

• The behavior looked improved compared with the previous run.
• Multiple alternating add/remove cycles with cam1 and cam2 completed successfully.
• The early removes were smooth.
• During the later remove, I did not observe a GPU spike; GPU/CPU looked normal.
• I am attaching the full 1.log captured from the patched debug image run.
• In this patched customhelper test run, 5 dynamic adds worked, 4 removes fully worked, and the 5th remove timed out. Including the initial startup source, 6 total source additions succeeded.
• Remove did not have any gpu spikes in runtime remove .
• I should not assume the issues on deepstream remove side please correct me if I’m doing anything wrong here all possible things I might have done wrong to lead this remove issue. Please suggest if anything to be understood by me.

Please confirm whether I should also replace libnvdsgst_multistream.so using a separate fixed library, and if so please provide the fixed libnvdsgst_multistream.so file.
nvidia_remove_fix_test_20260602_full_1log.zip (805.0 KB)
deepstream_runtime.log (34.7 KB)

[fanzh]

1. Thanks for the update! In deepstream_runtime.log, there is no new added log, such as “enter s_nvmultiurisrcbincreator_remove_source_impl”. could you recheck gst-nvmultiurisrcbincreator.cpp’s build and repalacement. you can compare the md5 value, and please copy .so to /opt/nvidia/deepstream/deepstream/lib, not /opt/nvidia/deepstream/deepstream/lib//opt/nvidia/deepstream/deepstream/lib/gst-plugins.
2. regarding “Not applied:”, I mean, I shared the modified nvstreammux.cpp on May 27, please build and repalce libnvdsgst_multistream.so by the method on that day.
3. then run the server after running “GST_DEBUG=nvmultiurisrcbin:5,nvstreammux:5”, then share the running log after issue recurs. the output log should include the following printings.

enter s_nvmultiurisrcbincreator_remove_source_impl   #add in nvmultiurisrcbincreator.cpp
Waiting for pad queue to drain   #add in nvstreammux.cpp

4. If the issue recurs, plese capture a stuck-process backtrace during the hang:

gdb -p <deepstream_pid>
thread apply all bt

This is probably the fastest way to identify the exact blocking call, Thanks!

[fanzh]

Additionally, As states in the guide, Currently validated Driver Version and GPU info: GameReady Driver version 572.60 with GeForce RTX-3050. driver 572.60 is recommended‌.

[yingliu]

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks.