Notcing you are testing a custom code, can you use DeepStream sample to reprodcue this issue? are you using old nvstreammux or new nvstreammux? Please refer to this faq How to set parameters reasonably to improve the efficiency of nvstreammux in live mode.
Testing with deepstream-app, I am not able to reproduce this issue, after one rtsp source was closed, the fps of other source did not change. Here is the log deepstream-app.txt (5.4 KB)