FPS reduced to 0.2 -0.6 FPS from 10 FPS while running 4 rtsp streams on a single pipeline on nano

• Hardware Platform (Jetson / GPU) Jetson Nano 4GB arm64
• DeepStream Version Deepstream-6.0_6.0.0-1
• JetPack Version (valid for Jetson only) Jetpack version-4.6-b199
• TensorRT Version 8.2.1.8-1+cuda10.2
• NVIDIA GPU Driver Version (valid for GPU only)
• Issue Type( questions, new requirements, bugs)
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

Running 4 rtsp streams on a single deepstream pipeline including tracker , the pipeline initially runs on max of 8-10 fps but after 30-45 mins it reduced to 0.2-0.6 and even 0 fps sometimes , and it looks like the system is getting choked because it slowed down every process. Why this thing is happening ? any comments or solution would be appreciated.
Thanks