OK. Could you also check the CPU loading with htop?
Could you change the type of sink0 to Fakesink and if the fps changes with the number of sources increasing?
You can also open the latency log of the deepstream-app by referring to Enable Latency measurement for deepstream sample apps.