My goal is to keep GPU utilization stable below ~90%, not only maximize average throughput.
I tested two approaches:
streammux batch-size=80, nvinfer batch-size=80, FP16 TensorRT engine with max batch 80
streammux batch-size=32, nvinfer batch-size=32, FP16 TensorRT engine with max batch 32, while still connecting 80 RTSP sources
With batch 80, average throughput is good, but I see sudden GPU SM spikes. With batch 32 and interval=2, the runtime is much more stable in my tests.
My question:
For 80 live RTSP sources, is it generally better to build/use a batch-80 engine and let nvinfer process one large batch, or use a smaller batch-32 engine and let DeepStream/nvinfer process the 80 sources in smaller chunks? Is there a possibility that it will somehow “fall behind” and just inference wont keep up with decoded frames?
What are the practical advantages/disadvantages of each approach in DeepStream?
Specifically, I want to understand:
Does nvinfer internally split larger nvstreammux batches into smaller inference chunks when nvinfer batch-size is smaller than the number of sources?
Is using streammux batch-size=32 with 80 live sources a recommended/valid approach?
Can smaller nvinfer batches reduce GPU utilization spikes even if average utilization is similar?
Are there latency or frame-dropping side effects when streammux batch-size is smaller than the number of live sources?
For live RTSP surveillance, should batch size be optimized for throughput, latency, or GPU utilization stability?
Any guidance or best practices for 80-camera DeepStream deployments would be appreciated.
Can you explain what the “average throughput” mean? Does “the runtime is much more stable” mean the GPU usage is smooth?
From the model view, it depends on the model itself. Whether the batch size 80 engine for one time is better than batch size 32 engine for 3 times depends on the model itself.
From the whole pipeline’s view, if your concern is the GPU usage, the model is not the only component who uses GPU, the tracker and sometimes postprocessing if it is implemented with CUDA will also use GPU too. They work in parallel, so it is not calculable.
Nvinfer internally splits larger nvstreammux batches into smaller inference chunks. You have set the nvstreammux batch size as the same to the nvinfer batch size in your test, so nvinfer handle the nvstreammux batch one time.
With your test case, the split is done by nvstreammux, nvstreammux will compose the batch size 32 batch from your 80 input streams, if your live streams are stable enough and the nvstreammux parameters are set properly, 3 batches may be generated for your 80 streams each time.
No.
No. It also depends on your model itself and the objects detected since you have tracker in your pipeline.
Yes.
What does the “throughput” mean? Which latency do you mean in this sentence? The network latency, inferencing latency or other ?
I actually tested it further and just connected 80 rtsp channels without nvinfer or nvtracker - just connecting to cam and taking snapsot every second and I realized that decoder percentage stay constantly at 100%. It results in artifacts - like all pixels are mixed inside the snapshot taken by bestshot and it is impossible to see anything there clearly.
How can I fix this?
Does this decoder belong to nvstreammux, is there any way that I view how it works inside deepstream?
Can I change its configurations?
No, video decode is independent to nvstreammux, I even don’t know whether software decoder or hardware decoder you are using. Can you post the complete pipeline and configurations?
The snapshot shows that there is packet loss in the RTSP stream transferring.
As you have mentioned the decoder consumption is 100%, decoding may be delayed, then the RTSP stack need to buffer more and more received packets for the packets may not be consumed by the decoder as soon as possible. The network stack buffer size is limited, then packet loss happen when the network stack buffer is overflow.
As A100 video decoder can support up to 100x1080p@30fps H264 streams or 178x1080p@30fps HEVC streams, your 80 RTSP streams may exceed such limitation.
Okay thank you for the information, if my streams provide 4k quality images, is there a way to reduce the resolution, asides from creating a substream of 1080p quality?
Sorry for my confusing questions, just it is my first time trying to set up such pipeline. I mean my streams arrive at 2560x1440 quality 30fps. It may be the reason why my decoder is always at 100%, what are my options to reduce it before it reaches the decoder, do I have to access the camera settings and set up a new substream with 1080p settings or there is another way?
In your case, DeepStream pipeline is just a RTSP client, the client can only support up to 100x1080p@30fps H264 streams or 178x1080p@30fps HEVC streams.
For the RTSP server side set up, you may consult the server vendors or providers.
