Running multiple parallel pipeline with nvinfer vs batched inference

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Hello.

When I try to run many indepent gst pipelines in parallel using nvinfer and running inference on the same model file, the more cameras I add the bigger the inference time is.

For example: with one camera inference time is around 10ms, with two cameras it goes up to 18ms, with 3 cameras it goes up to 26ms. It roughly adds 8 ms of inference time for each new independent pipeline running in parallel.

I thought I could fix this issue by using nvstreammux and batching all frames of the cameras and running inference on all frames at the same time. Instead of having multiple independent gst pipelines, I have a single gst pipeline with many source branches that feed to a single nvstreammux element, which in turn feeds the nvinfer element.

Example: the cameras are all triggered at the same time. I get N frames and I set the batch-sizeproperty of both nvinfer and nvstreammux to N. So what happens is that the mux element will batch all N frames and then run inference on the batch of N frames at the same time. What I noticed is that the bigger the batch gets, the inference time also goes up, so it looks like the infernece is not running in parallel for each frame.

How can I obtain true parallel behaviour, meaning that the inference time will not go up as more cameras are added? If that’s possible, I believe that in this case I would be limited only by how many instances of my model can fit on the GPU at the same time, since they would all run in parallel.

Thanks!

Jose

4

nvinfer-parallel_two_models_diff_ds - Copy.zip (1.8 MB)

I’ve attached a .nsys-rep in this message. This was captured with two models running in parallel independent pipelines.

I am seeing some gaps in execution:

image
image1640×383 50.3 KB

Specially with thse sm50 and sm80 kernels. It looks like they cannot run in parallel. Does this means the GPU is starving and that’s why running more than one model in parallel increases inference time?

8

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU)
• DeepStream Version
• JetPack Version (valid for Jetson only)
• TensorRT Version
• NVIDIA GPU Driver Version (valid for GPU only)
• Issue Type( questions, new requirements, bugs)
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

9

Hardware Platform: Orin

===== L4T Version =====

R36 (release), REVISION: 4.7, GCID: 42132812

===== DeepStream Version =====
deepstream-app version 7.1.0
DeepStreamSDK 7.1.0
CUDA Driver Version: 12.6
CUDA Runtime Version: 12.6
TensorRT Version: 10.3
cuDNN Version not available
libNVWarp360 Version: 2.0.1d3

11

Could you provide the complete media pipeline and the nvinfer and nvstreammux configuration files? To verify that the TensorRT engine is loaded and configured correctly, please also provide the complete running logs. Thanks.

How are you measuring inference time? Does it include nvinfer preprocessing, queueing, synchronization, and postprocessing, or only TensorRT GPU execution? Nsight Systems or CUDA events around TensorRT enqueue would help distinguish these components.

What is the model? Performance also depends on the model’s GPU resource usage. Please use nvidia-smi dmon or tegrastats to monitor GPU and memory utilization while testing one, two, and three sources. If one inference already consumes most SM or memory-bandwidth resources, additional pipelines will contend and their latency will increase.

Similarly, a larger batch does not imply constant latency. TensorRT launches more thread blocks or larger matrix operations for [N,C,H,W]. CUDA schedules this work across a finite number of SMs; blocks run concurrently only while cores, registers, shared memory, and bandwidth remain available. Remaining blocks execute in later waves. Thus, batch latency can increase even though batching improves parallelism and overall throughput.

12

The screenshot does not prove that the two models cannot run in parallel. The sm50 and sm80 kernels shown on the same row belong to one CUDA stream; kernels within that stream execute sequentially because of CUDA stream ordering and TensorRT layer dependencies. Different model streams can still overlap, and this trace shows partial overlap.

This does not indicate GPU starvation. Starvation means the GPU is idle because the application has not supplied work. If a kernel from one model must wait because another kernel consumes the available SM, Tensor Core, register, memory, or bandwidth resources, that is GPU contention.

Multiple models can therefore increase individual inference latency because they share finite GPU resources. Their kernels may overlap partially, interleave, or wait, even though aggregate throughput improves.

The current report lacks GPU hardware metrics, so it cannot determine whether the GPU was saturated or idle during these intervals. Please recapture with --gpu-metrics-devices=0 and correlate SM Active, SM Issue, Tensor Active, and memory bandwidth with the same kernel interval:

sudo nsys profile   -t cuda,cudnn,osrt,nvtx   --gpu-metrics-devices=0   -o report  <application>
13

Thanks for the reply. I am talking about running multiple parallel pipelines like this one. Some pre-processing elements were ommited.

v4l2src device=/dev/videoX ! video/x-bayer,format=grbg10le,width=1920,height=1200 ! video/x-raw,format=RGBA ! nvvideoconvert ! video/x-raw(memory:NVMM),width=1920,height=1200 ! queue ! mux.sink_0 nvstreammux name=mux width=1920 height=1200 batched-push-timeout=0 batch-size=1 ! nvinfer output-tensor-meta=1 config-file-path=<path> model-engine-file=<path> unique-id=1 ! nvvideoconvert copy-hw=2 ! video/x-raw,format=RGBA ! appsink

Each pipeline is being fed from a different /dev/videoX device, but they are all using the same model file.

Versus doing something like this:

nvstreammux name=mux width=1920 height=1200 batch-size=4 live-source=true sync-inputs=false batched-push-timeout=0  ! nvinfer output-tensor-meta=1 config-file-path=<path> model-engine-file=<path> unique-id=1 batch-size=4  ! nvvideoconvert copy-hw=2  ! video/x-raw,format=RGBA  ! appsink name=sink emit-signals=true sync=false max-buffers=1 drop=true
v4l2src device=/dev/videoX ! video/x-bayer,format=grbg10le,width=1920,height=1200  ! bayer2rgb  ! video/x-raw,format=RGBA  ! nvvideoconvert copy-hw=2  ! video/x-raw(memory:NVMM),format=RGBA,width=1920,height=1200 ! queue ! mux.sink_0
v4l2src device=/dev/videoY ! video/x-bayer,format=grbg10le,width=1920,height=1200  ! bayer2rgb  ! video/x-raw,format=RGBA  ! nvvideoconvert copy-hw=2  ! video/x-raw(memory:NVMM),format=RGBA,width=1920,height=1200 ! queue ! mux.sink_1
v4l2src device=/dev/videoZ ! video/x-bayer,format=grbg10le,width=1920,height=1200  ! bayer2rgb  ! video/x-raw,format=RGBA  ! nvvideoconvert copy-hw=2  ! video/x-raw(memory:NVMM),format=RGBA,width=1920,height=1200 ! queue ! mux.sink_2
v4l2src device=/dev/videoW ! video/x-bayer,format=grbg10le,width=1920,height=1200  ! bayer2rgb  ! video/x-raw,format=RGBA  ! nvvideoconvert copy-hw=2  ! video/x-raw(memory:NVMM),format=RGBA,width=1920,height=1200 ! queue ! mux.sink_3

There’s one pipeline branch per camera, and they all sink into the same nvstreammux element which then feeds a single nvinfer element.

I am measuring how long the nvinfer element takes to run my parsing the output of GST_TRACE timestamps.

14

batched-push-timeout=0 makes nvstreammux stop waiting as soon as the first frame arrives. Small arrival differences between cameras can therefore produce partial batches instead of one full batch. This may reduce throughput because nvinfer performs several small inference calls rather than one efficient batched call.
You can check the frame number in a batch using a pad probe after nvstreammux:

NvDsBatchMeta\* meta = gst_buffer_get_nvds_batch_meta(buffer);
g_print("frames in batch: %u\\n", meta->num_frames_in_batch);

Please refer to “Solution 3” in the-deepstream-application-is-running-slowly. set the batched-push-timeout to the (1000000 us/maximum fps among the videos).

First, the latency measured with GST_TRACE includes nvinfer queueing, preprocessing, synchronization and postprocessing—not only TensorRT GPU execution. Second, parallel execution means inference workloads can overlap; it does not guarantee constant per-frame latency as more cameras share finite GPU resources. Latency may increase while aggregate throughput still improves.
please retry after correcting batched-push-timeout=0.

15

Hi, thanks for the answer.

I did batched-push-timeout=0 on purpose because for my application latency is more important than throughput. I noticed that if I increase the timeout to a value that guarrantees that frames from all cameras will arrive before running a batched inference, the time that nvinfer takes to run is bigger and it increases with each camera added. Which makes sense because now the GPU is running inference on a frame that’s N number of times bigger than a single camera frame, where N is the number of cameras.

The best latency scenario I got in my tests was using one nvinfer element being fed with multiple cameras (the second case I described).

In this scenario, when setting the timeout to zero, can the inference workload of each camera overlap with each other? For example:

  • Frame from camera 0 arrives
  • GPU starts inference on frame of camera 0
  • Frame from camera 1 arrives
  • GPU starts another inference on frame of camera 1
  • GPU finishes inference on frame of camera 0
  • GPU finishes inference on frame of camera 1
  • …

Also in this scenario, the inference time of each frame remained almost constant no matter how many cameras I am using (I tested with up to 5 cameras). So it looks like it is the scenario with the smallest latency.

I suppose that if I add too many cameras in a way that the GPU needs to run too many parallel inference workloads concurrently and in this scenario the many parallel instances of the model cannot fit in the GPU at the same time, that’s when I’ll start to see a slowdown.

But as a long as the GPU can run a certain number of parallel inference workloads without exhausting its resources, the processing time of each individual inference remains constant.

Are my observations correct?

16

With batched-push-timeout=0 and a single nvinfer, per-camera inference does not overlap as independent concurrent GPU workloads.
nvinfer uses one TensorRT context/stream. Batches (often size 1) are enqueued in order: the next enqueue can start before the previous finishes, but GPU inference on that stream still runs sequentially. That is pipeline overlap, not parallel per-camera inference.

With batched-push-timeout=0, nvstreammux often forms partial batches (commonly 1 frame). Each nvinfer call then runs inference on that single frame, so the measured per-call inference time can stay roughly constant as you add cameras.
You configured batch-size=N for both nvstreammux and nvinfer. If each call actually infers only one frame, that does not leverage the GPU parallelism the batch-N engine is meant to provide.

17

Thank you so much for all the context here. Its been really helpful!

So in the second case I provided , since inference is not running in parallel, theoretically frames will be waiting in a gstreamer queue before they can be processed? So I should be able to measure that wait time using GST traces and this time should increase as more cameras are added? Since a frame from a camera will need to wait, in the worst case scenario, (N-1) frames to be processed first before its own frame gets to be processed?

And in the first case, where I use multiple independent pipelines each one with its own nvinfer element, then the inferences will truly happen in parallel? Since now there will be one TensorRT context/stream per nvinfer element? And the reason why I see a slowdown in inference time is because now there’s multiple truly parallel inferences running and they are sharing the same limited amount of resources of the GPU?

18

Please refer to my comment on Aug 12, my suggetsion is setting batched-push-timeout correctly first. If latency is more important, please refer to the-deepstream-application-is-running-slowly-jetson-only for improving latency.
Yes, frames may wait either in an upstream GStreamer queue or in nvinfer’s internal input/process queues. GST traces or pad probes can measure the buffer’s wall-clock delay through explicit queues and nvinfer. However, nvinfer latency includes internal queueing, preprocessing, inference, and postprocessing, so it does not isolate queue wait alone. With batched-push-timeout=0, if the mux instead produces N separate single-frame batches, the last one may wait behind N−1 batches—assuming there is no previous backlog.
The method is the first case is less efficient because they duplicate TensorRT contexts and buffers and add per-frame launch/scheduling overhead. Yes. Each nvinfer instance has its own TensorRT execution context and CUDA stream, so independent pipelines can run inference concurrently. Actual kernel overlap depends on available GPU resources. The increased latency is likely caused by contention for SMs, Tensor Cores, and memory bandwidth; confirm this using Nsight Systems GPU metrics.

19

Hi, thanks for the info. In the first scenario (one nv infer element per camera). This is what I see in Nsight using 3 cameras. The screenshot spans 28ms which roughly matches my measurements of the nvinfer time. I can see that SM Warp occupancy is much bigger while the “ExecutionContext” (orange) is running. But I only see a spike in the “Tensor Active” metric after the “ExecutionContext” (orange) is done. Is that “ExecutionContext” representing the inference time? It takes around 8ms to run. The “Tensor Active” metric is at high levels for around 15ms in this screenshot.

3_cameras_28_ms
3_cameras_28_ms1321×868 103 KB

In comparison with two cameras:

2_cameras_21_ms
2_cameras_21_ms1644×837 68.9 KB

The screenshot spans about 21ms, which also my measurements of nvinfer execution time with gst tracers. Now the “ExecutionContext” runs in about 3ms. And I also see the same behaviour of the metric “Tensor Active” jumping only after the “ExecutionContext” runs. The Tensor Active metric is at high levels for around 9.5ms in this screenshot.

I’m also attaching both nsys logs. Does those logs/screenshots prove that there’s some contention happening due to lack of GPU resources? With that I mean the only way for this contention to not happen would be to move to a platform with a beefier GPU like the Jetson AGX Thor?

2_cameras.zip (5.1 MB)

3_cameras.zip (6.9 MB)

21

No. The orange bar is ExecutionContext::enqueue (TensorRT host enqueue), not full GPU inference. Kernels continue asynchronously after it. That is why Tensor Active rises later. Your ~21/28 ms GST nvinfer times match the whole GPU busy window, not the orange bar alone (~3/8 ms).

Yes — they support GPU resource contention, not starvation. Multiple nvinfer enqueues overlap; with 3 cameras enqueue and some kernels get longer, and the GPU stays busy across that window.

No. Jetson AGX Thor is not the only way. Your slowdown comes from several nvinfers running at the same time and competing for the same GPU on Orin. A bigger GPU can reduce that pressure, but try these first:

  1. Prefer one nvinfer with nvstreammux (batching) if that still meets your latency goal
  2. Use a lighter / faster network (smaller input, INT8/FP16, etc.)
  3. Run fewer nvinfers at the same time (don’t give every camera its own concurrent nvinfer if you can avoid it)
  4. Use DLA for some inference if your model supports it
    Upgrade the platform only if you still need many independent nvinfers running concurrently and Orin is still too slow after the above.
22

Thank you so much!

Regarding the bullet point 1:

I understood that if I run one infer with nvstreamux feeding data from multiple cameras to a single nvinfer and setting batched-push-timeout=0 it would push frames for inference as soon as any frame is available. It would be almost the same as having one nvinfer per camera, except that it wouldn’t be able to run inferences for each camera in parallel. So in the end, from a latency perspective, it would be worse because it would need to wait for one inference to finish before starting another one. This doesn’t happen in the scenario where there’s one nvinfer per camera. Is this correct? If so, and considering latency is more important, using one nvinfer per camera would be better?

In another scenario, if I set the batch-push-timeout to a value that would allow frames from all cameras to arrive before they are sent to the GPU as a single batch, the inference time will grow linearly with each new camera. Example: if a single frames takes 10 ms to run, if I have 5 cameras the time to run inference on all 5 frames as a batch would be 50ms. Since latency is the more important metric, this is also not ideal.

At least from my tests, using one nvinfer per camera seems to increase inference time by around 7ms every time a new camera is added. With a single camera the inference time is 10ms, so an increase of 70%.

With the batched approach, each new camera adds 10ms of inference time, so a 100% increase per camera. So it seems the approach using one nvinferis slightly faster because each camera increases the latency time by 70% of the inference time, instead of the 100% of the batched approach. At the cost of using more resources of the GPU

In the end this seems like a tradeoff between:

  • Using less GPU resources in the batched approach at the cost of a bigger latency
  • Using more GPU resources in the “one nvinfer per camera” approach at the cost of a smaller latency

Do you agree with this tradeoff?

24

If you only compare “one mux → one nvinfer (timeout=0)” with “one nvinfer per camera”, Partly. batched-push-timeout=0 does push as soon as a frame is available (often 1-frame batches). That part is right. It is not almost the same as one nvinfer per camera. One nvinfer = one CUDA stream, so cameras are inferred one after another on the GPU, not in parallel. The next batch can be queued before the previous GPU kernels finish, but it does not run beside them.

If in that same comparison, Yes.

I can’t get the same resuts. could you share the cfgs of nvtreammux and nvinfer, and runnning logs including “inference time”, mentioned in “I am measuring how long the nvinfer element takes to run my parsing the output of GST_TRACE timestamps.”?

25

I will get you this data today!

26

Hi

So the time that nvinfer only takes to run is fairly constant in this pipeline with 5 cameras. I was not measuring it correctly before. This is my test pipeline (with some elements and configs ommited).

nvstreammux name=mux batch-size=5 live-source=true sync-inputs=false batched-push-timeout=0 ! nvinfer output-tensor-meta=1 unique-id=1 batch-size=5 ! nvvideoconvert ! appsink
v4l2src device=/dev/video3 ! mux.sink_3
v4l2src device=/dev/video6 ! mux.sink_6
v4l2src device=/dev/video7 ! mux.sink_7
v4l2src device=/dev/video8 ! mux.sink_8
v4l2src device=/dev/video5 ! mux.sink_5

What actually increases with each new added camera is the total latency from source to appsink.

I instrumented my code to print the timestamps in nanoseconds:

Adding timestamp to camera buffer: 1787592403288421505
Adding timestamp to camera buffer: 1787592403288425409
Adding timestamp to camera buffer: 1787592403288426785
Adding timestamp to camera buffer: 1787592403288426785
Adding timestamp to camera buffer: 1787592403288421473

Frame timestamp on appsink: 1787592403288425409
Now timestamp on   appsink: 1787592403314692331
Appsink now - original buffer timestamp: 26 ms

Frame timestamp on appsink: 1787592403288426785
Now timestamp on   appsink: 1787592403324381953
Appsink now - original buffer timestamp: 35 ms

Frame timestamp on appsink: 1787592403288421473
Now timestamp on   appsink: 1787592403333876502
Appsink now - original buffer timestamp: 45 ms

Frame timestamp on appsink: 1787592403288426785
Now timestamp on   appsink: 1787592403343290058
Appsink now - original buffer timestamp: 54 ms

Frame timestamp on appsink: 1787592403288421505
Now timestamp on   appsink: 1787592403352478140
Appsink now - original buffer timestamp: 64 ms

The “Adding timestamp to camera buffer” messages are printed right after each buffer is created and they show the timestamp that’s stamped on each frame. One can check they differ by around 5 us at most. This is expected because all the cameras are trigged at the same time, so there’s only small noisy differences in the processing of each camera until the point they are time stamped.

Then, on the appsink callback, I take the current system time in ns and subtract it from the original timestamp of the frame (those are the “Appsink now - original buffer timestamp” messages).

One can see that each call to the appsink callback is roughly spaced 10ms in time. I guess that is expected since you said the inferences for each camera cannot run in parallel in this scenario. So the last frame that goes through needed to wait 40 ms for all the other N-1 inferences to finish (which in this case was 4 inferences, and each one delayed the last frame time roughly 10 ms).

So there’s a factor in play that increases the total latency of the pipeline, and that factor is proportional to the number of cameras in the whole pipeline.

I get it that when using only one nvinfer the GPU resource allocation is “smarter” since there’s only one CUDA stream and TensorRT execution context, thus overral the GPU is using less resources. But the original problem of “increasing end-to-end latency as more cameras are added” is still present when compared to having one independent pipeline per camera, each with its own nvinfer element.

It’s a difference between:

  • Running faster serialized inferences, but each camera having to wait for the previous inferences to finish vs
  • Running slower parallelized inferences (slower because each inference instance is competing for GPU resources with the other inferences), but each camera doesn’t need to wait for the inferences of other cameras to finish before starting its own.

Is that the results you would expected to see to?

Thanks for all the help so far, it’s been really useful!

27

How you add the timestamp printed as:
Adding timestamp to camera buffer: 1787592403288421505
On which plugin or pipeline element is the timestamp added? Is it set in a pad probe after v4l2src, in a custom plugin, or directly in GST_BUFFER_PTS/custom metadata?

If you still compare “one mux → one nvinfer (timeout=0)” with “one nvinfer per camera”, then yes, the general behavior you observed is expected. In my test based on deepstream-test3,
One-source/batch-1 nvinfer: median component latency was about 10.69 ms.
Five-source/batch-5 shared nvinfer: median was about 19.22 ms.
My batch test used batched-push-timeout=40000 and set the batch-sizeproperty of both nvinfer and nvstreammux to N, so it is not directly equivalent to your timeout-0 test. However, it also shows that a five-frame batch does not necessarily take five times the single-frame latency. Therefore, Please rerfer to my suggestion on Aug 14.

28

It is set on a custom “timestamp” element right after the v4l2src (which I omitted in my example for simplicity). I set it using the gst_buffer_add_reference_timestamp_meta API

Thanks for all the help, I think I have everything I needed to know for now!