Nvstreamux process 1 frame in a batch at a time instead of 32 frames

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

• Hardware Platform (Jetson / GPU) : NVIDIA Jetson AGX Orin
• DeepStream Version : 7.0
• JetPack Version (valid for Jetson only) : 6.0
• TensorRT Version : 8.6.2.3
• Issue Type( questions, new requirements, bugs) : question

I have such a code in Python for running model inference on Nvidia Jetson Orin:

...
Create nvstreammux element to form batches from one or more sources
streammux = create_pipeline_element("nvstreammux", "stream-muxer", "Stream Muxer")
streammux.set_property("width", 1920)
streammux.set_property("height", 1080)
streammux.set_property("batch-size", 32)  # Set batch size to 32
streammux.set_property("batched-push-timeout", 4000000)  # Adjust if needed
streammux.set_property("live-source", 1)

# Crate nvvidconv element to convert the input stream
nvvidconv = create_pipeline_element(
      "nvvideoconvert", "convertor", "Video Converter"
  )
nvvidconv.set_property("nvbuf-memory-type", 0)

  # Create caps filter to convert the input stream to RGBA format
caps_rgb = create_pipeline_element("capsfilter", "nvmm_caps_rgb", "CapsFilter")
caps_rgb.set_property(
      "caps",
      Gst.Caps.from_string("video/x-raw(memory:NVMM), format=RGBA"),
  )

# Create nvinfer element to run inference on the input stream from nvsreammux
pgie = create_pipeline_element("nvinfer", "primary-inference", "Primary Inference")
pgie.set_property("config-file-path", f"inference/{model_name}/config_infer.txt")

# Create fakesink element
sink = create_pipeline_element("fakesink", "waylandsink", "Wayland Sink")

This is configuration file for the model:

model-color-format=0 # 0=RGB, 1=BGR
onnx-file=models/new_arcing_model_nchw.onnx
model-engine-file=models/new_arcing_model_nchw.onnx_b32_gpu0_fp32.engine
labelfile-path=models/labels.txt
infer-dims=3;224;224
batch-size=32
network-mode=0 # 0=FP32, 1=INT8, 2=FP16 mode
network-type=1
num-detected-classes=1
process-mode=1
gie-unique-id=1
classifier-threshold=0.0

The problem that I have is number of batches that are being processed by nvstreamux. I set it to 32, increased the batched-push-timeout but on live camera source it processes only 1 frame in a batch instead of 32. When switching to file source it can process only 6-9 frames at a time no matter how big is the batched-push-timeout. What might be the cause of very few frames processed by nvstreamux in a batch? The model itself can take a batch of 32 images.

1 Like
2

How many sources do you have? my guess is they are not produced at a sufficient speed so batched-push-timeout kicks in and pushes the buffer, you can play with that parameter too

Regards,
Allan Navarro
Embedded SW Engineer at RidgeRun

Contact us: support@ridgerun.com
Developers wiki: https://developer.ridgerun.com/
Website: www.ridgerun.com

3

So i have only one camera source. When it comes to batched-push-timeout I tried playing with this parameter changing it from 0.04 s to 4 or even 10 seconds but it always has only 1 frame per batch when it is a live camera source, however when i switch it to file source then it is 6-9 frames per batch (still not 32 as specified in set_property but more compared to camera source.

4

Ah yes, sorry I didn’t see the last part of the message. How do you determine the number of frames that are being processed?

Regards,
Allan Navarro
Embedded SW Engineer at RidgeRun

Contact us: support@ridgerun.com
Developers wiki: https://developer.ridgerun.com/
Website: www.ridgerun.com

6

I think this is as expected. The batch-size parameter means the maximum number of frames in a batch.

For live stream(rtsp/camera), Only the frames before batched-push-timeout will be composed into a batch.

Setting batched-push-timeout to -1 will cause the nvstreammux to wait forever until the frames number reaches the maximum, but this will cause a delay.

https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvstreammux.html#gst-properties

7

Maybe I am confusing what is the batch-size parameter in nvstreammux.

According to this:
https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_FAQ.html?highlight=batch%20size#what-is-the-difference-between-batch-size-of-nvstreammux-and-nvinfer-what-are-the-recommended-values-for-nvstreammux-batch-size

Is the batch-size parameter related to the number of sources or to the number of batches i want to pass to the inference model?

When it comes to inference if batch-size in model config file is set to 32, it means that before inference frames are packed into batch of 32 frames and the nvinfer waits for those frames? Are the batches accumulated, if so how long does nvinfer waits?

1 Like
8

Yes, for example: if batch-size is 32, and there are 4 sources, each with 8 frames form a batch.
But not always, this is one of those situations

nvinfer does not wait, nvstreammux waits for frames and forms batches.
For nvstreammux, it will wait for up to 32 frames before the batched-push-timeout occurs.

If the batched-push-timeout is reached, but the batch-size is not 32, inference will still be performed.

9

Ok, I still cannot understand why when specifying batch-size of nvstreammux to 32 and setting its batched-push-timeout to high number like 5 seconds it does not pack 32 frames into batch but only 4-6.
As an example I modified python script deepstream-test1 app:

so that the model in nvinfer config has now batch-size of 32:
dstest1_pgie_config.txt (3.0 KB)
I’ve modified the deepstream_test_1.py script to incorporate a probe function on the pgie element’s source and sink pads. This function allows you to verify and potentially manipulate the batch size information retrieved from the batch_meta structure. The script now prints the batch size for each processed batch.
deepstream_test_1.py.txt (7.5 KB)

After a while of converting model to 32 batch_size this is the output I got:

[SNK PAD] Batch size: 4
[SRC PAD] Batch size: 4

The default value of batched-push-timeout is 33000 which is equivalent to 0.033 seconds. When I change the batched-push-timeout to 2000000 which is 2 seconds it still yields the same output for batch-size:

[SNK PAD] Batch size: 4
[SRC PAD] Batch size: 4

Also I do not know why the pipeline stops after the first batch.

My question is: how should I specify the batch-size of nvstreamux and what could I do to form a batch of 32 frames to pass it once to my model that takes as input only batch of 32 images?

1 Like
10

Do you mean your model only supports a fixed batch of 32 images as input? Does your model not support dynamic shape input?

nvstreammux doesn’t seem to guarantee this, as I mentioned above the batch-size property of nvstreammux is the maximum number of frames.
The multiple streams are the main scenario for DeepStream.

You can refer to this topic.

11

I have both models, one with fixed batch-size of 32 and the other one with dynamical input. Surprisingly the one with fixed input batch-size to 32 works even tho the batches are not formed in 32 frames. I print batch size before each inference and the size of a batch is around 6-9. This is however when the input stream is file source.

In the live case scenerio, when the input stream is a camera, my print statement just before the inference (on the sink pad of the nvinfer) yields that batch-size is 1.

Could I somehow queue frames just before the inference to create a batch of 32 images?

When it comes to the model, is it advised to use model with fixed input shape or dynamic one?

12

I can understand why batch-size is 1 in live camera scenerio. But I want to make a faster inference by leveraging batching.

My question is, how can I create a batch of 32 images so that I can pass it once to my model instead of passing 1 image at a time to my model.

13

You may need to set the value of buffer-pool-size

    streammux.set_property('batch-size', 32)
    streammux.set_property('buffer-pool-size', 64)

1.py (8.2 KB)

14

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks

15

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.