Nvinfer and streammux batch-size property doubts

[tvai]

Hi,

I have an application creating a gstreamer pipeline.
I set dynamically nvstreammux batch-size property according to the number of streams, this part is correct according to the documentation.

Then I have a yolo detector, with batch-size=16 in the config.txt file.
This model has a dynamic shape of min=8, opt=16, max=32.

Should I set nvinfer batch-size=32 (max batch size of the model) in the config.txt file?
Should I set nvinfer batch-size property at runtime to the nvinfer element?

If I run 33 streams, and have nvinfer max batch-size of 32, what does DeepStream:

• one batch of 32 images, then one batch of 1 image?
• 2 batches of 32 images (32 first, then 32 with 31 dummy buffers)?

If I run 1 stream, and have nvinfer max batch-size of 32, what does DeepStream:

• one batch of 1 image?
  -one batch of 32 images, (1 image, and 31 dummy buffers)?

I have of course the same doubts when I’m using classifiers after the detector, should I set dynamically in my code the batch-size of the classifiers according to the number of objects detected?

Thanks in advance for the help.

[Fiona.Chen]

@tvai

This model has a dynamic shape of min=8, opt=16, max=32.

Should I set nvinfer batch-size=32 (max batch size of the model) in the config.txt file?

You can set batch-size to any number between 8 and 32.

@tvai

Should I set nvinfer batch-size property at runtime to the nvinfer element?

No, it is not supported.

@tvai

If I run 33 streams, and have nvinfer max batch-size of 32, what does DeepStream:

• one batch of 32 images, then one batch of 1 image?
• 2 batches of 32 images (32 first, then 32 with 31 dummy buffers)?

2 batches of 32 images (32 first, then 32 with 31 dummy buffers)

@tvai

If I run 1 stream, and have nvinfer max batch-size of 32, what does DeepStream:

• one batch of 1 image?
  -one batch of 32 images, (1 image, and 31 dummy buffers)?

You can set nvinfer batch-size as 8(min). So there are 7 dummy buffers in the batch.

@tvai

should I set dynamically in my code the batch-size of the classifiers according to the number of objects detected?

No. It is not supported. You can set the batch-size of SGIE as the maximum number it can support.

[tvai]

Thanks @Fiona.Chen for the answer!