Color format options and impact on each other

Accelerated Computing Intelligent Video Analytics DeepStream SDK
,
1

In DeepStream config for nvdspreprocess we have network-color-format and for nvinfer we have model-color-format. Can these be used at the same time? or is one overruling the other?
From the documentation it is not really clear what happens if both are used in their components respectively. Furthermore, model-color-format is also depending on value of input-tensor-from-meta (via config) or input-tensor-meta (set at runtime).

My current assumption is that if an nvdspreprocess is used, then it is overruling model-color-format. If the component nvdspreprocess is not part of the pipeline, model-color-format is used depending if input-tensor-from-meta or input-tensor-meta is set to false. Is my understanding correct? (would be good to add this somewhere in the docs if not already present - at least I wasn’t able to find anything this specific).

Now to my use case: what if I have various models that use different color formats, one might use BGR while the other RGB - if the nvdspreprocess component is already set to RGB, will it be possible to set BGR for the other downstream model?

greetings from the depth of DeepStream _\\ //

3

In short: nvdspreprocess is used for custom preprocessing. If you use nvdspreprocess, the default preprocessing configurations in nvinfer will be disabled.

When the nvinfer input-tensor-meta property is true or input-tensor-from-meta=true in the configuration file, the tensors output by nvdspreprocess will be used for inference. In this case, nvinfer model-color-format will no longer take effect.

If the model input layer names are different, you can use the following pipeline.nvinfer reads the layer name to match the corresponding tensor.

nvdspreprocess tensor-name=input_rgb --> nvinfer1 --> nvdspreprocess tensor-name=input_bgr --> nvinfer2

If the layer names are the same, you can first call nvds_remove_user_meta_from_batch to remove the upstream NVDS_PREPROCESS_BATCH_META and then add the new NVDS_PREPROCESS_BATCH_META.

4

thx for confirmation and ideas how to approach this.

how much performance impact will there be if we have two nvdspreprocess nodes?

going from RGB to BGR should be just a channel swap but having another nvdspreprocess might add additional compute cost, correct?

5

It will consume some GPU resources.

You can check the GPU usage percentage using nvidia-smi(dGPU)/tergastat(jetson) to see the specific impact.

1 Like
6

is there a way to just swap color channels for downstream models after nvinfer?

7

If you are only adjusting the model’s input, the default preprocessing in nvinfer is sufficient to accomplish the task. This kind of pipeline can also work.

nvinfer1  model-color-format=0  -->  nvinfer2 model-color-format=1

nvdspreprocess is only needed when nvinfer’s default preprocessing cannot complete the task, such as in ROI scenarios or when there are special operations.

8

yes, exactly I rely on nvdspreprocess because I’m using ROIs. therefore not using it is not an option.

9

nvdspreprocess involves not only channel swapping but also normalization. These operations are all performed on the GPU, so the different rgb order should have minimal impact on performance.

10

sounds good.
not sure about the details of the downstream nvdspreprocess after first nvinfer.
if the first nvdspreprocess creates ROIs, I assume the following nvdspreprocess doesn’t need to do this again. Is it sufficient to only do the color conversion, is this possible?

11

You can customize the CUDA kernel of the nvdspreprocess custom library.

12

I will swap color channels in ONNX, then there will be no need to change the pipeline. thx for all the information.