Deploy EfficientNet-b0 in Deepstream

[jingnanopt]

Please provide complete information as applicable to your setup.

**• Hardware Platform (Jetson / GPU) AGX
**• DeepStream Version 5.0
**• JetPack Version (valid for Jetson only) 4.4
**• TensorRT Version 7.1
**• NVIDIA GPU Driver Version (valid for GPU only) 10.2
**• Issue Type questions
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)

I have implemented the EfficientNet-b0 classification network using the tensorRT C++ API, and the single image result is consistent with the Python result. However, when deploying the EfficientNet-b0 classification network to deepstream, it encountered a situation that did not match the calculation result of the Python. I think it may be inconsistent with Python image preprocessing in deepstream.

The primary-gie is yolov5

[primary-gie]
enable=1
gpu-id=0
model-engine-file=yolov5s_bn4.engine
labelfile-path=labels.txt
#batch-size=1
#Required by the app for OSD, not a plugin property
bbox-border-color0=1;0;0;1
bbox-border-color1=0;1;1;1
bbox-border-color2=0;0;1;1
bbox-border-color3=0;1;0;1
interval=0
gie-unique-id=1
nvbuf-memory-type=0
config-file=config_infer_primary_yoloV5.txt

config_infer_primary_yoloV5.txt:

[property]
gpu-id=0
net-scale-factor=0.0039215697906911373
#0=RGB, 1=BGR
model-color-format=0
model-engine-file=yolov5s_bn4.engine
labelfile-path=labels.txt
process-mode=1
network-mode=2
num-detected-classes=80
gie-unique-id=1
network-type=0
#is-classifier=0
output-blob-names=prob
#cluster-mode=2
maintain-aspect-ratio=1
parse-bbox-func-name=NvDsInferParseCustomYoloV5
custom-lib-path=nvdsinfer_custom_impl_Yolo/libnvdsinfer_custom_impl_Yolo.so
engine-create-func-name=NvDsInferYoloCudaEngineGet
#scaling-filter=0
#scaling-compute-hw=0
#force-implicit-batch-dim=1

[class-attrs-all]
nms-iou-threshold=0.5
pre-cluster-threshold=0.4

roi-top-offset=0
roi-bottom-offset=0
detected-min-w=0
detected-min-h=0
detected-max-w=0
detected-max-h=0

The secondary-gie2 is efficientnet-b0

[secondary-gie2]
enable=1
#model-engine-file=efficientnet-b0_3_channel_misheng.engine
model-engine-file=efficientnet-b0_single_channel_misheng_bn8.engine
batch-size=8
gpu-id=0
gie-unique-id=6
operate-on-gie-id=1
operate-on-class-ids=0;
config-file=config_infer_secondary_efficientnet_b0_bn8.txt
config_infer_secondary_efficientnet_b0_bn8.txt

[property]
gpu-id=0
net-scale-factor=0.0175
offsets=116.28

labelfile-path=efficientnet_b0_labels.txt
model-engine-file=efficientnet-b0_single_channel_misheng_bn8.engine

#input-dims=3;224;224;0
infer-dims=1;224;224
#uff-input-blob-name=input_1
batch-size=8
process-mode=2
model-color-format=2
network-mode=0
network-type=1
num-detected-classes=3
interval=0
gie-unique-id=1
output-blob-names=prob
classifier-threshold=0.1
force-implicit-batch-dim=1
maintain-aspect-ratio=0

#network-mode=1
#is-classifier=1

classifier-async-mode=1
#classifier-threshold=0.51
input-object-min-width=2
input-object-min-height=2

In secondary-gie2, image is gray. This is my preprocessing method using C++.

for (int i = 0; i < global_params.input_h * global_params.input_w; i++) {

     data[i] = ((float)img.at<cv::Vec3b>(i)[0] - 116.28) * 0.0175;

}

[AastaLLL]

Hi,

The pipeline only run the classifier on the ROI region detected by the YOLOv5.
Is this identical to your python example?

More, please check the preprocessing formula in Deepstream:
https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvinfer.html

Formula:

y = net scale factor*(x-mean)

Please align the formula by adjusting the parameters below:

net-scale-factor=0.0175
offsets=116.28

Thanks.

[jingnanopt]

Thanks.
In my pipeline, the primary detector is YOLOv5, secondary-gie2 is EfficientNet. EfficientNet input is the ROI region detected by the YOLOv5. The parameters is adjusted as preprocessing in my classifier python sample.
net-scale-factor=0.0175
offsets=116.28

This is my classifier preprocessing method using C++.

for (int i = 0; i < global_params.input_h * global_params.input_w; i++) {
data[i] = ((float)img.atcv::Vec3b(i)[0] - 116.28) * 0.0175;

}

[AastaLLL]

Hi,

It seems that the color formats in YOLOv5 and EfficientNet are different.
Is that correct? (RGB to GRAY ?)

If yes, could you check if the ROI input for secondary nvinfer is gray or not?
Since the preprocessing you mentioned is for channel=3 rather than channel=1.

Thanks.