A problem about ResNet50 for SGIE classification

[yangyi]

Please provide complete information as applicable to your setup.

**• Hardware Platform (Jetson / GPU)GPU
**• DeepStream Version 6.1
**• TensorRT Version 8.2E
**• NVIDIA GPU Driver Version (valid for GPU only) 515

Dear professor:
I meet a problem. I use pytorch to train a resnet50 for the SGIE. I hope to get the classification result of the secondary engine.
My way is:
(1) build resnet50 by the torchvision
(2) train the NN,
(3) translate NN into onnx
(4) translate the onnx model into engine file
(5) DS load the engine

but I meet a strange problem

(1) Because the resnet50 in torchvision does not has softmax. If I add the softmax layer to the end of resnet50, and train the NN->ONNX->ENGINE->DS, So I can get the secondary engine output label.

(2) But if I train resnet50 without softmax, while I add the softmax layer in the step of translation onnx as below:

class model_with_softmax(nn.Module):
def init(self):
super(model_with_softmax, self).init()
self.model = torch.load(‘saved_model.pth’)
def forward(self, x):
x = self.model(x)
x = F.softmax(x, dim=1)
return x

I can get onnx → engine → load by DS.
But this way, I can not get the output label of secondary engine!

What is the matter? Could you give me some help?

Thank you very much.

[yuweiw]

What pipeline did you use in DeepStream?
Did you use our demo code to test this model?

[yangyi]

Thank you for your quick response.
my pipeline is the default of DS. But I modified the deepstram-app-main.c.
the modify main function

//==== I  add the code about SGIE result analysis =====//
if (appCtx[i]->config.primary_gie_config.enable)
{
    GstPad *src_pad = NULL;
    GstElement *sgie_pad = appCtx[i]->pipeline.common_elements.secondary_gie_bin.sub_bins[0].secondary_gie;
    src_pad = gst_element_get_static_pad (sgie_pad, "src");
    if (!src_pad)
        g_print ("Unable to get second_gie src pad\n");
    else
    {
        gst_pad_add_probe(src_pad, GST_PAD_PROBE_TYPE_BUFFER,
                  sgie_pad_buffer_probe, NULL, NULL);
        gst_object_unref (src_pad);
    }
}

// ====================================================//
}

and I add the sige_pad_buffer_probe as below

//===========This function is added here======//
static GstPadProbeReturn
sgie_pad_buffer_probe (GstPad * pad, GstPadProbeInfo * info, gpointer u_data)
{
static guint use_device_mem = 0;
NvDsBatchMeta batch_meta = gst_buffer_get_nvds_batch_meta (GST_BUFFER (info->data));
/ Iterate each frame metadata in batch */
for(NvDsMetaList * l_frame = batch_meta->frame_meta_list; l_frame != NULL; l_frame = l_frame->next)
{
printf(“========================frame list========================\n”);
NvDsFrameMeta *frame_meta = (NvDsFrameMeta *) l_frame->data;

for(NvDsMetaList * l_obj = frame_meta->obj_meta_list; l_obj != NULL;  l_obj = l_obj->next)
{
  printf("---------------------list obj--------------------------\n");
  NvDsObjectMeta *obj_meta = (NvDsObjectMeta *) l_obj->data;
  printf("l_obj id = %d\n", obj_meta->class_id);
  printf("%d\n", obj_meta->classifier_meta_list);
  for(NvDsMetaList *l = obj_meta->classifier_meta_list; l != NULL; l = l->next)
  {
     printf("-------classifer_list-----\n");
     NvDsClassifierMeta * classifierMeta = (NvDsClassifierMeta *) (l->data);
     for (NvDsMetaList * n = classifierMeta->label_info_list; n != NULL; n = n->next)
     {
         NvDsLabelInfo *labelInfo = (NvDsLabelInfo*) (n->data);
         printf("label: %s \n ", &labelInfo->result_label[0]);
     }
  }
}

}
return GST_PAD_PROBE_OK;
}
//==============================================================//

The problem is :
if I use the Resnet50 with softmax, and I can get the print show: “lables=”

But is I use the softmax is not added in the end of resnet50, but add when translate into onnx, I can not get labels information.

Thank you very much.

[yangyi]

Dear Yuweiw
I test the two engines in deepstream-test2. The result is the same.
One SGIE can classify the object, but the other can not.

I don’t know what is the problem.

I waiting for your responce.

[yuweiw]

Why do you use the second method to train models?
We suggest you use the first method to train model NN->ONNX->ENGINE->DS.
Also you can verify your tained model first with TAO tools. If the output is correct, then you can use deepstream.

[yangyi]

Thank you for your response. Yes I use the method:NN->ONNX->ENGINE->DS.

The passed days, I tried many many times. And I found the following rule:
(1) If I do not add softmax to Resnet50, the DS can not get the result of SGIE.
(2) If I transformed the NN to ONNX with a large W and H, the DS can not get the result of SGIE.
(3) It is weirdest that, Some times if I reboot the system, and transform the NN to ONNX again, the DS can get the SGIE result.

So the reboot method makes me very confused. Thank you very much.

Before, I used the TAO method. However, it is inconvenient. especially establish the environment. So I hope to use the method of onnx and tensorrt.

[yuweiw]

You should verify your model is ok before running it in deepstream. The running of the deepstream has nothing to do with the reboot operation. Is it possible that reboot affects the transformation process?

[yangyi]

@yuweiw

ify your model is ok before running it in deepstream. The running of the deepstream has nothing to do with the reboot operation. Is it possible that

Thank you very much. I am trying to use ONNX directly.
But I meet a problem: It looks like the SGIE just can be used for special object.

I hope to use SIGE to classificat object of class_id=0, however, just the id=19 can be recognized.

I have created another topic here, despite the original issues are different, but now the problem is the same.

Could you kindly help me. Thank you very much.

My configure is
[property]
gpu-id=0
net-scale-factor=0.0039215697906911373
offsets=97.6980; 99.9760; 88.8259

labelfile-path=./SGIElabels.txt
onnx-file=./ResNet50_with_softmax_400.onnx
model-engine-file=./aa.engine
batch-size=1
force-implicit-batch-dim=0

0=FP32, 1=INT8, 2=FP16 mode

network-mode=2
input-object-min-width = 5
input-object-min-height = 5

process-mode=2
model-color-format=1
gie-unique-id=3
operate-on-gie-id=1
operate-on-class-ids=0
is-classifier=1
output-blob-names=predictions/Softmax
classifier-async-mode=0
classifier-threshold=0.2

network-type=1
#scaling-filter=0
#scaling-compute-hw=0

Thank you very much.

[yangyi]

Thank you very much. It is my mistake, I am trying another way