A confused problem about Pytorch -- ONNX -- TensorRT -- Deepstream SGIE

Please provide complete information as applicable to your setup.

**• Hardware Platform (Jetson / GPU)GPU
**• DeepStream Version 6.1
**• TensorRT Version 8.2E
**• NVIDIA GPU Driver Version (valid for GPU only)515

Dear professor, this is a hurry problem to me, it makes me sad server days.

I use the pytorch to load ResNet50 in torchvision. I hope to train it by my own dataset, and translate it by tensorrt for SIGE of DS. However, it is a crushing problem for me.

(1) My model is:

class ResNet50_with_softmax(nn.Module):
def init(self):
super(ResNet50_with_softmax, self).init()
self.net = models.resnet50(weights=models.ResNet50_Weights.DEFAULT)
fc_features = self.net.fc.in_features
self.net.fc = nn.Linear(fc_features, 3)
self.softmax = nn.Softmax(dim=1)

def forward(self, x):
    x = self.net(x)
    x = self.softmax(x)
    return x

(2) My onnx transform as below:
import torch
from myModel import ResNet50_with_softmax, ResNet50_No_softmax

input_names = [“input”]
output_names = [“predictions/Softmax”]
input_data = torch.randn((2, 3, 300, 200))
model = ResNet50_with_softmax()
model.eval()
output = model(input_data)
print(output)
torch.onnx.export(model,
input_data,
‘ResNet_with_softmax_D_300.onnx’,
export_params=True,
do_constant_folding=True,
input_names=input_names,
output_names=output_names,
dynamic_axes={‘input’: {0: ‘batch_size’, 2:‘width’, 3:‘height’},
‘predictions/Softmax’: {0: ‘batch_size’}})

(3) My command for translating the onnx into tensorrt as below

/usr/src/tensorrt/bin/trtexec --onnx=ResNet_with_softmax_D_300.onnx
–minShapes=input:1x3x300x200
–optShapes=input:8x3x300x200
–maxShapes=input:16x3x300x200
–workspace=14096
–saveEngine=SGIE_with_softmax_D_300.engine --fp16

(4) I copy the SGIE_with_softmax_D_300.engine to test2.
and modify the “dstest2_sgie1_config.txt” as below

[property]
gpu-id=0
net-scale-factor=1
model-engine-file=./SGIE_with_softmax_D_300.engine
labelfile-path=./SGIElabels.txt

force-implicit-batch-dim=1
batch-size=8

0=FP32 and 1=INT8 mode

network-mode=1
input-object-min-width=64
input-object-min-height=64
process-mode=2
model-color-format=1

gie-unique-id=2
operate-on-gie-id=1
operate-on-class-ids=0
is-classifier=1
output-blob-names=predictions/Softmax
classifier-async-mode=1
classifier-threshold=0.51
process-mode=2
#scaling-filter=0
#scaling-compute-hw=0

my problem is: I can not get the result of my SIGE. So I modify the Resnet50, such as delete the softmax, load my trained weight, delete the dynamic axis.
But few times, it can work, the SGIE can get result. It likes a small probability random event.
Even the code and processes are the same, a few times I can get the result of SGIE, but many times I can not.

It makes me crazy. Please kindly help me. Thank you very much

(1) If I download the pre-trained RestNet18 from pytorch automatically, and don’t train it by my dataset, DS SGIE can get the result.
(2) Based on (1), I train NN on my dataset, DS SGIE can get the result.
(3) Go on, I change NN to Resnet50, DS SGIE CAN Not get the result.
(4) I retry ResNet18, it can not work
(5) I reboot the computer, It can not work both ResNet50 and ResNet18.

The code and the process are the same!
what happen??

I tried another way. I follow the step of tao _voc/classification.ipynb and produce a “final_model.etlt”. The Dataset is default set as voc2012, and class number is 20.

I copy “final_model.etlt” to deepstream-test2, and set the SGIE1 loading this engine. But It seems like don’t work.

The Fig.1 is shown the original test-2, the SGIE1 can classify the car color.


After I modify the configure as below:
[property]
gpu-id=0
net-scale-factor=1
offsets=123.67;116.28;103.53

tlt-model-key=ZmFtdW43NmZqOWZkczc5cmZsczFyNGFoODo1MGRlNGE5Ny1mNmYzLTRlZWYtOD**********
tlt-encoded-model=./final_model.etlt
labelfile-path=./SGIElabels.txt

input-dims=3;224;224;0
uff-input-blob-name=input_1

force-implicit-batch-dim=1
batch-size=8

0=FP32 and 1=INT8 mode

network-mode=1
input-object-min-width=10
input-object-min-height=10
process-mode=2
model-color-format=1

gie-unique-id=2
operate-on-gie-id=1
operate-on-class-ids=0
is-classifier=1
output-blob-names=predictions/Softmax
classifier-async-mode=1
classifier-threshold=0.51
process-mode=2
#scaling-filter=0
#scaling-compute-hw=0

I just get the following result

That means the SGIE1 does not work. What is the matter? Could you kindly help me?
Thank you very much.

Have you run your case with TensorRT? How did you verify the trained model?

Thank you for your response. I verified the engine in Tensorrt. The output is normal. It can produce the classification result.

Now I doubt the “net-scale-factor” and “offsets” not match. I am trying. If my guess is wrong please kindly let me know. Thank you .

You need to check by yourself. Maybe this link can give you some clue. DeepStream SDK FAQ - Intelligent Video Analytics / DeepStream SDK - NVIDIA Developer Forums

Dear Fiona.Chen
Thank you for your response. I modify the engine form. This time I directly use ONNX in DS.
I hope to use SGIE to classificat the object class_id = 0; however, I find that there just one class_id=0 object can be recognized by SGIE.
If there some limitation? Please check the following picture.

My configuration is change as below:

[property]
gpu-id=0
net-scale-factor=0.0039215697906911373
offsets=97.6980; 99.9760; 88.8259

labelfile-path=./SGIElabels.txt
onnx-file=./ResNet50_with_softmax_400.onnx
model-engine-file=./aa.engine
batch-size=1
force-implicit-batch-dim=0

0=FP32, 1=INT8, 2=FP16 mode

network-mode=2
input-object-min-width = 5
input-object-min-height = 5

process-mode=2
model-color-format=1
gie-unique-id=3
operate-on-gie-id=1
operate-on-class-ids=0
is-classifier=1
output-blob-names=predictions/Softmax
classifier-async-mode=0
classifier-threshold=0.2

network-type=1
#scaling-filter=0
#scaling-compute-hw=0

Thank you very much.

You have set “operate-on-class-ids=0” in your config file. So SGIE only work on class id 0 objects.

https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvinfer.html

Thank you very much for you so quickly response.
Yes I set the “operate-on-class-ids=0”. However, you can see, just obj_id =19 can be recognized by SGIE. obj_id= 31\32\8… can not be classificated by SGIE.

Thank you very very much

I calculate the high/width ratio of the detected objcet. That the object that can be recognized the high/width near 0.65.
If the ratio is limited?

I test, if height / width of the object is nearly 0.6, the engine can be used.

There is no deepstream limitation, please check the model’s limitation.

Thank you very much. It is my mistake. I am trying another way.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.