bInferDone is false after nvinfer in deepstream_test1.py with pytorch based NN

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU)
T4

• DeepStream Version
5.0

• JetPack Version (valid for Jetson only)
NA

• TensorRT Version
7.0.0.11

• NVIDIA GPU Driver Version (valid for GPU only)
440.64.00

• Issue Type( questions, new requirements, bugs)
I am using deepstream_test1.py. If I run it as-is, it works fine. When I use by own NN, the inference is not happening.

How I know that inference is not happening?

  • The labels are not being printed
  • print(“bInferDone={}”.format(frame_meta.bInferDone)) in osd_sink_pad_buffer_probe() – this is printing 0 (while it prints 1 with the caffe-resnet).

What is my NN?
It is a simple variant of Resnet created in pytorch using the below code.

model = resnet18(pretrained=True)
model.fc = torch.nn.Linear(512, 4)
model = torch.nn.Sequential(model, torch.nn.Softmax(1))

Then I converted it to TRT format using torch2trt, ran the generated TRT model on one image and verified that the output/labels are fine. Then I saved the engine part of the TRT model into a file and set that as the engine file in deepstream_test1.py config file. Since my NN does not output bboxes, I have changed network-type to 1.

What is my config file?

[property]
gpu-id=0
net-scale-factor=0.0039215697906911373
model-engine-file=../../../../samples/models/pytorch-resnet-to-trt.engine
force-implicit-batch-dim=1
batch-size=1
network-mode=0
network-type=1
num-detected-classes=4
interval=0
gie-unique-id=1
output-blob-names=output_0

[class-attrs-all]
pre-cluster-threshold=0.2
eps=0.2
group-threshold=1

My guess is that the output layer of my network is not amenable to be used by nvinfer OR the engine file I saved is incorrect in some way. There are no error messages or warnings.

• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

3

How about output-blob-names=predictions/Softmax?

4

Using output-blob-names=predictions/Softmax throws the error below.

Could not find output layer 'predictions/Softmax' in engine

I am guessing that is because of the torch2trt not preserving the layer names. During startup the app prints the following:-

INFO: …/nvdsinfer/nvdsinfer_model_builder.cpp:685 [Implicit Engine Info]: layers num: 2
0 INPUT kFLOAT input_0 3x224x224
1 OUTPUT kFLOAT output_0 4

5

then should keep output_0,
documentation about how to use custom model,
https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_using_custom_model.html

6

Thanks, this seems very helpful and I will go through it.

But is there a way to find out via logs/counters why the inference is not happening? That way I know I am not making any other mistakes and I can focus on the documentation you sent. Among the customisations, the first one Custom Output Parsing seems most relevant and I will try that first.

7

I tried out a few things.

parse-classifier-func-name=JUNK
custom-lib-path=/opt/nvidia/deepstream/deepstream-5.0/sources/libs/nvdsinfer_customparser/libnvds_infercustomparser.so

I had expected this to fail but it did not have any impact.

parse-classifier-func-name=NvDsInferClassiferParseCustomSoftmax
custom-lib-path=/opt/nvidia/deepstream/deepstream-5.0/sources/libs/nvdsinfer_customparser/libnvds_infercustomparser.so

I had modified the code of NvDsInferClassiferParseCustomSoftmax by adding an assert(0), and then running make to generate the .so, but the code never got triggered when I ran my pipeline. I tried adding an fprintf(stderr, ...) and those messages did not get printed. But I could see that, in some cases, the class index was -1 and the python app failed to process the metadata of those frames only. Also bInferDone was 0 for all frames.

What is the simplest way to make sure that the NvDsInferClassiferParseCustomSoftmax is really being invoked? Is there any way to log or add some counters or fail in a sure shot way from within this function so that I can its effects also see if my modifications work?

8

You can add your print in sources/libs/nvdsinfer/nvdsinfer_context_impl_output_parsing.cpp
ClassifyPostprocessor::fillClassificationOutput
to check if it will be invoked if you use custom classifier.
also for output parsing, you also can refer to SSD parser,

9

Thanks a lot.

I was finally able to use the custom classifier function and make sure that it is being invoked. Also I was able to use the NN output and print the custom labels that I had using the ClassifierMeta and LabelInfo.