Yes, if you model needs prepross or postprocess, you should add the function to process it. Maybe you can refer the topic below:
https://forums.developer.nvidia.com/t/getting-0-0-0-0-output-from-tao-multitask-classifier/210302
I am familiar with this post. I have used it previously to develop a batched inferencing tool on a directory filled with images which has worked with this model I have created (and another one as well), but I am not sure if that post shows me how to use multi-task classification with deepstream directly. I imagine there must be some standard bbox parsing function for multitask classification in deepstream/deepstream_tao_apps?
For some further debugging, I tried to run my two models through the standard deepstream_tao_apps apps and I did replicated the same bounding box errors with the multi-task classifier app.
in ~/deepstream_tao_apps-master/apps/tao_detection, ./ds-tao-detection -c ~/deepstream-app/multitask/config_infer_primary.txt -i file:///home/nvidia/Videos/example.mp4 -d ran without a hitch, but I suppose this is expected since the object detector has already been confirmed to work.
whereas:
in ~/deepstream_tao_apps-master/apps/tao_classifier, ./ds-tao-classifier -c ~/deepstream-app/multitask/config_infer_secondary_MT_people.txt -i file:///home/nvidia/Videos/example.mp4 -d failed immediately with the exact same error as reported in the original post. I would imagine that this multi-task classifier app that is packaged with deepstream_tao_apps should not be running into a bounding box issue unless the model was bad, something in my configuration was incorrect or there is some reference to a library with that function that is missing and needs to be linked to my config (similar to what is required to run a yolo_v4 model). I have validated that the model is functional before using the post you have referenced earlier which makes be believe there must be a configuration problem or library reference that i am missing.
The multi-task classification config file packaged with deepstream_tao_apps is as follows:
[property]
gpu-id=0
net-scale-factor=1.0
offsets=103.939;116.779;123.68
model-color-format=1
labelfile-path=multi_task_labels.txt
tlt-encoded-model=../../models/multi_task/abc.etlt
tlt-model-key=nvidia_tlt
model-engine-file=../../models/multi_task/abc.etlt_b1_gpu0_fp16.engine
infer-dims=3;80;60
uff-input-blob-name=input_1
batch-size=1
## 0=FP32, 1=INT8, 2=FP16 mode
network-mode=2
interval=0
gie-unique-id=1
network-type=1
scaling-filter=1
scaling-compute-hw=1
output-blob-names=base_color/Softmax;category/Softmax;season/Softmax
classifier-threshold=0.5
The configuration file that I have is nearly identical, with the exception of the parameters that was in the nvinfer_config.txt that was generated by TAO.
Is there some model that you may have on hand that you have gotten to work for multi-task classification in deepstream? Deepstream_tao_apps reports that it doesn’t have a standard model for multi-task classification that I could use for testing purpose, but if you have one that you have confirmed to work properly, that may narrow down where I may be having an issue with my setup.
At last, There may be some problems about the order of the output layer. You can check it.
In your log:
That is quite interesting. The order of the output-blob-names i am using was a direct output from tao via nvinfer_config.txt as shown in my earlier post. I did try and swap the order to output-blob-names=pose/Softmax;object/Softmax;action/Softmax and ran into the same bounding box error.