Running customized poseclassification-net in deepstream

[edit_or]

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) Nvidia Orin AGX 64B
• DeepStream Version 7.0
• JetPack Version (valid for Jetson only) Version: 6.0+b106

Running customized poseclassification-net in deepstream using Orin AGX.
I have two issues.
First thing is slow speed in display. The display is about 30 sec delay of actual outside movement.
Second thing is that display label is not accurate.
I have trained poseclassificationnet with owned dataset with about 1000 short videos.
80% for trainin, 10% for validation and 10% for test.
Test results are good.

In actual test using cctv, the labelled output are wrong. Trained 4 classes for poseclassification like sit, stand, walk, one activitiy.
But display always show sit.
All config files are here.

[fanzh]

1. please refer to this link for performance improvement.
2. Is the model retrained based on poseclassificationnet? are you testing the ample deepstream-pose-classification?
3. about “Test results are good.”, do you mean using deepstream-pose-classification the application can give correct labels of trained 4 classes? if no, please make sure the application can recognize that 10% test videos correctly.

[edit_or]

1. I trained with TAO’s poseclassification net and tested with Tao’s poseclassification net.
2. Yes the model is retrained with Tao’s poseclassification net for four customized classes.
3. No. Test results are good meant evaluated with trained model in Tao’s running inference (not in deepstream). 10% test data has good accuracy in evaluation.

[edit_or]

How are activities labelling displayed on frames?
How do we configure activities labels? I can’t find in config files.
My activities are sitting, standing, walking, self-hitting. But only sitting is labelled for any actions.

[fanzh]

1. the nvinfer will add classification meta in attach_metadata_classifier of \opt\nvidia\deepstream\deepstream\sources\gst-plugins\gst-nvinfer\gstnvinfer_meta_utils.cpp, then nvosd plugin will display the label.
2. it is hardcoded in this code. you can modify and rebuild the code.
3. is cctv a new database? first please make sure the application can recognize that 10% test videos.

[edit_or]

The original poseclassificationnet has 6 activities.
The model I trained has only 4 activities.
Made changes in parcer as

const char classes_str[4][32] = {
  "selfhit", "walk", "sit", "stand"
};

Now display always has label for index 0, in this case “selfhit”

[fanzh]

Both Tao and deepstream will use NVIDIA TensorRT to do inference. if the preprocessed data is the same, the inference results should be the same. the preprocess code is opensource. Here will prepare the cooridates sequence. you can dump the coordinates to compare the coordinates using in TAO test.

[edit_or]

Thank you. For six activities to four activities, this is the only place to update?

const char classes_str[4][32] = {
  "selfhit", "walk", "sit", "stand"
};

[fanzh]

yes. only postprocess function NvDsParseCustomPoseClassification will use this string.