Challenges in Implementing PoseClassificationNet in DeepStream-6.2

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) - Jetson Xavier NX
• DeepStream Version - 6.2
• JetPack Version (valid for Jetson only) - 5.1.1
• TensorRT Version - 8.5.2.2
• NVIDIA GPU Driver Version (valid for GPU only) - NA
• Issue Type( questions, new requirements, bugs) - questions
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

Trying to implement Pose Classification Net with DeepStream Python. The current Pipeline looks like this-
streammux(multiple streams) → PGIE (PeopleNet) → tracker(NVDCF) → SGIE1 (BodyPose3D Net) ->nvdspreprocess-> SGIE2 (PoseClassificationNet) ->Tiler → nvvidconv(YUV->RGBA) ->OSD ->nvvidconv2(RGBA->I420) ->CapsFilter(I420) → Encoder → filesink.

The pipeline works seamlessly for all frames. but there is a warning for the nvdspreprocess function. the .so file being taken from deepstream_app_3daction_recognition sample application based on the forum question posted here . Changed the input layer shape in the config file as required by the PoseClassificationNet model. The warning is as follows:

/dvs/git/dirty/git-master_linux/deepstream/sdk/apps/deepstream/sample_apps/deepstream-3d-action-recognition/custom_sequence_preprocess/sequence_image_process.cpp:499, [INFO: CUSTOM_LIB] 2D custom sequence network shape NSHW[3, 300, 34, 1], reshaped as [N: 3, C: 3, S:100, H: 34, W:1]
/dvs/git/dirty/git-master_linux/deepstream/sdk/apps/deepstream/sample_apps/deepstream-3d-action-recognition/custom_sequence_preprocess/sequence_image_process.cpp:522, [INFO: CUSTOM_LIB] Sequence preprocess buffer manager initialized with stride: 1, subsample: 0
/dvs/git/dirty/git-master_linux/deepstream/sdk/apps/deepstream/sample_apps/deepstream-3d-action-recognition/custom_sequence_preprocess/sequence_image_process.cpp:526, [INFO: CUSTOM_LIB] SequenceImagePreprocess initialized successfully
Using user provided processing height = 192 and processing width = 256

Q1- I am not sure how to proceed to rectify the above warning since the input is being reshaped in a wrong way.
Q2- Although the pipeline runs without terminating, I am not sure if the classification is happening accurately. Kindly advice on how to get the SGIE2 classification(Action class) displayed on the frame.

pipeline.py (19.1 KB)

config_infer_secondary_bodypose3dnet.txt (2.5 KB)
config_infer_secondary_poseclassification.txt (2.3 KB)
config_preprocess_3d_custom.txt (3.2 KB)

2

I used C++, not Python, but I solved it with the same pipeline as presented. The most important thing is that you need to customize and build the library used by nvdspreprocess yourself. You need to allocate GPU memory and prepare 3x300x34 worth of data. Since we have to deal with GPU memory, the CUDA API must be used.

It would be very helpful to set sgie2’s raw-output-file-write to True to see the input and output.

1 Like
4

We are developing the DeepStream Pose Classification Net c++ sample. It may be available in the next release.

The log looks OK.

You need to customize the PoseClassificationNet postprocessing function. The NGC Pose Classification | NVIDIA NGC only output classification logits, you need to do softmax by yourself and label the final output.

5

Thanks Fiona .

I’m still has no clue how to proceed forward and currently stuck with the following issue.

  1. Even Though I am successfully able to run SGIE2 Pose Classification but I am not able to extract the metadata . I have tried to print the Tensor Metadata but that data is of BodyPose3D sgie1 not the PoseClassification sgie2 model. Could you please guide me on how to extract the PoseClassification Metadata?

Thanks

6

I was in the same situation.
After scouring the deepstream official documentation and inside the NvDsBatchMeta structure, I found that the tensors inferred by the preprocessd tensor are stored in the batched user meta list, not in the frame meta list. I can access both output and input tensors by separating the meta types.

7

How did you do the postprocessing with Pose Classification model? The corresponding metadata depends on the postprocessing.

8

Hi Fiona,

I thought of doing the post-processing on the tensor meta data of PoseClassification sgie2 model. In the config I set the network-type =100 and output-tensor-meta = 1 .
When I Tried to implement it, Still I was only able to get the tensor-meta data of BodyPose3D sgie1 model not the PoseClassification sgie2 model.

Then I tried setting the network-type=1 and output-tensor-meta=1 for PoseClassification Model to check if I can able to extract the metadata. I was successful in running the pipeline but still not able to extract the metadata.

Could you please let me know Is there any other way to implement the postprocessing in-order to get the PoseClassification Labels as an output??

Thanks a lot for your time. Looking forward to your reply.

9

Hi Fiona, After going through Forums for postprocessing in DeepStream for classifier models.

I have added two lines in config file for PoseClassification sgie-2 model. Here are the lines :-
parse-classifier-func-name=NvDsInferClassiferParseCustomSoftmax
custom-lib-path=/opt/nvidia/deepstream/deepstream/sources/libs/nvdsinfer_customparser/libnvds_infercustomparser.so

After adding these two lines in the config file , I am able to run the pipeline successfully but Still I’m not clear how to get the PoseClassification Output.

Could you please guide me what to do next?

Thanks

10

What did you do in NvDsInferClassiferParseCustomSoftmax？

11

I have just changed the names of the labels. Rest Everything is same.

Is anything else has to be changed?

image
image895×353 11.4 KB

12

There is a sample of customized classifier output parsing callback function deepstream_lpr_app/nvinfer_custom_lpr_parser/nvinfer_custom_lpr_parser.cpp at master · NVIDIA-AI-IOT/deepstream_lpr_app · GitHub. Please make sure the “attributeLabel” in struct NvDsInferAttribute has been given correct string. And you can debug with your “NvDsInferClassifierParseCustomSoftmax” function to check whether the results have been generated correctly.

13

Hi Fiona, Thanks for the suggestion.

When I am trying to run the pipeline , Here are the logs what I am getting for the PoseClassification sgie2 model.
As we can see from the logs Output is fc_pred with a value of 6 ( There are 6 classes in my labels file).

image
image1892×126 10.7 KB

Do you think it’s correct? If so then how did I extract my metadata ?

I tried obj_meta.classifier_meta_list to get the PoseClassification data but it is always None.

14

Did you notice the classifier_meta_list below in the metadata structure?

batch_meta
 └ frame_meta_list
  └ obj_meta_list
   └ classifier_meta_list

Check the user_meta_list below

batch_meta
 └ batch_user_meta_list

Find that the base_meta.meta_type of that user_meta is NVDSINFER_TENSOR_OUTPUT_META.

15

Hi ,

I also tried using l_user_batch = batch_meta.batch_user_meta_list but it is also None.
image

However if I use l_user = obj_meta.obj_user_meta_list I can get the user metadata but that metadata is of BodyPose3dNet i.e. ( 34 *4 ) key points .

I am attaching the probe function code and output what I am getting on the terminal.

def tiler_sink_pad_buffer_probe(pad, info, u_data):

frame_number=0
num_rects=0
gst_buffer = info.get_buffer()

if not gst_buffer:
    print("Unable to get GstBuffer ")
    return

batch_meta = pyds.gst_buffer_get_nvds_batch_meta(hash(gst_buffer))

l_user_batch = batch_meta.batch_user_meta_list
print('l_user_batch', l_user_batch)
l_frame = batch_meta.frame_meta_list
while l_frame is not None:
    try:
        frame_meta = pyds.NvDsFrameMeta.cast(l_frame.data)
    except StopIteration:
        break

    obj_counter ={
        PGIE_CLASS_ID_PERSON:0
     }
    frame_number = frame_meta.frame_num
    num_rects = frame_meta.num_obj_meta
    l_obj = frame_meta.obj_meta_list
    l_user_frame = frame_meta.frame_user_meta_list
    print('l_user_frame', l_user_frame)
    
    while l_obj is not None:
        try:
            obj_meta = pyds.NvDsObjectMeta.cast(l_obj.data)
        except StopIteration:
            break
        obj_counter[obj_meta.class_id] +=1
        l_user = obj_meta.obj_user_meta_list
        class_obj = obj_meta.classifier_meta_list
        print('class_obj', class_obj)
        print('l_user:', l_user)
        while l_user is not None:
            try:
                user_meta = pyds.NvDsUserMeta.cast(l_user.data)
            except StopIteration:
                break
        
            if (user_meta.base_meta.meta_type !=pyds.NvDsMetaType.NVDSINFER_TENSOR_OUTPUT_META):
                continue
               
            tensor_meta = pyds.NvDsInferTensorMeta.cast(user_meta.user_meta_data)

            '''
            layers_info = []
            for i in range(tensor_meta.num_output_layers):
                layer = pyds.get_nvds_LayerInfo(tensor_meta, i)
                layers_info.append(layer)
                print(f'Layer: {i}, Layer name: {layer.layerName}')
            '''
            network_info = tensor_meta.network_info
            input_shape = (network_info.width, network_info.height)
            pose25d_layer = pyds.get_nvds_LayerInfo(tensor_meta, 0)
            dims = np.trim_zeros(pose25d_layer.inferDims.d, 'b')
            print('dims', dims)
            print("input_shape", input_shape)
            print("Network Input : w=%d, h=%d, c=%d"%(network_info.width, network_info.height, network_info.channels))
            cdata_type = data_type_map[pose25d_layer.dataType]
            ptr = ctypes.cast(pyds.get_ptr(pose25d_layer.buffer), ctypes.POINTER(cdata_type))
            
            pose25d_output = np.ctypeslib.as_array(ptr, shape=dims)

    
            embedding_dims = pose25d_layer.inferDims
            width = embedding_dims.d[2]
            height = embedding_dims.d[1]
            numElements = embedding_dims.d[0]
 
            print("embedding_dims: ", embedding_dims, "width: ", width, "height: ", height, "num_elements: ", numElements)
            print('pose25d', pose25d_output)
            print('pose25d_shaoe', pose25d_output.shape)
            try:
                l_user = l_user.next
            except StopIteration:
                break
        try:
            l_obj = l_obj.next
        except StopIteration:
            break
    print("Frame Number=", frame_number, "Number of Objects=",num_rects, "Person_count=",obj_counter[PGIE_CLASS_ID_PERSON])

    stream_index = "stream{0}".format(frame_meta.pad_index)
    try:
        l_frame = l_frame.next
    except StopIteration:
        break

return Gst.PadProbeReturn.OK

image
image942×870 30.6 KB

16

Can you check the files that are output when the raw-output-file-write property of nvdsinfer(pose classification net) is set to True?
Also, did you set the output-tensor-meta to True?

17

Hi , Yes I have enabled both the raw-output-file-write and output-tensor-meta but it’s still output on the Command terminal remains same as I shared the screenshot in the last reply.

here is the config file for PoseClassification sgie2 model.
[property]
gpu-id=0
net-scale-factor=1

Accuracy mode: mode0; Performance mode: mode1

tlt-model-key=nvidia_tao
tlt-encoded-model=…/models/st-gcn_3dbp_nvidia.etlt
model-engine-file=…/models/st-gcn_3dbp_nvidia.etlt_b1_gpu0_fp16.engine
labelfile-path = pose_classification_labels.txt
batch-size=8

0=FP32, 1=INT8, 2=FP16 mode

network-mode=2
#infer-dims=3;300;34

0=Detection 1=Classifier 2=Segmentation 100=other

network-type=1
is-classifier=1
num-detected-classes=6
interval=0
input-tensor-from-meta=1
gie-unique-id=3
output-blob-names=fc_pred
classifier-threshold=0.2
operate-on-gie-id=2
#operate-on-class-ids=0

Integer 0:NCHW 1:NHWC

network-input-order=0

Enable tensor metadata output

output-tensor-meta=1
raw-output-file-write=1

1-Primary 2-Secondary

process-mode=2

0=RGB 1=BGR 2=GRAY

model-color-format=1
maintain-aspect-ratio=0
symmetric-padding=0
scaling-filter=1
parse-classifier-func-name=NvDsInferClassiferParseCustomSoftmax
custom-lib-path=/opt/nvidia/deepstream/deepstream/sources/libs/nvdsinfer_customparser/libnvds_infercustomparser.so

18

How did you process “fc_pred” in your “NvDsInferClassifierParseCustomSoftmax” function?

19

Note that raw-output-file-write cannot be set in the config file, it exists as a property of nvdsinfer’s GObject.
For C, set it as follows.

g_object_set(G_OBJECT(sgie2), 
    "config-file-path", SGIE2_CONFIG_FILE,
    "raw-output-file-write", 1,
    NULL);

In python, it would probably look something like this.

sgie2.set_property("raw-output-file-write", True)
20

Hi Fiona ,

Here is the code for the nvdsinfer_customclassifierparser.cpp

include
include
include “nvdsinfer_custom_impl.h”

/* C-linkage to prevent name-mangling */
extern “C”
bool NvDsInferClassiferParseCustomSoftmax (std::vector const &outputLayersInfo,
NvDsInferNetworkInfo const &networkInfo,
float classifierThreshold,
std::vector &attrList,
std::string &descString);

static std::vector < std::vector< std:: string > > labels { {
“sitting_down”, “getting_up”, “sitting”, “standing”, “walking”, “jumping”} };

extern “C”
bool NvDsInferClassiferParseCustomSoftmax (std::vector const &outputLayersInfo,
NvDsInferNetworkInfo const &networkInfo,
float classifierThreshold,
std::vector &attrList,
std::string &descString)
{
/* Get the number of attributes supported by the classifier. */
unsigned int numAttributes = outputLayersInfo.size();

/* Iterate through all the output coverage layers of the classifier.
*/
for (unsigned int l = 0; l < numAttributes; l++)
{
    /* outputCoverageBuffer for classifiers is usually a softmax layer.
     * The layer is an array of probabilities of the object belonging
     * to each class with each probability being in the range [0,1] and
     * sum all probabilities will be 1.
     */
    NvDsInferDimsCHW dims;

    getDimsCHWFromDims(dims, outputLayersInfo[l].inferDims);
    unsigned int numClasses = dims.c;
    float *outputCoverageBuffer = (float *)outputLayersInfo[l].buffer;
    float maxProbability = 0;
    bool attrFound = false;
    NvDsInferAttribute attr;

    /* Iterate through all the probabilities that the object belongs to
     * each class. Find the maximum probability and the corresponding class
     * which meets the minimum threshold. */
    for (unsigned int c = 0; c < numClasses; c++)
    {
        float probability = outputCoverageBuffer[c];
        if (probability > classifierThreshold
                && probability > maxProbability)
        {
            maxProbability = probability;
            attrFound = true;
            attr.attributeIndex = l;
            attr.attributeValue = c;
            attr.attributeConfidence = probability;
        }
    }
    if (attrFound)
    {
        if (labels.size() > attr.attributeIndex &&
                attr.attributeValue < labels[attr.attributeIndex].size())
            attr.attributeLabel =
                strdup(labels[attr.attributeIndex][attr.attributeValue].c_str());
        else
            attr.attributeLabel = nullptr;
        attrList.push_back(attr);
        if (attr.attributeLabel)
            descString.append(attr.attributeLabel).append(" ");
    }
}

return true;

}

/* Check that the custom function has been defined correctly */
CHECK_CUSTOM_CLASSIFIER_PARSE_FUNC_PROTOTYPE(NvDsInferClassiferParseCustomSoftmax);

21

Hi ,

sgie2.set_property(“raw-output-file-write”, True

I tried but it’s still the output on command terminal remains same, I can’t find any bin files