Custom Yolov8n-face and FER Model Integration into Deepstream

i modify the code to add this in file:gstnvinfer_meta_utils.cpp:
g_print(“[attach_metadata_classifier] Classifier metadata attached for object ID=%d\n”, object_meta->object_id);

and then i modify this file: nvdsinfer_context_impl_output_parsing.cpp:

std::cout << "[parseAttributesFromSoftmaxLayers] m_ClassifierThreshold: "
<< m_ClassifierThreshold << std::endl;
std::cout << "[parseAttributesFromSoftmaxLayers] attrFound: "
<< (attrFound ? “true” : “false”) << std::endl;
std::cout << "[parseAttributesFromSoftmaxLayers] attrString: "
<< attrString << std::endl;
std::cout << "[parseAttributesFromSoftmaxLayers] Final Attribute String: "
<< attrString << std::endl;
and finally i make clean these files and then make these files, and ran the pipeline code and these are the results:

[parseAttributesFromSoftmaxLayers] Debug: Layer index = 0, numClasses = 7
Class 0 => Probability = 0.0977492
Class 1 => Probability = 0.0181941
Class 2 => Probability = 0.130716
Class 3 => Probability = 0.0399927
Class 4 => Probability = 0.240364
Class 5 => Probability = 0.144125
Class 6 => Probability = 0.32886
[Tracker Probe] Object ID=0, class_id=0
[Tracker Probe] No classifier meta for ID=0
[Tracker Probe] Object ID=0, class_id=0
[Tracker Probe] No classifier meta for ID=0
[parseAttributesFromSoftmaxLayers] Debug: Layer index = 0, numClasses = 7
Class 0 => Probability = 0.105825
Class 1 => Probability = 0.0248201
Class 2 => Probability = 0.177451
Class 3 => Probability = 0.0921164
Class 4 => Probability = 0.21939
Class 5 => Probability = 0.132402
Class 6 => Probability = 0.247995
[parseAttributesFromSoftmaxLayers] Debug: Layer index = 0, numClasses = 7
Class 0 => Probability = 0.104213
Class 1 => Probability = 0.0377113
Class 2 => Probability = 0.125215
Class 3 => Probability = 0.107298
Class 4 => Probability = 0.19396
Class 5 => Probability = 0.126459
Class 6 => Probability = 0.305143
[SGIE Probe] Object ID=0, class_id=0
[SGIE Probe] No classifier meta for object ID=0
[SGIE Probe] Current display_text for object ID=0: face 0
[Tracker Probe] Object ID=0, class_id=0
[Tracker Probe] No classifier meta for ID=0
[parseAttributesFromSoftmaxLayers] Debug: Layer index = 0, numClasses = 7
Class 0 => Probability = 0.0881634
Class 1 => Probability = 0.0595162
Class 2 => Probability = 0.154713
Class 3 => Probability = 0.0769069
Class 4 => Probability = 0.201242
Class 5 => Probability = 0.122912
Class 6 => Probability = 0.296547

please replace the /opt/nvidia/deepstream/deepstream/lib/libnvds_infer.so and /opt/nvidia/deepstream/deepstream/lib/gst-plugins/libnvdsgst_infer.so with the new so. importantly please backup the original so lib.

1 Like

aren’t they replace automatically when i run the command make clean and then make them again?

when i modify the .cpp file, i make clean and the compile them using make command.

please refer to the makefile. make install will copy the so. please backup the original so.

sorry, i have mistakenly delete those files, now how can i recover them.

please download deepstream_sdk_v7.1.0_x86_64.tbz2, then copy the so after unzip.

1 Like

alright! i place those files back, and now im getting this output:
[Tracker Probe] Object ID=0, class_id=0
[Tracker Probe] No classifier meta for ID=0
[SGIE Probe] Object ID=0, class_id=0
[SGIE Probe] No classifier meta for object ID=0
[SGIE Probe] Current display_text for object ID=0: face 0
[Tracker Probe] Object ID=0, class_id=0
[Tracker Probe] No classifier meta for ID=0
[Tracker Probe] Object ID=0, class_id=0
[Tracker Probe] No classifier meta for ID=0
[SGIE Probe] Object ID=0, class_id=0
[SGIE Probe] No classifier meta for object ID=0
[SGIE Probe] Current display_text for object ID=0: face 0
[SGIE Probe] Object ID=0, class_id=0
[SGIE Probe] No classifier meta for object ID=0
[SGIE Probe] Current display_text for object ID=0: face 0
[Tracker Probe] Object ID=0, class_id=0
[Tracker Probe] No classifier meta for ID=0
[SGIE Probe] Object ID=0, class_id=0
[SGIE Probe] No classifier meta for object ID=0
[SGIE Probe] Current display_text for object ID=0: face 0
[Tracker Probe] Object ID=0, class_id=0
[Tracker Probe] No classifier meta for ID=0
[SGIE Probe] Object ID=0, class_id=0
[SGIE Probe] No classifier meta for object ID=0
[SGIE Probe] Current display_text for object ID=0: face 0
[Tracker Probe] Object ID=0, class_id=0
[Tracker Probe] No classifier meta for ID=0

im confused, im new to deepstream and not sure where is the problem here, im trying my best, kindly guide step by step.

i replace these new .so files:

root@AAM-LAPTOP-027:/opt/nvidia/deepstream/deepstream-7.1/sources/libs/nvdsinfer# cp libnvds_infer.so /opt/nvidia/deepstream/deepstream-7.1/lib

root@AAM-LAPTOP-027:/opt/nvidia/deepstream/deepstream-7.1/sources/gst-plugins/gst-nvinfer# cp libnvdsgst_infer.so /opt/nvidia/deepstream/deepstream-7.1/lib/gst-plugins

@fanzh after doing the chnages now im getting this output:

[parseAttributesFromSoftmaxLayers] attrFound: true
[parseAttributesFromSoftmaxLayers] attrString:
[parseAttributesFromSoftmaxLayers] Final Attribute String:
[SGIE Probe] Object ID=0, class_id=0
[SGIE Probe] No classifier meta for object ID=0
[SGIE Probe] Current display_text for object ID=0: face 0
[Tracker Probe] Object ID=0, class_id=0
[Tracker Probe] No classifier meta for ID=0
[parseAttributesFromSoftmaxLayers] m_ClassifierThreshold: 0.1
[parseAttributesFromSoftmaxLayers] Debug: Layer index = 0, numClasses = 7
Class 0 => Probability = 0.0204926
Class 1 => Probability = 0.00276932
Class 2 => Probability = 0.520572
Class 3 => Probability = 0.0613143
Class 4 => Probability = 0.350521
Class 5 => Probability = 0.0218569
Class 6 => Probability = 0.0224738
[parseAttributesFromSoftmaxLayers] attrFound: true
[parseAttributesFromSoftmaxLayers] attrString:
[parseAttributesFromSoftmaxLayers] Final Attribute String:
[SGIE Probe] Object ID=0, class_id=0
[SGIE Probe] No classifier meta for object ID=0
[SGIE Probe] Current display_text for object ID=0: face 0
[Tracker Probe] Object ID=0, class_id=0
[Tracker Probe] No classifier meta for ID=0

From the log, there is an abnormality. why is attrString empty while attrFound is true? please print m_Labels.size() ,tr.attributeLabel to narrow down. could you share the content of emotion_labels.txt?

1 Like

im trying to modify this: print m_Labels.size()

in the mean time here is my emotion_labels.txt
Angry
Disgust
Fear
Happy
Sad
Surprise
Neutral

@fanzh after modifying with your guidance, now im getting this output here:
[Debug] Layer index = 0, numClasses = 7
Class 0 => Probability = 0.0249321
Class 1 => Probability = 0.00453382
Class 2 => Probability = 0.464273
[Debug] Updated maxProbability for Class 2 => 0.464273
Class 3 => Probability = 0.0679257
Class 4 => Probability = 0.396726
Class 5 => Probability = 0.0154429
Class 6 => Probability = 0.0261669
[Debug] Attribute Label Not Found for index 0 and value 2
[Debug] Final Attribute String:
[Tracker Probe] Object ID=0, class_id=0
[Tracker Probe] No classifier meta for ID=0
[Tracker Probe] Object ID=0, class_id=0
[Tracker Probe] No classifier meta for ID=0
[Debug] m_ClassifierThreshold: 0.1
[Debug] Number of attributes (output layers): 1
[Debug] Layer index = 0, numClasses = 7
Class 0 => Probability = 0.0249355
Class 1 => Probability = 0.00478933
Class 2 => Probability = 0.467065
[Debug] Updated maxProbability for Class 2 => 0.467065
Class 3 => Probability = 0.0660178
Class 4 => Probability = 0.395619
Class 5 => Probability = 0.0157959
Class 6 => Probability = 0.0257773

i did some more debugging here are the logs:
[Debug] Attribute Label Not Found for index 0 and value 2
[Debug] Appended Attribute Label to attrString: Unknown
[Debug] Final Attribute String: Unknown
[attach_metadata_classifier] Display Text: Unknown
[attach_metadata_classifier] Classifier metadata attached for object ID=18446744073709551615
[SGIE Probe] Object ID=18446744073709551615, class_id=-1
[SGIE Probe] Object ID=0, class_id=0
[SGIE Probe] No classifier meta for object ID=0
[SGIE Probe] Current display_text for object ID=0: face 0
[Tracker Probe] Object ID=0, class_id=0
[Tracker Probe] No classifier meta for ID=0
[Debug] m_ClassifierThreshold: 0.1
[Debug] m_Labels structure:
Index 0 (size 1): Angry
Index 1 (size 1): Disgust
Index 2 (size 1): Fear
Index 3 (size 1): Happy
Index 4 (size 1): Sad
Index 5 (size 1): Surprise
Index 6 (size 1): Neutral
[Debug] Number of attributes (output layers): 1
[Debug] Layer index = 0, numClasses = 7
Class 0 => Probability = 0.0175451
Class 1 => Probability = 0.00174571
Class 2 => Probability = 0.534733
[Debug] Updated maxProbability for Class 2 => 0.534733
Class 3 => Probability = 0.0580574
Class 4 => Probability = 0.343904
Class 5 => Probability = 0.02219
Class 6 => Probability = 0.0218246
[Debug] Attribute Label Not Found for index 0 and value 2
[Debug] Appended Attribute Label to attrString: Unknown
[Debug] Final Attribute String: Unknown
[attach_metadata_classifier] Display Text: Unknown
[attach_metadata_classifier] Classifier metadata attached for object ID=18446744073709551615
[SGIE Probe] Object ID=18446744073709551615, class_id=-1
[SGIE Probe] Object ID=0, class_id=0
[SGIE Probe] No classifier meta for object ID=0
[SGIE Probe] Current display_text for object ID=0: face 0
[Tracker Probe] Object ID=0, class_id=0
[Tracker Probe] No classifier meta for ID=0

Attribute Label Not Found for index 0 and value 2: This indicates that the code is trying to access m_Labels[0][2], but m_Labels[0] contains only one element (Angry), making value 2 out of bounds.

It seems m_Labels is structured as a flat mapping for each class index (one label per class), but the code treats it as a 2D array where each index may have multiple labels.

to address this issue: I modify the code to correctly fetch labels based on the attr.attributeValue directly, as m_Labels[attr.attributeValue].

so now im facing two main major issues here:

  1. the emotion label is on the whole screen, like it should only cover the face detection bbox and label it on there.

  2. im not sure why but in the offline mode the results were good but now in the integration into the deepstream the results are not good, like very bad. even though i have mention all of the preprocessing stuff in the config, but still the results are very bad.

here is the image for reference:

as you can see at the top left, but i want the label to represent it self on the bbox, because when will i have two faces in the video it will overlap.

kindly guide here @fanzh

@fanzh it seems like the second model inference is not happening on the pgie detected bbox, but it is happening onto the whole frame and maybe thats why the results are not accurate,
here is my config which clearly demonstrates that to inference only on

Operate on PGIE’s detected objects (gie-unique-id=1)

operate-on-gie-id=1
operate-on-class-ids=0

now i dont why this issue is happening, kindly guide me im stuck here @fanzh

config file:

[property]
gpu-id=0
net-scale-factor=0.003921568
model-engine-file=emotion_classifier_transposed.onnx_b1_gpu0_fp16.engine
labelfile-path=emotion_labels.txt

batch-size=1
num-detected-classes=7

Disable minimum object size constraints

input-object-min-width=64
input-object-min-height=64

output-blob-names=output_0
network-mode=2
process-mode=1
model-color-format=2
gpu-id=0
gie-unique-id=2

Operate on PGIE’s detected objects (gie-unique-id=1)

operate-on-gie-id=1
operate-on-class-ids=0

Mark as a classifier

is-classifier=1

Synchronous classification

classifier-async-mode=1

Probability threshold for classification

classifier-threshold=0.1

Enable scaling

scaling-filter=0
scaling-compute-hw=0

this is the abnormal logs. please split classification label with “;” in emotion_labels.txt. please refer to \deepstream_sdk_v7.1.0_x86_64\opt\nvidia\deepstream\deepstream\samples\models\Secondary_VehicleMake\labels.txt.

1 Like

@fanzh please see the latest update i have published

please set process-mode=2, which mean processing the bbox got in pgie. then please check the issue in my last comment. why Attribute Label is not found, attrString is Unknown and Display Text is unknown.

1 Like

i have already change the logic to read the labels as i have already mention previous, now the main problem is that i think the sgie is happening onto the whole frame that’s why it is not giving good results, it should only perform the inference onto the detected faces but it is inferencing onto whoel frame, as seen in th eimage above the emotion able is also on the whole frame where as it should only be on the bbox of face detection.

and i try to change the process mode = 2 and the window got stuck with errors:

Traceback (most recent call last):
File “/opt/nvidia/deepstream/deepstream-7.1/sources/DeepStream-Yolo-Face/deepstream.py”, line 166, in sgie_src_pad_buffer_probe
classification_info_list = classifier_meta.class_info_list
AttributeError: ‘pyds.NvDsClassifierMeta’ object has no attribute ‘class_info_list’. Did you mean: ‘label_info_list’?
nvstreammux: Successfully handled EOS for source_id=0
[Tracker Probe] Object ID=0, class_id=0, left=811.5484619140625, top=229.41973876953125, width=337.8937683105469, height=459.11907958984375
[attach_metadata_classifier] Display Text: face 0 Neutral
[attach_metadata_classifier] Classifier metadata attached for object ID=0
[SGIE Probe] Processing frame number: 463
[SGIE Probe] Found classifier meta for object ID: 0
Traceback (most recent call last):
File “/opt/nvidia/deepstream/deepstream-7.1/sources/DeepStream-Yolo-Face/deepstream.py”, line 166, in sgie_src_pad_buffer_probe
classification_info_list = classifier_meta.class_info_list
AttributeError: ‘pyds.NvDsClassifierMeta’ object has no attribute ‘class_info_list’. Did you mean: ‘label_info_list’?
DEBUG: EOS
Pipeline stopped.

here is my config of sgie:

[property]
gpu-id=0
net-scale-factor=0.003921568
model-engine-file=emotion_classifier_transposed.onnx_b1_gpu0_fp16.engine
labelfile-path=emotion_labels.txt

batch-size=1
num-detected-classes=7

Disable minimum object size constraints

input-object-min-width=64
input-object-min-height=64

output-blob-names=output_0
network-mode=2
process-mode=2
model-color-format=2
gpu-id=0
gie-unique-id=2

Operate on PGIE’s detected objects (gie-unique-id=1)

operate-on-gie-id=1
operate-on-class-ids=0

Mark as a classifier

is-classifier=1

Synchronous classification

classifier-async-mode=1

Probability threshold for classification

classifier-threshold=0.1

Enable scaling

scaling-filter=0
scaling-compute-hw=0

Great! this time Classifier meta is not null. please fix the issue ‘pyds.NvDsClassifierMeta’ object has no attribute ‘class_info_list’. Did you mean: ‘label_info_list’.

1 Like