Custom Yolov8n-face and FER Model Integration into Deepstream

Accelerated Computing Intelligent Video Analytics DeepStream SDK
, , , , , , ,
44

this error came only when i try to run the process mode=2,

i want two things :

  1. inference is on the whole frame not on the detected faces, how can i achieve that?
  2. emotion labels are not on the bbox of faces but on the whole frame because of the inference is happening on the whoel frame, and thats why im getting the bad results as well.

you can check my config above which shows that i have already set the:

Operate on PGIE’s detected objects (gie-unique-id=1)

operate-on-gie-id=1
operate-on-class-ids=0

im not sure why the inference is not running on the detected objects instead its running on the whole frame, which is causing the bad results.

@fanzh kindly guide accordingly.

45
  1. you only need to set process-mode=2. nvinfer plugin is opensource. if process-mode is 2, nvinfer plugin will call gst_nvinfer_process_objects. you can add log to check.
  2. yes, if process-mode=1, attach_metadata_classifier add a new object bbox. from the log “Display Text: face 0 Neutral”, the video should be able to show the classification label, you can comment out “class_info_list = classifier_meta.class_info_list” to check. this code causes the error.
46

after modifying and running the code, im getting this output here:
Debug] m_Labels structure:
Index 0 (size 1): Angry
Index 1 (size 1): Disgust
Index 2 (size 1): Fear
Index 3 (size 1): Happy
Index 4 (size 1): Sad
Index 5 (size 1): Surprise
Index 6 (size 1): Neutral
[Debug] Number of attributes (output layers): 1
[Debug] Layer index = 0, numClasses = 7
Class 0 => Probability = 0.0164723
Class 1 => Probability = 0.00134423
Class 2 => Probability = 0.547622
[Debug] Updated maxProbability for Class 2 => 0.547622
Class 3 => Probability = 0.0404817
Class 4 => Probability = 0.351849
Class 5 => Probability = 0.0227359
Class 6 => Probability = 0.0194946
[Debug] Attribute Label Found: Fear
[Debug] Appended Attribute Label to attrString: Fear
[Debug] Final Attribute String: Fear
[attach_metadata_classifier] Display Text: Fear
[attach_metadata_classifier] Classifier metadata attached for object ID=18446744073709551615
[SGIE Probe] Processing frame number: 28
[SGIE Probe] Skipping invalid object ID: 18446744073709551615
[SGIE Probe] No classifier meta for object ID=0
[Tracker Probe] Object ID=0, class_id=0, left=760.7022705078125, top=246.95399475097656, width=335.0787048339844, height=470.024658203125
[Tracker Probe] Object ID=0, class_id=0, left=760.965576171875, top=246.3533477783203, width=335.30780029296875, height=470.7333984375

with this following output in the display:

image
image1907×1068 74.3 KB

the inference should be on the face bbox but it is on the whole frame which is causing the bad results as well.

@fanzh need your guidance here.

47

@fanzh waiting for your guidance

The logs indicate that classifier metadata is being attached to an invalid object_id=18446744073709551615, while the valid object_id=0 has no classifier metadata attached. This suggests that the SGIE is not correctly associating emotion classification results with the detected face objects.

i think thats the reason the inference is not happening onto the detected face area.

48

attach_metadata_classifier adds classification meta object meta, if sgie, this function will create a object meta first. please print nvinfer->process_full_frame in attach_metadata_classifier, and check if nvds_add_obj_meta_to_frame is called in this function.

1 Like
49

@fanzh here are the logs after modification:
[Debug] Number of attributes (output layers): 1
[Debug] Layer index = 0, numClasses = 7
Class 0 => Probability = 0.0155425
Class 1 => Probability = 0.00151505
Class 2 => Probability = 0.525873
[Debug] Updated maxProbability for Class 2 => 0.525873
Class 3 => Probability = 0.042043
Class 4 => Probability = 0.3729
Class 5 => Probability = 0.021017
Class 6 => Probability = 0.0211096
[Debug] Attribute Label Found: Fear
[Debug] Appended Attribute Label to attrString: Fear
[Debug] Final Attribute String: Fear
[attach_metadata_classifier] nvinfer->process_full_frame: 1
[attach_metadata_classifier] Called nvds_add_obj_meta_to_frame for full frame classification
[attach_metadata_classifier] Display Text: Fear
[attach_metadata_classifier] Classifier metadata attached for object ID=18446744073709551615
[SGIE Probe] Processing frame number: 30
[SGIE Probe] Object ID=18446744073709551615, class_id=-1
[SGIE Probe] Skipping invalid object ID: 18446744073709551615
[SGIE Probe] Object ID=0, class_id=0
[SGIE Probe] No classifier meta for object ID=0
[PGIE Probe] Detected Object ID=18446744073709551615, class_id=0, left=762.3215942382812, top=243.83807373046875, width=336.38995361328125, height=471.16461181640625
[Tracker Probe] Object ID=0, class_id=0, left=761.929931640625, top=245.08506774902344, width=335.9674987792969, height=471.15228271484375
[Debug] m_ClassifierThreshold: 0.1
[Debug] m_Labels structure:
Index 0 (size 1): Angry
Index 1 (size 1): Disgust
Index 2 (size 1): Fear
Index 3 (size 1): Happy
Index 4 (size 1): Sad
Index 5 (size 1): Surprise
Index 6 (size 1): Neutral

this assures that the logs have the following patterns:

  1. attach_metadata_classifier Function:
  • nvinfer->process_full_frame: 1: Indicates that full-frame classification is enabled.
  • Called nvds_add_obj_meta_to_frame for full frame classification: Confirms that metadata is being attached at the frame level.
  • Classifier metadata attached for object ID=18446744073709551615: Metadata is being attached to an object with object_id=-1.
  1. SGIE Probe:
  • Object ID=18446744073709551615, class_id=-1: SGIE detects an object with an invalid object_id and skips it.
  • Object ID=0, class_id=0: Another object with object_id=0 is detected but lacks classifier metadata.
  1. Tracker Probe:
  • Object ID=0, class_id=0, ...: Tracker assigns object_id=0, which is invalid in DeepStream (valid IDs start from 1).
50

well, this is the progress so far now:

image
image1715×988 203 KB
i

i try to disable the process_full_frame by enabling the process_mode=2, so now this is good, but the only issue left here is that the emotion labels are stuck, like the inference happened only at the start, and it is not fluctuating with the movement of face, like its stuck or something, it seems static but not dynamic, it should fluctuate a bit with movement of face.

here are the logs now:

[attach_metadata_classifier] nvinfer->process_full_frame: 0
[attach_metadata_classifier] Display Text: face 0 Neutral
[attach_metadata_classifier] Classifier metadata attached for object ID=0
[SGIE Probe] Processing frame number: 126
[SGIE Probe] Object ID=0, class_id=0
[SGIE Probe] Found classifier meta for object ID: 0
[SGIE Probe] Face ID 0: Neutral (0.39)
[Tracker Probe] Object ID=0, class_id=0, left=766.7105102539062, top=260.00836181640625, width=336.23980712890625, height=464.06378173828125
[PGIE Probe] Detected Object ID=18446744073709551615, class_id=0, left=767.3170166015625, top=260.01165771484375, width=336.41693115234375, height=461.91131591796875
i know this is almost done, kindly help what should i do now to make accurate.

@fanzh guide me

51

@fanzh waiting for your guidance

52

could you share video to elaborate on this?

when bbox moved, does classification result still appear? you can add log in attach_metadata_classifier to check if this function is called many time.

1 Like
53

@fanzh yes im sharing the video with you.

when bbox moved, does classification result still appear?
yes, it is consistent, as you can see this video here.

what should i do now, its like the inference is not running continuously.

54

@fanzh waiting for you reply

55

why classification probability is always 0.39? please print maxProbability, attr.attributeIndex , attr.attributeValue after attrFound is 1 in parseAttributesFromSoftmaxLayers.

1 Like
56

@fanzh
here are the logs of the first 6 frames:
[SGIE Probe] Processing frame number: 1
[Tracker Probe] Object ID=0, class_id=0, left=891.1195678710938, top=171.78897094726562, width=238.37091064453125, height=315.4529724121094
[PGIE Probe] Detected Object ID=18446744073709551615, class_id=0, left=897.8289184570312, top=171.30056762695312, width=236.43814086914062, height=313.49359130859375
[SGIE Probe] Processing frame number: 2
[SGIE Probe] Object ID=0, class_id=0
[SGIE Probe] No classifier meta for object ID=0
[Debug] m_ClassifierThreshold: 0.1
[Debug] m_Labels structure:
Index 0 (size 1): Angry
Index 1 (size 1): Disgust
Index 2 (size 1): Fear
Index 3 (size 1): Happy
Index 4 (size 1): Sad
Index 5 (size 1): Surprise
Index 6 (size 1): Neutral
[Debug] Number of attributes (output layers): 1
[Debug] Layer index = 0, numClasses = 7
Class 0 => Probability = 0.018792
Class 1 => Probability = 0.000389256
Class 2 => Probability = 0.0397342
Class 3 => Probability = 0.822814
[Debug] Updated maxProbability for Class 3 => 0.822814
[Debug] After attrFound=true: maxProbability=0.822814, attr.attributeIndex=0, attr.attributeValue=3
Class 4 => Probability = 0.043565
Class 5 => Probability = 0.000743018
Class 6 => Probability = 0.073962
[Debug] Attribute Label Found: Happy
[Debug] Appended Attribute Label to attrString: Happy
[Debug] Final Attribute String: Happy
[PGIE Probe] Detected Object ID=18446744073709551615, class_id=0, left=900.9107666015625, top=171.62457275390625, width=237.74990844726562, height=313.4322509765625
[Tracker Probe] Object ID=0, class_id=0, left=893.826416015625, top=171.9442901611328, width=238.27651977539062, height=315.1025390625
[attach_metadata_classifier] nvinfer->process_full_frame: 0
[attach_metadata_classifier] Display Text: face 0 Happy
[attach_metadata_classifier] Classifier metadata attached for object ID=0
[SGIE Probe] Processing frame number: 3
[SGIE Probe] Object ID=0, class_id=0
[SGIE Probe] Found classifier meta for object ID: 0
[SGIE Probe] Face ID 0: Happy (0.82)
[PGIE Probe] Detected Object ID=18446744073709551615, class_id=0, left=905.2012939453125, top=172.42538452148438, width=235.33200073242188, height=311.8395080566406
[Tracker Probe] Object ID=0, class_id=0, left=896.8917236328125, top=172.0601806640625, width=237.5614013671875, height=314.4766540527344
[attach_metadata_classifier] nvinfer->process_full_frame: 0
[attach_metadata_classifier] Display Text: face 0 Happy
[attach_metadata_classifier] Classifier metadata attached for object ID=0
[SGIE Probe] Processing frame number: 4
[SGIE Probe] Object ID=0, class_id=0
[SGIE Probe] Found classifier meta for object ID: 0
[SGIE Probe] Face ID 0: Happy (0.82)
[PGIE Probe] Detected Object ID=18446744073709551615, class_id=0, left=907.9075927734375, top=174.13192749023438, width=235.07574462890625, height=309.75933837890625
[Tracker Probe] Object ID=0, class_id=0, left=900.0426635742188, top=172.52395629882812, width=237.63427734375, height=314.0727844238281
[attach_metadata_classifier] nvinfer->process_full_frame: 0
[attach_metadata_classifier] Display Text: face 0 Happy
[attach_metadata_classifier] Classifier metadata attached for object ID=0
[SGIE Probe] Processing frame number: 5
[SGIE Probe] Object ID=0, class_id=0
[SGIE Probe] Found classifier meta for object ID: 0
[SGIE Probe] Face ID 0: Happy (0.82)
[PGIE Probe] Detected Object ID=18446744073709551615, class_id=0, left=911.4287109375, top=175.189453125, width=235.1627197265625, height=307.842041015625
[Tracker Probe] Object ID=0, class_id=0, left=903.9085693359375, top=173.26666259765625, width=236.7451171875, height=313.21026611328125
[attach_metadata_classifier] nvinfer->process_full_frame: 0
[attach_metadata_classifier] Display Text: face 0 Happy
[attach_metadata_classifier] Classifier metadata attached for object ID=0

57

now what to do next here, to make sure the inference of emotions update itself continuously.
@fanzh kindly guide here

58

@fanzh waiting

59

please set classifier-async-mode=0.

60

i already did that but it is not working, it works for only 1st frame, but after that the emotion labels and their probs remain the same or static till the end.

@fanzh what can be the issue an dhow to resolve it, please reply ASAP.

61

here are the logs after the “classifier-async-mode=0”

[SGIE Probe] Processing frame number: 0
[Tracker Probe] Object ID=0, class_id=0, left=750.8934936523438, top=240.15335083007812, width=333.21234130859375, height=437.3307800292969
[Tracker Probe] Object ID=0, class_id=0, left=751.431640625, top=240.21600341796875, width=333.31170654296875, height=437.4458923339844
[SGIE Probe] Processing frame number: 1
[Debug] m_ClassifierThreshold: 0.1
[Debug] m_Labels structure:
Index 0 (size 1): Angry
Index 1 (size 1): Disgust
Index 2 (size 1): Fear
Index 3 (size 1): Happy
Index 4 (size 1): Sad
Index 5 (size 1): Surprise
Index 6 (size 1): Neutral
[Debug] Number of attributes (output layers): 1
[Debug] Layer index = 0, numClasses = 7
Class 0 => Probability = 0.00261071
Class 1 => Probability = 7.94431e-06
Class 2 => Probability = 0.197126
[Debug] Updated maxProbability for Class 2 => 0.197126
[Debug] After attrFound=true: maxProbability=0.197126, attr.attributeIndex=0, attr.attributeValue=2
Class 3 => Probability = 0.28822
[Debug] Updated maxProbability for Class 3 => 0.28822
[Debug] After attrFound=true: maxProbability=0.28822, attr.attributeIndex=0, attr.attributeValue=3
Class 4 => Probability = 0.0646253
Class 5 => Probability = 0.0622103
Class 6 => Probability = 0.3852
[Debug] Updated maxProbability for Class 6 => 0.3852
[Debug] After attrFound=true: maxProbability=0.3852, attr.attributeIndex=0, attr.attributeValue=6
[Debug] Attribute Label Found: Neutral
[Debug] Appended Attribute Label to attrString: Neutral
[Debug] Final Attribute String: Neutral
[attach_metadata_classifier] nvinfer->process_full_frame: 0
[attach_metadata_classifier] Display Text: face 0 Neutral
[attach_metadata_classifier] Classifier metadata attached for object ID=0
[SGIE Probe] Processing frame number: 2
[SGIE Probe] Found classifier meta for object ID: 0
[SGIE Probe] Face ID 0: Neutral (0.39)
[Tracker Probe] Object ID=0, class_id=0, left=751.1373291015625, top=240.59255981445312, width=333.4554443359375, height=437.25018310546875
[attach_metadata_classifier] nvinfer->process_full_frame: 0
[attach_metadata_classifier] Display Text: face 0 Neutral
[attach_metadata_classifier] Classifier metadata attached for object ID=0
[SGIE Probe] Processing frame number: 3
[SGIE Probe] Found classifier meta for object ID: 0
[SGIE Probe] Face ID 0: Neutral (0.39)
[Tracker Probe] Object ID=0, class_id=0, left=751.107421875, top=240.57855224609375, width=333.3847961425781, height=437.1232604980469
[attach_metadata_classifier] nvinfer->process_full_frame: 0
[attach_metadata_classifier] Display Text: face 0 Neutral
[attach_metadata_classifier] Classifier metadata attached for object ID=0
[SGIE Probe] Processing frame number: 4
[SGIE Probe] Found classifier meta for object ID: 0
[SGIE Probe] Face ID 0: Neutral (0.39)
[Tracker Probe] Object ID=0, class_id=0, left=751.4085083007812, top=240.62271118164062, width=333.52392578125, height=437.105224609375
[attach_metadata_classifier] nvinfer->process_full_frame: 0
[attach_metadata_classifier] Display Text: face 0 Neutral
[attach_metadata_classifier] Classifier metadata attached for object ID=0
[SGIE Probe] Processing frame number: 5
[SGIE Probe] Found classifier meta for object ID: 0
[SGIE Probe] Face ID 0: Neutral (0.39)
[Tracker Probe] Object ID=0, class_id=0, left=751.1649169921875, top=239.97702026367188, width=333.2769470214844, height=437.9465026855469
[attach_metadata_classifier] nvinfer->process_full_frame: 0
[attach_metadata_classifier] Display Text: face 0 Neutral
[attach_metadata_classifier] Classifier metadata attached for object ID=0

62

please print needs_infer and nvinfer->classifier_async_mode in gst_nvinfer_process_objects to check if each object is inferred.

1 Like
63

@fanzh these are the logs now:
[SGIE Probe] Processing frame number: 0
[Tracker Probe] Object ID=0, class_id=0, left=750.8934936523438, top=240.15335083007812, width=333.21234130859375, height=437.3307800292969
[Tracker Probe] Object ID=0, class_id=0, left=751.431640625, top=240.21600341796875, width=333.31170654296875, height=437.4458923339844
[SGIE Probe] Processing frame number: 1
[Debug] m_ClassifierThreshold: 0.1
[Debug] m_Labels structure:
Index 0 (size 1): Angry
Index 1 (size 1): Disgust
Index 2 (size 1): Fear
Index 3 (size 1): Happy
Index 4 (size 1): Sad
Index 5 (size 1): Surprise
Index 6 (size 1): Neutral
[Debug] Number of attributes (output layers): 1
[Debug] Layer index = 0, numClasses = 7
Class 0 => Probability = 0.00261071
Class 1 => Probability = 7.94431e-06
Class 2 => Probability = 0.197126
[Debug] Updated maxProbability for Class 2 => 0.197126
[Debug] After attrFound=true: maxProbability=0.197126, attr.attributeIndex=0, attr.attributeValue=2
Class 3 => Probability = 0.28822
[Debug] Updated maxProbability for Class 3 => 0.28822
[Debug] After attrFound=true: maxProbability=0.28822, attr.attributeIndex=0, attr.attributeValue=3
Class 4 => Probability = 0.0646253
Class 5 => Probability = 0.0622103
Class 6 => Probability = 0.3852
[Debug] Updated maxProbability for Class 6 => 0.3852
[Debug] After attrFound=true: maxProbability=0.3852, attr.attributeIndex=0, attr.attributeValue=6
[Debug] Attribute Label Found: Neutral
[Debug] Appended Attribute Label to attrString: Neutral
[Debug] Final Attribute String: Neutral
[attach_metadata_classifier] nvinfer->process_full_frame: 0
[attach_metadata_classifier] Display Text: face 0 Neutral
[attach_metadata_classifier] Classifier metadata attached for object ID=0
[SGIE Probe] Processing frame number: 2
[SGIE Probe] Found classifier meta for object ID: 0
[SGIE Probe] Face ID 0: Neutral (0.39)
[Tracker Probe] Object ID=0, class_id=0, left=751.1373291015625, top=240.59255981445312, width=333.4554443359375, height=437.25018310546875
[attach_metadata_classifier] nvinfer->process_full_frame: 0
[attach_metadata_classifier] Display Text: face 0 Neutral
[attach_metadata_classifier] Classifier metadata attached for object ID=0
[SGIE Probe] Processing frame number: 3
[SGIE Probe] Found classifier meta for object ID: 0
[SGIE Probe] Face ID 0: Neutral (0.39)
[Tracker Probe] Object ID=0, class_id=0, left=751.107421875, top=240.57855224609375, width=333.3847961425781, height=437.1232604980469
[attach_metadata_classifier] nvinfer->process_full_frame: 0
[attach_metadata_classifier] Display Text: face 0 Neutral
[attach_metadata_classifier] Classifier metadata attached for object ID=0
[SGIE Probe] Processing frame number: 4
[SGIE Probe] Found classifier meta for object ID: 0
[SGIE Probe] Face ID 0: Neutral (0.39)
[Tracker Probe] Object ID=0, class_id=0, left=751.4085083007812, top=240.62271118164062, width=333.52392578125, height=437.105224609375
[attach_metadata_classifier] nvinfer->process_full_frame: 0
[attach_metadata_classifier] Display Text: face 0 Neutral
[attach_metadata_classifier] Classifier metadata attached for object ID=0
[SGIE Probe] Processing frame number: 5
[SGIE Probe] Found classifier meta for object ID: 0
[SGIE Probe] Face ID 0: Neutral (0.39)
[Tracker Probe] Object ID=0, class_id=0, left=751.1649169921875, top=239.97702026367188, width=333.2769470214844, height=437.9465026855469
[attach_metadata_classifier] nvinfer->process_full_frame: 0
[attach_metadata_classifier] Display Text: face 0 Neutral
[attach_metadata_classifier] Classifier metadata attached for object ID=0
[SGIE Probe] Processing frame number: 6
[SGIE Probe] Found classifier meta for object ID: 0
[SGIE Probe] Face ID 0: Neutral (0.39)
[Tracker Probe] Object ID=0, class_id=0, left=751.59033203125, top=239.69998168945312, width=332.65118408203125, height=442.8916015625
[attach_metadata_classifier] nvinfer->process_full_frame: 0
[attach_metadata_classifier] Display Text: face 0 Neutral
[attach_metadata_classifier] Classifier metadata attached for object ID=0