Environment
NVIDIA DeepStream Version: 6.3.0
Hardware: NVIDIA Jetson (based on aarch64-linux-gnu outputs)
Operating System: Ubuntu (likely 20.04/22.04 on Jetson)
Problem Domain: Face Recognition Pipeline - PGIE (Face Detection) → Landmark SGIE (5-point Landmarks) → Preprocessing (Face Alignment & Normalization) → Embedding SGIE (Face Embeddings) → Saving Embeddings.
Summary of Problem
I’m building a NVIDIA DeepStream pipeline where I need to perform face alignment and normalization after landmark detection but before the face embedding model. My custom C++ library (libnvds_custom_face_align.so) contains the NvDsInferCustomPreprocess function designed to handle this.
Initially, the pipeline worked successfully without the landmark alignment step, going directly from face detection to embedding generation. However, due to low accuracy in face recognition without proper alignment, I am attempting to integrate this landmark alignment preprocessing step.
The primary blocker is getting the gst-nvdspreprocess plugin to correctly parse its configuration file (preprocess_config.txt) and then successfully load my custom library. I’ve encountered various parsing errors, and even when addressing them, I now face a “Could not open custom library” error, preventing the custom preprocessing function from being called. When gst-nvdspreprocess is removed and the embedding SGIE directly takes input from the landmark SGIE (without custom preprocessing), the pipeline runs, showing that detection and landmark models work.
Current State & Specific Errors
The pipeline currently fails at the gst-nvdspreprocess element’s configuration loading and custom library opening.
I’ve faced two main categories of persistent errors:
Configuration Parsing Errors (e.g., for process-on-all-objects):
ERROR nvdspreprocess nvdspreprocess_property_parser.cpp:629:nvdspreprocess_parse_common_group: Failed to parse config file /home/atmecs/Downloads/with_landmark/project/preprocess_config.txt: Error while setting property, in group group-0 Key file contains key "process-on-all-objects" which has a value that cannot be interpreted.
NVDSPREPROCESS_CFG_PARSER: Group 'group-0' parse failed
This occurred despite trying 0, 1, true, and false for boolean properties, and meticulously checking for hidden characters. It also appeared when [group-0] was temporarily removed, suggesting it’s mandatory.
Custom Library Loading Error:
Even after attempting to resolve the parsing errors, the pipeline then shows:
WARN nvdspreprocess gstnvdspreprocess.cpp:506:gst_nvdspreprocess_start:<nvdspreprocess-align> error: Could not open custom library
WARN GST_PADS gstpad.c:1142:gst_pad_set_active:<nvdspreprocess-align:sink> Failed to activate pad
This error reappears once the configuration parser gets past initial hurdles.
Detailed Pipeline Flow
filesrc -> decodebin -> nvvideoconvert -> capsfilter -> nvstreammux -> primary-inference (PGIE, YOLOv8 Face) -> secondary-inference-landmark (SGIE, 5-point landmarks) -> nvdspreprocess-align (Custom Preprocessing for Alignment/Normalization) -> secondary-inference-embedding (SGIE, Face Embeddings) -> nvdsosd -> nvegltransform -> nveglglessink
The osd_sink_pad_buffer_probe is attached to the secondary-inference-embedding’s source pad to extract and save the embeddings. Currently, no embeddings are saved because the nvdspreprocess-align element is failing, preventing data flow to the embedding SGIE. The g_print statements in NvDsInferCustomPreprocess are not being triggered, confirming the library isn’t being used.
Attempted Debugging & What’s Confirmed
gst-nvdspreprocess config parsing: This is the main blocker. The parser is extremely sensitive to unknown reasons for “value cannot be interpreted” or “group parse failed” errors, even with seemingly correct INI syntax. We have tried various boolean formats and meticulously cleaned the file for hidden characters.
gst-nvinfer custom input hook: gst-inspect-1.0 nvinfer confirms that there is no direct GObject property like custom-input-fn-name for gst-nvinfer (as indicated by older documentation/samples), meaning the custom preprocessing must happen via gst-nvdspreprocess for this pipeline structure.
Custom Library (libnvds_custom_face_align.so):
ls -l /home/atmecs/Downloads/with_landmark/project/libnvds_custom_face_align.soconfirms file existence and permissions (-rwxrwxr-x).ldd /home/atmecs/Downloads/with_landmark/project/libnvds_custom_face_align.soconfirms all direct dependencies are met (no not found errors).nm -D /home/atmecs/Downloads/with_landmark/project/libnvds_custom_face_align.so | grep NvDsInferCustomPreprocessconfirms NvDsInferCustomPreprocess is correctly exported and not name-mangled (T symbol present).- The C++ code (custom_face_align.cpp) includes g_print statements that are not appearing in the logs, indicating the library is not being loaded by gst-nvdspreprocess due to its config parsing or subsequent loading failure.
- The NvDsInferCustomPreprocess function expects NvBufSurface* in_surf, NvBufSurface* out_surf, NvDsInferContextInitParams* init_params, NvDsBatchMeta* batch_meta. It correctly identifies landmark SGIE’s unique ID as 3 and looks for fc1 layer (without hardcoding 10 elements due to 212 output).
Configuration Files & Code
1. Python Pipeline Script (with_landmark_train.py):
import sys
import gi
gi.require_version('Gst', '1.0')
from gi.repository import Gst, GLib
import pyds
import ctypes
import numpy as np
import json
import os
import logging
import configparser
logging.basicConfig(
level=logging.INFO,
format='[%(levelname)s] %(asctime)s %(message)s',
handlers=[
logging.FileHandler("app.log"),
logging.StreamHandler()
]
)
Gst.init(None)
embedding_store = {}
def osd_sink_pad_buffer_probe(pad, info, u_data):
gst_buffer = info.get_buffer()
if not gst_buffer:
logging.error("[ERROR] Unable to get GstBuffer")
return Gst.PadProbeReturn.OK
batch_meta = pyds.gst_buffer_get_nvds_batch_meta(hash(gst_buffer))
if not batch_meta:
logging.error("[ERROR] Unable to get batch meta")
return Gst.PadProbeReturn.OK
l_frame = batch_meta.frame_meta_list
while l_frame:
try:
frame_meta = pyds.NvDsFrameMeta.cast(l_frame.data)
except StopIteration:
break
frame_id = frame_meta.frame_num
logging.debug(f"[FRAME DEBUG] Processing Frame {frame_id}")
l_obj = frame_meta.obj_meta_list
while l_obj:
try:
obj_meta = pyds.NvDsObjectMeta.cast(l_obj.data)
except StopIteration:
break
if obj_meta.class_id == 0:
l_user = obj_meta.obj_user_meta_list
while l_user:
try:
user_meta = pyds.NvDsUserMeta.cast(l_user.data)
except StopIteration:
break
if user_meta.base_meta.meta_type == pyds.NvDsMetaType.NVDSINFER_TENSOR_OUTPUT_META:
tensor_meta = pyds.NvDsInferTensorMeta.cast(user_meta.user_meta_data)
if tensor_meta.unique_id == 2: # Embedding SGIE unique ID
for i in range(tensor_meta.num_output_layers):
layer_info = pyds.get_nvds_LayerInfo(tensor_meta, i)
if layer_info.inferDims.numElements > 0:
ptr = ctypes.cast(pyds.get_ptr(layer_info.buffer), ctypes.POINTER(ctypes.c_float))
np_array = np.ctypeslib.as_array(ptr, shape=(layer_info.inferDims.numElements,))
norm_embedding = np_array / np.linalg.norm(np_array) if np.linalg.norm(np_array) != 0 else np_array
video_id = u_data if u_data else "run"
if video_id not in embedding_store:
embedding_store[video_id] = []
embedding_store[video_id].append(norm_embedding.tolist())
logging.info(f"[INFO] Saved normalized embedding for '{video_id}' (Frame {frame_id}, Object {obj_meta.object_id})")
break
break
try:
l_obj = l_obj.next
except StopIteration:
break
try:
l_frame = l_frame.next
except StopIteration:
break
return Gst.PadProbeReturn.OK
def cb_newpad(decodebin, pad, data):
caps = pad.get_current_caps()
name = caps.to_string()
logging.info(f"[INFO] New pad detected: {name}")
if name.startswith("video/"):
sinkpad = data.get_static_pad("sink")
if not sinkpad.is_linked():
pad.link(sinkpad)
def main():
video_path = "video/sakshi.mp4"
config_path = "/home/atmecs/Downloads/with_landmark/project/dstest3_pgie_config.txt"
landmark_sgie_path = "/home/atmecs/Downloads/with_landmark/project/landmark_config.txt"
sgie_path = "/home/atmecs/Downloads/with_landmark/project/embed_config.txt"
output_json = "embeddings/generated_embeddings.json"
preprocess_config_path="/home/atmecs/Downloads/with_landmark/project/preprocess_config.txt"
pipeline = Gst.Pipeline()
source = Gst.ElementFactory.make("filesrc", "source")
source.set_property("location", video_path)
decodebin = Gst.ElementFactory.make("decodebin", "decodebin")
queue = Gst.ElementFactory.make("queue", "queue")
convert1 = Gst.ElementFactory.make("nvvideoconvert", "convert1")
capsfilter = Gst.ElementFactory.make("capsfilter", "capsfilter")
caps = Gst.Caps.from_string("video/x-raw(memory:NVMM), format=NV12")
capsfilter.set_property("caps", caps)
streammux = Gst.ElementFactory.make("nvstreammux", "streammux")
streammux.set_property("batch-size", 1)
streammux.set_property("width", 640)
streammux.set_property("height", 640)
streammux.set_property("batched-push-timeout", 40000)
pgie = Gst.ElementFactory.make("nvinfer", "primary-inference")
pgie.set_property("config-file-path", config_path)
landmark_sgie = Gst.ElementFactory.make("nvinfer", "secondary-inference-landmark")
landmark_sgie.set_property("config-file-path", landmark_sgie_path)
nvdspreprocess = Gst.ElementFactory.make("nvdspreprocess", "nvdspreprocess-align")
# Corrected property name for config file based on gst-inspect-1.0
nvdspreprocess.set_property("config-file", preprocess_config_path)
sgie = Gst.ElementFactory.make("nvinfer", "secondary-inference-embedding")
sgie.set_property("config-file-path", sgie_path)
nvdsosd = Gst.ElementFactory.make("nvdsosd", "osd")
transform = Gst.ElementFactory.make("nvegltransform", "nvegl-transform")
sink = Gst.ElementFactory.make("nveglglessink", "sink")
for elem in [source, decodebin, queue, convert1, capsfilter, streammux, pgie, landmark_sgie, nvdspreprocess, sgie, nvdsosd, transform, sink]:
if not elem:
logging.error(f"[ERROR] Failed to create element: {elem.get_name()}")
sys.exit(1)
pipeline.add(elem)
source.link(decodebin)
decodebin.connect("pad-added", cb_newpad, queue)
queue.link(convert1)
convert1.link(capsfilter)
sinkpad = streammux.get_request_pad("sink_0")
srcpad = capsfilter.get_static_pad("src")
if not srcpad or not sinkpad:
logging.error("[ERROR] Failed to get pads for streammux linking.")
sys.exit(1)
srcpad.link(sinkpad)
streammux.link(pgie)
pgie.link(landmark_sgie)
landmark_sgie.link(nvdspreprocess)
nvdspreprocess.link(sgie)
sgie.link(nvdsosd)
nvdsosd.link(transform)
transform.link(sink)
sgie_src_pad = sgie.get_static_pad("src")
if sgie_src_pad:
video_id = os.path.splitext(os.path.basename(video_path))[0]
sgie_src_pad.add_probe(Gst.PadProbeType.BUFFER, osd_sink_pad_buffer_probe, video_id)
else:
logging.error("[ERROR] Could not get Embedding SGIE src pad for probe.")
logging.info("[INFO] Starting pipeline for embedding generation and display...")
pipeline.set_state(Gst.State.PLAYING)
try:
loop = GLib.MainLoop()
loop.run()
except Exception as e:
logging.error(f"[ERROR] Interrupted: {e}")
finally:
pipeline.set_state(Gst.State.NULL)
logging.info("[INFO] Pipeline stopped.")
if os.path.exists(output_json):
try:
with open(output_json, "r") as f:
existing_embeddings = json.load(f)
except json.JSONDecodeError:
logging.error(f"[ERROR] Invalid JSON in {output_json}. Starting with empty existing embeddings.")
existing_embeddings = {}
else:
existing_embeddings = {}
final_embeddings = {}
for video_id, embs in embedding_store.items():
if not embs:
logging.warning(f"[WARN] No embeddings generated for video ID '{video_id}'. Skipping save.")
continue
final_embeddings[video_id] = embs
logging.info(f"[INFO] Stored {len(embs)} embeddings for '{video_id}'.")
merged_embeddings = {**existing_embeddings, **final_embeddings}
os.makedirs(os.path.dirname(output_json), exist_ok=True)
with open(output_json, "w") as f:
json.dump(merged_embeddings, f, indent=2)
logging.info(f"[INFO] Saved {len(final_embeddings)} new/updated embedding sets, total entries in JSON: {len(merged_embeddings)}")
if __name__ == '__main__':
main()
2. PGIE Config (dstest3_pgie_config.txt):
(You’ve provided this previously, assuming it detects faces and its unique ID is 1)
[property]
gpu-id=0
net-scale-factor=0.003921568
gie-unique-id=1
# Add your YOLOv8 face model specific properties here
onnx-file=models/yolov8n-face-lindevs.onnx
model-engine-file=models/yolov8n-face-lindevs.onnx_b1_gpu0_fp32.engine
labelfile-path=labels.txt
batch-size=1
network-mode=0 # FP32
num-detected-classes=1
interval=0
cluster-mode=2 # DBSCAN
output-blob-names=output0 # YOLOv8 output layer
network-type=0 # Detector
# ... other YOLOv8 specific parameters like custom parser, NMS thresholds, etc.
# For example, if you have a custom parser:
parse-bbox-func-name=NvDsInferParseYolo
custom-lib-path=/opt/nvidia/deepstream/deepstream/lib/libnvds_yoloparser.so
[class-attrs-all]
pre-cluster-threshold=0.25
post-cluster-threshold=0.4
# For YOLOv8, you might need to adjust detection and NMS thresholds
# nms-iou-threshold=0.5
# topk=100
3. Landmark SGIE Config (landmark_config.txt):
(You’ve provided this previously, assuming its unique ID is 3 and its output layer is fc1 with 212 elements)
[property]
gpu-id=0
# IMPORTANT: This Landmark SGIE's unique ID.
gie-unique-id=3
operate-on-gie-id=1 # Operate on objects from PGIE (UID 1)
operate-on-class-ids=0 # Only process class_id 0 (faces) from upstream GIE
onnx-file=models/landmark_5.onnx
model-engine-file=models/landmark_5.onnx_b1_gpu0_fp16.engine
labelfile-path=labels.txt
batch-size=1
network-mode=2 # FP16
num-detected-classes=1
output-blob-names=fc1 # Your landmark model's output layer for landmarks (212 elements)
infer-dims=3;192;192 # C;H;W for your landmark model input
network-type=1 # Classifier (often used for landmark prediction models)
output-tensor-meta=1 # Crucial to attach the landmark tensor for nvdspreprocess
net-scale-factor=1.0/255.0 # Example normalization for 0-255 to 0-1 range
offsets=0;0;0 # No offset if model expects 0-1 range
# mean-file= # No mean file if model expects 0-1 range
model-color-format=0 # 0=RGB, 1=BGR, 2=GRAY (set as per your model's requirement)
process-mode=2 # Secondary GIE mode (operating on objects)
classifier-async-mode=0
[class-attrs-all]
output-tensor-meta=1
verbose=1
4. gst-nvdspreprocess Config (preprocess_config.txt):
(The one we are currently trying to make parse correctly)
[property]
enable=1
gpu-id=0
unique-id=4
operate-on-gie-id=3
process-on-frame=0
target-unique-ids=2
custom-lib-path=/home/atmecs/Downloads/with_landmark/project/libnvds_custom_face_align.so
custom-tensor-preparation-function=NvDsInferCustomPreprocess
processing-width=112
processing-height=112
network-input-order=0
network-color-format=0
tensor-data-type=0
batch-size=1
network-input-shape=1;3;112;112
maintain-aspect-ratio=1
symmetric-padding=1
scaling-buf-pool-size=6
tensor-buf-pool-size=6
tensor-name=input_tensor
scaling-pool-memory-type=0
scaling-pool-compute-hw=0
scaling-filter=0
verbose=1
[user-configs]
pixel-normalization-factor=0.003921568
[group-0]
src-ids=0
operate-on-class-ids=0
process-on-all-objects=0
input-object-min-width=30
input-object-min-height=30
input-object-max-width=1000
input-object-max-height=1000
5. Embedding SGIE Config (embed_config.txt):
(This expects preprocessed tensor from nvdspreprocess)
[property]
gpu-id=0
gie-unique-id=2
operate-on-gie-id=4 # Operate on metadata from nvdspreprocess (unique-id=4)
operate-on-class-ids=0
process-mode=2
input-tensor-meta=1 # IMPORTANT: Tell nvinfer to expect preprocessed input tensors
onnx-file=models/model.onnx
model-engine-file=models/model.onnx_b1_gpu0_fp16.engine
labelfile-path=labels.txt
batch-size=1
network-mode=2
num-detected-classes=1
output-blob-names=1333
infer-dims=3;112;112
network-type=1
output-tensor-meta=1
classifier-async-mode=0
[class-attrs-all]
output-tensor-meta=1
verbose=1
6. Custom C++ Preprocessing Library Source (custom_face_align.cpp):
(This is the source for libnvds_custom_face_align.so)
// custom_face_align.cpp
// --- C++ Standard Libraries (Minimal) ---
#include <iostream> // For std::cout, std::cerr
#include <string> // For std::string
#include <cstring> // For memcpy, strcmp
#include <cmath> // For math functions
// --- GStreamer and GLib (Essential NVIDIA DeepStream Dependencies) ---
#include <gst/gst.h> // For g_print, guint (used by NVIDIA DeepStream types)
#include <glib.h> // For GList (used by NVIDIA DeepStream metadata)
// --- NVIDIA DeepStream Native Libraries (Core API) ---
// These define the primary metadata and buffer structures.
#include <nvbufsurface.h> // NvBufSurface, NvBufSurfaceMap/UnMap
#include <nvbufsurftransform.h> // NvBufSurfaceTransform (optional, for transformations)
#include <nvdsmeta.h> // NvDsBatchMeta, NvDsFrameMeta, NvDsObjectMeta, NvDsUserMeta, NVDSINFER_TENSOR_OUTPUT_META
#include <nvdsinfer_custom_impl.h> // NvDsInferCustomPreprocess signature, NvDsInferStatus
#include <nvdsinfer.h> // NvDsInferLayerInfo, NvDsInferNetworkInfo, NvDsInferTensorOrder
// **IMPORTANT**: Include nvdsmeta_schema.h here.
#include <nvdsmeta_schema.h> // For nvds_get_user_meta_data_for_unique_id
// --- OpenCV Libraries (For image processing) ---
#include <opencv2/core/core.hpp> // cv::Mat, cv::Point2f, cv::Rect, cv::Scalar
#include <opencv2/imgproc/imgproc.hpp> // cv::cvtColor, cv::warpAffine, cv::getAffineTransform
// --- NECESSARY MANUAL DECLARATIONS (for compatibility across NVIDIA DeepStream versions) ---
// Manual declaration for NvDsInferTensorOrder
// (Typically found in nvdsinfer.h)
typedef enum {
NVDSINFER_TENSOR_NCHW = 0, // Batches, Channels, Height, Width
NVDSINFER_TENSOR_NHWC = 1 // Batches, Height, Width, Channels
} NvDsInferTensorOrder;
// Manual declaration for NvDsInferTensorMeta
#ifndef NVDSINFER_TENSORMETA_DEFINED_BY_CUSTOM_LIB
#define NVDSINFER_TENSORMETA_DEFINED_BY_CUSTOM_LIB
struct NvDsInferTensorMeta {
guint num_output_layers;
NvDsInferLayerInfo *output_layers;
guint unique_id;
void *input_surf_buf_ptr;
guint frame_width;
guint frame_height;
NvBbox_Coords object_location;
NvBufSurface *in_nvbufsurface;
NvDsInferTensorOrder tensor_order;
};
#endif // NVDSINFER_TENSORMETA_DEFINED_BY_CUSTOM_LIB
// Manual declaration for nvds_get_user_meta_data_for_unique_id
// This is often in nvdsmeta_schema.h, but we declare it here as a fallback
// if your compiler can't find it there for some reason.
extern "C" void *
nvds_get_user_meta_data_for_unique_id (GList * user_meta_list, guint unique_id);
// --- Global Constants for Face Alignment ---
const float TARGET_LANDMARKS[10] = {
30.2946f, 51.6963f, // Left Eye (target X, Y)
82.5294f, 51.6963f, // Right Eye (target X, Y)
56.2272f, 75.6025f, // Nose (target X, Y)
40.1600f, 96.3025f, // Left Mouth Corner (target X, Y)
72.9150f, 96.3025f // Right Mouth Corner (target X, Y)
};
const int EMBEDDING_INPUT_WIDTH = 112;
const int EMBEDDING_INPUT_HEIGHT = 112;
const int EMBEDDING_INPUT_CHANNELS = 3;
// --- Helper Function: Convert NvBufSurface to OpenCV Mat ---
cv::Mat nvbufsurface_to_cvmat(NvBufSurface *surf, guint batch_idx) {
if (!surf || batch_idx >= surf->batchSize) {
g_print("NvBufSurfaceToCvMat Error: Invalid surf or batch_idx.\n");
return cv::Mat();
}
void* data_ptr = surf->surfaceList[batch_idx].dataPtr;
int width = surf->surfaceList[batch_idx].width;
int height = surf->surfaceList[batch_idx].height;
NvBufSurfaceColorFormat format = surf->surfaceList[batch_idx].colorFormat;
cv::Mat img;
if (format == NVBUF_COLOR_FORMAT_RGBA) {
img = cv::Mat(height, width, CV_8UC4, data_ptr);
g_print("NvBufSurfaceToCvMat: Converted RGBA to BGR.\n");
cv::cvtColor(img, img, cv::COLOR_RGBA2BGR);
} else if (format == NVBUF_COLOR_FORMAT_BGR) {
img = cv::Mat(height, width, CV_8UC3, data_ptr);
g_print("NvBufSurfaceToCvMat: Direct BGR format.\n");
} else if (format == NVBUF_COLOR_FORMAT_NV12) {
img = cv::Mat(height * 3 / 2, width, CV_8UC1, data_ptr);
g_print("NvBufSurfaceToCvMat: Converted NV12 to BGR.\n");
cv::cvtColor(img, img, cv::COLOR_YUV2BGR_NV12);
} else {
g_print("NvBufSurfaceToCvMat Warning: Unsupported NvBufSurface color format: %d. Returning empty Mat.\n", format);
return cv::Mat();
}
return img;
}
// --- Custom Context Initialization Function ---
extern "C" bool NvDsInferContextInit(NvDsInferContextInitParams* init_params) {
g_print("NvDsInferContextInit: Custom preprocessor initialized.\n");
return true;
}
// --- Custom Context De-initialization Function ---
extern "C" void NvDsInferContextDeinit(NvDsInferContextInitParams* init_params) {
g_print("NvDsInferContextDeinit: Custom preprocessor deinitialized.\n");
}
// --- The Core Custom Preprocessing Function ---
extern "C" NvDsInferStatus NvDsInferCustomPreprocess(
NvBufSurface* in_surf,
NvBufSurface* out_surf,
NvDsInferContextInitParams* init_params,
NvDsBatchMeta* batch_meta
) {
g_print("NvDsInferCustomPreprocess: Entered for batch_size=%d\n", batch_meta->num_frames_in_batch);
if (!in_surf || !out_surf || !init_params || !batch_meta) {
g_print("NvDsInferCustomPreprocess Error: Invalid arguments provided.\n");
return NVDSINFER_CUSTOM_LIB_FAILED;
}
if (NvBufSurfaceMap(in_surf, -1, -1, NVBUF_MAP_READ_WRITE) != 0) {
g_print("NvDsInferCustomPreprocess Error: Failed to map input NvBufSurface for read/write.\n");
return NVDSINFER_CUSTOM_LIB_FAILED;
}
if (NvBufSurfaceMap(out_surf, -1, -1, NVBUF_MAP_READ_WRITE) != 0) {
g_print("NvDsInferCustomPreprocess Error: Failed to map output NvBufSurface for writing.\n");
NvBufSurfaceUnMap(in_surf, -1, -1);
return NVDSINFER_CUSTOM_LIB_FAILED;
}
guint infer_output_batch_counter = 0;
for (GList* l_frame_meta = batch_meta->frame_meta_list; l_frame_meta != nullptr; l_frame_meta = l_frame_meta->next) {
NvDsFrameMeta* frame_meta = reinterpret_cast<NvDsFrameMeta*>(l_frame_meta->data);
guint current_frame_batch_idx = frame_meta->batch_id;
g_print("NvDsInferCustomPreprocess: Processing Frame %u (batch_idx %u).\n", frame_meta->frame_num, current_frame_batch_idx);
cv::Mat input_frame_mat = nvbufsurface_to_cvmat(in_surf, current_frame_batch_idx);
cv::Mat input_frame_mat = nvbufsurface_to_cvmat(in_surf, current_frame_batch_idx);
if (input_frame_mat.empty()) {
g_print("NvDsInferCustomPreprocess Error: Could not convert NvBufSurface to OpenCV Mat for frame batch_id %u. Skipping this frame.\n", current_frame_batch_idx);
GList* temp_obj_list = frame_meta->obj_meta_list;
while(temp_obj_list) {
if (infer_output_batch_counter < out_surf->batchSize) {
void* dest_ptr_obj = out_surf->surfaceList[infer_output_batch_counter].dataPtr;
if (dest_ptr_obj) {
memset(dest_ptr_obj, 0, EMBEDDING_INPUT_WIDTH * EMBEDDING_INPUT_HEIGHT * EMBEDDING_INPUT_CHANNELS * sizeof(float));
}
} else {
g_print("NvDsInferCustomPreprocess Error: Output buffer index out of bounds or null when frame conversion failed.\n");
}
infer_output_batch_counter++;
temp_obj_list = temp_obj_list->next;
}
continue;
}
GList* obj_meta_list = frame_meta->obj_meta_list;
for (GList* l_obj_meta = obj_meta_list; l_obj_meta != nullptr; l_obj_meta = l_obj_meta->next) {
NvDsObjectMeta* obj_meta = reinterpret_cast<NvDsObjectMeta*>(l_obj_meta->data);
g_print("NvDsInferCustomPreprocess: Processing Object %u (class_id %u).\n", obj_meta->object_id, obj_meta->class_id);
if (obj_meta->class_id != 0) {
g_print("NvDsInferCustomPreprocess: Skipping non-face object (class_id %u).\n", obj_meta->class_id);
if (infer_output_batch_counter < out_surf->batchSize) {
void* dest_ptr_obj = out_surf->surfaceList[infer_output_batch_counter].dataPtr;
if (dest_ptr_obj) {
memset(dest_ptr_obj, 0, EMBEDDING_INPUT_WIDTH * EMBEDDING_INPUT_HEIGHT * EMBEDDING_INPUT_CHANNELS * sizeof(float));
}
} else {
g_print("NvDsInferCustomPreprocess Error: Output buffer index out of bounds or null for non-face object.\n");
}
infer_output_batch_counter++;
continue;
}
float* landmark_data = nullptr;
// CRITICAL FIX: Landmark SGIE's unique ID is 3
NvDsInferTensorMeta *tensor_meta = (NvDsInferTensorMeta *)nvds_get_user_meta_data_for_unique_id(
obj_meta->obj_user_meta_list, 3);
if (tensor_meta) {
g_print("NvDsInferCustomPreprocess: Found tensor_meta from unique_id %u. Num output layers: %u.\n", tensor_meta->unique_id, tensor_meta->num_output_layers);
for (guint i = 0; i < tensor_meta->num_output_layers; ++i) {
NvDsInferLayerInfo* layer_info = &(tensor_meta->output_layers[i]);
g_print("NvDsInferCustomPreprocess: Checking Layer '%s' with %u elements.\n", layer_info->layerName, layer_info->inferDims.numElements);
if (strcmp(layer_info->layerName, "fc1") == 0) {
landmark_data = reinterpret_cast<float*>(layer_info->buffer);
g_print("NvDsInferCustomPreprocess: Found 'fc1' layer. Landmark data buffer: %p.\n", (void*)landmark_data);
break;
}
}
}
if (!landmark_data) {
g_print("NvDsInferCustomPreprocess Warning: No landmark data (from GIE ID 3) found for object ID %u in frame %u. Filling output slot with zeros.\n",
obj_meta->object_id, frame_meta->frame_num);
if (infer_output_batch_counter < out_surf->batchSize) {
void* dest_ptr_obj = out_surf->surfaceList[infer_output_batch_counter].dataPtr;
if (dest_ptr_obj) {
memset(dest_ptr_obj, 0, EMBEDDING_INPUT_WIDTH * EMBEDDING_INPUT_HEIGHT * EMBEDDING_INPUT_CHANNELS * sizeof(float));
}
} else {
g_print("NvDsInferCustomPreprocess Error: Output buffer index out of bounds or null for object without landmarks.\n");
}
infer_output_batch_counter++;
continue;
}
cv::Rect face_roi(
static_cast<int>(obj_meta->rect_params.left),
static_cast<int>(obj_meta->rect_params.top),
static_cast<int>(obj_meta->rect_params.width),
static_cast<int>(obj_meta->rect_params.height)
);
g_print("NvDsInferCustomPreprocess: Face ROI: (x:%d, y:%d, w:%d, h:%d).\n", face_roi.x, face_roi.y, face_roi.width, face_roi.height);
face_roi = face_roi & cv::Rect(0, 0, input_frame_mat.cols, input_frame_mat.rows);
if (face_roi.empty()) {
g_print("NvDsInferCustomPreprocess Warning: Empty or invalid face ROI for object ID %u. Filling output slot with zeros.\n", obj_meta->object_id);
if (infer_output_batch_counter < out_surf->batchSize) {
void* dest_ptr_obj = out_surf->surfaceList[infer_output_batch_counter].dataPtr;
if (dest_ptr_obj) {
memset(dest_ptr_obj, 0, EMBEDDING_INPUT_WIDTH * EMBEDDING_INPUT_HEIGHT * EMBEDDING_INPUT_CHANNELS * sizeof(float));
}
} else {
g_print("NvDsInferCustomPreprocess Error: Output buffer index out of bounds or null for invalid ROI.\n");
}
infer_output_batch_counter++;
continue;
}
cv::Mat face_crop = input_frame_mat(face_roi).clone();
g_print("NvDsInferCustomPreprocess: Face crop size: %dx%d.\n", face_crop.cols, face_crop.rows);
cv::Point2f src_pts[3];
cv::Point2f dst_pts[3];
src_pts[0] = cv::Point2f(landmark_data[0] - face_roi.x, landmark_data[1] - face_roi.y);
src_pts[1] = cv::Point2f(landmark_data[2] - face_roi.x, landmark_data[3] - face_roi.y);
src_pts[2] = cv::Point2f(landmark_data[4] - face_roi.x, landmark_data[5] - face_roi.y);
g_print("NvDsInferCustomPreprocess: Source Points: (%.2f,%.2f), (%.2f,%.2f), (%.2f,%.2f)\n",
src_pts[0].x, src_pts[0].y, src_pts[1].x, src_pts[1].y, src_pts[2].x, src_pts[2].y);
dst_pts[0] = cv::Point2f(TARGET_LANDMARKS[0], TARGET_LANDMARKS[1]);
dst_pts[1] = cv::Point2f(TARGET_LANDMARKS[2], TARGET_LANDMARKS[3]);
dst_pts[2] = cv::Point2f(TARGET_LANDMARKS[4], TARGET_LANDMARKS[5]);
g_print("NvDsInferCustomPreprocess: Target Points: (%.2f,%.2f), (%.2f,%.2f), (%.2f,%.2f)\n",
dst_pts[0].x, dst_pts[0].y, dst_pts[1].x, dst_pts[1].y, dst_pts[2].x, dst_pts[2].y);
cv::Mat affine_matrix = cv::getAffineTransform(src_pts, dst_pts);
g_print("NvDsInferCustomPreprocess: Affine Matrix computed.\n");
cv::Mat aligned_face;
aligned_face.create(EMBEDDING_INPUT_HEIGHT, EMBEDDING_INPUT_WIDTH, CV_8UC3);
cv::warpAffine(face_crop, aligned_face, affine_matrix, aligned_face.size(),
cv::INTER_LINEAR, cv::BORDER_CONSTANT, cv::Scalar(0, 0, 0));
g_print("NvDsInferCustomPreprocess: Face aligned to %dx%d.\n", aligned_face.cols, aligned_face.rows);
cv::Mat final_input_tensor;
cv::cvtColor(aligned_face, final_input_tensor, cv::COLOR_BGR2RGB);
g_print("NvDsInferCustomPreprocess: Converted to RGB.\n");
final_input_tensor.convertTo(final_input_tensor, CV_32FC3, 1.0/127.5, -1.0);
g_print("NvDsInferCustomPreprocess: Converted to FP32 and normalized to [-1, 1].\n");
size_t bytes_per_image = EMBEDDING_INPUT_WIDTH * EMBEDDING_INPUT_HEIGHT * EMBEDDING_INPUT_CHANNELS * sizeof(float);
if (infer_output_batch_counter < out_surf->batchSize) {
void* dest_ptr_obj = out_surf->surfaceList[infer_output_batch_counter].dataPtr;
if (dest_ptr_obj) {
memcpy(dest_ptr_obj, final_input_tensor.data, bytes_per_image);
g_print("NvDsInferCustomPreprocess: Copied aligned tensor to output NvBufSurface at index %u.\n", infer_output_batch_counter);
} else {
g_print("NvDsInferCustomPreprocess Error: Output buffer at index %u is null.\n", infer_output_batch_counter);
}
} else {
g_print("NvDsInferCustomPreprocess Error: Attempted to copy aligned face to out-of-bounds output buffer index: %u. Max: %u.\n",
infer_output_batch_counter, out_surf->batchSize);
}
infer_output_batch_counter++;
}
}
NvBufSurfaceUnMap(in_surf, -1, -1);
NvBufSurfaceUnMap(out_surf, -1, -1);
g_print("NvDsInferCustomPreprocess: Exiting. Total aligned objects processed in this batch: %u\n", infer_output_batch_counter);
return NVDSINFER_SUCCESS;
}