NVIDIA DeepStream 6.3: Persistent gst-nvdspreprocess Configuration Parsing & Library Loading Issues

Environment

NVIDIA DeepStream Version: 6.3.0

Hardware: NVIDIA Jetson (based on aarch64-linux-gnu outputs)

Operating System: Ubuntu (likely 20.04/22.04 on Jetson)

Problem Domain: Face Recognition Pipeline - PGIE (Face Detection) → Landmark SGIE (5-point Landmarks) → Preprocessing (Face Alignment & Normalization) → Embedding SGIE (Face Embeddings) → Saving Embeddings.

Summary of Problem

I’m building a NVIDIA DeepStream pipeline where I need to perform face alignment and normalization after landmark detection but before the face embedding model. My custom C++ library (libnvds_custom_face_align.so) contains the NvDsInferCustomPreprocess function designed to handle this.

Initially, the pipeline worked successfully without the landmark alignment step, going directly from face detection to embedding generation. However, due to low accuracy in face recognition without proper alignment, I am attempting to integrate this landmark alignment preprocessing step.

The primary blocker is getting the gst-nvdspreprocess plugin to correctly parse its configuration file (preprocess_config.txt) and then successfully load my custom library. I’ve encountered various parsing errors, and even when addressing them, I now face a “Could not open custom library” error, preventing the custom preprocessing function from being called. When gst-nvdspreprocess is removed and the embedding SGIE directly takes input from the landmark SGIE (without custom preprocessing), the pipeline runs, showing that detection and landmark models work.

Current State & Specific Errors

The pipeline currently fails at the gst-nvdspreprocess element’s configuration loading and custom library opening.

I’ve faced two main categories of persistent errors:

Configuration Parsing Errors (e.g., for process-on-all-objects):

ERROR      nvdspreprocess nvdspreprocess_property_parser.cpp:629:nvdspreprocess_parse_common_group: Failed to parse config file /home/atmecs/Downloads/with_landmark/project/preprocess_config.txt: Error while setting property, in group group-0 Key file contains key "process-on-all-objects" which has a value that cannot be interpreted.
NVDSPREPROCESS_CFG_PARSER: Group 'group-0' parse failed

This occurred despite trying 0, 1, true, and false for boolean properties, and meticulously checking for hidden characters. It also appeared when [group-0] was temporarily removed, suggesting it’s mandatory.

Custom Library Loading Error:

Even after attempting to resolve the parsing errors, the pipeline then shows:

WARN       nvdspreprocess gstnvdspreprocess.cpp:506:gst_nvdspreprocess_start:<nvdspreprocess-align> error: Could not open custom library
WARN       GST_PADS gstpad.c:1142:gst_pad_set_active:<nvdspreprocess-align:sink> Failed to activate pad

This error reappears once the configuration parser gets past initial hurdles.

Detailed Pipeline Flow

filesrc -> decodebin -> nvvideoconvert -> capsfilter -> nvstreammux -> primary-inference (PGIE, YOLOv8 Face) -> secondary-inference-landmark (SGIE, 5-point landmarks) -> nvdspreprocess-align (Custom Preprocessing for Alignment/Normalization) -> secondary-inference-embedding (SGIE, Face Embeddings) -> nvdsosd -> nvegltransform -> nveglglessink

The osd_sink_pad_buffer_probe is attached to the secondary-inference-embedding’s source pad to extract and save the embeddings. Currently, no embeddings are saved because the nvdspreprocess-align element is failing, preventing data flow to the embedding SGIE. The g_print statements in NvDsInferCustomPreprocess are not being triggered, confirming the library isn’t being used.

Attempted Debugging & What’s Confirmed

gst-nvdspreprocess config parsing: This is the main blocker. The parser is extremely sensitive to unknown reasons for “value cannot be interpreted” or “group parse failed” errors, even with seemingly correct INI syntax. We have tried various boolean formats and meticulously cleaned the file for hidden characters.

gst-nvinfer custom input hook: gst-inspect-1.0 nvinfer confirms that there is no direct GObject property like custom-input-fn-name for gst-nvinfer (as indicated by older documentation/samples), meaning the custom preprocessing must happen via gst-nvdspreprocess for this pipeline structure.

Custom Library (libnvds_custom_face_align.so):

  • ls -l /home/atmecs/Downloads/with_landmark/project/libnvds_custom_face_align.so confirms file existence and permissions (-rwxrwxr-x).
  • ldd /home/atmecs/Downloads/with_landmark/project/libnvds_custom_face_align.so confirms all direct dependencies are met (no not found errors).
  • nm -D /home/atmecs/Downloads/with_landmark/project/libnvds_custom_face_align.so | grep NvDsInferCustomPreprocess confirms NvDsInferCustomPreprocess is correctly exported and not name-mangled (T symbol present).
  • The C++ code (custom_face_align.cpp) includes g_print statements that are not appearing in the logs, indicating the library is not being loaded by gst-nvdspreprocess due to its config parsing or subsequent loading failure.
  • The NvDsInferCustomPreprocess function expects NvBufSurface* in_surf, NvBufSurface* out_surf, NvDsInferContextInitParams* init_params, NvDsBatchMeta* batch_meta. It correctly identifies landmark SGIE’s unique ID as 3 and looks for fc1 layer (without hardcoding 10 elements due to 212 output).

Configuration Files & Code

1. Python Pipeline Script (with_landmark_train.py):

import sys
import gi
gi.require_version('Gst', '1.0')
from gi.repository import Gst, GLib

import pyds
import ctypes
import numpy as np
import json
import os
import logging
import configparser

logging.basicConfig(
    level=logging.INFO,
    format='[%(levelname)s] %(asctime)s %(message)s',
    handlers=[
        logging.FileHandler("app.log"),
        logging.StreamHandler()
    ]
)

Gst.init(None)

embedding_store = {}

def osd_sink_pad_buffer_probe(pad, info, u_data):
    gst_buffer = info.get_buffer()
    if not gst_buffer:
        logging.error("[ERROR] Unable to get GstBuffer")
        return Gst.PadProbeReturn.OK

    batch_meta = pyds.gst_buffer_get_nvds_batch_meta(hash(gst_buffer))
    if not batch_meta:
        logging.error("[ERROR] Unable to get batch meta")
        return Gst.PadProbeReturn.OK

    l_frame = batch_meta.frame_meta_list
    while l_frame:
        try:
            frame_meta = pyds.NvDsFrameMeta.cast(l_frame.data)
        except StopIteration:
            break

        frame_id = frame_meta.frame_num
        logging.debug(f"[FRAME DEBUG] Processing Frame {frame_id}")

        l_obj = frame_meta.obj_meta_list
        while l_obj:
            try:
                obj_meta = pyds.NvDsObjectMeta.cast(l_obj.data)
            except StopIteration:
                break

            if obj_meta.class_id == 0:
                l_user = obj_meta.obj_user_meta_list
                while l_user:
                    try:
                        user_meta = pyds.NvDsUserMeta.cast(l_user.data)
                    except StopIteration:
                        break

                    if user_meta.base_meta.meta_type == pyds.NvDsMetaType.NVDSINFER_TENSOR_OUTPUT_META:
                        tensor_meta = pyds.NvDsInferTensorMeta.cast(user_meta.user_meta_data)
                        if tensor_meta.unique_id == 2: # Embedding SGIE unique ID
                            for i in range(tensor_meta.num_output_layers):
                                layer_info = pyds.get_nvds_LayerInfo(tensor_meta, i)
                                if layer_info.inferDims.numElements > 0:
                                    ptr = ctypes.cast(pyds.get_ptr(layer_info.buffer), ctypes.POINTER(ctypes.c_float))
                                    np_array = np.ctypeslib.as_array(ptr, shape=(layer_info.inferDims.numElements,))
                                    
                                    norm_embedding = np_array / np.linalg.norm(np_array) if np.linalg.norm(np_array) != 0 else np_array
                                    
                                    video_id = u_data if u_data else "run"

                                    if video_id not in embedding_store:
                                        embedding_store[video_id] = []
                                    embedding_store[video_id].append(norm_embedding.tolist())

                                    logging.info(f"[INFO] Saved normalized embedding for '{video_id}' (Frame {frame_id}, Object {obj_meta.object_id})")
                                    break
                            break
            try:
                l_obj = l_obj.next
            except StopIteration:
                break
        try:
            l_frame = l_frame.next
        except StopIteration:
            break
    return Gst.PadProbeReturn.OK

def cb_newpad(decodebin, pad, data):
    caps = pad.get_current_caps()
    name = caps.to_string()
    logging.info(f"[INFO] New pad detected: {name}")
    if name.startswith("video/"):
        sinkpad = data.get_static_pad("sink")
        if not sinkpad.is_linked():
            pad.link(sinkpad)

def main():
    video_path = "video/sakshi.mp4"
    config_path = "/home/atmecs/Downloads/with_landmark/project/dstest3_pgie_config.txt"
    landmark_sgie_path = "/home/atmecs/Downloads/with_landmark/project/landmark_config.txt"
    sgie_path = "/home/atmecs/Downloads/with_landmark/project/embed_config.txt"
    output_json = "embeddings/generated_embeddings.json"
    preprocess_config_path="/home/atmecs/Downloads/with_landmark/project/preprocess_config.txt"

    pipeline = Gst.Pipeline()

    source = Gst.ElementFactory.make("filesrc", "source")
    source.set_property("location", video_path)

    decodebin = Gst.ElementFactory.make("decodebin", "decodebin")
    queue = Gst.ElementFactory.make("queue", "queue")
    convert1 = Gst.ElementFactory.make("nvvideoconvert", "convert1")
    capsfilter = Gst.ElementFactory.make("capsfilter", "capsfilter")
    caps = Gst.Caps.from_string("video/x-raw(memory:NVMM), format=NV12")
    capsfilter.set_property("caps", caps)

    streammux = Gst.ElementFactory.make("nvstreammux", "streammux")
    streammux.set_property("batch-size", 1)
    streammux.set_property("width", 640)
    streammux.set_property("height", 640)
    streammux.set_property("batched-push-timeout", 40000)

    pgie = Gst.ElementFactory.make("nvinfer", "primary-inference")
    pgie.set_property("config-file-path", config_path)

    landmark_sgie = Gst.ElementFactory.make("nvinfer", "secondary-inference-landmark")
    landmark_sgie.set_property("config-file-path", landmark_sgie_path)

    nvdspreprocess = Gst.ElementFactory.make("nvdspreprocess", "nvdspreprocess-align")
    # Corrected property name for config file based on gst-inspect-1.0
    nvdspreprocess.set_property("config-file", preprocess_config_path)

    sgie = Gst.ElementFactory.make("nvinfer", "secondary-inference-embedding")
    sgie.set_property("config-file-path", sgie_path)
    
    nvdsosd = Gst.ElementFactory.make("nvdsosd", "osd")
    transform = Gst.ElementFactory.make("nvegltransform", "nvegl-transform")
    sink = Gst.ElementFactory.make("nveglglessink", "sink")

    for elem in [source, decodebin, queue, convert1, capsfilter, streammux, pgie, landmark_sgie, nvdspreprocess, sgie, nvdsosd, transform, sink]:
        if not elem:
            logging.error(f"[ERROR] Failed to create element: {elem.get_name()}")
            sys.exit(1)
        pipeline.add(elem)

    source.link(decodebin)
    decodebin.connect("pad-added", cb_newpad, queue)

    queue.link(convert1)
    convert1.link(capsfilter)

    sinkpad = streammux.get_request_pad("sink_0")
    srcpad = capsfilter.get_static_pad("src")
    if not srcpad or not sinkpad:
        logging.error("[ERROR] Failed to get pads for streammux linking.")
        sys.exit(1)
    srcpad.link(sinkpad)

    streammux.link(pgie)
    pgie.link(landmark_sgie)
    landmark_sgie.link(nvdspreprocess)
    nvdspreprocess.link(sgie)
    
    sgie.link(nvdsosd)
    nvdsosd.link(transform)
    transform.link(sink)

    sgie_src_pad = sgie.get_static_pad("src")
    if sgie_src_pad:
        video_id = os.path.splitext(os.path.basename(video_path))[0]
        sgie_src_pad.add_probe(Gst.PadProbeType.BUFFER, osd_sink_pad_buffer_probe, video_id)
    else:
        logging.error("[ERROR] Could not get Embedding SGIE src pad for probe.")

    logging.info("[INFO] Starting pipeline for embedding generation and display...")
    pipeline.set_state(Gst.State.PLAYING)

    try:
        loop = GLib.MainLoop()
        loop.run()
    except Exception as e:
        logging.error(f"[ERROR] Interrupted: {e}")
    finally:
        pipeline.set_state(Gst.State.NULL)
        logging.info("[INFO] Pipeline stopped.")

    if os.path.exists(output_json):
        try:
            with open(output_json, "r") as f:
                existing_embeddings = json.load(f)
        except json.JSONDecodeError:
            logging.error(f"[ERROR] Invalid JSON in {output_json}. Starting with empty existing embeddings.")
            existing_embeddings = {}
    else:
        existing_embeddings = {}

    final_embeddings = {}
    for video_id, embs in embedding_store.items():
        if not embs:
            logging.warning(f"[WARN] No embeddings generated for video ID '{video_id}'. Skipping save.")
            continue

        final_embeddings[video_id] = embs
        logging.info(f"[INFO] Stored {len(embs)} embeddings for '{video_id}'.")

    merged_embeddings = {**existing_embeddings, **final_embeddings}

    os.makedirs(os.path.dirname(output_json), exist_ok=True)
    with open(output_json, "w") as f:
        json.dump(merged_embeddings, f, indent=2)

    logging.info(f"[INFO] Saved {len(final_embeddings)} new/updated embedding sets, total entries in JSON: {len(merged_embeddings)}")

if __name__ == '__main__':
    main()

2. PGIE Config (dstest3_pgie_config.txt):

(You’ve provided this previously, assuming it detects faces and its unique ID is 1)

[property]
gpu-id=0
net-scale-factor=0.003921568
gie-unique-id=1
# Add your YOLOv8 face model specific properties here
onnx-file=models/yolov8n-face-lindevs.onnx
model-engine-file=models/yolov8n-face-lindevs.onnx_b1_gpu0_fp32.engine
labelfile-path=labels.txt
batch-size=1
network-mode=0 # FP32
num-detected-classes=1
interval=0
cluster-mode=2 # DBSCAN
output-blob-names=output0 # YOLOv8 output layer
network-type=0 # Detector
# ... other YOLOv8 specific parameters like custom parser, NMS thresholds, etc.
# For example, if you have a custom parser:
parse-bbox-func-name=NvDsInferParseYolo
custom-lib-path=/opt/nvidia/deepstream/deepstream/lib/libnvds_yoloparser.so

[class-attrs-all]
pre-cluster-threshold=0.25
post-cluster-threshold=0.4
# For YOLOv8, you might need to adjust detection and NMS thresholds
# nms-iou-threshold=0.5
# topk=100

3. Landmark SGIE Config (landmark_config.txt):

(You’ve provided this previously, assuming its unique ID is 3 and its output layer is fc1 with 212 elements)

[property]
gpu-id=0
# IMPORTANT: This Landmark SGIE's unique ID.
gie-unique-id=3
operate-on-gie-id=1 # Operate on objects from PGIE (UID 1)
operate-on-class-ids=0 # Only process class_id 0 (faces) from upstream GIE

onnx-file=models/landmark_5.onnx
model-engine-file=models/landmark_5.onnx_b1_gpu0_fp16.engine
labelfile-path=labels.txt
batch-size=1
network-mode=2 # FP16
num-detected-classes=1
output-blob-names=fc1 # Your landmark model's output layer for landmarks (212 elements)
infer-dims=3;192;192 # C;H;W for your landmark model input
network-type=1 # Classifier (often used for landmark prediction models)
output-tensor-meta=1 # Crucial to attach the landmark tensor for nvdspreprocess

net-scale-factor=1.0/255.0 # Example normalization for 0-255 to 0-1 range
offsets=0;0;0 # No offset if model expects 0-1 range
# mean-file= # No mean file if model expects 0-1 range
model-color-format=0 # 0=RGB, 1=BGR, 2=GRAY (set as per your model's requirement)

process-mode=2 # Secondary GIE mode (operating on objects)
classifier-async-mode=0

[class-attrs-all]
output-tensor-meta=1
verbose=1

4. gst-nvdspreprocess Config (preprocess_config.txt):

(The one we are currently trying to make parse correctly)

[property]
enable=1
gpu-id=0
unique-id=4
operate-on-gie-id=3
process-on-frame=0
target-unique-ids=2
custom-lib-path=/home/atmecs/Downloads/with_landmark/project/libnvds_custom_face_align.so
custom-tensor-preparation-function=NvDsInferCustomPreprocess
processing-width=112
processing-height=112
network-input-order=0
network-color-format=0
tensor-data-type=0
batch-size=1
network-input-shape=1;3;112;112
maintain-aspect-ratio=1
symmetric-padding=1
scaling-buf-pool-size=6
tensor-buf-pool-size=6
tensor-name=input_tensor
scaling-pool-memory-type=0
scaling-pool-compute-hw=0
scaling-filter=0
verbose=1

[user-configs]
pixel-normalization-factor=0.003921568

[group-0]
src-ids=0
operate-on-class-ids=0
process-on-all-objects=0
input-object-min-width=30
input-object-min-height=30
input-object-max-width=1000
input-object-max-height=1000

5. Embedding SGIE Config (embed_config.txt):

(This expects preprocessed tensor from nvdspreprocess)

[property]
gpu-id=0
gie-unique-id=2
operate-on-gie-id=4 # Operate on metadata from nvdspreprocess (unique-id=4)
operate-on-class-ids=0

process-mode=2
input-tensor-meta=1 # IMPORTANT: Tell nvinfer to expect preprocessed input tensors

onnx-file=models/model.onnx
model-engine-file=models/model.onnx_b1_gpu0_fp16.engine
labelfile-path=labels.txt
batch-size=1
network-mode=2
num-detected-classes=1
output-blob-names=1333
infer-dims=3;112;112
network-type=1
output-tensor-meta=1

classifier-async-mode=0

[class-attrs-all]
output-tensor-meta=1
verbose=1

6. Custom C++ Preprocessing Library Source (custom_face_align.cpp):

(This is the source for libnvds_custom_face_align.so)

// custom_face_align.cpp

// --- C++ Standard Libraries (Minimal) ---
#include <iostream>   // For std::cout, std::cerr
#include <string>     // For std::string
#include <cstring>    // For memcpy, strcmp
#include <cmath>      // For math functions

// --- GStreamer and GLib (Essential NVIDIA DeepStream Dependencies) ---
#include <gst/gst.h>  // For g_print, guint (used by NVIDIA DeepStream types)
#include <glib.h>     // For GList (used by NVIDIA DeepStream metadata)

// --- NVIDIA DeepStream Native Libraries (Core API) ---
// These define the primary metadata and buffer structures.
#include <nvbufsurface.h>        // NvBufSurface, NvBufSurfaceMap/UnMap
#include <nvbufsurftransform.h>  // NvBufSurfaceTransform (optional, for transformations)
#include <nvdsmeta.h>            // NvDsBatchMeta, NvDsFrameMeta, NvDsObjectMeta, NvDsUserMeta, NVDSINFER_TENSOR_OUTPUT_META
#include <nvdsinfer_custom_impl.h> // NvDsInferCustomPreprocess signature, NvDsInferStatus
#include <nvdsinfer.h>          // NvDsInferLayerInfo, NvDsInferNetworkInfo, NvDsInferTensorOrder

// **IMPORTANT**: Include nvdsmeta_schema.h here.
#include <nvdsmeta_schema.h>    // For nvds_get_user_meta_data_for_unique_id


// --- OpenCV Libraries (For image processing) ---
#include <opencv2/core/core.hpp>     // cv::Mat, cv::Point2f, cv::Rect, cv::Scalar
#include <opencv2/imgproc/imgproc.hpp> // cv::cvtColor, cv::warpAffine, cv::getAffineTransform


// --- NECESSARY MANUAL DECLARATIONS (for compatibility across NVIDIA DeepStream versions) ---

// Manual declaration for NvDsInferTensorOrder
// (Typically found in nvdsinfer.h)
typedef enum {
    NVDSINFER_TENSOR_NCHW = 0, // Batches, Channels, Height, Width
    NVDSINFER_TENSOR_NHWC = 1  // Batches, Height, Width, Channels
} NvDsInferTensorOrder;

// Manual declaration for NvDsInferTensorMeta
#ifndef NVDSINFER_TENSORMETA_DEFINED_BY_CUSTOM_LIB
#define NVDSINFER_TENSORMETA_DEFINED_BY_CUSTOM_LIB
struct NvDsInferTensorMeta {
    guint num_output_layers;
    NvDsInferLayerInfo *output_layers;
    guint unique_id;
    void *input_surf_buf_ptr;
    guint frame_width;
    guint frame_height;
    NvBbox_Coords object_location;
    NvBufSurface *in_nvbufsurface;
    NvDsInferTensorOrder tensor_order;
};
#endif // NVDSINFER_TENSORMETA_DEFINED_BY_CUSTOM_LIB

// Manual declaration for nvds_get_user_meta_data_for_unique_id
// This is often in nvdsmeta_schema.h, but we declare it here as a fallback
// if your compiler can't find it there for some reason.
extern "C" void *
nvds_get_user_meta_data_for_unique_id (GList * user_meta_list, guint unique_id);


// --- Global Constants for Face Alignment ---
const float TARGET_LANDMARKS[10] = {
    30.2946f, 51.6963f,  // Left Eye (target X, Y)
    82.5294f, 51.6963f,  // Right Eye (target X, Y)
    56.2272f, 75.6025f,  // Nose (target X, Y)
    40.1600f, 96.3025f,  // Left Mouth Corner (target X, Y)
    72.9150f, 96.3025f   // Right Mouth Corner (target X, Y)
};

const int EMBEDDING_INPUT_WIDTH = 112;
const int EMBEDDING_INPUT_HEIGHT = 112;
const int EMBEDDING_INPUT_CHANNELS = 3;


// --- Helper Function: Convert NvBufSurface to OpenCV Mat ---
cv::Mat nvbufsurface_to_cvmat(NvBufSurface *surf, guint batch_idx) {
    if (!surf || batch_idx >= surf->batchSize) {
        g_print("NvBufSurfaceToCvMat Error: Invalid surf or batch_idx.\n");
        return cv::Mat();
    }

    void* data_ptr = surf->surfaceList[batch_idx].dataPtr;
    int width = surf->surfaceList[batch_idx].width;
    int height = surf->surfaceList[batch_idx].height;
    NvBufSurfaceColorFormat format = surf->surfaceList[batch_idx].colorFormat;

    cv::Mat img;
    if (format == NVBUF_COLOR_FORMAT_RGBA) {
        img = cv::Mat(height, width, CV_8UC4, data_ptr);
        g_print("NvBufSurfaceToCvMat: Converted RGBA to BGR.\n");
        cv::cvtColor(img, img, cv::COLOR_RGBA2BGR);
    } else if (format == NVBUF_COLOR_FORMAT_BGR) {
        img = cv::Mat(height, width, CV_8UC3, data_ptr);
        g_print("NvBufSurfaceToCvMat: Direct BGR format.\n");
    } else if (format == NVBUF_COLOR_FORMAT_NV12) {
        img = cv::Mat(height * 3 / 2, width, CV_8UC1, data_ptr);
        g_print("NvBufSurfaceToCvMat: Converted NV12 to BGR.\n");
        cv::cvtColor(img, img, cv::COLOR_YUV2BGR_NV12);
    } else {
        g_print("NvBufSurfaceToCvMat Warning: Unsupported NvBufSurface color format: %d. Returning empty Mat.\n", format);
        return cv::Mat();
    }
    return img;
}

// --- Custom Context Initialization Function ---
extern "C" bool NvDsInferContextInit(NvDsInferContextInitParams* init_params) {
    g_print("NvDsInferContextInit: Custom preprocessor initialized.\n");
    return true;
}

// --- Custom Context De-initialization Function ---
extern "C" void NvDsInferContextDeinit(NvDsInferContextInitParams* init_params) {
    g_print("NvDsInferContextDeinit: Custom preprocessor deinitialized.\n");
}


// --- The Core Custom Preprocessing Function ---
extern "C" NvDsInferStatus NvDsInferCustomPreprocess(
    NvBufSurface* in_surf,
    NvBufSurface* out_surf,
    NvDsInferContextInitParams* init_params,
    NvDsBatchMeta* batch_meta
) {
    g_print("NvDsInferCustomPreprocess: Entered for batch_size=%d\n", batch_meta->num_frames_in_batch);

    if (!in_surf || !out_surf || !init_params || !batch_meta) {
        g_print("NvDsInferCustomPreprocess Error: Invalid arguments provided.\n");
        return NVDSINFER_CUSTOM_LIB_FAILED;
    }

    if (NvBufSurfaceMap(in_surf, -1, -1, NVBUF_MAP_READ_WRITE) != 0) {
        g_print("NvDsInferCustomPreprocess Error: Failed to map input NvBufSurface for read/write.\n");
        return NVDSINFER_CUSTOM_LIB_FAILED;
    }
    if (NvBufSurfaceMap(out_surf, -1, -1, NVBUF_MAP_READ_WRITE) != 0) {
        g_print("NvDsInferCustomPreprocess Error: Failed to map output NvBufSurface for writing.\n");
        NvBufSurfaceUnMap(in_surf, -1, -1);
        return NVDSINFER_CUSTOM_LIB_FAILED;
    }

    guint infer_output_batch_counter = 0;

    for (GList* l_frame_meta = batch_meta->frame_meta_list; l_frame_meta != nullptr; l_frame_meta = l_frame_meta->next) {
        NvDsFrameMeta* frame_meta = reinterpret_cast<NvDsFrameMeta*>(l_frame_meta->data);

        guint current_frame_batch_idx = frame_meta->batch_id;
        g_print("NvDsInferCustomPreprocess: Processing Frame %u (batch_idx %u).\n", frame_meta->frame_num, current_frame_batch_idx);

        cv::Mat input_frame_mat = nvbufsurface_to_cvmat(in_surf, current_frame_batch_idx);
cv::Mat input_frame_mat = nvbufsurface_to_cvmat(in_surf, current_frame_batch_idx);
        if (input_frame_mat.empty()) {
            g_print("NvDsInferCustomPreprocess Error: Could not convert NvBufSurface to OpenCV Mat for frame batch_id %u. Skipping this frame.\n", current_frame_batch_idx);
            GList* temp_obj_list = frame_meta->obj_meta_list;
            while(temp_obj_list) {
                if (infer_output_batch_counter < out_surf->batchSize) {
                    void* dest_ptr_obj = out_surf->surfaceList[infer_output_batch_counter].dataPtr;
                    if (dest_ptr_obj) {
                        memset(dest_ptr_obj, 0, EMBEDDING_INPUT_WIDTH * EMBEDDING_INPUT_HEIGHT * EMBEDDING_INPUT_CHANNELS * sizeof(float));
                    }
                } else {
                    g_print("NvDsInferCustomPreprocess Error: Output buffer index out of bounds or null when frame conversion failed.\n");
                }
                infer_output_batch_counter++;
                temp_obj_list = temp_obj_list->next;
            }
            continue;
        }

        GList* obj_meta_list = frame_meta->obj_meta_list;
        for (GList* l_obj_meta = obj_meta_list; l_obj_meta != nullptr; l_obj_meta = l_obj_meta->next) {
            NvDsObjectMeta* obj_meta = reinterpret_cast<NvDsObjectMeta*>(l_obj_meta->data);
            g_print("NvDsInferCustomPreprocess: Processing Object %u (class_id %u).\n", obj_meta->object_id, obj_meta->class_id);

            if (obj_meta->class_id != 0) {
                g_print("NvDsInferCustomPreprocess: Skipping non-face object (class_id %u).\n", obj_meta->class_id);
                if (infer_output_batch_counter < out_surf->batchSize) {
                    void* dest_ptr_obj = out_surf->surfaceList[infer_output_batch_counter].dataPtr;
                    if (dest_ptr_obj) {
                        memset(dest_ptr_obj, 0, EMBEDDING_INPUT_WIDTH * EMBEDDING_INPUT_HEIGHT * EMBEDDING_INPUT_CHANNELS * sizeof(float));
                    }
                } else {
                    g_print("NvDsInferCustomPreprocess Error: Output buffer index out of bounds or null for non-face object.\n");
                }
                infer_output_batch_counter++;
                continue;
            }

            float* landmark_data = nullptr;
            // CRITICAL FIX: Landmark SGIE's unique ID is 3
            NvDsInferTensorMeta *tensor_meta = (NvDsInferTensorMeta *)nvds_get_user_meta_data_for_unique_id(
                                                obj_meta->obj_user_meta_list, 3);

            if (tensor_meta) {
                g_print("NvDsInferCustomPreprocess: Found tensor_meta from unique_id %u. Num output layers: %u.\n", tensor_meta->unique_id, tensor_meta->num_output_layers);
                for (guint i = 0; i < tensor_meta->num_output_layers; ++i) {
                    NvDsInferLayerInfo* layer_info = &(tensor_meta->output_layers[i]);
                    g_print("NvDsInferCustomPreprocess: Checking Layer '%s' with %u elements.\n", layer_info->layerName, layer_info->inferDims.numElements);

                    if (strcmp(layer_info->layerName, "fc1") == 0) {
                        landmark_data = reinterpret_cast<float*>(layer_info->buffer);
                        g_print("NvDsInferCustomPreprocess: Found 'fc1' layer. Landmark data buffer: %p.\n", (void*)landmark_data);
                        break;
                    }
                }
            }

            if (!landmark_data) {
                g_print("NvDsInferCustomPreprocess Warning: No landmark data (from GIE ID 3) found for object ID %u in frame %u. Filling output slot with zeros.\n",
                          obj_meta->object_id, frame_meta->frame_num);
                if (infer_output_batch_counter < out_surf->batchSize) {
                    void* dest_ptr_obj = out_surf->surfaceList[infer_output_batch_counter].dataPtr;
                    if (dest_ptr_obj) {
                        memset(dest_ptr_obj, 0, EMBEDDING_INPUT_WIDTH * EMBEDDING_INPUT_HEIGHT * EMBEDDING_INPUT_CHANNELS * sizeof(float));
                    }
                } else {
                    g_print("NvDsInferCustomPreprocess Error: Output buffer index out of bounds or null for object without landmarks.\n");
                }
                infer_output_batch_counter++;
                continue;
            }

            cv::Rect face_roi(
                static_cast<int>(obj_meta->rect_params.left),
                static_cast<int>(obj_meta->rect_params.top),
                static_cast<int>(obj_meta->rect_params.width),
                static_cast<int>(obj_meta->rect_params.height)
            );
            g_print("NvDsInferCustomPreprocess: Face ROI: (x:%d, y:%d, w:%d, h:%d).\n", face_roi.x, face_roi.y, face_roi.width, face_roi.height);

            face_roi = face_roi & cv::Rect(0, 0, input_frame_mat.cols, input_frame_mat.rows);
            if (face_roi.empty()) {
                g_print("NvDsInferCustomPreprocess Warning: Empty or invalid face ROI for object ID %u. Filling output slot with zeros.\n", obj_meta->object_id);
                if (infer_output_batch_counter < out_surf->batchSize) {
                    void* dest_ptr_obj = out_surf->surfaceList[infer_output_batch_counter].dataPtr;
                    if (dest_ptr_obj) {
                        memset(dest_ptr_obj, 0, EMBEDDING_INPUT_WIDTH * EMBEDDING_INPUT_HEIGHT * EMBEDDING_INPUT_CHANNELS * sizeof(float));
                    }
                } else {
                    g_print("NvDsInferCustomPreprocess Error: Output buffer index out of bounds or null for invalid ROI.\n");
                }
                infer_output_batch_counter++;
                continue;
            }

            cv::Mat face_crop = input_frame_mat(face_roi).clone();
            g_print("NvDsInferCustomPreprocess: Face crop size: %dx%d.\n", face_crop.cols, face_crop.rows);

            cv::Point2f src_pts[3];
            cv::Point2f dst_pts[3];

            src_pts[0] = cv::Point2f(landmark_data[0] - face_roi.x, landmark_data[1] - face_roi.y);
            src_pts[1] = cv::Point2f(landmark_data[2] - face_roi.x, landmark_data[3] - face_roi.y);
            src_pts[2] = cv::Point2f(landmark_data[4] - face_roi.x, landmark_data[5] - face_roi.y);
            g_print("NvDsInferCustomPreprocess: Source Points: (%.2f,%.2f), (%.2f,%.2f), (%.2f,%.2f)\n",
                    src_pts[0].x, src_pts[0].y, src_pts[1].x, src_pts[1].y, src_pts[2].x, src_pts[2].y);

            dst_pts[0] = cv::Point2f(TARGET_LANDMARKS[0], TARGET_LANDMARKS[1]);
            dst_pts[1] = cv::Point2f(TARGET_LANDMARKS[2], TARGET_LANDMARKS[3]);
            dst_pts[2] = cv::Point2f(TARGET_LANDMARKS[4], TARGET_LANDMARKS[5]);
            g_print("NvDsInferCustomPreprocess: Target Points: (%.2f,%.2f), (%.2f,%.2f), (%.2f,%.2f)\n",
                    dst_pts[0].x, dst_pts[0].y, dst_pts[1].x, dst_pts[1].y, dst_pts[2].x, dst_pts[2].y);

            cv::Mat affine_matrix = cv::getAffineTransform(src_pts, dst_pts);
            g_print("NvDsInferCustomPreprocess: Affine Matrix computed.\n");

            cv::Mat aligned_face;
            aligned_face.create(EMBEDDING_INPUT_HEIGHT, EMBEDDING_INPUT_WIDTH, CV_8UC3);
            cv::warpAffine(face_crop, aligned_face, affine_matrix, aligned_face.size(),
                           cv::INTER_LINEAR, cv::BORDER_CONSTANT, cv::Scalar(0, 0, 0));
            g_print("NvDsInferCustomPreprocess: Face aligned to %dx%d.\n", aligned_face.cols, aligned_face.rows);

            cv::Mat final_input_tensor;
            cv::cvtColor(aligned_face, final_input_tensor, cv::COLOR_BGR2RGB);
            g_print("NvDsInferCustomPreprocess: Converted to RGB.\n");

            final_input_tensor.convertTo(final_input_tensor, CV_32FC3, 1.0/127.5, -1.0);
            g_print("NvDsInferCustomPreprocess: Converted to FP32 and normalized to [-1, 1].\n");

            size_t bytes_per_image = EMBEDDING_INPUT_WIDTH * EMBEDDING_INPUT_HEIGHT * EMBEDDING_INPUT_CHANNELS * sizeof(float);

            if (infer_output_batch_counter < out_surf->batchSize) {
                void* dest_ptr_obj = out_surf->surfaceList[infer_output_batch_counter].dataPtr;
                if (dest_ptr_obj) {
                    memcpy(dest_ptr_obj, final_input_tensor.data, bytes_per_image);
                    g_print("NvDsInferCustomPreprocess: Copied aligned tensor to output NvBufSurface at index %u.\n", infer_output_batch_counter);
                } else {
                    g_print("NvDsInferCustomPreprocess Error: Output buffer at index %u is null.\n", infer_output_batch_counter);
                }
            } else {
                g_print("NvDsInferCustomPreprocess Error: Attempted to copy aligned face to out-of-bounds output buffer index: %u. Max: %u.\n",
                          infer_output_batch_counter, out_surf->batchSize);
            }

            infer_output_batch_counter++;
        }
    }

    NvBufSurfaceUnMap(in_surf, -1, -1);
    NvBufSurfaceUnMap(out_surf, -1, -1);
    g_print("NvDsInferCustomPreprocess: Exiting. Total aligned objects processed in this batch: %u\n", infer_output_batch_counter);

    return NVDSINFER_SUCCESS;
}

In the customized library, there are missing functions and wrongNvDsInferCustomPreprocess function prototype.

The right function prototype(form batch to tensor):

nvdspreprocess->custom_tensor_function =
            dlsym_ptr<NvDsPreProcessStatus(CustomCtx *, NvDsPreProcessBatch *, NvDsPreProcessCustomBuf *&,
                                      CustomTensorParams &, NvDsPreProcessAcquirer *)>
            (nvdspreprocess->custom_lib_handle, nvdspreprocess->custom_tensor_function_name.c_str());

In addition, initLib/deInitLib is necessary

/**
 * custom library initialization function
 */
extern "C"
CustomCtx *initLib(CustomInitParams initparams);

/**
 * custom library deinitialization function
 */
extern "C"
void deInitLib(CustomCtx *ctx);

You can refer to
/opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-3d-action-recognition/custom_sequence_preprocess/sequence_image_process.h

Besides, there is a block diagram for nvdspreprocess.

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.