Nvinfer silently skips inference when using input-tensor-meta=1 with nvdspreprocess for embedding SGIE

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Please provide complete information as applicable to your setup.

• Hardware Platform GPU A6000 ADA
• DeepStream Version 8.0

• NVIDIA GPU Driver Version 13.0
Hi ,

I’ve been banging my head against this for a few days now and hoping someone can help point me in the right direction.

I’m trying to set up an embedding model (ResNet50 headless) as a secondary GIE in DeepStream 8.0, with a custom nvdspreprocess library that applies segmentation masks to detected objects before running the embedding inference. The idea is to mask out the background and only pass the segmented object to the embedding model .

The problem is that when I enable input-tensor-meta=1 on the nvinfer element, it completely skips inference - no errors, no warnings, just silently produces nothing. Without input-tensor-meta set, the model runs fine and I get output tensors.

Here’s what I’ve verified so far:

  1. The tensor IS being created and passed correctly - I can see GstNvDsPreProcessBatchMeta in the batch_user_meta_list when I probe the nvinfer sink pad
  2. target_unique_ids matches my gie-unique-id (both are 7)
  3. tensor_meta has a valid buffer pointer
  4. My custom tensor prep function is working - I’m dumping the tensor to a PPM file and it looks correct
  5. Both input-tensor-meta and output-tensor-meta are set to 1 (verified with g_object_get)

But on the source pad of nvinfer, there’s zero NVDSINFER_TENSOR_OUTPUT_META. It’s like nvinfer just ignores the tensor entirely.

My configs:

nvinfer config:

[property]
gpu-id=0
gie-unique-id=7
onnx-file=embedding_model/resnet50_headless.onnx
model-engine-file=embedding_model/resnet50_headless.onnx_b8_gpu0_fp32.engine
batch-size=8
net-scale-factor=1.0
offsets=123.675;116.28;103.53
infer-dims=3;224;224
network-type=100
output-tensor-meta=1
input-tensor-meta=1
operate-on-gie-id=1
process-mode=2
operate-on-class-ids=0
network-mode=2

nvdspreprocess config:

[property]
enable=1
target-unique-ids=7
unique-id=8
network-input-order=0
process-on-frame=0
operate-on-gie-id=1
maintain-aspect-ratio=1
symmetric-padding=1
processing-width=224
processing-height=224
network-input-shape=8;3;224;224
scaling-buf-pool-size=6
tensor-buf-pool-size=6
network-color-format=0
tensor-data-type=0
tensor-name=input
scaling-pool-memory-type=1
scaling-pool-compute-hw=0
scaling-filter=0
custom-lib-path=/opt/nvidia/deepstream/deepstream-8.0/kafka-cpp/gst-nvdspreprocess/nvdspreprocess_lib/libcustom2d_preprocess.so
custom-tensor-preparation-function=CustomTensorPreparation

[user-configs]
pixel-normalization-factor=1.0
offsets=123.675;116.28;103.53

Sink pad probe shows:

=== NVINFER SINK PAD PROBE ===
num_frames_in_batch: 2
  ✓ GstNvDsPreProcessBatchMeta FOUND!
    target_unique_ids count: 1
    target_unique_id[0]: 7
    ✓ tensor_meta EXISTS!
    tensor_meta->meta_id: 0
    tensor_meta->gpu_id: 0
    tensor_meta->raw_tensor_buffer: 0x7c5017200000
    roi_vector size: 1

Source pad probe shows:

=== NVINFER SOURCE PAD PROBE ===
num_frames_in_batch: 2
  Batch User meta type: 28  (NVDS_PREPROCESS_BATCH_META - carried over)
  Batch User meta type: 28
  ...
  *** NO NVDSINFER_TENSOR_OUTPUT_META! ***

I did notice one thing - the docs say “ensure that the batch-size of nvinfer is equal to the sum of ROIs”. My batch-size is 8 but roi_vector size varies depending on how many objects are detected. Could this be the issue? If so, how do you handle variable object counts in SGIE mode with nvdspreprocess?

I tried setting batch-size=1 but then I get a different issue - and I’d need to regenerate the ONNX with dynamic batch support anyway.

Has anyone successfully used nvdspreprocess with input-tensor-meta for a secondary GIE? Any pointers would be really appreciated.

Running on dGPU with DeepStream 8.0.

Thanks!

4

If I understand correctly, your pipeline looks like this, right?

PGIE --> nvdspreprocess(object to tensor for sgie) --> SGIE (input-tensor-meta=1, output-tensor-meta=1)

Are the name and datatype of your SGIE input layer correct? You can use this tool to check the input layer name of ONNX.

We also provide a sample application similar to your pipeline, which you can refer to.

5

Thank you to reply ,yes you are understand correctly pipeline ,and my model “input-name” are input not input_1;
i am use sgie model as pretrain resnet50 with last layer with nn.Identity() for extract feacture to save to embeding format like input are image and output are 2048d; below code are export to onnx file with useing pretrain model

def export_resnet50():

resnet=models.resnet50(pretrained=True)

resnet.fc=nn.Identity()

resnet.eval()

dummpy_input=torch.randn(1,3,224,224)

print(“Exporting headless onnx model…”)

torch.onnx.export(resnet,

dummpy_input,“resnet50_headless.onnx”,

input_names=[“input”],

output_names=[“embedding”],

opset_version=16 ,

dynamic_axes={

‘input’: {0: ‘batch_size’},

‘embedding’: {0: ‘batch_size’}

    })
6
  1. Check if you are looking for outputmeta in the wrong location.Based on your configuration, the code should look something like the following.
for (NvDsMetaList *l = batch_meta->batch_user_meta_list; l; l = l->next) {
    NvDsUserMeta *user_meta = (NvDsUserMeta *)l->data;
    if (user_meta->base_meta.meta_type != NVDS_PREPROCESS_BATCH_META)
        continue;
    GstNvDsPreProcessBatchMeta *preproc = 
        (GstNvDsPreProcessBatchMeta *)user_meta->user_meta_data;
    
    for (auto &roi : preproc->roi_vector) {
        for (GList *l_roi = roi.roi_user_meta_list; l_roi; l_roi = l_roi->next) {
            NvDsUserMeta *roi_user = (NvDsUserMeta *)l_roi->data;
            if (roi_user->base_meta.meta_type == NVDSINFER_TENSOR_OUTPUT_META) {
                NvDsInferTensorMeta *tensor = 
                    (NvDsInferTensorMeta *)roi_user->user_meta_data;
                //  tensor->out_buf_ptrs_host[0] embedding
                printf("FOUND embedding! unique_id=%d, layers=%d\n",
                    tensor->unique_id, tensor->num_output_layers);
            }
        }
    }
}
  1. If the inference didn’t happen, I still suspect it’s a tensor-name mismatch. Could you share the logs? Check this log : "could not find input layer with name = input"
GST_DEBUG=nvinfer:5 your_app  > log.log 2>&1
  1. Try to debug nvinfer plugin. The main focus is on checking the following functions. gst_nvinfer_process_tensor_input/attach_tensor_output_meta
7

yes i add your code into my probe i am also share what i step follow,
first i update the config file config_infer_sgie_resnet_embeding.txt
network-type=100
input-tensor-from-meta=1
output-tensor-meta=1

and this pipeline link

 if(!gst_element_link_many( streammux, 

queue_infer_, 

primary_nvinference_, 

nvtracker_, 

nvdsanalytics_, 

preprocess_embedding,

embedding_nvinfere_,

tee_,

and this probe i define

GstPad *src_pad = gst_element_get_static_pad(embedding_nvinfere_, "src");

if (src_pad) {

gst_pad_add_probe(src_pad, GST_PAD_PROBE_TYPE_BUFFER,

sgie_src_pad_buffer_probe, nullptr, nullptr);

gst_object_unref(src_pad);

printf("Probe added to embedding_nvinfere_ src pad\n");

and this probe i define

GstPadProbeReturn

sgie_src_pad_buffer_probe (GstPad * pad, GstPadProbeInfo * info, gpointer u_data)

{

GstBuffer *buf = (GstBuffer *) info->data;

if (!buf) {

return GST_PAD_PROBE_OK;

    }

NvDsBatchMeta *batch_meta = gst_buffer_get_nvds_batch_meta (buf);

if (!batch_meta) {

return GST_PAD_PROBE_OK;

    }

    // 1. Iterate Frames

for (NvDsMetaList *l_frame = batch_meta->frame_meta_list; l_frame != NULL; l_frame = l_frame->next) {

NvDsFrameMeta *frame_meta = (NvDsFrameMeta *) (l_frame->data);

        // 2. Iterate Objects (Detected by primary YOLO)

for (NvDsMetaList *l_obj = frame_meta->obj_meta_list; l_obj != NULL; l_obj = l_obj->next) {

NvDsObjectMeta *obj_meta = (NvDsObjectMeta *) (l_obj->data);

            ;

            // 3. Iterate User Meta

for (NvDsMetaList *l_user = obj_meta->obj_user_meta_list; l_user != NULL; l_user = l_user->next) {

NvDsUserMeta *user_meta = (NvDsUserMeta *) (l_user->data);

                // 4. Isolate the Tensor Meta

if (user_meta->base_meta.meta_type == NVDSINFER_TENSOR_OUTPUT_META) {

NvDsInferTensorMeta *tensor_meta = (NvDsInferTensorMeta *) user_meta->user_meta_data;

                    // 5. Locate specific output layer

for (unsigned int i = 0; i < tensor_meta->num_output_layers; i++) {

NvDsInferLayerInfo *layer_info = &tensor_meta->output_layers_info[i];

                    //     printf("FOUND embedding! unique_id=%d, layers=%d ,layer name=%s\n ",

                    // tensor_meta->unique_id, tensor_meta->num_output_layers,tensor_meta->output_layers_info);

std::cout << "Founding embeding! uniquie_id = "<<  tensor_meta->unique_id << ",layers = "<< tensor_meta->num_output_layers 

<< " ,layer names = "<<layer_info->layerName << std::endl; 

                        // Match the exact output name from your ONNX export

if (strcmp(layer_info->layerName, "embedding") == 0) {

std::cout << layer_info->layerName << std::endl;

                            // 6. Extract the memory pointer

                            // out_buf_ptrs_host holds the CPU-accessible memory

float *embeddings = (float *) tensor_meta->out_buf_ptrs_host[i];

                            // Construct a C++ vector from the raw memory block

                            // 2048 is your known dimension. Hardcoding is fragile; 

                            // layer_info->inferDims.numElements is safer.

std::vector<float> embed_vec(embeddings, embeddings + 2048);

                            // ACTION REQUIRED: Push 'embed_vec' and 'obj_meta->object_id' 

                            // to a thread-safe std::queue or message broker here.

                            // DO NOT WRITE TO DISK OR DB IN THIS FUNCTION.

                        }

                    }

                }

            }

        }

    }

return GST_PAD_PROBE_OK;

}

when i set input-tensor-from-meta=0 in config file i got this logs

input-tensor-from-meta_disable.log (24.9 KB)

when i set input-tensor-from-meta=1 in config file i got this logs

input-tensor-from-meta_enable.log (12.2 KB)

8

Based on the code snippet you provided, the solution I offered above should solve your problem, as mentioned in the documentation.
NVDSINFER_TENSOR_OUTPUT_META can only be found in obj_user_meta_list when the sgie input is an object; when the input is a tensor(nvdspreprocess), it needs to be searched in roi_user_meta_list.

When operating in preprocessed tensor input mode (using nvdspreprocess plugin), the tensor metadata is accessible through NvDsPreProcessorBatchMeta - first look for GstNvDsPreProcessBatchMeta in batch_meta->batch_user_meta_list , then access the roi_vector which contains NvDsRoiMeta objects.

9

your right - I missed that section to read . Thank you issue resolve