Unable to access classifier output tensor metadatawhen using `nvdspreprocess` with `input-tensor-meta=1`

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Hello,

I am using DeepStream 7.1 with Python bindings and I am trying to access the output tensors of a classifier when using nvdspreprocess together with input-tensor-meta=1. My objective is to retrieve one classifier output tensor for each ROI generated by the preprocessing stage, but I have been unable to access the corresponding NvDsInferTensorMeta.

• Hardware Platform: Jetson Orin NX 16GB
• DeepStream Version: 7.1
• TensorRT Version: 10.5
• Issue Type: Question

My pipeline is:

nvdspreprocess -> nvinfer (classifier)

The nvdspreprocess plugin generates multiple ROIs per frame, and nvinfer is configured with:

input-tensor-meta=1
output-tensor-meta=1

My goal is to obtain the classifier output tensor corresponding to each ROI generated by nvdspreprocess.

After reading this forum post: Nvdspreprocess + nvinfer(unet) ，model output error

I understand that when input-tensor-meta=1 is enabled in DeepStream 7.1, the tensor metadata is attached as user metadata inside NvDsRoiMeta, which is then attached to the frame metadata.

Since I am using Python, I found that NvDsRoiMeta bindings are available in DeepStream 8.0 (bindroimeta.cpp) but not in the DeepStream 7.1 Python bindings. NVIDIA suggested porting the relevant binding code from DeepStream 8.0 and rebuilding pyds.

I have ported the NvDsRoiMeta bindings from DeepStream 8.0 into the DeepStream 7.1 deepstream_python_apps source code and successfully rebuilt the Python bindings.

Requirement details:

  • Module: nvdspreprocess + nvinfer + Python bindings (pyds)

  • Functionality: Access the NvDsInferTensorMeta generated by nvinfer for each ROI produced by nvdspreprocess.

  • Expected result: Retrieve the output tensor and associate it with its corresponding ROI from Python.

At this point, I would appreciate some guidance on how to traverse the metadata hierarchy and retrieve the NvDsInferTensorMeta associated with each ROI. A Python example would be especially helpful.

Thank you.

2

Hello @djovediaz!

Based on the title and content of your topic, it looks like it may receive better visibility and feedback in a different category. We took the liberty of moving it for you.

If this was an incorrect assessment, please send me a direct message.

Disclaimer: this moderation suggestion and message were generated with AI assistance.

5

I don’t think additional binding is necessary; please refer to this FAQ.

I recommend upgrading to the latest versions: JP-7.2 and DS-9.1.

6

Thank you for your reply.
We have tested the solution you suggested, but when using the pre-processor to generate multiple ROIs, the metadata is shared differently.
Furthermore, the version of Deepstream you recommend is not available for the Jetson Orin Nx, the last one is DS 7.1.

image
image621×219 12 KB

7

Sorry for overlooking the fact that you are using Orin NX and cannot upgrade to the latest DeepStream version. Bindings for NvDsRoiMeta are already supported in the pyds for DS-8.0; you can simply port this functionality to DS-7.1.

git clone --branch v1.2.0 --depth 1 \
    https://github.com/NVIDIA-AI-IOT/deepstream_python_apps.git
cd deepstream_python_apps
git apply /path/to/roi_classifier_tensor_meta_sample/0001-add-preprocess-roi-bindings.patch
cd bindings
python3 -m build
python3 -m pip install dist/pyds-1.2.0*.whl

0001-add-preprocess-roi-bindings.patch (18.7 KB)

Here is how to access NvDsRoiMeta. link

batch_user_meta_list
  -> GstNvDsPreProcessBatchMeta
  -> roi_vector[*]
  -> roi_user_meta_list
  -> NVDSINFER_TENSOR_OUTPUT_META
  -> NvDsInferTensorMeta

You can refer to code snippet below.

import ctypes
import numpy as np
import pyds

def iter_glist(node):
    while node is not None:
        yield node.data
        try:
            node = node.next
        except StopIteration:
            return


def find_tensor_meta(user_meta_list):
    for node in iter_glist(user_meta_list):
        user_meta = pyds.NvDsUserMeta.cast(node)
        if user_meta.base_meta.meta_type == pyds.NVDSINFER_TENSOR_OUTPUT_META:
            yield pyds.NvDsInferTensorMeta.cast(user_meta.user_meta_data)


def layer_to_numpy(layer):
    c_type, np_type = {
        pyds.NvDsInferDataType.FLOAT: (ctypes.c_float, np.float32),
        pyds.NvDsInferDataType.HALF: (ctypes.c_uint16, np.float16),
        pyds.NvDsInferDataType.INT8: (ctypes.c_int8, np.int8),
        pyds.NvDsInferDataType.INT32: (ctypes.c_int32, np.int32),
    }[layer.dataType]
    shape = tuple(layer.inferDims.d[i] for i in range(layer.inferDims.numDims))
    count = int(np.prod(shape))
    address = pyds.get_ptr(layer.buffer)
    values = np.ctypeslib.as_array(
        ctypes.cast(address, ctypes.POINTER(c_type)), shape=(count,)
    )
    return values.view(np_type).reshape(shape)


def read_roi_tensors(batch_meta):
    for node in iter_glist(batch_meta.batch_user_meta_list):
        batch_user_meta = pyds.NvDsUserMeta.cast(node)
        if batch_user_meta.base_meta.meta_type != pyds.NVDS_PREPROCESS_BATCH_META:
            continue

        preprocess = pyds.GstNvDsPreProcessBatchMeta.cast(
            batch_user_meta.user_meta_data
        )
        for roi in preprocess.roi_vector:
            roi = pyds.NvDsRoiMeta.cast(roi)
            for tensor in find_tensor_meta(roi.roi_user_meta_list):
                for index in range(tensor.num_output_layers):
                    layer = pyds.get_nvds_LayerInfo(tensor, index)
                    array = layer_to_numpy(layer)
                    print(layer.layerName, array.shape, array)
8

I checked the DeepStream this and Orin Nano documentation this;

There appears to be an error in the documentation, as the Orin Nano does support JP-7.2 and DS-9.1.