Please provide complete information as applicable to your setup.
• Hardware Platform (Jetson / GPU) : RTX 3050, x86_64 • DeepStream Version : 7.1 • JetPack Version (valid for Jetson only) • TensorRT Version: 10.3.0.26 • NVIDIA GPU Driver Version (valid for GPU only): 565.57.01 • Issue Type( questions, new requirements, bugs) : I’m trying to convert the 3d-action-recognition pipeline in C++ to Python but I cant seem to extract the nvinfer output in pgie_src_pad function which I suppose is because meta_type NVDS_PREPROCESS has no binding in python hence I cant access the tensor_output. • How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing) • Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)
I just want guidance in the right direction that am I correct in assuming that the problem lies in the preprocess meta_type for python (In which case I’ll build the binding as specified in the deepstream_python repo) or does the problem lie somewhere else in which case all help is appreciated.
PS: I’ve currently made the pipeline in Python (as best I could from the C++ sample provided) which is running fine except the pgie_src_pad function where I’m unable to extract nvinfer data.
In its pgie_src_pad_buffer_probe function where they extract the nvinfer output from NvDsUserMeta type. Below is the part of the function where they extract tensor meta(nvinfer output) from GstNvDsPreProcessBatchMeta.
if (user_meta->base_meta.meta_type == NVDS_PREPROCESS_BATCH_META)
{
GstNvDsPreProcessBatchMeta *preprocess_batchmeta =
(GstNvDsPreProcessBatchMeta *)(user_meta->user_meta_data);
std::string model_dims = "";
if (preprocess_batchmeta->tensor_meta) {
if (preprocess_batchmeta->tensor_meta->tensor_shape.size() == MODEL_3D_SHAPES) {
model_dims = "3D: AR - ";
} else {
model_dims = "2D: AR - ";
}
}
Converting the same to python, user_meta.base_meta.meta_type gives me NvDsMeta.??? as the meta type which led me to believe that there must be no python binding for the NvDsPreprocessBatchMeta which was indeed the case when i looked at the official list.
So my main query is whether I am correct in assuming that creating my own python binding of NvDsPreprocessBatchMeta as specified here will solve my problem or is there something I’m missing.
Please let me know if you require any more details or clarity regarding the same.
Yes. You are right. There are no python binding for the NVDS_PREPROCESS_BATCH_META currently. You have to bind that yourself by referring the Guide in our bindings.
Created bindnvdspreprocess_meta.hpp in ./include/bin
Created bindnvdspreprocess_meta.cpp in ./src which is:
// NvDsPreProcessBatchMeta
#include "bind_string_property_definitions.h"
#include "bindnvdspreprocess_meta.hpp" // Include the corresponding header file
namespace py = pybind11;
namespace pydeepstream {
void bindnvdspreprocess_meta(py::module &m) {
py::class_<NvDsPreProcessBatchMeta>(m, "NvDsPreProcessBatchMeta",
pydsdoc::nvdspreprocessbatchmetadoc::NvDsPreProcessBatchMetaDoc::descr)
.def(py::init<>());
.def_readwrite("tensor_meta", &NvDsInferTensorMeta::tensor_meta);
.def_readwrite("roi_meta", &NvDsUserMeta::roi_meta);
.def("cast",
[](void *data) { // Function definition
return (NvDsPreProcessBatchMeta *) data; // Cast void* to NvDsObjectMeta*
},
py::return_value_policy::reference, // Ownership of return object must remain with C++
pydsdoc::nvdspreprocessbatchmetadoc::NvDsPreProcessBatchMetaDoc::cast); // Docstring
}
}
Created the docstrings nvdspreprocess_metadocs.h under ./docstrings containing
#pragma once // Ensure header file is included only once
namespace pydsdoc
{
namespace nvdspreprocessbatchmetadoc
{
namespace NvDsPreProcessBatchMetaDoc
{
constexpr const char* descr = R"pyds(
Holds information of object metadata in the frame.
:ivar tensor_meta: :class:`NvDsInferTensorMeta`, tensor_meta
:ivar roi_meta: the parent :class:`NvDsUserMeta` object.
Example usage:
::
batch_meta = pyds.gst_buffer_get_nvds_batch_meta(hash(gst_buffer))
l_user_meta = batch_meta.batch_user_meta_list
while l_user_meta:
try:
user_meta = pyds.NvDsUserMeta.cast(l_user_meta.data)
except StopIteration:
break
# Check if the meta is related to preprocessing
print(user_meta.user_meta)
if user_meta.base_meta.meta_type == pyds.NvDsMetaType.NVDS_PREPROCESS_BATCH_META:
preprocess_batchmeta = pyds.GstNvDsPreProcessBatchMeta.cast(user_meta.data)
model_dims = ""
if preprocess_batchmeta.tensor_meta:
if len(preprocess_batchmeta.tensor_meta.tensor_shape) == MODEL_3D_SHAPES:
model_dims = "3D: AR - "
else:
model_dims = "2D: AR - "
# Iterate over each ROI (Region of Interest)
for roi_meta in preprocess_batchmeta.roi_vector:
# Iterate over tensor metadata (from nvinfer)
l_user = roi_meta.roi_user_meta_list
while l_user:
try:
user_meta = pyds.NvDsUserMeta.cast(l_user.data)
except StopIteration:
break
# Check if this metadata contains tensor output
if user_meta.base_meta.meta_type == pyds.NvDsMetaType.NVDSINFER_TENSOR_OUTPUT_META:
tensor_meta = pyds.NvDsInferTensorMeta.cast(user_meta.user_meta_data)
try:
l_user=l_user.next
except StopIteration:
break)pyds";
}
}
}
Now according to the guide,
Any new bindings source file created must be added to ../docs/bindings_file_list.txt. After running ../docs/parse_bindings.py on the updated list, check ../docs/PYTHON_API to find the new toctree rst file. Add the file name to ../docs/index.rst. In our case, we added bindnvdspreprocess_meta.cpp to the file list, which generated the file ../docs/PYTHON_API/NvDsPreProcessBatchMeta/NvDsPreProcessBatchMeta_toc.rst
Now running my python sample still gives me meta_type as NvDsMetaType.???.
just added .value("NVDS_PREPROCESS_BATCH_META", NVDS_PREPROCESS_BATCH_META, pydsdoc::nvdspreprocessbatchmetadoc::NvDsPreProcessBatchMetaDoc::NVDS_PREPROCESS_BATCH_META) in bindgstnvdsmeta.cpp and included bindnvdspreprocess_meta.hpp as the header file.
Also added constexpr const char* NVDS_PREPROCESS_BATCH_META=R"pyds(metadata type to be set for extracting nvinfer data)pyds"; in gstnvdsmetadoc.h
After running parse_bindings.py, the 3d action script doesnt recognize NVDS_PREPROCESS_BATCH_META as a meta_type.
After your binding code is complete, do you recompile the whl file and install it? I just tried with our deepstream-preprocess-test, it can be recognized normally after the binding is added. The diff is like below. diff.txt (4.2 KB)
in the diff.txt file you shared, I see that you’ve made changes to the nvdsmetadoc.h and bindnvdsmeta.cpp, but in the previous reply you said to add the NVDS_PREPROCESS_BATCH_META in bindgstnvdsmeta.cpp file so I am a little confused.
Also in ./includes/nvdspreprocess_meta.h,
typedef struct
{
/** target unique ids for which meta is prepared */
std::vector<guint64> target_unique_ids;
/** pointer to tensor meta */
NvDsPreProcessTensorMeta *tensor_meta;
/** list of roi vectors per batch */
std::vector<NvDsRoiMeta> roi_vector;
/** pointer to buffer from scaling pool*/
void *private_data;
} GstNvDsPreProcessBatchMeta;
Here the type is GstNvDsPreProcessBatchMeta, so in bindnvdspreprocess_meta.cpp, should the class be GstNvDsPreProcessBatchMeta or NvDsPreProcessBatchMeta.
This file also has nvds_roi_meta.h header file which also doesnt have a python plugin so do I have to create it’s bindings as well or not.
You need to refer to our C code to bind your own meta.
The bindgstnvdsmeta.cpp is bound for the gstnvdsmeta.h and the bindnvdsmeta.cpp is bound for the nvdsmeta.h.
GstNvDsPreProcessBatchMeta.
Yes, you need to do the binding for the roi vector. You should to bind each struct you want to use.
I’ve successfully bound GstNvDsPreProcessBatchMeta and developed a 3D Action Recognition pipeline in Python. I know many community members have faced challenges with these bindings and the action recognition task, so I wanted to ask if there’s a possibility to add these preprocess bindings and the complete action recognition pipeline to the official repo.
I’m more than happy to share my codebase and collaborate to help improve it for everyone.
Sure. You can share your codebase here or “Message” to me directly by clicking my icon. We’ll review and test that. We will discuss internally whether we need to officially release it later. Thanks in advance.
Hi @ereshmittal5 , we’ll add the binding related code to our project. Could you just attach the sample code here for others to refer to?