• Hardware Platform (Jetson / GPU) Jetson
• DeepStream Version 6.2
• JetPack Version (valid for Jetson only) 3.1
• TensorRT Version8.5.2.2
**• Issue Type( questions, new requirements, bugs)**questions
Hi, i want deploy a classification network on jetson-orin using deepstream 6.2. So, the pipline of this situation is kind of like: source->decoder->classification network->osd->sink. The primary mode is a classification network instead a object detector. The classification network will use full image to get the classification result. My question is how do i get the classifition result from the pipline without object detection? I didn’t see any data structure could apply this imformation cause i have find in NvDsBatchMeta, NvDsFrameMeta, NvDsLabelInfo,NvDsUserMeta etc.
pipline.py (8.6 KB)
config.txt (421 Bytes)
config_preprocess.txt (3.3 KB)
please refer to DeepStream SDK FAQ - #25 by fanzh
nvinfer plugin is opensource. In function attach_metadata_classifier, one object meta will be added if classification model works as pgie. you can get classification information from NvDsObjectMeta’s NvDsClassifierMetaList.
Thanks for reply！I have read the FAQ and looking for some other replies. The answer I have find is
seems that when I use the classification model as pgie and set process-mode to 1, the deepstream will attach an object to whole frame automatically and the classification result will in frame_meta->ojb_meta_list->classifier_meta_list. But I have found that the ojb_meta_list is None and the classifier_meta_list is not exist.
Here is the config.txt
[property]
gpu-id=0
net-scale-factor=1
onnx-file=./vis_net.onnx
model-engine-file=./vis_net_1.engine
labelfile-path=./label_vis.txt
force-implicit-batch-dim=1
batch-size=1
network-mode=0
process-mode=1
network-type=1
interval=0
gie-unique-id=1
model-color-format=0
output-blob-names=output
input-tensor-from-meta=1
output-tensor-meta=1
classifier-threshold=0.1
please set input-tensor-from-meta to 0, please refer to the nvinfer for parameter explanation.
I want use preprocess plugin, so I have to set input-tensor-from-meta to 1.
I have tried set input-tensor-from-meta to 0, and the result is still None.
For classification, in my understand, one should set the classification network as the primary mode in main() function and set process-mode=1, network-type=1, model-engine-file and labelfile-path correctly in config.txt file. What else should I pay attention to?
https://forums.developer.nvidia.com/t/how-to-make-metadata-probe-for-classification-only-pipeline/171126
I also tired above method, and still got None result.
can you see the output bbox? in the FAQ above, you can see a binding box on the output picture, if there is no object meta, the nvdsosd will not draw the bbox.
I can’t see the bbox in picture
I modify the deepstream-test2 to get the result. Here is the modified files.
test.py (14.6 KB)
dstest2_sgie1_config.txt (3.2 KB)
GitHub - nvqiezi/deepstream-classify-pgie-test: use classification model as pgie
I have tried this project and got the following result:
It seems that something wrong with the video loading. Then, I changed another video, and got the following result
Seems still wrong with the video loading
if classification works as pgie, obj_meta.class_id will be -1. you can see the logics in function attach_metadata_classifier. you can get the classfication result from NvDsLabelInfo’s result_class_id in NvDsClassifierMeta.
in your python code, "obj_counter[obj_meta.class_id] += 1” will crash because obj_meta.class_id is -1.
I have print the result_class_id and obj_meta.class_id, the results are:
yes, the obj_meta.class_id is -1, but the values of result_class_id are confused me
batch_meta = pyds.gst_buffer_get_nvds_batch_meta(hash(gst_buffer))
l_batch_class_meta = batch_meta.classifier_meta_pool
class_meta_data = pyds.NvDsClassifierMeta.cast(l_batch_class_meta)
l_batch_label = class_meta_data.label_info_list
label_info = pyds.NvDsLabelInfo.cast(l_batch_label)
print('result_class_id: {}'.format(label_info.result_class_id))
l_frame = batch_meta.frame_meta_list
while l_frame is not None:
try:
frame_meta = pyds.NvDsFrameMeta.cast(l_frame.data)
except StopIteration:
break
frame_number=frame_meta.frame_num
num_rects = frame_meta.num_obj_meta
l_obj=frame_meta.obj_meta_list
print(frame_number)
while l_obj is not None:
try:
obj_meta=pyds.NvDsObjectMeta.cast(l_obj.data)
except StopIteration:
break
print('obj_meta.class_id: {}'.format(obj_meta.class_id))
try:
l_obj=l_obj.next
except StopIteration:
break
NvDsClassifierMeta is in NvDsObjectMeta. please refer to \opt\nvidia\deepstream\deepstream-6.2\sources\apps\sample_apps\deepstream-preprocess-test\deepstream_preprocess_test.cpp for how to access NvDsClassifierMeta.
I have modify the deepstream-test2 according to deepstream_preprocess_test.cpp, the result is:
my code is :
osd_sink_pad_buffer_probe (GstPad * pad, GstPadProbeInfo * info,
gpointer u_data)
{
GstBuffer *buf = (GstBuffer *) info->data;
guint num_rects = 0;
NvDsObjectMeta *obj_meta = NULL;
guint vehicle_count = 0;
guint person_count = 0;
NvDsMetaList * l_frame = NULL;
NvDsMetaList * l_obj = NULL;
NvDsBatchMeta *batch_meta = gst_buffer_get_nvds_batch_meta (buf);
std::cout << "batch in" << std::endl;
for (l_frame = batch_meta->frame_meta_list; l_frame != NULL;
l_frame = l_frame->next) {
std::cout << "frame in" << std::endl;
NvDsFrameMeta *frame_meta = (NvDsFrameMeta *) (l_frame->data);
int offset = 0;
for (l_obj = frame_meta->obj_meta_list; l_obj != NULL;
l_obj = l_obj->next)
{
std::cout << "object in" << std::endl;
obj_meta = (NvDsObjectMeta *) (l_obj->data);
NvDsMetaList *l_classifier = NULL;
for (l_classifier = obj_meta->classifier_meta_list; l_classifier != NULL;
l_classifier = l_classifier->next)
{
std::cout << "class in" << std::endl;
NvDsClassifierMeta *classifier_meta = (NvDsClassifierMeta *)(l_classifier->data);
NvDsLabelInfoList *l_label;
for (l_label = classifier_meta->label_info_list; l_label != NULL;
l_label = l_classifier->next)
{
std::cout << "label in" << std::endl;
NvDsLabelInfo *label_info = (NvDsLabelInfo *)l_label->data;
std::cout << "label info:" << label_info << std::endl;
}
}
}
}
g_print ("Frame Number = %d Number of objects = %d "
"Vehicle Count = %d Person Count = %d\n",
frame_number, num_rects, vehicle_count, person_count);
frame_number++;
return GST_PAD_PROBE_OK;
}
Couldn’t access the attributes of the object
I mean, you can refer to deepstream-preprocess-test for how to access NvDsClassifierMeta. in the python code, you already can access NvDsObjectMeta . you can port the the logic of accessing NvDsObjectMeta 's NvDsClassifierMeta to the python code.
Did you mean in this way?
l_batch_class_meta = batch_meta.classifier_meta_pool
class_meta_data = pyds.NvDsClassifierMeta.cast(l_batch_class_meta)
l_batch_label = class_meta_data.label_info_list
label_info = pyds.NvDsLabelInfo.cast(l_batch_label)
print('result_class_id: {}'.format(label_info.result_class_id))
the output still confusing:
NvDsClassifierMeta is in NvDsObjectMeta. from this code, it is using a new NvDsClassifierMeta. please port this logics:
NvDsMetaList *l_classifier = NULL;
for (l_classifier = obj_meta->classifier_meta_list; l_classifier != NULL;
l_classifier = l_classifier->next)
Am i right?
l_batch_obj_meta_pool = batch_meta.obj_meta_pool
obj_meta_data = pyds.NvDsObjectMeta.cast(l_batch_obj_meta_pool)
print('classifier_meta_list : {}'.format(obj_meta_data.classifier_meta_list ))
obj_class_meta = obj_meta_data.classifier_meta_list
print("obj_class_meta: {}".format(type(obj_class_meta)))
class_meta_data = pyds.NvDsClassifierMeta.cast(obj_class_meta)
print(class_meta_data.label_info_list)
can’t see the right results
no, it is wrong. in the python code, you already can access obj_meta. you should loop NvDsClassifierMeta from obj_meta’s classifier_meta_list.
please refer to the struct of NvDsObjectMeta in \opt\nvidia\deepstream\deepstream-6.2\sources\includes\nvdsmeta.h.
Thanks for your patience. I get the result from the following code, hope it is useful for new commer.
gst_buffer = info.get_buffer()
batch_meta = pyds.gst_buffer_get_nvds_batch_meta(hash(gst_buffer))
l_frame = batch_meta.frame_meta_list
while l_frame is not None:
try:
frame_meta = pyds.NvDsFrameMeta.cast(l_frame.data)
except StopIteration:
break
frame_number=frame_meta.frame_num
num_rects = frame_meta.num_obj_meta
l_obj=frame_meta.obj_meta_list
print('frame number: {}'.format(frame_number))
while l_obj is not None:
try:
obj_meta=pyds.NvDsObjectMeta.cast(l_obj.data)
except StopIteration:
break
obj_class_list = pyds.NvDsClassifierMeta.cast(obj_meta.classifier_meta_list.data)
label_info_list = pyds.NvDsLabelInfo.cast(obj_class_list.label_info_list.data)
print('label_info_list.result_class_id: {}'.format(label_info_list.result_class_id ))
try:
l_obj=l_obj.next
except StopIteration:
break
try:
l_frame=l_frame.next
except StopIteration:
break
return Gst.PadProbeReturn.OK
The classification result is in label_info_list.result_class_id
Thanks for sharing, please open a new topic if you encounter new problems.