Hi, my question is - how does one access metadata when there are only classsification networks in pipeline?
NvDsFrameMeta seems to contain only object level inference results.
Is there an attribute one can access to get classification metadata when there is 1 or more classification networks in a pipeline (and no primary object detection)?
Good question, the classification meta should be also in the object meta even there is no detector model in the pipeline, you can refer gstnvinfer_meta_utils.cpp → attach_metadata_classifier for how nvinfer attach the classsification meta data.
if (nvinfer->process_full_frame) {
/* Attach only one object in the meta since this is a full frame
* classification. */
object_meta = nvds_acquire_obj_meta_from_pool (batch_meta);
Thank you for quick answer.
That’s the problem for me. I try to iterate over obj_meta_list and it turns out to be empty:
for (l_frame = batch_meta->frame_meta_list; l_frame != NULL;
l_frame = l_frame->next) {
NvDsFrameMeta *frame_meta = (NvDsFrameMeta *) (l_frame->data);
int offset = 0;for (l_obj = frame_meta->obj_meta_list; l_obj != NULL; l_obj = l_obj->next) { obj_meta = (NvDsObjectMeta *) (l_obj->data); g_print ("Frame Number = %d Class_id= %d ",frame_number, obj_meta->class_id); if (obj_meta->class_id == PGIE_CLASS_ID_FOR_REGRESSION || obj_meta->class_id == PGIE_CLASS_ID_COMISSION )$ non_empty_count++; } }
I start my script, it runs, but I only see frame number printed and nothing else (frame number printing statement is at the end of the probe function, just like in deepstream-test-1 example):
Now playing: …/…/…/…/samples/streams/sample_720p.h264
0:00:09.467327410 24536 0x55a872505d50 INFO nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger: NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1702> [UID = 1]: deserialized trt engine from :/opt/nvidia/deepstream/deepstream-5.1/sources/apps/sample_apps/deepstream-test0/test.engine
INFO: …/nvdsinfer/nvdsinfer_model_builder.cpp:685 [Implicit Engine Info]: layers num: 2
0 INPUT kFLOAT input 3x512x512
1 OUTPUT kFLOAT sequential_5 30:00:09.467374989 24536 0x55a872505d50 INFO nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger: NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:1806> [UID = 1]: Use deserialized engine model: /opt/nvidia/deepstream/deepstream-5.1/sources/apps/sample_apps/deepstream-test0/test.engine
0:00:09.468018392 24536 0x55a872505d50 INFO nvinfer gstnvinfer_impl.cpp:313:notifyLoadModelStatus: [UID 1]: Load new model:dstest1_pgie_config.txt sucessfully
Running…
Frame Number = 0
Frame Number = 1
Frame Number = 2
Frame Number = 3
…
Am I doing it wrong, may be?
The code seems ok, where did you install the probe? Also can you explain more about your pipeline and share me with the config file.
And side question from a novice in C/C++
If I make changes to gstnvinfer_meta_utils.cpp - what steps do I need to take to apply changes (aside from “make clean” and “make” inside gstnvinfer dir)?
I tried to add some g_print statements into attach_metadata_classifier to see what steps were executed in each frame, but after “make clean” and “make” for both my app and gstnvinfer folder I did not see any expected output in std after stating my app?
You need to do make install after rebuild the lib. Yeah, will be better to do debug by yourself, it’s also a chance to get more familiar with DS SDK.
Sure.
I took deepstream_test1 files as a starting point and changed basically only dstest1_pgie_config.txt to:
[property]
gpu-id=0
classifier-threshold = 0.5
is-classifier=true
onnx-file=./sequential_5.onnx
model-engine-file=./test.engine
batch-size=1
Probe is attached where it was in the example - to nvosd sink
Thanks, that worked.
So I added g_print like that:
/**
- Update string label in an existing object metadata. If processing on full
- frames, need to attach a new metadata. Assume only one label per object is generated.
*/
void
attach_metadata_classifier (GstNvInfer * nvinfer, GstMiniObject * tensor_out_object,
GstNvInferFrame & frame, GstNvInferObjectInfo & object_info)
{
NvDsObjectMeta *object_meta = frame.obj_meta;
NvDsBatchMeta *batch_meta = (nvinfer->process_full_frame) ?
frame.frame_meta->base_meta.batch_meta : object_meta->base_meta.batch_meta;
g_print(“Accesing meta\n”);
if (object_info.attributes.size () == 0 ||
object_info.label.length() == 0){
g_print(“No object metadata, returning\n”);
return;
}
And after running app i see the following ouput:
Now playing: …/…/…/…/samples/streams/sample_720p.h264
0:00:10.041046515 24642 0x56274ce43560 INFO nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger: NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1702> [UID = 1]: deserialized trt engine from :/opt/nvidia/deepstream/deepstream-5.1/sources/apps/sample_apps/deepstream-test0/test.engine
INFO: …/nvdsinfer/nvdsinfer_model_builder.cpp:685 [Implicit Engine Info]: layers num: 2
0 INPUT kFLOAT input 3x512x512
1 OUTPUT kFLOAT sequential_5 30:00:10.041094612 24642 0x56274ce43560 INFO nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger: NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:1806> [UID = 1]: Use deserialized engine model: /opt/nvidia/deepstream/deepstream-5.1/sources/apps/sample_apps/deepstream-test0/test.engine
0:00:10.041622216 24642 0x56274ce43560 INFO nvinfer gstnvinfer_impl.cpp:313:notifyLoadModelStatus: [UID 1]: Load new model:dstest1_pgie_config.txt sucessfully
Running…
Accesing meta
No object metadata, returning
Frame Number = 0
Accesing meta
No object metadata, returning
Frame Number = 1
Accesing meta
No object metadata, returning
That means that no objects in a frame would mean no metadata attached after classifiction, If I’m not wrong.
What can I do from here?
Oh, you need to customized your post process parser for your model, you can refer nvdsinfer_customclassifierparser.cpp under $DS_TOP/sources/libs/nvdsinfer_customparser/ for how to do that
Thanks, will do that.
I somehow knew it wouldn’t be that easy.
I"ll try to update this post after figuring out what needed to be done for the sake of this topic’s completeness.
Yeah, you can check nvdsinfer_context_impl_output_parsing.cpp for how gst-nvinfer to parse the model’s output, good luck.
Overall - there was no need for extensive custom parsing (At least in my case);
When there are only classificaiton NNs in a pipeline - 1 object will be created in frame_meta->ojb_meta_list to which all classification results would be appended into obj_meta->classifier_meta_list
Therefor, I just needed to modify probe function to access Classifier metadata list and retrieve my values:
static GstPadProbeReturn
osd_sink_pad_buffer_probe (GstPad * pad, GstPadProbeInfo * info,
gpointer u_data)
{
GstBuffer *buf = (GstBuffer *) info->data;
NvDsObjectMeta *obj_meta = NULL;
guint non_empty_count = 0;
guint _count = 0;
NvDsMetaList *l_frame = NULL, *l_obj = NULL, *cl_obj = NULL, *id_obj = NULL;
NvDsClassifierMeta * cl_meta = NULL;
NvDsLabelInfo * id_meta = NULL;
NvDsDisplayMeta *display_meta = NULL;
NvDsBatchMeta *batch_meta = gst_buffer_get_nvds_batch_meta (buf);for (l_frame = batch_meta->frame_meta_list; l_frame != NULL; l_frame = l_frame->next) { NvDsFrameMeta *frame_meta = (NvDsFrameMeta *) (l_frame->data); int offset = 0; for (l_obj = frame_meta->obj_meta_list; l_obj != NULL; l_obj = l_obj->next) { obj_meta = (NvDsObjectMeta *) (l_obj->data); for (cl_obj = obj_meta->classifier_meta_list; cl_obj != NULL; cl_obj = cl_obj->next) { cl_meta = (NvDsClassifierMeta *) (cl_obj->data); for (id_obj = cl_meta->label_info_list; id_obj != NULL; id_obj = id_obj->next) { id_meta = (NvDsLabelInfo *) (id_obj->data); g_print("Frame number = %d, train_car_status: %s \n", frame_number, id_meta->result_label); } } }}
After delving deeper into the belly of the beast I found that my classification results were waiting for me right there.
Thanks again for help. Your pointers to the right files allowed me to understand where to look for answers.
Great work, however one thing still confused is that why following code print ““No object metadata, returning\n””
Yeah, that was also originally a question for me.
But after tinkering with ClassifyPostprocessor::parseAttributesFromSoftmaxLayers function in nvdsinfer_context_impl_output_parsing.cpp I understood that I forgot to add path to labelfile in my config.
labelfile-path=./labels.txt
Without it, no classification data gets attached to MetaData.
After adding label file - everything started to work OK.
Cool, thanks for the info.