Use Gst-nvinfer to classify a cropped ROI in the input stream

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU)
• DeepStream Version
• JetPack Version (valid for Jetson only)
• TensorRT Version
• NVIDIA GPU Driver Version (valid for GPU only)
• Issue Type( questions, new requirements, bugs)
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

I am using DS 6.2 SDK to built the traffic light classifier. I was able to successfully implement it.

For the development, i used the cropped portion of video since my custom model was trained on the cropped image dataset and it worked perfectly.

But for the deployment, we will have a overview camera of the street with Traffic light occupying a small area, so i want to pass the cropped segment of the video in the stream so that predicted labels are correct.

How can we implement the ROI settings in the input so that the classifier model only infer to the portion instead of the whole frame of the video input.

Thanks in advance :)

you can use nvdspreprocess+nvinfer instead of nvinfer. nvdspreprocess will do ROI cropping. please refer to nvdspreprocess doc. please refer to sample deepstream-preprocess-test in deepstram sdk.

Thanks for the prompt reply. I will look into the documentation and test app.

Just an additional query, after the classification of the traffic light in the cropped area, my objective is to detect the objects in the whole frame when the traffic light turns red. So if i preprocess to do ROI cropping, is there any chance of doing the object detection in whole frame?

Please suggest on the efficient approach that i need to take in such case! Thank you :)

what is the whole media pipeline? dose it look like “source->decoder->nvstreammux->nvpreprocess->gie1(light checking)–>gie2(detection model)->osd->sink”? if yes, you can set gie2’s process-mode to 1, then gie2 will do inference on the whole frame. if the gie1’s output is red, you can set gie2’s interval to 0, then gie2 will do inference on every frame. if the gie1’s output is not red, you can set gie2’s interval to max, then gie2 will not do inference. please refer to nvinfer for nvinfer’s parameters.

Yes the pipeline is similar as you stated, with more additional blocks after gie2. After the detection results in gie2 from 2 different camera angles (streaming via nvstreammux), i am planning to use a secondary gie (gie3) for the reid model to match the detected objects with same identity in two different views followed by osd and sink.

I will try to implement the part as you suggested and in case of any complications, i will revert back to you. Thanks alot for the support.

Hi fanzh,

I am getting the following error for floating point value.

What might be the reason for this? Any idea for debug!

Here are my config file:
config_preprocess.txt (2.6 KB)
pgie_config.txt (3.4 KB)

This was because of my configuration on the config_preprocess.txt. It always expects some value for scaling factor. When i enabled the property with this value, i could run the program.

Glad to know you fixed it, thanks for the update! Is this still an DeepStream issue to support?

Yes sir. I am abit stuck to use the output meta tensor using NvdsUserMeta to get the classification label while using preprocessor and pgie since this data is no more available in framemeta–>objectmeta–>classifiermeta–>labelmeta when used without the preprocessor. I refer to the user meta documentation but i couldnt figure my way out. Can you please help me on this!

I went through the documentation of NvDsInferTensorMeta — Deepstream Deepstream Version: 6.2 documentation . Through this i could get through to get the object from the code snippet of function osd_sink_pad_buffer_probe below:

def osd_sink_pad_buffer_probe(pad,info,u_data):

frame_number=0
num_rects=0

gst_buffer = info.get_buffer()
if not gst_buffer:
    print("Unable to get GstBuffer ")
    return

# Retrieve batch metadata from the gst_buffer
# Note that pyds.gst_buffer_get_nvds_batch_meta() expects the
# C address of gst_buffer as input, which is obtained with hash(gst_buffer)
batch_meta = pyds.gst_buffer_get_nvds_batch_meta(hash(gst_buffer))

#check if you have to get the batch user meta list or frame user meta list in the frame
#l_user = batch_meta.batch_user_meta_list

l_frame = batch_meta.frame_meta_list
l_user = batch_meta.batch_user_meta_list
while l_user is not None:
    try:
        user_meta= pyds.NvDsUserMeta.cast(l_user.data)
    except StopIteration:
        break

    if user_meta and user_meta.base_meta.meta_type == pyds.NVDSINFER_TENSOR_OUTPUT_META:
        try:
            classi = pyds.NvDsInferTensorMeta.cast(user_meta.user_meta_data)
            print(classi)
        except StopIteration:
            break
    try:
        l_user=l_user.next
    except StopIteration:
        break
return Gst.PadProbeReturn.OK

This could give me the object but i didnt find any function to retrieve the object data which i guess contains the classification label.

In case of segmentation, using the link , we can get mask data using get_segmentation_mask() on the object.

I tried to check the output of my nvinfer by setting raw-output-file-write=1 for which i get the many files in batch buffer. Posting one of them so that it might help you to debug:
gstnvdsinfer_uid-01_layer-output_batch-0000000300_batchsize-01.bin (16 Bytes)

please help me out on this.

Thank you.

there is NvDsClassifierMeta in object meta, please refer to opt\nvidia\deepstream\deepstream-6.2\sources\includes\nvdsmeta.h, here is a sample pgie_src_pad_buffer_probe in opt\nvidia\deepstream\deepstream-6.2\sources\apps\sample_apps\deepstream-preprocess-test\deepstream_preprocess_test.cpp

Hi. Thanks for your support but with the example in preprocess test there is a way to get the Meta data for the detector but i found it hard to get the metadata for the classifier.

But i found another way to obtain the classification label. I enabled output-tensor-meta to 1 in pgie configuration and used this meta to find the label. I guess i did the hard approach to get my labels and there should be a better way to do this.

Here is my code for at the sink of the tiler to obtain the classification labels of ROI:

def tiler_sink_pad_buffer_probe(pad,info,u_data):
    frame_number=0
    gst_buffer = info.get_buffer()
    if not gst_buffer:
        print("Unable to get GstBuffer ")
        return

    # Retrieve batch metadata from the gst_buffer
    # Note that pyds.gst_buffer_get_nvds_batch_meta() expects the
    # C address of gst_buffer as input, which is obtained with hash(gst_buffer)
    batch_meta = pyds.gst_buffer_get_nvds_batch_meta(hash(gst_buffer))
    l_frame = batch_meta.frame_meta_list
    frame_meta = pyds.NvDsFrameMeta.cast(l_frame.data)
    frame_number=frame_meta.frame_num
    l_user = batch_meta.batch_user_meta_list
    while l_user is not None:
        try:
            user_meta= pyds.NvDsUserMeta.cast(l_user.data)
        except StopIteration:
            break
        if user_meta and user_meta.base_meta.meta_type == pyds.NVDSINFER_TENSOR_OUTPUT_META:
            try:
                tensor_meta = pyds.NvDsInferTensorMeta.cast(user_meta.user_meta_data)
                #the below code refrence taken from reid pipeline- #
                # https://github.com/ml6team/deepstream-python/blob/master/deepstream/app/pipelines/re_identification.py #
                layer = pyds.get_nvds_LayerInfo(tensor_meta, 0)
                ptr = ctypes.cast(pyds.get_ptr(layer.buffer), ctypes.POINTER(ctypes.c_float))
                features = np.ctypeslib.as_array(ptr, shape=(4,))
                labels = ['green', 'none', 'red', 'yellow']
                max_index = np.argmax(features)
                print("Frame Number={} label={}".format(frame_number, labels[max_index]))
            except StopIteration:
                break
        try:
            l_user=l_user.next
        except StopIteration:
            break
    return Gst.PadProbeReturn.OK

Please suggest if there is a better way to get the labels

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.