How to bundle up a sequence of images for action recognition system in deepstream

seonhoon1002 · December 18, 2020, 5:13am

I want to bundle up a sequence of images for action recognition system.
But I couldn’t find method that bundle up the sequence of images.
Because If I try to extract images before ‘nvinfer’, I couldn’t extract images(but I can extract images after ‘nvdsosd’)
How can I bundle up a sequence of images in deepstream before nvinfer?

AastaLLL · December 18, 2020, 7:50am

Hi,

You can use a sink component to save the sequence into images (File Save).

Thanks.

seonhoon1002 · December 18, 2020, 8:50am

Thank you for quick response.
But I want to not save images(because It is important to keep a high speed), but directly convert a image format that can be used by numpy or cv2.
And How to convert “stream -->image --> modified image–> stream”, Before ‘nvinfer’ I can’t find the way.

AastaLLL · December 29, 2020, 5:33am

Hi,

To use cv2 or numpy, you may want our Deepstream python interface.
You can find some examples on our GitHub.

For example:

github.com

NVIDIA-AI-IOT/deepstream_python_apps/blob/master/apps/deepstream-imagedata-multistream/deepstream_imagedata-multistream.py#L123


# Periodically check for objects with borderline confidence value that may be false positive detections.
# If such detections are found, annoate the frame with bboxes and confidence value.
# Save the annotated frame to file.
if((saved_count["stream_"+str(frame_meta.pad_index)]%30==0) and (obj_meta.confidence>0.3 and obj_meta.confidence<0.31)):
    if is_first_obj:
        is_first_obj = False
        # Getting Image data using nvbufsurface
        # the input should be address of buffer and batch_id
        n_frame=pyds.get_nvds_buf_surface(hash(gst_buffer),frame_meta.batch_id)
        #convert python array into numy array format.
        frame_image=np.array(n_frame,copy=True,order='C')
        #covert the array into cv2 default color format
        frame_image=cv2.cvtColor(frame_image,cv2.COLOR_RGBA2BGRA)
    save_image = True
    frame_image=draw_bounding_boxes(frame_image,obj_meta,obj_meta.confidence)
try: 
    l_obj=l_obj.next
except StopIteration:
    break

Thanks.

seonhoon1002 · December 29, 2020, 6:01am

But It isn’t work before nvinfer. How can I extract image before nvinfer?

AastaLLL · January 12, 2021, 2:14am

Hi,

You can save the raw image in a similar way.
The frame buffer can be accessed with the following function:

n_frame=pyds.get_nvds_buf_surface(hash(gst_buffer),frame_meta.batch_id)
frame_image=np.array(n_frame,copy=True,order='C')
frame_image=cv2.cvtColor(frame_image,cv2.COLOR_RGBA2BGRA)

For example, the deepstream_imagedata-multistream.py (16.1 KB) is to dump each frame into the same folder.

Thanks.