Access metadata for segmentation mask

Accelerated Computing Intelligent Video Analytics DeepStream SDK
,
1

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU): RTX 2080
• DeepStream Version: 10.2
• JetPack Version (valid for Jetson only)
• TensorRT Version: 7.0
• NVIDIA GPU Driver Version (valid for GPU only):410.xx

I want to access the output segmentation mask using python API through probe function and save it to disk as image with Opencv and numpy array. As I read, the metadata belongs to NvDsUserMeta which can be achieved through frame_user_meta_list to get the NvDsInferSegmentationMeta. So the flow I understand is somehow like this:

def tiler_sink_pad_buffer_probe(pad,info,u_data):
    frame_number=0
    num_rects=0
    gst_buffer = info.get_buffer()
if not gst_buffer:
    print("Unable to get GstBuffer ")
    return
    
# Retrieve batch metadata from the gst_buffer
# Note that pyds.gst_buffer_get_nvds_batch_meta() expects the
# C address of gst_buffer as input, which is obtained with hash(gst_buffer)
batch_meta = pyds.gst_buffer_get_nvds_batch_meta(hash(gst_buffer))

l_frame = batch_meta.frame_meta_list
user_meta_list = l_frame.frame_user_meta_list
while user_meta_list is not None:
         user_meta = pyds.NvDsUserMeta.cast(user_meta_list.data)
         mask = user_meta.user_meta_data.class_map

So the mask value here is currently the pointer to the map containing class type like 0 or 1 of the output segmentation mask. So I would really want to know how to get the data into numpy array from the pointer value. Or can you give me another proper approach to this. I follow the deepstream-imagemeta-multistream.py example but this only convert the original image from gst_buffer.

3

Unfortunately, current pyds binding does not support mask in numpy array.

6

Thank you for informing me that. Currently I am working on segmentation task as I already described. The problem here is that the output mask showing when I run the deepstream is blank, just an empty black screen without any value. The model I use is a custom ResNet Unet model written in pytorch, then I convert it to tensorrt. On the other hand, when I run another model which is trained on different dataset and has nothing to do with my dataset, it randomly generated noised output, still there is output, so i guess my pipeline is correct. But as you say there is no way to get the output predicted from deepstream, so I dont really know what is the output. Is it the model problem or the preprocessing steps. Do you have any idea bout this. Thanks again

7

@redalert1013

I think you can refer to TensorRT’s python APIs, python samples, or C++ APIs and C++ samples to run your models in standalone mode that is independent of DS pipeline to check the outputs.