Why does nvmultistreamtiler reverse object order in frame_meta->obj_meta_list?

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU): GeForce RTX 3090
• DeepStream Version: 6.1
• TensorRT Version: 8.2
• NVIDIA GPU Driver Version (valid for GPU only): 510
• Issue Type( questions, new requirements, bugs): question

I understand that the tiler merges all frames in a batched buffer into a single frame, and the doc says the tiler combines “NvDsBatchMeta and NvDsFrameMeta in row-major order (starting from source 0, left to right across the top row, then across the next row)” so object index change before and after the tiler plugin is expected.

In my pipeline I have 2 sources, I noticed that the order of the objects in frame_meta->obj_meta_list is reversed after the tiler plugin. Please see example bellow.

Where:

  • batch = current batched buffer
  • batch_id = frame_meta->batch_id
  • src_id = frame_meta->source_id
  • obj_index = object index in frame_meta->obj_meta_list of current frame in batch_meta->frame_meta_list
  • x = obj_meta->detector_bbox_info.org_bbox_coords.top
  • y = obj_meta->detector_bbox_info.org_bbox_coords.left

Case 1: frame_meta->batch_id match frame_meta->source_id

----------
before tiler [batch = 19, batch_id = 0, src_id = 0, obj_index = 0] x = 453.75, y = 312.69
before tiler [batch = 19, batch_id = 0, src_id = 0, obj_index = 1] x = 277.85, y = 379.27
before tiler [batch = 19, batch_id = 0, src_id = 0, obj_index = 2] x = 169.19, y = 201.79
before tiler [batch = 19, batch_id = 0, src_id = 0, obj_index = 3] x = 2.62, y = 693.85
before tiler [batch = 19, batch_id = 0, src_id = 0, obj_index = 4] x = 216.08, y = 869.71
before tiler [batch = 19, batch_id = 1, src_id = 1, obj_index = 0] x = 0.00, y = 378.50
before tiler [batch = 19, batch_id = 1, src_id = 1, obj_index = 1] x = 156.22, y = 890.23
before tiler [batch = 19, batch_id = 1, src_id = 1, obj_index = 2] x = 344.46, y = 331.87
before tiler [batch = 19, batch_id = 1, src_id = 1, obj_index = 3] x = 36.58, y = 863.97
----------
----------
after tiler [batch = 19, batch_id = 0, src_id = 0, obj_index = 0], x = 36.58, y = 863.97
after tiler [batch = 19, batch_id = 0, src_id = 0, obj_index = 1], x = 344.46, y = 331.87
after tiler [batch = 19, batch_id = 0, src_id = 0, obj_index = 2], x = 156.22, y = 890.23
after tiler [batch = 19, batch_id = 0, src_id = 0, obj_index = 3], x = 0.00, y = 378.50
after tiler [batch = 19, batch_id = 0, src_id = 0, obj_index = 4], x = 216.08, y = 869.71
after tiler [batch = 19, batch_id = 0, src_id = 0, obj_index = 5], x = 2.62, y = 693.85
after tiler [batch = 19, batch_id = 0, src_id = 0, obj_index = 6], x = 169.19, y = 201.79
after tiler [batch = 19, batch_id = 0, src_id = 0, obj_index = 7], x = 277.85, y = 379.27
after tiler [batch = 19, batch_id = 0, src_id = 0, obj_index = 8], x = 453.75, y = 312.69
----------

Case 2: frame_meta->batch_id does not match frame_meta->source_id

----------
before tiler [batch = 0, batch_id = 0, src_id = 1, obj_index = 0] x = 0.00, y = 425.23
before tiler [batch = 0, batch_id = 0, src_id = 1, obj_index = 1] x = 203.79, y = 936.85
before tiler [batch = 0, batch_id = 0, src_id = 1, obj_index = 2] x = 38.64, y = 865.28
before tiler [batch = 0, batch_id = 1, src_id = 0, obj_index = 0] x = 152.98, y = 68.65
before tiler [batch = 0, batch_id = 1, src_id = 0, obj_index = 1] x = 202.32, y = 423.64
before tiler [batch = 0, batch_id = 1, src_id = 0, obj_index = 2] x = 2.01, y = 688.54
before tiler [batch = 0, batch_id = 1, src_id = 0, obj_index = 3] x = 211.92, y = 889.19
----------
----------
after tiler [batch = 0, batch_id = 0, src_id = 0, obj_index = 0], x = 211.92, y = 889.19
after tiler [batch = 0, batch_id = 0, src_id = 0, obj_index = 1], x = 2.01, y = 688.54
after tiler [batch = 0, batch_id = 0, src_id = 0, obj_index = 2], x = 202.32, y = 423.64
after tiler [batch = 0, batch_id = 0, src_id = 0, obj_index = 3], x = 152.98, y = 68.65
after tiler [batch = 0, batch_id = 0, src_id = 0, obj_index = 4], x = 38.64, y = 865.28
after tiler [batch = 0, batch_id = 0, src_id = 0, obj_index = 5], x = 203.79, y = 936.85
after tiler [batch = 0, batch_id = 0, src_id = 0, obj_index = 6], x = 0.00, y = 425.23
----------

Focus on batch_id, src_id, obj_index before and after the tiler:

Case 1:

(0, 0, 0) → (0, 0, 8)
(0, 0, 1) → (0, 0, 7)
(0, 0, 2) → (0, 0, 6)
(0, 0, 3) → (0, 0, 5)
(0, 0, 4) → (0, 0, 4)
(1, 1, 0) → (0, 0, 3)
(1, 1, 1) → (0, 0, 2)
(1, 1, 2) → (0, 0, 1)
(1, 1, 3) → (0, 0, 0)

Conceptually, the order of object index is reverse based on batch id (instead of source id) from last to first batch (instead of first to last):

[0]->[1]->[2]->[3]->[4]->[0]->[1]->[2]->[3]
[8]<-[7]<-[6]<-[5]<-[4]<-[3]<-[2]<-[1]<-[0]

Case 2:

(0, 1, 0) → (0, 0, 6)
(0, 1, 1) → (0, 0, 5)
(0, 1, 2) → (0, 0, 4)
(1, 0, 0) → (0, 0, 3)
(1, 0, 1) → (0, 0, 2)
(1, 0, 2) → (0, 0, 1)
(1, 0, 3) → (0, 0, 0)

Same as case 1, conceptually, the order of object index is reverse based on batch id (instead of source id) from last to first batch (instead of first to last):

[0]->[1]->[2]->[0]->[1]->[2]->[3]
[6]<-[5]<-[4]<-[3]<-[2]<-[1]<-[0]

Why does the order of object metadatas within each original frame is completely reversed after the tiler? Is this expected behaviour?

I also noticed that this reversing order starts in version 6.1, in version 5.1 the order of objects within each original frame is preserved by the tiler plugin.

Did you run our demo code or your own code? Could you show your code and config files to us?

@yuweiw,

Thank you for the response. Upon closer inspection, the plugin that reverse the object metadata order is actually the nvvideoconvert plugin, not the tiler. Please find the modified sample app bellow. The modified deepstream-test1:

deepstream_test1_app.c (9.6 KB)

To build & run the app:

make CUDA_VER=11.6
./deepstream-test1-app ../../../../samples/streams/sample_720p.h264 ../../../../samples/streams/sample_720p.h264

Example output:

----------
before nvvideoconvert [batch = 0, batch_id = 0, src_id = 0, object_index = 0], x = 473.05, y = 423.51
before nvvideoconvert [batch = 0, batch_id = 0, src_id = 0, object_index = 1], x = 475.33, y = 340.03
before nvvideoconvert [batch = 0, batch_id = 0, src_id = 0, object_index = 2], x = 401.43, y = 2.44
before nvvideoconvert [batch = 0, batch_id = 0, src_id = 0, object_index = 3], x = 443.65, y = 306.57
before nvvideoconvert [batch = 0, batch_id = 0, src_id = 0, object_index = 4], x = 463.44, y = 640.84
before nvvideoconvert [batch = 0, batch_id = 0, src_id = 0, object_index = 5], x = 463.01, y = 645.63
before nvvideoconvert [batch = 0, batch_id = 0, src_id = 0, object_index = 6], x = 79.00, y = 870.23
before nvvideoconvert [batch = 0, batch_id = 0, src_id = 0, object_index = 7], x = 471.33, y = 571.71
before nvvideoconvert [batch = 0, batch_id = 0, src_id = 0, object_index = 8], x = 459.04, y = 512.17
before nvvideoconvert [batch = 0, batch_id = 0, src_id = 0, object_index = 9], x = 468.47, y = 636.49
before nvvideoconvert [batch = 0, batch_id = 0, src_id = 0, object_index = 10], x = 470.92, y = 540.98
before nvvideoconvert [batch = 0, batch_id = 0, src_id = 0, object_index = 11], x = 478.00, y = 593.12
before nvvideoconvert [batch = 0, batch_id = 0, src_id = 0, object_index = 12], x = 495.75, y = 618.40
before nvvideoconvert [batch = 0, batch_id = 0, src_id = 0, object_index = 13], x = 473.05, y = 423.51
before nvvideoconvert [batch = 0, batch_id = 0, src_id = 0, object_index = 14], x = 475.33, y = 340.03
before nvvideoconvert [batch = 0, batch_id = 0, src_id = 0, object_index = 15], x = 401.43, y = 2.44
before nvvideoconvert [batch = 0, batch_id = 0, src_id = 0, object_index = 16], x = 443.65, y = 306.57
before nvvideoconvert [batch = 0, batch_id = 0, src_id = 0, object_index = 17], x = 463.44, y = 640.84
before nvvideoconvert [batch = 0, batch_id = 0, src_id = 0, object_index = 18], x = 463.01, y = 645.63
before nvvideoconvert [batch = 0, batch_id = 0, src_id = 0, object_index = 19], x = 79.00, y = 870.23
before nvvideoconvert [batch = 0, batch_id = 0, src_id = 0, object_index = 20], x = 471.33, y = 571.71
before nvvideoconvert [batch = 0, batch_id = 0, src_id = 0, object_index = 21], x = 459.04, y = 512.17
before nvvideoconvert [batch = 0, batch_id = 0, src_id = 0, object_index = 22], x = 468.47, y = 636.49
before nvvideoconvert [batch = 0, batch_id = 0, src_id = 0, object_index = 23], x = 470.92, y = 540.98
before nvvideoconvert [batch = 0, batch_id = 0, src_id = 0, object_index = 24], x = 478.00, y = 593.12
before nvvideoconvert [batch = 0, batch_id = 0, src_id = 0, object_index = 25], x = 495.75, y = 618.40
----------
----------
after nvvideoconvert [batch = 0, batch_id = 0, src_id = 0, object_index = 0], x = 495.75, y = 618.40
after nvvideoconvert [batch = 0, batch_id = 0, src_id = 0, object_index = 1], x = 478.00, y = 593.12
after nvvideoconvert [batch = 0, batch_id = 0, src_id = 0, object_index = 2], x = 470.92, y = 540.98
after nvvideoconvert [batch = 0, batch_id = 0, src_id = 0, object_index = 3], x = 468.47, y = 636.49
after nvvideoconvert [batch = 0, batch_id = 0, src_id = 0, object_index = 4], x = 459.04, y = 512.17
after nvvideoconvert [batch = 0, batch_id = 0, src_id = 0, object_index = 5], x = 471.33, y = 571.71
after nvvideoconvert [batch = 0, batch_id = 0, src_id = 0, object_index = 6], x = 79.00, y = 870.23
after nvvideoconvert [batch = 0, batch_id = 0, src_id = 0, object_index = 7], x = 463.01, y = 645.63
after nvvideoconvert [batch = 0, batch_id = 0, src_id = 0, object_index = 8], x = 463.44, y = 640.84
after nvvideoconvert [batch = 0, batch_id = 0, src_id = 0, object_index = 9], x = 443.65, y = 306.57
after nvvideoconvert [batch = 0, batch_id = 0, src_id = 0, object_index = 10], x = 401.43, y = 2.44
after nvvideoconvert [batch = 0, batch_id = 0, src_id = 0, object_index = 11], x = 475.33, y = 340.03
after nvvideoconvert [batch = 0, batch_id = 0, src_id = 0, object_index = 12], x = 473.05, y = 423.51
after nvvideoconvert [batch = 0, batch_id = 0, src_id = 0, object_index = 13], x = 495.75, y = 618.40
after nvvideoconvert [batch = 0, batch_id = 0, src_id = 0, object_index = 14], x = 478.00, y = 593.12
after nvvideoconvert [batch = 0, batch_id = 0, src_id = 0, object_index = 15], x = 470.92, y = 540.98
after nvvideoconvert [batch = 0, batch_id = 0, src_id = 0, object_index = 16], x = 468.47, y = 636.49
after nvvideoconvert [batch = 0, batch_id = 0, src_id = 0, object_index = 17], x = 459.04, y = 512.17
after nvvideoconvert [batch = 0, batch_id = 0, src_id = 0, object_index = 18], x = 471.33, y = 571.71
after nvvideoconvert [batch = 0, batch_id = 0, src_id = 0, object_index = 19], x = 79.00, y = 870.23
after nvvideoconvert [batch = 0, batch_id = 0, src_id = 0, object_index = 20], x = 463.01, y = 645.63
after nvvideoconvert [batch = 0, batch_id = 0, src_id = 0, object_index = 21], x = 463.44, y = 640.84
after nvvideoconvert [batch = 0, batch_id = 0, src_id = 0, object_index = 22], x = 443.65, y = 306.57
after nvvideoconvert [batch = 0, batch_id = 0, src_id = 0, object_index = 23], x = 401.43, y = 2.44
after nvvideoconvert [batch = 0, batch_id = 0, src_id = 0, object_index = 24], x = 475.33, y = 340.03
after nvvideoconvert [batch = 0, batch_id = 0, src_id = 0, object_index = 25], x = 473.05, y = 423.51
----------

The before probe is added at nvvidconv sink pad, the after probe is added at nvvidconv source pad. As you can see from inspecting the output, the object metadata order is reversed. Is this expected behaviour?

We will create new buffer in nvvidconvert plugin. So it maybe change. Does this affect your user case?

@yuweiw,

Hi, thank you for the reply. Yes, this new behaviour of nvvidconvert affects my use case. I run the same app on DS 5.1 and I observed this new behaviour in version 6.1, not 5.1. In version 5.1, nvvidconvert maintains the order object metadata instead of reversing the original order.

Also just a follow up, does your answer imply that the order may or may not change?

Ok,Could you tell us your specific user case? We’ll check it. Thanks

@yuweiw

I run a custom algorithm on the detected object that’s specific to their locations in the frame and I want to check the correctness visually using the nvosd plugin and a display sink, I use an object’s index to quickly identify it. The visual debug process was straightforward in version 5.1, but in version 6.1, I need to manually reverse the indexes to check for correction.

You can try to use object_id from the NvDsObjectMeta structure instead of your own object_index paras. The order of objects may change in the list.

@yuweiw,

Thank you for the suggestion. I don’t use the tracker plugin in my pipeline so the object_id value is always -1 for all objects. Do you meant to overwrite that value for my own use?

There is no update from you for a period, assuming this is not an issue anymore.
Hence we are closing this topic. If need further support, please open a new one.
Thanks

We suggest you identify it from the existing structure instead of your own paras. Cause the order in the linked list may change though some plugins.

1 Like

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.