Metadata wrongly positioned when using a tee with nvmultistreamtiler and nvstreamdemux

[flavio.mello]

Please provide complete information as applicable to your setup.
• Hardware Platform (Jetson / GPU) Jetson Xavier AGX
• DeepStream Version 6.3.0
• JetPack Version (valid for Jetson only) 5.1
• TensorRT Version 8.5.2.2

I have multiple sources, let’s consider 4 sources. I write metadata to the display of each batch using a buffer probe as in deepstream_python_apps/apps/deepstream-test3/deepstream_test_3.py at master · NVIDIA-AI-IOT/deepstream_python_apps · GitHub, But, I see metadata being written in the correct position at the tiled view, and with a wrong offset at the one batch view.

Streams 0, 1, 2, 3 at tiled view

Stream 2 with wrong offset

My pipeline is like this:

[uridecodebin] -> streammux -> pgie -> tracker -> queue -> tee -|

|-> nvmultistreamtiler -> nvvideoconvert -> nvosd -> (RTSP stuff)
|-> nvstreamdemux -> [nvvideoconvert] -> [nvosd] -> ([similar RTSP stuff])

I placed my buffer probe to write metadata to display at the sink pad from my queue, so that I can write at each separately by iterating over batch_meta.frame_meta_list.

By doing this, I understood that I would be writing in the metadata from each batch separately, and I thought that was right since the tiler was fusing those batches in one image and showing the metadata at the right position. But now, when I inspect one batch alone, I see the metadata with an offset, and probably with the wrong size too.

I don’t understand what I am missing here.

[flavio.mello]

So, I am still trying to find out the problem. I noticed that metadata is correctly positioned when I do NOT link nvmultistreamtiler, that is:

[uridecodebin] → streammux → pgie → tracker → queue → tee -|

|XX nvmultistreamtiler → nvvideoconvert → nvosd → (RTSP stuff)
|-> nvstreamdemux → [nvvideoconvert] → [nvosd] → ([similar RTSP stuff])

So, the nvmultistreamtiler is the aggressor here, but I don’t understand how to avoid it. I understood from manual that the tiler converts all images into one. I wrongly supposed that the frame and its metadata would be merged, and then, the result would be tiled with all other images, but its not. Frames are merged and metadata suffers some kind of transformation.

How to adjust this?

[yuweiw]

Sorry for the late response. Could you attach your diffs from our demo code so that we can run that on our side and analyze that? Thanks

[flavio.mello]

I am using Yolov5, but I changed my config file to resnet10 for your convenience. This is the config file for detector dstest3_pgie_config.txt (3.1 KB), and these are the config files for the tracker dstest2_tracker_config.txt (1.7 KB), config_tracker_NvDCF_perf.yml.txt (4.5 KB).

The scriptfrom the pipeline I described is this one pipeline.py.txt (29.1 KB).

You may run with 4 sources like this:
python3 pipeline.py -o rtsp -i rtsp://user:pass@10.21.45.19 rtsp://user:pass@10.21.45.19 rtsp://user:pass@10.21.45.19 rtsp://user:pass@10.21.45.19

And the RTSP stream is accessible through:

*** Will launch RTSP Streaming at rtsp://localhost:8554/tiled ***
*** Will launch RTSP Streaming at rtsp://localhost:8554/stream0 ***
*** Will launch RTSP Streaming at rtsp://localhost:8554/stream1 ***
*** Will launch RTSP Streaming at rtsp://localhost:8554/stream2 ***
*** Will launch RTSP Streaming at rtsp://localhost:8554/stream3 ***

[yuweiw]

I have tried your code with our model: samples/models/Primary_Detector/resnet18_trafficcamnet.etlt. It works normally.
Command: python3 pipeline.py -o screen -i rtsp://xxx rtsp://xxx
Could you try to upgrade the deepstream to 6.4 and try that? Thanks

[flavio.mello]

It is OK if you use your model.

The -o stream option doesn’t create/add/link the tee branch for nvstreamdemux, you will only have the nvmultistreamtiler branch. You need to change it to -o rtsp.

You also need two sources, so you command must be python3 pipeline.py -o rtsp -i rtsp://xxx rtsp://xxx

[yuweiw]

OK. We can repoduce the issue on our side. We’ll analyze this problem as soon as possible.

[yuweiw]

The tee plugin is just add the ref count of the gstbuffer. It’s not really copy the gstbuffer. So if you processed the image in one branch, the image in the other branch will be affected.

You need to add nvvideoconvert plugin with disable-passthrough=1 after the tee plugin.