Gst-nvdsanalytics roi filtering objects presence inconsistent

[melissa.montero]

I have been testing the nvdsanalytics element to determine the presence of persons in a region, however I am having inconsistent results. It was my understanding that the analytics will determine that an object is in the ROI if the bottom center coordinate is inside the defined region, however I get object bounding boxes classified in the ROI when the bottom line is outside the region and bounding boxes not detected in the ROI when the bottom line is inside the region.

Following example shows how in some occasions the man is detected inside the RT ROI when the bottom bounding box line is outside the region:

On the other hand next video shows the man is not detected in ROI even when the bottom line of the bounding box is inside the ROI-RF:

Also I am getting different results depending on the detector that I use, I tested with peoplenet and rtmdet, with the same video and pipeline, even tough I get similar bounding boxes, the analytics ROI detections are different, see below:

• rtmdet
  

• peoplenet
  

I tested with the following gstreamer pipeline:
gst-launch-1.0 filesrc location= test-walking.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! .sink_0 nvstreammux name=m width=1920 height=1080 batch-size=1 live-source=1 ! nvinfer config-file-path= peoplenet/config_infer_primary_peoplenet.txt ! nvdsanalytics config-file=config.txt ! nvvideoconvert ! nvdsosd ! nvvideoconvert ! nvv4l2h264enc ! h264parse ! avimux ! filesink location=/tmp/test.avi

Using the following analytics configuration file:

[property]
enable=1
config-width=1920
config-height=1080
osd-mode=2
display-font-size=12

[roi-filtering-stream-0]
enable=1
roi-RF=0;643;1920;643;1920;890;0;890
inverse-roi=0
class-id=-1

Is my understanding of how the analytics element should work wrong? Do I have a configuration error?

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) Jetson AGX Orin
• DeepStream Version 7.0
• JetPack Version (valid for Jetson only) 6.0
• TensorRT Version
• NVIDIA GPU Driver Version (valid for GPU only)
• Issue Type( questions, new requirements, bugs)
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

Melissa Montero
www.ridgerun.ai

[fanzh]

Thanks for the sharing! I am checking.

[fanzh]

could you share the test videos? Thanks! let’s test the same videos. which peoplenet version are you testing?

[MarcoMadrigal]

Hi fanzh,

You can find the sample videos here (actually it happens mostly with any video):

Video 1

Video 2

The version is

Latest Version
pruned_quantized_decrypted_v2.3.4

In any case, after testing with RTM and peoplenet we determined that it does not matter the detector used (or version) these issues happen anyway.

Best Regards,
Marco Madrigal

RidgeRun AI | Artificial Intelligence Services | United States

[fanzh]

Using your configuration file and peoplenet v2.3.4 model, I tested deepstream-nvdsanalytics-test on Orin with DS7.1. my test results are correct. Here are the log log1028.txt (32.1 KB) and screenshots. when he bottom center coordinate (x_left + width/2, y_top + height) entered the ROI, the count will increase.

[MarcoMadrigal]

Hi fanzh,

Thanks for looking at this. We are using DS7.0 and a GStreamer pipeline, would this make any difference? Anyway, we will look at this example but just wondering if there might be any different with the DS version and/or using a GStreamer pipeline instead of the sample application.

Best Regards,
Marco Madrigal

RidgeRun AI | Artificial Intelligence Services | United States

[fanzh]

Thanks for the sharing! nvdsanalytics plugin is opensource. there is no difference between DS7.0 and DS7.1 in nvdsanalytics low-level lib.
could you use DS7.1 to reproduce this accuracy issue? Thanks! please refer to my last comment. I used your test video and configuration file without modifying the code.

[MarcoMadrigal]

Hi @fanzh ,

We completed the tests. Here is what we found:

1. We tested the sample application you suggested with the videos and we reproduced your results. Using the application the ROI detects the bounding box correctly.

2. When checking the application, we realized it uses the tracker after the detector.

3. When we add the tracker to the original GStreamer pipeline (as shown below) the results are the same than when using the application i.e. the detection within the ROI is correct.

gst-launch-1.0 filesrc location= test-walking1.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! .sink_0 nvstreammux name=m width=1920 height=1080 batch-size=1 ! nvinfer config-file-path= peoplenet/config_infer_primary_peoplenet.txt ! nvtracker ll-lib-file=“/opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so” ll-config-file=“/opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_tracker_NvDCF_perf.yml” tracker-width=640 tracker-height=480 ! nvdsanalytics config-file=config.txt ! nvvideoconvert ! nvdsosd ! nvvideoconvert ! nvv4l2h264enc ! h264parse ! avimux ! filesink location=/tmp/test.avi

4. The following videos show the difference between using the tracker and not using it with the sample videos:

For our use case, using the tracker is not necessary, we only need to detect whether the detection occurs within the ROI or not. That said, our questions is:

Is it required to use the tracker to get the analytics detecting the bounding box in the ROI correctly?

You can easily test this by removing the tracker in your application or using the GStreamer pipeline without tracker.

Best Regards,
Marco

RidgeRun AI | Artificial Intelligence Services | United States

[fanzh]

without tracker, using the same video and cfg, I can reproduce this issue. that is, when the bottom center coordinate (x_left + width/2, y_top + height) did not enter the ROI, the ROI count increased. we are checking then get back to you. Thanks!

[simon.renuart]

Hello,

I’ve encountered similar issues.
I was a bit surprised to see you confirm the results as correct with the tracker… What are your thoughts on these screenshots from certain frames?
On my side, I’ve also noticed some inconsistencies in line crossings.

@fanzh Is there a patch available?

Thank you

[fanzh]

@simon.renuart

@fanzh Is there a patch available?

@simon.renuart I am not able to reprodcue this issue with the latest version. could you try the latest nvcr.io/nvidia/deepstream:8.0-triton-multiarch? If the issue persists at your end, please share the test video, cfg. Thanks!

[simon.renuart]

Hi,

On my side, the issue is still there :

How to replicate :

deepstream-nvdsanalytics-test.zip (75.2 MB)

1. Start DS 8 container :

xhost +

docker run --gpus all -it --rm --runtime=nvidia --network=host --privileged -v /tmp/.X11-unix:/tmp/.X11-unix -e DISPLAY=$DISPLAY -w /opt/nvidia/deepstream/deepstream-8.0 nvcr.io/nvidia/deepstream:8.0-triton-multiarch

2. Replace deepstream-nvdsanalytics-test folder with mine so that you can produce video output as well.

3. Go in the folder

cd /opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-nvdsanalytics-test

export CUDA_VER=12.8

make

./deepstream-nvdsanalytics-test file:///opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-nvdsanalytics-test/vid1.mp4

4. Analyze frames of /opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-nvdsanalytics-test/deepstream_output.mp4

This issue occurs multiple time.

Thanks

[fanzh]

Thanks for the sharing! To be precise, the nvdsanalytcs uses the bboxes of the same object in the consective frames, not only one object. hence tracker is needed for recognizing the same object in consecutive frames. Since the nvdsanalytcs low-level is closed source, I may not be able to share further information. My apology for this.

1. As the stated above, please add tracker for the accuracy issue. Please let me know if there is still accuracy issue after adding tracker.
2. nvdsanalytics plugn is opensource. plese refer to topic for how to add custom algorithm.

[simon.renuart]

@fanzh

the nvdsanalytcs uses the bboxes of the same object in the consective frames, not only one object. hence tracker is needed for recognizing the same object in consecutive frames.

Hi,

Just to confirm: in the sample I am testing, the tracker is already enabled, so the current accuracy issue is not related to the absence of tracking. The bug is still present even with the tracker in the pipeline.

Regarding the point about “nvdsanalytics uses the bboxes of the same object in consecutive frames, not only one object, hence a tracker is needed for recognizing the same object in consecutive frames”: I would like to clarify that statement. By design, the tracker is precisely what ensures that bounding boxes from frame to frame are associated with the same object ID. nvdsanalytics then consumes these tracked objects. So the supposed requirement (“tracker is needed”) is already satisfied in the tested setup.

In other words, the accuracy issue we are observing cannot be explained simply by the lack of a tracker, since one is already present and functioning.

Could you please confirm if there are any known limitations or additional configuration parameters in nvdsanalytics that might explain this behavior?

Thanks

[fanzh]

@simon.renuart

Just to confirm: in the sample I am testing, the tracker is already enabled, so the current accuracy issue is not related to the absence of tracking. The bug is still present even with the tracker in the pipeline.

Since the original issue is regarding accuracy issue when tracker is disabled. could you open a new topic for this new issue? Thank you!

[simon.renuart]

Created here : Gst-nvdsanalytics roi filtering objects presence inconsistent

Thanks