Problem with detection in the ROI using nvdsanalytics

[vikas.wfh]

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU): Jetson Nano
• DeepStream Version: 5.1
• JetPack Version (valid for Jetson only): 4.5.1
• TensorRT Version: 7.1.3
• Issue Type( questions, new requirements, bugs): Object detection in the ROI using nvdsanalytics plugin

Hello,

I am trying to develop an application for vehicle counting in a specific region(ROI). I have take deepstream-nvdsanalytics-test application from deepstream sample application for my reference.

I am using nvdsanalytics plugin for the detection of vehicles in the ROI using Nvidia TrafficCamNet. As per the below post, the detection of vehicles in the ROI happens when the bottom line’s center of the object’s bounding box enters the edge(line) of ROI. And the detected object is removed from ROI when the bottom line’s center of the object’s bounding box exits the edge(line) of ROI.

nvdsanalytics-element-roi-strange-behaviour

When I run the application with my own configuration files, the following errors were observed :

1. Lag in between OSD and Terminal logs, logs are ahead of OSD.
2. The objects are still detected to be in the ROI even when the bottom bounding box line has exited ROI. This problem occurs 2 out of 3 times.

I have attached my logs for error 2 as above. My defined ROI is roi-ROI2=685;300;1060;300;1070;790;685;790, The vehicles are moving from top to bottom in the frames. After frame number 589, the vehicle bottom line of the bounding box exits from ROI2 but ROI2 status is still shown 1 for next 3 frames.

Frame Number = 575 of Stream = 0,    Objs in ROI3 = 0  Objs in ROI1 = 0  Objs in ROI2 = 0
Frame Number = 576 of Stream = 0,    Objs in ROI3 = 0  Objs in ROI1 = 0  Objs in ROI2 = 1 object 16678981843013337127 in ROI2 -  Bbox-> Left:846 Top:141 Width:155 Height:228
Frame Number = 577 of Stream = 0,    Objs in ROI3 = 0  Objs in ROI1 = 0  Objs in ROI2 = 1 object 16678981843013337127 in ROI2 -  Bbox-> Left:845 Top:148 Width:158 Height:232
Frame Number = 578 of Stream = 0,    Objs in ROI3 = 0  Objs in ROI1 = 0  Objs in ROI2 = 1 object 16678981843013337127 in ROI2 -  Bbox-> Left:843 Top:162 Width:166 Height:247
Frame Number = 579 of Stream = 0,    Objs in ROI3 = 0  Objs in ROI1 = 0  Objs in ROI2 = 1 object 16678981843013337127 in ROI2 -  Bbox-> Left:841 Top:170 Width:169 Height:251
Frame Number = 580 of Stream = 0,    Objs in ROI3 = 0  Objs in ROI1 = 0  Objs in ROI2 = 1 object 16678981843013337127 in ROI2 -  Bbox-> Left:839 Top:186 Width:175 Height:270
**PERF:  19.05 (20.03)	
Frame Number = 581 of Stream = 0,    Objs in ROI3 = 0  Objs in ROI1 = 0  Objs in ROI2 = 1 object 16678981843013337127 in ROI2 -  Bbox-> Left:837 Top:195 Width:179 Height:275
Frame Number = 582 of Stream = 0,    Objs in ROI3 = 0  Objs in ROI1 = 0  Objs in ROI2 = 1 object 16678981843013337127 in ROI2 -  Bbox-> Left:835 Top:212 Width:185 Height:298
Frame Number = 583 of Stream = 0,    Objs in ROI3 = 0  Objs in ROI1 = 0  Objs in ROI2 = 1 object 16678981843013337127 in ROI2 -  Bbox-> Left:834 Top:222 Width:189 Height:304
Frame Number = 584 of Stream = 0,    Objs in ROI3 = 0  Objs in ROI1 = 0  Objs in ROI2 = 1 object 16678981843013337127 in ROI2 -  Bbox-> Left:829 Top:242 Width:198 Height:329
Frame Number = 585 of Stream = 0,    Objs in ROI3 = 0  Objs in ROI1 = 0  Objs in ROI2 = 1 object 16678981843013337127 in ROI2 -  Bbox-> Left:828 Top:252 Width:203 Height:336
Frame Number = 586 of Stream = 0,    Objs in ROI3 = 0  Objs in ROI1 = 0  Objs in ROI2 = 1 object 16678981843013337127 in ROI2 -  Bbox-> Left:825 Top:276 Width:214 Height:361
Frame Number = 587 of Stream = 0,    Objs in ROI3 = 0  Objs in ROI1 = 0  Objs in ROI2 = 1 object 16678981843013337127 in ROI2 -  Bbox-> Left:824 Top:288 Width:218 Height:369
Frame Number = 588 of Stream = 0,    Objs in ROI3 = 0  Objs in ROI1 = 0  Objs in ROI2 = 1 object 16678981843013337127 in ROI2 -  Bbox-> Left:820 Top:315 Width:230 Height:399
Frame Number = 589 of Stream = 0,    Objs in ROI3 = 0  Objs in ROI1 = 0  Objs in ROI2 = 1 object 16678981843013337127 in ROI2 -  Bbox-> Left:818 Top:329 Width:235 Height:408
Frame Number = 590 of Stream = 0,    Objs in ROI3 = 0  Objs in ROI1 = 0  Objs in ROI2 = 1 object 16678981843013337127 in ROI2 -  Bbox-> Left:808 Top:362 Width:256 Height:446
Frame Number = 591 of Stream = 0,    Objs in ROI3 = 0  Objs in ROI1 = 0  Objs in ROI2 = 1 object 16678981843013337127 in ROI2 -  Bbox-> Left:805 Top:380 Width:262 Height:456
Frame Number = 592 of Stream = 0,    Objs in ROI3 = 0  Objs in ROI1 = 0  Objs in ROI2 = 1 object 16678981843013337127 in ROI2 -  Bbox-> Left:795 Top:417 Width:285 Height:503
Frame Number = 593 of Stream = 0,    Objs in ROI3 = 0  Objs in ROI1 = 0  Objs in ROI2 = 0
Frame Number = 594 of Stream = 0,    Objs in ROI3 = 0  Objs in ROI1 = 0  Objs in ROI2 = 0

For my application, I have customized the ROI coordinates in the config_nvdsanalytics.txt file shown below:

# The values in the config file are overridden by values set through GObject
# properties.

[property]
enable=1
#Width height used for configuration to which below configs are configured
config-width=1920
config-height=1080
#osd-mode 0: Dont display any lines, rois and text
#         1: Display only lines, rois and static text i.e. labels
#         2: Display all info from 1 plus information about counts
#osd-mode=2
osd-mode=2
#Set OSD font size that has to be displayed
display-font-size=12

## Per stream configuration
[roi-filtering-stream-0]
#enable or disable following feature
enable=1
#ROI to filter select objects, and remove from meta data
roi-ROI1=350;510;680;510;680;790;300;790
roi-ROI2=685;300;1060;300;1070;790;685;790
roi-ROI3=1075;420;1300;420;1550;790;1100;790
#remove objects in the ROI
inverse-roi=0
class-id=0

[overcrowding-stream-1]
enable=0
roi-OC=295;643;579;634;642;913;56;828
#no of objects that will trigger OC
object-threshold=3
class-id=0

[line-crossing-stream-0]
enable=0
line-crossing-Exit2=800;600;800;700;250;860;1550;860
class-id=0
#extended when 0- only counts crossing on the configured Line
#              1- assumes extended Line crossing counts all the crossing
extended=0
#LC modes supported:
#loose   : counts all crossing without strong adherence to direction
#balanced: Strict direction adherence expected compared to mode=loose
#strict  : Strict direction adherence expected compared to mode=balanced
mode=strict

[direction-detection-stream-0]
enable=0
#Label;direction;
direction-South=284;840;360;662;
direction-North=1106;622;1312;701;
class-id=0

Here is my main configuration file which is used to run the application.

[application]
enable-perf-measurement=1
perf-measurement-interval-sec=1

[tiled-display]
enable=1
rows=1
columns=1
width=1280
height=720
gpu-id=0

[source0]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=3
num-sources=1
uri=file:///home/spectross/Videos/640x480/250821/out_640x480_Day-Mor_Cut.mp4
gpu-id=0

[streammux]
gpu-id=0
batch-size=1
batched-push-timeout=40000
## Set muxer output width and height
width=1920
height=1080

[sink0]
enable=1
# AI: Disable the sink0
#enable=0
#Type - 1=FakeSink 2=EglSink 3=File
type=2
sync=0
source-id=0
gpu-id=0

[osd]
enable=1
# AI: Disable the osd
#enable=0
gpu-id=0
border-width=3
text-size=15
text-color=1;1;1;1;
text-bg-color=0.3;0.3;0.3;1
font=Arial

[primary-gie]
enable=1
gpu-id=0
# Modify as necessary
model-engine-file=../../../../samples/models/tlt_pretrained_models/trafficcamnet/resnet18_trafficcamnet_pruned.etlt_b1_gpu0_int8.engine
batch-size=1
#Required by the app for OSD, not a plugin property
bbox-border-color0=1;0;0;1
bbox-border-color1=0;1;1;1
bbox-border-color2=0;0;1;1
bbox-border-color3=0;1;0;1
gie-unique-id=1
config-file=config/config_infer_primary_trafficcamnet.txt

[tracker]
enable=1
tracker-width=640
tracker-height=384
# AI: Change tracker to nvdcf
#ll-lib-file=/opt/nvidia/deepstream/deepstream-5.1/lib/libnvds_mot_iou.so
ll-lib-file=/opt/nvidia/deepstream/deepstream-5.1/lib/libnvds_nvdcf.so
#ll-lib-file=/opt/nvidia/deepstream/deepstream-5.1/lib/libnvds_mot_klt.so
#ll-config-file required for DCF/IOU only
#ll-config-file=../deepstream-app/tracker_config.yml
ll-config-file=config/tracker_config.yml
#ll-config-file=iou_config.txt
gpu-id=0
#enable-batch-process applicable to DCF only
enable-batch-process=1
display-tracking-id=1

[nvds-analytics]
enable=1
config-file=config/config_nvdsanalytics.txt

[tests]
file-loop=0

I have the following queries

1. Is the lag stated above a normal behavior that could be possibly due to the lag between video pipeline and OSD pipeline or some other pipeline/reason.
2. Is my understanding correct about the center of the bottom line of the bounding box being used for detecting it in the ROI at both entry and exit from the ROI ?
3. How can I fix the error of the object being shown as detected in the ROI when it has already exited the ROI ?

Please help me to fix the problem.

Thanks
Vikas Dwivedi

[hitesh.pant21]

Hey Vikas,
I will have to replicate the issue to answer your 1st question.

Regarding the 2nd question:
Yes, the object is considered inside the ROI when more than 50% of the bottom line of the box**(foot coordinate of the bbox)** is inside the ROI.
Attaching a few images for your reference:

1. Since more than 50% of the foot coordinate of the bbox is out of both the ROI therefore count inside the roi is not incresed.
   

   

2. Similarly, in the following the though the 90% of the box is indise the ROI but the foot coordinate of the bbox is outside the ROI, hence no increase in the number inside the ROI.
   

   

3.This is an ideal case.

Regarding the 3rd question:
The issue is due to the delay in the pipeline,
when using deepstream, metadata is generated for every frame. Whichever object you are trying to detect is assigned an object_id.

Here is an other way round for this issue:
1.maintain a queue where you will append the object present in the frame, this data you can fetch from the metadata. Use the len of the queue to show the live count for the ROI.

2. from the incoming metadata for the next frames, check if the object_id stored in the queue is also present in the metadata.

3. if the object_id is not present in the metadata then pop out the object _id from the queue, else leave the object_id inside the queue.