•Nvidia Jetson Xavier
•Deepstream 6.1.1
•Jetpack 5.0.2-b231
•TensorRT 8.4.1-1
•Issue Type-Question
• Does the NvDCF Tracker have the ability to output metadata between Object Detection intervals? For example, if I set interval=5 for the PGIE, this means that Object Detection will only run inference every 5 frames. If Object Detection detects a target on one frame will the tracker output metadata with estimated bbox data between each inference interval?
Yes, please refer: Gst-nvtracker — DeepStream 6.2 Release documentation
I am currently running the max_perf configuration and in between frames I don’t see any metadata being output for the tracker. Does it have to be the accuracy configuration for it to produce tracker metadata between non-inferenced frames?
Also, if the interval=0 does the NvDCF tracker still produce metadata after inference has lost the object? For example, lets say object detection detects a human at frame N=0, will the NvDCF continue to track this object after, even if Object Detection doesn’t detect the object again till frame N=5?
Thanks
Yes, please ensure maxShadowTrackingAge in tracker configure file is larger then PGIE inference interval.
Thanks! And just to confirm, the NvDsObjectMeta will contain tracker information even if the object isn’t present from inference?
I’m seeing this in my logs:
Mar 01 11:01:33 ubuntu deepstream-app[583482]: DSLOG:NVDS_REDIS_PROTO: nvds_msgapi_do_work: Message sent = {
"version" : "4.0",
"id" : "1433",
"timestamp" : "2023-03-01T16:01:33.008Z",
"sensorId" : "onvif_std",
"videoURL" : "rtsp://onviftest:grumpycat@10.1.10.58:8554/XOD",
"frameWidth" : 1920,
"frameHeight" : 1080,
"objects" : [
{
"objectId" : "2",
"label" : "drone",
"confidence" : "0.922604",
"topLeftX" : "1191.564941",
"topLeftY" : "233.309372",
"bottomRightX" : "1296.991455",
"bottomRightY" : "294.690613",
"tracker_bbox" : {
"topleftx" : 1182,
"toplefty" : 233,
"bottomrightx" : 1304,
"bottomrighty" : 295,
"confidence" : 1.0441417694091797
}
}
]
}
Mar 01 11:01:33 ubuntu deepstream-app[583482]: DSLOG:NVDS_REDIS_PROTO: nvds_msgapi_do_work: Message sent = {
"version" : "4.0",
"id" : "1434",
"timestamp" : "2023-03-01T16:01:33.031Z",
"sensorId" : "onvif_std",
"videoURL" : "rtsp://onviftest:grumpycat@10.1.10.58:8554/XOD",
"frameWidth" : 1920,
"frameHeight" : 1080,
"objects" : [
]
}
Mar 01 11:01:33 ubuntu deepstream-app[583482]: DSLOG:NVDS_REDIS_PROTO: nvds_msgapi_do_work: Message sent = {
"version" : "4.0",
"id" : "1435",
"timestamp" : "2023-03-01T16:01:33.058Z",
"sensorId" : "onvif_std",
"videoURL" : "rtsp://onviftest:grumpycat@10.1.10.58:8554/XOD",
"frameWidth" : 1920,
"frameHeight" : 1080,
"objects" : [
]
}
As soon as the object isn’t detected by inference anymore the tracker information also disappears. How do I get the tracker information between frames when the object is not being detected?
Now if I change the interval of my PGIE to interval=3(meaning inference will only be ran every 4 frames), then I do see the tracker information like so:
Mar 01 11:08:18 ubuntu deepstream-app[583574]: DSLOG:NVDS_REDIS_PROTO: nvds_msgapi_do_work: Message sent = {
"version" : "4.0",
"id" : "748",
"timestamp" : "2023-03-01T16:08:18.463Z",
"sensorId" : "onvif_std",
"videoURL" : "rtsp://onviftest:grumpycat@10.1.10.58:8554/XOD",
"frameWidth" : 1920,
"frameHeight" : 1080,
"objects" : [
{
"objectId" : "1",
"label" : "drone",
"confidence" : "0.890529",
"topLeftX" : "914.882996",
"topLeftY" : "209.746399",
"bottomRightX" : "1028.759521",
"bottomRightY" : "257.995789",
"tracker_bbox" : {
"topleftx" : 914,
"toplefty" : 209,
"bottomrightx" : 1029,
"bottomrighty" : 258,
"confidence" : 0.96500313282012939
}
}
]
}
Mar 01 11:08:18 ubuntu deepstream-app[583574]: DSLOG:NVDS_REDIS_PROTO: nvds_msgapi_do_work: Message sent = {
"version" : "4.0",
"id" : "749",
"timestamp" : "2023-03-01T16:08:18.484Z",
"sensorId" : "onvif_std",
"videoURL" : "rtsp://onviftest:grumpycat@10.1.10.58:8554/XOD",
"frameWidth" : 1920,
"frameHeight" : 1080,
"objects" : [
{
"objectId" : "1",
"label" : "drone",
"confidence" : "-0.100000",
"topLeftX" : "0.000000",
"topLeftY" : "0.000000",
"bottomRightX" : "0.000000",
"bottomRightY" : "0.000000",
"tracker_bbox" : {
"topleftx" : 914,
"toplefty" : 209,
"bottomrightx" : 1029,
"bottomrighty" : 258,
"confidence" : 1.1426326036453247
}
}
]
}
Mar 01 11:08:18 ubuntu deepstream-app[583574]: DSLOG:NVDS_REDIS_PROTO: nvds_msgapi_do_work: Message sent = {
"version" : "4.0",
"id" : "750",
"timestamp" : "2023-03-01T16:08:18.515Z",
"sensorId" : "onvif_std",
"videoURL" : "rtsp://onviftest:grumpycat@10.1.10.58:8554/XOD",
"frameWidth" : 1920,
"frameHeight" : 1080,
"objects" : [
{
"objectId" : "1",
"label" : "drone",
"confidence" : "-0.100000",
"topLeftX" : "0.000000",
"topLeftY" : "0.000000",
"bottomRightX" : "0.000000",
"bottomRightY" : "0.000000",
"tracker_bbox" : {
"topleftx" : 914,
"toplefty" : 209,
"bottomrightx" : 1029,
"bottomrighty" : 258,
"confidence" : 1.1311823129653931
}
}
]
}
Mar 01 11:08:18 ubuntu deepstream-app[583574]: DSLOG:NVDS_REDIS_PROTO: nvds_msgapi_do_work: Message sent = {
"version" : "4.0",
"id" : "751",
"timestamp" : "2023-03-01T16:08:18.566Z",
"sensorId" : "onvif_std",
"videoURL" : "rtsp://onviftest:grumpycat@10.1.10.58:8554/XOD",
"frameWidth" : 1920,
"frameHeight" : 1080,
"objects" : [
{
"objectId" : "1",
"label" : "drone",
"confidence" : "-0.100000",
"topLeftX" : "0.000000",
"topLeftY" : "0.000000",
"bottomRightX" : "0.000000",
"bottomRightY" : "0.000000",
"tracker_bbox" : {
"topleftx" : 914,
"toplefty" : 209,
"bottomrightx" : 1029,
"bottomrighty" : 258,
"confidence" : 1.1242440938949585
}
}
]
}
Mar 01 11:08:18 ubuntu deepstream-app[583574]: DSLOG:NVDS_REDIS_PROTO: nvds_msgapi_do_work: Message sent = {
"version" : "4.0",
"id" : "752",
"timestamp" : "2023-03-01T16:08:18.617Z",
"sensorId" : "onvif_std",
"videoURL" : "rtsp://onviftest:grumpycat@10.1.10.58:8554/XOD",
"frameWidth" : 1920,
"frameHeight" : 1080,
"objects" : [
{
"objectId" : "1",
"label" : "drone",
"confidence" : "0.914171",
"topLeftX" : "911.987915",
"topLeftY" : "209.465012",
"bottomRightX" : "1031.654663",
"bottomRightY" : "260.147156",
"tracker_bbox" : {
"topleftx" : 913,
"toplefty" : 209,
"bottomrightx" : 1030,
"bottomrighty" : 259,
"confidence" : 1.0783514976501465
}
}
]
}
Let’s say for example I am tracking an Object with a PTZ camera and it has autofocus that causes the image to blur for a few frames which causes inference to not detect the object for a few frames. However, we’ve already detected this object before, will the tracker still maintain this blurred object after an inference call doesn’t detect anything?
I think tracker prediction is still an internal data right now. You can’t access it. I had the same problem and implemented a kalman filter Gst-nvtracker - #8 by pshin
Thanks!
So, what I am gathering is that it only reports tracker data when you set the interval of the PGIE higher than 0 and even then it only reports the tracker information between non-inferenced frames. It will NOT report tracker information again if the latest inference doesn’t detect the object again.
Which contradicts the documentation found here for the NvDCF tracker:
https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvtracker.html
It states:
The NvDCF tracker is an online multi-object tracker that employes a discriminative correlation filter for visual object tracking, which allows indepedent object tracking even when detection results are not available. It uses the combination of the correlation filter responses and bounding box proximity for data association.
How does one get the Tracker results if detection results aren’t available?
FYI:
I had to go into /opt/Nvidia/deepstream/deepstream-6.2/sources/libs/nvmsgconv/ and edit dsmeta_payload.c to report the Tracker data by the way.
I am brokering the tracker/detection info to another program using Redis Streams that uses a PTZ camera to track the bounding box/tracker info.
Can you have a try with NV sample in below link? So we can check it with NV sample.
There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks
Hello,
NvDCF tracker performs Shadow Tracking even when an object is not detected due to false negatives or PGIE interval > 0. The tracking data during Shadow Tracking is not reported as metadata to the downstream (so not shown on OSD nor appear in msgBroker messages), but once the object is activated again (i.e., re-detected and identified as the same object), such “past-frame data” is reported to the downstream as usermeta. Please search “past-frame” in our tracker doc Gst-nvtracker — DeepStream 6.2 Release documentation, and make sure you enable past-frame data in the pipeline.
Pls check write_kitti_past_track_output() in deepstream_app.c file to see how to access the past-frame data in usermeta.
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.