Adding tracker after two back to back detectors?

So we were trying to have two object detectors working inside deepstream and we were able to get that to work but the only issue we are facing right now is with the tracker. The issue what we are currently facing is that,

let’s say the first detector is person and the second detector is a face, the moment we have the tracker after the two object detectors in the pipeline, the label of the face is getting flipped between person and face, which kinda looks strange, so due to the tracking interval being set, the tracker is making the label of the face to person. Any idea as to how we could fix this?

Is there any problem if only person detector with tracker ?