NvDCF Tracker After nvdsmetamux: Can It Handle Objects from Multiple GIE Sources?

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU): Orin AGX
• DeepStream Version: 7.0
• JetPack Version (valid for Jetson only): 6.0

Context - Working Configuration

We have a DeepStream application with **parallel inference using 2 models** running successfully:

Pipeline architecture:

  • source → streammux
    • Branch 1: pgie1 (unique_id=1) → tracker0 → nvdsmetamux
    • Branch 2: pgie2 (unique_id=2) → tracker1 → nvdsmetamux
  • nvdsmetamux → streamdemux → OSD → sinks

Configuration details:

  • 2 primary GIEs with different models (unique_component_id: 1 and 2)
  • Each branch has its own NvDCF tracker (tracker0 and tracker1)
  • nvdsmetamux aggregates metadata from both branches
  • This configuration works correctly: bboxes are displayed with tracking IDs on correct RTSP Out

Problem Encountered

We’re experiencing PTS tolerance timeout issues where:

  • When one model’s inference takes longer, its metadata arrives late to the metamux
  • The metamux drops the late metadata entirely (exceeds `pts-tolerance`)
  • Result: complete loss of information from one model, including tracker interpolation data
  • Even with `pts-tolerance=200000`, the issue persists under load

Attempted Solution - Single Tracker After Metamux

To solve the PTS issue, we want to place a single tracker AFTER the metamux to aggregate objects from both models before tracking:

  • source → streammux
    • Branch 1: pgie1 (unique_id=1) → nvdsmetamux
    • Branch 2: pgie2 (unique_id=2) → nvdsmetamux
  • nvdsmetamux → SINGLE TRACKER → streamdemux → OSD → sinks

Implentation:

  • Removed tracker0 and tracker1 from branches
  • Created single tracker bin after nvdsmetamux using standard NvDCF tracker
  • Tracker configuration: `ll-lib-file: /opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so`

nvdsmetamux configuration

[property]

enable=1

# active-pad commented out to allow metadata fusion

# active-pad=sink_0 #tested with and without

pts-tolerance=200000

[group-0]

src-ids-model-1=0;1

src-ids-model-2=0;1

Current Issue

The application runs without errors, but:

What works:

  • Both GIEs detect objects correctly
  • Metadata is aggregated by nvdsmetamux (verified in logs)
  • Example: `Frame 1 Source 0: Total=17 (GIE1=4, GIE2=13)

What does not work:

  • No bboxes appear on video output
  • The tracker appears to not process the aggregated objects

Questions

1. Is NvDCF tracker designed to handle objects from multiple GIE sources (different unique_component_id values) in a single frame?

2. Does the tracker filter objects based on `gie_unique_id` even when no `gie-id` filter is specified in the config?

3. Is our architecture valid? Should metadata from multiple parallel GIEs be aggregated BEFORE or AFTER tracking?

4. Are there any specific tracker configuration parameters required to accept objects from multiple GIE sources?

5. Is there an alternative approach to solve the PTS timeout issue while maintaining tracking across both model outputs?

Thanks for any help.

3

Why the metadata arrives late? We need to check this before change the pipeline. Are you set Orin to MAXN and boost the clock?

4

Yes, nvp model is MAXN and jetson clocks is running. Here is the performance measured at the sink pad of the nvstreamdemux element:

**PERF: 25.13 (25.33) 25.13 (25.31)

5

Here are the results:

  • A frame is inferred when a bounding box is drawn and the confidence is > 0.

  • An object is only tracked when the bounding box is drawn and the confidence is -1.

  • Frame metadata is skipped by the metamux when nothing appears.

I also logged the presence of objects in my logic placed after the metamux from both branches:

Source 0, Frame 346: branch_0_meta=1, branch_1_meta=1
Source 1, Frame 342: branch_0_meta=1, branch_1_meta=0
Source 1, Frame 343: branch_0_meta=1, branch_1_meta=0
Source 0, Frame 347: branch_0_meta=1, branch_1_meta=1
Source 1, Frame 344: branch_0_meta=1, branch_1_meta=0
Source 0, Frame 348: branch_0_meta=1, branch_1_meta=1
Source 1, Frame 345: branch_0_meta=1, branch_1_meta=0
Source 0, Frame 349: branch_0_meta=1, branch_1_meta=1
Source 1, Frame 346: branch_0_meta=0, branch_1_meta=0
Source 0, Frame 350: branch_0_meta=0, branch_1_meta=1
Source 1, Frame 347: branch_0_meta=0, branch_1_meta=0
Source 0, Frame 351: branch_0_meta=0, branch_1_meta=1
Source 1, Frame 348: branch_0_meta=0, branch_1_meta=0
Source 0, Frame 352: branch_0_meta=0, branch_1_meta=0
Source 1, Frame 349: branch_0_meta=0, branch_1_meta=0
Source 0, Frame 353: branch_0_meta=0, branch_1_meta=0
Source 1, Frame 350: branch_0_meta=1, branch_1_meta=0
Source 0, Frame 354: branch_0_meta=1, branch_1_meta=1
Source 1, Frame 351: branch_0_meta=1, branch_1_meta=0

Very often, metadata from one branch or both does not pass through the metamux.

6

The problem stays even with only one branch for two streams but does not appear any more with one single stream.

7

I added a callback on each branch to compare pts:

[Branch 1] Source 0, Frame 97, pts=4875749432
[Branch 0] Source 0, Frame 97, pts=4875749432

I compared many pairs of the same image passing through each branch. Each time, both frames had the same PTS.