Please provide complete information as applicable to your setup.
• Hardware Platform (Jetson / GPU): Jetson Orin
• DeepStream Version: 5.2
• JetPack Version (valid for Jetson only): 5.2
• TensorRT Version: Unknown
• NVIDIA GPU Driver Version (valid for GPU only): Unknown
• Issue Type( questions, new requirements, bugs): Question
I have this trace I obtained when using nvidia nsight.
I am wondering if anyone has any insight into reasons why nvtracker might take 9.702s for the first batch and then proceed quickly from there.
It appears to be loading yaml.
Is this expected? Is there any way to move this loading time to earlier so I don’t hold up processing waiting for it to complete once we start receiving frames?
I am running this for my custom application but can attempt to reproduce with an existing sample if this is not expected.
Best regards.
Which DeepStream version and JetPack are you using? Can you have a check on latest DeepStream 7.0 on JetPack 6.0 GA?
Can you have a check on latest DeepStream 7.0 on JetPack 6.0 GA?
I am not sure. I don’t control the OS on the boards. I can look into this.
do you know if what I am seeing is expected or not? I am just trying to get a little bit of clarity before I start doing an investigation.
thanks.
Re-ID model build will consume time. Can you share the nsys profile data? So we can analysis it.
I can, is there a way I can send it to you privately?
I have send one private message to you. You can reply in that private channel.
Thanks. I have sent you the report + logs.
In case it is helpful to others, the command I was using is as follows:
gst-launch-1.0 rtspsrc name=src-rtspsrc location=rtsp://10.101.8.120:554/stream_0 protocols=tcp latency=5000 drop-on-latency=True \
! rtph264depay name=src-rtph264depay \
! h264parse name=src-hparse config-interval=-1 \
! queue name=src-queue max-size-buffers=0 max-size-time=120000000000 max-size-bytes=0 \
! nvv4l2decoder name=nvv4l2decoder num-extra-surfaces=55 enable-max-performance=1 disable-dpb=True \
! nvvideoconvert name=src-nvvideoconvert gpu-id=0 \
! i-mux.sink_0 nvstreammux name=i-mux gpu-id=0 batch-size=16 width=1920 height=1080 attach-sys-ts=1 num-surfaces-per-frame=1 live-source=0 sync-inputs=0 buffer-pool-size=128 batched-push-timeout=916667 max-latency=3000000000 frame-duration=-1 \
! progressreport \
! queue name=infer-queue max-size-buffers=0 max-size-bytes=0 max-size-time=0 \
! nvinfer name=i-infer gpu-id=0 batch-size=16 config-file-path=/etc/processor/outputs/2024-05-30/17-07-36-824549/config_infer_primary.yaml \
! queue name=track-queue max-size-buffers=0 max-size-bytes=0 max-size-time=0 \
! nvtracker name=t-nvtracker gpu-id=0 ll-lib-file=/opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so ll-config-file=/etc/processor/outputs/2024-05-30/17-07-36-824549/config_tracker_NvSORT_custom.yaml \
! queue name=sink-queue max-size-buffers=0 max-size-bytes=0 max-size-time=0 \
! fakesink name=sk-fakesink sync=False enable-last-sample=0
edit: Also, I don’t think I am using Re-ID as we are not using DCF.
Also, here is the NVSORT_custom_template.yml contents
BaseConfig:
# If the confidence of a detector bbox is lower than this, then it won't be
# considered for tracking
minDetectorConfidence: 0.2
TargetManagement:
# In case the bbox is likely to be clipped by image border, unclip bbox
enableBboxUnClipping: 0
# Max number of targets to track per stream. Recommended to set >10.
# Note: this value should account for the targets being tracked in shadow mode
# as well. Max value depends on the GPU memory capacity
maxTargetsPerStream: 100
#
# [Creation & Termination Policy]
#
# If the IOU between the newly detected object and any of the existing targets
# is higher than this threshold, this newly detected object will be discarded.
minIouDiff4NewTarget: 0.26006486048167077
# If the confidence of an object tracker is lower than this on the fly, then
# it will be tracked in shadow mode. Valid Range: [0.0, 1.0]
minTrackerConfidence: 0.008134201107677205
# If the target's age exceeds this, the target will be considered to be valid.
probationAge: 1
# Max length of shadow tracking. If the shadowTrackingAge exceeds this limit,
# the tracker will be terminated.
maxShadowTrackingAge: 5
# If the shadowTrackingAge reaches this threshold while in TENTATIVE period,
# the target will be terminated prematurely.
earlyTerminationAge: 1
TrajectoryManagement:
# Use 64-bit long Unique ID when assignining tracker ID. Default is [true]
useUniqueID: 1
DataAssociator:
# the type of data associator among { DEFAULT= 0 }
dataAssociatorType: 0
# the type of matching algorithm among { GREEDY=0, CASCADED=1 }
associationMatcherType: 0
# If checked, only the same-class objects are associated with each other.
# Default: true
checkClassMatch: 1
#
# [Association Metric: Thresholds for valid candidates]
#
# Min total score
minMatchingScore4Overall: 0.0026861559707920825
# Min bbox size similarity score
minMatchingScore4SizeSimilarity: 0.1 # Size similarity in our case is important
# Min IOU score
minMatchingScore4Iou: 0.006837245935615654
# Weight for the Size-similarity score
matchingScoreWeight4SizeSimilarity: 0.1
# Weight for the IOU score
matchingScoreWeight4Iou: 0.04708585446530863
#
# [Association Metric: Tentative detections] only uses iou similarity for
# tentative detections
#
# If a detection's confidence is lower than this but higher than
# minDetectorConfidence, then it's considered as a tentative detection
tentativeDetectorConfidence: 0.7628657651628549
# Min iou threshold to match targets and tentative detection
minMatchingScore4TentativeIou: 0.08268307009366113
# use the predicted state info for association instead of the past known
# states
usePrediction4Assoc: 1
StateEstimator:
# the type of state estimator among { DUMMY=0, SIMPLE=1, REGULAR=2 }
stateEstimatorType: 2
#
# [Dynamics Modeling]
#
# weight of process and measurement noise for bbox center; if set, location
# noise will be proportional to box height
noiseWeightVar4Loc: 0.13762054930701298
# weight of process and measurement noise for velocity; if set, velocity noise
# will be proportional to box height
noiseWeightVar4Vel: 0.01 # Presumably much less noise compared to water
# use aspect ratio in Kalman filter's observation
useAspectRatio: 0
