Invalid device ordinal in nvtracker

When running deepstream pipeline with custom plugins I get (with 50% chance)

gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so
gstnvtracker: Batch processing is ON
gstnvtracker: Past frame output is OFF
GPUassert: invalid device ordinal src/modules/NvMultiObjectTracker/context.cpp 61

What causes this error? Our other plugins run with no problems. I get this error only if one concrete plugin (but not much different from the others) comes after NvDCF tracker in the pipeline, no other situation causes this.

Config of the tracker is here

%YAML:1.0
################################################################################
# Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved.
#
# Permission is hereby granted, free of charge, to any person obtaining a
# copy of this software and associated documentation files (the "Software"),
# to deal in the Software without restriction, including without limitation
# the rights to use, copy, modify, merge, publish, distribute, sublicense,
# and/or sell copies of the Software, and to permit persons to whom the
# Software is furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
# DEALINGS IN THE SOFTWARE.
################################################################################

#./samples/configs/deepstream-app/config_tracker_NvDCF_accuracy.yml

enable: 1
# For the case of NvDCF tracker, tracker-width and tracker-height must be a multiple of 32, respectively
tracker-width: 640
tracker-height: 384
ll-lib-file: "/opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so"
ll-config-file: "config/tracker_config.yml"
gpu-id: 0
#enable-batch-process and enable-past-frame applicable to DCF only
enable-batch-process: 1
enable-past-frame: 0
display-tracking-id: 1
compute-hw: 0
tracking-id-reset-mode: 0
tracking-surface-type: 0

BaseConfig:
    minDetectorConfidence: 0   # If the confidence of a detector bbox is lower than this, then it won't be considered for tracking

TargetManagement:
    enableBboxUnClipping: 1   # In case the bbox is likely to be clipped by image border, unclip bbox
    preserveStreamUpdateOrder: 0 # When assigning new target ids, preserve input streams' order to keep target ids in a deterministic order over multuple runs
    maxTargetsPerStream: 150  # Max number of targets to track per stream. Recommended to set >10. Note: this value should account for the targets being tracked in shadow mode as well. Max value depends on the GPU memory capacity

    # [Creation & Termination Policy]
    minIouDiff4NewTarget: 0.5   # If the IOU between the newly detected object and any of the existing targets is higher than this threshold, this newly detected object will be discarded.
    minTrackerConfidence: 0.2   # If the confidence of an object tracker is lower than this on the fly, then it will be tracked in shadow mode. Valid Range: [0.0, 1.0]
    probationAge: 3 # If the target's age exceeds this, the target will be considered to be valid.
    maxShadowTrackingAge: 5   # Max length of shadow tracking. If the shadowTrackingAge exceeds this limit, the tracker will be terminated.
    earlyTerminationAge: 1      # If the shadowTrackingAge reaches this threshold while in TENTATIVE period, the target will be terminated prematurely.

TrajectoryManagement:
    useUniqueID: 1   # Use 64-bit long Unique ID when assignining tracker ID. Default is [true]
    enableReAssoc: 1    # Enable Re-Assoc

    # [Re-Assoc: Motion-based]
    minTrajectoryLength4Projection: 20  # min trajectory length required to make projected trajectory
    prepLength4TrajectoryProjection: 10  # the length of the trajectory during which the state estimator is updated to make projections
    trajectoryProjectionLength: 90  # the length of the projected trajectory

    # [Re-Assoc: Trajectory Similarity]
    minTrackletMatchingScore: 0.5   # min tracklet similarity score for matching
    maxAngle4TrackletMatching: 20   # max angle difference for tracklet matching [degree]
    minSpeedSimilarity4TrackletMatching: 0.1 # min speed similarity for tracklet matching
    minBboxSizeSimilarity4TrackletMatching: 0.6 # min bbox size similarity for tracklet matching
    maxTrackletMatchingTimeSearchRange: 20      # the search space in time for max tracklet similarity

DataAssociator:
    dataAssociatorType: 0 # the type of data associator among { DEFAULT= 0 }
    associationMatcherType: 0 # the type of matching algorithm among { GREEDY=0, GLOBAL=1 }
    checkClassMatch: 1  # If checked, only the same-class objects are associated with each other. Default: true

    # [Association Metric: Thresholds for valid candidates]
    minMatchingScore4Overall: 0.0   # Min total score
    minMatchingScore4SizeSimilarity: 0.5  # Min bbox size similarity score
    minMatchingScore4Iou: 0.3       # Min IOU score
    minMatchingScore4VisualSimilarity: 0.6  # Min visual similarity score

    # [Association Metric: Weights]
    matchingScoreWeight4VisualSimilarity: 0.5  # Weight for the visual similarity (in terms of correlation response ratio)
    matchingScoreWeight4SizeSimilarity: 0.0    # Weight for the Size-similarity score
    matchingScoreWeight4Iou: 0.1   # Weight for the IOU score

StateEstimator:
    stateEstimatorType: 1  # the type of state estimator among { DUMMY=0, SIMPLE=1, REGULAR=2 }

    # [Dynamics Modeling]
    processNoiseVar4Loc: 2.0    # Process noise variance for bbox center
    processNoiseVar4Size: 1.0   # Process noise variance for bbox size
    processNoiseVar4Vel: 0.1    # Process noise variance for velocity
    measurementNoiseVar4Detector: 2.0    # Measurement noise variance for detector's detection
    measurementNoiseVar4Tracker: 10.0    # Measurement noise variance for tracker's localization

VisualTracker:
    visualTrackerType: 1 # the type of visual tracker among { DUMMY=0, NvDCF=1 }

    # [NvDCF: Feature Extraction]
    useColorNames: 1     # Use ColorNames feature
    useHog: 1            # Use Histogram-of-Oriented-Gradient (HOG) feature
    featureImgSizeLevel: 3  # Size of a feature image. Valid range: {1, 2, 3, 4, 5}, from the smallest to the largest
    featureFocusOffsetFactor_y: -0.2 # The offset for the center of hanning window relative to the feature height. The center of hanning window would move by (featureFocusOffsetFactor_y*featureMatSize.height) in vertical direction

    # [NvDCF: Correlation Filter]
    filterLr: 0.075 # learning rate for DCF filter in exponential moving average. Valid Range: [0.0, 1.0]
    filterChannelWeightsLr: 0.1 # learning rate for the channel weights among feature channels. Valid Range: [0.0, 1.0]
    gaussianSigma: 0.75 # Standard deviation for Gaussian for desired response when creating DCF filter [pixels]

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU):
• DeepStream Version:
• TensorRT Version:
• NVIDIA GPU Driver Version (valid for GPU only):

I have found what caused the problem.

In the plugin we call cudaSetDevice(gpuId), where gpuId was sometimes of non-visible gpu. When gpuId was not valid, tracker later crashed with aforementioned error. When I corrected the gpuId to valid numbers, problem solved.

It seems that cudaSetDevice used elsewhere has some effect on tracker, even when we use g_object_set to set gpu-id for it.

Platform: GPU
Deepstream 6.2
TensorRT 8.5.2
gpu driver version: 525.89.02

Yes, please set right GPU ID. Thanks for reporting it.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.