Cannot load built engine resnet50_market1501_aicity156

• 3060
• DeepStream Version 6.4
• Bug

For a deepstream python application, I use the config_tracker_NvDCF_accuracy tracker.

Here’s the config file used:

%YAML:1.0
################################################################################
# Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved.
#
# Permission is hereby granted, free of charge, to any person obtaining a
# copy of this software and associated documentation files (the "Software"),
# to deal in the Software without restriction, including without limitation
# the rights to use, copy, modify, merge, publish, distribute, sublicense,
# and/or sell copies of the Software, and to permit persons to whom the
# Software is furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
# DEALINGS IN THE SOFTWARE.
################################################################################

BaseConfig:
  minDetectorConfidence: 0.1894    # If the confidence of a detector bbox is lower than this, then it won't be considered for tracking

TargetManagement:
  enableBboxUnClipping: 1    # In case the bbox is likely to be clipped by image border, unclip bbox
  preserveStreamUpdateOrder: 0    # When assigning new target ids, preserve input streams' order to keep target ids in a deterministic order over multuple runs
  maxTargetsPerStream: 150    # Max number of targets to track per stream. Recommended to set >10. Note: this value should account for the targets being tracked in shadow mode as well. Max value depends on the GPU memory capacity

  # [Creation & Termination Policy]
  minIouDiff4NewTarget: 0.3686    # If the IOU between the newly detected object and any of the existing targets is higher than this threshold, this newly detected object will be discarded.
  minTrackerConfidence: 0.1513    # If the confidence of an object tracker is lower than this on the fly, then it will be tracked in shadow mode. Valid Range: [0.0, 1.0]
  probationAge: 2    # If the target's age exceeds this, the target will be considered to be valid.
  maxShadowTrackingAge: 42    # Max length of shadow tracking. If the shadowTrackingAge exceeds this limit, the tracker will be terminated.
  earlyTerminationAge: 1    # If the shadowTrackingAge reaches this threshold while in TENTATIVE period, the target will be terminated prematurely.

TrajectoryManagement:
  useUniqueID: 0    # Use 64-bit long Unique ID when assignining tracker ID. Default is [true]
  enableReAssoc: 1    # Enable Re-Assoc

  # [Re-Assoc Metric: Thresholds for valid candidates]
  minMatchingScore4Overall: 0.6622    # min matching score for overall
  minTrackletMatchingScore: 0.2940    # min tracklet similarity score for re-assoc
  minMatchingScore4ReidSimilarity: 0.0771    # min reid similarity score for re-assoc

  # [Re-Assoc Metric: Weights]
  matchingScoreWeight4TrackletSimilarity: 0.7981    # weight for tracklet similarity score
  matchingScoreWeight4ReidSimilarity: 0.3848    # weight for reid similarity score

  # [Re-Assoc: Motion-based]
  minTrajectoryLength4Projection: 34    # min trajectory length required to make projected trajectory
  prepLength4TrajectoryProjection: 58    # the length of the trajectory during which the state estimator is updated to make projections
  trajectoryProjectionLength: 33    # the length of the projected trajectory
  maxAngle4TrackletMatching: 67    # max angle difference for tracklet matching [degree]
  minSpeedSimilarity4TrackletMatching: 0.0574    # min speed similarity for tracklet matching
  minBboxSizeSimilarity4TrackletMatching: 0.1013    # min bbox size similarity for tracklet matching
  maxTrackletMatchingTimeSearchRange: 27    # the search space in time for max tracklet similarity
  trajectoryProjectionProcessNoiseScale: 0.0100    # trajectory projector's process noise scale w.r.t. state estimator
  trajectoryProjectionMeasurementNoiseScale: 100    # trajectory projector's measurement noise scale w.r.t. state estimator
  trackletSpacialSearchRegionScale: 0.0100    # the search region scale for peer tracklet

  # [Re-Assoc: Reid based. Reid model params are set in ReID section]
  reidExtractionInterval: 8    # frame interval to extract reid features per target

DataAssociator:
  dataAssociatorType: 0    # the type of data associator among { DEFAULT= 0 }
  associationMatcherType: 1    # the type of matching algorithm among { GREEDY=0, CASCADED=1 }
  checkClassMatch: 1    # If checked, only the same-class objects are associated with each other. Default: true

  # [Association Metric: Thresholds for valid candidates]
  minMatchingScore4Overall: 0.0222    # Min total score
  minMatchingScore4SizeSimilarity: 0.3552    # Min bbox size similarity score
  minMatchingScore4Iou: 0.0548   # Min IOU score
  minMatchingScore4VisualSimilarity: 0.5043    # Min visual similarity score

  # [Association Metric: Weights]
  matchingScoreWeight4VisualSimilarity: 0.3951    # Weight for the visual similarity (in terms of correlation response ratio)
  matchingScoreWeight4SizeSimilarity: 0.6003    # Weight for the Size-similarity score
  matchingScoreWeight4Iou: 0.4033    # Weight for the IOU score

  # [Association Metric: Tentative detections] only uses iou similarity for tentative detections
  tentativeDetectorConfidence: 0.1024    # If a detection's confidence is lower than this but higher than minDetectorConfidence, then it's considered as a tentative detection
  minMatchingScore4TentativeIou: 0.2852    # Min iou threshold to match targets and tentative detection

StateEstimator:
  stateEstimatorType: 1    # the type of state estimator among { DUMMY=0, SIMPLE=1, REGULAR=2 }

  # [Dynamics Modeling]
  processNoiseVar4Loc: 6810.8668    # Process noise variance for bbox center
  processNoiseVar4Size: 1541.8647   # Process noise variance for bbox size
  processNoiseVar4Vel: 1348.4874    # Process noise variance for velocity
  measurementNoiseVar4Detector: 100.0000   # Measurement noise variance for detector's detection
  measurementNoiseVar4Tracker: 293.3238    # Measurement noise variance for tracker's localization

VisualTracker:
  visualTrackerType: 1    # the type of visual tracker among { DUMMY=0, NvDCF=1 }

  # [NvDCF: Feature Extraction]
  useColorNames: 1    # Use ColorNames feature
  useHog: 1    # Use Histogram-of-Oriented-Gradient (HOG) feature
  featureImgSizeLevel: 3    # Size of a feature image. Valid range: {1, 2, 3, 4, 5}, from the smallest to the largest
  featureFocusOffsetFactor_y: -0.1054    # The offset for the center of hanning window relative to the feature height. The center of hanning window would move by (featureFocusOffsetFactor_y*featureMatSize.height) in vertical direction

  # [NvDCF: Correlation Filter]
  filterLr: 0.0767    # learning rate for DCF filter in exponential moving average. Valid Range: [0.0, 1.0]
  filterChannelWeightsLr: 0.0339    # learning rate for the channel weights among feature channels. Valid Range: [0.0, 1.0]
  gaussianSigma: 0.5687    # Standard deviation for Gaussian for desired response when creating DCF filter [pixels]   

ReID:
  reidType: 2    # The type of reid among { DUMMY=0, NvDEEPSORT=1, Reid based reassoc=2, both NvDEEPSORT and reid based reassoc=3}

  # [Reid Network Info]
  batchSize: 100    # Batch size of reid network
  workspaceSize: 1000    # Workspace size to be used by reid engine, in MB
  reidFeatureSize: 256    # Size of reid feature
  reidHistorySize: 100    # Max number of reid features kept for one object
  inferDims: [3, 256, 128]    # Reid network input dimension CHW or HWC based on inputOrder
  networkMode: 1    # Reid network inference precision mode among {fp32=0, fp16=1, int8=2 }

  # [Input Preprocessing]
  inputOrder: 0    # Reid network input order among { NCHW=0, NHWC=1 }. Batch will be converted to the specified order before reid input.
  colorFormat: 0    # Reid network input color format among {RGB=0, BGR=1 }. Batch will be converted to the specified color before reid input.
  offsets: [123.6750, 116.2800, 103.5300]    # Array of values to be subtracted from each input channel, with length equal to number of channels
  netScaleFactor: 0.01735207    # Scaling factor for reid network input after substracting offsets
  keepAspc: 1    # Whether to keep aspc ratio when resizing input objects for reid

  # [Output Postprocessing]
  addFeatureNormalization: 1       # If reid feature is not normalized in network, adding normalization on output so each reid feature has l2 norm equal to 1
  minVisibility4GalleryUpdate: 0.6 # Add ReID embedding to the gallery only if the visibility is not lower than this

  # [Paths and Names]
  onnxFile: "/opt/nvidia/deepstream/deepstream/deepstreamInterface/models/resnet50_market1501_aicity156.onnx"
  modelEngineFile: "/opt/nvidia/deepstream/deepstream/deepstreamInterface/models/resnet50_market1501_aicity156.onnx_b100_gpu0_fp16.engine" # Engine file path

As you can see, I’ve just modified the end of the file to use the re-identification model provided at the following address (onnx):

When I start my pipeline, the engine is well built and my pipeline runs smoothly.

However, if the engine file (previously built, so no need to build the engine again) is loaded by the pipeline, I get an error :

deepstream-1  | gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so
deepstream-1  | [NvMultiObjectTracker] Loading TRT Engine for tracker ReID...
deepstream-1  | [NvMultiObjectTracker] Loading Complete!
deepstream-1  | terminate called after throwing an instance of 'nvinfer1::APIUsageError'
deepstream-1  |   what():  Parameter check failed at: runtime/rt/runtime.cpp::~Runtime::346, condition: mEngineCounter.use_count() == 1. Destroying a runtime before destroying deserialized engines created by the runtime leads to undefined behavior.
deepstream-1  | 
deepstream-1 exited with code 139

This error doesn’t seem to prevent the pipeline from starting up for the sample:
/opt/nvidia/deepstream/deepstream-6.4/sources/apps/sample_apps/deepstream-test2
But this seems to be a problem when using python.

Thanks for your help

It seems that the above link cannot be opened. It seems to be an error of tensorrt.

By the way, what error does python sample report?

It is the exact same error in the c code sample.
Also, I was not able to use the python sample deepstream-test2 with a tracker using Reid.
Can you provide help on that side too? So we can replicate that bug.
Thanks

I also encounter the problem with the deepstream-nvdsanalytics Python sample.
Here’s how to reproduce:

  • Download the ReId model:
cd /opt/nvidia/deepstream/deepstream/sources/tracker_ReID # follow the README instructions in this directory:
mkdir ../../samples/models/Tracker/
wget 'https://api.ngc.nvidia.com/v2/models/nvidia/tao/reidentificationnet/versions/deployable_v1.0/files/resnet50_market1501.etlt' -P ../../samples/models/Tracker/
  • Copy config_tracker_NvDCF_accuracy.yml into the deepstream-nvdsanalytics sample directory :
cp /opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_tracker_NvDCF_accuracy.yml /opt/nvidia/deepstream/deepstream/sources/deepstream_python_apps/apps/deepstream-nvdsanalytics/
  • Edit ndsnvanalytics_tracker_config.txt on line 30:
    ll-config-file=config_tracker_NvDCF_accuracy.yml

  • Start the sample:

python3 deepstream_nvdsanalytics.py file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.h264
  • The sample is launched after the reidentification engine model is built.

  • Stop and restart the sample. The inference engine will no longer try to build the engine because it already exists.

  • Error :

gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so
[NvMultiObjectTracker] Loading TRT Engine for tracker ReID...
[NvMultiObjectTracker] Loading Complete!
Segmentation fault (core dumped)

Following your steps, I can run it normally, but I only have a T4 GPU. I will look for a 3060 to try.

What are your driver version and OS version? These usually have an impact

Thu Jan 25 16:41:59 2024       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.154.05             Driver Version: 535.154.05   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 3060 ...    Off | 00000000:01:00.0  On |                  N/A |
| N/A   56C    P5              13W /  60W |    129MiB /  6144MiB |     33%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A      4350      G   /usr/lib/xorg/Xorg                          125MiB |
+---------------------------------------------------------------------------------------+

I’am using Ubuntu 22.04.3 LTS

any news ?
Thanks

Are you running on host or docker? How do you start docker?

If you use docker, try the following command to start docker

docker run --gpus all -it --rm  --net=host --privileged -v /tmp/.X11-unix:/tmp/.X11-unix -e DISPLAY=$DISPLAY -w /opt/nvidia/deepstream/deepstream nvcr.io/nvidia/deepstream:6.4-triton-multiarch 

What is your kernel version? You can get it through uname -a. I think there is something issue with the installation

I am running with Docker

Same error when I am starting docker with your command.

Here is the kernel :

Linux trudemo-Inspiron-16-Plus-7620 6.5.0-15-generic #15~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Fri Jan 12 18:54:30 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

It may be that your kernel version is too high, causing incompatibility.

The default kernel version of 22.04.3 is 6.2. I currently use the 5.15.x version of the kernel. You can try this.

Have you tried to replicate the steps ?
Is it working on your side (can you run multiple time the sample) ?
What do you think @yuweiw

Yes, of cource. I tried following your steps, including multi-stream.works fine.

By the way, can you run test1 normally?

I tried on another computer with 5.15.x version of the kernel.

Have the same issue.

I can run the sample.
When I try to run it a second time, it is not working.

What problems did you encounter?

First execution of python3 deepstream_nvdsanalytics.py file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.h264 :

gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so
~~ CLOG[src/modules/ReID/ReID.cpp, loadTRTEngine() @line 598]: Engine file does not exist
[NvMultiObjectTracker] Load engine failed. Create engine again.
WARNING: [TRT]: onnx2trt_utils.cpp:374: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
WARNING: ../nvdsinfer/nvdsinfer_model_builder.cpp:1208 INT8 calibration file not specified. Trying FP16 mode.
WARNING: [TRT]: TensorRT encountered issues when converting weights between types and that could affect accuracy.
WARNING: [TRT]: If this is not the desired behavior, please modify the weights or retrain with regularization to adjust the magnitude of the weights.
WARNING: [TRT]: Check verbose logs for the list of affected weights.
WARNING: [TRT]: - 87 weights are affected by this issue: Detected subnormal FP16 values.
WARNING: [TRT]: - 77 weights are affected by this issue: Detected values less than smallest positive FP16 subnormal value and converted them to the FP16 minimum subnormalized value.
[NvMultiObjectTracker] Serialized plan file cached at location: /opt/nvidia/deepstream/deepstream-6.4/samples/models/Tracker/resnet50_market1501.etlt_b100_gpu0_fp16.engine
[NvMultiObjectTracker] Initialized
0:03:04.121176173   413 0x56429e313470 INFO                 nvinfer gstnvinfer.cpp:682:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:2092> [UID = 1]: deserialized trt engine from :/opt/nvidia/deepstream/deepstream-6.4/samples/models/Primary_Detector/resnet18_trafficcamnet.etlt_b1_gpu0_int8.engine
INFO: ../nvdsinfer/nvdsinfer_model_builder.cpp:610 [Implicit Engine Info]: layers num: 3
0   INPUT  kFLOAT input_1         3x544x960       
1   OUTPUT kFLOAT output_bbox/BiasAdd 16x34x60        
2   OUTPUT kFLOAT output_cov/Sigmoid 4x34x60         

0:03:04.401834940   413 0x56429e313470 INFO                 nvinfer gstnvinfer.cpp:682:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2195> [UID = 1]: Use deserialized engine model: /opt/nvidia/deepstream/deepstream-6.4/samples/models/Primary_Detector/resnet18_trafficcamnet.etlt_b1_gpu0_int8.engine
0:03:04.413691488   413 0x56429e313470 INFO                 nvinfer gstnvinfer_impl.cpp:328:notifyLoadModelStatus:<primary-inference> [UID 1]: Load new model:dsnvanalytics_pgie_config.txt sucessfully

Second execution of python3 deepstream_nvdsanalytics.py file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.h264 :

gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so
[NvMultiObjectTracker] Loading TRT Engine for tracker ReID...
[NvMultiObjectTracker] Loading Complete!
Aborted (core dumped)

I receive this on

  • 3060 kernel 6.5.0-15-generic Ubuntu 22.04.3 LTS
  • T4 kernel 5.15.0-91-generic Ubuntu 22.04.3 LTS

I mean what errors were encountered when running the docker command above.

I think there should be no problem with the above command.

If there is a problem, I suspect there is something wrong with docker.

I can successfully run this command :

docker run --gpus all -it --rm  --net=host --privileged -v /tmp/.X11-unix:/tmp/.X11-unix -e DISPLAY=$DISPLAY -w /opt/nvidia/deepstream/deepstream nvcr.io/nvidia/deepstream:6.4-triton-multiarch 

I get the error I just mentioned before inside that container.

@yuweiw

Can you send me your logs of your second execution of :

python3 deepstream_nvdsanalytics.py file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.h264