when I change my precision from init8 to fp16 || 32, tracker id is stable but bounding box jitters a lot, how do we fix it?
Could you share us a video for the issue you mentioned?
So we can check it with our internal team for some suggestion.
We want to reproduce this issue in our environment.
Would you mind to share your platform and configuration with us?
I am using
AGX, 24 * 1920 *1080(H.264/High Profile) @ 60 FPS, bitrate 47,195kbps. 1 detector and 3 classifiers
I have offloaded 3 classifiers to DLA 0 and 1
I have tried with H265 * 30FPS and I had a similar issue.
I am using KLT now
It would be best if you post your entire deepstream-app config file so we can reproduce. Are you inferencing at every frame?
@ i have shared privately[dm]
NvDcf does multi camera re-identification ?
If you still sees the issue when using KLT, pls try increasing
NvDCF and other trackers in DeepStream are all single-camera tracker.
KLT is stable but tracker id changes often this will be an issue in vehicle counting, how do we fix the jitter issue with nvdcf
Could you modify the configure file based on our source30_1080p_dec_infer-resnet_tiled_display_int8.txt
We want to reproduce this issue in our environment first.
I am using modified source30_1080p_dec_infer-resnet_tiled_display_int8.txt, I have changed from int 8 to FP16
source30_1080p_dec_infer-resnet_tiled_display_int8.txt default config with NVDCF works fine, but its single-source video and a single detector
@AastaLLL we replicated the issue on 8 * AGX kits
Thanks for your update.
We are investigating this issue internally.
Will update more information once we got any news.
We cannot reproduce this issue in our side.
Here are our configure for your reference:
# Copyright (c) 2018 NVIDIA Corporation. All rights reserved. # # NVIDIA Corporation and its licensors retain all intellectual property # and proprietary rights in and to this software, related documentation # and any modifications thereto. Any use, reproduction, disclosure or # distribution of this software and related documentation without an express # license agreement from NVIDIA Corporation is strictly prohibited. [application] enable-perf-measurement=1 perf-measurement-interval-sec=5 #gie-kitti-output-dir=streamscl [tiled-display] enable=1 rows=1 columns=1 width=1280 height=720 gpu-id=0 #(0): nvbuf-mem-default - Default memory allocated, specific to particular platform #(1): nvbuf-mem-cuda-pinned - Allocate Pinned/Host cuda memory, applicable for Tesla #(2): nvbuf-mem-cuda-device - Allocate Device cuda memory, applicable for Tesla #(3): nvbuf-mem-cuda-unified - Allocate Unified cuda memory, applicable for Tesla #(4): nvbuf-mem-surface-array - Allocate Surface Array memory, applicable for Jetson nvbuf-memory-type=0 [source0] enable=1 #Type - 1=CameraV4L2 2=URI 3=MultiURI 4=RTSP type=2 uri=file://../../streams/sample_1080p_h264.mp4 #drop-frame-interval=2 gpu-id=0 # (0): memtype_device - Memory type Device # (1): memtype_pinned - Memory type Host Pinned # (2): memtype_unified - Memory type Unified cudadec-memtype=0 [sink0] enable=1 #Type - 1=FakeSink 2=EglSink 3=File type=2 sync=1 source-id=0 gpu-id=0 nvbuf-memory-type=0 [osd] enable=1 gpu-id=0 border-width=1 text-size=15 text-color=1;1;1;1; text-bg-color=0.3;0.3;0.3;1 font=Serif show-clock=0 clock-x-offset=800 clock-y-offset=820 clock-text-size=12 clock-color=1;0;0;0 nvbuf-memory-type=0 [streammux] gpu-id=0 ##Boolean property to inform muxer that sources are live live-source=0 batch-size=1 ##time out in usec, to wait after the first buffer is available ##to push the batch even if the complete batch is not formed batched-push-timeout=40000 ## Set muxer output width and height width=1920 height=1080 ##Enable to maintain aspect ratio wrt source, and allow black borders, works ##along with width, height properties enable-padding=0 nvbuf-memory-type=0 # config-file property is mandatory for any gie section. # Other properties are optional and if set will override the properties set in # the infer config file. [primary-gie] enable=1 gpu-id=0 model-engine-file=../../models/Primary_Detector/resnet10.caffemodel_b30_int8.engine #Required to display the PGIE labels, should be added even when using config-file #property batch-size=1 #Required by the app for OSD, not a plugin property bbox-border-color0=1;0;0;1 bbox-border-color1=0;1;1;1 bbox-border-color2=0;0;1;1 bbox-border-color3=0;1;0;1 interval=0 #Required by the app for SGIE, when used along with config-file property gie-unique-id=1 nvbuf-memory-type=0 config-file=config_infer_primary.txt [tracker] enable=1 tracker-width=480 tracker-height=272 #ll-lib-file=/opt/nvidia/deepstream/deepstream-4.0/lib/libnvds_mot_iou.so ll-lib-file=/opt/nvidia/deepstream/deepstream-4.0/lib/libnvds_nvdcf.so #ll-config-file required for IOU only #ll-config-file=iou_config.txt gpu-id=0 [tests] file-loop=0
# Copyright (c) 2018 NVIDIA Corporation. All rights reserved. # # NVIDIA Corporation and its licensors retain all intellectual property # and proprietary rights in and to this software, related documentation # and any modifications thereto. Any use, reproduction, disclosure or # distribution of this software and related documentation without an express # license agreement from NVIDIA Corporation is strictly prohibited. # Following properties are mandatory when engine files are not specified: # int8-calib-file(Only in INT8) # Caffemodel mandatory properties: model-file, proto-file, output-blob-names # UFF: uff-file, input-dims, uff-input-blob-name, output-blob-names # ONNX: onnx-file # # Mandatory properties for detectors: # num-detected-classes # # Optional properties for detectors: # enable-dbscan(Default=false), interval(Primary mode only, Default=0) # custom-lib-path, # parse-bbox-func-name # # Mandatory properties for classifiers: # classifier-threshold, is-classifier # # Optional properties for classifiers: # classifier-async-mode(Secondary mode only, Default=false) # # Optional properties in secondary mode: # operate-on-gie-id(Default=0), operate-on-class-ids(Defaults to all classes), # input-object-min-width, input-object-min-height, input-object-max-width, # input-object-max-height # # Following properties are always recommended: # batch-size(Default=1) # # Other optional properties: # net-scale-factor(Default=1), network-mode(Default=0 i.e FP32), # model-color-format(Default=0 i.e. RGB) model-engine-file, labelfile-path, # mean-file, gie-unique-id(Default=0), offsets, gie-mode (Default=1 i.e. primary), # custom-lib-path, network-mode(Default=0 i.e FP32) # # The values in the config file are overridden by values set through GObject # properties. [property] gpu-id=0 net-scale-factor=0.0039215697906911373 model-file=../../models/Primary_Detector/resnet10.caffemodel proto-file=../../models/Primary_Detector/resnet10.prototxt model-engine-file=../../models/Primary_Detector/resnet10.caffemodel_b30_int8.engine labelfile-path=../../models/Primary_Detector/labels.txt int8-calib-file=../../models/Primary_Detector/cal_trt.bin batch-size=30 process-mode=1 model-color-format=0 ## 0=FP32, 1=INT8, 2=FP16 mode network-mode=2 num-detected-classes=4 interval=0 gie-unique-id=1 output-blob-names=conv2d_bbox;conv2d_cov/Sigmoid #parse-bbox-func-name=NvDsInferParseCustomResnet #custom-lib-path=/path/to/libnvdsparsebbox.so #enable-dbscan=1 [class-attrs-all] threshold=0.2 group-threshold=1 ## Set eps=0.7 and minBoxes for enable-dbscan=1 eps=0.2 #minBoxes=3 roi-top-offset=0 roi-bottom-offset=0 detected-min-w=0 detected-min-h=0 detected-max-w=0 detected-max-h=0 ## Per class configuration #[class-attrs-2] #threshold=0.6 #eps=0.5 #group-threshold=3 #roi-top-offset=20 #roi-bottom-offset=10 #detected-min-w=40 #detected-min-h=40 #detected-max-w=400 #detected-max-h=800
Is there any missing in our test?
Default demo runs fine with a single video
source30_1080p_dec_infer-resnet_tiled_display_int8.txt with singe source(video) works fine.
*source30_1080p_dec_infer-resnet_tiled_display_int8.txt has only 1 detector
Problem is only when we add multiple sources with different videos
I can give you VPN access to our lab
Can you try with
- Multiple video sources with different videos.
- Add secondary classifiers.
[primary-gie] enable=1 gpu-id=0 model-engine-file=/opt/nvidia/deepstream/deepstream-4.0/samples/models/Primary_Detector/resnet10.caffemodel_b16_fp16.engine batch-size=16 #Required by the app for OSD, not a plugin property bbox-border-color0=1;0;0;1 bbox-border-color1=0;1;1;1 bbox-border-color2=0;0;1;1 bbox-border-color3=0;1;0;1 interval=4 gie-unique-id=1 nvbuf-memory-type=0 config-file=config_infer_primary.txt
[secondary-gie0] enable=1 model-engine-file=/opt/nvidia/deepstream/deepstream-4.0/samples/models/Secondary_VehicleTypes/resnet18.caffemodel_b16_int8.engine labelfile-path=/opt/nvidia/deepstream/deepstream-4.0/samples/models/Secondary_VehicleTypes/labels.txt gpu-id=0 batch-size=16 gie-unique-id=4 operate-on-gie-id=1 operate-on-class-ids=0; config-file=config_infer_secondary_vehicletypes.txt [secondary-gie1] enable=1 model-engine-file=/opt/nvidia/deepstream/deepstream-4.0/samples/models/Secondary_CarColor/resnet18.caffemodel_b16_int8.engine labelfile-path=/opt/nvidia/deepstream/deepstream-4.0/samples/models/Secondary_CarColor/labels.txt batch-size=16 gpu-id=0 gie-unique-id=5 operate-on-gie-id=1 operate-on-class-ids=0; config-file=config_infer_secondary_carcolor.txt [secondary-gie2] enable=1 model-engine-file=/opt/nvidia/deepstream/deepstream-4.0/samples/models/Secondary_CarMake/resnet18.caffemodel_b16_int8.engine labelfile-path=/opt/nvidia/deepstream/deepstream-4.0/samples/models/Secondary_CarMake/labels.txt batch-size=16 gpu-id=0 gie-unique-id=6 operate-on-gie-id=1 operate-on-class-ids=0; config-file=config_infer_secondary_carmake.txt
I noticed that the PGIE interval is set to 4:
interval=4 Can you reduce this to 2 or 1?
We found a bug in the release version of NvDCF, and it may cause such jitter in case of interval>0. We fixed it and the next release version will be much smoother.
Can you let me know what does interval mean in this context, does it mean the object detector running for 4 frames or of that sort?
Could you also specify where the documentation for config files are present?
interval=0 is stable.
Hey @pshin, can we get any info on the timeline for next release?
Please refer to
interval at PGIE section.
I cannot disclose any future plans, sorry. Please refer to a public announcement related to DeepStream. But, I guess we are trying to make the releases in a regular basis.
Hello @pshin, 6 months later I still experience the jitter issue described above with the nvdcf tracker that came with DeepStream 5.0DP if I set the interval parameter of the tracker to anything greater than zero. Do you have some update on the fix? Maybe we can patch the tracker somehow?
I created a small log that shows how many objects are tracked reported on the metadata stream for each from, running on the sample_720p.h264 example stream. I set
interval=3 in the tracker config so the tracker should run only on each 4th frame. It is clearly visible that the number of objects reported is fluctuating: every 4th frame (I guess those ones when the tracker run) reports significantly more objects.
Frame Number = 496, FPS = 11.13, objects = 15 Frame Number = 497, FPS = 11.10, objects = 8 Frame Number = 498, FPS = 11.12, objects = 8 Frame Number = 499, FPS = 11.13, objects = 9 Frame Number = 500, FPS = 11.13, objects = 15 Frame Number = 501, FPS = 11.10, objects = 7 Frame Number = 502, FPS = 11.12, objects = 8 Frame Number = 503, FPS = 11.14, objects = 6 Frame Number = 504, FPS = 11.13, objects = 14 Frame Number = 505, FPS = 11.10, objects = 6 Frame Number = 506, FPS = 11.12, objects = 8 Frame Number = 507, FPS = 11.13, objects = 8 Frame Number = 508, FPS = 11.13, objects = 14 Frame Number = 509, FPS = 11.10, objects = 6 Frame Number = 510, FPS = 11.12, objects = 8 Frame Number = 511, FPS = 11.13, objects = 8 Frame Number = 512, FPS = 11.13, objects = 13 Frame Number = 513, FPS = 11.10, objects = 7 Frame Number = 514, FPS = 11.12, objects = 6 Frame Number = 515, FPS = 11.13, objects = 7 Frame Number = 516, FPS = 11.13, objects = 12 Frame Number = 517, FPS = 11.10, objects = 5 Frame Number = 518, FPS = 11.12, objects = 5 Frame Number = 519, FPS = 11.13, objects = 6