Minor frame "vibration" in output video when using NvDCF tracker and nvinfer interval>0

• Hardware Platform (Jetson / GPU) GPU
• DeepStream Version 6.1.1
Docker container nvcr.io/nvidia/deepstream:6.1.1-samples
• JetPack Version (valid for Jetson only)
• TensorRT Version 8.4.1.5
• NVIDIA GPU Driver Version (valid for GPU only) 515.65.01
• Issue Type( questions, new requirements, bugs) Maybe bug
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)

I’m getting output videos with unexpected minor “vibration” (apologies for not able to explain it precisely, but should be easy to notice if you watch and compare the attached video files), only when using NvDCF tracker and setting nvinfer interval > 0.

Model: PeopleNet deployable_quantized_v2.6
Config:

config_infer_primary_peoplenet.txt
[property]
gpu-id=0
net-scale-factor=0.0039215697906911373
tlt-model-key=tlt_encode
tlt-encoded-model=/host/test/peoplenet/resnet34_peoplenet_int8.etlt
labelfile-path=/host/test/peoplenet/labels_peoplenet.txt
model-engine-file=/host/test/peoplenet/resnet34_peoplenet_int8.etlt_b1_gpu0_int8.engine
int8-calib-file=/host/test/peoplenet/resnet34_peoplenet_int8.txt
input-dims=3;544;960;0
uff-input-blob-name=input_1
batch-size=1
process-mode=1
model-color-format=0
## 0=FP32, 1=INT8, 2=FP16 mode
network-mode=1
num-detected-classes=3
cluster-mode=2
interval=0
gie-unique-id=1
output-blob-names=output_bbox/BiasAdd;output_cov/Sigmoid

#Use the config params below for dbscan clustering mode
#[class-attrs-all]
#detected-min-w=4
#detected-min-h=4
#minBoxes=3
#eps=0.7

#Use the config params below for NMS clustering mode
[class-attrs-all]
topk=20
nms-iou-threshold=0.5
pre-cluster-threshold=0.2

## Per class configurations
[class-attrs-0]
topk=20
nms-iou-threshold=0.5
pre-cluster-threshold=0.4

#[class-attrs-1]
#pre-cluster-threshold=0.05
#eps=0.7
#dbscan-min-score=0.5

Input video:

  • NvDCF tracker + nvinfer interval=1 = vibration
gst-launch-1.0 \
    uridecodebin uri="file:///host/test/record.mp4" ! m.sink_0 \
    nvstreammux name=m batch-size=1 width=1920 height=1080 \
    ! nvinfer \
        config-file-path=/host/test/peoplenet/config_infer_primary_peoplenet.txt \
        interval=1 \
    ! nvtracker \
        ll-lib-file=/opt/nvidia/deepstream/deepstream-6.1/lib/libnvds_nvmultiobjecttracker.so \
        ll-config-file=/opt/nvidia/deepstream/deepstream-6.1/samples/configs/deepstream-app/config_tracker_NvDCF_max_perf.yml \
    ! nvvideoconvert \
    ! nvdsosd \
    ! nvvideoconvert \
    ! nvv4l2h264enc bitrate=4000000 \
    ! h264parse \
    ! mp4mux \
    ! filesink location="/host/test/nvdcf_interval1_vibration.mp4"

And all of the below pipelines produce normal videos with no vibration.

NvDCF tracker + nvinfer interval=0
gst-launch-1.0 \
    uridecodebin uri="file:///host/test/record.mp4" ! m.sink_0 \
    nvstreammux name=m batch-size=1 width=1920 height=1080 \
    ! nvinfer \
        config-file-path=/host/test/peoplenet/config_infer_primary_peoplenet.txt \
        interval=1 \
    ! nvtracker \
        ll-lib-file=/opt/nvidia/deepstream/deepstream-6.1/lib/libnvds_nvmultiobjecttracker.so \
        ll-config-file=/opt/nvidia/deepstream/deepstream-6.1/samples/configs/deepstream-app/config_tracker_NvDCF_max_perf.yml \
    ! nvvideoconvert \
    ! nvdsosd \
    ! nvvideoconvert \
    ! nvv4l2h264enc bitrate=4000000 \
    ! h264parse \
    ! mp4mux \
    ! filesink location="/host/test/nvdcf_interval0_novibration.mp4"
IOU tracker + nvinfer interval=0
gst-launch-1.0 \
    uridecodebin uri="file:///host/test/record.mp4" ! m.sink_0 \
    nvstreammux name=m batch-size=1 width=1920 height=1080 \
    ! nvinfer \
        config-file-path=/host/test/peoplenet/config_infer_primary_peoplenet.txt \
        interval=0 \
    ! nvtracker \
        ll-lib-file=/opt/nvidia/deepstream/deepstream-6.1/lib/libnvds_nvmultiobjecttracker.so \
        ll-config-file=/opt/nvidia/deepstream/deepstream-6.1/samples/configs/deepstream-app/config_tracker_IOU.yml \
    ! nvvideoconvert \
    ! nvdsosd \
    ! nvvideoconvert \
    ! nvv4l2h264enc bitrate=4000000 \
    ! h264parse \
    ! mp4mux \
    ! filesink location="/host/test/iou_interval0_novibration.mp4"
IOU tracker + nvinfer interval=1
gst-launch-1.0 \
    uridecodebin uri="file:///host/test/record.mp4" ! m.sink_0 \
    nvstreammux name=m batch-size=1 width=1920 height=1080 \
    ! nvinfer \
        config-file-path=/host/test/peoplenet/config_infer_primary_peoplenet.txt \
        interval=1 \
    ! nvtracker \
        ll-lib-file=/opt/nvidia/deepstream/deepstream-6.1/lib/libnvds_nvmultiobjecttracker.so \
        ll-config-file=/opt/nvidia/deepstream/deepstream-6.1/samples/configs/deepstream-app/config_tracker_IOU.yml \
    ! nvvideoconvert \
    ! nvdsosd \
    ! nvvideoconvert \
    ! nvv4l2h264enc bitrate=4000000 \
    ! h264parse \
    ! mp4mux \
    ! filesink location="/host/test/iou_interval1_novibration.mp4"
No tracker + nvinfer interval=0
gst-launch-1.0 \
    uridecodebin uri="file:///host/test/record.mp4" ! m.sink_0 \
    nvstreammux name=m batch-size=1 width=1920 height=1080 \
    ! nvinfer \
        config-file-path=/host/test/peoplenet/config_infer_primary_peoplenet.txt \
        interval=0 \
    ! nvvideoconvert \
    ! nvdsosd \
    ! nvvideoconvert \
    ! nvv4l2h264enc bitrate=4000000 \
    ! h264parse \
    ! mp4mux \
    ! filesink location="/host/test/notracker_interval0_novibration.mp4"
No tracker + nvinfer interval=1
gst-launch-1.0 \
    uridecodebin uri="file:///host/test/record.mp4" ! m.sink_0 \
    nvstreammux name=m batch-size=1 width=1920 height=1080 \
    ! nvinfer \
        config-file-path=/host/test/peoplenet/config_infer_primary_peoplenet.txt \
        interval=1 \
    ! nvvideoconvert \
    ! nvdsosd \
    ! nvvideoconvert \
    ! nvv4l2h264enc bitrate=4000000 \
    ! h264parse \
    ! mp4mux \
    ! filesink location="/host/test/notracker_interval1_novibration.mp4"

Suppose nvtracker don’t modify the output video. It is strange. Can you double check? Or run your command line in other setup, such as Jetson device.

I have tested the NvDCF tracker + interval=1 pipeline on multiple GPU machines:

  • Quadro P2000 515.65.01
  • GTX1060 515.48.07
  • GTX1070Mobile 515.76
  • Quadrio RTX4000 515.65.01
  • Tesla T4 470.103.01
  • RTX3060LHR 515.48.07
    All of the output videos have the weird vibration issue.
    I don’t have any Jetson device to test on, though.

Can you have a try with: nvstreammux name=m batch-size=1 width=1920 height=1088?

There is no vibration with nvstreammux name=m batch-size=1 width=1920 height=1088. I also tried different nvstreammux resolution settings (640x360, 640x384, 960x540, 960x544, 1280x720, 1280x736, etc), it seems like the vibration issue only happens when resolution is 1920x1080.

Can the workaround unblock you?

Yes, the workaround is great. But I am still curious about what is causing the vibration in this corner case, is it somehow related to nvtracker’s requirement of “frame width and height must be multiple of 32”?

Glad to know the workaround works for you. We will continue to check the root cause of the issue. Thanks.