Delay, randomness and dropped frames in RTSP output Stream

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) dGPU
• DeepStream Version 5.0 Latest NGC container devel
• JetPack Version (valid for Jetson only)
• TensorRT Version
• NVIDIA GPU Driver Version (valid for GPU only) 440.118.02
• Issue Type( questions, new requirements, bugs) Bug
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

I am running 30 RTSP Input streams with 30 output RTSP Streams on P4 Card. Have also enabled img-save, NVDSAnalytics, Tracker and smart record generation.
I have customized the NVDS-Analytics deepstream-app to include the above.
Here are the few observations:

  1. Top utilization gives 294% (its a 20 core server - 2 CPU 10 core each) so this seems ok.
  2. nvidia-smi shows 80-90% GPU utilization with 92% memory utilization. This is stable and does not rise much. Tracker used IOU.
  3. NvDCF leads to out of memory errors as it takes around 1GB memory.
  4. KLT fails to initialize sometimes.

My major problem is that Delay, randomness and dropped frames in RTSP output Stream.
Here is the config.

Pipeline Config

[application]
enable-perf-measurement=1
perf-measurement-interval-sec=5
#gie-kitti-output-dir=streamscl

[source0]
enable=1
type=4
uri=rtsp://admin:admin@192.168.203.197
select-rtp-protocol=4
rtsp-reconnect-interval-sec=60
num-sources=1
camera-id=53
drop-frame-interval=0
gpu-id=0
cudadec-memtype=0
smart-record=1
smart-rec-duration=5
smart-rec-video-cache=5
smart-rec-start-time=0
smart-rec-default-duration=5
smart-rec-container=0
smart-rec-dir-path=./logs
smart-rec-file-prefix=camera100

[source1]
enable=1
type=4
uri=rtsp://admin:admin@192.168.203.199
select-rtp-protocol=4
rtsp-reconnect-interval-sec=60
num-sources=1
camera-id=51
drop-frame-interval=0
gpu-id=0
cudadec-memtype=0
smart-record=1
smart-rec-duration=5
smart-rec-video-cache=5
smart-rec-start-time=0
smart-rec-default-duration=5
smart-rec-container=0
smart-rec-dir-path=./logs
smart-rec-file-prefix=camera101

[source2]
enable=1
type=4
uri=rtsp://admin:admin@192.168.203.131
select-rtp-protocol=4
rtsp-reconnect-interval-sec=60
num-sources=1
camera-id=54
drop-frame-interval=0
gpu-id=0
cudadec-memtype=0
smart-record=1
smart-rec-duration=5
smart-rec-video-cache=5
smart-rec-start-time=0
smart-rec-default-duration=5
smart-rec-container=0
smart-rec-dir-path=./logs
smart-rec-file-prefix=camera102

[source3]
enable=1
type=4
uri=rtsp://admin:admin@192.168.203.67
select-rtp-protocol=4
rtsp-reconnect-interval-sec=60
num-sources=1
camera-id=55
drop-frame-interval=0
gpu-id=0
cudadec-memtype=0
smart-record=1
smart-rec-duration=5
smart-rec-video-cache=5
smart-rec-start-time=0
smart-rec-default-duration=5
smart-rec-container=0
smart-rec-dir-path=./logs
smart-rec-file-prefix=camera103

[source4]
enable=1
type=4
uri=rtsp://admin:admin@192.168.202.67
select-rtp-protocol=4
rtsp-reconnect-interval-sec=60
num-sources=1
camera-id=56
drop-frame-interval=0
gpu-id=0
cudadec-memtype=0
smart-record=1
smart-rec-duration=5
smart-rec-video-cache=5
smart-rec-start-time=0
smart-rec-default-duration=5
smart-rec-container=0
smart-rec-dir-path=./logs
smart-rec-file-prefix=camera104

[source5]
enable=1
type=4
uri=rtsp://admin:admin@192.168.201.195
select-rtp-protocol=4
rtsp-reconnect-interval-sec=60
num-sources=1
camera-id=57
drop-frame-interval=0
gpu-id=0
cudadec-memtype=0
smart-record=1
smart-rec-duration=5
smart-rec-video-cache=5
smart-rec-start-time=0
smart-rec-default-duration=5
smart-rec-container=0
smart-rec-dir-path=./logs
smart-rec-file-prefix=camera105

[source6]
enable=1
type=4
uri=rtsp://admin:admin@192.168.203.3
select-rtp-protocol=4
rtsp-reconnect-interval-sec=60
num-sources=1
camera-id=58
drop-frame-interval=0
gpu-id=0
cudadec-memtype=0
smart-record=1
smart-rec-duration=5
smart-rec-video-cache=5
smart-rec-start-time=0
smart-rec-default-duration=5
smart-rec-container=0
smart-rec-dir-path=./logs
smart-rec-file-prefix=camera106

[source7]
enable=1
type=4
uri=rtsp://admin:admin@192.168.204.67
select-rtp-protocol=4
rtsp-reconnect-interval-sec=60
num-sources=1
camera-id=60
drop-frame-interval=0
gpu-id=0
cudadec-memtype=0
smart-record=1
smart-rec-duration=5
smart-rec-video-cache=5
smart-rec-start-time=0
smart-rec-default-duration=5
smart-rec-container=0
smart-rec-dir-path=./logs
smart-rec-file-prefix=camera107

[source8]
enable=1
type=4
uri=rtsp://admin:admin@192.168.205.134
select-rtp-protocol=4
rtsp-reconnect-interval-sec=60
num-sources=1
camera-id=61
drop-frame-interval=0
gpu-id=0
cudadec-memtype=0
smart-record=1
smart-rec-duration=5
smart-rec-video-cache=5
smart-rec-start-time=0
smart-rec-default-duration=5
smart-rec-container=0
smart-rec-dir-path=./logs
smart-rec-file-prefix=camera108

[source9]
enable=1
type=4
uri=rtsp://admin:admin@192.168.205.197
select-rtp-protocol=4
rtsp-reconnect-interval-sec=60
num-sources=1
camera-id=62
drop-frame-interval=0
gpu-id=0
cudadec-memtype=0
smart-record=1
smart-rec-duration=5
smart-rec-video-cache=5
smart-rec-start-time=0
smart-rec-default-duration=5
smart-rec-container=0
smart-rec-dir-path=./logs
smart-rec-file-prefix=camera109

[source10]
enable=1
type=4
uri=rtsp://admin:admin@192.168.205.133
select-rtp-protocol=4
rtsp-reconnect-interval-sec=60
num-sources=1
camera-id=63
drop-frame-interval=0
gpu-id=0
cudadec-memtype=0
smart-record=1
smart-rec-duration=5
smart-rec-video-cache=5
smart-rec-start-time=0
smart-rec-default-duration=5
smart-rec-container=0
smart-rec-dir-path=./logs
smart-rec-file-prefix=camera110

[source11]
enable=1
type=4
uri=rtsp://admin:admin@192.168.205.132
select-rtp-protocol=4
rtsp-reconnect-interval-sec=60
num-sources=1
camera-id=65
drop-frame-interval=0
gpu-id=0
cudadec-memtype=0
smart-record=1
smart-rec-duration=5
smart-rec-video-cache=5
smart-rec-start-time=0
smart-rec-default-duration=5
smart-rec-container=0
smart-rec-dir-path=./logs
smart-rec-file-prefix=camera111

[source12]
enable=1
type=4
uri=rtsp://admin:admin@192.168.205.198
select-rtp-protocol=4
rtsp-reconnect-interval-sec=60
num-sources=1
camera-id=64
drop-frame-interval=0
gpu-id=0
cudadec-memtype=0
smart-record=1
smart-rec-duration=5
smart-rec-video-cache=5
smart-rec-start-time=0
smart-rec-default-duration=5
smart-rec-container=0
smart-rec-dir-path=./logs
smart-rec-file-prefix=camera112

[source13]
enable=1
type=4
uri=rtsp://admin:admin@192.168.205.131
select-rtp-protocol=4
rtsp-reconnect-interval-sec=60
num-sources=1
camera-id=66
drop-frame-interval=0
gpu-id=0
cudadec-memtype=0
smart-record=1
smart-rec-duration=5
smart-rec-video-cache=5
smart-rec-start-time=0
smart-rec-default-duration=5
smart-rec-container=0
smart-rec-dir-path=./logs
smart-rec-file-prefix=camera113

[source14]
enable=1
type=4
uri=rtsp://admin:admin@192.168.194.15
select-rtp-protocol=4
rtsp-reconnect-interval-sec=60
num-sources=1
camera-id=68
drop-frame-interval=0
gpu-id=0
cudadec-memtype=0
smart-record=1
smart-rec-duration=5
smart-rec-video-cache=5
smart-rec-start-time=0
smart-rec-default-duration=5
smart-rec-container=0
smart-rec-dir-path=./logs
smart-rec-file-prefix=camera114

[source15]
enable=1
type=4
uri=rtsp://admin:admin@192.168.194.4
select-rtp-protocol=4
rtsp-reconnect-interval-sec=60
num-sources=1
camera-id=67
drop-frame-interval=0
gpu-id=0
cudadec-memtype=0
smart-record=1
smart-rec-duration=5
smart-rec-video-cache=5
smart-rec-start-time=0
smart-rec-default-duration=5
smart-rec-container=0
smart-rec-dir-path=./logs
smart-rec-file-prefix=camera115

[source16]
enable=1
type=4
uri=rtsp://admin:admin@192.168.205.70
select-rtp-protocol=4
rtsp-reconnect-interval-sec=60
num-sources=1
camera-id=69
drop-frame-interval=0
gpu-id=0
cudadec-memtype=0
smart-record=1
smart-rec-duration=5
smart-rec-video-cache=5
smart-rec-start-time=0
smart-rec-default-duration=5
smart-rec-container=0
smart-rec-dir-path=./logs
smart-rec-file-prefix=camera116

[source17]
enable=1
type=4
uri=rtsp://admin:admin@192.168.205.69
select-rtp-protocol=4
rtsp-reconnect-interval-sec=60
num-sources=1
camera-id=70
drop-frame-interval=0
gpu-id=0
cudadec-memtype=0
smart-record=1
smart-rec-duration=5
smart-rec-video-cache=5
smart-rec-start-time=0
smart-rec-default-duration=5
smart-rec-container=0
smart-rec-dir-path=./logs
smart-rec-file-prefix=camera117

[source18]
enable=1
type=4
uri=rtsp://admin:admin@192.168.205.68
select-rtp-protocol=4
rtsp-reconnect-interval-sec=60
num-sources=1
camera-id=72
drop-frame-interval=0
gpu-id=0
cudadec-memtype=0
smart-record=1
smart-rec-duration=5
smart-rec-video-cache=5
smart-rec-start-time=0
smart-rec-default-duration=5
smart-rec-container=0
smart-rec-dir-path=./logs
smart-rec-file-prefix=camera118

[source19]
enable=1
type=4
uri=rtsp://admin:admin@192.168.194.22
select-rtp-protocol=4
rtsp-reconnect-interval-sec=60
num-sources=1
camera-id=71
drop-frame-interval=0
gpu-id=0
cudadec-memtype=0
smart-record=1
smart-rec-duration=5
smart-rec-video-cache=5
smart-rec-start-time=0
smart-rec-default-duration=5
smart-rec-container=0
smart-rec-dir-path=./logs
smart-rec-file-prefix=camera119

[source20]
enable=1
type=4
uri=rtsp://admin:admin@192.168.207.67
select-rtp-protocol=4
rtsp-reconnect-interval-sec=60
num-sources=1
camera-id=73
drop-frame-interval=0
gpu-id=0
cudadec-memtype=0
smart-record=1
smart-rec-duration=5
smart-rec-video-cache=5
smart-rec-start-time=0
smart-rec-default-duration=5
smart-rec-container=0
smart-rec-dir-path=./logs
smart-rec-file-prefix=camera120

[source21]
enable=1
type=4
uri=rtsp://admin:admin@192.168.207.68
select-rtp-protocol=4
rtsp-reconnect-interval-sec=60
num-sources=1
camera-id=75
drop-frame-interval=0
gpu-id=0
cudadec-memtype=0
smart-record=1
smart-rec-duration=5
smart-rec-video-cache=5
smart-rec-start-time=0
smart-rec-default-duration=5
smart-rec-container=0
smart-rec-dir-path=./logs
smart-rec-file-prefix=camera121

[source22]
enable=1
type=4
uri=rtsp://admin:admin@192.168.202.196
select-rtp-protocol=4
rtsp-reconnect-interval-sec=60
num-sources=1
camera-id=76
drop-frame-interval=0
gpu-id=0
cudadec-memtype=0
smart-record=1
smart-rec-duration=5
smart-rec-video-cache=5
smart-rec-start-time=0
smart-rec-default-duration=5
smart-rec-container=0
smart-rec-dir-path=./logs
smart-rec-file-prefix=camera122

[source23]
enable=1
type=4
uri=rtsp://admin:admin@192.168.207.69
select-rtp-protocol=4
rtsp-reconnect-interval-sec=60
num-sources=1
camera-id=78
drop-frame-interval=0
gpu-id=0
cudadec-memtype=0
smart-record=1
smart-rec-duration=5
smart-rec-video-cache=5
smart-rec-start-time=0
smart-rec-default-duration=5
smart-rec-container=0
smart-rec-dir-path=./logs
smart-rec-file-prefix=camera123

[source24]
enable=1
type=4
uri=rtsp://admin:admin@192.168.202.195
select-rtp-protocol=4
rtsp-reconnect-interval-sec=60
num-sources=1
camera-id=77
drop-frame-interval=0
gpu-id=0
cudadec-memtype=0
smart-record=1
smart-rec-duration=5
smart-rec-video-cache=5
smart-rec-start-time=0
smart-rec-default-duration=5
smart-rec-container=0
smart-rec-dir-path=./logs
smart-rec-file-prefix=camera124

[source25]
enable=1
type=4
uri=rtsp://admin:admin@192.168.202.4
select-rtp-protocol=4
rtsp-reconnect-interval-sec=60
num-sources=1
camera-id=80
drop-frame-interval=0
gpu-id=0
cudadec-memtype=0
smart-record=1
smart-rec-duration=5
smart-rec-video-cache=5
smart-rec-start-time=0
smart-rec-default-duration=5
smart-rec-container=0
smart-rec-dir-path=./logs
smart-rec-file-prefix=camera125

[source26]
enable=1
type=4
uri=rtsp://admin:admin@192.168.207.70
select-rtp-protocol=4
rtsp-reconnect-interval-sec=60
num-sources=1
camera-id=79
drop-frame-interval=0
gpu-id=0
cudadec-memtype=0
smart-record=1
smart-rec-duration=5
smart-rec-video-cache=5
smart-rec-start-time=0
smart-rec-default-duration=5
smart-rec-container=0
smart-rec-dir-path=./logs
smart-rec-file-prefix=camera126

[source27]
enable=1
type=4
uri=rtsp://admin:admin@192.168.202.3
select-rtp-protocol=4
rtsp-reconnect-interval-sec=60
num-sources=1
camera-id=81
drop-frame-interval=0
gpu-id=0
cudadec-memtype=0
smart-record=1
smart-rec-duration=5
smart-rec-video-cache=5
smart-rec-start-time=0
smart-rec-default-duration=5
smart-rec-container=0
smart-rec-dir-path=./logs
smart-rec-file-prefix=camera127

[source28]
enable=1
type=4
uri=rtsp://admin:admin@192.168.205.196
select-rtp-protocol=4
rtsp-reconnect-interval-sec=60
num-sources=1
camera-id=59
drop-frame-interval=0
gpu-id=0
cudadec-memtype=0
smart-record=1
smart-rec-duration=5
smart-rec-video-cache=5
smart-rec-start-time=0
smart-rec-default-duration=5
smart-rec-container=0
smart-rec-dir-path=./logs
smart-rec-file-prefix=camera128

[source29]
enable=1
type=4
uri=rtsp://admin:admin@192.168.205.67
select-rtp-protocol=4
rtsp-reconnect-interval-sec=60
num-sources=1
camera-id=74
drop-frame-interval=0
gpu-id=0
cudadec-memtype=0
smart-record=1
smart-rec-duration=5
smart-rec-video-cache=5
smart-rec-start-time=0
smart-rec-default-duration=5
smart-rec-container=0
smart-rec-dir-path=./logs
smart-rec-file-prefix=camera129

[osd]
enable=1
gpu-id=0
border-width=1
text-size=15
text-color=1;1;1;1;
text-bg-color=0.3;0.3;0.3;1
font=Serif
show-clock=0
clock-x-offset=800
clock-y-offset=820
clock-text-size=12
clock-color=1;0;0;0
nvbuf-memory-type=0

[streammux]
gpu-id=0
live-source=1
batch-size=30
batched-push-timeout=60000
width=1920
height=1080
enable-padding=0
nvbuf-memory-type=0

[primary-gie]
enable=1
gpu-id=0
batch-size=30
bbox-border-color0=1;0;0;1
bbox-border-color1=0;1;1;1
bbox-border-color2=0;0;1;1
bbox-border-color3=0;1;0;1
interval=4
gie-unique-id=1
nvbuf-memory-type=0
config-file=pgie-config.txt

[tests]
file-loop=0

[nvds-analytics]
enable=1
config-file=roi-config.txt

[tracker]
enable=1

For the case of NvDCF tracker, tracker-width and tracker-height must be a multiple of 32, respectively

tracker-width=640
tracker-height=384
ll-lib-file=/opt/nvidia/deepstream/deepstream-5.0/lib/libnvds_mot_iou.so
#ll-lib-file=/opt/nvidia/deepstream/deepstream-5.0/lib/libnvds_nvdcf.so
#ll-lib-file=/opt/nvidia/deepstream/deepstream-5.0/lib/libnvds_mot_klt.so
#ll-config-file required for DCF/IOU only
#ll-config-file=tracker_config.yml
#ll-config-file=iou_config.txt
gpu-id=0
#enable-batch-process and enable-past-frame applicable to DCF only
enable-batch-process=1
enable-past-frame=0
display-tracking-id=0

[img-save]
enable=1
output-folder-path=./logs
save-img-cropped-obj=0
save-img-full-frame=1
second-to-skip-interval=600
min-confidence=0.9
max-confidence=1.0
min-box-width=5
min-box-height=5

[sink0]
enable=1
type=4
source-id=0
codec=1
enc-type=0
sync=0
bitrate=4000000
profile=0
rtsp-port=8554
udp-port=5000
gpu-id=0

[sink1]
enable=1
type=4
source-id=1
codec=1
enc-type=0
sync=0
bitrate=4000000
profile=0
rtsp-port=8555
udp-port=5001
gpu-id=0

[sink2]
enable=1
type=4
source-id=2
codec=1
enc-type=0
sync=0
bitrate=4000000
profile=0
rtsp-port=8556
udp-port=5002
gpu-id=0

[sink3]
enable=1
type=4
source-id=3
codec=1
enc-type=0
sync=0
bitrate=4000000
profile=0
rtsp-port=8557
udp-port=5003
gpu-id=0

[sink4]
enable=1
type=4
source-id=4
codec=1
enc-type=0
sync=0
bitrate=4000000
profile=0
rtsp-port=8558
udp-port=5004
gpu-id=0

[sink5]
enable=1
type=4
source-id=5
codec=1
enc-type=0
sync=0
bitrate=4000000
profile=0
rtsp-port=8559
udp-port=5005
gpu-id=0

[sink6]
enable=1
type=4
source-id=6
codec=1
enc-type=0
sync=0
bitrate=4000000
profile=0
rtsp-port=8560
udp-port=5006
gpu-id=0

[sink7]
enable=1
type=4
source-id=7
codec=1
enc-type=0
sync=0
bitrate=4000000
profile=0
rtsp-port=8561
udp-port=5007
gpu-id=0

[sink8]
enable=1
type=4
source-id=8
codec=1
enc-type=0
sync=0
bitrate=4000000
profile=0
rtsp-port=8562
udp-port=5008
gpu-id=0

[sink9]
enable=1
type=4
source-id=9
codec=1
enc-type=0
sync=0
bitrate=4000000
profile=0
rtsp-port=8563
udp-port=5009
gpu-id=0

[sink10]
enable=1
type=4
source-id=10
codec=1
enc-type=0
sync=0
bitrate=4000000
profile=0
rtsp-port=8564
udp-port=5010
gpu-id=0

[sink11]
enable=1
type=4
source-id=11
codec=1
enc-type=0
sync=0
bitrate=4000000
profile=0
rtsp-port=8565
udp-port=5011
gpu-id=0

[sink12]
enable=1
type=4
source-id=12
codec=1
enc-type=0
sync=0
bitrate=4000000
profile=0
rtsp-port=8566
udp-port=5012
gpu-id=0

[sink13]
enable=1
type=4
source-id=13
codec=1
enc-type=0
sync=0
bitrate=4000000
profile=0
rtsp-port=8567
udp-port=5013
gpu-id=0

[sink14]
enable=1
type=4
source-id=14
codec=1
enc-type=0
sync=0
bitrate=4000000
profile=0
rtsp-port=8568
udp-port=5014
gpu-id=0

[sink15]
enable=1
type=4
source-id=15
codec=1
enc-type=0
sync=0
bitrate=4000000
profile=0
rtsp-port=8569
udp-port=5015
gpu-id=0

[sink16]
enable=1
type=4
source-id=16
codec=1
enc-type=0
sync=0
bitrate=4000000
profile=0
rtsp-port=8570
udp-port=5016
gpu-id=0

[sink17]
enable=1
type=4
source-id=17
codec=1
enc-type=0
sync=0
bitrate=4000000
profile=0
rtsp-port=8571
udp-port=5017
gpu-id=0

[sink18]
enable=1
type=4
source-id=18
codec=1
enc-type=0
sync=0
bitrate=4000000
profile=0
rtsp-port=8572
udp-port=5018
gpu-id=0

[sink19]
enable=1
type=4
source-id=19
codec=1
enc-type=0
sync=0
bitrate=4000000
profile=0
rtsp-port=8573
udp-port=5019
gpu-id=0

[sink20]
enable=1
type=4
source-id=20
codec=1
enc-type=0
sync=0
bitrate=4000000
profile=0
rtsp-port=8574
udp-port=5020
gpu-id=0

[sink21]
enable=1
type=4
source-id=21
codec=1
enc-type=0
sync=0
bitrate=4000000
profile=0
rtsp-port=8575
udp-port=5021
gpu-id=0

[sink22]
enable=1
type=4
source-id=22
codec=1
enc-type=0
sync=0
bitrate=4000000
profile=0
rtsp-port=8576
udp-port=5022
gpu-id=0

[sink23]
enable=1
type=4
source-id=23
codec=1
enc-type=0
sync=0
bitrate=4000000
profile=0
rtsp-port=8577
udp-port=5023
gpu-id=0

[sink24]
enable=1
type=4
source-id=24
codec=1
enc-type=0
sync=0
bitrate=4000000
profile=0
rtsp-port=8578
udp-port=5024
gpu-id=0

[sink25]
enable=1
type=4
source-id=25
codec=1
enc-type=0
sync=0
bitrate=4000000
profile=0
rtsp-port=8579
udp-port=5025
gpu-id=0

[sink26]
enable=1
type=4
source-id=26
codec=1
enc-type=0
sync=0
bitrate=4000000
profile=0
rtsp-port=8580
udp-port=5026
gpu-id=0

[sink27]
enable=1
type=4
source-id=27
codec=1
enc-type=0
sync=0
bitrate=4000000
profile=0
rtsp-port=8581
udp-port=5027
gpu-id=0

[sink28]
enable=1
type=4
source-id=28
codec=1
enc-type=0
sync=0
bitrate=4000000
profile=0
rtsp-port=8582
udp-port=5028
gpu-id=0

[sink29]
enable=1
type=4
source-id=29
codec=1
enc-type=0
sync=0
bitrate=4000000
profile=0
rtsp-port=8583
udp-port=5029
gpu-id=0

PGIE Config

[property]
gpu-id=0
net-scale-factor=0.0039215697906911373
tlt-encoded-model=resnet10_detector.etlt
tlt-model-key=XYZ
labelfile-path=labels.txt
int8-calib-file=calibration.bin
batch-size=30
force-implicit-batch-dim=1
process-mode=1
model-color-format=0
network-mode=1
num-detected-classes=1
interval=4
gie-unique-id=1
infer-dims=3;1080;1920
uff-input-blob-name=input_1
output-blob-names=output_cov/Sigmoid;output_bbox/BiasAdd

[class-attrs-all]
pre-cluster-threshold=0.3
eps=0.3
group-threshold=1

3

Do you mean your server has 20 GPUs?

Since you have enabled 30 rtsp sources(decoding), 30 smart recording(encoding), 30 rtsp outputs(encoding), it is better to share the decoding/encoding loading to different GPUs. You may try to set it with “gpu-id” parameter.

4

20 Core → 2 CPUs with 10 core each.
I have to load each P4 GPU (there are 4 GPUs in the server) with around 30 streams.
I have to run total of 100+ streams on each server.
I have observed that if I disable NVDSAnalytics then I can run 30 streams but given the nature of my requirement I need to enable NVDSAnalytics.

After running the code it takes a while (around 20-30 secs) to observe the RTSP output.

Can I share my code privately for you to have a look.

5

For performance tuning, there are some tips:

https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_troubleshooting.html#the-deepstream-application-is-running-slowly

6

As you can see in the config I have already used all the suggestions. I have experimented with even the batched-push-timeout value.
Can you please review the code and see if you can find any bottlenecks.

7

Have you measured the latency with NVDS_ENABLE_LATENCY_MEASUREMENT and NVDS_ENABLE_COMPONENT_LATENCY_MEASUREMENT?

8

Why do you think the bottleneck is related to your codes? Can this be reproduced with our original deepstream-app or deepstream-test5-app?

9

I will try this.

10

Here is the output with latency ON.

root@va03:/opt/nvidia/deepstream/deepstream-5.0/sources/apps/sample_apps/de
epstream-nvdsanalytics-test# export NVDS_ENABLE_LATENCY_MEASUREMENT=1
root@va03:/opt/nvidia/deepstream/deepstream-5.0/sources/apps/sample_apps/de
epstream-nvdsanalytics-test# export NVDS_ENABLE_COMPONENT_LATENCY_MEASUREMENT=1
root@va03:/opt/nvidia/deepstream/deepstream-5.0/sources/apps/sample_apps/de
epstream-nvdsanalytics-test# ./deepstream-app -c ./assets/topconfig30.txt
*** DeepStream: Launched RTSP Streaming at rtsp://localhost:8554/ds-test ***

*** DeepStream: Launched RTSP Streaming at rtsp://localhost:8555/ds-test ***

*** DeepStream: Launched RTSP Streaming at rtsp://localhost:8556/ds-test ***

*** DeepStream: Launched RTSP Streaming at rtsp://localhost:8557/ds-test ***

*** DeepStream: Launched RTSP Streaming at rtsp://localhost:8558/ds-test ***

*** DeepStream: Launched RTSP Streaming at rtsp://localhost:8559/ds-test ***

*** DeepStream: Launched RTSP Streaming at rtsp://localhost:8560/ds-test ***

*** DeepStream: Launched RTSP Streaming at rtsp://localhost:8561/ds-test ***

*** DeepStream: Launched RTSP Streaming at rtsp://localhost:8562/ds-test ***

*** DeepStream: Launched RTSP Streaming at rtsp://localhost:8563/ds-test ***

*** DeepStream: Launched RTSP Streaming at rtsp://localhost:8564/ds-test ***

*** DeepStream: Launched RTSP Streaming at rtsp://localhost:8565/ds-test ***

*** DeepStream: Launched RTSP Streaming at rtsp://localhost:8566/ds-test ***

*** DeepStream: Launched RTSP Streaming at rtsp://localhost:8567/ds-test ***

*** DeepStream: Launched RTSP Streaming at rtsp://localhost:8568/ds-test ***

*** DeepStream: Launched RTSP Streaming at rtsp://localhost:8569/ds-test ***

*** DeepStream: Launched RTSP Streaming at rtsp://localhost:8570/ds-test ***

*** DeepStream: Launched RTSP Streaming at rtsp://localhost:8571/ds-test ***

*** DeepStream: Launched RTSP Streaming at rtsp://localhost:8572/ds-test ***

*** DeepStream: Launched RTSP Streaming at rtsp://localhost:8573/ds-test ***

*** DeepStream: Launched RTSP Streaming at rtsp://localhost:8574/ds-test ***

*** DeepStream: Launched RTSP Streaming at rtsp://localhost:8575/ds-test ***

*** DeepStream: Launched RTSP Streaming at rtsp://localhost:8576/ds-test ***

*** DeepStream: Launched RTSP Streaming at rtsp://localhost:8577/ds-test ***

*** DeepStream: Launched RTSP Streaming at rtsp://localhost:8578/ds-test ***

*** DeepStream: Launched RTSP Streaming at rtsp://localhost:8579/ds-test ***

*** DeepStream: Launched RTSP Streaming at rtsp://localhost:8580/ds-test ***

*** DeepStream: Launched RTSP Streaming at rtsp://localhost:8581/ds-test ***

*** DeepStream: Launched RTSP Streaming at rtsp://localhost:8582/ds-test ***

*** DeepStream: Launched RTSP Streaming at rtsp://localhost:8583/ds-test ***

gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream-5.0/lib/libnvds_mot_iou.so
gstnvtracker: Optional NvMOT_RemoveStreams not implemented
gstnvtracker: Batch processing is OFF
gstnvtracker: Past frame output is OFF
0:00:00.588232976 653 0x5599737b2690 INFO nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1715> [UID = 1]: Trying to create engine from model files
INFO: …/nvdsinfer/nvdsinfer_func_utils.cpp:39 [TRT]: Reading Calibration Cache for calibrator: EntropyCalibration2
INFO: …/nvdsinfer/nvdsinfer_func_utils.cpp:39 [TRT]: Generated calibration scales using calibration cache. Make sure that calibration cache has latest scales.
INFO: …/nvdsinfer/nvdsinfer_func_utils.cpp:39 [TRT]: To regenerate calibration cache, please delete the existing one. TensorRT will generate a new calibration cache.
INFO: …/nvdsinfer/nvdsinfer_func_utils.cpp:39 [TRT]: Some tactics do not have sufficient workspace memory to run. Increasing workspace size may increase performance, please check verbose output.
INFO: …/nvdsinfer/nvdsinfer_func_utils.cpp:39 [TRT]: Detected 1 inputs and 2 output network tensors.
0:01:14.120155576 653 0x5599737b2690 INFO nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1748> [UID = 1]: serialize cuda engine to file: /opt/nvidia/deepstream/deepstream-5.0/sources/apps/sample_apps/deepstream-nvdsanalytics-test/assets/resnet10_detector.etlt_b30_gpu0_int8.engine successfully
WARNING: …/nvdsinfer/nvdsinfer_func_utils.cpp:36 [TRT]: Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles
INFO: …/nvdsinfer/nvdsinfer_model_builder.cpp:685 [Implicit Engine Info]: layers num: 3
0 INPUT kFLOAT input_1 3x1080x1920
1 OUTPUT kFLOAT output_bbox/BiasAdd 4x68x120
2 OUTPUT kFLOAT output_cov/Sigmoid 1x68x120

0:01:14.132631208 653 0x5599737b2690 INFO nvinfer gstnvinfer_impl.cpp:313:notifyLoadModelStatus:<primary_gie> [UID 1]: Load new model:/opt/nvidia/deepstream/deepstream-5.0/sources/apps/sample_apps/deepstream-nvdsanalytics-test/assets/pgie-config.txt sucessfully

Runtime commands:
h: Print this help
q: Quit

    p: Pause
    r: Resume

**PERF: FPS 0 (Avg) FPS 1 (Avg) FPS 2 (Avg) FPS 3 (Avg) FPS 4 (Avg) FPS 5 (Avg) FPS 6 (Avg) FPS 7 (Avg) FPS 8 (Avg)FPS 9 (Avg) FPS 10 (Avg) FPS 11 (Avg) FPS 12 (Avg) FPS 13 (Avg) FPS 14 (Avg) FPS 15 (Avg) FPS 16 (Avg) FPS 17 (Avg) FPS 18 (Avg) FPS 19 (Avg) FPS 20 (Avg) FPS 21 (Avg) FPS 22 (Avg) FPS 23 (Avg) FPS 24 (Avg) FPS 25 (Avg) FPS 26 (Avg) FPS 27 (Avg) FPS 28 (Avg) FPS 29 (Avg)
**PERF: 0.00 (0.00) 0.00 (0.00) 0.00 (0.00) 0.00 (0.00) 0.00 (0.00) 0.00 (0.00) 0.00 (0.00) 0.00 (0.00) 0.00 (0.00)0.00 (0.00) 0.00 (0.00) 0.00 (0.00) 0.00 (0.00) 0.00 (0.00)0.00 (0.00) 0.00 (0.00) 0.00 (0.00) 0.00 (0.00) 0.00 (0.00)0.00 (0.00) 0.00 (0.00) 0.00 (0.00) 0.00 (0.00) 0.00 (0.00)0.00 (0.00) 0.00 (0.00) 0.00 (0.00) 0.00 (0.00) 0.00 (0.00)0.00 (0.00)
** INFO: <bus_callback:181>: Pipeline ready

ERROR from src_elem6: Unauthorized
Debug info: gstrtspsrc.c(6116): gst_rtspsrc_send (): /GstPipeline:pipeline/GstBin:multi_src_bin/GstBin:src_sub_bin6/GstRTSPSrc:src_elem6:
Unauthorized (401)
** INFO: <reset_source_pipeline:1154>: Resetting source 6
ERROR from src_elem14: Unauthorized
Debug info: gstrtspsrc.c(6116): gst_rtspsrc_send (): /GstPipeline:pipeline/GstBin:multi_src_bin/GstBin:src_sub_bin14/GstRTSPSrc:src_elem14:
Unauthorized (401)
** INFO: <reset_source_pipeline:1154>: Resetting source 14
ERROR from src_elem6: Unauthorized
Debug info: gstrtspsrc.c(6116): gst_rtspsrc_send (): /GstPipeline:pipeline/GstBin:multi_src_bin/GstBin:src_sub_bin6/GstRTSPSrc:src_elem6:
Unauthorized (401)
ERROR from src_elem14: Unauthorized
Debug info: gstrtspsrc.c(6116): gst_rtspsrc_send (): /GstPipeline:pipeline/GstBin:multi_src_bin/GstBin:src_sub_bin14/GstRTSPSrc:src_elem14:
Unauthorized (401)
IOU Tracker Init with threshold 0.100000
IOU Tracker Init with threshold 0.100000
IOU Tracker Init with threshold 0.100000
IOU Tracker Init with threshold 0.100000
Alerting Set after 200 frames
IOU Tracker Init with threshold 0.100000
IOU Tracker Init with threshold 0.100000
IOU Tracker Init with threshold 0.100000
IOU Tracker Init with threshold 0.100000
IOU Tracker Init with threshold 0.100000
IOU Tracker Init with threshold 0.100000
IOU Tracker Init with threshold 0.100000
IOU Tracker Init with threshold 0.100000
IOU Tracker Init with threshold 0.100000
IOU Tracker Init with threshold 0.100000
IOU Tracker Init with threshold 0.100000
IOU Tracker Init with threshold 0.100000
IOU Tracker Init with threshold 0.100000
IOU Tracker Init with threshold 0.100000
IOU Tracker Init with threshold 0.100000
IOU Tracker Init with threshold 0.100000
IOU Tracker Init with threshold 0.100000
IOU Tracker Init with threshold 0.100000
IOU Tracker Init with threshold 0.100000
IOU Tracker Init with threshold 0.100000
IOU Tracker Init with threshold 0.100000
IOU Tracker Init with threshold 0.100000
IOU Tracker Init with threshold 0.100000

BATCH-NUM = 0**
Batch meta not found for buffer 0x7f588c09c610
BATCH-NUM = 1**
Batch meta not found for buffer 0x7f5a7009bc50
BATCH-NUM = 2**
Batch meta not found for buffer 0x7f5a54069c90
BATCH-NUM = 3**
Batch meta not found for buffer 0x7f5a7009c000
BATCH-NUM = 4**
Batch meta not found for buffer 0x7f57d013b8c0
BATCH-NUM = 5**
Batch meta not found for buffer 0x7f57c0094440
BATCH-NUM = 6**
Batch meta not found for buffer 0x7f5900081580
BATCH-NUM = 7**
Batch meta not found for buffer 0x7f59cc09c800
BATCH-NUM = 8**
Batch meta not found for buffer 0x7f5a540ba290
BATCH-NUM = 9**
Batch meta not found for buffer 0x7f599c080670
BATCH-NUM = 10**
Batch meta not found for buffer 0x7f582007c930
BATCH-NUM = 11**
Batch meta not found for buffer 0x7f5968090560
BATCH-NUM = 12**
Batch meta not found for buffer 0x7f593c0aa410
BATCH-NUM = 13**
Batch meta not found for buffer 0x7f5a70081700
BATCH-NUM = 14**
Batch meta not found for buffer 0x7f590007e080
BATCH-NUM = 15**
Batch meta not found for buffer 0x7f58cc077220
BATCH-NUM = 16**
Batch meta not found for buffer 0x7f5a540cfc80
BATCH-NUM = 17**
Batch meta not found for buffer 0x7f596805eb30
BATCH-NUM = 18**
Batch meta not found for buffer 0x7f59b409b4b0
BATCH-NUM = 19**
Batch meta not found for buffer 0x7f58c008c9e0
BATCH-NUM = 20**
Batch meta not found for buffer 0x7f588c08b070
BATCH-NUM = 21**
Batch meta not found for buffer 0x7f587412cab0
BATCH-NUM = 22**
Batch meta not found for buffer 0x7f593c070cc0
BATCH-NUM = 23**
Batch meta not found for buffer 0x7f57dc05a1e0
BATCH-NUM = 24**
Batch meta not found for buffer 0x7f59cc09c5e0
BATCH-NUM = 25**
Batch meta not found for buffer 0x7f584406a5c0
BATCH-NUM = 26**
Batch meta not found for buffer 0x7f5a54064bf0
BATCH-NUM = 27**
Batch meta not found for buffer 0x7f5a24086770
BATCH-NUM = 28**
Batch meta not found for buffer 0x7f58c006cb10
BATCH-NUM = 29**
Batch meta not found for buffer 0x7f5a70104180
BATCH-NUM = 30**
Batch meta not found for buffer 0x7f57d0106d90
BATCH-NUM = 31**
Batch meta not found for buffer 0x7f59cc0d1b20
BATCH-NUM = 32**
Batch meta not found for buffer 0x7f590008d810
BATCH-NUM = 33**
Batch meta not found for buffer 0x7f58cc0f11d0
BATCH-NUM = 34**
Batch meta not found for buffer 0x7f57c00b98f0
BATCH-NUM = 35**
Batch meta not found for buffer 0x7f5a700852e0
BATCH-NUM = 36**
Batch meta not found for buffer 0x7f5a180f3c10
BATCH-NUM = 37**
Batch meta not found for buffer 0x7f596c0a1090
BATCH-NUM = 38**
Batch meta not found for buffer 0x7f599c0c4030
BATCH-NUM = 39**
Batch meta not found for buffer 0x7f5844079510
BATCH-NUM = 40**
Batch meta not found for buffer 0x7f58cc0c5850
BATCH-NUM = 41**
Batch meta not found for buffer 0x7f59f409ce50
BATCH-NUM = 42**
Batch meta not found for buffer 0x7f57d0087bf0
BATCH-NUM = 43**
Batch meta not found for buffer 0x7f5a180ae720
BATCH-NUM = 44**
Batch meta not found for buffer 0x7f57ec0654d0
BATCH-NUM = 45**
Batch meta not found for buffer 0x7f59680a8060
BATCH-NUM = 46**
Batch meta not found for buffer 0x7f5a54069a70
BATCH-NUM = 47**
Batch meta not found for buffer 0x7f59b40a0b30
BATCH-NUM = 48**
Batch meta not found for buffer 0x7f57a40576c0**PERF: 5.02 (4.98) 3.79 (3.77) 0.00 (0.00) 2.99 (2.96) 7.90 (7.79) 3.14 (3.12)0.00 (0.00) 6.48 (5.88) 5.91 (5.44) 3.36 (3.34) 4.87 (4.72)4.37 (4.34) 4.18 (3.92) 4.16 (4.13) 0.00 (0.00) 11.37 (10.30) 3.48 (3.41) 0.96 (0.67) 8.68 (7.63) 16.57 (3.45) 5.15 (5.11) 6.98 (6.33) 3.52 (3.50) 3.84 (3.65) 3.80 (3.60)6.17 (5.70) 3.67 (3.65) 9.98 (9.42) 3.43 (3.41) 3.62 (3.60)

BATCH-NUM = 49**
Batch meta not found for buffer 0x7f58cc1085f0
BATCH-NUM = 50**
Batch meta not found for buffer 0x7f588c0e2a00
BATCH-NUM = 51**
Batch meta not found for buffer 0x7f58741321d0
BATCH-NUM = 52**
Batch meta not found for buffer 0x7f59840b9a70
BATCH-NUM = 53**
Batch meta not found for buffer 0x7f58e808ca50
BATCH-NUM = 54**
Batch meta not found for buffer 0x7f5a18083d80
BATCH-NUM = 55**
Batch meta not found for buffer 0x7f57d015a260
BATCH-NUM = 56**
Batch meta not found for buffer 0x7f593c0a6bd0
BATCH-NUM = 57**
Batch meta not found for buffer 0x7f5a7009c000
BATCH-NUM = 58**
Batch meta not found for buffer 0x7f57ec0e1460
BATCH-NUM = 59**
Batch meta not found for buffer 0x7f57ec0b3a60
BATCH-NUM = 60**
Batch meta not found for buffer 0x7f57a8111890
BATCH-NUM = 61**
Batch meta not found for buffer 0x7f59cc0d1b20
BATCH-NUM = 62**
Batch meta not found for buffer 0x7f588c0e2a00
BATCH-NUM = 63**
Batch meta not found for buffer 0x7f58a80b5030
BATCH-NUM = 64**
Batch meta not found for buffer 0x7f5a540ef9b0
BATCH-NUM = 65**
Batch meta not found for buffer 0x7f57a8111cd0
BATCH-NUM = 66**
Batch meta not found for buffer 0x7f59680bc010
BATCH-NUM = 67**
Batch meta not found for buffer 0x7f58a8090cd0
BATCH-NUM = 68**
Batch meta not found for buffer 0x7f5a1805e5b0
BATCH-NUM = 69**
Batch meta not found for buffer 0x7f593c08f5f0
BATCH-NUM = 70**
Batch meta not found for buffer 0x7f5a18083d80
BATCH-NUM = 71**
Batch meta not found for buffer 0x7f57a8069360
BATCH-NUM = 72**
Batch meta not found for buffer 0x7f58e808f560
BATCH-NUM = 73**
Batch meta not found for buffer 0x7f58c006a370
BATCH-NUM = 74**
Batch meta not found for buffer 0x7f5874089af0
BATCH-NUM = 75**
Batch meta not found for buffer 0x7f59f409ce50
BATCH-NUM = 76**
Batch meta not found for buffer 0x7f59641919c0
BATCH-NUM = 77**
Batch meta not found for buffer 0x7f59f408d010
BATCH-NUM = 78**
Batch meta not found for buffer 0x7f584407cbb0
BATCH-NUM = 79**
Batch meta not found for buffer 0x7f59f409c7f0
BATCH-NUM = 80**
Batch meta not found for buffer 0x7f59cc0ba930
BATCH-NUM = 81**
Batch meta not found for buffer 0x7f57a80c5830
BATCH-NUM = 82**
Batch meta not found for buffer 0x7f585c117ac0
BATCH-NUM = 83**
Batch meta not found for buffer 0x7f5a18080bb0
BATCH-NUM = 84**
Batch meta not found for buffer 0x7f5a14053d70
BATCH-NUM = 85**
Batch meta not found for buffer 0x7f59cc0ce060
BATCH-NUM = 86**
Batch meta not found for buffer 0x7f596807ea60
BATCH-NUM = 87**
Batch meta not found for buffer 0x7f59cc059520
BATCH-NUM = 88**
Batch meta not found for buffer 0x7f58a811a470
BATCH-NUM = 89**
Batch meta not found for buffer 0x7f59cc0c8230
BATCH-NUM = 90**
Batch meta not found for buffer 0x7f599c0997b0
BATCH-NUM = 91**
Batch meta not found for buffer 0x7f57d0094400
BATCH-NUM = 92**
Batch meta not found for buffer 0x7f5bd00c77c0
BATCH-NUM = 93**
Batch meta not found for buffer 0x7f5a70089bc0
BATCH-NUM = 94**
Batch meta not found for buffer 0x7f59000c30c0
BATCH-NUM = 95**
Batch meta not found for buffer 0x7f58740b9e50
BATCH-NUM = 96**
Batch meta not found for buffer 0x7f59cc07c460
BATCH-NUM = 97**
Batch meta not found for buffer 0x7f59cc08d4b0
BATCH-NUM = 98**
Batch meta not found for buffer 0x7f59840b9410
BATCH-NUM = 99**
Batch meta not found for buffer 0x7f58c006c3a0
BATCH-NUM = 100**
Batch meta not found for buffer 0x7f59b4082b50
BATCH-NUM = 101**
Batch meta not found for buffer 0x7f599c099bf0
BATCH-NUM = 102**
Batch meta not found for buffer 0x7f585c0a8740
BATCH-NUM = 103**
Batch meta not found for buffer 0x7f58e806d9f0
BATCH-NUM = 104**
Batch meta not found for buffer 0x7f57d0094400
BATCH-NUM = 105**
Batch meta not found for buffer 0x7f587413fe00
BATCH-NUM = 106**
Batch meta not found for buffer 0x7f5a540da4f0
BATCH-NUM = 107**
Batch meta not found for buffer 0x7f58c007b9e0
BATCH-NUM = 108**
Batch meta not found for buffer 0x7f58e8085cd0
BATCH-NUM = 109**
Batch meta not found for buffer 0x7f5a180f1910
BATCH-NUM = 110**
Batch meta not found for buffer 0x7f57d0157660
BATCH-NUM = 111**
Batch meta not found for buffer 0x7f593c0e5e70
BATCH-NUM = 112**
Batch meta not found for buffer 0x7f596806be20
BATCH-NUM = 113**
Batch meta not found for buffer 0x7f57ec0d4880
BATCH-NUM = 114**
Batch meta not found for buffer 0x7f57c007b260
BATCH-NUM = 115**
Batch meta not found for buffer 0x7f57a80a6500
BATCH-NUM = 116**
Batch meta not found for buffer 0x7f574c011520
BATCH-NUM = 117**
Batch meta not found for buffer 0x7f5968090560
BATCH-NUM = 118**
Batch meta not found for buffer 0x7f5a540919f0
BATCH-NUM = 119**
Batch meta not found for buffer 0x7f57ec084c00
BATCH-NUM = 120**
Batch meta not found for buffer 0x7f5a701048f0
BATCH-NUM = 121**
Batch meta not found for buffer 0x7f585c0fa520
BATCH-NUM = 122**
Batch meta not found for buffer 0x7f585c088c80
BATCH-NUM = 123**
Batch meta not found for buffer 0x7f59b4070660
BATCH-NUM = 124**
Batch meta not found for buffer 0x7f58cc0b1a10
BATCH-NUM = 125**
Batch meta not found for buffer 0x7f5a18076720
BATCH-NUM = 126**
Batch meta not found for buffer 0x7f5a701076f0
BATCH-NUM = 127**
Batch meta not found for buffer 0x7f57f800dd70
BATCH-NUM = 128**
Batch meta not found for buffer 0x7f5a540cf840
BATCH-NUM = 129**
Batch meta not found for buffer 0x7f59840f8d90
BATCH-NUM = 130**
Batch meta not found for buffer 0x7f58cc077110
BATCH-NUM = 131**
Batch meta not found for buffer 0x7f599c0922e0
BATCH-NUM = 132**
Batch meta not found for buffer 0x7f59b40823e0
BATCH-NUM = 133**
Batch meta not found for buffer 0x7f585c078110
BATCH-NUM = 134**
Batch meta not found for buffer 0x7f57d0106730
BATCH-NUM = 135**
Batch meta not found for buffer 0x7f5a54064480
BATCH-NUM = 136**
Batch meta not found for buffer 0x7f57d00a1260
BATCH-NUM = 137**
Batch meta not found for buffer 0x7f57c00aae40
BATCH-NUM = 138**
Batch meta not found for buffer 0x7f57d00940d0
BATCH-NUM = 139**
Batch meta not found for buffer 0x7f58740ac850
BATCH-NUM = 140**
Batch meta not found for buffer 0x7f596c0a2e90
BATCH-NUM = 141**
Batch meta not found for buffer 0x7f57ec084c00
BATCH-NUM = 142**
Batch meta not found for buffer 0x7f58cc06ed70
BATCH-NUM = 143**
Batch meta not found for buffer 0x7f59b407ad90
BATCH-NUM = 144**
Batch meta not found for buffer 0x7f5a1805ee30
BATCH-NUM = 145**
Batch meta not found for buffer 0x7f588c0bf030
BATCH-NUM = 146**
Batch meta not found for buffer 0x7f593c0d5520
BATCH-NUM = 147**
Batch meta not found for buffer 0x7f585c0ab460
BATCH-NUM = 148**
Batch meta not found for buffer 0x7f59840b43e0
BATCH-NUM = 149**
Batch meta not found for buffer 0x7f5a180b5250
BATCH-NUM = 150**
Batch meta not found for buffer 0x7f5a540791c0
BATCH-NUM = 151**
Batch meta not found for buffer 0x7f5a700a3700
BATCH-NUM = 152**
Batch meta not found for buffer 0x7f5a24086dd0
BATCH-NUM = 153**
Batch meta not found for buffer 0x7f58cc110d70**PERF: 5.54 (5.12) 5.54 (4.59) 0.00 (0.00) 6.39 (5.12) 5.54 (5.85) 5.75 (4.41)0.00 (0.00) 5.54 (5.55) 5.54 (5.41) 5.54 (4.39) 5.68 (5.16)5.54 (4.88) 5.33 (4.69) 5.75 (4.91) 0.00 (0.00) 5.44 (6.16)5.47 (4.39) 5.67 (4.62) 5.54 (5.97) 3.50 (4.48) 5.54 (5.17)5.74 (5.82) 5.54 (4.47) 5.54 (4.61) 5.54 (4.61) 5.54 (5.50)5.54 (4.50) 5.68 (6.32) 5.54 (4.39) 5.54 (4.50)

BATCH-NUM = 154**
Batch meta not found for buffer 0x7f59840ef370
BATCH-NUM = 155**
Batch meta not found for buffer 0x7f57ec0dbe80
BATCH-NUM = 156**
Batch meta not found for buffer 0x7f58740b94c0
BATCH-NUM = 157**
Batch meta not found for buffer 0x7f57d00a1e10
BATCH-NUM = 158**
Batch meta not found for buffer 0x7f58a8098d70
BATCH-NUM = 159**
Batch meta not found for buffer 0x7f57a812ce00
BATCH-NUM = 160**
Batch meta not found for buffer 0x7f5900106ea0
BATCH-NUM = 161**
Batch meta not found for buffer 0x7f59000a6620
BATCH-NUM = 162**
Batch meta not found for buffer 0x7f57d015ae10
BATCH-NUM = 163**
Batch meta not found for buffer 0x7f59b4070660
BATCH-NUM = 164**
Batch meta not found for buffer 0x7f5a180df370
BATCH-NUM = 165**
Batch meta not found for buffer 0x7f5b50082b10
BATCH-NUM = 166**
Batch meta not found for buffer 0x7f59b407f5d0
BATCH-NUM = 167**
Batch meta not found for buffer 0x7f58cc06bac0
BATCH-NUM = 168**
Batch meta not found for buffer 0x7f5a541151b0
BATCH-NUM = 169**
Batch meta not found for buffer 0x7f57a807f4c0
BATCH-NUM = 170**
Batch meta not found for buffer 0x7f58a8087a60
BATCH-NUM = 171**
Batch meta not found for buffer 0x7f59cc0bc2d0
BATCH-NUM = 172**
Batch meta not found for buffer 0x7f5a180ae610
BATCH-NUM = 173**
Batch meta not found for buffer 0x7f599c0c4140
BATCH-NUM = 174**
Batch meta not found for buffer 0x7f585c0fa520
BATCH-NUM = 175**
Batch meta not found for buffer 0x7f585c09db20
BATCH-NUM = 176**
Batch meta not found for buffer 0x7f58740b8300
BATCH-NUM = 177**
Batch meta not found for buffer 0x7f58cc0f1500
BATCH-NUM = 178**
Batch meta not found for buffer 0x7f5a540e2700
BATCH-NUM = 179**
Batch meta not found for buffer 0x7f58a811a470
BATCH-NUM = 180**
Batch meta not found for buffer 0x7f59f40ceb80
BATCH-NUM = 181**
Batch meta not found for buffer 0x7f59840b9b80
BATCH-NUM = 182**
Batch meta not found for buffer 0x7f585c070e40
BATCH-NUM = 183**
Batch meta not found for buffer 0x7f599c080670
BATCH-NUM = 184**
Batch meta not found for buffer 0x7f58cc080130
BATCH-NUM = 185**
Batch meta not found for buffer 0x7f5900119600
BATCH-NUM = 186**
Batch meta not found for buffer 0x7f588c0c2660
BATCH-NUM = 187**
Batch meta not found for buffer 0x7f57c0072bf0
BATCH-NUM = 188**
Batch meta not found for buffer 0x7f5a180e9230
BATCH-NUM = 189**
Batch meta not found for buffer 0x7f5968256150
BATCH-NUM = 190**
Batch meta not found for buffer 0x7f59cc072740
BATCH-NUM = 191**
Batch meta not found for buffer 0x7f584410de70
BATCH-NUM = 192**
Batch meta not found for buffer 0x7f59840ae4b0
BATCH-NUM = 193**
Batch meta not found for buffer 0x7f5a18073480
BATCH-NUM = 194**
Batch meta not found for buffer 0x7f599c092830
BATCH-NUM = 195**
Batch meta not found for buffer 0x7f599c07e4a0
BATCH-NUM = 196**
Batch meta not found for buffer 0x7f57a812c140
BATCH-NUM = 197**
Batch meta not found for buffer 0x7f599c0c4470
BATCH-NUM = 198**
Batch meta not found for buffer 0x7f5a180e9230
BATCH-NUM = 199**
Batch meta not found for buffer 0x7f5a18073480
BATCH-NUM = 200**
Batch meta not found for buffer 0x7f587412c9a0
BATCH-NUM = 201**
Batch meta not found for buffer 0x7f5a1809a080
BATCH-NUM = 202**
Batch meta not found for buffer 0x7f57a80c5720
BATCH-NUM = 203**
Batch meta not found for buffer 0x7f585c09c460
BATCH-NUM = 204**
Batch meta not found for buffer 0x7f59b40d9950
BATCH-NUM = 205**
Batch meta not found for buffer 0x7f5a180fb0c0
BATCH-NUM = 206**
Batch meta not found for buffer 0x7f5a18073480
BATCH-NUM = 207**
Batch meta not found for buffer 0x7f58740f2450
BATCH-NUM = 208**
Batch meta not found for buffer 0x7f590008db40
BATCH-NUM = 209**
Batch meta not found for buffer 0x7f5820078e30
BATCH-NUM = 210**
Batch meta not found for buffer 0x7f58a80c8bf0
BATCH-NUM = 211**
Batch meta not found for buffer 0x7f59cc059a70
BATCH-NUM = 212**
Batch meta not found for buffer 0x7f58cc06ee80
BATCH-NUM = 213**
Batch meta not found for buffer 0x7f585c0fab80
BATCH-NUM = 214**
Batch meta not found for buffer 0x7f59f40d4230
BATCH-NUM = 215**
Batch meta not found for buffer 0x7f5a7009d7e0
BATCH-NUM = 216**
Batch meta not found for buffer 0x7f57d007f920
BATCH-NUM = 217**
Batch meta not found for buffer 0x7f5a540f0810
BATCH-NUM = 218**
Batch meta not found for buffer 0x7f599c0b57c0
BATCH-NUM = 219**
Batch meta not found for buffer 0x7f57d009d140
BATCH-NUM = 220**
Batch meta not found for buffer 0x7f5a18076830
BATCH-NUM = 221**
Batch meta not found for buffer 0x7f59f4086350
BATCH-NUM = 222**
Batch meta not found for buffer 0x7f587413c560
BATCH-NUM = 223**
Batch meta not found for buffer 0x7f59cc0baa40
BATCH-NUM = 224**
Batch meta not found for buffer 0x7f59f4067bd0
BATCH-NUM = 225**
Batch meta not found for buffer 0x7f596c08a7d0
BATCH-NUM = 226**
Batch meta not found for buffer 0x7f5a7009d7e0
BATCH-NUM = 227**
Batch meta not found for buffer 0x7f587411c900
BATCH-NUM = 228**
Batch meta not found for buffer 0x7f59f40d4230
BATCH-NUM = 229**
Batch meta not found for buffer 0x7f596806d0b0
BATCH-NUM = 230**
Batch meta not found for buffer 0x7f5a18076830
BATCH-NUM = 231**
Batch meta not found for buffer 0x7f57c00bfe80
BATCH-NUM = 232**
Batch meta not found for buffer 0x7f5820078e30
BATCH-NUM = 233**
Batch meta not found for buffer 0x7f585c08bc10
BATCH-NUM = 234**
Batch meta not found for buffer 0x7f584406d1b0
BATCH-NUM = 235**
Batch meta not found for buffer 0x7f585c088620
BATCH-NUM = 236**
Batch meta not found for buffer 0x7f59000a2140
BATCH-NUM = 237**
Batch meta not found for buffer 0x7f585c0a8c90
BATCH-NUM = 238**
Batch meta not found for buffer 0x7f58cc0a06d0
BATCH-NUM = 239**
Batch meta not found for buffer 0x7f59001194f0
BATCH-NUM = 240**
Batch meta not found for buffer 0x7f587413c560
BATCH-NUM = 241**
Batch meta not found for buffer 0x7f58c007ed10
BATCH-NUM = 242**
Batch meta not found for buffer 0x7f57a4057e30
BATCH-NUM = 243**
Batch meta not found for buffer 0x7f5a701000c0
BATCH-NUM = 244**
Batch meta not found for buffer 0x7f59cc0baa40
BATCH-NUM = 245**
Batch meta not found for buffer 0x7f57a80757d0
BATCH-NUM = 246**
Batch meta not found for buffer 0x7f5a540b28f0**PERF: 5.26 (5.31) 5.26 (4.96) 0.00 (0.00) 5.26 (5.32) 5.26 (5.74) 5.26 (4.84)0.00 (0.00) 5.26 (5.57) 5.26 (5.49) 5.26 (4.82) 5.26 (5.33)5.26 (5.16) 5.26 (5.03) 5.26 (5.17) 0.00 (0.00) 5.26 (5.91)5.45 (4.89) 6.39 (5.57) 5.26 (5.81) 5.26 (4.90) 5.26 (5.33)5.26 (5.73) 5.45 (4.96) 5.45 (5.03) 5.26 (4.97) 5.26 (5.54)5.45 (4.96) 5.26 (6.01) 5.45 (4.89) 5.45 (4.97)

BATCH-NUM = 247**
Batch meta not found for buffer 0x7f57a809a670
BATCH-NUM = 248**
Batch meta not found for buffer 0x7f59840ef6a0
BATCH-NUM = 249**
Batch meta not found for buffer 0x7f58e8085890
BATCH-NUM = 250**
Batch meta not found for buffer 0x7f59b40861c0
BATCH-NUM = 251**
Batch meta not found for buffer 0x7f59000c3e90
BATCH-NUM = 252**
Batch meta not found for buffer 0x7f57c006f140
BATCH-NUM = 253**
Batch meta not found for buffer 0x7f5a180e69e0
BATCH-NUM = 254**
Batch meta not found for buffer 0x7f58740b9a10
BATCH-NUM = 255**
Batch meta not found for buffer 0x7f5a540b73f0
BATCH-NUM = 256**
Batch meta not found for buffer 0x7f57ec0cc780
BATCH-NUM = 257**
Batch meta not found for buffer 0x7f59cc0e6240
BATCH-NUM = 258**
Batch meta not found for buffer 0x7f58cc0f1610
BATCH-NUM = 259**
Batch meta not found for buffer 0x7f59b40743d0
BATCH-NUM = 260**
Batch meta not found for buffer 0x7f58740bc570
BATCH-NUM = 261**
Batch meta not found for buffer 0x7f5a180e4240
BATCH-NUM = 262**
Batch meta not found for buffer 0x7f58200f0050
BATCH-NUM = 263**
Batch meta not found for buffer 0x7f58cc0dc300
BATCH-NUM = 264**
Batch meta not found for buffer 0x7f58200cad30
BATCH-NUM = 265**
Batch meta not found for buffer 0x7f58e80805c0
BATCH-NUM = 266**
Batch meta not found for buffer 0x7f57a8082ce0
BATCH-NUM = 267**
Batch meta not found for buffer 0x7f58e807a9e0
BATCH-NUM = 268**
Batch meta not found for buffer 0x7f584406d1b0
BATCH-NUM = 269**
Batch meta not found for buffer 0x7f58c007ed10
BATCH-NUM = 270**
Batch meta not found for buffer 0x7f59680bc670
BATCH-NUM = 271**
Batch meta not found for buffer 0x7f58a80ab850
BATCH-NUM = 272**
Batch meta not found for buffer 0x7f59f40719f0
BATCH-NUM = 273**
Batch meta not found for buffer 0x7f5900081cf0
BATCH-NUM = 274**
Batch meta not found for buffer 0x7f57ec087880
BATCH-NUM = 275**
Batch meta not found for buffer 0x7f5844073520
BATCH-NUM = 276**
Batch meta not found for buffer 0x7f598409b300
BATCH-NUM = 277**
Batch meta not found for buffer 0x7f58cc0e0580
BATCH-NUM = 278**
Batch meta not found for buffer 0x7f57a81280c0
BATCH-NUM = 279**
Batch meta not found for buffer 0x7f58cc0ba710
BATCH-NUM = 280**
Batch meta not found for buffer 0x7f5a180737b0
BATCH-NUM = 281**
Batch meta not found for buffer 0x7f58cc0b1080
BATCH-NUM = 282**
Batch meta not found for buffer 0x7f58a80f6270
BATCH-NUM = 283**
Batch meta not found for buffer 0x7f58cc0f9e70
BATCH-NUM = 284**
Batch meta not found for buffer 0x7f58cc0e9410
BATCH-NUM = 285**
Batch meta not found for buffer 0x7f58a805fa60
BATCH-NUM = 286**
Batch meta not found for buffer 0x7f59000f0560
BATCH-NUM = 287**
Batch meta not found for buffer 0x7f5a180aee90
BATCH-NUM = 288**
Batch meta not found for buffer 0x7f588c0aaae0
BATCH-NUM = 289**
Batch meta not found for buffer 0x7f59000ebea0
BATCH-NUM = 290**
Batch meta not found for buffer 0x7f58a8095010
BATCH-NUM = 291**
Batch meta not found for buffer 0x7f57ec0937b0
BATCH-NUM = 292**
Batch meta not found for buffer 0x7f585c082e00
BATCH-NUM = 293**
Batch meta not found for buffer 0x7f582007b4b0
BATCH-NUM = 294**
Batch meta not found for buffer 0x7f58440759a0
BATCH-NUM = 295**
Batch meta not found for buffer 0x7f5a180df6a0
BATCH-NUM = 296**
Batch meta not found for buffer 0x7f599c093d60
BATCH-NUM = 297**
Batch meta not found for buffer 0x7f5a182903c0
BATCH-NUM = 298**
Batch meta not found for buffer 0x7f5820080460
BATCH-NUM = 299**
Batch meta not found for buffer 0x7f5a541052f0
BATCH-NUM = 300**
Batch meta not found for buffer 0x7f58cc0808a0
BATCH-NUM = 301**
Batch meta not found for buffer 0x7f596c07e880
BATCH-NUM = 302**
Batch meta not found for buffer 0x7f585c0fec00
BATCH-NUM = 303**
Batch meta not found for buffer 0x7f59b40d9620
BATCH-NUM = 304**
Batch meta not found for buffer 0x7f593c08a4c0
BATCH-NUM = 305**
Batch meta not found for buffer 0x7f5a70095c20
BATCH-NUM = 306**
Batch meta not found for buffer 0x7f5a1807c8e0
BATCH-NUM = 307**
Batch meta not found for buffer 0x7f5a540b7720
BATCH-NUM = 308**
Batch meta not found for buffer 0x7f57ec0c6390
BATCH-NUM = 309**
Batch meta not found for buffer 0x7f5a700852e0
BATCH-NUM = 310**
Batch meta not found for buffer 0x7f585c088b70
BATCH-NUM = 311**
Batch meta not found for buffer 0x7f58c00747a0
BATCH-NUM = 312**
Batch meta not found for buffer 0x7f57d011bea0
BATCH-NUM = 313**
Batch meta not found for buffer 0x7f593c0a6ac0
BATCH-NUM = 314**
Batch meta not found for buffer 0x7f59f4067df0
BATCH-NUM = 315**
Batch meta not found for buffer 0x7f59b4070220
BATCH-NUM = 316**
Batch meta not found for buffer 0x7f5a540e22c0
BATCH-NUM = 317**
Batch meta not found for buffer 0x7f57d00aa170
BATCH-NUM = 318**
Batch meta not found for buffer 0x7f58c00747a0
BATCH-NUM = 319**
Batch meta not found for buffer 0x7f59cc0a6d60
BATCH-NUM = 320**
Batch meta not found for buffer 0x7f5bd00c75a0
BATCH-NUM = 321**
Batch meta not found for buffer 0x7f599c07ad50
BATCH-NUM = 322**
Batch meta not found for buffer 0x7f59b40c2090
BATCH-NUM = 323**
Batch meta not found for buffer 0x7f57d00aa170
BATCH-NUM = 324**
Batch meta not found for buffer 0x7f596c07d0e0
BATCH-NUM = 325**
Batch meta not found for buffer 0x7f58200dd550
BATCH-NUM = 326**
Batch meta not found for buffer 0x7f582007a240
BATCH-NUM = 327**
Batch meta not found for buffer 0x7f58740bc570
BATCH-NUM = 328**
Batch meta not found for buffer 0x7f58c007bd10
BATCH-NUM = 329**
Batch meta not found for buffer 0x7f58cc10c030
BATCH-NUM = 330**
Batch meta not found for buffer 0x7f588c0bf360
BATCH-NUM = 331**
Batch meta not found for buffer 0x7f5a540be460
BATCH-NUM = 332**
Batch meta not found for buffer 0x7f57ec0e1570
BATCH-NUM = 333**
Batch meta not found for buffer 0x7f59cc0a6d60
BATCH-NUM = 334**
Batch meta not found for buffer 0x7f57c007b7b0
BATCH-NUM = 335**
Batch meta not found for buffer 0x7f587413c890
BATCH-NUM = 336**
Batch meta not found for buffer 0x7f59b40d0660
BATCH-NUM = 337**
Batch meta not found for buffer 0x7f58a809c340
BATCH-NUM = 338**
Batch meta not found for buffer 0x7f5a540be460
BATCH-NUM = 339**
Batch meta not found for buffer 0x7f58cc102d60
BATCH-NUM = 340**
Batch meta not found for buffer 0x7f57c00a2b00
BATCH-NUM = 341**
Batch meta not found for buffer 0x7f59b40d9400
BATCH-NUM = 342**
Batch meta not found for buffer 0x7f587413c890
BATCH-NUM = 343**
Batch meta not found for buffer 0x7f582012e570
BATCH-NUM = 344**
Batch meta not found for buffer 0x7f588c0dd450
BATCH-NUM = 345**
Batch meta not found for buffer 0x7f57a8069e00
BATCH-NUM = 346**
Batch meta not found for buffer 0x7f5a18080220
BATCH-NUM = 347**
Batch meta not found for buffer 0x7f5a540f02c0
BATCH-NUM = 348**
Batch meta not found for buffer 0x7f58201070c0
BATCH-NUM = 349**
Batch meta not found for buffer 0x7f588c0e6330ERROR from src_elem2: Could not open resource for reading and writing.
Debug info: gstrtspsrc.c(7469): gst_rtspsrc_retrieve_sdp (): /GstPipeline:pipeline/GstBin:multi_src_bin/GstBin:src_sub_bin2/GstRTSPSrc:src_elem2:
Failed to connect. (Generic error)
** INFO: <reset_source_pipeline:1154>: Resetting source 2

BATCH-NUM = 350**
Batch meta not found for buffer 0x7f5a180f3170**PERF: 5.08 (5.22) 5.08 (4.97) 0.00 (0.00) 5.28 (5.28) 5.28 (5.58) 5.07 (4.88)0.00 (0.00) 4.87 (5.34) 5.28 (5.41) 5.08 (4.87) 5.08 (5.24)5.08 (5.11) 5.28 (5.08) 5.08 (5.12) 0.00 (0.00) 5.28 (5.69)5.08 (4.92) 5.07 (5.40) 5.28 (5.63) 5.08 (4.92) 5.08 (5.24)5.28 (5.58) 5.08 (4.97) 5.08 (5.02) 5.28 (5.03) 5.08 (5.39)5.07 (4.97) 5.08 (5.70) 5.28 (4.97) 5.08 (4.98)

BATCH-NUM = 351**
Batch meta not found for buffer 0x7f58e807d250
BATCH-NUM = 352**
Batch meta not found for buffer 0x7f58200c0900
BATCH-NUM = 353**
Batch meta not found for buffer 0x7f58201143b0
BATCH-NUM = 354**
Batch meta not found for buffer 0x7f5bd00c76b0
BATCH-NUM = 355**
Batch meta not found for buffer 0x7f5a181101b0
BATCH-NUM = 356**
Batch meta not found for buffer 0x7f5964191580
BATCH-NUM = 357**
Batch meta not found for buffer 0x7f596c0a73d0
BATCH-NUM = 358**
Batch meta not found for buffer 0x7f585c070180
BATCH-NUM = 359**
Batch meta not found for buffer 0x7f59b40ca270
BATCH-NUM = 360**
Batch meta not found for buffer 0x7f57d0157990
BATCH-NUM = 361**
Batch meta not found for buffer 0x7f59b407fe50
BATCH-NUM = 362**
Batch meta not found for buffer 0x7f593c0c88b0
BATCH-NUM = 363**
Batch meta not found for buffer 0x7f5968256260
BATCH-NUM = 364**
Batch meta not found for buffer 0x7f58440584c0
BATCH-NUM = 365**
Batch meta not found for buffer 0x7f59f4089030
BATCH-NUM = 366**
Batch meta not found for buffer 0x7f596c0abbd0
BATCH-NUM = 367**
Batch meta not found for buffer 0x7f59b4070220
BATCH-NUM = 368**
Batch meta not found for buffer 0x7f5a5407dbb0
BATCH-NUM = 369**
Batch meta not found for buffer 0x7f584411a7e0
BATCH-NUM = 370**
Batch meta not found for buffer 0x7f57d01469a0
BATCH-NUM = 371**
Batch meta not found for buffer 0x7f58c008c7c0
BATCH-NUM = 372**
Batch meta not found for buffer 0x7f585c0a8c90
BATCH-NUM = 373**
Batch meta not found for buffer 0x7f596c0a2720
BATCH-NUM = 374**
Batch meta not found for buffer 0x7f5984071590
BATCH-NUM = 375**
Batch meta not found for buffer 0x7f58a80c2510
BATCH-NUM = 376**
Batch meta not found for buffer 0x7f5a540cf620
BATCH-NUM = 377**
Batch meta not found for buffer 0x7f57d0074550
BATCH-NUM = 378**
Batch meta not found for buffer 0x7f593c0aaeb0
BATCH-NUM = 379**
Batch meta not found for buffer 0x7f599c093810
BATCH-NUM = 380**
Batch meta not found for buffer 0x7f587411cb20
BATCH-NUM = 381**
Batch meta not found for buffer 0x7f58741362b0
BATCH-NUM = 382**
Batch meta not found for buffer 0x7f57d0078cf0
BATCH-NUM = 383**
Batch meta not found for buffer 0x7f585c117570
BATCH-NUM = 384**
Batch meta not found for buffer 0x7f5900078ab0
BATCH-NUM = 385**
Batch meta not found for buffer 0x7f596c0702e0
BATCH-NUM = 386**
Batch meta not found for buffer 0x7f5820101e40
BATCH-NUM = 387**
Batch meta not found for buffer 0x7f585c09c460
BATCH-NUM = 388**
Batch meta not found for buffer 0x7f5a540e23d0
BATCH-NUM = 389**
Batch meta not found for buffer 0x7f599c0c4140
BATCH-NUM = 390**
Batch meta not found for buffer 0x7f584407be50
BATCH-NUM = 391**
Batch meta not found for buffer 0x7f59cc0a6a30
BATCH-NUM = 392**
Batch meta not found for buffer 0x7f596c0b67d0
BATCH-NUM = 393**
Batch meta not found for buffer 0x7f5a540e2a30
BATCH-NUM = 394**
Batch meta not found for buffer 0x7f5964191cf0
BATCH-NUM = 395**
Batch meta not found for buffer 0x7f596807a6a0
BATCH-NUM = 396**
Batch meta not found for buffer 0x7f593c0d5630
BATCH-NUM = 397**
Batch meta not found for buffer 0x7f588c077c90
BATCH-NUM = 398**
Batch meta not found for buffer 0x7f588c0aa370
BATCH-NUM = 399**
Batch meta not found for buffer 0x7f57a8077600
BATCH-NUM = 400**
Batch meta not found for buffer 0x7f5900090d10
BATCH-NUM = 401**
Batch meta not found for buffer 0x7f57d00f6290
BATCH-NUM = 402**
Batch meta not found for buffer 0x7f5a1809a080
BATCH-NUM = 403**
Batch meta not found for buffer 0x7f58c0097d30
BATCH-NUM = 404**
Batch meta not found for buffer 0x7f584410d2c0
BATCH-NUM = 405**
Batch meta not found for buffer 0x7f57ec0c2d40
BATCH-NUM = 406**
Batch meta not found for buffer 0x7f58e8109660
BATCH-NUM = 407**
Batch meta not found for buffer 0x7f5874078340
BATCH-NUM = 408**
Batch meta not found for buffer 0x7f585c089b10
BATCH-NUM = 409**
Batch meta not found for buffer 0x7f596c09e090
BATCH-NUM = 410**
Batch meta not found for buffer 0x7f5844089890
BATCH-NUM = 411**
Batch meta not found for buffer 0x7f5a70085d80
BATCH-NUM = 412**
Batch meta not found for buffer 0x7f5a240697d0
BATCH-NUM = 413**
Batch meta not found for buffer 0x7f588c09c2e0
BATCH-NUM = 414**
Batch meta not found for buffer 0x7f5874132500

11

With the original Deepstream-app in the nvdsanalytics folder running with the same config that is present above I see a similar delay.

Server Config: 2 CPU 10 core, GPU Nvidia P4, TOP shows deepstream-app taking 200% (which is ok) and GPU util around 80-90% with mem around 7180MB on 7611MB.

Here is the findings:

All cases ran on original deepstream-nvdsanalytics-app with roi enabled on all streams.

Case-1

Tracker (ibnvds_mot_iou.so) - N
nvds-enable - N
no. of streams - 30
Performance: Frames hang, frames skip is observed, not profound however. 2-3 secs hang/skip sometimes

Case-2

Tracker (ibnvds_mot_iou.so) - Y
nvds-enable - Y
no. of streams - 30
Performance: Frames hang, frames skip is observed, very profound however. 5-10 secs hang/skip often.

Case-3

Tracker (libnvds_nvdcf.so) - Y
nvds-enable - Y
no. of streams - 30
Performance: Application fails to start and hangs with memory error. Had to restart then.

Case-4

Tracker (ibnvds_mot_iou.so) - Y
nvds-enable - Y
no. of streams - 20
Performance: Frames hang, frames skip is observed, performance improved however. 3-4 secs hang/skip sometimes.

12

There is no latency information print out. Can you refer to DeepStream SDK FAQ - Intelligent Video Analytics / DeepStream SDK - NVIDIA Developer Forums?

13

root@va03:/opt/nvidia/deepstream/deepstream-5.0/sources/apps/sample_apps/deepstream-nvdsanalytics-test# export NVDS_ENABLE_COMPONENT_LATENCY_MEASUREMENT=1
root@va03:/opt/nvidia/deepstream/deepstream-5.0/sources/apps/sample_apps/deepstream-nvdsanalytics-test# export NVDS_ENABLE_LATENCY_MEASUREMENT=1
root@va03:/opt/nvidia/deepstream/deepstream-5.0/sources/apps/sample_apps/deepstream-nvdsanalytics-test# ./deepstream-app -c ./assets/topconfig30.txt

*** DeepStream: Launched RTSP Streaming at rtsp://localhost:8554/ds-test ***

*** DeepStream: Launched RTSP Streaming at rtsp://localhost:8555/ds-test ***

*** DeepStream: Launched RTSP Streaming at rtsp://localhost:8556/ds-test ***

*** DeepStream: Launched RTSP Streaming at rtsp://localhost:8557/ds-test ***

*** DeepStream: Launched RTSP Streaming at rtsp://localhost:8558/ds-test ***

*** DeepStream: Launched RTSP Streaming at rtsp://localhost:8559/ds-test ***

*** DeepStream: Launched RTSP Streaming at rtsp://localhost:8560/ds-test ***

*** DeepStream: Launched RTSP Streaming at rtsp://localhost:8561/ds-test ***

*** DeepStream: Launched RTSP Streaming at rtsp://localhost:8562/ds-test ***

*** DeepStream: Launched RTSP Streaming at rtsp://localhost:8563/ds-test ***

*** DeepStream: Launched RTSP Streaming at rtsp://localhost:8564/ds-test ***

*** DeepStream: Launched RTSP Streaming at rtsp://localhost:8565/ds-test ***

*** DeepStream: Launched RTSP Streaming at rtsp://localhost:8566/ds-test ***

*** DeepStream: Launched RTSP Streaming at rtsp://localhost:8567/ds-test ***

*** DeepStream: Launched RTSP Streaming at rtsp://localhost:8568/ds-test ***

*** DeepStream: Launched RTSP Streaming at rtsp://localhost:8569/ds-test ***

*** DeepStream: Launched RTSP Streaming at rtsp://localhost:8570/ds-test ***

*** DeepStream: Launched RTSP Streaming at rtsp://localhost:8571/ds-test ***

*** DeepStream: Launched RTSP Streaming at rtsp://localhost:8572/ds-test ***

*** DeepStream: Launched RTSP Streaming at rtsp://localhost:8573/ds-test ***

*** DeepStream: Launched RTSP Streaming at rtsp://localhost:8574/ds-test ***

*** DeepStream: Launched RTSP Streaming at rtsp://localhost:8575/ds-test ***

*** DeepStream: Launched RTSP Streaming at rtsp://localhost:8576/ds-test ***

*** DeepStream: Launched RTSP Streaming at rtsp://localhost:8577/ds-test ***

*** DeepStream: Launched RTSP Streaming at rtsp://localhost:8578/ds-test ***

*** DeepStream: Launched RTSP Streaming at rtsp://localhost:8579/ds-test ***

*** DeepStream: Launched RTSP Streaming at rtsp://localhost:8580/ds-test ***

*** DeepStream: Launched RTSP Streaming at rtsp://localhost:8581/ds-test ***

*** DeepStream: Launched RTSP Streaming at rtsp://localhost:8582/ds-test ***

*** DeepStream: Launched RTSP Streaming at rtsp://localhost:8583/ds-test ***

gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream-5.0/lib/libnvds_mot_iou.so
gstnvtracker: Optional NvMOT_RemoveStreams not implemented
gstnvtracker: Batch processing is OFF
gstnvtracker: Past frame output is OFF
0:00:00.555561239 262 0x55f2401e1e30 INFO nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1715> [UID = 1]: Trying to create engine from model files
INFO: …/nvdsinfer/nvdsinfer_func_utils.cpp:39 [TRT]: Reading Calibration Cache for calibrator: EntropyCalibration2
INFO: …/nvdsinfer/nvdsinfer_func_utils.cpp:39 [TRT]: Generated calibration scales using calibration cache. Make sure that calibration cache has latest scales.
INFO: …/nvdsinfer/nvdsinfer_func_utils.cpp:39 [TRT]: To regenerate calibration cache, please delete the existing one. TensorRT will generate a new calibration cache.
INFO: …/nvdsinfer/nvdsinfer_func_utils.cpp:39 [TRT]: Some tactics do not have sufficient workspace memory to run. Increasing workspace size may increase performance, please check verbose output.
INFO: …/nvdsinfer/nvdsinfer_func_utils.cpp:39 [TRT]: Detected 1 inputs and 2 output network tensors.
0:01:14.313613478 262 0x55f2401e1e30 INFO nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1748> [UID = 1]: serialize cuda engine to file: /opt/nvidia/deepstream/deepstream-5.0/sources/apps/sample_apps/deepstream-nvdsanalytics-test/assets/resnet10_detector.etlt_b30_gpu0_int8.engine successfully
WARNING: …/nvdsinfer/nvdsinfer_func_utils.cpp:36 [TRT]: Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles
INFO: …/nvdsinfer/nvdsinfer_model_builder.cpp:685 [Implicit Engine Info]: layers num: 3
0 INPUT kFLOAT input_1 3x1080x1920
1 OUTPUT kFLOAT output_bbox/BiasAdd 4x68x120
2 OUTPUT kFLOAT output_cov/Sigmoid 1x68x120

0:01:14.327121747 262 0x55f2401e1e30 INFO nvinfer gstnvinfer_impl.cpp:313:notifyLoadModelStatus:<primary_gie> [UID 1]: Load new model:/opt/nvidia/deepstream/deepstream-5.0/sources/apps/sample_apps/deepstream-nvdsanalytics-test/assets/pgie-config.txt sucessfully

Runtime commands:
h: Print this help
q: Quit

    p: Pause
    r: Resume

**PERF: FPS 0 (Avg) FPS 1 (Avg) FPS 2 (Avg) FPS 3 (Avg) FPS 4 (Avg) FPS 5 (Avg) FPS 6 (Avg) FPS 7 (Avg) FPS 8 (Avg) FPS 9 (Avg) FPS 10 (Avg) FPS 11 (Avg) FPS 12 (Avg) FPS 13 (Avg) FPS 14 (Avg) FPS 15 (Avg) FPS 16 (Avg) FPS 17 (Avg) FPS 18 (Avg) FPS 19 (Avg) FPS 20 (Avg) FPS 21 (Avg) FPS 22 (Avg) FPS 23 (Avg) FPS 24 (Avg) FPS 25 (Avg) FPS 26 (Avg) FPS 27 (Avg) FPS 28 (Avg) FPS 29 (Avg)
**PERF: 0.00 (0.00) 0.00 (0.00) 0.00 (0.00) 0.00 (0.00) 0.00 (0.00) 0.00 (0.00) 0.00 (0.00) 0.00 (0.00) 0.00 (0.00) 0.00 (0.00) 0.00 (0.00) 0.00 (0.00) 0.00 (0.00) 0.00 (0.00) 0.00 (0.00) 0.00 (0.00) 0.00 (0.00) 0.00 (0.00) 0.00 (0.00) 0.00 (0.00) 0.00 (0.00) 0.00 (0.00) 0.00 (0.00) 0.00 (0.00) 0.00 (0.00) 0.00 (0.00) 0.00 (0.00) 0.00 (0.00) 0.00 (0.00) 0.00 (0.00)
** INFO: <bus_callback:181>: Pipeline ready

ERROR from src_elem6: Unauthorized
Debug info: gstrtspsrc.c(6116): gst_rtspsrc_send (): /GstPipeline:pipeline/GstBin:multi_src_bin/GstBin:src_sub_bin6/GstRTSPSrc:src_elem6:
Unauthorized (401)
** INFO: <reset_source_pipeline:1154>: Resetting source 6
ERROR from src_elem14: Unauthorized
Debug info: gstrtspsrc.c(6116): gst_rtspsrc_send (): /GstPipeline:pipeline/GstBin:multi_src_bin/GstBin:src_sub_bin14/GstRTSPSrc:src_elem14:
Unauthorized (401)
** INFO: <reset_source_pipeline:1154>: Resetting source 14
ERROR from src_elem6: Unauthorized
Debug info: gstrtspsrc.c(6116): gst_rtspsrc_send (): /GstPipeline:pipeline/GstBin:multi_src_bin/GstBin:src_sub_bin6/GstRTSPSrc:src_elem6:
Unauthorized (401)
ERROR from src_elem14: Unauthorized
Debug info: gstrtspsrc.c(6116): gst_rtspsrc_send (): /GstPipeline:pipeline/GstBin:multi_src_bin/GstBin:src_sub_bin14/GstRTSPSrc:src_elem14:
Unauthorized (401)
IOU Tracker Init with threshold 0.100000
IOU Tracker Init with threshold 0.100000
IOU Tracker Init with threshold 0.100000
IOU Tracker Init with threshold 0.100000
IOU Tracker Init with threshold 0.100000
IOU Tracker Init with threshold 0.100000
IOU Tracker Init with threshold 0.100000
Alerting Set after 200 frames
IOU Tracker Init with threshold 0.100000
IOU Tracker Init with threshold 0.100000
IOU Tracker Init with threshold 0.100000
IOU Tracker Init with threshold 0.100000
IOU Tracker Init with threshold 0.100000
IOU Tracker Init with threshold 0.100000
IOU Tracker Init with threshold 0.100000
IOU Tracker Init with threshold 0.100000
IOU Tracker Init with threshold 0.100000
IOU Tracker Init with threshold 0.100000
IOU Tracker Init with threshold 0.100000
IOU Tracker Init with threshold 0.100000
IOU Tracker Init with threshold 0.100000
IOU Tracker Init with threshold 0.100000
IOU Tracker Init with threshold 0.100000

BATCH-NUM = 0**
Batch meta not found for buffer 0x7fcddc0ba330
BATCH-NUM = 1**
Batch meta not found for buffer 0x7fcc1c1831f0
BATCH-NUM = 2**
Batch meta not found for buffer 0x7fcd7c078340
BATCH-NUM = 3**
Batch meta not found for buffer 0x7fccdc07e380
BATCH-NUM = 0**
Comp name = nvv4l2decoder19 in_system_timestamp = 1609917746673.521973 out_system_timestamp = 1609917747694.389893 component latency= 1020.867920
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 0 in_system_timestamp = 1609917747694.603027 out_system_timestamp = 1609917747717.592041 component_latency = 22.989014
Comp name = primary_gie in_system_timestamp = 1609917747717.650879 out_system_timestamp = 1609917747717.752930 component latency= 0.102051
Comp name = tracking_tracker in_system_timestamp = 1609917747717.777100 out_system_timestamp = 1609917747718.412109 component latency= 0.635010
Comp name = dsanalytics0 in_system_timestamp = 1609917747718.465088 out_system_timestamp = 1609917747718.739014 component latency= 0.273926
Comp name = demuxer in_system_timestamp = 1609917747718.762939 out_system_timestamp = 1609917747720.201904 component latency= 1.438965
Comp name = osd_conv in_system_timestamp = 1609917750091.076904 out_system_timestamp = 1609917750093.023926 component latency= 1.947021
Comp name = nvosd0 in_system_timestamp = 1609917750093.191895 out_system_timestamp = 1609917750099.647949 component latency= 6.456055
Source id = 0 Frame_num = 0 Frame latency = 3426.184082 (ms)

BATCH-NUM = 1**
Comp name = nvv4l2decoder19 in_system_timestamp = 1609917747662.115967 out_system_timestamp = 1609917747706.394043 component latency= 44.278076
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 1 in_system_timestamp = 1609917747706.454102 out_system_timestamp = 1609917747777.795898 component_latency = 71.341797
Comp name = primary_gie in_system_timestamp = 1609917747777.840088 out_system_timestamp = 1609917747777.904053 component latency= 0.063965
Comp name = tracking_tracker in_system_timestamp = 1609917747777.918945 out_system_timestamp = 1609917747778.257080 component latency= 0.338135
Comp name = dsanalytics0 in_system_timestamp = 1609917747778.297119 out_system_timestamp = 1609917747778.445068 component latency= 0.147949
Comp name = demuxer in_system_timestamp = 1609917747778.462891 out_system_timestamp = 1609917747778.566895 component latency= 0.104004
Comp name = osd_conv in_system_timestamp = 1609917750093.408936 out_system_timestamp = 1609917750095.134033 component latency= 1.725098
Comp name = nvosd0 in_system_timestamp = 1609917750099.985107 out_system_timestamp = 1609917750109.362061 component latency= 9.376953
Source id = 0 Frame_num = 1 Frame latency = 2447.281982 (ms)

BATCH-NUM = 2**
Comp name = nvv4l2decoder19 in_system_timestamp = 1609917747662.195068 out_system_timestamp = 1609917747742.929932 component latency= 80.734863
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 2 in_system_timestamp = 1609917747742.995117 out_system_timestamp = 1609917747838.042969 component_latency = 95.047852
Comp name = primary_gie in_system_timestamp = 1609917747838.095947 out_system_timestamp = 1609917747898.062988 component latency= 59.967041
Comp name = tracking_tracker in_system_timestamp = 1609917747898.077881 out_system_timestamp = 1609917747898.617920 component latency= 0.540039
Comp name = dsanalytics0 in_system_timestamp = 1609917747898.667969 out_system_timestamp = 1609917747898.949951 component latency= 0.281982
Comp name = demuxer in_system_timestamp = 1609917747898.977051 out_system_timestamp = 1609917747899.079102 component latency= 0.102051
Comp name = osd_conv in_system_timestamp = 1609917750095.345947 out_system_timestamp = 1609917750099.846924 component latency= 4.500977
Comp name = nvosd0 in_system_timestamp = 1609917750109.589111 out_system_timestamp = 1609917750120.373047 component latency= 10.783936
Source id = 0 Frame_num = 2 Frame latency = 2458.210938 (ms)

BATCH-NUM = 3**
Comp name = nvv4l2decoder19 in_system_timestamp = 1609917747671.285889 out_system_timestamp = 1609917747788.690918 component latency= 117.405029
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 3 in_system_timestamp = 1609917747788.749023 out_system_timestamp = 1609917747898.499023 component_latency = 109.750000
Comp name = primary_gie in_system_timestamp = 1609917747899.841064 out_system_timestamp = 1609917747899.886963 component latency= 0.045898
Comp name = tracking_tracker in_system_timestamp = 1609917747899.916016 out_system_timestamp = 1609917747900.510986 component latency= 0.594971
Comp name = dsanalytics0 in_system_timestamp = 1609917747907.808105 out_system_timestamp = 1609917747908.052979 component latency= 0.244873
Comp name = demuxer in_system_timestamp = 1609917747908.075928 out_system_timestamp = 1609917747908.185059 component latency= 0.109131
Comp name = osd_conv in_system_timestamp = 1609917750100.205078 out_system_timestamp = 1609917750111.209961 component latency= 11.004883
Comp name = nvosd0 in_system_timestamp = 1609917750120.599121 out_system_timestamp = 1609917750123.888916 component latency= 3.289795
Source id = 0 Frame_num = 3 Frame latency = 2452.626221 (ms)

BATCH-NUM = 4**
Comp name = nvv4l2decoder19 in_system_timestamp = 1609917747674.221924 out_system_timestamp = 1609917747844.929932 component latency= 170.708008
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 4 in_system_timestamp = 1609917747844.989990 out_system_timestamp = 1609917747958.750977 component_latency = 113.760986
Comp name = primary_gie in_system_timestamp = 1609917747958.886963 out_system_timestamp = 1609917747959.011963 component latency= 0.125000
Comp name = tracking_tracker in_system_timestamp = 1609917747959.034912 out_system_timestamp = 1609917747959.477051 component latency= 0.442139
Comp name = dsanalytics0 in_system_timestamp = 1609917747959.518066 out_system_timestamp = 1609917747959.770996 component latency= 0.252930
Comp name = demuxer in_system_timestamp = 1609917747959.803955 out_system_timestamp = 1609917747959.946045 component latency= 0.142090
Comp name = osd_conv in_system_timestamp = 1609917750113.194092 out_system_timestamp = 1609917750121.354004 component latency= 8.159912
Comp name = nvosd0 in_system_timestamp = 1609917750124.051025 out_system_timestamp = 1609917750127.382080 component latency= 3.331055
Source id = 0 Frame_num = 4 Frame latency = 2453.177002 (ms)

BATCH-NUM = 4**
Batch meta not found for buffer 0x7fcc20068d70
BATCH-NUM = 5**
Batch meta not found for buffer 0x7fcddc0e1c90
BATCH-NUM = 6**
Batch meta not found for buffer 0x7fcbdc06d8d0
BATCH-NUM = 7**
Batch meta not found for buffer 0x7fcd100871b0
BATCH-NUM = 8**
Batch meta not found for buffer 0x7fcd98067e40
BATCH-NUM = 9**
Batch meta not found for buffer 0x7fcd3c084120
BATCH-NUM = 10**
Batch meta not found for buffer 0x7fcd100eb1d0
BATCH-NUM = 11**
Batch meta not found for buffer 0x7fcd100e78f0
BATCH-NUM = 12**
Batch meta not found for buffer 0x7fcbf80ca800
BATCH-NUM = 13**
Batch meta not found for buffer 0x7fcddc07ea00IOU Tracker Init with threshold 0.100000
IOU Tracker Init with threshold 0.100000
IOU Tracker Init with threshold 0.100000
IOU Tracker Init with threshold 0.100000
IOU Tracker Init with threshold 0.100000

BATCH-NUM = 5**
Comp name = nvv4l2decoder19 in_system_timestamp = 1609917747701.020996 out_system_timestamp = 1609917750094.988037 component latency= 2393.967041
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 5 in_system_timestamp = 1609917750095.050049 out_system_timestamp = 1609917750428.993896 component_latency = 333.943848
Comp name = primary_gie in_system_timestamp = 1609917750429.121094 out_system_timestamp = 1609917750429.275879 component latency= 0.154785
Comp name = tracking_tracker in_system_timestamp = 1609917750429.309082 out_system_timestamp = 1609917750431.073975 component latency= 1.764893
Comp name = dsanalytics0 in_system_timestamp = 1609917750431.123047 out_system_timestamp = 1609917750431.875977 component latency= 0.752930
Comp name = demuxer in_system_timestamp = 1609917750431.923096 out_system_timestamp = 1609917750432.012939 component latency= 0.089844
Comp name = osd_conv in_system_timestamp = 1609917750432.150879 out_system_timestamp = 1609917750452.225098 component latency= 20.074219
Comp name = nvosd0 in_system_timestamp = 1609917750452.412109 out_system_timestamp = 1609917750517.302979 component latency= 64.890869
Source id = 0 Frame_num = 5 Frame latency = 2816.333984 (ms)

BATCH-NUM = 14**
Batch meta not found for buffer 0x7fcb4c0b5e80
BATCH-NUM = 15**
Batch meta not found for buffer 0x7fcbdc0636d0
BATCH-NUM = 16**
Batch meta not found for buffer 0x7fcd1008f170
BATCH-NUM = 17**
Batch meta not found for buffer 0x7fcb940d4990
BATCH-NUM = 18**
Batch meta not found for buffer 0x7fcd3c0eb1a0
BATCH-NUM = 19**
Batch meta not found for buffer 0x7fcddc07f010
BATCH-NUM = 20**
Batch meta not found for buffer 0x7fcb4c09da40
BATCH-NUM = 21**
Batch meta not found for buffer 0x7fcd1c072a60
BATCH-NUM = 22**
Batch meta not found for buffer 0x7fccdc27b5b0
BATCH-NUM = 23**
Batch meta not found for buffer 0x7fcb70130170
BATCH-NUM = 24**
Batch meta not found for buffer 0x7fcb7c082910
BATCH-NUM = 25**
Batch meta not found for buffer 0x7fcbf80bbaa0
BATCH-NUM = 6**
Comp name = nvv4l2decoder19 in_system_timestamp = 1609917747742.655029 out_system_timestamp = 1609917750127.500977 component latency= 2384.845947
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 6 in_system_timestamp = 1609917750127.610107 out_system_timestamp = 1609917750489.757080 component_latency = 362.146973
Comp name = primary_gie in_system_timestamp = 1609917750489.822998 out_system_timestamp = 1609917750859.125977 component latency= 369.302979
Comp name = tracking_tracker in_system_timestamp = 1609917750859.162109 out_system_timestamp = 1609917750860.435059 component latency= 1.272949
Comp name = dsanalytics0 in_system_timestamp = 1609917750860.514893 out_system_timestamp = 1609917750861.246094 component latency= 0.731201
Comp name = demuxer in_system_timestamp = 1609917750861.297119 out_system_timestamp = 1609917750861.381104 component latency= 0.083984
Comp name = osd_conv in_system_timestamp = 1609917750861.518066 out_system_timestamp = 1609917750867.666016 component latency= 6.147949
Comp name = nvosd0 in_system_timestamp = 1609917750868.000000 out_system_timestamp = 1609917750903.022949 component latency= 35.022949
Source id = 0 Frame_num = 6 Frame latency = 3160.417969 (ms)

BATCH-NUM = 7**
Comp name = nvv4l2decoder19 in_system_timestamp = 1609917747762.872070 out_system_timestamp = 1609917750128.175049 component latency= 2365.302979
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 7 in_system_timestamp = 1609917750128.220947 out_system_timestamp = 1609917750550.370117 component_latency = 422.149170
Comp name = primary_gie in_system_timestamp = 1609917750639.070068 out_system_timestamp = 1609917750859.809082 component latency= 220.739014
Comp name = tracking_tracker in_system_timestamp = 1609917750859.836914 out_system_timestamp = 1609917750861.200928 component latency= 1.364014
Comp name = dsanalytics0 in_system_timestamp = 1609917750863.207031 out_system_timestamp = 1609917750863.766113 component latency= 0.559082
Comp name = demuxer in_system_timestamp = 1609917750863.823975 out_system_timestamp = 1609917750863.894043 component latency= 0.070068
Comp name = osd_conv in_system_timestamp = 1609917750867.982910 out_system_timestamp = 1609917750881.125000 component latency= 13.142090
Comp name = nvosd0 in_system_timestamp = 1609917750903.333984 out_system_timestamp = 1609917751087.846924 component latency= 184.512939
Source id = 0 Frame_num = 7 Frame latency = 3325.024902 (ms)
**PERF: 3.24 (3.15) 2.66 (2.60) 0.00 (0.00) 4.52 (4.32) 5.03 (4.80) 5.53 (5.28) 0.00 (0.00) 0.47 (0.46) 8.80 (7.75) 3.39 (3.31) 3.24 (2.96) 7.35 (6.45) 3.01 (2.77) 5.03 (4.80) 0.00 (0.00) 8.80 (7.75) 3.14 (3.07) 3.83 (3.73) 9.57 (8.49) 4.28 (3.88) 5.53 (5.28) 7.55 (3.27) 5.53 (5.28) 8.80 (7.75) 4.06 (3.94) 4.47 (3.42) 4.72 (4.23) 8.80 (7.75) 3.71 (3.62) 4.94 (4.74)

BATCH-NUM = 26**
Batch meta not found for buffer 0x7fcd94355a10
BATCH-NUM = 27**
Batch meta not found for buffer 0x7fcb4c069880
BATCH-NUM = 28**
Batch meta not found for buffer 0x7fcc100857e0
BATCH-NUM = 29**
Batch meta not found for buffer 0x7fcd3c069bd0
BATCH-NUM = 30**
Batch meta not found for buffer 0x7fcb4c056710
BATCH-NUM = 31**
Batch meta not found for buffer 0x7fcdec103070
BATCH-NUM = 32**
Batch meta not found for buffer 0x7fcb700c09d0
BATCH-NUM = 33**
Batch meta not found for buffer 0x7fcd4c078800
BATCH-NUM = 8**
Comp name = nvv4l2decoder19 in_system_timestamp = 1609917747839.372070 out_system_timestamp = 1609917750128.548096 component latency= 2289.176025
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 8 in_system_timestamp = 1609917750128.572998 out_system_timestamp = 1609917750610.851074 component_latency = 482.278076
Comp name = primary_gie in_system_timestamp = 1609917750639.267090 out_system_timestamp = 1609917750860.327881 component latency= 221.060791
Comp name = tracking_tracker in_system_timestamp = 1609917750860.345947 out_system_timestamp = 1609917750861.215088 component latency= 0.869141
Comp name = dsanalytics0 in_system_timestamp = 1609917750865.343994 out_system_timestamp = 1609917750865.733887 component latency= 0.389893
Comp name = demuxer in_system_timestamp = 1609917750865.771973 out_system_timestamp = 1609917750865.991943 component latency= 0.219971
Comp name = osd_conv in_system_timestamp = 1609917751360.977051 out_system_timestamp = 1609917751382.937012 component latency= 21.959961
Comp name = nvosd0 in_system_timestamp = 1609917751383.097900 out_system_timestamp = 1609917751407.334961 component latency= 24.237061
Source id = 0 Frame_num = 8 Frame latency = 3568.004883 (ms)

BATCH-NUM = 34**
Batch meta not found for buffer 0x7fcbf813a1c0
BATCH-NUM = 35**
Batch meta not found for buffer 0x7fcd7c06fb80
BATCH-NUM = 36**
Batch meta not found for buffer 0x7fcbdc069410
BATCH-NUM = 37**
Batch meta not found for buffer 0x7fccf807d150
BATCH-NUM = 38**
Batch meta not found for buffer 0x7fcc7006bc50
BATCH-NUM = 39**
Batch meta not found for buffer 0x7fccc40787b0
BATCH-NUM = 40**
Batch meta not found for buffer 0x7fccf8079950
BATCH-NUM = 41**
Batch meta not found for buffer 0x7fcb7c0751b0
BATCH-NUM = 42**
Batch meta not found for buffer 0x7fccf8079620
BATCH-NUM = 43**
Batch meta not found for buffer 0x7fcd980729f0
BATCH-NUM = 44**
Batch meta not found for buffer 0x7fcc200cc8a0
BATCH-NUM = 45**
Batch meta not found for buffer 0x7fcbdc107290
BATCH-NUM = 46**
Batch meta not found for buffer 0x7fcd3c0c79e0
BATCH-NUM = 47**
Batch meta not found for buffer 0x7fccc40c33e0
BATCH-NUM = 48**
Batch meta not found for buffer 0x7fcd98070d10
BATCH-NUM = 49**
Batch meta not found for buffer 0x7fcdec07f220
BATCH-NUM = 9**
Comp name = nvv4l2decoder19 in_system_timestamp = 1609917747846.094971 out_system_timestamp = 1609917750130.898926 component latency= 2284.803955
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 9 in_system_timestamp = 1609917750131.020996 out_system_timestamp = 1609917750671.472900 component_latency = 540.451904
Comp name = primary_gie in_system_timestamp = 1609917750671.531006 out_system_timestamp = 1609917751107.552979 component latency= 436.021973
Comp name = tracking_tracker in_system_timestamp = 1609917751107.589111 out_system_timestamp = 1609917751109.280029 component latency= 1.690918
Comp name = dsanalytics0 in_system_timestamp = 1609917751109.326904 out_system_timestamp = 1609917751109.951904 component latency= 0.625000
Comp name = demuxer in_system_timestamp = 1609917751110.007080 out_system_timestamp = 1609917751110.083984 component latency= 0.076904
Comp name = osd_conv in_system_timestamp = 1609917751488.172119 out_system_timestamp = 1609917751503.897949 component latency= 15.725830
Comp name = nvosd0 in_system_timestamp = 1609917751504.086914 out_system_timestamp = 1609917751517.459961 component latency= 13.373047
Source id = 0 Frame_num = 9 Frame latency = 3671.416992 (ms)

BATCH-NUM = 50**
Batch meta not found for buffer 0x7fcb4c0d53f0
BATCH-NUM = 51**
Batch meta not found for buffer 0x7fcc1007d090
BATCH-NUM = 52**
Batch meta not found for buffer 0x7fcbc408d390
BATCH-NUM = 53**
Batch meta not found for buffer 0x7fcb400ef620
BATCH-NUM = 54**
Batch meta not found for buffer 0x7fcc10061d10
BATCH-NUM = 10**
Comp name = nvv4l2decoder19 in_system_timestamp = 1609917750125.593994 out_system_timestamp = 1609917750496.285889 component latency= 370.691895
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 10 in_system_timestamp = 1609917750496.386963 out_system_timestamp = 1609917750923.930908 component_latency = 427.543945
Comp name = primary_gie in_system_timestamp = 1609917750924.061035 out_system_timestamp = 1609917751108.143066 component latency= 184.082031
Comp name = tracking_tracker in_system_timestamp = 1609917751108.169922 out_system_timestamp = 1609917751109.331055 component latency= 1.161133
Comp name = dsanalytics0 in_system_timestamp = 1609917751111.485107 out_system_timestamp = 1609917751111.891113 component latency= 0.406006
Comp name = demuxer in_system_timestamp = 1609917751111.947998 out_system_timestamp = 1609917751112.022949 component latency= 0.074951
Comp name = osd_conv in_system_timestamp = 1609917751678.229004 out_system_timestamp = 1609917751682.959961 component latency= 4.730957
Comp name = nvosd0 in_system_timestamp = 1609917751683.139893 out_system_timestamp = 1609917751701.711914 component latency= 18.572021
Source id = 0 Frame_num = 10 Frame latency = 1576.170898 (ms)

BATCH-NUM = 55**
Batch meta not found for buffer 0x7fcc2008bc20
BATCH-NUM = 56**
Batch meta not found for buffer 0x7fcc1006e340
BATCH-NUM = 11**
Comp name = nvv4l2decoder19 in_system_timestamp = 1609917750127.669922 out_system_timestamp = 1609917750891.996094 component latency= 764.326172
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 11 in_system_timestamp = 1609917750892.168945 out_system_timestamp = 1609917750984.525879 component_latency = 92.356934
Comp name = primary_gie in_system_timestamp = 1609917750984.629883 out_system_timestamp = 1609917751108.820068 component latency= 124.190186
Comp name = tracking_tracker in_system_timestamp = 1609917751108.854980 out_system_timestamp = 1609917751109.460938 component latency= 0.605957
Comp name = dsanalytics0 in_system_timestamp = 1609917751113.412109 out_system_timestamp = 1609917751113.712891 component latency= 0.300781
Comp name = demuxer in_system_timestamp = 1609917751113.742920 out_system_timestamp = 1609917751113.789062 component latency= 0.046143
Comp name = osd_conv in_system_timestamp = 1609917751867.625977 out_system_timestamp = 1609917751889.282959 component latency= 21.656982
Comp name = nvosd0 in_system_timestamp = 1609917751889.465088 out_system_timestamp = 1609917751921.023926 component latency= 31.558838
Source id = 0 Frame_num = 11 Frame latency = 1793.402100 (ms)

BATCH-NUM = 57**
Batch meta not found for buffer 0x7fccf80841c0
BATCH-NUM = 58**
Batch meta not found for buffer 0x7fcb7c07f600
BATCH-NUM = 12**
Comp name = nvv4l2decoder19 in_system_timestamp = 1609917750128.373047 out_system_timestamp = 1609917750968.844971 component latency= 840.471924
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 12 in_system_timestamp = 1609917750968.971924 out_system_timestamp = 1609917751044.935059 component_latency = 75.963135
Comp name = primary_gie in_system_timestamp = 1609917751045.031006 out_system_timestamp = 1609917751280.352051 component latency= 235.321045
Comp name = tracking_tracker in_system_timestamp = 1609917751280.377930 out_system_timestamp = 1609917751281.404053 component latency= 1.026123
Comp name = dsanalytics0 in_system_timestamp = 1609917751281.437988 out_system_timestamp = 1609917751281.908936 component latency= 0.470947
Comp name = demuxer in_system_timestamp = 1609917751281.937012 out_system_timestamp = 1609917751282.020996 component latency= 0.083984
Comp name = osd_conv in_system_timestamp = 1609917751999.677979 out_system_timestamp = 1609917752076.033936 component latency= 76.355957
Comp name = nvosd0 in_system_timestamp = 1609917752076.175049 out_system_timestamp = 1609917752107.439941 component latency= 31.264893
Source id = 0 Frame_num = 12 Frame latency = 1979.091064 (ms)

BATCH-NUM = 13**
Comp name = nvv4l2decoder19 in_system_timestamp = 1609917750128.636963 out_system_timestamp = 1609917751406.641113 component latency= 1278.004150
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 13 in_system_timestamp = 1609917751406.802979 out_system_timestamp = 1609917751413.513916 component_latency = 6.710938
Comp name = primary_gie in_system_timestamp = 1609917751413.587891 out_system_timestamp = 1609917751413.688965 component latency= 0.101074
Comp name = tracking_tracker in_system_timestamp = 1609917751413.718018 out_system_timestamp = 1609917751414.198975 component latency= 0.480957
Comp name = dsanalytics0 in_system_timestamp = 1609917751414.221924 out_system_timestamp = 1609917751414.349121 component latency= 0.127197
Comp name = demuxer in_system_timestamp = 1609917751414.361084 out_system_timestamp = 1609917751415.108887 component latency= 0.747803
Comp name = osd_conv in_system_timestamp = 1609917752259.351074 out_system_timestamp = 1609917752260.587891 component latency= 1.236816
Comp name = nvosd0 in_system_timestamp = 1609917752260.761963 out_system_timestamp = 1609917752269.358887 component latency= 8.596924
Source id = 0 Frame_num = 13 Frame latency = 2140.766113 (ms)

BATCH-NUM = 59**
Batch meta not found for buffer 0x7fcd3c0ef490
BATCH-NUM = 14**
Comp name = nvv4l2decoder19 in_system_timestamp = 1609917750142.106934 out_system_timestamp = 1609917751510.584961 component latency= 1368.478027
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 14 in_system_timestamp = 1609917751510.708984 out_system_timestamp = 1609917751534.309082 component_latency = 23.600098
Comp name = primary_gie in_system_timestamp = 1609917751534.556885 out_system_timestamp = 1609917751621.252930 component latency= 86.696045
Comp name = tracking_tracker in_system_timestamp = 1609917751621.268066 out_system_timestamp = 1609917751622.022949 component latency= 0.754883
Comp name = dsanalytics0 in_system_timestamp = 1609917751624.094971 out_system_timestamp = 1609917751624.365967 component latency= 0.270996
Comp name = demuxer in_system_timestamp = 1609917751624.392090 out_system_timestamp = 1609917751625.438965 component latency= 1.046875
Comp name = osd_conv in_system_timestamp = 1609917752358.011963 out_system_timestamp = 1609917752360.635010 component latency= 2.623047
Comp name = nvosd0 in_system_timestamp = 1609917752360.782959 out_system_timestamp = 1609917752364.558105 component latency= 3.775146
Source id = 0 Frame_num = 14 Frame latency = 2222.474121 (ms)

BATCH-NUM = 60**
Batch meta not found for buffer 0x7fcbdc137cf0
BATCH-NUM = 15**
Comp name = nvv4l2decoder19 in_system_timestamp = 1609917750496.520996 out_system_timestamp = 1609917751691.385010 component latency= 1194.864014
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 15 in_system_timestamp = 1609917751691.551025 out_system_timestamp = 1609917751747.686035 component_latency = 56.135010
Comp name = primary_gie in_system_timestamp = 1609917751747.747070 out_system_timestamp = 1609917751840.955078 component latency= 93.208008
Comp name = tracking_tracker in_system_timestamp = 1609917751840.966064 out_system_timestamp = 1609917751841.376953 component latency= 0.410889
Comp name = dsanalytics0 in_system_timestamp = 1609917751842.530029 out_system_timestamp = 1609917751842.593994 component latency= 0.063965
Comp name = demuxer in_system_timestamp = 1609917751842.601074 out_system_timestamp = 1609917751842.644043 component latency= 0.042969
Comp name = osd_conv in_system_timestamp = 1609917752464.437988 out_system_timestamp = 1609917752465.812988 component latency= 1.375000
Comp name = nvosd0 in_system_timestamp = 1609917752465.982910 out_system_timestamp = 1609917752479.345947 component latency= 13.363037
Source id = 0 Frame_num = 15 Frame latency = 1982.853027 (ms)

BATCH-NUM = 61**
Batch meta not found for buffer 0x7fcc3c122e50
BATCH-NUM = 16**
Comp name = nvv4l2decoder19 in_system_timestamp = 1609917750892.716064 out_system_timestamp = 1609917751911.808105 component latency= 1019.092041
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 16 in_system_timestamp = 1609917751911.991943 out_system_timestamp = 1609917751959.333984 component_latency = 47.342041
Comp name = primary_gie in_system_timestamp = 1609917751959.404053 out_system_timestamp = 1609917752143.811035 component latency= 184.406982
Comp name = tracking_tracker in_system_timestamp = 1609917752143.841064 out_system_timestamp = 1609917752144.724121 component latency= 0.883057
Comp name = dsanalytics0 in_system_timestamp = 1609917752144.945068 out_system_timestamp = 1609917752145.663086 component latency= 0.718018
Comp name = demuxer in_system_timestamp = 1609917752145.702881 out_system_timestamp = 1609917752346.802979 component latency= 201.100098
Comp name = osd_conv in_system_timestamp = 1609917752571.272949 out_system_timestamp = 1609917752571.870117 component latency= 0.597168
Comp name = nvosd0 in_system_timestamp = 1609917752572.000977 out_system_timestamp = 1609917752576.441895 component latency= 4.440918
Source id = 0 Frame_num = 16 Frame latency = 1683.766846 (ms)

BATCH-NUM = 62**
Batch meta not found for buffer 0x7fcc480826c0
BATCH-NUM = 63**
Batch meta not found for buffer 0x7fcd4c083bb0
BATCH-NUM = 64**
Batch meta not found for buffer 0x7fcb940d4cc0
BATCH-NUM = 17**
Comp name = nvv4l2decoder19 in_system_timestamp = 1609917750968.969971 out_system_timestamp = 1609917752146.974121 component latency= 1178.004150
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 17 in_system_timestamp = 1609917752147.102051 out_system_timestamp = 1609917752148.367920 component_latency = 1.265869
Comp name = primary_gie in_system_timestamp = 1609917752148.414062 out_system_timestamp = 1609917752195.197998 component latency= 46.783936
Comp name = tracking_tracker in_system_timestamp = 1609917752195.218994 out_system_timestamp = 1609917752195.517090 component latency= 0.298096
Comp name = dsanalytics0 in_system_timestamp = 1609917752379.514893 out_system_timestamp = 1609917752379.633057 component latency= 0.118164
Comp name = demuxer in_system_timestamp = 1609917752379.648926 out_system_timestamp = 1609917752379.896973 component latency= 0.248047
Comp name = osd_conv in_system_timestamp = 1609917752852.683105 out_system_timestamp = 1609917752859.122070 component latency= 6.438965
Comp name = nvosd0 in_system_timestamp = 1609917752859.299072 out_system_timestamp = 1609917752872.873047 component latency= 13.573975
Source id = 0 Frame_num = 17 Frame latency = 1903.944092 (ms)

BATCH-NUM = 18**
Comp name = nvv4l2decoder19 in_system_timestamp = 1609917751510.802979 out_system_timestamp = 1609917752263.032959 component latency= 752.229980
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 18 in_system_timestamp = 1609917752263.148926 out_system_timestamp = 1609917752437.061035 component_latency = 173.912109
Comp name = primary_gie in_system_timestamp = 1609917752437.138916 out_system_timestamp = 1609917752437.229980 component latency= 0.091064
Comp name = tracking_tracker in_system_timestamp = 1609917752437.273926 out_system_timestamp = 1609917752438.757080 component latency= 1.483154
Comp name = dsanalytics0 in_system_timestamp = 1609917752438.806885 out_system_timestamp = 1609917752439.593018 component latency= 0.786133
Comp name = demuxer in_system_timestamp = 1609917752439.635986 out_system_timestamp = 1609917752439.709961 component latency= 0.073975
Comp name = osd_conv in_system_timestamp = 1609917753137.178955 out_system_timestamp = 1609917753146.693115 component latency= 9.514160
Comp name = nvosd0 in_system_timestamp = 1609917753146.850098 out_system_timestamp = 1609917753168.073975 component latency= 21.223877
Source id = 0 Frame_num = 18 Frame latency = 1657.317139 (ms)

BATCH-NUM = 65**
Batch meta not found for buffer 0x7fcc3c0b80d0
BATCH-NUM = 66**
Batch meta not found for buffer 0x7fcb7c0992e0
BATCH-NUM = 67**
Batch meta not found for buffer 0x7fcb940c9ad0
BATCH-NUM = 68**
Batch meta not found for buffer 0x7fcb4c0563e0
BATCH-NUM = 19**
Comp name = nvv4l2decoder19 in_system_timestamp = 1609917751510.896973 out_system_timestamp = 1609917752363.552979 component latency= 852.656006
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 19 in_system_timestamp = 1609917752363.689941 out_system_timestamp = 1609917752498.132080 component_latency = 134.442139
Comp name = primary_gie in_system_timestamp = 1609917752498.205078 out_system_timestamp = 1609917752498.313965 component latency= 0.108887
Comp name = tracking_tracker in_system_timestamp = 1609917752498.356934 out_system_timestamp = 1609917752499.549072 component latency= 1.192139
Comp name = dsanalytics0 in_system_timestamp = 1609917752499.594971 out_system_timestamp = 1609917752500.354004 component latency= 0.759033
Comp name = demuxer in_system_timestamp = 1609917752500.415039 out_system_timestamp = 1609917752500.517090 component latency= 0.102051
Comp name = osd_conv in_system_timestamp = 1609917753344.157959 out_system_timestamp = 1609917753344.597900 component latency= 0.439941
Comp name = nvosd0 in_system_timestamp = 1609917753344.800049 out_system_timestamp = 1609917753351.562988 component latency= 6.762939
Source id = 0 Frame_num = 19 Frame latency = 1840.708984 (ms)

BATCH-NUM = 69**
Batch meta not found for buffer 0x7fcc10074260
BATCH-NUM = 70**
Batch meta not found for buffer 0x7fcdec0f88c0
BATCH-NUM = 71**
Batch meta not found for buffer 0x7fcc200bde00
BATCH-NUM = 20**
Comp name = nvv4l2decoder19 in_system_timestamp = 1609917751691.493896 out_system_timestamp = 1609917752473.782959 component latency= 782.289062
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 20 in_system_timestamp = 1609917752473.941895 out_system_timestamp = 1609917752558.846924 component_latency = 84.905029
Comp name = primary_gie in_system_timestamp = 1609917752558.906006 out_system_timestamp = 1609917752787.049072 component latency= 228.143066
Comp name = tracking_tracker in_system_timestamp = 1609917752787.082031 out_system_timestamp = 1609917752788.837891 component latency= 1.755859
Comp name = dsanalytics0 in_system_timestamp = 1609917752848.873047 out_system_timestamp = 1609917752849.414062 component latency= 0.541016
Comp name = demuxer in_system_timestamp = 1609917752849.458008 out_system_timestamp = 1609917752849.524902 component latency= 0.066895
Comp name = osd_conv in_system_timestamp = 1609917753571.541992 out_system_timestamp = 1609917753580.865967 component latency= 9.323975
Comp name = nvosd0 in_system_timestamp = 1609917753581.001953 out_system_timestamp = 1609917753588.173096 component latency= 7.171143
Source id = 0 Frame_num = 20 Frame latency = 1896.722168 (ms)

BATCH-NUM = 72**
Batch meta not found for buffer 0x7fcddc0799b0
BATCH-NUM = 73**
Batch meta not found for buffer 0x7fcb7c0992e0
BATCH-NUM = 74**
Batch meta not found for buffer 0x7fcb4c072370
BATCH-NUM = 75**
Batch meta not found for buffer 0x7fcbc40d5630
BATCH-NUM = 76**
Batch meta not found for buffer 0x7fcd10272190
BATCH-NUM = 77**
Batch meta not found for buffer 0x7fcb940fb710
BATCH-NUM = 21**
Comp name = nvv4l2decoder19 in_system_timestamp = 1609917751912.060059 out_system_timestamp = 1609917752575.031006 component latency= 662.970947
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 21 in_system_timestamp = 1609917752575.197021 out_system_timestamp = 1609917752619.439941 component_latency = 44.242920
Comp name = primary_gie in_system_timestamp = 1609917752619.562012 out_system_timestamp = 1609917752787.779053 component latency= 168.217041
Comp name = tracking_tracker in_system_timestamp = 1609917752787.805908 out_system_timestamp = 1609917752788.896973 component latency= 1.091064
Comp name = dsanalytics0 in_system_timestamp = 1609917752979.996094 out_system_timestamp = 1609917752980.745117 component latency= 0.749023
Comp name = demuxer in_system_timestamp = 1609917752980.795898 out_system_timestamp = 1609917752982.062012 component latency= 1.266113
Comp name = osd_conv in_system_timestamp = 1609917753780.909912 out_system_timestamp = 1609917753806.355957 component latency= 25.446045
Comp name = nvosd0 in_system_timestamp = 1609917753806.531982 out_system_timestamp = 1609917753818.664062 component latency= 12.132080
Source id = 0 Frame_num = 21 Frame latency = 1906.647949 (ms)

BATCH-NUM = 78**
Batch meta not found for buffer 0x7fcdec082ce0
BATCH-NUM = 79**
Batch meta not found for buffer 0x7fccdc080770
BATCH-NUM = 22**
Comp name = nvv4l2decoder19 in_system_timestamp = 1609917752147.077881 out_system_timestamp = 1609917752870.451904 component latency= 723.374023
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 22 in_system_timestamp = 1609917752870.570068 out_system_timestamp = 1609917752902.186035 component_latency = 31.615967
Comp name = primary_gie in_system_timestamp = 1609917752902.260986 out_system_timestamp = 1609917753142.527100 component latency= 240.266113
Comp name = tracking_tracker in_system_timestamp = 1609917753142.547119 out_system_timestamp = 1609917753143.593994 component latency= 1.046875
Comp name = dsanalytics0 in_system_timestamp = 1609917753143.637939 out_system_timestamp = 1609917753144.343018 component latency= 0.705078
Comp name = demuxer in_system_timestamp = 1609917753144.405029 out_system_timestamp = 1609917753146.102051 component latency= 1.697021
Comp name = osd_conv in_system_timestamp = 1609917753897.156006 out_system_timestamp = 1609917753902.124023 component latency= 4.968018
Comp name = nvosd0 in_system_timestamp = 1609917753902.262939 out_system_timestamp = 1609917753908.687012 component latency= 6.424072
Source id = 0 Frame_num = 22 Frame latency = 1761.650146 (ms)

BATCH-NUM = 80**
Batch meta not found for buffer 0x7fcd7c0750c0
BATCH-NUM = 81**
Batch meta not found for buffer 0x7fcb940665c0
BATCH-NUM = 82**
Batch meta not found for buffer 0x7fcd982627e0
BATCH-NUM = 83**
Batch meta not found for buffer 0x7fcd3c05c050
BATCH-NUM = 84**
Batch meta not found for buffer 0x7fcb2809d220
BATCH-NUM = 85**
Batch meta not found for buffer 0x7fcb940e5990
BATCH-NUM = 23**
Comp name = nvv4l2decoder19 in_system_timestamp = 1609917752263.249023 out_system_timestamp = 1609917753158.826904 component latency= 895.577881
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 23 in_system_timestamp = 1609917753158.955078 out_system_timestamp = 1609917753190.022949 component_latency = 31.067871
Comp name = primary_gie in_system_timestamp = 1609917753190.090088 out_system_timestamp = 1609917753309.740967 component latency= 119.650879
Comp name = tracking_tracker in_system_timestamp = 1609917753309.767090 out_system_timestamp = 1609917753310.512939 component latency= 0.745850
Comp name = dsanalytics0 in_system_timestamp = 1609917753311.395020 out_system_timestamp = 1609917753311.674072 component latency= 0.279053
Comp name = demuxer in_system_timestamp = 1609917753311.701904 out_system_timestamp = 1609917753312.129883 component latency= 0.427979
Comp name = osd_conv in_system_timestamp = 1609917754125.372070 out_system_timestamp = 1609917754135.540039 component latency= 10.167969
Comp name = nvosd0 in_system_timestamp = 1609917754135.722900 out_system_timestamp = 1609917754145.656982 component latency= 9.934082
Source id = 0 Frame_num = 23 Frame latency = 1882.445068 (ms)

BATCH-NUM = 86**
Batch meta not found for buffer 0x7fcbac102960
BATCH-NUM = 87**
Batch meta not found for buffer 0x7fcd980747f0
BATCH-NUM = 24**
Comp name = nvv4l2decoder19 in_system_timestamp = 1609917752363.689941 out_system_timestamp = 1609917753347.206055 component latency= 983.516113
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 24 in_system_timestamp = 1609917753347.370117 out_system_timestamp = 1609917753371.904053 component_latency = 24.533936
Comp name = primary_gie in_system_timestamp = 1609917753371.972900 out_system_timestamp = 1609917753542.464111 component latency= 170.491211
Comp name = tracking_tracker in_system_timestamp = 1609917753542.521973 out_system_timestamp = 1609917753543.701904 component latency= 1.179932
Comp name = dsanalytics0 in_system_timestamp = 1609917753544.061035 out_system_timestamp = 1609917753544.600098 component latency= 0.539062
Comp name = demuxer in_system_timestamp = 1609917753544.639893 out_system_timestamp = 1609917753545.656982 component latency= 1.017090
Comp name = osd_conv in_system_timestamp = 1609917754232.389893 out_system_timestamp = 1609917754235.907959 component latency= 3.518066
Comp name = nvosd0 in_system_timestamp = 1609917754236.045898 out_system_timestamp = 1609917754245.970947 component latency= 9.925049
Source id = 0 Frame_num = 24 Frame latency = 1882.322998 (ms)

BATCH-NUM = 88**
Batch meta not found for buffer 0x7fcdb007e880
BATCH-NUM = 89**
Batch meta not found for buffer 0x7fcd1c0bf260
BATCH-NUM = 90**
Batch meta not found for buffer 0x7fcb7c09c150
BATCH-NUM = 25**
Comp name = nvv4l2decoder19 in_system_timestamp = 1609917752474.002930 out_system_timestamp = 1609917753586.460938 component latency= 1112.458008
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 25 in_system_timestamp = 1609917753586.620117 out_system_timestamp = 1609917753604.885986 component_latency = 18.265869
Comp name = primary_gie in_system_timestamp = 1609917753604.941895 out_system_timestamp = 1609917753772.582031 component latency= 167.640137
Comp name = tracking_tracker in_system_timestamp = 1609917753772.612061 out_system_timestamp = 1609917753773.763916 component latency= 1.151855
Comp name = dsanalytics0 in_system_timestamp = 1609917754202.743896 out_system_timestamp = 1609917754203.208984 component latency= 0.465088
Comp name = demuxer in_system_timestamp = 1609917754203.263916 out_system_timestamp = 1609917754204.218994 component latency= 0.955078
Comp name = osd_conv in_system_timestamp = 1609917754340.840088 out_system_timestamp = 1609917754348.427002 component latency= 7.586914
Comp name = nvosd0 in_system_timestamp = 1609917754348.645020 out_system_timestamp = 1609917754362.333008 component latency= 13.687988
Source id = 0 Frame_num = 25 Frame latency = 1888.370117 (ms)

BATCH-NUM = 91**
Batch meta not found for buffer 0x7fcbdc1076d0
BATCH-NUM = 26**
Comp name = nvv4l2decoder19 in_system_timestamp = 1609917752870.659912 out_system_timestamp = 1609917753817.552002 component latency= 946.892090
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 26 in_system_timestamp = 1609917753817.718994 out_system_timestamp = 1609917753831.073975 component_latency = 13.354980
Comp name = primary_gie in_system_timestamp = 1609917753831.190918 out_system_timestamp = 1609917753831.263916 component latency= 0.072998
Comp name = tracking_tracker in_system_timestamp = 1609917753831.285889 out_system_timestamp = 1609917753832.018066 component latency= 0.732178
Comp name = dsanalytics0 in_system_timestamp = 1609917754209.593994 out_system_timestamp = 1609917754209.949951 component latency= 0.355957
Comp name = demuxer in_system_timestamp = 1609917754209.985107 out_system_timestamp = 1609917754210.794922 component latency= 0.809814
Comp name = osd_conv in_system_timestamp = 1609917754519.014893 out_system_timestamp = 1609917754615.635986 component latency= 96.621094
Comp name = nvosd0 in_system_timestamp = 1609917754615.849121 out_system_timestamp = 1609917754630.840088 component latency= 14.990967
Source id = 0 Frame_num = 26 Frame latency = 1760.247070 (ms)

BATCH-NUM = 92**
Batch meta not found for buffer 0x7fccc40d0de0
BATCH-NUM = 93**
Batch meta not found for buffer 0x7fcbc408ec00
BATCH-NUM = 94**
Batch meta not found for buffer 0x7fcdb007e880
BATCH-NUM = 95**
Batch meta not found for buffer 0x7fccdc08dcc0
BATCH-NUM = 96**
Batch meta not found for buffer 0x7fcd7c06a170
BATCH-NUM = 97**
Batch meta not found for buffer 0x7fcc20076de0
BATCH-NUM = 27**
Comp name = nvv4l2decoder19 in_system_timestamp = 1609917753158.981934 out_system_timestamp = 1609917753911.149902 component latency= 752.167969
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 27 in_system_timestamp = 1609917753911.253906 out_system_timestamp = 1609917754263.987061 component_latency = 352.733154
Comp name = primary_gie in_system_timestamp = 1609917754264.081055 out_system_timestamp = 1609917754264.211914 component latency= 0.130859
Comp name = tracking_tracker in_system_timestamp = 1609917754264.260010 out_system_timestamp = 1609917754265.500977 component latency= 1.240967
Comp name = dsanalytics0 in_system_timestamp = 1609917754292.775879 out_system_timestamp = 1609917754293.116943 component latency= 0.341064
Comp name = demuxer in_system_timestamp = 1609917754293.139893 out_system_timestamp = 1609917754293.194092 component latency= 0.054199
Comp name = osd_conv in_system_timestamp = 1609917754740.602051 out_system_timestamp = 1609917754741.007080 component latency= 0.405029
Comp name = nvosd0 in_system_timestamp = 1609917754741.227051 out_system_timestamp = 1609917754749.175049 component latency= 7.947998
Source id = 0 Frame_num = 27 Frame latency = 1590.228027 (ms)

BATCH-NUM = 98**
Batch meta not found for buffer 0x7fcb400771f0
BATCH-NUM = 99**
Batch meta not found for buffer 0x7fcddc0e1410
BATCH-NUM = 100**
Batch meta not found for buffer 0x7fcb4c078c80
BATCH-NUM = 101**
Batch meta not found for buffer 0x7fcc1006e890
BATCH-NUM = 28**
Comp name = nvv4l2decoder19 in_system_timestamp = 1609917753347.385010 out_system_timestamp = 1609917754137.417969 component latency= 790.032959
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 28 in_system_timestamp = 1609917754137.524902 out_system_timestamp = 1609917754324.845947 component_latency = 187.321045
Comp name = primary_gie in_system_timestamp = 1609917754324.930908 out_system_timestamp = 1609917754325.022949 component latency= 0.092041
Comp name = tracking_tracker in_system_timestamp = 1609917754325.066895 out_system_timestamp = 1609917754326.256104 component latency= 1.189209
Comp name = dsanalytics0 in_system_timestamp = 1609917754355.499023 out_system_timestamp = 1609917754356.166016 component latency= 0.666992
Comp name = demuxer in_system_timestamp = 1609917754356.221924 out_system_timestamp = 1609917754356.310059 component latency= 0.088135
Comp name = osd_conv in_system_timestamp = 1609917754960.891113 out_system_timestamp = 1609917755016.753906 component latency= 55.862793
Comp name = nvosd0 in_system_timestamp = 1609917755016.899902 out_system_timestamp = 1609917755032.287109 component latency= 15.387207
Source id = 0 Frame_num = 28 Frame latency = 1684.937988 (ms)

BATCH-NUM = 102**
Batch meta not found for buffer 0x7fcdec0847f0
BATCH-NUM = 103**
Batch meta not found for buffer 0x7fcab000aa70
BATCH-NUM = 104**
Batch meta not found for buffer 0x7fcdec0fc600
BATCH-NUM = 105**
Batch meta not found for buffer 0x7fcd9807beb0
BATCH-NUM = 29**
Comp name = nvv4l2decoder19 in_system_timestamp = 1609917753586.647949 out_system_timestamp = 1609917754245.068115 component latency= 658.420166
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 29 in_system_timestamp = 1609917754245.197998 out_system_timestamp = 1609917754385.571045 component_latency = 140.373047
Comp name = primary_gie in_system_timestamp = 1609917754385.666992 out_system_timestamp = 1609917754621.506104 component latency= 235.839111
Comp name = tracking_tracker in_system_timestamp = 1609917754621.543945 out_system_timestamp = 1609917754623.251953 component latency= 1.708008
Comp name = dsanalytics0 in_system_timestamp = 1609917754719.405029 out_system_timestamp = 1609917754719.931885 component latency= 0.526855
Comp name = demuxer in_system_timestamp = 1609917754719.970947 out_system_timestamp = 1609917754720.041992 component latency= 0.071045
Comp name = osd_conv in_system_timestamp = 1609917755257.181885 out_system_timestamp = 1609917755257.678955 component latency= 0.497070
Comp name = nvosd0 in_system_timestamp = 1609917755257.809082 out_system_timestamp = 1609917755264.979980 component latency= 7.170898
Source id = 0 Frame_num = 29 Frame latency = 1678.375000 (ms)

BATCH-NUM = 106**
Batch meta not found for buffer 0x7fcd100c49b0
BATCH-NUM = 107**
Batch meta not found for buffer 0x7fcb940665c0
BATCH-NUM = 108**
Batch meta not found for buffer 0x7fcc10067770
BATCH-NUM = 109**
Batch meta not found for buffer 0x7fcdb007e880
BATCH-NUM = 110**
Batch meta not found for buffer 0x7fcb7c0881f0
BATCH-NUM = 111**
Batch meta not found for buffer 0x7fccf810db10
BATCH-NUM = 112**
Batch meta not found for buffer 0x7fcb7c076b80
BATCH-NUM = 30**
Comp name = nvv4l2decoder19 in_system_timestamp = 1609917753817.705078 out_system_timestamp = 1609917754357.974121 component latency= 540.269043
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 30 in_system_timestamp = 1609917754358.047119 out_system_timestamp = 1609917754445.995117 component_latency = 87.947998
Comp name = primary_gie in_system_timestamp = 1609917754446.123047 out_system_timestamp = 1609917754622.270996 component latency= 176.147949
Comp name = tracking_tracker in_system_timestamp = 1609917754622.299072 out_system_timestamp = 1609917754623.293945 component latency= 0.994873
Comp name = dsanalytics0 in_system_timestamp = 1609917754816.322021 out_system_timestamp = 1609917754816.923096 component latency= 0.601074
Comp name = demuxer in_system_timestamp = 1609917754816.978027 out_system_timestamp = 1609917754817.073975 component latency= 0.095947
Comp name = osd_conv in_system_timestamp = 1609917755354.208984 out_system_timestamp = 1609917755356.583984 component latency= 2.375000
Comp name = nvosd0 in_system_timestamp = 1609917755356.722900 out_system_timestamp = 1609917755375.129883 component latency= 18.406982
Source id = 0 Frame_num = 30 Frame latency = 1557.471924 (ms)

BATCH-NUM = 113**
Batch meta not found for buffer 0x7fcd982821a0
BATCH-NUM = 114**
Batch meta not found for buffer 0x7fcd9826b550
BATCH-NUM = 115**
Batch meta not found for buffer 0x7fcb5c059c70
BATCH-NUM = 31**
Comp name = nvv4l2decoder19 in_system_timestamp = 1609917753911.251953 out_system_timestamp = 1609917754633.112061 component latency= 721.860107
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 31 in_system_timestamp = 1609917754633.219971 out_system_timestamp = 1609917754773.253906 component_latency = 140.033936
Comp name = primary_gie in_system_timestamp = 1609917754773.335938 out_system_timestamp = 1609917755020.487061 component latency= 247.151123
Comp name = tracking_tracker in_system_timestamp = 1609917755020.539062 out_system_timestamp = 1609917755022.561035 component latency= 2.021973
Comp name = dsanalytics0 in_system_timestamp = 1609917755022.938965 out_system_timestamp = 1609917755023.591064 component latency= 0.652100
Comp name = demuxer in_system_timestamp = 1609917755023.654053 out_system_timestamp = 1609917755023.733887 component latency= 0.079834
Comp name = osd_conv in_system_timestamp = 1609917755583.809082 out_system_timestamp = 1609917755589.152100 component latency= 5.343018
Comp name = nvosd0 in_system_timestamp = 1609917755589.292969 out_system_timestamp = 1609917755603.610107 component latency= 14.317139
Source id = 0 Frame_num = 31 Frame latency = 1692.431152 (ms)

BATCH-NUM = 116**
Batch meta not found for buffer 0x7fcd3c0e3660
BATCH-NUM = 117**
Batch meta not found for buffer 0x7fcbdc0858e0
BATCH-NUM = 118**
Batch meta not found for buffer 0x7fcb2806c180
BATCH-NUM = 119**
Batch meta not found for buffer 0x7fccdc07b150
BATCH-NUM = 120**
Batch meta not found for buffer 0x7fcb940ca580
BATCH-NUM = 32**
Comp name = nvv4l2decoder19 in_system_timestamp = 1609917754137.673096 out_system_timestamp = 1609917754746.957031 component latency= 609.283936
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 32 in_system_timestamp = 1609917754747.073975 out_system_timestamp = 1609917754833.827881 component_latency = 86.753906
Comp name = primary_gie in_system_timestamp = 1609917754833.959961 out_system_timestamp = 1609917755021.367920 component latency= 187.407959
Comp name = tracking_tracker in_system_timestamp = 1609917755021.439941 out_system_timestamp = 1609917755022.635010 component latency= 1.195068
Comp name = dsanalytics0 in_system_timestamp = 1609917755121.055908 out_system_timestamp = 1609917755121.840088 component latency= 0.784180
Comp name = demuxer in_system_timestamp = 1609917755121.904053 out_system_timestamp = 1609917755121.985107 component latency= 0.081055
Comp name = osd_conv in_system_timestamp = 1609917755697.281006 out_system_timestamp = 1609917755701.147949 component latency= 3.866943
Comp name = nvosd0 in_system_timestamp = 1609917755701.308105 out_system_timestamp = 1609917755705.940918 component latency= 4.632812
Source id = 0 Frame_num = 32 Frame latency = 1568.305908 (ms)

BATCH-NUM = 121**
Batch meta not found for buffer 0x7fcb4c07ee10
BATCH-NUM = 122**
Batch meta not found for buffer 0x7fcb7c073d50
BATCH-NUM = 123**
Batch meta not found for buffer 0x7fcddc104790
BATCH-NUM = 124**
Batch meta not found for buffer 0x7fccc4086d00
BATCH-NUM = 33**
Comp name = nvv4l2decoder19 in_system_timestamp = 1609917754245.335938 out_system_timestamp = 1609917755019.340088 component latency= 774.004150
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 33 in_system_timestamp = 1609917755019.450928 out_system_timestamp = 1609917755073.511963 component_latency = 54.061035
Comp name = primary_gie in_system_timestamp = 1609917755073.574951 out_system_timestamp = 1609917755259.562012 component latency= 185.987061
Comp name = tracking_tracker in_system_timestamp = 1609917755259.596924 out_system_timestamp = 1609917755260.397949 component latency= 0.801025
Comp name = dsanalytics0 in_system_timestamp = 1609917755282.680908 out_system_timestamp = 1609917755282.979004 component latency= 0.298096
Comp name = demuxer in_system_timestamp = 1609917755283.004883 out_system_timestamp = 1609917755283.085938 component latency= 0.081055
Comp name = osd_conv in_system_timestamp = 1609917755791.791016 out_system_timestamp = 1609917755793.395020 component latency= 1.604004
Comp name = nvosd0 in_system_timestamp = 1609917755793.523926 out_system_timestamp = 1609917755804.326904 component latency= 10.802979
Source id = 0 Frame_num = 33 Frame latency = 1559.041992 (ms)

BATCH-NUM = 125**
Batch meta not found for buffer 0x7fcbf80e82a0
BATCH-NUM = 126**
Batch meta not found for buffer 0x7fcbf80c2d50
BATCH-NUM = 127**
Batch meta not found for buffer 0x7fcd10088280
BATCH-NUM = 34**
Comp name = nvv4l2decoder19 in_system_timestamp = 1609917754358.166016 out_system_timestamp = 1609917755259.085938 component latency= 900.919922
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 34 in_system_timestamp = 1609917755259.252930 out_system_timestamp = 1609917755334.030029 component_latency = 74.777100
Comp name = primary_gie in_system_timestamp = 1609917755334.113037 out_system_timestamp = 1609917755559.299072 component latency= 225.186035
Comp name = tracking_tracker in_system_timestamp = 1609917755559.318115 out_system_timestamp = 1609917755560.245117 component latency= 0.927002
Comp name = dsanalytics0 in_system_timestamp = 1609917755680.076904 out_system_timestamp = 1609917755680.467041 component latency= 0.390137
Comp name = demuxer in_system_timestamp = 1609917755680.520020 out_system_timestamp = 1609917755680.577881 component latency= 0.057861
Comp name = osd_conv in_system_timestamp = 1609917756094.486084 out_system_timestamp = 1609917756099.237061 component latency= 4.750977
Comp name = nvosd0 in_system_timestamp = 1609917756099.444092 out_system_timestamp = 1609917756112.250000 component latency= 12.805908
Source id = 0 Frame_num = 34 Frame latency = 1754.112061 (ms)

BATCH-NUM = 128**
Batch meta not found for buffer 0x7fcbf813a0b0
BATCH-NUM = 129**
Batch meta not found for buffer 0x7fcb4c07ed00
BATCH-NUM = 130**
Batch meta not found for buffer 0x7fcbdc0858e0
BATCH-NUM = 131**
Batch meta not found for buffer 0x7fccf8081290
BATCH-NUM = 132**
Batch meta not found for buffer 0x7fcd3c089be0
BATCH-NUM = 133**
Batch meta not found for buffer 0x7fcd3c0994e0
BATCH-NUM = 134**
Batch meta not found for buffer 0x7fcb5c059c70
BATCH-NUM = 135**
Batch meta not found for buffer 0x7fcc48082390
BATCH-NUM = 136**
Batch meta not found for buffer 0x7fcd1c0bfe10
BATCH-NUM = 137**
Batch meta not found for buffer 0x7fcd3c0a5930**PERF: 5.27 (4.36) 5.47 (4.12) 0.00 (0.00) 5.47 (5.08) 5.67 (5.37) 5.27 (5.22) 0.00 (0.00) 5.47 (3.11) 6.08 (6.23) 5.06 (4.23) 4.82 (4.09) 6.28 (6.23) 4.44 (3.75) 5.88 (5.51) 0.00 (0.00) 5.88 (6.06) 4.86 (4.01) 5.06 (4.48) 5.67 (6.01) 5.02 (4.65) 5.67 (5.51) 3.47 (4.15) 5.07 (5.08) 6.08 (6.23) 5.06 (4.58) 4.24 (4.23) 5.02 (4.81) 5.88 (6.06) 4.86 (4.29) 5.47 (5.19)

BATCH-NUM = 35**
Comp name = nvv4l2decoder19 in_system_timestamp = 1609917754633.322998 out_system_timestamp = 1609917755358.801025 component latency= 725.478027
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 35 in_system_timestamp = 1609917755358.912109 out_system_timestamp = 1609917755394.457031 component_latency = 35.544922
Comp name = primary_gie in_system_timestamp = 1609917755401.384033 out_system_timestamp = 1609917755559.643066 component latency= 158.259033
Comp name = tracking_tracker in_system_timestamp = 1609917755559.669922 out_system_timestamp = 1609917755560.299072 component latency= 0.629150
Comp name = dsanalytics0 in_system_timestamp = 1609917755735.114014 out_system_timestamp = 1609917755735.461914 component latency= 0.347900
Comp name = demuxer in_system_timestamp = 1609917755735.493896 out_system_timestamp = 1609917755771.550049 component latency= 36.056152
Comp name = osd_conv in_system_timestamp = 1609917756183.709961 out_system_timestamp = 1609917756197.220947 component latency= 13.510986
Comp name = nvosd0 in_system_timestamp = 1609917756197.339111 out_system_timestamp = 1609917756210.053955 component latency= 12.714844
Source id = 0 Frame_num = 35 Frame latency = 1576.770020 (ms)

BATCH-NUM = 138**
Batch meta not found for buffer 0x7fcbf80cf390
BATCH-NUM = 139**
Batch meta not found for buffer 0x7fcd3c079c90
BATCH-NUM = 140**
Batch meta not found for buffer 0x7fcc3c0b5cc0
BATCH-NUM = 141**
Batch meta not found for buffer 0x7fcdec0f02d0
BATCH-NUM = 142**
Batch meta not found for buffer 0x7fcb940ca580
BATCH-NUM = 143**
Batch meta not found for buffer 0x7fcdb006e7a0
BATCH-NUM = 144**
Batch meta not found for buffer 0x7fcd3c0ce9c0
BATCH-NUM = 145**
Batch meta not found for buffer 0x7fcd3c084230
BATCH-NUM = 146**
Batch meta not found for buffer 0x7fcd1008b9f0
BATCH-NUM = 147**
Batch meta not found for buffer 0x7fcb4c07ee10
BATCH-NUM = 148**
Batch meta not found for buffer 0x7fccf80d01c0
BATCH-NUM = 149**
Batch meta not found for buffer 0x7fcc100fa890
BATCH-NUM = 150**
Batch meta not found for buffer 0x7fccc40d0de0
BATCH-NUM = 151**
Batch meta not found for buffer 0x7fcbc409c240
BATCH-NUM = 152**
Batch meta not found for buffer 0x7fcd3c0cead0
BATCH-NUM = 36**
Comp name = nvv4l2decoder19 in_system_timestamp = 1609917754747.211914 out_system_timestamp = 1609917755594.887939 component latency= 847.676025
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 36 in_system_timestamp = 1609917755595.003906 out_system_timestamp = 1609917755731.148926 component_latency = 136.145020
Comp name = primary_gie in_system_timestamp = 1609917755731.227051 out_system_timestamp = 1609917756014.604004 component latency= 283.376953
Comp name = tracking_tracker in_system_timestamp = 1609917756014.632080 out_system_timestamp = 1609917756015.652100 component latency= 1.020020
Comp name = dsanalytics0 in_system_timestamp = 1609917756015.675049 out_system_timestamp = 1609917756016.104004 component latency= 0.428955
Comp name = demuxer in_system_timestamp = 1609917756016.129883 out_system_timestamp = 1609917756016.200928 component latency= 0.071045
Comp name = osd_conv in_system_timestamp = 1609917756306.759033 out_system_timestamp = 1609917756324.241943 component latency= 17.482910
Comp name = nvosd0 in_system_timestamp = 1609917756324.389893 out_system_timestamp = 1609917756345.419922 component latency= 21.030029
Source id = 0 Frame_num = 36 Frame latency = 1598.255127 (ms)

BATCH-NUM = 153**
Batch meta not found for buffer 0x7fccf8075300
BATCH-NUM = 154**
Batch meta not found for buffer 0x7fcc20076010
BATCH-NUM = 155**
Batch meta not found for buffer 0x7fcc7006d3b0
BATCH-NUM = 156**
Batch meta not found for buffer 0x7fccdc084520
BATCH-NUM = 157**
Batch meta not found for buffer 0x7fcd7c078890
BATCH-NUM = 158**
Batch meta not found for buffer 0x7fcd1c066c40
BATCH-NUM = 159**
Batch meta not found for buffer 0x7fcd1c072400
BATCH-NUM = 37**
Comp name = nvv4l2decoder19 in_system_timestamp = 1609917755019.511963 out_system_timestamp = 1609917755702.500000 component latency= 682.988037
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 37 in_system_timestamp = 1609917755702.569092 out_system_timestamp = 1609917755795.293945 component_latency = 92.724854
Comp name = primary_gie in_system_timestamp = 1609917755795.427979 out_system_timestamp = 1609917756015.159912 component latency= 219.731934
Comp name = tracking_tracker in_system_timestamp = 1609917756015.183105 out_system_timestamp = 1609917756016.560059 component latency= 1.376953
Comp name = dsanalytics0 in_system_timestamp = 1609917756048.801025 out_system_timestamp = 1609917756049.218994 component latency= 0.417969
Comp name = demuxer in_system_timestamp = 1609917756049.250977 out_system_timestamp = 1609917756049.318115 component latency= 0.067139
Comp name = osd_conv in_system_timestamp = 1609917756586.992920 out_system_timestamp = 1609917756597.852051 component latency= 10.859131
Comp name = nvosd0 in_system_timestamp = 1609917756597.996094 out_system_timestamp = 1609917756604.767090 component latency= 6.770996
Source id = 0 Frame_num = 37 Frame latency = 1585.318115 (ms)

BATCH-NUM = 160**
Batch meta not found for buffer 0x7fcc3c0970c0
BATCH-NUM = 161**
Batch meta not found for buffer 0x7fcd1c0b6ce0
BATCH-NUM = 162**
Batch meta not found for buffer 0x7fcdec0ba3c0
BATCH-NUM = 163**
Batch meta not found for buffer 0x7fcdec0847f0
BATCH-NUM = 164**
Batch meta not found for buffer 0x7fcb4c07ed00
BATCH-NUM = 165**
Batch meta not found for buffer 0x7fcb4c0c2e30
BATCH-NUM = 166**
Batch meta not found for buffer 0x7fcc200cdd10
BATCH-NUM = 167**
Batch meta not found for buffer 0x7fcb4c0dc120
BATCH-NUM = 168**
Batch meta not found for buffer 0x7fcd3c090aa0
BATCH-NUM = 169**
Batch meta not found for buffer 0x7fccdc084c90
BATCH-NUM = 170**
Batch meta not found for buffer 0x7fcddc11e1e0
BATCH-NUM = 171**
Batch meta not found for buffer 0x7fcd1c093140
BATCH-NUM = 172**
Batch meta not found for buffer 0x7fcc100617c0
BATCH-NUM = 173**
Batch meta not found for buffer 0x7fcc200aa9a0
BATCH-NUM = 38**
Comp name = nvv4l2decoder19 in_system_timestamp = 1609917755259.282959 out_system_timestamp = 1609917755804.675049 component latency= 545.392090
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 38 in_system_timestamp = 1609917755804.797119 out_system_timestamp = 1609917755855.750977 component_latency = 50.953857
Comp name = primary_gie in_system_timestamp = 1609917755855.868896 out_system_timestamp = 1609917756015.991943 component latency= 160.123047
Comp name = tracking_tracker in_system_timestamp = 1609917756016.009033 out_system_timestamp = 1609917756016.762939 component latency= 0.753906
Comp name = dsanalytics0 in_system_timestamp = 1609917756217.899902 out_system_timestamp = 1609917756218.374023 component latency= 0.474121
Comp name = demuxer in_system_timestamp = 1609917756218.426025 out_system_timestamp = 1609917756253.575928 component latency= 35.149902
Comp name = osd_conv in_system_timestamp = 1609917756802.668945 out_system_timestamp = 1609917756806.224121 component latency= 3.555176
Comp name = nvosd0 in_system_timestamp = 1609917756806.382080 out_system_timestamp = 1609917756817.645020 component latency= 11.262939
Source id = 0 Frame_num = 38 Frame latency = 1558.406982 (ms)

BATCH-NUM = 174**
Batch meta not found for buffer 0x7fcb2806f4f0
BATCH-NUM = 175**
Batch meta not found for buffer 0x7fcb280780d0
BATCH-NUM = 176**
Batch meta not found for buffer 0x7fcae800dc30
BATCH-NUM = 177**
Batch meta not found for buffer 0x7fcb7012aab0
BATCH-NUM = 178**
Batch meta not found for buffer 0x7fcc10080030
BATCH-NUM = 179**
Batch meta not found for buffer 0x7fcbf80c25e0
BATCH-NUM = 180**
Batch meta not found for buffer 0x7fcbdc0957b0
BATCH-NUM = 181**
Batch meta not found for buffer 0x7fcb940ca580
BATCH-NUM = 182**
Batch meta not found for buffer 0x7fcb400ef730
BATCH-NUM = 183**
Batch meta not found for buffer 0x7fcd100bab30
BATCH-NUM = 183**
Batch meta not found for buffer 0x7fcc7006b0a0
BATCH-NUM = 185**
Batch meta not found for buffer 0x7fcbdc085c10
BATCH-NUM = 186**
Batch meta not found for buffer 0x7fcdb007d400
BATCH-NUM = 187**
Batch meta not found for buffer 0x7fcc4806e270
BATCH-NUM = 188**
Batch meta not found for buffer 0x7fcb7c0748f0
BATCH-NUM = 189**
Batch meta not found for buffer 0x7fcd980728e0
BATCH-NUM = 190**
Batch meta not found for buffer 0x7fcdec0fc600
BATCH-NUM = 191**
Batch meta not found for buffer 0x7fcc48084a20
BATCH-NUM = 192**
Batch meta not found for buffer 0x7fcd9826f9c0
BATCH-NUM = 193**
Batch meta not found for buffer 0x7fcb4c073ab0
BATCH-NUM = 194**
Batch meta not found for buffer 0x7fcc200cd270
BATCH-NUM = 195**
Batch meta not found for buffer 0x7fcbdc07eb10
BATCH-NUM = 196**
Batch meta not found for buffer 0x7fcbdc08c5d0
BATCH-NUM = 39**
Comp name = nvv4l2decoder19 in_system_timestamp = 1609917755358.943115 out_system_timestamp = 1609917756120.384033 component latency= 761.440918
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 39 in_system_timestamp = 1609917756120.455078 out_system_timestamp = 1609917756278.733887 component_latency = 158.278809
Comp name = primary_gie in_system_timestamp = 1609917756278.809082 out_system_timestamp = 1609917756278.886963 component latency= 0.077881
Comp name = tracking_tracker in_system_timestamp = 1609917756278.933105 out_system_timestamp = 1609917756280.081055 component latency= 1.147949
Comp name = dsanalytics0 in_system_timestamp = 1609917756280.105957 out_system_timestamp = 1609917756280.458984 component latency= 0.353027
Comp name = demuxer in_system_timestamp = 1609917756280.481934 out_system_timestamp = 1609917756280.540039 component latency= 0.058105
Comp name = osd_conv in_system_timestamp = 1609917756939.879883 out_system_timestamp = 1609917756956.801025 component latency= 16.921143
Comp name = nvosd0 in_system_timestamp = 1609917756956.928955 out_system_timestamp = 1609917756991.604004 component latency= 34.675049
Source id = 0 Frame_num = 39 Frame latency = 1632.704834 (ms)

BATCH-NUM = 197**
Batch meta not found for buffer 0x7fcb940ca690
BATCH-NUM = 198**
Batch meta not found for buffer 0x7fcbc408e7c0
BATCH-NUM = 199**
Batch meta not found for buffer 0x7fcb28101270
BATCH-NUM = 200**
Batch meta not found for buffer 0x7fcd4c0f8300
BATCH-NUM = 40**
Comp name = nvv4l2decoder19 in_system_timestamp = 1609917755595.103027 out_system_timestamp = 1609917756199.827881 component latency= 604.724854
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 40 in_system_timestamp = 1609917756199.951904 out_system_timestamp = 1609917756339.187988 component_latency = 139.236084
Comp name = primary_gie in_system_timestamp = 1609917756339.329102 out_system_timestamp = 1609917756586.309082 component latency= 246.979980
Comp name = tracking_tracker in_system_timestamp = 1609917756586.345947 out_system_timestamp = 1609917756587.507080 component latency= 1.161133
Comp name = dsanalytics0 in_system_timestamp = 1609917756587.615967 out_system_timestamp = 1609917756588.653076 component latency= 1.037109
Comp name = demuxer in_system_timestamp = 1609917756588.722900 out_system_timestamp = 1609917756588.814941 component latency= 0.092041
Comp name = osd_conv in_system_timestamp = 1609917757079.437988 out_system_timestamp = 1609917757081.007080 component latency= 1.569092
Comp name = nvosd0 in_system_timestamp = 1609917757081.141113 out_system_timestamp = 1609917757093.131104 component latency= 11.989990
Source id = 0 Frame_num = 40 Frame latency = 1498.067871 (ms)

BATCH-NUM = 201**
Batch meta not found for buffer 0x7fcb4c07abd0
BATCH-NUM = 202**
Batch meta not found for buffer 0x7fcbc408d170
BATCH-NUM = 203**
Batch meta not found for buffer 0x7fcb280bc0e0
BATCH-NUM = 41**
Comp name = nvv4l2decoder19 in_system_timestamp = 1609917755702.638916 out_system_timestamp = 1609917756343.581055 component latency= 640.942139
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 41 in_system_timestamp = 1609917756343.697998 out_system_timestamp = 1609917756399.607910 component_latency = 55.909912
Comp name = primary_gie in_system_timestamp = 1609917756399.746094 out_system_timestamp = 1609917756587.039062 component latency= 187.292969
Comp name = tracking_tracker in_system_timestamp = 1609917756587.062988 out_system_timestamp = 1609917756588.141113 component latency= 1.078125
Comp name = dsanalytics0 in_system_timestamp = 1609917756590.709961 out_system_timestamp = 1609917756591.102051 component latency= 0.392090
Comp name = demuxer in_system_timestamp = 1609917756591.138916 out_system_timestamp = 1609917756591.478027 component latency= 0.339111
Comp name = osd_conv in_system_timestamp = 1609917757273.423096 out_system_timestamp = 1609917757291.152100 component latency= 17.729004
Comp name = nvosd0 in_system_timestamp = 1609917757291.350098 out_system_timestamp = 1609917757323.905029 component latency= 32.554932
Source id = 0 Frame_num = 41 Frame latency = 1621.312012 (ms)

BATCH-NUM = 204**
Batch meta not found for buffer 0x7fccdc08ea60
BATCH-NUM = 42**
Comp name = nvv4l2decoder19 in_system_timestamp = 1609917755804.944092 out_system_timestamp = 1609917756609.947021 component latency= 805.002930
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 42 in_system_timestamp = 1609917756610.079102 out_system_timestamp = 1609917756644.858887 component_latency = 34.779785
Comp name = primary_gie in_system_timestamp = 1609917756644.983887 out_system_timestamp = 1609917756789.157959 component latency= 144.174072
Comp name = tracking_tracker in_system_timestamp = 1609917756789.186035 out_system_timestamp = 1609917756789.881104 component latency= 0.695068
Comp name = dsanalytics0 in_system_timestamp = 1609917756789.993896 out_system_timestamp = 1609917756790.645996 component latency= 0.652100
Comp name = demuxer in_system_timestamp = 1609917756790.686035 out_system_timestamp = 1609917756791.240967 component latency= 0.554932
Comp name = osd_conv in_system_timestamp = 1609917757556.306885 out_system_timestamp = 1609917757565.666016 component latency= 9.359131
Comp name = nvosd0 in_system_timestamp = 1609917757565.822021 out_system_timestamp = 1609917757579.299072 component latency= 13.477051
Source id = 0 Frame_num = 42 Frame latency = 1774.419922 (ms)

BATCH-NUM = 205**
Batch meta not found for buffer 0x7fcd10087d60
BATCH-NUM = 206**
Batch meta not found for buffer 0x7fcb94137600
BATCH-NUM = 43**
Comp name = nvv4l2decoder19 in_system_timestamp = 1609917756120.541016 out_system_timestamp = 1609917756807.608887 component latency= 687.067871
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 43 in_system_timestamp = 1609917756807.666992 out_system_timestamp = 1609917756828.982910 component_latency = 21.315918
Comp name = primary_gie in_system_timestamp = 1609917756829.052002 out_system_timestamp = 1609917756938.541992 component latency= 109.489990
Comp name = tracking_tracker in_system_timestamp = 1609917756938.564941 out_system_timestamp = 1609917756939.066895 component latency= 0.501953
Comp name = dsanalytics0 in_system_timestamp = 1609917756939.090088 out_system_timestamp = 1609917756939.364990 component latency= 0.274902
Comp name = demuxer in_system_timestamp = 1609917756939.402100 out_system_timestamp = 1609917756939.764893 component latency= 0.362793
Comp name = osd_conv in_system_timestamp = 1609917757659.420898 out_system_timestamp = 1609917757659.799072 component latency= 0.378174
Comp name = nvosd0 in_system_timestamp = 1609917757659.914062 out_system_timestamp = 1609917757668.841064 component latency= 8.927002
Source id = 0 Frame_num = 43 Frame latency = 1548.343018 (ms)

BATCH-NUM = 207**
Batch meta not found for buffer 0x7fcae801cc80
BATCH-NUM = 44**
Comp name = nvv4l2decoder19 in_system_timestamp = 1609917756199.949951 out_system_timestamp = 1609917757003.363037 component latency= 803.413086
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 44 in_system_timestamp = 1609917757003.534912 out_system_timestamp = 1609917757057.465088 component_latency = 53.930176
Comp name = primary_gie in_system_timestamp = 1609917757057.593994 out_system_timestamp = 1609917757232.635986 component latency= 175.041992
Comp name = tracking_tracker in_system_timestamp = 1609917757232.650879 out_system_timestamp = 1609917757233.498047 component latency= 0.847168
Comp name = dsanalytics0 in_system_timestamp = 1609917757233.604980 out_system_timestamp = 1609917757234.419922 component latency= 0.814941
Comp name = demuxer in_system_timestamp = 1609917757234.489014 out_system_timestamp = 1609917757234.711914 component latency= 0.222900
Comp name = osd_conv in_system_timestamp = 1609917757789.731934 out_system_timestamp = 1609917757819.201904 component latency= 29.469971
Comp name = nvosd0 in_system_timestamp = 1609917757819.364014 out_system_timestamp = 1609917757875.028076 component latency= 55.664062
Source id = 0 Frame_num = 44 Frame latency = 1675.110107 (ms)

BATCH-NUM = 208**
Batch meta not found for buffer 0x7fcd98078c80
BATCH-NUM = 209**
Batch meta not found for buffer 0x7fcb94137600
BATCH-NUM = 45**
Comp name = nvv4l2decoder19 in_system_timestamp = 1609917756343.750977 out_system_timestamp = 1609917757093.410889 component latency= 749.659912
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 45 in_system_timestamp = 1609917757093.466064 out_system_timestamp = 1609917757117.783936 component_latency = 24.317871
Comp name = primary_gie in_system_timestamp = 1609917757117.899902 out_system_timestamp = 1609917757232.933105 component latency= 115.033203
Comp name = tracking_tracker in_system_timestamp = 1609917757232.940918 out_system_timestamp = 1609917757233.556885 component latency= 0.615967
Comp name = dsanalytics0 in_system_timestamp = 1609917757235.833008 out_system_timestamp = 1609917757236.097900 component latency= 0.264893
Comp name = demuxer in_system_timestamp = 1609917757236.125977 out_system_timestamp = 1609917757236.739014 component latency= 0.613037
Comp name = osd_conv in_system_timestamp = 1609917758006.749023 out_system_timestamp = 1609917758031.490967 component latency= 24.741943
Comp name = nvosd0 in_system_timestamp = 1609917758031.614014 out_system_timestamp = 1609917758099.768066 component latency= 68.154053
Source id = 0 Frame_num = 45 Frame latency = 1756.059082 (ms)

BATCH-NUM = 210**
Batch meta not found for buffer 0x7fcc2007ce70
BATCH-NUM = 46**
Comp name = nvv4l2decoder19 in_system_timestamp = 1609917756610.214111 out_system_timestamp = 1609917757296.863037 component latency= 686.648926
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 46 in_system_timestamp = 1609917757297.018066 out_system_timestamp = 1609917757357.208984 component_latency = 60.190918
Comp name = primary_gie in_system_timestamp = 1609917757359.102051 out_system_timestamp = 1609917757495.125977 component latency= 136.023926
Comp name = tracking_tracker in_system_timestamp = 1609917757495.220947 out_system_timestamp = 1609917757495.881104 component latency= 0.660156
Comp name = dsanalytics0 in_system_timestamp = 1609917757497.562988 out_system_timestamp = 1609917757497.873047 component latency= 0.310059
Comp name = demuxer in_system_timestamp = 1609917757497.905029 out_system_timestamp = 1609917757497.958008 component latency= 0.052979
Comp name = osd_conv in_system_timestamp = 1609917758181.531982 out_system_timestamp = 1609917758191.289062 component latency= 9.757080
Comp name = nvosd0 in_system_timestamp = 1609917758191.413086 out_system_timestamp = 1609917758199.367920 component latency= 7.954834
Source id = 0 Frame_num = 46 Frame latency = 1589.194824 (ms)

BATCH-NUM = 211**
Batch meta not found for buffer 0x7fcae801cc80
BATCH-NUM = 47**
Comp name = nvv4l2decoder19 in_system_timestamp = 1609917756807.740967 out_system_timestamp = 1609917757567.740967 component latency= 760.000000
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 47 in_system_timestamp = 1609917757567.812988 out_system_timestamp = 1609917757628.044922 component_latency = 60.231934
Comp name = primary_gie in_system_timestamp = 1609917757628.159912 out_system_timestamp = 1609917757628.259033 component latency= 0.099121
Comp name = tracking_tracker in_system_timestamp = 1609917757628.287109 out_system_timestamp = 1609917757629.102051 component latency= 0.814941
Comp name = dsanalytics0 in_system_timestamp = 1609917757641.406982 out_system_timestamp = 1609917757641.836914 component latency= 0.429932
Comp name = demuxer in_system_timestamp = 1609917757641.874023 out_system_timestamp = 1609917757641.941895 component latency= 0.067871
Comp name = osd_conv in_system_timestamp = 1609917758435.478027 out_system_timestamp = 1609917758450.093018 component latency= 14.614990
Comp name = nvosd0 in_system_timestamp = 1609917758450.259033 out_system_timestamp = 1609917758560.843018 component latency= 110.583984
Source id = 0 Frame_num = 47 Frame latency = 1753.151123 (ms)

^C^C^C

14

The video decoder is very slow. Seems 30 decoding+60 encoding exhaust the nvdec capability.

15

Here is the output with 2 RTSP IN and OUT

root@va03:/opt/nvidia/deepstream/deepstream-5.0/sources/apps/sample_apps/deepstream-nvdsanalytics-test# ./deepstream-app -c ./assets/topconfig.txt

*** DeepStream: Launched RTSP Streaming at rtsp://localhost:8554/ds-test ***

*** DeepStream: Launched RTSP Streaming at rtsp://localhost:8555/ds-test ***

gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream-5.0/lib/libnvds_nvdcf.so
gstnvtracker: Batch processing is ON
gstnvtracker: Past frame output is OFF
[NvDCF] Initialized
ERROR: …/nvdsinfer/nvdsinfer_func_utils.cpp:33 [TRT]: …/rtSafe/coreReadArchive.cpp (31) - Serialization Error in verifyHeader: 0 (Magic tag does not match)
ERROR: …/nvdsinfer/nvdsinfer_func_utils.cpp:33 [TRT]: INVALID_STATE: std::exception
ERROR: …/nvdsinfer/nvdsinfer_func_utils.cpp:33 [TRT]: INVALID_CONFIG: Deserialize the cuda engine failed.
ERROR: …/nvdsinfer/nvdsinfer_model_builder.cpp:1567 Deserialize engine failed from file: /opt/nvidia/deepstream/deepstream-5.0/sources/apps/sample_apps/deepstream-nvdsanalytics-test/assets/pgie-config.txt
0:00:01.329449793 773 0x55ba196cf8c0 WARN nvinfer gstnvinfer.cpp:616:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1690> [UID = 1]: deserialize engine from file :/opt/nvidia/deepstream/deepstream-5.0/sources/apps/sample_apps/deepstream-nvdsanalytics-test/assets/pgie-config.txt failed
0:00:01.329511308 773 0x55ba196cf8c0 WARN nvinfer gstnvinfer.cpp:616:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:1797> [UID = 1]: deserialize backend context from engine from file :/opt/nvidia/deepstream/deepstream-5.0/sources/apps/sample_apps/deepstream-nvdsanalytics-test/assets/pgie-config.txt failed, try rebuild
0:00:01.329541012 773 0x55ba196cf8c0 INFO nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1715> [UID = 1]: Trying to create engine from model files
INFO: …/nvdsinfer/nvdsinfer_func_utils.cpp:39 [TRT]: Reading Calibration Cache for calibrator: EntropyCalibration2
INFO: …/nvdsinfer/nvdsinfer_func_utils.cpp:39 [TRT]: Generated calibration scales using calibration cache. Make sure that calibration cache has latest scales.
INFO: …/nvdsinfer/nvdsinfer_func_utils.cpp:39 [TRT]: To regenerate calibration cache, please delete the existing one. TensorRT will generate a new calibration cache.
INFO: …/nvdsinfer/nvdsinfer_func_utils.cpp:39 [TRT]: Some tactics do not have sufficient workspace memory to run. Increasing workspace size may increase performance, please check verbose output.
INFO: …/nvdsinfer/nvdsinfer_func_utils.cpp:39 [TRT]: Detected 1 inputs and 2 output network tensors.
0:00:23.396202714 773 0x55ba196cf8c0 INFO nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1748> [UID = 1]: serialize cuda engine to file: /opt/nvidia/deepstream/deepstream-5.0/sources/apps/sample_apps/deepstream-nvdsanalytics-test/assets/resnet10_detector.etlt_b2_gpu0_int8.engine successfully
WARNING: …/nvdsinfer/nvdsinfer_func_utils.cpp:36 [TRT]: Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles
INFO: …/nvdsinfer/nvdsinfer_model_builder.cpp:685 [Implicit Engine Info]: layers num: 3
0 INPUT kFLOAT input_1 3x1080x1920
1 OUTPUT kFLOAT output_bbox/BiasAdd 4x68x120
2 OUTPUT kFLOAT output_cov/Sigmoid 1x68x120

0:00:23.401008024 773 0x55ba196cf8c0 INFO nvinfer gstnvinfer_impl.cpp:313:notifyLoadModelStatus:<primary_gie> [UID 1]: Load new model:/opt/nvidia/deepstream/deepstream-5.0/sources/apps/sample_apps/deepstream-nvdsanalytics-test/assets/pgie-config.txt sucessfully

Runtime commands:
h: Print this help
q: Quit

    p: Pause
    r: Resume

**PERF: FPS 0 (Avg) FPS 1 (Avg)
**PERF: 0.00 (0.00) 0.00 (0.00)
** INFO: <bus_callback:181>: Pipeline ready

** INFO: <bus_callback:167>: Pipeline running

Alerting Set after 6000 frames

BATCH-NUM = 0**
Comp name = nvv4l2decoder1 in_system_timestamp = 1609918678766.760986 out_system_timestamp = 1609918678898.365967 component latency= 131.604980
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 0 in_system_timestamp = 1609918678898.592041 out_system_timestamp = 1609918678901.191895 component_latency = 2.599854
Comp name = primary_gie in_system_timestamp = 1609918678902.510010 out_system_timestamp = 1609918678921.615967 component latency= 19.105957
Comp name = tracking_tracker in_system_timestamp = 1609918678921.625977 out_system_timestamp = 1609918678923.310059 component latency= 1.684082
Comp name = dsanalytics0 in_system_timestamp = 1609918678923.402100 out_system_timestamp = 1609918678923.572021 component latency= 0.169922
Comp name = demuxer in_system_timestamp = 1609918678923.593994 out_system_timestamp = 1609918678923.684082 component latency= 0.090088
Comp name = osd_conv in_system_timestamp = 1609918678934.627930 out_system_timestamp = 1609918678936.066895 component latency= 1.438965
Comp name = nvosd0 in_system_timestamp = 1609918678936.201904 out_system_timestamp = 1609918678953.002930 component latency= 16.801025
Source id = 0 Frame_num = 0 Frame latency = 186.312988 (ms)

BATCH-NUM = 0**
Batch meta not found for buffer 0x7fbbd408b930
BATCH-NUM = 1**
Batch meta not found for buffer 0x7fbbd408b600
BATCH-NUM = 1**
Comp name = nvv4l2decoder1 in_system_timestamp = 1609918678878.520996 out_system_timestamp = 1609918678912.104004 component latency= 33.583008
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 1 in_system_timestamp = 1609918678912.231934 out_system_timestamp = 1609918678933.271973 component_latency = 21.040039
Comp name = primary_gie in_system_timestamp = 1609918678933.325928 out_system_timestamp = 1609918678952.658936 component latency= 19.333008
Comp name = tracking_tracker in_system_timestamp = 1609918678952.676025 out_system_timestamp = 1609918678954.058105 component latency= 1.382080
Comp name = dsanalytics0 in_system_timestamp = 1609918678954.082031 out_system_timestamp = 1609918678954.120117 component latency= 0.038086
Comp name = demuxer in_system_timestamp = 1609918678954.128906 out_system_timestamp = 1609918678954.164062 component latency= 0.035156
Comp name = osd_conv in_system_timestamp = 1609918678954.354980 out_system_timestamp = 1609918678955.071045 component latency= 0.716064
Comp name = nvosd0 in_system_timestamp = 1609918678955.230957 out_system_timestamp = 1609918678960.770996 component latency= 5.540039
Source id = 0 Frame_num = 1 Frame latency = 82.291016 (ms)

BATCH-NUM = 2**
Batch meta not found for buffer 0x7fbbd40838b0
BATCH-NUM = 3**
Batch meta not found for buffer 0x7fbbd4084ae0
BATCH-NUM = 2**
Comp name = nvv4l2decoder1 in_system_timestamp = 1609918678878.592041 out_system_timestamp = 1609918678953.545898 component latency= 74.953857
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 2 in_system_timestamp = 1609918678953.656006 out_system_timestamp = 1609918678993.895996 component_latency = 40.239990
Comp name = primary_gie in_system_timestamp = 1609918678994.006104 out_system_timestamp = 1609918679002.895996 component latency= 8.889893
Comp name = tracking_tracker in_system_timestamp = 1609918679002.916016 out_system_timestamp = 1609918679003.325928 component latency= 0.409912
Comp name = dsanalytics0 in_system_timestamp = 1609918679003.366943 out_system_timestamp = 1609918679003.425049 component latency= 0.058105
Comp name = demuxer in_system_timestamp = 1609918679003.438965 out_system_timestamp = 1609918679003.504883 component latency= 0.065918
Comp name = osd_conv in_system_timestamp = 1609918679003.666992 out_system_timestamp = 1609918679003.845947 component latency= 0.178955
Comp name = nvosd0 in_system_timestamp = 1609918679004.009033 out_system_timestamp = 1609918679007.936035 component latency= 3.927002
Source id = 0 Frame_num = 2 Frame latency = 129.383057 (ms)

BATCH-NUM = 3**
Comp name = nvv4l2decoder1 in_system_timestamp = 1609918678879.896973 out_system_timestamp = 1609918679031.507080 component latency= 151.610107
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 3 in_system_timestamp = 1609918679031.620117 out_system_timestamp = 1609918679031.721924 component_latency = 0.101807
Comp name = primary_gie in_system_timestamp = 1609918679031.832031 out_system_timestamp = 1609918679048.607910 component latency= 16.775879
Comp name = tracking_tracker in_system_timestamp = 1609918679048.626953 out_system_timestamp = 1609918679049.178955 component latency= 0.552002
Comp name = dsanalytics0 in_system_timestamp = 1609918679049.268066 out_system_timestamp = 1609918679049.360107 component latency= 0.092041
Comp name = demuxer in_system_timestamp = 1609918679049.376953 out_system_timestamp = 1609918679049.545898 component latency= 0.168945
Comp name = osd_conv in_system_timestamp = 1609918679049.705078 out_system_timestamp = 1609918679049.930908 component latency= 0.225830
Comp name = nvosd0 in_system_timestamp = 1609918679050.095947 out_system_timestamp = 1609918679056.487061 component latency= 6.391113
Source id = 0 Frame_num = 3 Frame latency = 176.655029 (ms)

BATCH-NUM = 4**
Comp name = nvv4l2decoder1 in_system_timestamp = 1609918678910.426025 out_system_timestamp = 1609918679111.273926 component latency= 200.847900
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 4 in_system_timestamp = 1609918679111.343018 out_system_timestamp = 1609918679111.427002 component_latency = 0.083984
Comp name = primary_gie in_system_timestamp = 1609918679111.535889 out_system_timestamp = 1609918679128.266113 component latency= 16.730225
Comp name = tracking_tracker in_system_timestamp = 1609918679128.283936 out_system_timestamp = 1609918679128.931885 component latency= 0.647949
Comp name = dsanalytics0 in_system_timestamp = 1609918679129.020996 out_system_timestamp = 1609918679129.111084 component latency= 0.090088
Comp name = demuxer in_system_timestamp = 1609918679129.128906 out_system_timestamp = 1609918679129.272949 component latency= 0.144043
Comp name = osd_conv in_system_timestamp = 1609918679129.433105 out_system_timestamp = 1609918679129.685059 component latency= 0.251953
Comp name = nvosd0 in_system_timestamp = 1609918679129.853027 out_system_timestamp = 1609918679136.247070 component latency= 6.394043
Source id = 0 Frame_num = 4 Frame latency = 225.863037 (ms)

BATCH-NUM = 5**
Comp name = nvv4l2decoder1 in_system_timestamp = 1609918678950.784912 out_system_timestamp = 1609918679150.839111 component latency= 200.054199
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 5 in_system_timestamp = 1609918679150.947021 out_system_timestamp = 1609918679151.033936 component_latency = 0.086914
Comp name = primary_gie in_system_timestamp = 1609918679151.101074 out_system_timestamp = 1609918679167.868896 component latency= 16.767822
Comp name = tracking_tracker in_system_timestamp = 1609918679167.885986 out_system_timestamp = 1609918679168.416016 component latency= 0.530029
Comp name = dsanalytics0 in_system_timestamp = 1609918679168.522949 out_system_timestamp = 1609918679168.610107 component latency= 0.087158
Comp name = demuxer in_system_timestamp = 1609918679168.626953 out_system_timestamp = 1609918679168.766113 component latency= 0.139160
Comp name = osd_conv in_system_timestamp = 1609918679168.922119 out_system_timestamp = 1609918679169.175049 component latency= 0.252930
Comp name = nvosd0 in_system_timestamp = 1609918679169.333008 out_system_timestamp = 1609918679175.797119 component latency= 6.464111
Source id = 0 Frame_num = 5 Frame latency = 225.055176 (ms)

BATCH-NUM = 6**
Comp name = nvv4l2decoder1 in_system_timestamp = 1609918679031.094971 out_system_timestamp = 1609918679231.454102 component latency= 200.359131
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 6 in_system_timestamp = 1609918679231.581055 out_system_timestamp = 1609918679231.688965 component_latency = 0.107910
Comp name = primary_gie in_system_timestamp = 1609918679231.752930 out_system_timestamp = 1609918679248.572021 component latency= 16.819092
Comp name = tracking_tracker in_system_timestamp = 1609918679248.590088 out_system_timestamp = 1609918679249.186035 component latency= 0.595947
Comp name = dsanalytics0 in_system_timestamp = 1609918679249.277100 out_system_timestamp = 1609918679249.364990 component latency= 0.087891
Comp name = demuxer in_system_timestamp = 1609918679249.383057 out_system_timestamp = 1609918679249.544922 component latency= 0.161865
Comp name = osd_conv in_system_timestamp = 1609918679249.704102 out_system_timestamp = 1609918679249.944092 component latency= 0.239990
Comp name = nvosd0 in_system_timestamp = 1609918679250.107910 out_system_timestamp = 1609918679256.528076 component latency= 6.420166
Source id = 0 Frame_num = 6 Frame latency = 225.474121 (ms)

BATCH-NUM = 7**
Comp name = nvv4l2decoder1 in_system_timestamp = 1609918679110.853027 out_system_timestamp = 1609918679311.164062 component latency= 200.311035
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 7 in_system_timestamp = 1609918679311.272949 out_system_timestamp = 1609918679311.368896 component_latency = 0.095947
Comp name = primary_gie in_system_timestamp = 1609918679311.434082 out_system_timestamp = 1609918679328.202881 component latency= 16.768799
Comp name = tracking_tracker in_system_timestamp = 1609918679328.217041 out_system_timestamp = 1609918679328.775879 component latency= 0.558838
Comp name = dsanalytics0 in_system_timestamp = 1609918679328.875977 out_system_timestamp = 1609918679328.964111 component latency= 0.088135
Comp name = demuxer in_system_timestamp = 1609918679328.980957 out_system_timestamp = 1609918679329.127930 component latency= 0.146973
Comp name = osd_conv in_system_timestamp = 1609918679329.288086 out_system_timestamp = 1609918679329.552002 component latency= 0.263916
Comp name = nvosd0 in_system_timestamp = 1609918679329.715088 out_system_timestamp = 1609918679336.196045 component latency= 6.480957
Source id = 0 Frame_num = 7 Frame latency = 225.385010 (ms)

BATCH-NUM = 8**
Comp name = nvv4l2decoder1 in_system_timestamp = 1609918679150.458984 out_system_timestamp = 1609918679351.094971 component latency= 200.635986
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 8 in_system_timestamp = 1609918679351.202881 out_system_timestamp = 1609918679351.297119 component_latency = 0.094238
Comp name = primary_gie in_system_timestamp = 1609918679351.363037 out_system_timestamp = 1609918679368.197021 component latency= 16.833984
Comp name = tracking_tracker in_system_timestamp = 1609918679368.215088 out_system_timestamp = 1609918679368.747070 component latency= 0.531982
Comp name = dsanalytics0 in_system_timestamp = 1609918679368.798096 out_system_timestamp = 1609918679368.884033 component latency= 0.085938
Comp name = demuxer in_system_timestamp = 1609918679368.899902 out_system_timestamp = 1609918679369.041016 component latency= 0.141113
Comp name = osd_conv in_system_timestamp = 1609918679369.197998 out_system_timestamp = 1609918679369.400879 component latency= 0.202881
Comp name = nvosd0 in_system_timestamp = 1609918679369.565918 out_system_timestamp = 1609918679376.028076 component latency= 6.462158
Source id = 0 Frame_num = 8 Frame latency = 225.611084 (ms)

BATCH-NUM = 9**
Comp name = nvv4l2decoder1 in_system_timestamp = 1609918679231.056885 out_system_timestamp = 1609918679431.218994 component latency= 200.162109
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 9 in_system_timestamp = 1609918679431.397949 out_system_timestamp = 1609918679431.490967 component_latency = 0.093018
Comp name = primary_gie in_system_timestamp = 1609918679431.540039 out_system_timestamp = 1609918679448.355957 component latency= 16.815918
Comp name = tracking_tracker in_system_timestamp = 1609918679448.374023 out_system_timestamp = 1609918679448.992920 component latency= 0.618896
Comp name = dsanalytics0 in_system_timestamp = 1609918679449.083984 out_system_timestamp = 1609918679449.173096 component latency= 0.089111
Comp name = demuxer in_system_timestamp = 1609918679449.189941 out_system_timestamp = 1609918679449.334961 component latency= 0.145020
Comp name = osd_conv in_system_timestamp = 1609918679449.497070 out_system_timestamp = 1609918679449.790039 component latency= 0.292969
Comp name = nvosd0 in_system_timestamp = 1609918679449.955078 out_system_timestamp = 1609918679456.438965 component latency= 6.483887
Source id = 0 Frame_num = 9 Frame latency = 225.424072 (ms)

BATCH-NUM = 10**
Comp name = nvv4l2decoder1 in_system_timestamp = 1609918679310.803955 out_system_timestamp = 1609918679511.331055 component latency= 200.527100
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 10 in_system_timestamp = 1609918679511.499023 out_system_timestamp = 1609918679511.591064 component_latency = 0.092041
Comp name = primary_gie in_system_timestamp = 1609918679511.638916 out_system_timestamp = 1609918679528.275879 component latency= 16.636963
Comp name = tracking_tracker in_system_timestamp = 1609918679528.291992 out_system_timestamp = 1609918679528.877930 component latency= 0.585938
Comp name = dsanalytics0 in_system_timestamp = 1609918679528.916992 out_system_timestamp = 1609918679529.003906 component latency= 0.086914
Comp name = demuxer in_system_timestamp = 1609918679529.019043 out_system_timestamp = 1609918679529.145996 component latency= 0.126953
Comp name = osd_conv in_system_timestamp = 1609918679529.304932 out_system_timestamp = 1609918679529.553955 component latency= 0.249023
Comp name = nvosd0 in_system_timestamp = 1609918679529.718994 out_system_timestamp = 1609918679539.125977 component latency= 9.406982
Source id = 0 Frame_num = 10 Frame latency = 228.369141 (ms)

BATCH-NUM = 11**
Comp name = nvv4l2decoder1 in_system_timestamp = 1609918679350.752930 out_system_timestamp = 1609918679551.354004 component latency= 200.601074
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 11 in_system_timestamp = 1609918679551.519043 out_system_timestamp = 1609918679551.613037 component_latency = 0.093994
Comp name = primary_gie in_system_timestamp = 1609918679551.664062 out_system_timestamp = 1609918679568.410889 component latency= 16.746826
Comp name = tracking_tracker in_system_timestamp = 1609918679568.427002 out_system_timestamp = 1609918679568.937988 component latency= 0.510986
Comp name = dsanalytics0 in_system_timestamp = 1609918679568.989014 out_system_timestamp = 1609918679569.079102 component latency= 0.090088
Comp name = demuxer in_system_timestamp = 1609918679569.094971 out_system_timestamp = 1609918679569.266113 component latency= 0.171143
Comp name = osd_conv in_system_timestamp = 1609918679569.416992 out_system_timestamp = 1609918679569.600098 component latency= 0.183105
Comp name = nvosd0 in_system_timestamp = 1609918679569.773926 out_system_timestamp = 1609918679576.114014 component latency= 6.340088
Source id = 0 Frame_num = 11 Frame latency = 225.407959 (ms)

BATCH-NUM = 12**
Comp name = nvv4l2decoder1 in_system_timestamp = 1609918679430.871094 out_system_timestamp = 1609918679632.470947 component latency= 201.599854
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 12 in_system_timestamp = 1609918679632.666016 out_system_timestamp = 1609918679632.760010 component_latency = 0.093994
Comp name = primary_gie in_system_timestamp = 1609918679632.809082 out_system_timestamp = 1609918679649.579102 component latency= 16.770020
Comp name = tracking_tracker in_system_timestamp = 1609918679649.596924 out_system_timestamp = 1609918679650.083008 component latency= 0.486084
Comp name = dsanalytics0 in_system_timestamp = 1609918679650.172119 out_system_timestamp = 1609918679650.261963 component latency= 0.089844
Comp name = demuxer in_system_timestamp = 1609918679650.278076 out_system_timestamp = 1609918679650.416992 component latency= 0.138916
Comp name = osd_conv in_system_timestamp = 1609918679650.572998 out_system_timestamp = 1609918679650.824951 component latency= 0.251953
Comp name = nvosd0 in_system_timestamp = 1609918679650.993896 out_system_timestamp = 1609918679657.377930 component latency= 6.384033
Source id = 0 Frame_num = 12 Frame latency = 226.552979 (ms)

BATCH-NUM = 13**
Comp name = nvv4l2decoder1 in_system_timestamp = 1609918679510.972900 out_system_timestamp = 1609918679838.851074 component latency= 327.878174
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 13 in_system_timestamp = 1609918679839.023926 out_system_timestamp = 1609918679839.168945 component_latency = 0.145020
Comp name = primary_gie in_system_timestamp = 1609918679839.218018 out_system_timestamp = 1609918679857.125977 component latency= 17.907959
Comp name = tracking_tracker in_system_timestamp = 1609918679857.145020 out_system_timestamp = 1609918679860.666992 component latency= 3.521973
Comp name = dsanalytics0 in_system_timestamp = 1609918679860.759033 out_system_timestamp = 1609918679860.854980 component latency= 0.095947
Comp name = demuxer in_system_timestamp = 1609918679860.872070 out_system_timestamp = 1609918679861.031006 component latency= 0.158936
Comp name = osd_conv in_system_timestamp = 1609918679861.198975 out_system_timestamp = 1609918679862.032959 component latency= 0.833984
Comp name = nvosd0 in_system_timestamp = 1609918679862.197021 out_system_timestamp = 1609918679867.191895 component latency= 4.994873
Source id = 0 Frame_num = 13 Frame latency = 356.260010 (ms)

BATCH-NUM = 14**
Comp name = nvv4l2decoder1 in_system_timestamp = 1609918679550.990967 out_system_timestamp = 1609918679838.920898 component latency= 287.929932
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 14 in_system_timestamp = 1609918679839.058105 out_system_timestamp = 1609918679839.318115 component_latency = 0.260010
Comp name = primary_gie in_system_timestamp = 1609918679839.594971 out_system_timestamp = 1609918679872.543945 component latency= 32.948975
Comp name = tracking_tracker in_system_timestamp = 1609918679872.563965 out_system_timestamp = 1609918679872.989990 component latency= 0.426025
Comp name = dsanalytics0 in_system_timestamp = 1609918679873.082031 out_system_timestamp = 1609918679873.176025 component latency= 0.093994
Comp name = demuxer in_system_timestamp = 1609918679873.193115 out_system_timestamp = 1609918679873.360107 component latency= 0.166992
Comp name = osd_conv in_system_timestamp = 1609918679873.478027 out_system_timestamp = 1609918679873.752930 component latency= 0.274902
Comp name = nvosd0 in_system_timestamp = 1609918679873.906982 out_system_timestamp = 1609918679878.851074 component latency= 4.944092
Source id = 0 Frame_num = 14 Frame latency = 327.895020 (ms)

BATCH-NUM = 15**
Comp name = nvv4l2decoder1 in_system_timestamp = 1609918679632.083984 out_system_timestamp = 1609918679840.540039 component latency= 208.456055
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 15 in_system_timestamp = 1609918679840.696045 out_system_timestamp = 1609918679880.993896 component_latency = 40.297852
Comp name = primary_gie in_system_timestamp = 1609918679881.102051 out_system_timestamp = 1609918679890.211914 component latency= 9.109863
Comp name = tracking_tracker in_system_timestamp = 1609918679890.230957 out_system_timestamp = 1609918679890.656982 component latency= 0.426025
Comp name = dsanalytics0 in_system_timestamp = 1609918679890.748047 out_system_timestamp = 1609918679890.815918 component latency= 0.067871
Comp name = demuxer in_system_timestamp = 1609918679890.832031 out_system_timestamp = 1609918679890.917969 component latency= 0.085938
Comp name = osd_conv in_system_timestamp = 1609918679891.083984 out_system_timestamp = 1609918679891.258057 component latency= 0.174072
Comp name = nvosd0 in_system_timestamp = 1609918679891.430908 out_system_timestamp = 1609918679895.314941 component latency= 3.884033
Source id = 0 Frame_num = 15 Frame latency = 263.269043 (ms)

BATCH-NUM = 16**
Comp name = nvv4l2decoder1 in_system_timestamp = 1609918679836.805908 out_system_timestamp = 1609918679910.870117 component latency= 74.064209
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 16 in_system_timestamp = 1609918679911.042969 out_system_timestamp = 1609918679911.186035 component_latency = 0.143066
Comp name = primary_gie in_system_timestamp = 1609918679911.231934 out_system_timestamp = 1609918679929.674072 component latency= 18.442139
Comp name = tracking_tracker in_system_timestamp = 1609918679929.695068 out_system_timestamp = 1609918679930.175049 component latency= 0.479980
Comp name = dsanalytics0 in_system_timestamp = 1609918679930.270020 out_system_timestamp = 1609918679930.364990 component latency= 0.094971
Comp name = demuxer in_system_timestamp = 1609918679930.382080 out_system_timestamp = 1609918679930.538086 component latency= 0.156006
Comp name = osd_conv in_system_timestamp = 1609918679930.701904 out_system_timestamp = 1609918679930.892090 component latency= 0.190186
Comp name = nvosd0 in_system_timestamp = 1609918679931.069092 out_system_timestamp = 1609918679936.052002 component latency= 4.982910
Source id = 0 Frame_num = 16 Frame latency = 99.285156 (ms)

BATCH-NUM = 17**
Comp name = nvv4l2decoder1 in_system_timestamp = 1609918679837.003906 out_system_timestamp = 1609918679951.180908 component latency= 114.177002
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 17 in_system_timestamp = 1609918679951.343018 out_system_timestamp = 1609918679951.485107 component_latency = 0.142090
Comp name = primary_gie in_system_timestamp = 1609918679951.525879 out_system_timestamp = 1609918679968.972900 component latency= 17.447021
Comp name = tracking_tracker in_system_timestamp = 1609918679968.993896 out_system_timestamp = 1609918679969.478027 component latency= 0.484131
Comp name = dsanalytics0 in_system_timestamp = 1609918679969.575928 out_system_timestamp = 1609918679969.676025 component latency= 0.100098
Comp name = demuxer in_system_timestamp = 1609918679969.694092 out_system_timestamp = 1609918679969.879883 component latency= 0.185791
Comp name = osd_conv in_system_timestamp = 1609918679970.058105 out_system_timestamp = 1609918679970.235107 component latency= 0.177002
Comp name = nvosd0 in_system_timestamp = 1609918679970.408936 out_system_timestamp = 1609918679976.885010 component latency= 6.476074
Source id = 0 Frame_num = 17 Frame latency = 139.923096 (ms)

BATCH-NUM = 18**
Comp name = nvv4l2decoder1 in_system_timestamp = 1609918679839.801025 out_system_timestamp = 1609918680030.602051 component latency= 190.801025
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 18 in_system_timestamp = 1609918680030.775879 out_system_timestamp = 1609918680030.912109 component_latency = 0.136230
Comp name = primary_gie in_system_timestamp = 1609918680030.965088 out_system_timestamp = 1609918680049.361084 component latency= 18.395996
Comp name = tracking_tracker in_system_timestamp = 1609918680049.383057 out_system_timestamp = 1609918680049.879883 component latency= 0.496826
Comp name = dsanalytics0 in_system_timestamp = 1609918680049.978027 out_system_timestamp = 1609918680050.073975 component latency= 0.095947
Comp name = demuxer in_system_timestamp = 1609918680050.092041 out_system_timestamp = 1609918680050.254883 component latency= 0.162842
Comp name = osd_conv in_system_timestamp = 1609918680050.420898 out_system_timestamp = 1609918680050.616943 component latency= 0.196045
Comp name = nvosd0 in_system_timestamp = 1609918680050.793945 out_system_timestamp = 1609918680055.772949 component latency= 4.979004
Source id = 0 Frame_num = 18 Frame latency = 216.010010 (ms)

BATCH-NUM = 4**
Batch meta not found for buffer 0x7fbbd4083be0
BATCH-NUM = 19**
Comp name = nvv4l2decoder1 in_system_timestamp = 1609918679910.569092 out_system_timestamp = 1609918680110.750000 component latency= 200.180908
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 19 in_system_timestamp = 1609918680110.917969 out_system_timestamp = 1609918680111.054932 component_latency = 0.136963
Comp name = primary_gie in_system_timestamp = 1609918680111.107910 out_system_timestamp = 1609918680129.280029 component latency= 18.172119
Comp name = tracking_tracker in_system_timestamp = 1609918680129.302002 out_system_timestamp = 1609918680129.821045 component latency= 0.519043
Comp name = dsanalytics0 in_system_timestamp = 1609918680129.917969 out_system_timestamp = 1609918680130.021973 component latency= 0.104004
Comp name = demuxer in_system_timestamp = 1609918680130.040039 out_system_timestamp = 1609918680130.199951 component latency= 0.159912
Comp name = osd_conv in_system_timestamp = 1609918680130.364014 out_system_timestamp = 1609918680130.583984 component latency= 0.219971
Comp name = nvosd0 in_system_timestamp = 1609918680130.753906 out_system_timestamp = 1609918680135.731934 component latency= 4.978027
Source id = 0 Frame_num = 19 Frame latency = 225.200928 (ms)

BATCH-NUM = 20**
Comp name = nvv4l2decoder1 in_system_timestamp = 1609918679950.878906 out_system_timestamp = 1609918680150.988037 component latency= 200.109131
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 20 in_system_timestamp = 1609918680151.168945 out_system_timestamp = 1609918680151.314941 component_latency = 0.145996
Comp name = primary_gie in_system_timestamp = 1609918680151.364014 out_system_timestamp = 1609918680170.261963 component latency= 18.897949
Comp name = tracking_tracker in_system_timestamp = 1609918680170.282959 out_system_timestamp = 1609918680170.760010 component latency= 0.477051
Comp name = dsanalytics0 in_system_timestamp = 1609918680170.850098 out_system_timestamp = 1609918680170.940918 component latency= 0.090820
Comp name = demuxer in_system_timestamp = 1609918680170.958008 out_system_timestamp = 1609918680171.115967 component latency= 0.157959
Comp name = osd_conv in_system_timestamp = 1609918680171.273926 out_system_timestamp = 1609918680171.517090 component latency= 0.243164
Comp name = nvosd0 in_system_timestamp = 1609918680171.684082 out_system_timestamp = 1609918680178.122070 component latency= 6.437988
Source id = 0 Frame_num = 20 Frame latency = 227.285156 (ms)

BATCH-NUM = 5**
Batch meta not found for buffer 0x7fbbd4083360
BATCH-NUM = 21**
Comp name = nvv4l2decoder1 in_system_timestamp = 1609918680030.294922 out_system_timestamp = 1609918680231.034912 component latency= 200.739990
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 21 in_system_timestamp = 1609918680231.200928 out_system_timestamp = 1609918680231.337891 component_latency = 0.136963
Comp name = primary_gie in_system_timestamp = 1609918680231.387939 out_system_timestamp = 1609918680249.422119 component latency= 18.034180
Comp name = tracking_tracker in_system_timestamp = 1609918680249.443115 out_system_timestamp = 1609918680249.983887 component latency= 0.540771
Comp name = dsanalytics0 in_system_timestamp = 1609918680250.080078 out_system_timestamp = 1609918680250.174072 component latency= 0.093994
Comp name = demuxer in_system_timestamp = 1609918680250.191895 out_system_timestamp = 1609918680250.357910 component latency= 0.166016
Comp name = osd_conv in_system_timestamp = 1609918680250.527100 out_system_timestamp = 1609918680250.739990 component latency= 0.212891
Comp name = nvosd0 in_system_timestamp = 1609918680250.927002 out_system_timestamp = 1609918680255.920898 component latency= 4.993896
Source id = 0 Frame_num = 21 Frame latency = 225.665039 (ms)

BATCH-NUM = 22**
Comp name = nvv4l2decoder1 in_system_timestamp = 1609918680110.455078 out_system_timestamp = 1609918680310.446045 component latency= 199.990967
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 22 in_system_timestamp = 1609918680310.612061 out_system_timestamp = 1609918680310.760986 component_latency = 0.148926
Comp name = primary_gie in_system_timestamp = 1609918680310.812012 out_system_timestamp = 1609918680330.011963 component latency= 19.199951
Comp name = tracking_tracker in_system_timestamp = 1609918680330.033936 out_system_timestamp = 1609918680330.545898 component latency= 0.511963
Comp name = dsanalytics0 in_system_timestamp = 1609918680330.644043 out_system_timestamp = 1609918680330.740967 component latency= 0.096924
Comp name = demuxer in_system_timestamp = 1609918680330.758057 out_system_timestamp = 1609918680330.958984 component latency= 0.200928
Comp name = osd_conv in_system_timestamp = 1609918680331.129883 out_system_timestamp = 1609918680331.352051 component latency= 0.222168
Comp name = nvosd0 in_system_timestamp = 1609918680331.525879 out_system_timestamp = 1609918680336.528076 component latency= 5.002197
Source id = 0 Frame_num = 22 Frame latency = 226.114014 (ms)

BATCH-NUM = 23**
Comp name = nvv4l2decoder1 in_system_timestamp = 1609918680150.694092 out_system_timestamp = 1609918680350.753906 component latency= 200.059814
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 23 in_system_timestamp = 1609918680350.846924 out_system_timestamp = 1609918680351.000000 component_latency = 0.153076
Comp name = primary_gie in_system_timestamp = 1609918680351.041016 out_system_timestamp = 1609918680370.579102 component latency= 19.538086
Comp name = tracking_tracker in_system_timestamp = 1609918680370.600098 out_system_timestamp = 1609918680371.086914 component latency= 0.486816
Comp name = dsanalytics0 in_system_timestamp = 1609918680371.180908 out_system_timestamp = 1609918680371.280029 component latency= 0.099121
Comp name = demuxer in_system_timestamp = 1609918680371.297119 out_system_timestamp = 1609918680371.450928 component latency= 0.153809
Comp name = osd_conv in_system_timestamp = 1609918680371.611084 out_system_timestamp = 1609918680371.818115 component latency= 0.207031
Comp name = nvosd0 in_system_timestamp = 1609918680371.991943 out_system_timestamp = 1609918680378.457031 component latency= 6.465088
Source id = 0 Frame_num = 23 Frame latency = 227.804932 (ms)

BATCH-NUM = 24**
Comp name = nvv4l2decoder1 in_system_timestamp = 1609918680230.741943 out_system_timestamp = 1609918680431.055908 component latency= 200.313965
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 24 in_system_timestamp = 1609918680431.222900 out_system_timestamp = 1609918680431.370117 component_latency = 0.147217
Comp name = primary_gie in_system_timestamp = 1609918680431.409912 out_system_timestamp = 1609918680449.868896 component latency= 18.458984
Comp name = tracking_tracker in_system_timestamp = 1609918680449.891113 out_system_timestamp = 1609918680450.393066 component latency= 0.501953
Comp name = dsanalytics0 in_system_timestamp = 1609918680450.491943 out_system_timestamp = 1609918680450.587891 component latency= 0.095947
Comp name = demuxer in_system_timestamp = 1609918680450.605957 out_system_timestamp = 1609918680450.769043 component latency= 0.163086
Comp name = osd_conv in_system_timestamp = 1609918680450.934082 out_system_timestamp = 1609918680451.147949 component latency= 0.213867
Comp name = nvosd0 in_system_timestamp = 1609918680451.325928 out_system_timestamp = 1609918680456.279053 component latency= 4.953125
Source id = 0 Frame_num = 24 Frame latency = 225.576172 (ms)

BATCH-NUM = 6**
Batch meta not found for buffer 0x7fbbd4086c10
BATCH-NUM = 25**
Comp name = nvv4l2decoder1 in_system_timestamp = 1609918680310.149902 out_system_timestamp = 1609918680512.287109 component latency= 202.137207
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 25 in_system_timestamp = 1609918680512.451904 out_system_timestamp = 1609918680512.638916 component_latency = 0.187012
Comp name = primary_gie in_system_timestamp = 1609918680512.689941 out_system_timestamp = 1609918680529.658936 component latency= 16.968994
Comp name = tracking_tracker in_system_timestamp = 1609918680529.676025 out_system_timestamp = 1609918680530.221924 component latency= 0.545898
Comp name = dsanalytics0 in_system_timestamp = 1609918680530.322998 out_system_timestamp = 1609918680530.427979 component latency= 0.104980
Comp name = demuxer in_system_timestamp = 1609918680530.447021 out_system_timestamp = 1609918680530.642090 component latency= 0.195068
Comp name = osd_conv in_system_timestamp = 1609918680530.816895 out_system_timestamp = 1609918680531.052979 component latency= 0.236084
Comp name = nvosd0 in_system_timestamp = 1609918680531.173096 out_system_timestamp = 1609918680539.297119 component latency= 8.124023
Source id = 0 Frame_num = 25 Frame latency = 229.196045 (ms)

BATCH-NUM = 26**
Comp name = nvv4l2decoder1 in_system_timestamp = 1609918680350.470947 out_system_timestamp = 1609918680550.654053 component latency= 200.183105
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 26 in_system_timestamp = 1609918680550.772949 out_system_timestamp = 1609918680550.898926 component_latency = 0.125977
Comp name = primary_gie in_system_timestamp = 1609918680550.947021 out_system_timestamp = 1609918680570.972900 component latency= 20.025879
Comp name = tracking_tracker in_system_timestamp = 1609918680570.990967 out_system_timestamp = 1609918680571.511963 component latency= 0.520996
Comp name = dsanalytics0 in_system_timestamp = 1609918680571.602051 out_system_timestamp = 1609918680571.695068 component latency= 0.093018
Comp name = demuxer in_system_timestamp = 1609918680571.712891 out_system_timestamp = 1609918680571.874023 component latency= 0.161133
Comp name = osd_conv in_system_timestamp = 1609918680572.037109 out_system_timestamp = 1609918680572.274902 component latency= 0.237793
Comp name = nvosd0 in_system_timestamp = 1609918680572.431885 out_system_timestamp = 1609918680577.402100 component latency= 4.970215
Source id = 0 Frame_num = 26 Frame latency = 226.969971 (ms)

BATCH-NUM = 7**
Batch meta not found for buffer 0x7fbbd40858d0
BATCH-NUM = 27**
Comp name = nvv4l2decoder1 in_system_timestamp = 1609918680430.760986 out_system_timestamp = 1609918680631.037109 component latency= 200.276123
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 27 in_system_timestamp = 1609918680631.211914 out_system_timestamp = 1609918680631.353027 component_latency = 0.141113
Comp name = primary_gie in_system_timestamp = 1609918680631.403076 out_system_timestamp = 1609918680650.563965 component latency= 19.160889
Comp name = tracking_tracker in_system_timestamp = 1609918680650.582031 out_system_timestamp = 1609918680651.113037 component latency= 0.531006
Comp name = dsanalytics0 in_system_timestamp = 1609918680651.218994 out_system_timestamp = 1609918680651.316895 component latency= 0.097900
Comp name = demuxer in_system_timestamp = 1609918680651.335938 out_system_timestamp = 1609918680651.543945 component latency= 0.208008
Comp name = osd_conv in_system_timestamp = 1609918680651.708984 out_system_timestamp = 1609918680651.892090 component latency= 0.183105
Comp name = nvosd0 in_system_timestamp = 1609918680652.064941 out_system_timestamp = 1609918680658.501953 component latency= 6.437012
Source id = 0 Frame_num = 27 Frame latency = 227.786133 (ms)

BATCH-NUM = 28**
Comp name = nvv4l2decoder1 in_system_timestamp = 1609918680511.898926 out_system_timestamp = 1609918680711.041016 component latency= 199.142090
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 28 in_system_timestamp = 1609918680711.208984 out_system_timestamp = 1609918680711.351074 component_latency = 0.142090
Comp name = primary_gie in_system_timestamp = 1609918680711.402100 out_system_timestamp = 1609918680730.639893 component latency= 19.237793
Comp name = tracking_tracker in_system_timestamp = 1609918680730.657959 out_system_timestamp = 1609918680731.159912 component latency= 0.501953
Comp name = dsanalytics0 in_system_timestamp = 1609918680731.253906 out_system_timestamp = 1609918680731.351074 component latency= 0.097168
Comp name = demuxer in_system_timestamp = 1609918680731.370117 out_system_timestamp = 1609918680731.544922 component latency= 0.174805
Comp name = osd_conv in_system_timestamp = 1609918680731.708984 out_system_timestamp = 1609918680731.981934 component latency= 0.272949
Comp name = nvosd0 in_system_timestamp = 1609918680732.105957 out_system_timestamp = 1609918680736.418945 component latency= 4.312988
Source id = 0 Frame_num = 28 Frame latency = 224.552979 (ms)

BATCH-NUM = 29**
Comp name = nvv4l2decoder1 in_system_timestamp = 1609918680550.403076 out_system_timestamp = 1609918680884.269043 component latency= 333.865967
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 29 in_system_timestamp = 1609918680884.426025 out_system_timestamp = 1609918680884.579102 component_latency = 0.153076
Comp name = primary_gie in_system_timestamp = 1609918680884.621094 out_system_timestamp = 1609918680935.745117 component latency= 51.124023
Comp name = tracking_tracker in_system_timestamp = 1609918680935.761963 out_system_timestamp = 1609918680939.952881 component latency= 4.190918
Comp name = dsanalytics0 in_system_timestamp = 1609918680940.042969 out_system_timestamp = 1609918680940.142090 component latency= 0.099121
Comp name = demuxer in_system_timestamp = 1609918680940.158936 out_system_timestamp = 1609918680940.322021 component latency= 0.163086
Comp name = osd_conv in_system_timestamp = 1609918680940.479980 out_system_timestamp = 1609918680941.294922 component latency= 0.814941
Comp name = nvosd0 in_system_timestamp = 1609918680941.402100 out_system_timestamp = 1609918680946.325928 component latency= 4.923828
Source id = 0 Frame_num = 29 Frame latency = 395.958008 (ms)

BATCH-NUM = 30**
Comp name = nvv4l2decoder1 in_system_timestamp = 1609918680630.706055 out_system_timestamp = 1609918680884.400879 component latency= 253.694824
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 30 in_system_timestamp = 1609918680884.459961 out_system_timestamp = 1609918680886.656006 component_latency = 2.196045
Comp name = primary_gie in_system_timestamp = 1609918680886.693115 out_system_timestamp = 1609918680951.924072 component latency= 65.230957
Comp name = tracking_tracker in_system_timestamp = 1609918680951.937988 out_system_timestamp = 1609918680955.449951 component latency= 3.511963
Comp name = dsanalytics0 in_system_timestamp = 1609918680955.540039 out_system_timestamp = 1609918680955.635986 component latency= 0.095947
Comp name = demuxer in_system_timestamp = 1609918680955.654053 out_system_timestamp = 1609918680955.731934 component latency= 0.077881
Comp name = osd_conv in_system_timestamp = 1609918680955.906006 out_system_timestamp = 1609918680956.764893 component latency= 0.858887
Comp name = nvosd0 in_system_timestamp = 1609918680956.929932 out_system_timestamp = 1609918680962.706055 component latency= 5.776123
Source id = 0 Frame_num = 30 Frame latency = 332.040039 (ms)

BATCH-NUM = 8**
Batch meta not found for buffer 0x7fbbd407f950
BATCH-NUM = 9**
Batch meta not found for buffer 0x7fbbd4087c20
BATCH-NUM = 31**
Comp name = nvv4l2decoder1 in_system_timestamp = 1609918680710.749023 out_system_timestamp = 1609918680910.760010 component latency= 200.010986
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 31 in_system_timestamp = 1609918680910.908936 out_system_timestamp = 1609918680926.343018 component_latency = 15.434082
Comp name = primary_gie in_system_timestamp = 1609918680926.389893 out_system_timestamp = 1609918680967.370117 component latency= 40.980225
Comp name = tracking_tracker in_system_timestamp = 1609918680967.387939 out_system_timestamp = 1609918680968.038086 component latency= 0.650146
Comp name = dsanalytics0 in_system_timestamp = 1609918680968.134033 out_system_timestamp = 1609918680968.229980 component latency= 0.095947
Comp name = demuxer in_system_timestamp = 1609918680968.247070 out_system_timestamp = 1609918680968.326904 component latency= 0.079834
Comp name = osd_conv in_system_timestamp = 1609918680968.496094 out_system_timestamp = 1609918680968.739990 component latency= 0.243896
Comp name = nvosd0 in_system_timestamp = 1609918680968.893066 out_system_timestamp = 1609918680974.540039 component latency= 5.646973
Source id = 0 Frame_num = 31 Frame latency = 263.822998 (ms)

BATCH-NUM = 10**
Batch meta not found for buffer 0x7fbbd4083580
BATCH-NUM = 11**
Batch meta not found for buffer 0x7fbbec078b60
BATCH-NUM = 32**
Comp name = nvv4l2decoder1 in_system_timestamp = 1609918680882.135986 out_system_timestamp = 1609918680951.189941 component latency= 69.053955
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 32 in_system_timestamp = 1609918680951.352051 out_system_timestamp = 1609918680991.554932 component_latency = 40.202881
Comp name = primary_gie in_system_timestamp = 1609918680991.591064 out_system_timestamp = 1609918681000.476074 component latency= 8.885010
Comp name = tracking_tracker in_system_timestamp = 1609918681000.492920 out_system_timestamp = 1609918681000.895020 component latency= 0.402100
Comp name = dsanalytics0 in_system_timestamp = 1609918681000.946045 out_system_timestamp = 1609918681001.010986 component latency= 0.064941
Comp name = demuxer in_system_timestamp = 1609918681001.027100 out_system_timestamp = 1609918681001.114990 component latency= 0.087891
Comp name = osd_conv in_system_timestamp = 1609918681001.282959 out_system_timestamp = 1609918681001.457031 component latency= 0.174072
Comp name = nvosd0 in_system_timestamp = 1609918681001.626953 out_system_timestamp = 1609918681005.349121 component latency= 3.722168
Source id = 0 Frame_num = 32 Frame latency = 123.258057 (ms)

BATCH-NUM = 33**
Comp name = nvv4l2decoder1 in_system_timestamp = 1609918680882.345947 out_system_timestamp = 1609918681031.344971 component latency= 148.999023
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 33 in_system_timestamp = 1609918681031.541992 out_system_timestamp = 1609918681031.685059 component_latency = 0.143066
Comp name = primary_gie in_system_timestamp = 1609918681031.736084 out_system_timestamp = 1609918681051.381104 component latency= 19.645020
Comp name = tracking_tracker in_system_timestamp = 1609918681051.397949 out_system_timestamp = 1609918681051.870117 component latency= 0.472168
Comp name = dsanalytics0 in_system_timestamp = 1609918681051.961914 out_system_timestamp = 1609918681052.054932 component latency= 0.093018
Comp name = demuxer in_system_timestamp = 1609918681052.072021 out_system_timestamp = 1609918681052.225098 component latency= 0.153076
Comp name = osd_conv in_system_timestamp = 1609918681052.387939 out_system_timestamp = 1609918681052.652100 component latency= 0.264160
Comp name = nvosd0 in_system_timestamp = 1609918681052.825928 out_system_timestamp = 1609918681059.289062 component latency= 6.463135
Source id = 0 Frame_num = 33 Frame latency = 176.985107 (ms)

BATCH-NUM = 12**
Batch meta not found for buffer 0x7fbbd40877e0
BATCH-NUM = 34**
Comp name = nvv4l2decoder1 in_system_timestamp = 1609918680910.480957 out_system_timestamp = 1609918681110.983887 component latency= 200.502930
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 34 in_system_timestamp = 1609918681111.150879 out_system_timestamp = 1609918681111.284912 component_latency = 0.134033
Comp name = primary_gie in_system_timestamp = 1609918681111.335938 out_system_timestamp = 1609918681131.642090 component latency= 20.306152
Comp name = tracking_tracker in_system_timestamp = 1609918681131.659912 out_system_timestamp = 1609918681132.146973 component latency= 0.487061
Comp name = dsanalytics0 in_system_timestamp = 1609918681132.242920 out_system_timestamp = 1609918681132.336914 component latency= 0.093994
Comp name = demuxer in_system_timestamp = 1609918681132.354980 out_system_timestamp = 1609918681132.543945 component latency= 0.188965
Comp name = osd_conv in_system_timestamp = 1609918681132.714111 out_system_timestamp = 1609918681132.905029 component latency= 0.190918
Comp name = nvosd0 in_system_timestamp = 1609918681133.079102 out_system_timestamp = 1609918681139.574951 component latency= 6.495850
Source id = 0 Frame_num = 34 Frame latency = 229.135010 (ms)

BATCH-NUM = 35**
Comp name = nvv4l2decoder1 in_system_timestamp = 1609918680950.503906 out_system_timestamp = 1609918681150.860107 component latency= 200.356201
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 35 in_system_timestamp = 1609918681151.031006 out_system_timestamp = 1609918681151.179932 component_latency = 0.148926
Comp name = primary_gie in_system_timestamp = 1609918681151.233887 out_system_timestamp = 1609918681167.996094 component latency= 16.762207
Comp name = tracking_tracker in_system_timestamp = 1609918681168.012939 out_system_timestamp = 1609918681168.486084 component latency= 0.473145
Comp name = dsanalytics0 in_system_timestamp = 1609918681168.581055 out_system_timestamp = 1609918681168.699951 component latency= 0.118896
Comp name = demuxer in_system_timestamp = 1609918681168.721924 out_system_timestamp = 1609918681168.927979 component latency= 0.206055
Comp name = osd_conv in_system_timestamp = 1609918681169.094971 out_system_timestamp = 1609918681169.272949 component latency= 0.177979
Comp name = nvosd0 in_system_timestamp = 1609918681169.440918 out_system_timestamp = 1609918681175.947021 component latency= 6.506104
Source id = 0 Frame_num = 35 Frame latency = 225.485107 (ms)

BATCH-NUM = 13**
Batch meta not found for buffer 0x7fbbd4081560
BATCH-NUM = 36**
Comp name = nvv4l2decoder1 in_system_timestamp = 1609918681031.014893 out_system_timestamp = 1609918681231.416016 component latency= 200.401123
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 36 in_system_timestamp = 1609918681231.581055 out_system_timestamp = 1609918681231.710938 component_latency = 0.129883
Comp name = primary_gie in_system_timestamp = 1609918681231.759033 out_system_timestamp = 1609918681251.522949 component latency= 19.763916
Comp name = tracking_tracker in_system_timestamp = 1609918681251.540039 out_system_timestamp = 1609918681252.077881 component latency= 0.537842
Comp name = dsanalytics0 in_system_timestamp = 1609918681252.172119 out_system_timestamp = 1609918681252.283936 component latency= 0.111816
Comp name = demuxer in_system_timestamp = 1609918681252.302002 out_system_timestamp = 1609918681252.468018 component latency= 0.166016
Comp name = osd_conv in_system_timestamp = 1609918681252.637939 out_system_timestamp = 1609918681252.894043 component latency= 0.256104
Comp name = nvosd0 in_system_timestamp = 1609918681253.069092 out_system_timestamp = 1609918681259.643066 component latency= 6.573975
Source id = 0 Frame_num = 36 Frame latency = 228.670166 (ms)

BATCH-NUM = 37**
Comp name = nvv4l2decoder1 in_system_timestamp = 1609918681110.654053 out_system_timestamp = 1609918681311.143066 component latency= 200.489014
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 37 in_system_timestamp = 1609918681311.314941 out_system_timestamp = 1609918681311.447998 component_latency = 0.133057
Comp name = primary_gie in_system_timestamp = 1609918681311.497070 out_system_timestamp = 1609918681328.406006 component latency= 16.908936
Comp name = tracking_tracker in_system_timestamp = 1609918681328.425049 out_system_timestamp = 1609918681328.920898 component latency= 0.495850
Comp name = dsanalytics0 in_system_timestamp = 1609918681328.972900 out_system_timestamp = 1609918681329.097900 component latency= 0.125000
Comp name = demuxer in_system_timestamp = 1609918681329.117920 out_system_timestamp = 1609918681329.316895 component latency= 0.198975
Comp name = osd_conv in_system_timestamp = 1609918681329.479980 out_system_timestamp = 1609918681329.689941 component latency= 0.209961
Comp name = nvosd0 in_system_timestamp = 1609918681329.862061 out_system_timestamp = 1609918681336.417969 component latency= 6.555908
Source id = 0 Frame_num = 37 Frame latency = 225.826904 (ms)

BATCH-NUM = 14**
Batch meta not found for buffer 0x7fbbd407b7a0
BATCH-NUM = 38**
Comp name = nvv4l2decoder1 in_system_timestamp = 1609918681150.537109 out_system_timestamp = 1609918681351.218994 component latency= 200.681885
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 38 in_system_timestamp = 1609918681351.385986 out_system_timestamp = 1609918681351.518066 component_latency = 0.132080
Comp name = primary_gie in_system_timestamp = 1609918681351.571045 out_system_timestamp = 1609918681368.352051 component latency= 16.781006
Comp name = tracking_tracker in_system_timestamp = 1609918681368.368896 out_system_timestamp = 1609918681368.855957 component latency= 0.487061
Comp name = dsanalytics0 in_system_timestamp = 1609918681368.914062 out_system_timestamp = 1609918681369.031006 component latency= 0.116943
Comp name = demuxer in_system_timestamp = 1609918681369.052002 out_system_timestamp = 1609918681369.258057 component latency= 0.206055
Comp name = osd_conv in_system_timestamp = 1609918681369.451904 out_system_timestamp = 1609918681369.641113 component latency= 0.189209
Comp name = nvosd0 in_system_timestamp = 1609918681369.823975 out_system_timestamp = 1609918681376.342041 component latency= 6.518066
Source id = 0 Frame_num = 38 Frame latency = 225.845947 (ms)

BATCH-NUM = 15**
Batch meta not found for buffer 0x7fbbd40819a0
BATCH-NUM = 39**
Comp name = nvv4l2decoder1 in_system_timestamp = 1609918681231.064941 out_system_timestamp = 1609918681431.188965 component latency= 200.124023
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 39 in_system_timestamp = 1609918681431.305908 out_system_timestamp = 1609918681431.439941 component_latency = 0.134033
Comp name = primary_gie in_system_timestamp = 1609918681431.500977 out_system_timestamp = 1609918681448.361084 component latency= 16.860107
Comp name = tracking_tracker in_system_timestamp = 1609918681448.377930 out_system_timestamp = 1609918681448.940918 component latency= 0.562988
Comp name = dsanalytics0 in_system_timestamp = 1609918681449.006104 out_system_timestamp = 1609918681449.128906 component latency= 0.122803
Comp name = demuxer in_system_timestamp = 1609918681449.150879 out_system_timestamp = 1609918681449.347900 component latency= 0.197021
Comp name = osd_conv in_system_timestamp = 1609918681449.531006 out_system_timestamp = 1609918681449.727051 component latency= 0.196045
Comp name = nvosd0 in_system_timestamp = 1609918681449.912109 out_system_timestamp = 1609918681456.402100 component latency= 6.489990
Source id = 0 Frame_num = 39 Frame latency = 225.378174 (ms)

BATCH-NUM = 40**
Comp name = nvv4l2decoder1 in_system_timestamp = 1609918681310.841064 out_system_timestamp = 1609918681511.072021 component latency= 200.230957
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 40 in_system_timestamp = 1609918681511.247070 out_system_timestamp = 1609918681511.382080 component_latency = 0.135010
Comp name = primary_gie in_system_timestamp = 1609918681511.430908 out_system_timestamp = 1609918681528.188965 component latency= 16.758057
Comp name = tracking_tracker in_system_timestamp = 1609918681528.207031 out_system_timestamp = 1609918681528.726074 component latency= 0.519043
Comp name = dsanalytics0 in_system_timestamp = 1609918681528.803955 out_system_timestamp = 1609918681528.924072 component latency= 0.120117
Comp name = demuxer in_system_timestamp = 1609918681528.946045 out_system_timestamp = 1609918681529.187012 component latency= 0.240967
Comp name = osd_conv in_system_timestamp = 1609918681529.364990 out_system_timestamp = 1609918681529.552979 component latency= 0.187988
Comp name = nvosd0 in_system_timestamp = 1609918681529.733887 out_system_timestamp = 1609918681539.020996 component latency= 9.287109
Source id = 0 Frame_num = 40 Frame latency = 228.219971 (ms)

BATCH-NUM = 16**
Batch meta not found for buffer 0x7fbbd4081560
BATCH-NUM = 41**
Comp name = nvv4l2decoder1 in_system_timestamp = 1609918681350.889893 out_system_timestamp = 1609918681550.906982 component latency= 200.017090
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 41 in_system_timestamp = 1609918681551.070068 out_system_timestamp = 1609918681551.212891 component_latency = 0.142822
Comp name = primary_gie in_system_timestamp = 1609918681551.252930 out_system_timestamp = 1609918681568.058105 component latency= 16.805176
Comp name = tracking_tracker in_system_timestamp = 1609918681568.074951 out_system_timestamp = 1609918681568.575928 component latency= 0.500977
Comp name = dsanalytics0 in_system_timestamp = 1609918681568.664062 out_system_timestamp = 1609918681568.747070 component latency= 0.083008
Comp name = demuxer in_system_timestamp = 1609918681568.762939 out_system_timestamp = 1609918681568.909912 component latency= 0.146973
Comp name = osd_conv in_system_timestamp = 1609918681569.076904 out_system_timestamp = 1609918681569.264893 component latency= 0.187988
Comp name = nvosd0 in_system_timestamp = 1609918681569.445068 out_system_timestamp = 1609918681576.000977 component latency= 6.555908
Source id = 0 Frame_num = 41 Frame latency = 225.157227 (ms)

BATCH-NUM = 17**
Batch meta not found for buffer 0x7fbbd4088b20
BATCH-NUM = 42**
Comp name = nvv4l2decoder1 in_system_timestamp = 1609918681430.857910 out_system_timestamp = 1609918681631.027100 component latency= 200.169189
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 42 in_system_timestamp = 1609918681631.195068 out_system_timestamp = 1609918681631.339111 component_latency = 0.144043
Comp name = primary_gie in_system_timestamp = 1609918681631.388916 out_system_timestamp = 1609918681648.356934 component latency= 16.968018
Comp name = tracking_tracker in_system_timestamp = 1609918681648.374023 out_system_timestamp = 1609918681648.882080 component latency= 0.508057
Comp name = dsanalytics0 in_system_timestamp = 1609918681648.943115 out_system_timestamp = 1609918681649.074951 component latency= 0.131836
Comp name = demuxer in_system_timestamp = 1609918681649.097900 out_system_timestamp = 1609918681649.301025 component latency= 0.203125
Comp name = osd_conv in_system_timestamp = 1609918681649.474121 out_system_timestamp = 1609918681649.666992 component latency= 0.192871
Comp name = nvosd0 in_system_timestamp = 1609918681649.843018 out_system_timestamp = 1609918681656.262939 component latency= 6.419922
Source id = 0 Frame_num = 42 Frame latency = 225.448975 (ms)

BATCH-NUM = 43**
Comp name = nvv4l2decoder1 in_system_timestamp = 1609918681510.737061 out_system_timestamp = 1609918681710.956055 component latency= 200.218994
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 43 in_system_timestamp = 1609918681711.107910 out_system_timestamp = 1609918681711.243896 component_latency = 0.135986
Comp name = primary_gie in_system_timestamp = 1609918681711.352051 out_system_timestamp = 1609918681728.228027 component latency= 16.875977
Comp name = tracking_tracker in_system_timestamp = 1609918681728.246094 out_system_timestamp = 1609918681728.752930 component latency= 0.506836
Comp name = dsanalytics0 in_system_timestamp = 1609918681728.795898 out_system_timestamp = 1609918681728.885010 component latency= 0.089111
Comp name = demuxer in_system_timestamp = 1609918681728.900879 out_system_timestamp = 1609918681729.062988 component latency= 0.162109
Comp name = osd_conv in_system_timestamp = 1609918681729.235107 out_system_timestamp = 1609918681729.467041 component latency= 0.231934
Comp name = nvosd0 in_system_timestamp = 1609918681729.639893 out_system_timestamp = 1609918681736.099121 component latency= 6.459229
Source id = 0 Frame_num = 43 Frame latency = 225.420898 (ms)

BATCH-NUM = 18**
Batch meta not found for buffer 0x7fbbd4089c40
BATCH-NUM = 44**
Comp name = nvv4l2decoder1 in_system_timestamp = 1609918681550.577881 out_system_timestamp = 1609918681751.191895 component latency= 200.614014
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 44 in_system_timestamp = 1609918681751.322021 out_system_timestamp = 1609918681751.449951 component_latency = 0.127930
Comp name = primary_gie in_system_timestamp = 1609918681751.502930 out_system_timestamp = 1609918681768.371094 component latency= 16.868164
Comp name = tracking_tracker in_system_timestamp = 1609918681768.387939 out_system_timestamp = 1609918681768.898926 component latency= 0.510986
Comp name = dsanalytics0 in_system_timestamp = 1609918681768.935059 out_system_timestamp = 1609918681769.004883 component latency= 0.069824
Comp name = demuxer in_system_timestamp = 1609918681769.017090 out_system_timestamp = 1609918681769.146973 component latency= 0.129883
Comp name = osd_conv in_system_timestamp = 1609918681769.308105 out_system_timestamp = 1609918681769.563965 component latency= 0.255859
Comp name = nvosd0 in_system_timestamp = 1609918681769.738037 out_system_timestamp = 1609918681776.144043 component latency= 6.406006
Source id = 0 Frame_num = 44 Frame latency = 225.611084 (ms)

BATCH-NUM = 19**
Batch meta not found for buffer 0x7fbbd4088b20
BATCH-NUM = 45**
Comp name = nvv4l2decoder1 in_system_timestamp = 1609918681630.697021 out_system_timestamp = 1609918681977.834961 component latency= 347.137939
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 45 in_system_timestamp = 1609918681977.906982 out_system_timestamp = 1609918681978.043945 component_latency = 0.136963
Comp name = primary_gie in_system_timestamp = 1609918681978.145020 out_system_timestamp = 1609918682047.751953 component latency= 69.606934
Comp name = tracking_tracker in_system_timestamp = 1609918682047.769043 out_system_timestamp = 1609918682048.693115 component latency= 0.924072
Comp name = dsanalytics0 in_system_timestamp = 1609918682048.779053 out_system_timestamp = 1609918682048.872070 component latency= 0.093018
Comp name = demuxer in_system_timestamp = 1609918682048.888916 out_system_timestamp = 1609918682049.043945 component latency= 0.155029
Comp name = osd_conv in_system_timestamp = 1609918682049.239990 out_system_timestamp = 1609918682050.519043 component latency= 1.279053
Comp name = nvosd0 in_system_timestamp = 1609918682050.621094 out_system_timestamp = 1609918682054.922119 component latency= 4.301025
Source id = 0 Frame_num = 45 Frame latency = 424.246094 (ms)

BATCH-NUM = 46**
Comp name = nvv4l2decoder1 in_system_timestamp = 1609918681710.622070 out_system_timestamp = 1609918681978.678955 component latency= 268.056885
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 46 in_system_timestamp = 1609918681978.768066 out_system_timestamp = 1609918682005.985107 component_latency = 27.217041
Comp name = primary_gie in_system_timestamp = 1609918682006.040039 out_system_timestamp = 1609918682048.718018 component latency= 42.677979
Comp name = tracking_tracker in_system_timestamp = 1609918682048.729004 out_system_timestamp = 1609918682049.113037 component latency= 0.384033
Comp name = dsanalytics0 in_system_timestamp = 1609918682049.143066 out_system_timestamp = 1609918682049.201904 component latency= 0.058838
Comp name = demuxer in_system_timestamp = 1609918682049.211914 out_system_timestamp = 1609918682049.374023 component latency= 0.162109
Comp name = osd_conv in_system_timestamp = 1609918682050.709961 out_system_timestamp = 1609918682055.677979 component latency= 4.968018
Comp name = nvosd0 in_system_timestamp = 1609918682055.790039 out_system_timestamp = 1609918682061.820068 component latency= 6.030029
Source id = 0 Frame_num = 46 Frame latency = 351.221924 (ms)

BATCH-NUM = 47**
Comp name = nvv4l2decoder1 in_system_timestamp = 1609918681750.866943 out_system_timestamp = 1609918681979.874023 component latency= 229.007080
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 47 in_system_timestamp = 1609918681979.983887 out_system_timestamp = 1609918682046.260986 component_latency = 66.277100
Comp name = primary_gie in_system_timestamp = 1609918682046.372070 out_system_timestamp = 1609918682057.663086 component latency= 11.291016
Comp name = tracking_tracker in_system_timestamp = 1609918682057.684082 out_system_timestamp = 1609918682062.311035 component latency= 4.626953
Comp name = dsanalytics0 in_system_timestamp = 1609918682062.392090 out_system_timestamp = 1609918682062.455078 component latency= 0.062988
Comp name = demuxer in_system_timestamp = 1609918682062.468994 out_system_timestamp = 1609918682062.553955 component latency= 0.084961
Comp name = osd_conv in_system_timestamp = 1609918682062.745117 out_system_timestamp = 1609918682063.027100 component latency= 0.281982
Comp name = nvosd0 in_system_timestamp = 1609918682063.125000 out_system_timestamp = 1609918682066.202881 component latency= 3.077881
Source id = 0 Frame_num = 47 Frame latency = 315.349121 (ms)

BATCH-NUM = 20**
Batch meta not found for buffer 0x7fbbd407fa60
BATCH-NUM = 21**
Batch meta not found for buffer 0x7fbbd4088b20
BATCH-NUM = 22**
Batch meta not found for buffer 0x7fbbd4081560
BATCH-NUM = 48**
Comp name = nvv4l2decoder1 in_system_timestamp = 1609918681975.475098 out_system_timestamp = 1609918682031.925049 component latency= 56.449951
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 48 in_system_timestamp = 1609918682032.083008 out_system_timestamp = 1609918682085.904053 component_latency = 53.821045
Comp name = primary_gie in_system_timestamp = 1609918682085.960938 out_system_timestamp = 1609918682102.905029 component latency= 16.944092
Comp name = tracking_tracker in_system_timestamp = 1609918682102.926025 out_system_timestamp = 1609918682103.440918 component latency= 0.514893
Comp name = dsanalytics0 in_system_timestamp = 1609918682103.529053 out_system_timestamp = 1609918682103.625000 component latency= 0.095947
Comp name = demuxer in_system_timestamp = 1609918682103.641113 out_system_timestamp = 1609918682103.716064 component latency= 0.074951
Comp name = osd_conv in_system_timestamp = 1609918682103.912109 out_system_timestamp = 1609918682104.093994 component latency= 0.181885
Comp name = nvosd0 in_system_timestamp = 1609918682104.268066 out_system_timestamp = 1609918682108.572021 component latency= 4.303955
Source id = 0 Frame_num = 48 Frame latency = 133.151855 (ms)

BATCH-NUM = 49**
Comp name = nvv4l2decoder1 in_system_timestamp = 1609918681975.690918 out_system_timestamp = 1609918682110.820068 component latency= 135.129150
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 49 in_system_timestamp = 1609918682110.978027 out_system_timestamp = 1609918682126.072998 component_latency = 15.094971
Comp name = primary_gie in_system_timestamp = 1609918682126.131104 out_system_timestamp = 1609918682143.029053 component latency= 16.897949
Comp name = tracking_tracker in_system_timestamp = 1609918682143.051025 out_system_timestamp = 1609918682143.572021 component latency= 0.520996
Comp name = dsanalytics0 in_system_timestamp = 1609918682143.658936 out_system_timestamp = 1609918682143.746094 component latency= 0.087158
Comp name = demuxer in_system_timestamp = 1609918682143.762939 out_system_timestamp = 1609918682143.849121 component latency= 0.086182
Comp name = osd_conv in_system_timestamp = 1609918682144.031006 out_system_timestamp = 1609918682144.270996 component latency= 0.239990
Comp name = nvosd0 in_system_timestamp = 1609918682144.452881 out_system_timestamp = 1609918682148.761963 component latency= 4.309082
Source id = 0 Frame_num = 49 Frame latency = 173.112061 (ms)

BATCH-NUM = 23**
Batch meta not found for buffer 0x7fbbec079550
BATCH-NUM = 50**
Comp name = nvv4l2decoder1 in_system_timestamp = 1609918681978.936035 out_system_timestamp = 1609918682151.047119 component latency= 172.111084
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 50 in_system_timestamp = 1609918682151.219971 out_system_timestamp = 1609918682191.467041 component_latency = 40.247070
Comp name = primary_gie in_system_timestamp = 1609918682191.582031 out_system_timestamp = 1609918682200.656006 component latency= 9.073975
Comp name = tracking_tracker in_system_timestamp = 1609918682200.677002 out_system_timestamp = 1609918682201.059082 component latency= 0.382080
Comp name = dsanalytics0 in_system_timestamp = 1609918682201.105957 out_system_timestamp = 1609918682201.173096 component latency= 0.067139
Comp name = demuxer in_system_timestamp = 1609918682201.187988 out_system_timestamp = 1609918682201.272949 component latency= 0.084961
Comp name = osd_conv in_system_timestamp = 1609918682201.464111 out_system_timestamp = 1609918682201.643066 component latency= 0.178955
Comp name = nvosd0 in_system_timestamp = 1609918682201.822998 out_system_timestamp = 1609918682205.707031 component latency= 3.884033
Source id = 0 Frame_num = 50 Frame latency = 226.810059 (ms)

BATCH-NUM = 24**
Batch meta not found for buffer 0x7fbbec07fb80
BATCH-NUM = 51**
Comp name = nvv4l2decoder1 in_system_timestamp = 1609918682030.185059 out_system_timestamp = 1609918682231.086914 component latency= 200.901855
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 51 in_system_timestamp = 1609918682231.194092 out_system_timestamp = 1609918682231.341064 component_latency = 0.146973
Comp name = primary_gie in_system_timestamp = 1609918682231.398926 out_system_timestamp = 1609918682248.283936 component latency= 16.885010
Comp name = tracking_tracker in_system_timestamp = 1609918682248.311035 out_system_timestamp = 1609918682248.837891 component latency= 0.526855
Comp name = dsanalytics0 in_system_timestamp = 1609918682248.925049 out_system_timestamp = 1609918682249.020996 component latency= 0.095947
Comp name = demuxer in_system_timestamp = 1609918682249.037109 out_system_timestamp = 1609918682249.184082 component latency= 0.146973
Comp name = osd_conv in_system_timestamp = 1609918682249.356934 out_system_timestamp = 1609918682249.572021 component latency= 0.215088
Comp name = nvosd0 in_system_timestamp = 1609918682249.752930 out_system_timestamp = 1609918682254.628906 component latency= 4.875977
Source id = 0 Frame_num = 51 Frame latency = 224.484863 (ms)

BATCH-NUM = 25**
Batch meta not found for buffer 0x7fbbd4097d40
BATCH-NUM = 52**
Comp name = nvv4l2decoder1 in_system_timestamp = 1609918682110.429932 out_system_timestamp = 1609918682311.196045 component latency= 200.766113
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 52 in_system_timestamp = 1609918682311.381104 out_system_timestamp = 1609918682311.519043 component_latency = 0.137939
Comp name = primary_gie in_system_timestamp = 1609918682311.576904 out_system_timestamp = 1609918682328.732910 component latency= 17.156006
Comp name = tracking_tracker in_system_timestamp = 1609918682328.754883 out_system_timestamp = 1609918682329.447998 component latency= 0.693115
Comp name = dsanalytics0 in_system_timestamp = 1609918682329.535889 out_system_timestamp = 1609918682329.656982 component latency= 0.121094
Comp name = demuxer in_system_timestamp = 1609918682329.674072 out_system_timestamp = 1609918682329.843994 component latency= 0.169922
Comp name = osd_conv in_system_timestamp = 1609918682330.022949 out_system_timestamp = 1609918682330.362061 component latency= 0.339111
Comp name = nvosd0 in_system_timestamp = 1609918682330.570068 out_system_timestamp = 1609918682335.629883 component latency= 5.059814
Source id = 0 Frame_num = 52 Frame latency = 225.246094 (ms)

BATCH-NUM = 26**
Batch meta not found for buffer 0x7fbbec07fb80
BATCH-NUM = 53**
Comp name = nvv4l2decoder1 in_system_timestamp = 1609918682150.731934 out_system_timestamp = 1609918682350.803955 component latency= 200.072021
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 53 in_system_timestamp = 1609918682350.910889 out_system_timestamp = 1609918682351.058105 component_latency = 0.147217
Comp name = primary_gie in_system_timestamp = 1609918682351.163086 out_system_timestamp = 1609918682368.079102 component latency= 16.916016
Comp name = tracking_tracker in_system_timestamp = 1609918682368.100098 out_system_timestamp = 1609918682368.648926 component latency= 0.548828
Comp name = dsanalytics0 in_system_timestamp = 1609918682368.738037 out_system_timestamp = 1609918682368.826904 component latency= 0.088867
Comp name = demuxer in_system_timestamp = 1609918682368.843994 out_system_timestamp = 1609918682369.003906 component latency= 0.159912
Comp name = osd_conv in_system_timestamp = 1609918682369.178955 out_system_timestamp = 1609918682369.409912 component latency= 0.230957
Comp name = nvosd0 in_system_timestamp = 1609918682369.590088 out_system_timestamp = 1609918682375.906006 component latency= 6.315918
Source id = 0 Frame_num = 53 Frame latency = 225.218994 (ms)

16

The video decoder is the bottle neck. You can try to enable some software decoders if the CPU performance is OK.

17

How can I do that? Do you have some pointers?

18

Sorry, current deepstream-app can not set SW decoders. You may try to set some SW encoder with rtsp sink, DeepStream Reference Application - deepstream-app — DeepStream 6.1.1 Release documentation, just set ‘enc-type=1’.

20

We did try the enc-type=1 and even for a while tried to switch to codec=2 but neither of them solve the issue.

Looks like more optimization needs to be done. Here are few things that we are trying:

  1. Current model is trained at 1920X1080 images, resnet10. Should we resize the images and labels to 960x544 pixels and then train.
  2. Change streammux to 960x544 pixels
  3. Introduce drop-frame-interval to higher value.
  4. interval in pgie set to around 4
  5. Batch-size match the stream numbers
  6. Timeout = 1/fps in millisecs

Anything else that we can do.
We also have to write images (using the transfer learning app scheme to write images rather than using GST-DSExample optimized) and that also seems to delay the pipeline.

21

@Fiona.Chen Can you please check if this helps. One more observation was that when we add img-save=1 even though we trigger

23

Have you monitored the GPU loading and usage while running your application?