With 4 sources the problem is not observed.
We noticed that the problem is aggravated at 30 rtsp sources and 30 rtsp sinks when we are viewing all 30 rtsp sinks. When we view one or two rtsp sinks then we don’t observe this issue.
We also notice heavy pixelation or ghosting in viewing just one feed as well.
It may be caused by streaming packets lost.
What does “When we view one or two rtsp sinks then we don’t observe this issue.” mean?
Do you mean you have configured to output 30 streams in different ports, when you observe one or two of the streams with stream player(e.g. VLC player), the streams display compatible to the source id. But when you open 30 player, some players(or all players） show the content in mess?
Any ideas how we can debug and resolve this. Need to stabilise the production system. In running 2-3 streams (source and sink) this problem is not observed but at 30 this becomes prominent.
Is my above description correct for your case?
Yes. The moment the app starts running we use ffmpeg to capture the RTSP feeds nd convert to HLS to be displayed on our UI webpage. We typically form groups of 30 RTSP sources with 30 RTSP sinks each with unique RTSP nd UDP ports.
Can you view the rtsp stream directly with vlc player? When you view two of the 30 streams, they are correct, but when you view 30 of the 30 streams, they are mess. If my description is correct, the problem may not in deepstream side. Once the stream is generated, the port number can not change, and the rtsp server just send the designated stream to the designated port. So whether you view 2 of the 30 streams or you view 30 of the 30 streams, there is no difference.
Till now we tried changing live-source from 0 to 1 and vice versa.
Tried bit-rate from 4000000 to 500000 in sink.
Tried different codec 1 or 2 in sink.
Tried different qos 0 or 1 in sink.
Tried different enc-type 1 or 2 in sink.
Tried different iframeinterval from 30 to 5 in sink.
Tried profile 0 or 2 in sink.
Tried changing drop-on-latency from TRUE to FALSE.
Here is out default config.
[sink0]
enable=1
type=4
source-id=0
codec=1
enc-type=0
qos=0
sync=0
bitrate=4000000
profile=0
rtsp-port=8554
udp-port=5000
gpu-id=0
One thing that we have noticed is that along with ghosting we also observe a lag of around 10-15 secs between input and output stream.
The problem is observed in VLC as well.
We have been reporting this issue from 5.0 GA. Based on the announcement in 5.0.1 we though the problem is solved but we have faced this problem in 5.0.1. Then we read about release noted of 5.1 talking about RTSP stability. We face this issue in 5.1 also.
We chose Deepstream over OpenCV for our implementation but facing lots of issues when running 30 streams. We are losing customer confidence. Any help is resolving the issue will be highly appreciated please.
I’ve tested 30 sources and 30 rtsp sinks case in our dGPU platform with DS5.1. It works well. The content for every stream is correct when I play them in the same time with 30 vlc players.
Which dGPU are you using T4? We are using P4. Can you check and confirm same with P4? If you can also share the dumps and videos. Which app are you using. Here are my details:
Also please share the details of your setup.
Just config 30 sources and 30 rtsp sinks in deepstream-app config file and run deepstream-app.
Let me also test the same and report results.
Did you run the normal deepstream-app or the deepstream-app compiled in the nvdsanalytics example.
analytics is not enabled
I enabled Latency Measurement and am astonished to observe the delay in processing.
When the current batch number is 170K at that time the batch being processed is 7.4K.
See the logs:
BATCH-NUM = 7468**
Comp name = nvv4l2decoder25 in_system_timestamp = 1618800729082.727051 out_system_timestamp = 1618800729745.093018 component latency= 662.365967
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 7468 in_system_timestamp = 1618800730396.593018 out_system_timestamp = 1618800730673.933105 component_latency = 277.340088
Comp name = primary_gie in_system_timestamp = 1618800730998.992920 out_system_timestamp = 1618800731163.216064 component latency= 164.223145
Comp name = tracking_tracker in_system_timestamp = 1618800731163.272949 out_system_timestamp = 1618800731173.643066 component latency= 10.370117
Comp name = dsanalytics0 in_system_timestamp = 1618800731173.941895 out_system_timestamp = 1618800731174.930908 component latency= 0.989014
Comp name = demuxer in_system_timestamp = 1618800731174.955078 out_system_timestamp = 1618800731176.783936 component latency= 1.828857
Comp name = osd_conv in_system_timestamp = 1618800731176.944092 out_system_timestamp = 1618800731191.633057 component latency= 14.688965
Comp name = nvosd0 in_system_timestamp = 1618800731191.924072 out_system_timestamp = 1618800731208.094971 component latency= 16.170898
Source id = 0 Frame_num = 7468 Frame latency = 2125.425049 (ms)
BATCH-NUM = 179148**
Batch meta not found for buffer 0x7f142c1ea7c0
BATCH-NUM = 179149**
Batch meta not found for buffer 0x7f14b021e800
BATCH-NUM = 179150**
Batch meta not found for buffer 0x7f13802791d0
BATCH-NUM = 179151**
Batch meta not found for buffer 0x7f14e40b8490
BATCH-NUM = 179152**
Batch meta not found for buffer 0x7f13f0135780
BATCH-NUM = 179153**
Batch meta not found for buffer 0x7f13804cea30
BATCH-NUM = 179154**
Batch meta not found for buffer 0x7f13e40a04a0
BATCH-NUM = 179155**
Batch meta not found for buffer 0x7f14942db6e0
BATCH-NUM = 179156**
Batch meta not found for buffer 0x7f14f01b4290
BATCH-NUM = 179157**
Batch meta not found for buffer 0x7f164415a5b0
BATCH-NUM = 179158**
Batch meta not found for buffer 0x7f1414091c30
BATCH-NUM = 179159**
Batch meta not found for buffer 0x7f13e40e5460
BATCH-NUM = 179160**
Batch meta not found for buffer 0x7f14140f3d60
BATCH-NUM = 179161**
Batch meta not found for buffer 0x7f1494093960
BATCH-NUM = 179162**
Batch meta not found for buffer 0x7f15c00d68c0
BATCH-NUM = 179163**
Batch meta not found for buffer 0x7f14cc0afd60
BATCH-NUM = 7469**
Comp name = nvv4l2decoder25 in_system_timestamp = 1618800729174.388916 out_system_timestamp = 1618800729860.698975 component latency= 686.310059
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 7469 in_system_timestamp = 1618800730524.701904 out_system_timestamp = 1618800730816.050049 component_latency = 291.348145
Comp name = primary_gie in_system_timestamp = 1618800731120.529053 out_system_timestamp = 1618800731306.233887 component latency= 185.704834
Comp name = tracking_tracker in_system_timestamp = 1618800731306.304932 out_system_timestamp = 1618800731318.632080 component latency= 12.327148
Comp name = dsanalytics0 in_system_timestamp = 1618800731318.849121 out_system_timestamp = 1618800731320.181885 component latency= 1.332764
Comp name = demuxer in_system_timestamp = 1618800731320.207031 out_system_timestamp = 1618800731322.388916 component latency= 2.181885
Comp name = osd_conv in_system_timestamp = 1618800731322.541992 out_system_timestamp = 1618800731329.017090 component latency= 6.475098
Comp name = nvosd0 in_system_timestamp = 1618800731329.200928 out_system_timestamp = 1618800731348.412109 component latency= 19.211182
Source id = 0 Frame_num = 7469 Frame latency = 2174.077148 (ms)
BATCH-NUM = 179164**
Batch meta not found for buffer 0x7f14b02315f0
BATCH-NUM = 179165**
Batch meta not found for buffer 0x7f139c13fe00
BATCH-NUM = 179166**
Batch meta not found for buffer 0x7f14f021b2f0
BATCH-NUM = 179167**
Batch meta not found for buffer 0x7f1524235600
BATCH-NUM = 179168**
Batch meta not found for buffer 0x7f15c00cae70
BATCH-NUM = 179169**
Batch meta not found for buffer 0x7f142c1e0920
BATCH-NUM = 179170**
Batch meta not found for buffer 0x7f15540bf270
BATCH-NUM = 179171**
Batch meta not found for buffer 0x7f15380b2580
BATCH-NUM = 179172**
Batch meta not found for buffer 0x7f15c01238f0
BATCH-NUM = 179173**
Batch meta not found for buffer 0x7f15ec0fb240
BATCH-NUM = 179174**
Batch meta not found for buffer 0x7f15b008acd0
BATCH-NUM = 179175**
Batch meta not found for buffer 0x7f14f009c0b0
BATCH-NUM = 179176**
Batch meta not found for buffer 0x7f14b02278a0
BATCH-NUM = 179177**
Batch meta not found for buffer 0x7f15c00f4bd0
BATCH-NUM = 179178**
Batch meta not found for buffer 0x7f139c08f7c0
BATCH-NUM = 179179**
Batch meta not found for buffer 0x7f14702277b0
BATCH-NUM = 179180**
Batch meta not found for buffer 0x7f15c00f16d0
BATCH-NUM = 179181**
Batch meta not found for buffer 0x7f15c00d01a0
BATCH-NUM = 179182**
Batch meta not found for buffer 0x7f15900700d0
BATCH-NUM = 179183**
Batch meta not found for buffer 0x7f15ec0855b0
BATCH-NUM = 179184**
Batch meta not found for buffer 0x7f1508197d30
BATCH-NUM = 179185**
Batch meta not found for buffer 0x7f14c80730a0
BATCH-NUM = 179186**
Batch meta not found for buffer 0x7f13f013fa00
BATCH-NUM = 7470**
Comp name = nvv4l2decoder25 in_system_timestamp = 1618800729278.361084 out_system_timestamp = 1618800730000.909912 component latency= 722.548828
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 7470 in_system_timestamp = 1618800730682.492920 out_system_timestamp = 1618800730938.618896 component_latency = 256.125977
Comp name = primary_gie in_system_timestamp = 1618800731264.253906 out_system_timestamp = 1618800731458.028076 component latency= 193.774170
Comp name = tracking_tracker in_system_timestamp = 1618800731458.093994 out_system_timestamp = 1618800731470.851074 component latency= 12.757080
Comp name = dsanalytics0 in_system_timestamp = 1618800731471.179932 out_system_timestamp = 1618800731472.658936 component latency= 1.479004
Comp name = demuxer in_system_timestamp = 1618800731472.686035 out_system_timestamp = 1618800731474.802002 component latency= 2.115967
Comp name = osd_conv in_system_timestamp = 1618800731474.960938 out_system_timestamp = 1618800731486.528076 component latency= 11.567139
Comp name = nvosd0 in_system_timestamp = 1618800731486.739990 out_system_timestamp = 1618800731502.288086 component latency= 15.548096
Source id = 0 Frame_num = 7470 Frame latency = 2223.975830 (ms)
BATCH-NUM = 179187**
Batch meta not found for buffer 0x7f14f01e1280
BATCH-NUM = 179188**
Batch meta not found for buffer 0x7f15242266f0
BATCH-NUM = 179189**
Batch meta not found for buffer 0x7f14501ee850
BATCH-NUM = 179190**
Batch meta not found for buffer 0x7f15d409ee30
BATCH-NUM = 179191**
Batch meta not found for buffer 0x7f1438237c70
BATCH-NUM = 179192**
Batch meta not found for buffer 0x7f17582427a0
BATCH-NUM = 179193**
Batch meta not found for buffer 0x7f14940a43b0
BATCH-NUM = 179194**
Batch meta not found for buffer 0x7f14f00bdb30
BATCH-NUM = 179195**
Batch meta not found for buffer 0x7f15ec0e6bb0
BATCH-NUM = 179196**
Batch meta not found for buffer 0x7f15b00c8510
BATCH-NUM = 179197**
Batch meta not found for buffer 0x7f1584079bd0
BATCH-NUM = 179198**
Batch meta not found for buffer 0x7f139c124220
BATCH-NUM = 179199**
Batch meta not found for buffer 0x7f147017c5d0
BATCH-NUM = 179200**
Batch meta not found for buffer 0x7f14700d5bd0
BATCH-NUM = 179201**
Batch meta not found for buffer 0x7f15ec0a3400
BATCH-NUM = 179202**
Batch meta not found for buffer 0x7f13e40e5460
BATCH-NUM = 179203**
Batch meta not found for buffer 0x7f147015f400
BATCH-NUM = 179204**
Batch meta not found for buffer 0x7f15080d1370
BATCH-NUM = 179205**
Batch meta not found for buffer 0x7f15d406e120
BATCH-NUM = 179206**
Batch meta not found for buffer 0x7f1494223df0
BATCH-NUM = 179207**
Batch meta not found for buffer 0x7f14880e5a20
BATCH-NUM = 179208**
Batch meta not found for buffer 0x7f16341073f0
BATCH-NUM = 179209**
Batch meta not found for buffer 0x7f14701e4b60
BATCH-NUM = 179210**
Batch meta not found for buffer 0x7f14e411f850
BATCH-NUM = 179211**
Batch meta not found for buffer 0x7f15d40bbcd0
BATCH-NUM = 179212**
Batch meta not found for buffer 0x7f14b02315f0
BATCH-NUM = 179213**
Batch meta not found for buffer 0x7f15c00a02e0
BATCH-NUM = 179214**
Batch meta not found for buffer 0x7f1438184c60
BATCH-NUM = 179215**
Batch meta not found for buffer 0x7f14140f3d60
BATCH-NUM = 179216**
Batch meta not found for buffer 0x7f16441cbde0
BATCH-NUM = 179217**
Batch meta not found for buffer 0x7f1470098550
BATCH-NUM = 179218**
Batch meta not found for buffer 0x7f1414072340
BATCH-NUM = 179219**
Batch meta not found for buffer 0x7f13e4074670
BATCH-NUM = 179220**
Batch meta not found for buffer 0x7f14f009c0b0
BATCH-NUM = 179221**
Batch meta not found for buffer 0x7f1618092d40
BATCH-NUM = 179222**
Batch meta not found for buffer 0x7f1470204300
BATCH-NUM = 179223**
Batch meta not found for buffer 0x7f148808f0c0
BATCH-NUM = 179224**
Batch meta not found for buffer 0x7f15080f5260
BATCH-NUM = 179225**
Batch meta not found for buffer 0x7f14701c6cf0
BATCH-NUM = 179226**
Batch meta not found for buffer 0x7f1644204c50
BATCH-NUM = 179227**
Batch meta not found for buffer 0x7f139c0ac730
BATCH-NUM = 7471**
Comp name = nvv4l2decoder25 in_system_timestamp = 1618800729582.349121 out_system_timestamp = 1618800730235.187012 component latency= 652.837891
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 7471 in_system_timestamp = 1618800730829.599121 out_system_timestamp = 1618800731198.695068 component_latency = 369.095947
Comp name = primary_gie in_system_timestamp = 1618800731553.403076 out_system_timestamp = 1618800731741.645020 component latency= 188.241943
Comp name = tracking_tracker in_system_timestamp = 1618800731741.701904 out_system_timestamp = 1618800731752.868896 component latency= 11.166992
Comp name = dsanalytics0 in_system_timestamp = 1618800731753.159912 out_system_timestamp = 1618800731754.403076 component latency= 1.243164
Comp name = demuxer in_system_timestamp = 1618800731754.430908 out_system_timestamp = 1618800731756.196045 component latency= 1.765137
Comp name = osd_conv in_system_timestamp = 1618800731756.398926 out_system_timestamp = 1618800731762.404053 component latency= 6.005127
Comp name = nvosd0 in_system_timestamp = 1618800731762.638916 out_system_timestamp = 1618800731771.895996 component latency= 9.257080
Source id = 0 Frame_num = 7471 Frame latency = 2189.595947 (ms)
BATCH-NUM = 179228**
Batch meta not found for buffer 0x7f17f81c0e80
BATCH-NUM = 179229**
Batch meta not found for buffer 0x7f13f016f120
BATCH-NUM = 179230**
Batch meta not found for buffer 0x7f14f0164a00
BATCH-NUM = 179231**
Batch meta not found for buffer 0x7f1508197d30
BATCH-NUM = 179232**
Batch meta not found for buffer 0x7f15242266f0
BATCH-NUM = 179233**
Batch meta not found for buffer 0x7f15c00f16d0
BATCH-NUM = 179234**
Batch meta not found for buffer 0x7f139c08d6e0
BATCH-NUM = 179235**
Batch meta not found for buffer 0x7f14e4074bd0
BATCH-NUM = 179236**
Batch meta not found for buffer 0x7f1608099e80
BATCH-NUM = 179237**
Batch meta not found for buffer 0x7f1758321020
BATCH-NUM = 179238**
Batch meta not found for buffer 0x7f14701db250
BATCH-NUM = 179239**
Batch meta not found for buffer 0x7f16441729b0
BATCH-NUM = 179240**
Batch meta not found for buffer 0x7f1590085020
BATCH-NUM = 179241**
Batch meta not found for buffer 0x7f16440aabb0
BATCH-NUM = 179242**
Batch meta not found for buffer 0x7f14942db6e0
BATCH-NUM = 179243**
Batch meta not found for buffer 0x7f1470204a70
BATCH-NUM = 179244**
Batch meta not found for buffer 0x7f139c0ed2a0
BATCH-NUM = 179245**
Batch meta not found for buffer 0x7f15c00ea4e0
BATCH-NUM = 179246**
Batch meta not found for buffer 0x7f1590070ea0
BATCH-NUM = 179247**
Batch meta not found for buffer 0x7f150815e8a0
BATCH-NUM = 179248**
Batch meta not found for buffer 0x7f138009a0b0
BATCH-NUM = 179249**
Batch meta not found for buffer 0x7f14cc0afd60
BATCH-NUM = 179250**
Batch meta not found for buffer 0x7f14e40e4e30
BATCH-NUM = 179251**
Batch meta not found for buffer 0x7f14f0203840
BATCH-NUM = 7472**
Comp name = nvv4l2decoder25 in_system_timestamp = 1618800729745.205078 out_system_timestamp = 1618800730406.769043 component latency= 661.563965
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 7472 in_system_timestamp = 1618800731094.271973 out_system_timestamp = 1618800731343.866943 component_latency = 249.594971
Comp name = primary_gie in_system_timestamp = 1618800731690.594971 out_system_timestamp = 1618800731843.049072 component latency= 152.454102
Comp name = tracking_tracker in_system_timestamp = 1618800731843.097900 out_system_timestamp = 1618800731852.260010 component latency= 9.162109
Comp name = dsanalytics0 in_system_timestamp = 1618800731852.527100 out_system_timestamp = 1618800731853.466064 component latency= 0.938965
Comp name = demuxer in_system_timestamp = 1618800731853.489990 out_system_timestamp = 1618800731854.937988 component latency= 1.447998
Comp name = osd_conv in_system_timestamp = 1618800731855.093994 out_system_timestamp = 1618800731872.303955 component latency= 17.209961
Comp name = nvosd0 in_system_timestamp = 1618800731872.556885 out_system_timestamp = 1618800731881.344971 component latency= 8.788086
Source id = 0 Frame_num = 7472 Frame latency = 2136.183838 (ms)
BATCH-NUM = 179252**
Batch meta not found for buffer 0x7f1414119790
BATCH-NUM = 179253**
Batch meta not found for buffer 0x7f14b02278a0
BATCH-NUM = 179254**
Batch meta not found for buffer 0x7f147017c5d0
BATCH-NUM = 179255**
Batch meta not found for buffer 0x7f1608080790
BATCH-NUM = 179256**
Batch meta not found for buffer 0x7f1524235600
BATCH-NUM = 179257**
Batch meta not found for buffer 0x7f13e4074670
BATCH-NUM = 179258**
Batch meta not found for buffer 0x7f139c1303b0
BATCH-NUM = 179259**
Batch meta not found for buffer 0x7f142c1e65b0
BATCH-NUM = 179260**
Batch meta not found for buffer 0x7f14501ee850
BATCH-NUM = 179261**
Batch meta not found for buffer 0x7f15080a3690
BATCH-NUM = 179262**
Batch meta not found for buffer 0x7f163409e500
BATCH-NUM = 179263**
Batch meta not found for buffer 0x7f13f01748b0
BATCH-NUM = 179264**
Batch meta not found for buffer 0x7f1618079e50
BATCH-NUM = 179265**
Batch meta not found for buffer 0x7f14880f7140
BATCH-NUM = 179266**
Batch meta not found for buffer 0x7f175810d1f0
BATCH-NUM = 179267**
Batch meta not found for buffer 0x7f14b02278a0
BATCH-NUM = 179268**
Batch meta not found for buffer 0x7f14e40ef470
BATCH-NUM = 179269**
Batch meta not found for buffer 0x7f152422c8b0
BATCH-NUM = 179270**
Batch meta not found for buffer 0x7f14700d5bd0
BATCH-NUM = 7473**
Comp name = nvv4l2decoder25 in_system_timestamp = 1618800729860.842041 out_system_timestamp = 1618800730533.669922 component latency= 672.827881
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 7473 in_system_timestamp = 1618800731206.789062 out_system_timestamp = 1618800731497.044922 component_latency = 290.255859
Comp name = primary_gie in_system_timestamp = 1618800731807.085938 out_system_timestamp = 1618800731968.114990 component latency= 161.029053
Comp name = tracking_tracker in_system_timestamp = 1618800731968.172119 out_system_timestamp = 1618800731978.781006 component latency= 10.608887
Comp name = dsanalytics0 in_system_timestamp = 1618800731978.951904 out_system_timestamp = 1618800731980.210938 component latency= 1.259033
Comp name = demuxer in_system_timestamp = 1618800731980.237061 out_system_timestamp = 1618800731982.672119 component latency= 2.435059
Comp name = osd_conv in_system_timestamp = 1618800731982.895996 out_system_timestamp = 1618800732015.475098 component latency= 32.579102
Comp name = nvosd0 in_system_timestamp = 1618800732015.677002 out_system_timestamp = 1618800732022.535889 component latency= 6.858887
Source id = 0 Frame_num = 7473 Frame latency = 2161.738037 (ms)
BATCH-NUM = 179271**
Batch meta not found for buffer 0x7f14940a2280
BATCH-NUM = 179272**
Batch meta not found for buffer 0x7f15080fd500
BATCH-NUM = 179273**
Batch meta not found for buffer 0x7f147020b600
BATCH-NUM = 179274**
Batch meta not found for buffer 0x7f16341073f0
BATCH-NUM = 179275**
Batch meta not found for buffer 0x7f15ec0855b0
BATCH-NUM = 179276**
Batch meta not found for buffer 0x7f16441d67f0
BATCH-NUM = 179277**
Batch meta not found for buffer 0x7f15d40bbcd0
BATCH-NUM = 179278**
Batch meta not found for buffer 0x7f14e408c030
BATCH-NUM = 179279**
Batch meta not found for buffer 0x7f15c00bdc30
BATCH-NUM = 179280**
Batch meta not found for buffer 0x7f17580da410
BATCH-NUM = 179281**
Batch meta not found for buffer 0x7f1414118c90
BATCH-NUM = 179282**
Batch meta not found for buffer 0x7f15540bf270
BATCH-NUM = 179283**
Batch meta not found for buffer 0x7f147019f510
BATCH-NUM = 7474**
Comp name = nvv4l2decoder25 in_system_timestamp = 1618800730001.082031 out_system_timestamp = 1618800730692.530029 component latency= 691.447998
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 7474 in_system_timestamp = 1618800731347.635986 out_system_timestamp = 1618800731630.197021 component_latency = 282.561035
Comp name = primary_gie in_system_timestamp = 1618800731925.885986 out_system_timestamp = 1618800732119.287109 component latency= 193.401123
Comp name = tracking_tracker in_system_timestamp = 1618800732119.343994 out_system_timestamp = 1618800732131.650879 component latency= 12.306885
Comp name = dsanalytics0 in_system_timestamp = 1618800732131.928955 out_system_timestamp = 1618800732133.365967 component latency= 1.437012
Comp name = demuxer in_system_timestamp = 1618800732133.392090 out_system_timestamp = 1618800732136.653076 component latency= 3.260986
Comp name = osd_conv in_system_timestamp = 1618800732136.815918 out_system_timestamp = 1618800732156.084961 component latency= 19.269043
Comp name = nvosd0 in_system_timestamp = 1618800732156.319092 out_system_timestamp = 1618800732161.543945 component latency= 5.224854
Source id = 0 Frame_num = 7474 Frame latency = 2160.531006 (ms)
BATCH-NUM = 179284**
Batch meta not found for buffer 0x7f14701b9c70
BATCH-NUM = 179285**
Batch meta not found for buffer 0x7f14b006f5b0
BATCH-NUM = 179286**
Batch meta not found for buffer 0x7f14141429b0
BATCH-NUM = 179287**
Batch meta not found for buffer 0x7f142c1d2a00
BATCH-NUM = 179288**
Batch meta not found for buffer 0x7f1590085020
BATCH-NUM = 179289**
Batch meta not found for buffer 0x7f15ec0fb240
BATCH-NUM = 179290**
Batch meta not found for buffer 0x7f161806bc10
BATCH-NUM = 179291**
Batch meta not found for buffer 0x7f13f013c4c0
BATCH-NUM = 179292**
Batch meta not found for buffer 0x7f14cc08d820
BATCH-NUM = 179293**
Batch meta not found for buffer 0x7f15c00cae70
BATCH-NUM = 179294**
Batch meta not found for buffer 0x7f1438237c70
BATCH-NUM = 179295**
Batch meta not found for buffer 0x7f14140828e0
BATCH-NUM = 179296**
Batch meta not found for buffer 0x7f14940aea30
BATCH-NUM = 179297**
Batch meta not found for buffer 0x7f13800e8b40
BATCH-NUM = 179298**
Batch meta not found for buffer 0x7f13f0147330
BATCH-NUM = 179299**
Batch meta not found for buffer 0x7f148808f0c0
BATCH-NUM = 179300**
Batch meta not found for buffer 0x7f15900859b0
BATCH-NUM = 179301**
Batch meta not found for buffer 0x7f149421c720
BATCH-NUM = 179302**
Batch meta not found for buffer 0x7f14f01b4070
BATCH-NUM = 179303**
Batch meta not found for buffer 0x7f15080dbe70
BATCH-NUM = 179304**
Batch meta not found for buffer 0x7f175828c0c0
BATCH-NUM = 179305**
Batch meta not found for buffer 0x7f15ec06fe90
BATCH-NUM = 179306**
Batch meta not found for buffer 0x7f148809d250
BATCH-NUM = 179307**
Batch meta not found for buffer 0x7f1470204300
BATCH-NUM = 179308**
Batch meta not found for buffer 0x7f13f013fa00
BATCH-NUM = 179309**
Batch meta not found for buffer 0x7f1608080790
BATCH-NUM = 7475**
Comp name = nvv4l2decoder25 in_system_timestamp = 1618800730235.313965 out_system_timestamp = 1618800730838.477051 component latency= 603.163086
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 7475 in_system_timestamp = 1618800731497.616943 out_system_timestamp = 1618800731778.259033 component_latency = 280.642090
Comp name = primary_gie in_system_timestamp = 1618800732082.625000 out_system_timestamp = 1618800732270.678955 component latency= 188.053955
Comp name = tracking_tracker in_system_timestamp = 1618800732270.738037 out_system_timestamp = 1618800732282.637939 component latency= 11.899902
Comp name = dsanalytics0 in_system_timestamp = 1618800732282.888916 out_system_timestamp = 1618800732283.825928 component latency= 0.937012
Comp name = demuxer in_system_timestamp = 1618800732283.847900 out_system_timestamp = 1618800732285.810059 component latency= 1.962158
Comp name = osd_conv in_system_timestamp = 1618800732285.914062 out_system_timestamp = 1618800732298.741943 component latency= 12.827881
Comp name = nvosd0 in_system_timestamp = 1618800732298.957031 out_system_timestamp = 1618800732312.779053 component latency= 13.822021
Source id = 0 Frame_num = 7475 Frame latency = 2077.505127 (ms)
BATCH-NUM = 179310**
Batch meta not found for buffer 0x7f147022b7f0
BATCH-NUM = 179311**
Batch meta not found for buffer 0x7f16440b30b0
BATCH-NUM = 179312**
Batch meta not found for buffer 0x7f13e4115e00
BATCH-NUM = 179313**
Batch meta not found for buffer 0x7f14e4087220
BATCH-NUM = 179314**
Batch meta not found for buffer 0x7f1494071e50
BATCH-NUM = 179315**
Batch meta not found for buffer 0x7f14e40f8c00
BATCH-NUM = 179316**
Batch meta not found for buffer 0x7f15ec14e8b0
BATCH-NUM = 179317**
Batch meta not found for buffer 0x7f175803bb00
BATCH-NUM = 179318**
Batch meta not found for buffer 0x7f141409c710
BATCH-NUM = 179319**
Batch meta not found for buffer 0x7f139c0a76d0
BATCH-NUM = 179320**
Batch meta not found for buffer 0x7f14e40e46c0
BATCH-NUM = 179321**
Batch meta not found for buffer 0x7f148808f0c0
BATCH-NUM = 179322**
Batch meta not found for buffer 0x7f15ac055b40
BATCH-NUM = 179323**
Batch meta not found for buffer 0x7f139c12d800
BATCH-NUM = 179324**
Batch meta not found for buffer 0x7f1414104d10
BATCH-NUM = 179325**
Batch meta not found for buffer 0x7f16340618f0
BATCH-NUM = 179326**
Batch meta not found for buffer 0x7f149409b950
BATCH-NUM = 179327**
Batch meta not found for buffer 0x7f1608099e80
BATCH-NUM = 179328**
Batch meta not found for buffer 0x7f15081f8d10
BATCH-NUM = 179329**
Batch meta not found for buffer 0x7f14f009c0b0
BATCH-NUM = 179330**
Batch meta not found for buffer 0x7f14880e51a0
BATCH-NUM = 179331**
Batch meta not found for buffer 0x7f15080a3690
BATCH-NUM = 179332**
Batch meta not found for buffer 0x7f14e4074bd0
BATCH-NUM = 179333**
Batch meta not found for buffer 0x7f14f021b2f0
BATCH-NUM = 179334**
Batch meta not found for buffer 0x7f139c12d5e0
BATCH-NUM = 179335**
Batch meta not found for buffer 0x7f1758205480
BATCH-NUM = 179336**
Batch meta not found for buffer 0x7f1644128670
BATCH-NUM = 179337**
Batch meta not found for buffer 0x7f1494244c70
BATCH-NUM = 179338**
Batch meta not found for buffer 0x7f14e40f62a0
BATCH-NUM = 179339**
Batch meta not found for buffer 0x7f1470163c80
BATCH-NUM = 179340**
Batch meta not found for buffer 0x7f147009e620
BATCH-NUM = 179341**
Batch meta not found for buffer 0x7f143817c3e0
BATCH-NUM = 179342**
Batch meta not found for buffer 0x7f1470246b60
BATCH-NUM = 179343**
Batch meta not found for buffer 0x7f1508185e00
BATCH-NUM = 179344**
Batch meta not found for buffer 0x7f1444053190
BATCH-NUM = 179345**
Batch meta not found for buffer 0x7f1584079bd0
BATCH-NUM = 179346**
Batch meta not found for buffer 0x7f15c00ea4e0
BATCH-NUM = 179347**
Batch meta not found for buffer 0x7f139c11b450
BATCH-NUM = 179348**
Batch meta not found for buffer 0x7f164415a6c0
BATCH-NUM = 179349**
Batch meta not found for buffer 0x7f14e406ad70
BATCH-NUM = 179350**
Batch meta not found for buffer 0x7f175810d1f0
BATCH-NUM = 179351**
Batch meta not found for buffer 0x7f13e410f5b0
BATCH-NUM = 179352**
Batch meta not found for buffer 0x7f13804cea30
BATCH-NUM = 179353**
Batch meta not found for buffer 0x7f14f0203840^C** ERROR: <_intr_handler:705>: User Interrupted…
BATCH-NUM = 179354**
Batch meta not found for buffer 0x7f13f009b340
BATCH-NUM = 7476**
Comp name = nvv4l2decoder25 in_system_timestamp = 1618800730406.879883 out_system_timestamp = 1618800731099.218018 component latency= 692.338135
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 7476 in_system_timestamp = 1618800731774.629883 out_system_timestamp = 1618800732003.989014 component_latency = 229.359131
Comp name = primary_gie in_system_timestamp = 1618800732377.273926 out_system_timestamp = 1618800732526.816895 component latency= 149.542969
Comp name = tracking_tracker in_system_timestamp = 1618800732526.866943 out_system_timestamp = 1618800732537.237061 component latency= 10.370117
Comp name = dsanalytics0 in_system_timestamp = 1618800732537.502930 out_system_timestamp = 1618800732539.169922 component latency= 1.666992
Comp name = demuxer in_system_timestamp = 1618800732539.207031 out_system_timestamp = 1618800732540.435059 component latency= 1.228027
Comp name = osd_conv in_system_timestamp = 1618800732540.605957 out_system_timestamp = 1618800732542.894043 component latency= 2.288086
Comp name = nvosd0 in_system_timestamp = 1618800732543.069092 out_system_timestamp = 1618800732558.089111 component latency= 15.020020
Source id = 0 Frame_num = 7476 Frame latency = 2151.264160 (ms)
BATCH-NUM = 179355**
Batch meta not found for buffer 0x7f15c00d4610
BATCH-NUM = 179356**
Batch meta not found for buffer 0x7f142c26e150
BATCH-NUM = 179357**
Batch meta not found for buffer 0x7f14cc0afd60
BATCH-NUM = 179358**
Batch meta not found for buffer 0x7f14940aea30
BATCH-NUM = 179359**
Batch meta not found for buffer 0x7f14702161d0
BATCH-NUM = 179360**
Batch meta not found for buffer 0x7f14f0203840
BATCH-NUM = 179361**
Batch meta not found for buffer 0x7f1494223df0
BATCH-NUM = 179362**
Batch meta not found for buffer 0x7f142c1e3090
BATCH-NUM = 179363**
Batch meta not found for buffer 0x7f15c00a02e0
BATCH-NUM = 179364**
Batch meta not found for buffer 0x7f149421c720
BATCH-NUM = 179365**
Batch meta not found for buffer 0x7f138010f9f0
BATCH-NUM = 179366**
Batch meta not found for buffer 0x7f1494093960
BATCH-NUM = 179367**
Batch meta not found for buffer 0x7f14f021bd90
BATCH-NUM = 179368**
Batch meta not found for buffer 0x7f14701f7000
BATCH-NUM = 179369**
Batch meta not found for buffer 0x7f14701e4b60
BATCH-NUM = 179370**
Batch meta not found for buffer 0x7f139c0ed2a0
BATCH-NUM = 179371**
Batch meta not found for buffer 0x7f13e40ca0b0Quitting
Do you mean you find a new issue? Please create new ticket for new issue.