Test the ai_nvr of the latest jps, in the case of adding multiple streams, it is found that deepstream stops inference, Active sources: 0, sourceid still exists, please see log
My steps
Step 1: Standard configuration work
Step 2: Add the rtsp stream
I found that dla and cpu were still running, but at a lower frequency
Step 3: Modify the vst configuration to support more streams. I changed it to 16 streams
Step 4: Restart ai-nvr
Step 5: Add more paths to the rtsp flow
Step 6: Try playing the video
Step 7: Check with jtop and find that it is not running anymore, then check the log of deepstream to find the problem
deepstream.log (6.0 MB)
I will check and feedback later.
I can run 8 cameras in one DLA with default JPS release. The stream will route to another DLA if you add more then 8 cameras.
66[385734ac-bb20-4514-8acc-105d4614f345] 30.05 (30.13) 11[dfde20b0-f1e7-42fe-899b-3d11c3ae4ecd] 30.25 (30.06) 44[284ee652-1208-4847-bb95-8d775260b738] 30.05 (30.13) 33[0e264899-465c-4234-bf0a-68882e02b240] 30.05 (30.13) 55[66c43f0a-f28b-4a74-aece-a73a9a901f51] 30.25 (30.06) 77[85b681e4-f83a-4554-be82-f069a8b361f2] 30.05 (30.05) 22[8c2363a6-6f86-46a9-a3f0-575eb94c5dd4] 30.10 (30.05) 00[be307291-892c-401e-82e1-3c216bccca88] 30.05 (30.05)
Active sources : 0
Thu Aug 22 08:16:42 2024
**PERF:
Active sources : 8
Thu Aug 22 08:16:47 2024
**PERF:
66[385734ac-bb20-4514-8acc-105d4614f345] 29.94 (30.13) 11[dfde20b0-f1e7-42fe-899b-3d11c3ae4ecd] 29.74 (30.06) 44[284ee652-1208-4847-bb95-8d775260b738] 29.94 (30.13) 33[0e264899-465c-4234-bf0a-68882e02b240] 29.94 (30.13) 55[66c43f0a-f28b-4a74-aece-a73a9a901f51] 29.74 (30.06) 77[85b681e4-f83a-4554-be82-f069a8b361f2] 29.94 (30.06) 22[8c2363a6-6f86-46a9-a3f0-575eb94c5dd4] 29.89 (30.06) 00[be307291-892c-401e-82e1-3c216bccca88] 29.74 (30.05)
Active sources : 0
Thu Aug 22 08:16:47 2024
**PERF:
Active sources : 8
Thu Aug 22 08:16:52 2024
**PERF:
66[385734ac-bb20-4514-8acc-105d4614f345] 30.06 (30.13) 11[dfde20b0-f1e7-42fe-899b-3d11c3ae4ecd] 30.23 (30.06) 44[284ee652-1208-4847-bb95-8d775260b738] 30.06 (30.13) 33[0e264899-465c-4234-bf0a-68882e02b240] 30.06 (30.13) 55[66c43f0a-f28b-4a74-aece-a73a9a901f51] 30.23 (30.06) 77[85b681e4-f83a-4554-be82-f069a8b361f2] 30.03 (30.05) 22[8c2363a6-6f86-46a9-a3f0-575eb94c5dd4] 30.23 (30.06) 00[be307291-892c-401e-82e1-3c216bccca88] 30.23 (30.05)
Active sources : 0
Thu Aug 22 08:16:52 2024
**PERF:
Active sources : 8
Thu Aug 22 08:16:57 2024
**PERF:
66[385734ac-bb20-4514-8acc-105d4614f345] 30.06 (30.13) 11[dfde20b0-f1e7-42fe-899b-3d11c3ae4ecd] 29.93 (30.05) 44[284ee652-1208-4847-bb95-8d775260b738] 30.06 (30.12) 33[0e264899-465c-4234-bf0a-68882e02b240] 30.06 (30.13) 55[66c43f0a-f28b-4a74-aece-a73a9a901f51] 29.89 (30.05) 77[85b681e4-f83a-4554-be82-f069a8b361f2] 30.09 (30.05) 22[8c2363a6-6f86-46a9-a3f0-575eb94c5dd4] 29.89 (30.05) 00[be307291-892c-401e-82e1-3c216bccca88] 30.09 (30.05)
Active sources : 0
Thu Aug 22 08:16:57 2024
**PERF:
Active sources : 8
Thu Aug 22 08:17:02 2024
**PERF:
66[385734ac-bb20-4514-8acc-105d4614f345] 30.02 (30.13) 11[dfde20b0-f1e7-42fe-899b-3d11c3ae4ecd] 29.99 (30.05) 44[284ee652-1208-4847-bb95-8d775260b738] 30.02 (30.12) 33[0e264899-465c-4234-bf0a-68882e02b240] 29.86 (30.12) 55[66c43f0a-f28b-4a74-aece-a73a9a901f51] 30.02 (30.05) 77[85b681e4-f83a-4554-be82-f069a8b361f2] 29.86 (30.05) 22[8c2363a6-6f86-46a9-a3f0-575eb94c5dd4] 30.02 (30.05) 00[be307291-892c-401e-82e1-3c216bccca88] 29.83 (30.05)
Active sources : 0
Thu Aug 22 08:17:02 2024
**PERF:
Active sources : 8
Thu Aug 22 08:17:07 2024
**PERF:
66[385734ac-bb20-4514-8acc-105d4614f345] 29.84 (30.13) 11[dfde20b0-f1e7-42fe-899b-3d11c3ae4ecd] 29.84 (30.05) 44[284ee652-1208-4847-bb95-8d775260b738] 29.84 (30.12) 33[0e264899-465c-4234-bf0a-68882e02b240] 29.81 (30.12) 55[66c43f0a-f28b-4a74-aece-a73a9a901f51] 29.84 (30.05) 77[85b681e4-f83a-4554-be82-f069a8b361f2] 30.01 (30.05) 22[8c2363a6-6f86-46a9-a3f0-575eb94c5dd4] 29.84 (30.05) 00[be307291-892c-401e-82e1-3c216bccca88] 29.84 (30.05)
Active sources : 0
Thu Aug 22 08:17:07 2024
**PERF:
Active sources : 8
Thu Aug 22 08:17:12 2024
**PERF:
66[385734ac-bb20-4514-8acc-105d4614f345] 30.13 (30.12) 11[dfde20b0-f1e7-42fe-899b-3d11c3ae4ecd] 30.13 (30.05) 44[284ee652-1208-4847-bb95-8d775260b738] 30.13 (30.12) 33[0e264899-465c-4234-bf0a-68882e02b240] 30.16 (30.12) 55[66c43f0a-f28b-4a74-aece-a73a9a901f51] 30.13 (30.05) 77[85b681e4-f83a-4554-be82-f069a8b361f2] 29.93 (30.05) 22[8c2363a6-6f86-46a9-a3f0-575eb94c5dd4] 30.13 (30.05) 00[be307291-892c-401e-82e1-3c216bccca88] 30.13 (30.05)
Active sources : 0
Thu Aug 22 08:17:12 2024
**PERF:
Active sources : 8
Thu Aug 22 08:17:17 2024
**PERF:
66[385734ac-bb20-4514-8acc-105d4614f345] 29.94 (30.12) 11[dfde20b0-f1e7-42fe-899b-3d11c3ae4ecd] 29.94 (30.05) 44[284ee652-1208-4847-bb95-8d775260b738] 29.94 (30.12) 33[0e264899-465c-4234-bf0a-68882e02b240] 29.95 (30.12) 55[66c43f0a-f28b-4a74-aece-a73a9a901f51] 29.94 (30.05) 77[85b681e4-f83a-4554-be82-f069a8b361f2] 30.14 (30.05) 22[8c2363a6-6f86-46a9-a3f0-575eb94c5dd4] 29.94 (30.05) 00[be307291-892c-401e-82e1-3c216bccca88] 30.14 (30.05)
Active sources : 0
Thu Aug 22 08:17:17 2024
**PERF:
Active sources : 8
**PERF: FPS 2 (Avg) FPS 7 (Avg) FPS 1 (Avg) FPS 0 (Avg) FPS 3 (Avg) FPS 4 (Avg) FPS 5 (Avg) FPS 6 (Avg)
Thu Aug 22 08:17:22 2024
**PERF:
66[385734ac-bb20-4514-8acc-105d4614f345] 30.03 (30.12) 11[dfde20b0-f1e7-42fe-899b-3d11c3ae4ecd] 29.87 (30.05) 44[284ee652-1208-4847-bb95-8d775260b738] 30.03 (30.12) 33[0e264899-465c-4234-bf0a-68882e02b240] 29.99 (30.12) 55[66c43f0a-f28b-4a74-aece-a73a9a901f51] 30.03 (30.05) 77[85b681e4-f83a-4554-be82-f069a8b361f2] 29.83 (30.05) 22[8c2363a6-6f86-46a9-a3f0-575eb94c5dd4] 30.03 (30.05) 00[be307291-892c-401e-82e1-3c216bccca88] 29.83 (30.05)
Active sources : 0
Thu Aug 22 08:17:22 2024
**PERF:
Active sources : 8
Thu Aug 22 08:17:27 2024
**PERF:
66[385734ac-bb20-4514-8acc-105d4614f345] 29.93 (30.12) 11[dfde20b0-f1e7-42fe-899b-3d11c3ae4ecd] 30.10 (30.05) 44[284ee652-1208-4847-bb95-8d775260b738] 29.93 (30.11) 33[0e264899-465c-4234-bf0a-68882e02b240] 30.13 (30.12) 55[66c43f0a-f28b-4a74-aece-a73a9a901f51] 29.93 (30.05) 77[85b681e4-f83a-4554-be82-f069a8b361f2] 30.13 (30.05) 22[8c2363a6-6f86-46a9-a3f0-575eb94c5dd4] 29.93 (30.05) 00[be307291-892c-401e-82e1-3c216bccca88] 29.93 (30.05)
Active sources : 0
Thu Aug 22 08:17:27 2024
**PERF:
Active sources : 8
Thu Aug 22 08:17:32 2024
**PERF:
66[385734ac-bb20-4514-8acc-105d4614f345] 29.91 (30.12) 11[dfde20b0-f1e7-42fe-899b-3d11c3ae4ecd] 30.11 (30.05) 44[284ee652-1208-4847-bb95-8d775260b738] 30.11 (30.12) 33[0e264899-465c-4234-bf0a-68882e02b240] 29.91 (30.12) 55[66c43f0a-f28b-4a74-aece-a73a9a901f51] 30.11 (30.05) 77[85b681e4-f83a-4554-be82-f069a8b361f2] 29.93 (30.05) 22[8c2363a6-6f86-46a9-a3f0-575eb94c5dd4] 30.11 (30.05) 00[be307291-892c-401e-82e1-3c216bccca88] 30.11 (30.05)
Active sources : 0
I believe you have seen the log I sent. It shows that there is no reasoning at all
I just did another test, after adding 16 ways, I restarted the service, observed its hardware and log, and found the problem
1:There is a serious mismatch between the number of analyzed video streams shown in the log and the actual additions to the vst
2:No analysis also no gpu usage, log shows no analysis
Please add 8 hours in the log to be Beijing time，The test ends at 19:30 on August 22
deepstream.log (163.7 KB)
Yes, I can see the log which you shared. But I need reproduce the issue in my side with JPS release. I added one same video stream into NVStreamer 8 times. Then I get 8 RTSP streams. Than I add the 8 RTSP stream to VST. The 8 RTSP streams work fine in DeepStream. I will have a test with restart DeepStream: docker restart deepstream.
Can you please help to reproduce the issue with JPS release?
I restarted DeepStream with: docker restart deepstream . DeepStream works fine after restart.
Fri Aug 23 01:43:16 2024
**PERF:
66[385734ac-bb20-4514-8acc-105d4614f345] 29.86 (30.68) 11[dfde20b0-f1e7-42fe-899b-3d11c3ae4ecd] 29.86 (30.66) 44[284ee652-1208-4847-bb95-8d775260b738] 29.86 (30.68) 33[0e264899-465c-4234-bf0a-68882e02b240] 30.06 (30.70) 55[66c43f0a-f28b-4a74-aece-a73a9a901f51] 29.86 (30.68) 77[85b681e4-f83a-4554-be82-f069a8b361f2] 29.89 (30.66) 22[8c2363a6-6f86-46a9-a3f0-575eb94c5dd4] 30.06 (30.70) 00[be307291-892c-401e-82e1-3c216bccca88] 29.89 (30.68)
Active sources : 0
Fri Aug 23 01:43:16 2024
**PERF:
Active sources : 8
Fri Aug 23 01:43:21 2024
**PERF:
66[385734ac-bb20-4514-8acc-105d4614f345] 30.10 (30.64) 11[dfde20b0-f1e7-42fe-899b-3d11c3ae4ecd] 30.10 (30.62) 44[284ee652-1208-4847-bb95-8d775260b738] 30.10 (30.64) 33[0e264899-465c-4234-bf0a-68882e02b240] 29.90 (30.64) 55[66c43f0a-f28b-4a74-aece-a73a9a901f51] 29.90 (30.62) 77[85b681e4-f83a-4554-be82-f069a8b361f2] 30.08 (30.62) 22[8c2363a6-6f86-46a9-a3f0-575eb94c5dd4] 29.93 (30.64) 00[be307291-892c-401e-82e1-3c216bccca88] 30.08 (30.64)
Active sources : 0
Please share the configuration of your jps. The vst cannot be accessed through port 30080, and the smart can be accessed through port 81
Uploading: 13时41分14秒_副本_deepstream.log…
I created 10 streams using nvstreamer, and added 8 streams in vst first, and found that the gpu was working. When I continued to add, I found that the log of deepstream started to remove all the existing streams, and there was no inference stream in it. The time was about 11:30 to 11 noon: 40, the time in the log is about 03:30-03:40
I can’t reproduce the issue with 10 streams. Here is the log of DeepStream with 10 streams. 8 on one DLA. 2 on another DLA.
**PERF:
22[8c2363a6-6f86-46a9-a3f0-575eb94c5dd4] 29.96 (30.00) 33[0e264899-465c-4234-bf0a-68882e02b240] 30.13 (29.99) 11[dfde20b0-f1e7-42fe-899b-3d11c3ae4ecd] 29.96 (29.99) 44[284ee652-1208-4847-bb95-8d775260b738] 29.92 (29.99) 66[385734ac-bb20-4514-8acc-105d4614f345] 29.96 (29.99) 55[66c43f0a-f28b-4a74-aece-a73a9a901f51] 29.96 (29.98) 00[be307291-892c-401e-82e1-3c216bccca88] 29.96 (29.98) 77[85b681e4-f83a-4554-be82-f069a8b361f2] 29.96 (29.98)
Active sources : 2
Fri Aug 23 07:15:03 2024
**PERF:
99[65a953d9-b63f-462d-ac04-8ab0cdb811cd] 30.08 (30.08) 88[71d01706-753d-4b2e-a17e-87ee96b55241] 30.29 (30.04)
Active sources : 8
Fri Aug 23 07:15:08 2024
**PERF:
22[8c2363a6-6f86-46a9-a3f0-575eb94c5dd4] 30.00 (30.00) 33[0e264899-465c-4234-bf0a-68882e02b240] 29.80 (29.99) 11[dfde20b0-f1e7-42fe-899b-3d11c3ae4ecd] 30.00 (30.00) 44[284ee652-1208-4847-bb95-8d775260b738] 30.00 (29.99) 66[385734ac-bb20-4514-8acc-105d4614f345] 30.00 (30.00) 55[66c43f0a-f28b-4a74-aece-a73a9a901f51] 29.80 (29.98) 00[be307291-892c-401e-82e1-3c216bccca88] 30.00 (29.99) 77[85b681e4-f83a-4554-be82-f069a8b361f2] 29.80 (29.98)
Active sources : 2
Fri Aug 23 07:15:08 2024
**PERF:
99[65a953d9-b63f-462d-ac04-8ab0cdb811cd] 29.92 (30.08) 88[71d01706-753d-4b2e-a17e-87ee96b55241] 29.92 (30.04)
Active sources : 8
Fri Aug 23 07:15:13 2024
**PERF:
22[8c2363a6-6f86-46a9-a3f0-575eb94c5dd4] 29.91 (30.00) 33[0e264899-465c-4234-bf0a-68882e02b240] 30.11 (29.99) 11[dfde20b0-f1e7-42fe-899b-3d11c3ae4ecd] 29.95 (29.99) 44[284ee652-1208-4847-bb95-8d775260b738] 30.11 (29.99) 66[385734ac-bb20-4514-8acc-105d4614f345] 29.91 (29.99) 55[66c43f0a-f28b-4a74-aece-a73a9a901f51] 30.15 (29.98) 00[be307291-892c-401e-82e1-3c216bccca88] 29.91 (29.98) 77[85b681e4-f83a-4554-be82-f069a8b361f2] 30.31 (29.99)
Active sources : 2
Fri Aug 23 07:15:13 2024
**PERF:
99[65a953d9-b63f-462d-ac04-8ab0cdb811cd] 30.08 (30.07) 88[71d01706-753d-4b2e-a17e-87ee96b55241] 30.08 (30.04)
Active sources : 8
Fri Aug 23 07:15:18 2024
I’m glad you can reproduce this problem. I just did it all over again. I did it all over again.
In severe cases, it will delete all streams. Or if the activity flow is 0, in a state of suspended animation. This is the same phenomenon that happens when we add streams directly to deepstream via the api
Sorry. I can’t reproduce the issue with 10 streams. I modified my previous comment.
Are you trying to add streams directly to deepstream via api
能否分享下您的pn26的配置，另外您可以使用其他机器的nvstreamer作为视频源
I download all the config files(ai_nvr-1.1.0.tar.gz) here without any changes: Jetson Platform Services Reference Workflow & Resources | NVIDIA NGC
Here is the supported stream count. Orin AGX can support 16 streams: AI NVR — Metropolis on Jetson documentation 0.1.0 documentation
Yes, we need those VST changes. But DeepStream haven’t any changes.
The problem with the number of streams in vst and deepstream was repeated
1: Since 10 channels have been added to the vst before, now restart the container of ai-nvr,
2: Remove the three sensors found by onvif and then add 6-way rtsp video streams
A mismatch was found between vst and deepstream
deepstream.log (2.1 MB)
emdx_01.log (22.7 KB)
emdx_02.log (21.6 KB)
redis.log (1.8 MB)
sdr-emdx.log (146.4 KB)
Please refer this topic: Testing ai-nvr adds 16-way video streams and deepstream does not perform inference - #4 by kesong
There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks