Milestone VPS - removing a stream/camera doesn't free the GPU RAM

Accelerated Computing Intelligent Video Analytics General
#1

We are using the Milestone VPS plugin with Deepstream Pipeline that includes different Detection/Classification Models. When we connect a camera through Milestone everything is working fine except when disconnecting it. Upon disconnection, the reserved GPU memory is not freed which leads to the accumulation of reserved but unused memory. This is what is printed on the IVA/Pipeline side:

ReceiveLoop websocket exception: The remote party closed the WebSocket connection without completing the close handshake.

In a big system with many connections/disconnections happening, this leads to full GPU memory and the crash of the system.

Can you please assist on this?

image

image

image

#3

Hi @mohammad1 ,
COuld you please provide the setup info as other topics?

Is this log from Milestone VPS plugin?

#4

Thanks for following up.

Please note that we are using Milestone 2020 R2 and VPS From MIPSDK 2020 R1. Milestone is running on a Windows OS while the VPS/Deepstream Pipeline is running on Ubuntu 18.04.
Are these informative or do you require more info about the setup?

The log is from the server running VPS/Deepstream pipeline.

Also note that this issue is happening whether we manually delete hardware from Milestone or there was a disconnection between the Milestone and the VPS/Deepstream server (e.g. Milestone server down).

Delete VPS
Delete VPS1920×1080 200 KB

#5

Sorry! This is the first time I heard about Milestone 2020 R2 and VPS From MIPSDK 2020 R1 are. And I have no idea what it will happen to DeepStream when “Delete Hardware”.
If you could help me understand what happens to DeepStream when disconnecting it, I may could try to check why DeepStream behaves as what you see.

#7

According to the discussion with the Milestone team, it was a bug from the VPS code. They released the version that corrects this error. It can be found here.