ClientManager - INFO - Re-activate dead client:org1-a with token

Sometime during our previous FL run, the client org1-a was attached to the server and it comes up on running a new instance of the server container each time.

To elaborate further, even on shutting down the server and client from the previous run and then starting just the server for a new FL run throws the following message:

2021-06-08 07:09:00,597 - ClientManager - INFO - Re-activate dead client:org1-a with token: e146801a-a9e3-434f-914c-3251458625be

The issue is that we aren’t then able to get the org1-a client joined as it would throw

start fl because process of 239 does not exist
new pid 262
2021-06-08 07:14:59,219 - Communicator - INFO - Action: client_registration grpc communication error. retry: 1500, First start till now: 0.013784170150756836 seconds.
2021-06-08 07:14:59,219 - Communicator - INFO - <_InactiveRpcError of RPC that terminated with:
status = StatusCode.FAILED_PRECONDITION
details = "Client ID already registered as a client: org1-a"
`debug_error_string = "{"created":"@1623136499.218869147","description":"Error received from peer ipv4:10.65.199.142:8002","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Client ID already registered as a client: org1-a","grpc_status":9}"`
  1. How we can start a fresh session of the server without this previous dead client being added? Unable to deregister the client from the server.

  2. How can we reactivate a dead client, given the token?

The reason the “re-active dead client” after the server restarted is because the previous client process is still running when the server got restarted. In this case, if you restart the client again. the server will show the “Client ID already registered.” message.

Please check on the client machine to make sure all the process have been killed properly. And you can issue the admin command “remove_client Cliend-Name” to allow the new start client to register.

Also to use the “shutdown client” command to ensure properly client shutdown before restarting the client process.

Thank you, this fixes it! Many thanks!