Issues with starting federated learning server training

At the moment I am testing out federated learning locally with two clients and a server. Everything is working as expected up until I run the start server command in order begin training. This is the entire log from the training run that pops up right before the server stops training again:

2021-03-09 13:47:42,279 - FLServerConfiger - INFO - DETERMINISM IS OFF
2021-03-09 13:47:42,290 - BaseServer - INFO - Round time: 1615297662 second(s).
2021-03-09 13:47:42,290 - BaseServer - INFO - Round time: 54 second(s).
Traceback (most recent call last):
File “server/sai.py”, line 385, in start_server_training
File “server/fed_server.py”, line 839, in start
File “server/server_model_manager.py”, line 89, in initialize
AttributeError: ‘NoneType’ object has no attribute ‘initialize’
FL server execution exception: ‘NoneType’ object has no attribute ‘initialize’
2021-03-09 13:47:42,290 - BaseServer - INFO - Stopping server training…
2021-03-09 13:47:42,290 - ServerModelManager - INFO - closing the model manager
2021-03-09 13:47:42,290 - BaseServer - INFO - Round time: less than a second(s).

The server status is then

FL run number:1
FL server status: training stopped
Registered clients: 2
------------------------------------------------------------------------
| CLIENT NAME | TOKEN | SUBMITTED MODEL |
------------------------------------------------------------------------
| flclient1 | 3d5d666b-48c5-4ba0-a7be-4325d763578b | |
| flclient2 | 78764b2e-7dcf-4db0-843a-ec3326f35433 | |
------------------------------------------------------------------------

The run_1 folder is looking just as expected, and the server got its MMAR using the upload_folder command. Both clients are connected, have deployed MMARs in the correct format, and is otherwise ready to go. Obviously I cannot start client training before the server training has been started. Does anyone know what might be the issue here?

Hi
Thanks for your interest in clara train.
Could you verify using v3.1.01 and also the latest notebooks for FL found at clara-train-examples/NoteBooks/FL at master · NVIDIA/clara-train-examples · GitHub
Please check the adminMMAR folder as the configs has changed in V3.1 from V3.0

Could you verify that the FL notebooks run as expected before going forward

Thank you for the reply! I am not certain exactly what caused the errors as all the configurations were correctly set, but I resolved the issue by regenerating all the packages and trying with a new set of clients/server.