How to compare different federated algorithm

I am trying to implement and compare some different federated learning algorithms using Clara training SDK. I have some questions:

  • How to choose subset of client to perform local update? (As paper Communication-Efficient Learning of Deep Networks from Decentralized Data mentioned). It very important to keep the same clients chosen at the same round when compare different algorithms.
    Currently, The FL server begins model aggregation after accepting updates from the minimum number of FL clients required. Not all clients have the same amount of data in the FL setting, i.e., the client containing less data will quicker send its update back to the FL server. The FL server will aggregate the updates sent by clients with less data at every round.

For example, there are 6 clients and min_num_clients is 2. The 6 clients got their token:

2020-10-10 08:12:29,299 - FederatedServer - INFO - Client: New client client0@172.17.0.3 joined. Sent token: 93cd4bd0-4fdd-407b-a4da-facb22a9c89d.  Total clients: 1
2020-10-10 08:12:34,439 - FederatedServer - INFO - Client: New client client1@172.17.0.4 joined. Sent token: 03a3e6a9-89db-447d-8a06-65e2499b4809.  Total clients: 2
2020-10-10 08:12:39,122 - FederatedServer - INFO - Client: New client client2@172.17.0.5 joined. Sent token: e57e866c-678e-47db-a475-3bba4745464f.  Total clients: 3
2020-10-10 08:12:40,988 - FederatedServer - INFO - Client: New client client3@172.17.0.6 joined. Sent token: 22c4665e-5574-43ef-8a3d-1c612fb53d45.  Total clients: 4
2020-10-10 08:12:44,382 - FederatedServer - INFO - Client: New client client4@172.17.0.7 joined. Sent token: 50858324-ccbd-4f74-84e0-e1c14d694d1f.  Total clients: 5
2020-10-10 08:12:50,104 - FederatedServer - INFO - Client: New client client5@172.17.0.9 joined. Sent token: b1c83614-3c31-43bb-a8a5-9cdd6a044411.  Total clients: 6

Client 2(starts with e57e) and 4(starts with 5085) have less data, so FL server only aggregated the updates sent by client 2 and 4

2020-10-10 08:13:32,711 - ServerModelManager - INFO - aggregating 2 updates at round 0
2020-10-10 08:13:33,094 - ServerModelManager - INFO - Saving model checkpoint at: /mmar/clara_seg_ct_brats_adaptive_fl/commands/../models/fedadam_server_lr_1.0_client_lr_0.0001_eps_0.01/FL_global_model.ckpt
2020-10-10 08:13:34,083 - AssignVariables - INFO - Vars from remote 252, Vars from local 252, vars matched 252 of 252 local
2020-10-10 08:13:34,796 - FederatedServer - INFO - Round time: 106 second(s).
2020-10-10 08:14:04,442 - FederatedServer - INFO - received brats_segmentation_50858324-ccbd-4f74-84e0-e1c14d694d1f_1 (56441952 Bytes, 30 seconds)
2020-10-10 08:14:14,003 - FederatedServer - INFO - received brats_segmentation_e57e866c-678e-47db-a475-3bba4745464f_1 (56441952 Bytes, 40 seconds)
2020-10-10 08:14:24,013 - FederatedServer - INFO - > aggregating: 1
2020-10-10 08:14:24,014 - ServerModelManager - INFO - aggregating 2 updates at round 1
2020-10-10 08:14:24,333 - ServerModelManager - INFO - Saving model checkpoint at: /mmar/clara_seg_ct_brats_adaptive_fl/commands/../models/fedadam_server_lr_1.0_client_lr_0.0001_eps_0.01/FL_global_model.ckpt
2020-10-10 08:14:24,402 - AssignVariables - INFO - Vars from remote 252, Vars from local 252, vars matched 252 of 252 local
2020-10-10 08:14:24,817 - FederatedServer - INFO - Round time: 50 second(s).
2020-10-10 08:14:57,777 - FederatedServer - INFO - received brats_segmentation_50858324-ccbd-4f74-84e0-e1c14d694d1f_2 (56441952 Bytes, 33 seconds)
2020-10-10 08:15:04,682 - FederatedServer - INFO - received brats_segmentation_e57e866c-678e-47db-a475-3bba4745464f_2 (56441952 Bytes, 40 seconds)
2020-10-10 08:15:14,693 - FederatedServer - INFO - > aggregating: 2
2020-10-10 08:15:14,693 - ServerModelManager - INFO - aggregating 2 updates at round 2
2020-10-10 08:15:15,044 - ServerModelManager - INFO - Saving model checkpoint at: /mmar/clara_seg_ct_brats_adaptive_fl/commands/../models/fedadam_server_lr_1.0_client_lr_0.0001_eps_0.01/FL_global_model.ckpt
2020-10-10 08:15:15,114 - AssignVariables - INFO - Vars from remote 252, Vars from local 252, vars matched 252 of 252 local
2020-10-10 08:15:15,548 - FederatedServer - INFO - Round time: 51 second(s).
2020-10-10 08:15:45,511 - FederatedServer - INFO - received brats_segmentation_50858324-ccbd-4f74-84e0-e1c14d694d1f_3 (56441952 Bytes, 30 seconds)
2020-10-10 08:15:55,924 - FederatedServer - INFO - received brats_segmentation_e57e866c-678e-47db-a475-3bba4745464f_3 (56441952 Bytes, 40 seconds)
2020-10-10 08:16:05,934 - FederatedServer - INFO - > aggregating: 3
2020-10-10 08:16:05,934 - ServerModelManager - INFO - aggregating 2 updates at round 3
2020-10-10 08:16:06,281 - ServerModelManager - INFO - Saving model checkpoint at: /mmar/clara_seg_ct_brats_adaptive_fl/commands/../models/fedadam_server_lr_1.0_client_lr_0.0001_eps_0.01/FL_global_model.ckpt
2020-10-10 08:16:06,376 - AssignVariables - INFO - Vars from remote 252, Vars from local 252, vars matched 252 of 252 local
2020-10-10 08:16:06,841 - FederatedServer - INFO - Round time: 51 second(s).
2020-10-10 08:16:37,946 - FederatedServer - INFO - received brats_segmentation_50858324-ccbd-4f74-84e0-e1c14d694d1f_4 (56441952 Bytes, 31 seconds)
2020-10-10 08:16:45,902 - FederatedServer - INFO - received brats_segmentation_e57e866c-678e-47db-a475-3bba4745464f_4 (56441952 Bytes, 39 seconds)
2020-10-10 08:16:55,913 - FederatedServer - INFO - > aggregating: 4
2020-10-10 08:16:55,913 - ServerModelManager - INFO - aggregating 2 updates at round 4
2020-10-10 08:16:56,237 - ServerModelManager - INFO - Saving model checkpoint at: /mmar/clara_seg_ct_brats_adaptive_fl/commands/../models/fedadam_server_lr_1.0_client_lr_0.0001_eps_0.01/FL_global_model.ckpt
2020-10-10 08:16:56,308 - AssignVariables - INFO - Vars from remote 252, Vars from local 252, vars matched 252 of 252 local
2020-10-10 08:16:56,793 - FederatedServer - INFO - Round time: 50 second(s).
2020-10-10 08:17:25,979 - FederatedServer - INFO - received brats_segmentation_50858324-ccbd-4f74-84e0-e1c14d694d1f_5 (56441952 Bytes, 29 seconds)
2020-10-10 08:17:36,025 - FederatedServer - INFO - received brats_segmentation_e57e866c-678e-47db-a475-3bba4745464f_5 (56441952 Bytes, 40 seconds)
2020-10-10 08:17:46,036 - FederatedServer - INFO - > aggregating: 5
2020-10-10 08:17:46,036 - ServerModelManager - INFO - aggregating 2 updates at round 5
2020-10-10 08:17:46,395 - ServerModelManager - INFO - Saving model checkpoint at: /mmar/clara_seg_ct_brats_adaptive_fl/commands/../models/fedadam_server_lr_1.0_client_lr_0.0001_eps_0.01/FL_global_model.ckpt
2020-10-10 08:17:46,466 - AssignVariables - INFO - Vars from remote 252, Vars from local 252, vars matched 252 of 252 local
  • Is it possible for us to implement third-part FL server?

Thanks. Currently, the only way to guarantee that the same clients are contributing to each round is to set the min_num_clients to the same number as registered clients.