Missing key(s) in state_dict

We are facing the following issue during server aggregation (after local client training) on 4.0 GA Federated Learning.

Kindly review the configs (especially config_fed_server.json & config_fed_client.json) & logs as attached below.

PS: It seems to be using the default Monai UNet instead of our model

config.zip (14.7 KB)

config_fed_server.json (1005 Bytes)
config_fed_client.json (773 Bytes)

	Missing key(s) in state_dict: "segmenter.0.down.0.conv.weight", "segmenter.0.down.0.conv.bias", "segmenter.0.down.0.res.weight", "segmenter.0.down.0.res.bias", "segmenter.0.down.0.end_seq.1.weight", "segmenter.0.down.0.end_seq.1.bias", "segmenter.0.down.1.conv.weight", "segmenter.0.down.1.conv.bias", "segmenter.0.down.1.res.weight", "segmenter.0.down.1.res.bias", "segmenter.0.down.1.end_seq.1.weight", "segmenter.0.down.1.end_seq.1.bias", "segmenter.0.down.2.conv.weight", "segmenter.0.down.2.conv.bias", "segmenter.0.down.2.res.weight", "segmenter.0.down.2.res.bias", "segmenter.0.down.2.end_seq.1.weight", "segmenter.0.down.2.end_seq.1.bias", "segmenter.0.down.3.conv.weight", "segmenter.0.down.3.conv.bias", "segmenter.0.down.3.res.weight", "segmenter.0.down.3.res.bias", "segmenter.0.down.3.end_seq.1.weight", "segmenter.0.down.3.end_seq.1.bias", "segmenter.0.u.conv.weight", "segmenter.0.u.conv.bias", "segmenter.0.u.res.weight", "segmenter.0.u.res.bias", "segmenter.0.u.end_seq.1.weight", "segmenter.0.u.end_seq.1.bias", "segmenter.0.up.0.conv.weight", "segmenter.0.up.0.conv.bias", "segmenter.0.up.0.res.weight", "segmenter.0.up.0.res.bias", "segmenter.0.up.0.end_seq.1.weight", "segmenter.0.up.0.end_seq.1.bias", "segmenter.0.up.1.conv.weight", "segmenter.0.up.1.conv.bias", "segmenter.0.up.1.res.weight", "segmenter.0.up.1.res.bias", "segmenter.0.up.1.end_seq.1.weight", "segmenter.0.up.1.end_seq.1.bias", "segmenter.0.up.2.conv.weight", "segmenter.0.up.2.conv.bias", "segmenter.0.up.2.res.weight", "segmenter.0.up.2.res.bias", "segmenter.0.up.2.end_seq.1.weight", "segmenter.0.up.2.end_seq.1.bias", "segmenter.0.up.3.conv.weight", "segmenter.0.up.3.conv.bias", "segmenter.0.up.3.res.weight", "segmenter.0.up.3.res.bias", "segmenter.0.up.3.end_seq.1.weight", "segmenter.0.up.3.end_seq.1.bias", "segmenter.1.weight", "segmenter.1.bias". 
	Unexpected key(s) in state_dict: "model.0.conv.unit0.conv.weight", "model.0.conv.unit0.conv.bias", "model.0.conv.unit0.adn.N.weight", "model.0.conv.unit0.adn.N.bias", "model.0.conv.unit0.adn.N.running_mean", "model.0.conv.unit0.adn.N.running_var", "model.0.conv.unit0.adn.N.num_batches_tracked", "model.0.conv.unit0.adn.A.weight", "model.0.conv.unit1.conv.weight", "model.0.conv.unit1.conv.bias", "model.0.conv.unit1.adn.N.weight", "model.0.conv.unit1.adn.N.bias", "model.0.conv.unit1.adn.N.running_mean", "model.0.conv.unit1.adn.N.running_var", "model.0.conv.unit1.adn.N.num_batches_tracked", "model.0.conv.unit1.adn.A.weight", "model.0.residual.weight", "model.0.residual.bias", "model.1.submodule.0.conv.unit0.conv.weight", "model.1.submodule.0.conv.unit0.conv.bias", "model.1.submodule.0.conv.unit0.adn.N.weight", "model.1.submodule.0.conv.unit0.adn.N.bias", "model.1.submodule.0.conv.unit0.adn.N.running_mean", "model.1.submodule.0.conv.unit0.adn.N.running_var", "model.1.submodule.0.conv.unit0.adn.N.num_batches_tracked", "model.1.submodule.0.conv.unit0.adn.A.weight", "model.1.submodule.0.conv.unit1.conv.weight", "model.1.submodule.0.conv.unit1.conv.bias", "model.1.submodule.0.conv.unit1.adn.N.weight", "model.1.submodule.0.conv.unit1.adn.N.bias", "model.1.submodule.0.conv.unit1.adn.N.running_mean", "model.1.submodule.0.conv.unit1.adn.N.running_var", "model.1.submodule.0.conv.unit1.adn.N.num_batches_tracked", "model.1.submodule.0.conv.unit1.adn.A.weight", "model.1.submodule.0.residual.weight", "model.1.submodule.0.residual.bias", "model.1.submodule.1.submodule.0.conv.unit0.conv.weight", "model.1.submodule.1.submodule.0.conv.unit0.conv.bias", "model.1.submodule.1.submodule.0.conv.unit0.adn.N.weight", "model.1.submodule.1.submodule.0.conv.unit0.adn.N.bias", "model.1.submodule.1.submodule.0.conv.unit0.adn.N.running_mean", "model.1.submodule.1.submodule.0.conv.unit0.adn.N.running_var", "model.1.submodule.1.submodule.0.conv.unit0.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.0.conv.unit0.adn.A.weight", "model.1.submodule.1.submodule.0.conv.unit1.conv.weight", "model.1.submodule.1.submodule.0.conv.unit1.conv.bias", "model.1.submodule.1.submodule.0.conv.unit1.adn.N.weight", "model.1.submodule.1.submodule.0.conv.unit1.adn.N.bias", "model.1.submodule.1.submodule.0.conv.unit1.adn.N.running_mean", "model.1.submodule.1.submodule.0.conv.unit1.adn.N.running_var", "model.1.submodule.1.submodule.0.conv.unit1.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.0.conv.unit1.adn.A.weight", "model.1.submodule.1.submodule.0.residual.weight", "model.1.submodule.1.submodule.0.residual.bias", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.conv.weight", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.conv.bias", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.adn.N.weight", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.adn.N.bias", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.adn.N.running_mean", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.adn.N.running_var", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.adn.A.weight", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.conv.weight", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.conv.bias", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.adn.N.weight", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.adn.N.bias", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.adn.N.running_mean", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.adn.N.running_var", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.adn.A.weight", "model.1.submodule.1.submodule.1.submodule.0.residual.weight", "model.1.submodule.1.submodule.1.submodule.0.residual.bias", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.conv.weight", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.conv.bias", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.adn.N.weight", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.adn.N.bias", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.adn.N.running_mean", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.adn.N.running_var", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.adn.A.weight", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.conv.weight", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.conv.bias", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.adn.N.weight", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.adn.N.bias", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.adn.N.running_mean", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.adn.N.running_var", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.adn.A.weight", "model.1.submodule.1.submodule.1.submodule.1.submodule.residual.weight", "model.1.submodule.1.submodule.1.submodule.1.submodule.residual.bias", "model.1.submodule.1.submodule.1.submodule.2.0.conv.weight", "model.1.submodule.1.submodule.1.submodule.2.0.conv.bias", "model.1.submodule.1.submodule.1.submodule.2.0.adn.N.weight", "model.1.submodule.1.submodule.1.submodule.2.0.adn.N.bias", "model.1.submodule.1.submodule.1.submodule.2.0.adn.N.running_mean", "model.1.submodule.1.submodule.1.submodule.2.0.adn.N.running_var", "model.1.submodule.1.submodule.1.submodule.2.0.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.1.submodule.2.0.adn.A.weight", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.conv.weight", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.conv.bias", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.adn.N.weight", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.adn.N.bias", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.adn.N.running_mean", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.adn.N.running_var", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.adn.A.weight", "model.1.submodule.1.submodule.2.0.conv.weight", "model.1.submodule.1.submodule.2.0.conv.bias", "model.1.submodule.1.submodule.2.0.adn.N.weight", "model.1.submodule.1.submodule.2.0.adn.N.bias", "model.1.submodule.1.submodule.2.0.adn.N.running_mean", "model.1.submodule.1.submodule.2.0.adn.N.running_var", "model.1.submodule.1.submodule.2.0.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.2.0.adn.A.weight", "model.1.submodule.1.submodule.2.1.conv.unit0.conv.weight", "model.1.submodule.1.submodule.2.1.conv.unit0.conv.bias", "model.1.submodule.1.submodule.2.1.conv.unit0.adn.N.weight", "model.1.submodule.1.submodule.2.1.conv.unit0.adn.N.bias", "model.1.submodule.1.submodule.2.1.conv.unit0.adn.N.running_mean", "model.1.submodule.1.submodule.2.1.conv.unit0.adn.N.running_var", "model.1.submodule.1.submodule.2.1.conv.unit0.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.2.1.conv.unit0.adn.A.weight", "model.1.submodule.2.0.conv.weight", "model.1.submodule.2.0.conv.bias", "model.1.submodule.2.0.adn.N.weight", "model.1.submodule.2.0.adn.N.bias", "model.1.submodule.2.0.adn.N.running_mean", "model.1.submodule.2.0.adn.N.running_var", "model.1.submodule.2.0.adn.N.num_batches_tracked", "model.1.submodule.2.0.adn.A.weight", "model.1.submodule.2.1.conv.unit0.conv.weight", "model.1.submodule.2.1.conv.unit0.conv.bias", "model.1.submodule.2.1.conv.unit0.adn.N.weight", "model.1.submodule.2.1.conv.unit0.adn.N.bias", "model.1.submodule.2.1.conv.unit0.adn.N.running_mean", "model.1.submodule.2.1.conv.unit0.adn.N.running_var", "model.1.submodule.2.1.conv.unit0.adn.N.num_batches_tracked", "model.1.submodule.2.1.conv.unit0.adn.A.weight", "model.2.0.conv.weight", "model.2.0.conv.bias", "model.2.0.adn.N.weight", "model.2.0.adn.N.bias", "model.2.0.adn.N.running_mean", "model.2.0.adn.N.running_var", "model.2.0.adn.N.num_batches_tracked", "model.2.0.adn.A.weight", "model.2.1.conv.unit0.conv.weight", "model.2.1.conv.unit0.conv.bias". 
2021-06-21 00:27:54,449 - ignite.engine.engine.SupervisedEvaluator - ERROR - Exception: Error(s) in loading state_dict for NewLesionsUNet:
	Missing key(s) in state_dict: "segmenter.0.down.0.conv.weight", "segmenter.0.down.0.conv.bias", "segmenter.0.down.0.res.weight", "segmenter.0.down.0.res.bias", "segmenter.0.down.0.end_seq.1.weight", "segmenter.0.down.0.end_seq.1.bias", "segmenter.0.down.1.conv.weight", "segmenter.0.down.1.conv.bias", "segmenter.0.down.1.res.weight", "segmenter.0.down.1.res.bias", "segmenter.0.down.1.end_seq.1.weight", "segmenter.0.down.1.end_seq.1.bias", "segmenter.0.down.2.conv.weight", "segmenter.0.down.2.conv.bias", "segmenter.0.down.2.res.weight", "segmenter.0.down.2.res.bias", "segmenter.0.down.2.end_seq.1.weight", "segmenter.0.down.2.end_seq.1.bias", "segmenter.0.down.3.conv.weight", "segmenter.0.down.3.conv.bias", "segmenter.0.down.3.res.weight", "segmenter.0.down.3.res.bias", "segmenter.0.down.3.end_seq.1.weight", "segmenter.0.down.3.end_seq.1.bias", "segmenter.0.u.conv.weight", "segmenter.0.u.conv.bias", "segmenter.0.u.res.weight", "segmenter.0.u.res.bias", "segmenter.0.u.end_seq.1.weight", "segmenter.0.u.end_seq.1.bias", "segmenter.0.up.0.conv.weight", "segmenter.0.up.0.conv.bias", "segmenter.0.up.0.res.weight", "segmenter.0.up.0.res.bias", "segmenter.0.up.0.end_seq.1.weight", "segmenter.0.up.0.end_seq.1.bias", "segmenter.0.up.1.conv.weight", "segmenter.0.up.1.conv.bias", "segmenter.0.up.1.res.weight", "segmenter.0.up.1.res.bias", "segmenter.0.up.1.end_seq.1.weight", "segmenter.0.up.1.end_seq.1.bias", "segmenter.0.up.2.conv.weight", "segmenter.0.up.2.conv.bias", "segmenter.0.up.2.res.weight", "segmenter.0.up.2.res.bias", "segmenter.0.up.2.end_seq.1.weight", "segmenter.0.up.2.end_seq.1.bias", "segmenter.0.up.3.conv.weight", "segmenter.0.up.3.conv.bias", "segmenter.0.up.3.res.weight", "segmenter.0.up.3.res.bias", "segmenter.0.up.3.end_seq.1.weight", "segmenter.0.up.3.end_seq.1.bias", "segmenter.1.weight", "segmenter.1.bias". 
	Unexpected key(s) in state_dict: "model.0.conv.unit0.conv.weight", "model.0.conv.unit0.conv.bias", "model.0.conv.unit0.adn.N.weight", "model.0.conv.unit0.adn.N.bias", "model.0.conv.unit0.adn.N.running_mean", "model.0.conv.unit0.adn.N.running_var", "model.0.conv.unit0.adn.N.num_batches_tracked", "model.0.conv.unit0.adn.A.weight", "model.0.conv.unit1.conv.weight", "model.0.conv.unit1.conv.bias", "model.0.conv.unit1.adn.N.weight", "model.0.conv.unit1.adn.N.bias", "model.0.conv.unit1.adn.N.running_mean", "model.0.conv.unit1.adn.N.running_var", "model.0.conv.unit1.adn.N.num_batches_tracked", "model.0.conv.unit1.adn.A.weight", "model.0.residual.weight", "model.0.residual.bias", "model.1.submodule.0.conv.unit0.conv.weight", "model.1.submodule.0.conv.unit0.conv.bias", "model.1.submodule.0.conv.unit0.adn.N.weight", "model.1.submodule.0.conv.unit0.adn.N.bias", "model.1.submodule.0.conv.unit0.adn.N.running_mean", "model.1.submodule.0.conv.unit0.adn.N.running_var", "model.1.submodule.0.conv.unit0.adn.N.num_batches_tracked", "model.1.submodule.0.conv.unit0.adn.A.weight", "model.1.submodule.0.conv.unit1.conv.weight", "model.1.submodule.0.conv.unit1.conv.bias", "model.1.submodule.0.conv.unit1.adn.N.weight", "model.1.submodule.0.conv.unit1.adn.N.bias", "model.1.submodule.0.conv.unit1.adn.N.running_mean", "model.1.submodule.0.conv.unit1.adn.N.running_var", "model.1.submodule.0.conv.unit1.adn.N.num_batches_tracked", "model.1.submodule.0.conv.unit1.adn.A.weight", "model.1.submodule.0.residual.weight", "model.1.submodule.0.residual.bias", "model.1.submodule.1.submodule.0.conv.unit0.conv.weight", "model.1.submodule.1.submodule.0.conv.unit0.conv.bias", "model.1.submodule.1.submodule.0.conv.unit0.adn.N.weight", "model.1.submodule.1.submodule.0.conv.unit0.adn.N.bias", "model.1.submodule.1.submodule.0.conv.unit0.adn.N.running_mean", "model.1.submodule.1.submodule.0.conv.unit0.adn.N.running_var", "model.1.submodule.1.submodule.0.conv.unit0.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.0.conv.unit0.adn.A.weight", "model.1.submodule.1.submodule.0.conv.unit1.conv.weight", "model.1.submodule.1.submodule.0.conv.unit1.conv.bias", "model.1.submodule.1.submodule.0.conv.unit1.adn.N.weight", "model.1.submodule.1.submodule.0.conv.unit1.adn.N.bias", "model.1.submodule.1.submodule.0.conv.unit1.adn.N.running_mean", "model.1.submodule.1.submodule.0.conv.unit1.adn.N.running_var", "model.1.submodule.1.submodule.0.conv.unit1.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.0.conv.unit1.adn.A.weight", "model.1.submodule.1.submodule.0.residual.weight", "model.1.submodule.1.submodule.0.residual.bias", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.conv.weight", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.conv.bias", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.adn.N.weight", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.adn.N.bias", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.adn.N.running_mean", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.adn.N.running_var", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.adn.A.weight", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.conv.weight", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.conv.bias", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.adn.N.weight", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.adn.N.bias", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.adn.N.running_mean", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.adn.N.running_var", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.adn.A.weight", "model.1.submodule.1.submodule.1.submodule.0.residual.weight", "model.1.submodule.1.submodule.1.submodule.0.residual.bias", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.conv.weight", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.conv.bias", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.adn.N.weight", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.adn.N.bias", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.adn.N.running_mean", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.adn.N.running_var", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.adn.A.weight", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.conv.weight", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.conv.bias", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.adn.N.weight", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.adn.N.bias", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.adn.N.running_mean", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.adn.N.running_var", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.adn.A.weight", "model.1.submodule.1.submodule.1.submodule.1.submodule.residual.weight", "model.1.submodule.1.submodule.1.submodule.1.submodule.residual.bias", "model.1.submodule.1.submodule.1.submodule.2.0.conv.weight", "model.1.submodule.1.submodule.1.submodule.2.0.conv.bias", "model.1.submodule.1.submodule.1.submodule.2.0.adn.N.weight", "model.1.submodule.1.submodule.1.submodule.2.0.adn.N.bias", "model.1.submodule.1.submodule.1.submodule.2.0.adn.N.running_mean", "model.1.submodule.1.submodule.1.submodule.2.0.adn.N.running_var", "model.1.submodule.1.submodule.1.submodule.2.0.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.1.submodule.2.0.adn.A.weight", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.conv.weight", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.conv.bias", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.adn.N.weight", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.adn.N.bias", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.adn.N.running_mean", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.adn.N.running_var", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.adn.A.weight", "model.1.submodule.1.submodule.2.0.conv.weight", "model.1.submodule.1.submodule.2.0.conv.bias", "model.1.submodule.1.submodule.2.0.adn.N.weight", "model.1.submodule.1.submodule.2.0.adn.N.bias", "model.1.submodule.1.submodule.2.0.adn.N.running_mean", "model.1.submodule.1.submodule.2.0.adn.N.running_var", "model.1.submodule.1.submodule.2.0.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.2.0.adn.A.weight", "model.1.submodule.1.submodule.2.1.conv.unit0.conv.weight", "model.1.submodule.1.submodule.2.1.conv.unit0.conv.bias", "model.1.submodule.1.submodule.2.1.conv.unit0.adn.N.weight", "model.1.submodule.1.submodule.2.1.conv.unit0.adn.N.bias", "model.1.submodule.1.submodule.2.1.conv.unit0.adn.N.running_mean", "model.1.submodule.1.submodule.2.1.conv.unit0.adn.N.running_var", "model.1.submodule.1.submodule.2.1.conv.unit0.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.2.1.conv.unit0.adn.A.weight", "model.1.submodule.2.0.conv.weight", "model.1.submodule.2.0.conv.bias", "model.1.submodule.2.0.adn.N.weight", "model.1.submodule.2.0.adn.N.bias", "model.1.submodule.2.0.adn.N.running_mean", "model.1.submodule.2.0.adn.N.running_var", "model.1.submodule.2.0.adn.N.num_batches_tracked", "model.1.submodule.2.0.adn.A.weight", "model.1.submodule.2.1.conv.unit0.conv.weight", "model.1.submodule.2.1.conv.unit0.conv.bias", "model.1.submodule.2.1.conv.unit0.adn.N.weight", "model.1.submodule.2.1.conv.unit0.adn.N.bias", "model.1.submodule.2.1.conv.unit0.adn.N.running_mean", "model.1.submodule.2.1.conv.unit0.adn.N.running_var", "model.1.submodule.2.1.conv.unit0.adn.N.num_batches_tracked", "model.1.submodule.2.1.conv.unit0.adn.A.weight", "model.2.0.conv.weight", "model.2.0.conv.bias", "model.2.0.adn.N.weight", "model.2.0.adn.N.bias", "model.2.0.adn.N.running_mean", "model.2.0.adn.N.running_var", "model.2.0.adn.N.num_batches_tracked", "model.2.0.adn.A.weight", "model.2.1.conv.unit0.conv.weight", "model.2.1.conv.unit0.conv.bias".

lesion-activity-{Epoch-1,Num_of_rounds-1}-GA-Server.txt (10.6 KB)
lesion-activity-{Epoch-1,Num_of_rounds-1}-GA-ClientA.txt (108.8 KB)
lesion-activity-{Epoch-1,Num_of_rounds-1}-GA-ClientB.txt (91.0 KB)

It appears that the “lesion-activity-clara-fl.NewLesionsUNet.model.NewLesionsUNet” model data didn’t got saved correctly on the server by the “PTFileModelPersistor”. Both clients were not able to load the final global model for validation. Could you please check if the final global model has the correct contents.

Reply 1. Yes. We need ModelPersistor in both server and client. During FL, there is only model parameters pushed and pulled. The model structure is in ModelPersistor, which interprets the parameters and make the complete model for training.
Reply 2. Yes, the reason is same as the previous. we need this function to complete the model for understanding right whole one.
Reply 3. Two arguments that you claimed are correct. I also check the path pointing to your customized model structure. I don’t see the reason why default Unet comes out the right. will check with EE again.

PTFileModelPersistor extracts all the parameters from the model weights then save in the FL_global_model.pt. Can you check if your model also contains these additional parameters from UNet model?