NOTE: MOFED driver for multi-node communication was not detected.
      Multi-node communication performance may be reduced.

siddharth@snac-dell:/workspace$ ls
audit.log	  data			    run_1    run_44   run_517  run_613	run_619  run_712  run_72   run_810  run_85   run_921  run_964  transfer
challenge_data	  lesion-activity-clara-fl  run_100  run_502  run_524  run_614	run_626  run_713  run_764  run_814  run_88   run_946  run_988
client_token.txt  log.txt		    run_23   run_514  run_552  run_616	run_66	 run_716  run_773  run_816  run_916  run_948  run_998
daemon_pid.fl	  pid.fl		    run_416  run_516  run_612  run_618	run_686  run_718  run_784  run_84   run_918  run_949  startup
siddharth@snac-dell:/workspace$ cd startup/
siddharth@snac-dell:/workspace/startup$ 
siddharth@snac-dell:/workspace/startup$ ls
client.crt  client.key	client_context.tenseal	docker.sh  fed_client.json  log.config	readme.txt  rootCA.pem	signature.pkl  start.sh  sub_start.sh
siddharth@snac-dell:/workspace/startup$ 
siddharth@snac-dell:/workspace/startup$ 
siddharth@snac-dell:/workspace/startup$ ./start.sh 
WORKSPACE set to /workspace/startup/..
siddharth@snac-dell:/workspace/startup$ WORKSPACE set to /workspace/startup/..
PYTHONPATH is /local/custom::/opt/nvidia/medical
start fl because process of 262 does not exist
new pid 239
2021-06-20 23:32:54,556 - FederatedClient - INFO - Successfully registered client:org1-b for lesion_activity. Got token:6d00231d-4c8f-458e-8e50-e70518a046af
created /tmp/fl/org1-b/comm/training/x
created /tmp/fl/org1-b/comm/training/y
created /tmp/fl/org1-b/comm/training/t
^C
siddharth@snac-dell:/workspace/startup$ exit
exit

siddharth@snac-dell:~/packages7/org1-b/startup$ 
siddharth@snac-dell:~/packages7/org1-b/startup$ cler

Command 'cler' not found, did you mean:

  command 'clex' from deb clex
  command 'clear' from deb ncurses-bin
  command 'cver' from deb gplcver
  command 'cleo' from deb cleo

Try: sudo apt install <deb name>

siddharth@snac-dell:~/packages7/org1-b/startup$ clear

siddharth@snac-dell:~/packages7/org1-b/startup$ 
siddharth@snac-dell:~/packages7/org1-b/startup$ 
siddharth@snac-dell:~/packages7/org1-b/startup$ ls
client_context.tenseal	client.crt  client.key	docker.sh  fed_client.json  log.config	readme.txt  rootCA.pem	signature.pkl  start.sh  sub_start.sh
siddharth@snac-dell:~/packages7/org1-b/startup$ 
siddharth@snac-dell:~/packages7/org1-b/startup$ ./docker.sh 
Starting docker with nvcr.io/nvidia/clara-train-sdk:v4.0

=============
== PyTorch ==
=============

NVIDIA Release 21.02 (build 20138038)
PyTorch Version 1.8.0a0+52ea372

Container image Copyright (c) 2021, NVIDIA CORPORATION.  All rights reserved.

Copyright (c) 2014-2021 Facebook Inc.
Copyright (c) 2011-2014 Idiap Research Institute (Ronan Collobert)
Copyright (c) 2012-2014 Deepmind Technologies    (Koray Kavukcuoglu)
Copyright (c) 2011-2012 NEC Laboratories America (Koray Kavukcuoglu)
Copyright (c) 2011-2013 NYU                      (Clement Farabet)
Copyright (c) 2006-2010 NEC Laboratories America (Ronan Collobert, Leon Bottou, Iain Melvin, Jason Weston)
Copyright (c) 2006      Idiap Research Institute (Samy Bengio)
Copyright (c) 2001-2004 Idiap Research Institute (Ronan Collobert, Samy Bengio, Johnny Mariethoz)
Copyright (c) 2015      Google Inc.
Copyright (c) 2015      Yangqing Jia
Copyright (c) 2013-2016 The Caffe contributors
All rights reserved.

NVIDIA Deep Learning Profiler (dlprof) Copyright (c) 2021, NVIDIA CORPORATION.  All rights reserved.

Various files include modifications (c) NVIDIA CORPORATION.  All rights reserved.

This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:
https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license

NOTE: MOFED driver for multi-node communication was not detected.
      Multi-node communication performance may be reduced.

siddharth@snac-dell:/workspace$ ls
audit.log	  data			    run_1    run_44   run_517  run_613	run_619  run_686  run_718  run_784  run_84   run_918  run_949  startup
challenge_data	  lesion-activity-clara-fl  run_100  run_502  run_524  run_614	run_626  run_712  run_72   run_810  run_85   run_921  run_964  transfer
client_token.txt  log.txt		    run_23   run_514  run_552  run_616	run_66	 run_713  run_764  run_814  run_88   run_946  run_988
daemon_pid.fl	  pid.fl		    run_416  run_516  run_612  run_618	run_671  run_716  run_773  run_816  run_916  run_948  run_998
siddharth@snac-dell:/workspace$ cd startup/
siddharth@snac-dell:/workspace/startup$ ls
client.crt  client.key	client_context.tenseal	docker.sh  fed_client.json  log.config	readme.txt  rootCA.pem	signature.pkl  start.sh  sub_start.sh
siddharth@snac-dell:/workspace/startup$ ./start.sh 
WORKSPACE set to /workspace/startup/..
siddharth@snac-dell:/workspace/startup$ WORKSPACE set to /workspace/startup/..
PYTHONPATH is /local/custom::/opt/nvidia/medical
There seems to be one instance, pid=221, running.
If you are sure it's not the case, please kill process 221.

siddharth@snac-dell:/workspace/startup$ ./start.sh 
WORKSPACE set to /workspace/startup/..
siddharth@snac-dell:/workspace/startup$ WORKSPACE set to /workspace/startup/..
PYTHONPATH is /local/custom::/opt/nvidia/medical
start fl because process of 239 does not exist
new pid 262
2021-06-20 23:37:50,263 - FederatedClient - INFO - Successfully registered client:org1-b for lesion_activity. Got token:70a43d1e-9233-4ca6-954c-fb168a8b2dad
created /tmp/fl/org1-b/comm/training/x
created /tmp/fl/org1-b/comm/training/y
created /tmp/fl/org1-b/comm/training/t
2021-06-20 23:38:47,779 - ClientAdminInterface - INFO - Starting client training. rank: 0
training child process ID: 405
starting the client .....
token is: 70a43d1e-9233-4ca6-954c-fb168a8b2dad run_number is: 711 uid: org1-b listen_port: 60431
2021-06-20 23:38:48,738 - matplotlib - WARNING - Matplotlib created a temporary config/cache directory at /tmp/matplotlib-ihofvdpb because the default path (/home/siddharth/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.
2021-06-20 23:38:48,772 - matplotlib.font_manager - INFO - Generating new fontManager, this may take some time...
2021-06-20 23:38:48,838 - ProcessExecutor - INFO - waiting for process to finish
Created the listener on port: 60431
2021-06-20 23:38:52,458 - torch.distributed.nn.jit.instantiator - INFO - Created a temporary directory at /tmp/tmp_tjuhwtj
2021-06-20 23:38:52,459 - torch.distributed.nn.jit.instantiator - INFO - Writing /tmp/tmp_tjuhwtj/_remote_module_non_sriptable.py
no max_epochs specified for SupervisedTrainer args, use global var 'epochs: 10'.
========== Train Config Result ===========
Num Epochs:  10
Use GPU:  True
Multi GPU:  False
Automatic Mixed Precision:  Disabled
Determinism Training:  Enabled
cuDNN BenchMark:  False
CUDA Matmul Allow TF32:  True
cuDNN Allow TF32:  True
Model:  <class 'lesion-activity-clara-fl.NewLesionsUNet.model.NewLesionsUNet'>
Loss:  <class 'lesion-activity-clara-fl.NewLesionsUNet.model.MyDiCELoss'>
Optimizer:  <class 'torch.optim.adam.Adam'>
LR Scheduler:  <class 'NoneType'>
Train Dataset:  <class 'lesion-activity-clara-fl.NewLesionsUNet.dataset.LongitudinalCroppingDataset'>
Train DataLoader:  <class 'monai.data.dataloader.DataLoader'>
Train Transform #1: <class 'monai.transforms.utility.dictionary.ToTensord'>
Validate Dataset:  <class 'monai.data.dataset.Dataset'>
Validate DataLoader:  <class 'monai.data.dataloader.DataLoader'>
Validate Transform #1: <class 'monai.transforms.io.dictionary.LoadImaged'>
Validate Transform #2: <class 'lesion-activity-clara-fl.NewLesionsUNet.pre_transform.GetMask'>
Validate Transform #3: <class 'monai.transforms.intensity.dictionary.NormalizeIntensityd'>
Validate Transform #4: <class 'monai.transforms.utility.dictionary.ToTensord'>
Train Handler #1: <class 'monai.handlers.validation_handler.ValidationHandler'>
Train Handler #2: <class 'monai.handlers.checkpoint_saver.CheckpointSaver'>
Train Handler #3: <class 'monai.handlers.stats_handler.StatsHandler'>
Train Handler #4: <class 'monai.handlers.tensorboard_handlers.TensorBoardStatsHandler'>
Validate Handler #1: <class 'monai.handlers.stats_handler.StatsHandler'>
Validate Handler #2: <class 'monai.handlers.tensorboard_handlers.TensorBoardStatsHandler'>
Validate Handler #3: <class 'monai.handlers.checkpoint_saver.CheckpointSaver'>
Validate Post Transforms #1: <class 'monai.transforms.post.dictionary.Activationsd'>
Validate Post Transforms #2: <class 'monai.transforms.post.dictionary.AsDiscreted'>
Validate Inferer:  <class 'monai.inferers.inferer.SlidingWindowInferer'>
Validate Key Metric:  <class 'monai.handlers.mean_dice.MeanDice'>
Validate Additional Metric #My own validation mean dice loss: <class 'lesion-activity-clara-fl.NewLesionsUNet.model.MyValLoss'>
Train Inferer:  <class 'monai.inferers.inferer.SimpleInferer'>
========== End of Train Config Result ===========
2021-06-20 23:40:42,261 - FederatedClient - INFO - Starting to fetch global model.
2021-06-20 23:40:45,151 - Communicator - INFO - Received lesion_activity model at round 0 (19259350 Bytes). GetModel time: 2.8815925121307373 seconds
Get global model for round: 0
pull_models completed. Status:True rank:0
2021-06-20 23:40:45,171 - ClientTrainer - INFO - ClientTrainer abort signal: False
2021-06-20 23:40:45,172 - AssignVariables - INFO - Vars 148 of 0 assigned.
2021-06-20 23:40:45,175 - ignite.engine.engine.SupervisedEvaluator - INFO - Engine run resuming from iteration 0, epoch 0 until 1 epochs
2021-06-20 23:46:59,760 - ignite.engine.engine.SupervisedEvaluator - INFO - Got new best metric of Monai's validation mean dice loss: 2.1376723452704027e-05
2021-06-20 23:46:59,760 - ignite.engine.engine.SupervisedEvaluator - INFO - Epoch[1] Metrics -- Monai's validation mean dice loss: 0.0000 My own validation mean dice loss: 1.0000 
2021-06-20 23:46:59,760 - ignite.engine.engine.SupervisedEvaluator - INFO - Key metric: Monai's validation mean dice loss best value: 2.1376723452704027e-05 at epoch: 1
2021-06-20 23:47:00,083 - ignite.engine.engine.SupervisedEvaluator - INFO - Epoch[1] Complete. Time taken: 00:06:15
2021-06-20 23:47:00,083 - ignite.engine.engine.SupervisedEvaluator - INFO - Engine run complete. Time taken: 00:06:15
2021-06-20 23:47:00,176 - ignite.engine.engine.SupervisedTrainer - INFO - Engine run resuming from iteration 0, epoch 0 until 1 epochs
2021-06-20 23:47:02,229 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 1/133 -- train_loss: 1.7602 
2021-06-20 23:47:02,446 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 2/133 -- train_loss: 1.1917 
2021-06-20 23:47:02,640 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 3/133 -- train_loss: 1.1054 
2021-06-20 23:47:02,847 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 4/133 -- train_loss: 1.0922 
2021-06-20 23:47:03,036 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 5/133 -- train_loss: 1.0849 
2021-06-20 23:47:03,411 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 6/133 -- train_loss: 1.0681 
2021-06-20 23:47:03,599 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 7/133 -- train_loss: 1.0757 
2021-06-20 23:47:03,787 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 8/133 -- train_loss: 1.0723 
2021-06-20 23:47:03,974 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 9/133 -- train_loss: 1.0671 
2021-06-20 23:47:04,164 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 10/133 -- train_loss: 1.0625 
2021-06-20 23:47:04,353 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 11/133 -- train_loss: 1.0464 
2021-06-20 23:47:04,544 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 12/133 -- train_loss: 1.0545 
2021-06-20 23:47:04,734 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 13/133 -- train_loss: 1.0330 
2021-06-20 23:47:04,939 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 14/133 -- train_loss: 1.0220 
2021-06-20 23:47:05,129 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 15/133 -- train_loss: 1.0325 
2021-06-20 23:47:05,318 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 16/133 -- train_loss: 1.0218 
2021-06-20 23:47:05,602 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 17/133 -- train_loss: 1.0125 
2021-06-20 23:47:05,791 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 18/133 -- train_loss: 1.0105 
2021-06-20 23:47:05,980 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 19/133 -- train_loss: 1.0324 
2021-06-20 23:47:06,170 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 20/133 -- train_loss: 1.0251 
2021-06-20 23:47:06,359 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 21/133 -- train_loss: 1.0258 
2021-06-20 23:47:06,681 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 22/133 -- train_loss: 1.0104 
2021-06-20 23:47:06,870 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 23/133 -- train_loss: 0.9855 
2021-06-20 23:47:07,058 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 24/133 -- train_loss: 1.0196 
2021-06-20 23:47:07,247 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 25/133 -- train_loss: 1.0266 
2021-06-20 23:47:07,437 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 26/133 -- train_loss: 0.9902 
2021-06-20 23:47:07,625 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 27/133 -- train_loss: 1.0016 
2021-06-20 23:47:07,815 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 28/133 -- train_loss: 0.9824 
2021-06-20 23:47:08,005 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 29/133 -- train_loss: 0.9758 
2021-06-20 23:47:08,220 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 30/133 -- train_loss: 0.9695 
2021-06-20 23:47:08,410 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 31/133 -- train_loss: 0.9700 
2021-06-20 23:47:08,600 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 32/133 -- train_loss: 0.9924 
2021-06-20 23:47:08,848 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 33/133 -- train_loss: 0.9754 
2021-06-20 23:47:09,037 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 34/133 -- train_loss: 0.9798 
2021-06-20 23:47:09,227 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 35/133 -- train_loss: 0.9701 
2021-06-20 23:47:09,415 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 36/133 -- train_loss: 0.9739 
2021-06-20 23:47:09,604 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 37/133 -- train_loss: 0.9674 
2021-06-20 23:47:09,941 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 38/133 -- train_loss: 0.9594 
2021-06-20 23:47:10,131 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 39/133 -- train_loss: 0.9535 
2021-06-20 23:47:10,320 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 40/133 -- train_loss: 0.9826 
2021-06-20 23:47:10,509 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 41/133 -- train_loss: 0.9722 
2021-06-20 23:47:10,699 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 42/133 -- train_loss: 0.9539 
2021-06-20 23:47:10,888 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 43/133 -- train_loss: 0.9462 
2021-06-20 23:47:11,078 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 44/133 -- train_loss: 0.9190 
2021-06-20 23:47:11,267 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 45/133 -- train_loss: 0.9481 
2021-06-20 23:47:11,483 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 46/133 -- train_loss: 0.9581 
2021-06-20 23:47:11,673 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 47/133 -- train_loss: 0.9331 
2021-06-20 23:47:11,862 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 48/133 -- train_loss: 0.9552 
2021-06-20 23:47:12,111 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 49/133 -- train_loss: 0.9654 
2021-06-20 23:47:12,300 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 50/133 -- train_loss: 0.9291 
2021-06-20 23:47:12,491 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 51/133 -- train_loss: 0.9999 
2021-06-20 23:47:12,680 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 52/133 -- train_loss: 0.9700 
2021-06-20 23:47:12,869 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 53/133 -- train_loss: 0.9540 
2021-06-20 23:47:13,205 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 54/133 -- train_loss: 0.9637 
2021-06-20 23:47:13,395 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 55/133 -- train_loss: 0.9156 
2021-06-20 23:47:13,588 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 56/133 -- train_loss: 0.9251 
2021-06-20 23:47:13,778 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 57/133 -- train_loss: 0.8865 
2021-06-20 23:47:13,968 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 58/133 -- train_loss: 0.8766 
2021-06-20 23:47:14,157 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 59/133 -- train_loss: 0.9588 
2021-06-20 23:47:14,347 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 60/133 -- train_loss: 0.8601 
2021-06-20 23:47:14,537 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 61/133 -- train_loss: 0.9463 
2021-06-20 23:47:14,752 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 62/133 -- train_loss: 0.8923 
2021-06-20 23:47:14,943 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 63/133 -- train_loss: 0.9488 
2021-06-20 23:47:15,132 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 64/133 -- train_loss: 0.9399 
2021-06-20 23:47:15,382 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 65/133 -- train_loss: 0.9121 
2021-06-20 23:47:15,573 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 66/133 -- train_loss: 0.9106 
2021-06-20 23:47:15,764 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 67/133 -- train_loss: 0.8787 
2021-06-20 23:47:15,955 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 68/133 -- train_loss: 0.8295 
2021-06-20 23:47:16,145 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 69/133 -- train_loss: 0.8938 
2021-06-20 23:47:16,483 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 70/133 -- train_loss: 0.8682 
2021-06-20 23:47:16,673 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 71/133 -- train_loss: 0.9311 
2021-06-20 23:47:16,862 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 72/133 -- train_loss: 0.9382 
2021-06-20 23:47:17,051 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 73/133 -- train_loss: 0.7982 
2021-06-20 23:47:17,242 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 74/133 -- train_loss: 0.8667 
2021-06-20 23:47:17,430 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 75/133 -- train_loss: 0.9382 
2021-06-20 23:47:17,621 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 76/133 -- train_loss: 0.8640 
2021-06-20 23:47:17,811 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 77/133 -- train_loss: 0.9258 
2021-06-20 23:47:18,026 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 78/133 -- train_loss: 0.8814 
2021-06-20 23:47:18,216 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 79/133 -- train_loss: 0.8917 
2021-06-20 23:47:18,405 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 80/133 -- train_loss: 0.9138 
2021-06-20 23:47:18,657 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 81/133 -- train_loss: 0.8583 
2021-06-20 23:47:18,846 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 82/133 -- train_loss: 0.7946 
2021-06-20 23:47:19,036 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 83/133 -- train_loss: 0.9572 
2021-06-20 23:47:19,225 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 84/133 -- train_loss: 0.8400 
2021-06-20 23:47:19,415 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 85/133 -- train_loss: 0.8598 
2021-06-20 23:47:19,749 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 86/133 -- train_loss: 0.8361 
2021-06-20 23:47:19,938 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 87/133 -- train_loss: 0.8913 
2021-06-20 23:47:20,128 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 88/133 -- train_loss: 0.8593 
2021-06-20 23:47:20,317 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 89/133 -- train_loss: 0.8356 
2021-06-20 23:47:20,507 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 90/133 -- train_loss: 0.8472 
2021-06-20 23:47:20,698 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 91/133 -- train_loss: 0.7454 
2021-06-20 23:47:20,889 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 92/133 -- train_loss: 0.8699 
2021-06-20 23:47:21,078 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 93/133 -- train_loss: 0.7920 
2021-06-20 23:47:21,294 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 94/133 -- train_loss: 0.8331 
2021-06-20 23:47:21,490 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 95/133 -- train_loss: 0.8341 
2021-06-20 23:47:21,682 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 96/133 -- train_loss: 0.8154 
2021-06-20 23:47:21,937 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 97/133 -- train_loss: 0.7854 
2021-06-20 23:47:22,128 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 98/133 -- train_loss: 0.7463 
2021-06-20 23:47:22,319 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 99/133 -- train_loss: 0.8374 
2021-06-20 23:47:22,510 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 100/133 -- train_loss: 0.7787 
2021-06-20 23:47:22,701 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 101/133 -- train_loss: 0.8126 
2021-06-20 23:47:23,034 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 102/133 -- train_loss: 0.7186 
2021-06-20 23:47:23,225 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 103/133 -- train_loss: 0.8285 
2021-06-20 23:47:23,416 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 104/133 -- train_loss: 0.8196 
2021-06-20 23:47:23,606 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 105/133 -- train_loss: 0.8312 
2021-06-20 23:47:23,796 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 106/133 -- train_loss: 0.8645 
2021-06-20 23:47:23,986 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 107/133 -- train_loss: 0.8532 
2021-06-20 23:47:24,176 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 108/133 -- train_loss: 0.7329 
2021-06-20 23:47:24,367 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 109/133 -- train_loss: 0.8053 
2021-06-20 23:47:24,584 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 110/133 -- train_loss: 0.7682 
2021-06-20 23:47:24,774 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 111/133 -- train_loss: 0.8501 
2021-06-20 23:47:24,963 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 112/133 -- train_loss: 0.7342 
2021-06-20 23:47:25,212 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 113/133 -- train_loss: 0.8207 
2021-06-20 23:47:25,402 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 114/133 -- train_loss: 0.7834 
2021-06-20 23:47:25,591 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 115/133 -- train_loss: 0.8044 
2021-06-20 23:47:25,780 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 116/133 -- train_loss: 0.7388 
2021-06-20 23:47:25,972 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 117/133 -- train_loss: 0.8734 
2021-06-20 23:47:26,310 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 118/133 -- train_loss: 0.7999 
2021-06-20 23:47:26,499 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 119/133 -- train_loss: 0.7686 
2021-06-20 23:47:26,687 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 120/133 -- train_loss: 0.7821 
2021-06-20 23:47:26,875 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 121/133 -- train_loss: 0.8390 
2021-06-20 23:47:27,067 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 122/133 -- train_loss: 0.6208 
2021-06-20 23:47:27,254 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 123/133 -- train_loss: 0.7853 
2021-06-20 23:47:27,442 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 124/133 -- train_loss: 0.7650 
2021-06-20 23:47:27,629 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 125/133 -- train_loss: 0.7147 
2021-06-20 23:47:27,844 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 126/133 -- train_loss: 0.7832 
2021-06-20 23:47:28,031 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 127/133 -- train_loss: 0.7886 
2021-06-20 23:47:28,218 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 128/133 -- train_loss: 0.7514 
2021-06-20 23:47:28,469 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 129/133 -- train_loss: 0.8141 
2021-06-20 23:47:28,657 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 130/133 -- train_loss: 0.8206 
2021-06-20 23:47:28,845 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 131/133 -- train_loss: 0.8017 
2021-06-20 23:47:29,032 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 132/133 -- train_loss: 0.7441 
2021-06-20 23:47:29,153 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 133/133 -- train_loss: 0.8183 
2021-06-20 23:47:29,154 - ignite.engine.engine.SupervisedEvaluator - INFO - Engine run resuming from iteration 0, epoch 0 until 1 epochs
2021-06-20 23:53:41,131 - ignite.engine.engine.SupervisedEvaluator - INFO - Got new best metric of Monai's validation mean dice loss: 0.13218934834003448
2021-06-20 23:53:41,132 - ignite.engine.engine.SupervisedEvaluator - INFO - Epoch[1] Metrics -- Monai's validation mean dice loss: 0.1322 My own validation mean dice loss: 0.8585 
2021-06-20 23:53:41,132 - ignite.engine.engine.SupervisedEvaluator - INFO - Key metric: Monai's validation mean dice loss best value: 0.13218934834003448 at epoch: 1
2021-06-20 23:53:41,292 - ignite.engine.engine.SupervisedEvaluator - INFO - Epoch[1] Complete. Time taken: 00:06:12
2021-06-20 23:53:41,293 - ignite.engine.engine.SupervisedEvaluator - INFO - Engine run complete. Time taken: 00:06:12
2021-06-20 23:53:41,413 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch[1] Complete. Time taken: 00:06:40
2021-06-20 23:53:41,621 - ignite.engine.engine.SupervisedTrainer - INFO - Train completed, saved final checkpoint: checkpoint_final_iteration=133.pt
2021-06-20 23:53:41,621 - ignite.engine.engine.SupervisedTrainer - INFO - Engine run complete. Time taken: 00:06:41
Send model to server.
2021-06-20 23:53:41,783 - FederatedClient - INFO - Starting to push model.
2021-06-20 23:53:41,786 - Communicator - INFO - Send lesion_activity at round 0
2021-06-20 23:53:41,809 - Communicator - INFO - Received comments: lesion_activity Received round 0 from org1-b (713 Bytes, 907 seconds). SubmitUpdate time: 0.022276639938354492 seconds
2021-06-20 23:53:41,811 - ignite.engine.engine.SupervisedTrainer - INFO - Terminate signaled. Engine will stop after current iteration is finished.
2021-06-20 23:53:41,811 - ignite.engine.engine.SupervisedEvaluator - INFO - Terminate signaled. Engine will stop after current iteration is finished.
2021-06-20 23:53:45,569 - ProcessExecutor - INFO - process finished with return code 0
cross_validation child process ID: 3267
starting the client .....
token is: 70a43d1e-9233-4ca6-954c-fb168a8b2dad run_number is: 711 uid: org1-b
2021-06-20 23:53:46,438 - matplotlib - WARNING - Matplotlib created a temporary config/cache directory at /tmp/matplotlib-t1q8wzcc because the default path (/home/siddharth/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.
2021-06-20 23:53:46,475 - matplotlib.font_manager - INFO - Generating new fontManager, this may take some time...
2021-06-20 23:53:49,380 - CrossSiteValManager - INFO - Cross validation timeout is set to 10 minutes.
2021-06-20 23:53:56,431 - Communicator - INFO - Server reply to SubmitBestLocalModel:  Received best model from org1-b.. SubmitBestLocalModel time: 6.918471097946167 seconds
2021-06-20 23:53:56,452 - FederatedClient - INFO - Getting other models from server for cross validation.
2021-06-20 23:53:56,470 - Communicator - INFO - Received 0 models for validation. GetValidationModels time: 0.01690506935119629 seconds
2021-06-20 23:53:56,473 - FederatedClient - INFO - Server has no models available currently. Waiting 60 secs before asking again.
2021-06-20 23:54:56,539 - FederatedClient - INFO - Getting other models from server for cross validation.
2021-06-20 23:54:56,559 - Communicator - INFO - Received 0 models for validation. GetValidationModels time: 0.018537044525146484 seconds
2021-06-20 23:54:56,562 - FederatedClient - INFO - Server has no models available currently. Waiting 60 secs before asking again.
2021-06-20 23:55:56,628 - FederatedClient - INFO - Getting other models from server for cross validation.
2021-06-20 23:55:56,645 - Communicator - INFO - Received 0 models for validation. GetValidationModels time: 0.015695810317993164 seconds
2021-06-20 23:55:56,647 - FederatedClient - INFO - Server has no models available currently. Waiting 60 secs before asking again.
2021-06-20 23:56:56,714 - FederatedClient - INFO - Getting other models from server for cross validation.
2021-06-20 23:56:56,735 - Communicator - INFO - Received 0 models for validation. GetValidationModels time: 0.019411802291870117 seconds
2021-06-20 23:56:56,737 - FederatedClient - INFO - Server has no models available currently. Waiting 60 secs before asking again.
2021-06-20 23:57:56,804 - FederatedClient - INFO - Getting other models from server for cross validation.
2021-06-20 23:57:56,824 - Communicator - INFO - Received 0 models for validation. GetValidationModels time: 0.017854690551757812 seconds
2021-06-20 23:57:56,826 - FederatedClient - INFO - Server has no models available currently. Waiting 60 secs before asking again.
2021-06-20 23:58:56,893 - FederatedClient - INFO - Getting other models from server for cross validation.
2021-06-20 23:58:56,912 - Communicator - INFO - Received 0 models for validation. GetValidationModels time: 0.018385648727416992 seconds
2021-06-20 23:58:56,915 - FederatedClient - INFO - Server has no models available currently. Waiting 60 secs before asking again.
2021-06-20 23:59:56,981 - FederatedClient - INFO - Getting other models from server for cross validation.
2021-06-21 00:00:04,204 - Communicator - INFO - Received 1 models for validation. GetValidationModels time: 7.221736907958984 seconds
2021-06-21 00:00:04,874 - torch.distributed.nn.jit.instantiator - INFO - Created a temporary directory at /tmp/tmpoota0tly
2021-06-21 00:00:04,875 - torch.distributed.nn.jit.instantiator - INFO - Writing /tmp/tmpoota0tly/_remote_module_non_sriptable.py
========== Validate Config Result ===========
Use GPU:  True
Multi GPU:  False
Automatic Mixed Precision:  Enabled
Determinism Evaluation:  Disabled
cuDNN BenchMark:  False
CUDA Matmul Allow TF32:  True
cuDNN Allow TF32:  True
Model:  <class 'lesion-activity-clara-fl.NewLesionsUNet.model.NewLesionsUNet'>
Dataset:  <class 'monai.data.dataset.Dataset'>
DataLoader:  <class 'monai.data.dataloader.DataLoader'>
Validate Transform #1: <class 'monai.transforms.io.dictionary.LoadImaged'>
Validate Transform #2: <class 'lesion-activity-clara-fl.NewLesionsUNet.pre_transform.GetMask'>
Validate Transform #3: <class 'monai.transforms.utility.dictionary.ToTensord'>
Validate Handler #1: <class 'monai.handlers.stats_handler.StatsHandler'>
Validate Handler #2: <class 'monai.handlers.checkpoint_loader.CheckpointLoader'>
Validate Handler #3: <class 'monai.handlers.segmentation_saver.SegmentationSaver'>
Validate Post Transforms #1: <class 'monai.transforms.post.dictionary.Activationsd'>
Validate Post Transforms #2: <class 'lesion-activity-clara-fl.NewLesionsUNet.pre_transform.RemoveSmallRegions'>
Validate Inferer:  <class 'monai.inferers.inferer.SlidingWindowInferer'>
Validate Key Metric:  <class 'monai.handlers.mean_dice.MeanDice'>
Validate Additional Metric #My own validation mean dice loss: <class 'lesion-activity-clara-fl.NewLesionsUNet.model.MyValLoss'>
========== End of Validate Config Result ===========
2021-06-21 00:00:08,207 - ignite.engine.engine.SupervisedEvaluator - INFO - Engine run resuming from iteration 0, epoch 0 until 1 epochs
2021-06-21 00:00:08,250 - ignite.engine.engine.SupervisedEvaluator - INFO - Restored all variables from /tmp/tmp5viuh_i3/mmar/models/local_best_model.PT_MODEL
siddharth@snac-dell:/workspace/startup$ file written: /tmp/tmp5viuh_i3/mmar/eval/brain_mask/brain_mask_seg.nii.gz.
2021-06-21 00:04:00,483 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files.
file written: /tmp/tmp5viuh_i3/mmar/eval/brain_mask/brain_mask_seg.nii.gz.
2021-06-21 00:05:09,891 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files.
file written: /tmp/tmp5viuh_i3/mmar/eval/brain_mask/brain_mask_seg.nii.gz.
2021-06-21 00:09:13,670 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files.
file written: /tmp/tmp5viuh_i3/mmar/eval/brain_mask/brain_mask_seg.nii.gz.
2021-06-21 00:12:18,834 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files.
file written: /tmp/tmp5viuh_i3/mmar/eval/brain_mask/brain_mask_seg.nii.gz.
2021-06-21 00:15:12,517 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files.
file written: /tmp/tmp5viuh_i3/mmar/eval/brain_mask/brain_mask_seg.nii.gz.
2021-06-21 00:15:16,468 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files.
file written: /tmp/tmp5viuh_i3/mmar/eval/brain_mask/brain_mask_seg.nii.gz.
2021-06-21 00:16:24,759 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files.
file written: /tmp/tmp5viuh_i3/mmar/eval/brain_mask/brain_mask_seg.nii.gz.
2021-06-21 00:20:01,139 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files.
file written: /tmp/tmp5viuh_i3/mmar/eval/brain_mask/brain_mask_seg.nii.gz.
2021-06-21 00:24:12,635 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files.
file written: /tmp/tmp5viuh_i3/mmar/eval/brain_mask/brain_mask_seg.nii.gz.
2021-06-21 00:30:23,799 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files.
file written: /tmp/tmp5viuh_i3/mmar/eval/brain_mask/brain_mask_seg.nii.gz.
2021-06-21 00:33:25,295 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files.
file written: /tmp/tmp5viuh_i3/mmar/eval/brain_mask/brain_mask_seg.nii.gz.
2021-06-21 00:33:50,718 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files.
file written: /tmp/tmp5viuh_i3/mmar/eval/brain_mask/brain_mask_seg.nii.gz.
2021-06-21 00:33:53,256 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files.
file written: /tmp/tmp5viuh_i3/mmar/eval/brain_mask/brain_mask_seg.nii.gz.
2021-06-21 00:36:39,563 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files.
file written: /tmp/tmp5viuh_i3/mmar/eval/brain_mask/brain_mask_seg.nii.gz.
2021-06-21 00:37:18,336 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files.
file written: /tmp/tmp5viuh_i3/mmar/eval/brain_mask/brain_mask_seg.nii.gz.
2021-06-21 00:37:27,715 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files.
file written: /tmp/tmp5viuh_i3/mmar/eval/brain_mask/brain_mask_seg.nii.gz.
2021-06-21 00:37:38,597 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files.
file written: /tmp/tmp5viuh_i3/mmar/eval/brain_mask/brain_mask_seg.nii.gz.
2021-06-21 00:41:58,063 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files.
file written: /tmp/tmp5viuh_i3/mmar/eval/brain_mask/brain_mask_seg.nii.gz.
2021-06-21 00:44:10,747 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files.
file written: /tmp/tmp5viuh_i3/mmar/eval/brain_mask/brain_mask_seg.nii.gz.
2021-06-21 00:44:14,708 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files.
2021-06-21 00:44:14,711 - ignite.engine.engine.SupervisedEvaluator - INFO - Got new best metric of Monai's validation mean dice loss: 0.12495676428079605
2021-06-21 00:44:14,712 - ignite.engine.engine.SupervisedEvaluator - INFO - Epoch[1] Metrics -- Monai's validation mean dice loss: 0.1250 My own validation mean dice loss: 0.8689 
2021-06-21 00:44:14,712 - ignite.engine.engine.SupervisedEvaluator - INFO - Key metric: Monai's validation mean dice loss best value: 0.12495676428079605 at epoch: 1
2021-06-21 00:44:14,712 - ignite.engine.engine.SupervisedEvaluator - INFO - Epoch[1] Complete. Time taken: 00:44:06
2021-06-21 00:44:14,712 - ignite.engine.engine.SupervisedEvaluator - INFO - Engine run complete. Time taken: 00:44:07
2021-06-21 00:44:14,771 - medl.apps.mmar_conf - INFO - Total Evaluation Time 2646.5634293556213
2021-06-21 00:44:14,779 - ClientValidator - INFO - Do validation ...
2021-06-21 00:44:14,787 - FederatedClient - INFO - Submitting cross validation results to server.
2021-06-21 00:44:14,808 - Communicator - INFO - Received comments:  Received Cross Validation results from org1-b.. SubmitCrossSiteValidationResults time: 0.018944978713989258 seconds
2021-06-21 00:44:14,809 - FederatedClient - INFO - More models available with server.
2021-06-21 00:44:14,811 - FederatedClient - INFO - Getting other models from server for cross validation.
2021-06-21 00:44:17,658 - Communicator - INFO - Received 1 models for validation. GetValidationModels time: 2.8453869819641113 seconds
========== Validate Config Result ===========
Use GPU:  True
Multi GPU:  False
Automatic Mixed Precision:  Enabled
Determinism Evaluation:  Disabled
cuDNN BenchMark:  False
CUDA Matmul Allow TF32:  True
cuDNN Allow TF32:  True
Model:  <class 'lesion-activity-clara-fl.NewLesionsUNet.model.NewLesionsUNet'>
Dataset:  <class 'monai.data.dataset.Dataset'>
DataLoader:  <class 'monai.data.dataloader.DataLoader'>
Validate Transform #1: <class 'monai.transforms.io.dictionary.LoadImaged'>
Validate Transform #2: <class 'lesion-activity-clara-fl.NewLesionsUNet.pre_transform.GetMask'>
Validate Transform #3: <class 'monai.transforms.utility.dictionary.ToTensord'>
Validate Handler #1: <class 'monai.handlers.stats_handler.StatsHandler'>
Validate Handler #2: <class 'monai.handlers.checkpoint_loader.CheckpointLoader'>
Validate Handler #3: <class 'monai.handlers.segmentation_saver.SegmentationSaver'>
Validate Post Transforms #1: <class 'monai.transforms.post.dictionary.Activationsd'>
Validate Post Transforms #2: <class 'lesion-activity-clara-fl.NewLesionsUNet.pre_transform.RemoveSmallRegions'>
Validate Inferer:  <class 'monai.inferers.inferer.SlidingWindowInferer'>
Validate Key Metric:  <class 'monai.handlers.mean_dice.MeanDice'>
Validate Additional Metric #My own validation mean dice loss: <class 'lesion-activity-clara-fl.NewLesionsUNet.model.MyValLoss'>
========== End of Validate Config Result ===========
2021-06-21 00:44:18,031 - ignite.engine.engine.SupervisedEvaluator - INFO - Engine run resuming from iteration 0, epoch 0 until 1 epochs
2021-06-21 00:44:18,048 - ignite.engine.engine.SupervisedEvaluator - ERROR - Engine run is terminating due to exception: Error(s) in loading state_dict for NewLesionsUNet:
	Missing key(s) in state_dict: "segmenter.0.down.0.conv.weight", "segmenter.0.down.0.conv.bias", "segmenter.0.down.0.res.weight", "segmenter.0.down.0.res.bias", "segmenter.0.down.0.end_seq.1.weight", "segmenter.0.down.0.end_seq.1.bias", "segmenter.0.down.1.conv.weight", "segmenter.0.down.1.conv.bias", "segmenter.0.down.1.res.weight", "segmenter.0.down.1.res.bias", "segmenter.0.down.1.end_seq.1.weight", "segmenter.0.down.1.end_seq.1.bias", "segmenter.0.down.2.conv.weight", "segmenter.0.down.2.conv.bias", "segmenter.0.down.2.res.weight", "segmenter.0.down.2.res.bias", "segmenter.0.down.2.end_seq.1.weight", "segmenter.0.down.2.end_seq.1.bias", "segmenter.0.down.3.conv.weight", "segmenter.0.down.3.conv.bias", "segmenter.0.down.3.res.weight", "segmenter.0.down.3.res.bias", "segmenter.0.down.3.end_seq.1.weight", "segmenter.0.down.3.end_seq.1.bias", "segmenter.0.u.conv.weight", "segmenter.0.u.conv.bias", "segmenter.0.u.res.weight", "segmenter.0.u.res.bias", "segmenter.0.u.end_seq.1.weight", "segmenter.0.u.end_seq.1.bias", "segmenter.0.up.0.conv.weight", "segmenter.0.up.0.conv.bias", "segmenter.0.up.0.res.weight", "segmenter.0.up.0.res.bias", "segmenter.0.up.0.end_seq.1.weight", "segmenter.0.up.0.end_seq.1.bias", "segmenter.0.up.1.conv.weight", "segmenter.0.up.1.conv.bias", "segmenter.0.up.1.res.weight", "segmenter.0.up.1.res.bias", "segmenter.0.up.1.end_seq.1.weight", "segmenter.0.up.1.end_seq.1.bias", "segmenter.0.up.2.conv.weight", "segmenter.0.up.2.conv.bias", "segmenter.0.up.2.res.weight", "segmenter.0.up.2.res.bias", "segmenter.0.up.2.end_seq.1.weight", "segmenter.0.up.2.end_seq.1.bias", "segmenter.0.up.3.conv.weight", "segmenter.0.up.3.conv.bias", "segmenter.0.up.3.res.weight", "segmenter.0.up.3.res.bias", "segmenter.0.up.3.end_seq.1.weight", "segmenter.0.up.3.end_seq.1.bias", "segmenter.1.weight", "segmenter.1.bias". 
	Unexpected key(s) in state_dict: "model.0.conv.unit0.conv.weight", "model.0.conv.unit0.conv.bias", "model.0.conv.unit0.adn.N.weight", "model.0.conv.unit0.adn.N.bias", "model.0.conv.unit0.adn.N.running_mean", "model.0.conv.unit0.adn.N.running_var", "model.0.conv.unit0.adn.N.num_batches_tracked", "model.0.conv.unit0.adn.A.weight", "model.0.conv.unit1.conv.weight", "model.0.conv.unit1.conv.bias", "model.0.conv.unit1.adn.N.weight", "model.0.conv.unit1.adn.N.bias", "model.0.conv.unit1.adn.N.running_mean", "model.0.conv.unit1.adn.N.running_var", "model.0.conv.unit1.adn.N.num_batches_tracked", "model.0.conv.unit1.adn.A.weight", "model.0.residual.weight", "model.0.residual.bias", "model.1.submodule.0.conv.unit0.conv.weight", "model.1.submodule.0.conv.unit0.conv.bias", "model.1.submodule.0.conv.unit0.adn.N.weight", "model.1.submodule.0.conv.unit0.adn.N.bias", "model.1.submodule.0.conv.unit0.adn.N.running_mean", "model.1.submodule.0.conv.unit0.adn.N.running_var", "model.1.submodule.0.conv.unit0.adn.N.num_batches_tracked", "model.1.submodule.0.conv.unit0.adn.A.weight", "model.1.submodule.0.conv.unit1.conv.weight", "model.1.submodule.0.conv.unit1.conv.bias", "model.1.submodule.0.conv.unit1.adn.N.weight", "model.1.submodule.0.conv.unit1.adn.N.bias", "model.1.submodule.0.conv.unit1.adn.N.running_mean", "model.1.submodule.0.conv.unit1.adn.N.running_var", "model.1.submodule.0.conv.unit1.adn.N.num_batches_tracked", "model.1.submodule.0.conv.unit1.adn.A.weight", "model.1.submodule.0.residual.weight", "model.1.submodule.0.residual.bias", "model.1.submodule.1.submodule.0.conv.unit0.conv.weight", "model.1.submodule.1.submodule.0.conv.unit0.conv.bias", "model.1.submodule.1.submodule.0.conv.unit0.adn.N.weight", "model.1.submodule.1.submodule.0.conv.unit0.adn.N.bias", "model.1.submodule.1.submodule.0.conv.unit0.adn.N.running_mean", "model.1.submodule.1.submodule.0.conv.unit0.adn.N.running_var", "model.1.submodule.1.submodule.0.conv.unit0.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.0.conv.unit0.adn.A.weight", "model.1.submodule.1.submodule.0.conv.unit1.conv.weight", "model.1.submodule.1.submodule.0.conv.unit1.conv.bias", "model.1.submodule.1.submodule.0.conv.unit1.adn.N.weight", "model.1.submodule.1.submodule.0.conv.unit1.adn.N.bias", "model.1.submodule.1.submodule.0.conv.unit1.adn.N.running_mean", "model.1.submodule.1.submodule.0.conv.unit1.adn.N.running_var", "model.1.submodule.1.submodule.0.conv.unit1.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.0.conv.unit1.adn.A.weight", "model.1.submodule.1.submodule.0.residual.weight", "model.1.submodule.1.submodule.0.residual.bias", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.conv.weight", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.conv.bias", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.adn.N.weight", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.adn.N.bias", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.adn.N.running_mean", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.adn.N.running_var", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.adn.A.weight", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.conv.weight", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.conv.bias", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.adn.N.weight", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.adn.N.bias", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.adn.N.running_mean", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.adn.N.running_var", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.adn.A.weight", "model.1.submodule.1.submodule.1.submodule.0.residual.weight", "model.1.submodule.1.submodule.1.submodule.0.residual.bias", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.conv.weight", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.conv.bias", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.adn.N.weight", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.adn.N.bias", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.adn.N.running_mean", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.adn.N.running_var", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.adn.A.weight", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.conv.weight", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.conv.bias", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.adn.N.weight", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.adn.N.bias", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.adn.N.running_mean", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.adn.N.running_var", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.adn.A.weight", "model.1.submodule.1.submodule.1.submodule.1.submodule.residual.weight", "model.1.submodule.1.submodule.1.submodule.1.submodule.residual.bias", "model.1.submodule.1.submodule.1.submodule.2.0.conv.weight", "model.1.submodule.1.submodule.1.submodule.2.0.conv.bias", "model.1.submodule.1.submodule.1.submodule.2.0.adn.N.weight", "model.1.submodule.1.submodule.1.submodule.2.0.adn.N.bias", "model.1.submodule.1.submodule.1.submodule.2.0.adn.N.running_mean", "model.1.submodule.1.submodule.1.submodule.2.0.adn.N.running_var", "model.1.submodule.1.submodule.1.submodule.2.0.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.1.submodule.2.0.adn.A.weight", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.conv.weight", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.conv.bias", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.adn.N.weight", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.adn.N.bias", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.adn.N.running_mean", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.adn.N.running_var", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.adn.A.weight", "model.1.submodule.1.submodule.2.0.conv.weight", "model.1.submodule.1.submodule.2.0.conv.bias", "model.1.submodule.1.submodule.2.0.adn.N.weight", "model.1.submodule.1.submodule.2.0.adn.N.bias", "model.1.submodule.1.submodule.2.0.adn.N.running_mean", "model.1.submodule.1.submodule.2.0.adn.N.running_var", "model.1.submodule.1.submodule.2.0.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.2.0.adn.A.weight", "model.1.submodule.1.submodule.2.1.conv.unit0.conv.weight", "model.1.submodule.1.submodule.2.1.conv.unit0.conv.bias", "model.1.submodule.1.submodule.2.1.conv.unit0.adn.N.weight", "model.1.submodule.1.submodule.2.1.conv.unit0.adn.N.bias", "model.1.submodule.1.submodule.2.1.conv.unit0.adn.N.running_mean", "model.1.submodule.1.submodule.2.1.conv.unit0.adn.N.running_var", "model.1.submodule.1.submodule.2.1.conv.unit0.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.2.1.conv.unit0.adn.A.weight", "model.1.submodule.2.0.conv.weight", "model.1.submodule.2.0.conv.bias", "model.1.submodule.2.0.adn.N.weight", "model.1.submodule.2.0.adn.N.bias", "model.1.submodule.2.0.adn.N.running_mean", "model.1.submodule.2.0.adn.N.running_var", "model.1.submodule.2.0.adn.N.num_batches_tracked", "model.1.submodule.2.0.adn.A.weight", "model.1.submodule.2.1.conv.unit0.conv.weight", "model.1.submodule.2.1.conv.unit0.conv.bias", "model.1.submodule.2.1.conv.unit0.adn.N.weight", "model.1.submodule.2.1.conv.unit0.adn.N.bias", "model.1.submodule.2.1.conv.unit0.adn.N.running_mean", "model.1.submodule.2.1.conv.unit0.adn.N.running_var", "model.1.submodule.2.1.conv.unit0.adn.N.num_batches_tracked", "model.1.submodule.2.1.conv.unit0.adn.A.weight", "model.2.0.conv.weight", "model.2.0.conv.bias", "model.2.0.adn.N.weight", "model.2.0.adn.N.bias", "model.2.0.adn.N.running_mean", "model.2.0.adn.N.running_var", "model.2.0.adn.N.num_batches_tracked", "model.2.0.adn.A.weight", "model.2.1.conv.unit0.conv.weight", "model.2.1.conv.unit0.conv.bias". 
2021-06-21 00:44:18,048 - ignite.engine.engine.SupervisedEvaluator - ERROR - Exception: Error(s) in loading state_dict for NewLesionsUNet:
	Missing key(s) in state_dict: "segmenter.0.down.0.conv.weight", "segmenter.0.down.0.conv.bias", "segmenter.0.down.0.res.weight", "segmenter.0.down.0.res.bias", "segmenter.0.down.0.end_seq.1.weight", "segmenter.0.down.0.end_seq.1.bias", "segmenter.0.down.1.conv.weight", "segmenter.0.down.1.conv.bias", "segmenter.0.down.1.res.weight", "segmenter.0.down.1.res.bias", "segmenter.0.down.1.end_seq.1.weight", "segmenter.0.down.1.end_seq.1.bias", "segmenter.0.down.2.conv.weight", "segmenter.0.down.2.conv.bias", "segmenter.0.down.2.res.weight", "segmenter.0.down.2.res.bias", "segmenter.0.down.2.end_seq.1.weight", "segmenter.0.down.2.end_seq.1.bias", "segmenter.0.down.3.conv.weight", "segmenter.0.down.3.conv.bias", "segmenter.0.down.3.res.weight", "segmenter.0.down.3.res.bias", "segmenter.0.down.3.end_seq.1.weight", "segmenter.0.down.3.end_seq.1.bias", "segmenter.0.u.conv.weight", "segmenter.0.u.conv.bias", "segmenter.0.u.res.weight", "segmenter.0.u.res.bias", "segmenter.0.u.end_seq.1.weight", "segmenter.0.u.end_seq.1.bias", "segmenter.0.up.0.conv.weight", "segmenter.0.up.0.conv.bias", "segmenter.0.up.0.res.weight", "segmenter.0.up.0.res.bias", "segmenter.0.up.0.end_seq.1.weight", "segmenter.0.up.0.end_seq.1.bias", "segmenter.0.up.1.conv.weight", "segmenter.0.up.1.conv.bias", "segmenter.0.up.1.res.weight", "segmenter.0.up.1.res.bias", "segmenter.0.up.1.end_seq.1.weight", "segmenter.0.up.1.end_seq.1.bias", "segmenter.0.up.2.conv.weight", "segmenter.0.up.2.conv.bias", "segmenter.0.up.2.res.weight", "segmenter.0.up.2.res.bias", "segmenter.0.up.2.end_seq.1.weight", "segmenter.0.up.2.end_seq.1.bias", "segmenter.0.up.3.conv.weight", "segmenter.0.up.3.conv.bias", "segmenter.0.up.3.res.weight", "segmenter.0.up.3.res.bias", "segmenter.0.up.3.end_seq.1.weight", "segmenter.0.up.3.end_seq.1.bias", "segmenter.1.weight", "segmenter.1.bias". 
	Unexpected key(s) in state_dict: "model.0.conv.unit0.conv.weight", "model.0.conv.unit0.conv.bias", "model.0.conv.unit0.adn.N.weight", "model.0.conv.unit0.adn.N.bias", "model.0.conv.unit0.adn.N.running_mean", "model.0.conv.unit0.adn.N.running_var", "model.0.conv.unit0.adn.N.num_batches_tracked", "model.0.conv.unit0.adn.A.weight", "model.0.conv.unit1.conv.weight", "model.0.conv.unit1.conv.bias", "model.0.conv.unit1.adn.N.weight", "model.0.conv.unit1.adn.N.bias", "model.0.conv.unit1.adn.N.running_mean", "model.0.conv.unit1.adn.N.running_var", "model.0.conv.unit1.adn.N.num_batches_tracked", "model.0.conv.unit1.adn.A.weight", "model.0.residual.weight", "model.0.residual.bias", "model.1.submodule.0.conv.unit0.conv.weight", "model.1.submodule.0.conv.unit0.conv.bias", "model.1.submodule.0.conv.unit0.adn.N.weight", "model.1.submodule.0.conv.unit0.adn.N.bias", "model.1.submodule.0.conv.unit0.adn.N.running_mean", "model.1.submodule.0.conv.unit0.adn.N.running_var", "model.1.submodule.0.conv.unit0.adn.N.num_batches_tracked", "model.1.submodule.0.conv.unit0.adn.A.weight", "model.1.submodule.0.conv.unit1.conv.weight", "model.1.submodule.0.conv.unit1.conv.bias", "model.1.submodule.0.conv.unit1.adn.N.weight", "model.1.submodule.0.conv.unit1.adn.N.bias", "model.1.submodule.0.conv.unit1.adn.N.running_mean", "model.1.submodule.0.conv.unit1.adn.N.running_var", "model.1.submodule.0.conv.unit1.adn.N.num_batches_tracked", "model.1.submodule.0.conv.unit1.adn.A.weight", "model.1.submodule.0.residual.weight", "model.1.submodule.0.residual.bias", "model.1.submodule.1.submodule.0.conv.unit0.conv.weight", "model.1.submodule.1.submodule.0.conv.unit0.conv.bias", "model.1.submodule.1.submodule.0.conv.unit0.adn.N.weight", "model.1.submodule.1.submodule.0.conv.unit0.adn.N.bias", "model.1.submodule.1.submodule.0.conv.unit0.adn.N.running_mean", "model.1.submodule.1.submodule.0.conv.unit0.adn.N.running_var", "model.1.submodule.1.submodule.0.conv.unit0.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.0.conv.unit0.adn.A.weight", "model.1.submodule.1.submodule.0.conv.unit1.conv.weight", "model.1.submodule.1.submodule.0.conv.unit1.conv.bias", "model.1.submodule.1.submodule.0.conv.unit1.adn.N.weight", "model.1.submodule.1.submodule.0.conv.unit1.adn.N.bias", "model.1.submodule.1.submodule.0.conv.unit1.adn.N.running_mean", "model.1.submodule.1.submodule.0.conv.unit1.adn.N.running_var", "model.1.submodule.1.submodule.0.conv.unit1.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.0.conv.unit1.adn.A.weight", "model.1.submodule.1.submodule.0.residual.weight", "model.1.submodule.1.submodule.0.residual.bias", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.conv.weight", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.conv.bias", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.adn.N.weight", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.adn.N.bias", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.adn.N.running_mean", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.adn.N.running_var", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.adn.A.weight", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.conv.weight", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.conv.bias", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.adn.N.weight", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.adn.N.bias", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.adn.N.running_mean", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.adn.N.running_var", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.adn.A.weight", "model.1.submodule.1.submodule.1.submodule.0.residual.weight", "model.1.submodule.1.submodule.1.submodule.0.residual.bias", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.conv.weight", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.conv.bias", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.adn.N.weight", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.adn.N.bias", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.adn.N.running_mean", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.adn.N.running_var", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.adn.A.weight", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.conv.weight", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.conv.bias", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.adn.N.weight", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.adn.N.bias", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.adn.N.running_mean", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.adn.N.running_var", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.adn.A.weight", "model.1.submodule.1.submodule.1.submodule.1.submodule.residual.weight", "model.1.submodule.1.submodule.1.submodule.1.submodule.residual.bias", "model.1.submodule.1.submodule.1.submodule.2.0.conv.weight", "model.1.submodule.1.submodule.1.submodule.2.0.conv.bias", "model.1.submodule.1.submodule.1.submodule.2.0.adn.N.weight", "model.1.submodule.1.submodule.1.submodule.2.0.adn.N.bias", "model.1.submodule.1.submodule.1.submodule.2.0.adn.N.running_mean", "model.1.submodule.1.submodule.1.submodule.2.0.adn.N.running_var", "model.1.submodule.1.submodule.1.submodule.2.0.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.1.submodule.2.0.adn.A.weight", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.conv.weight", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.conv.bias", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.adn.N.weight", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.adn.N.bias", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.adn.N.running_mean", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.adn.N.running_var", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.adn.A.weight", "model.1.submodule.1.submodule.2.0.conv.weight", "model.1.submodule.1.submodule.2.0.conv.bias", "model.1.submodule.1.submodule.2.0.adn.N.weight", "model.1.submodule.1.submodule.2.0.adn.N.bias", "model.1.submodule.1.submodule.2.0.adn.N.running_mean", "model.1.submodule.1.submodule.2.0.adn.N.running_var", "model.1.submodule.1.submodule.2.0.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.2.0.adn.A.weight", "model.1.submodule.1.submodule.2.1.conv.unit0.conv.weight", "model.1.submodule.1.submodule.2.1.conv.unit0.conv.bias", "model.1.submodule.1.submodule.2.1.conv.unit0.adn.N.weight", "model.1.submodule.1.submodule.2.1.conv.unit0.adn.N.bias", "model.1.submodule.1.submodule.2.1.conv.unit0.adn.N.running_mean", "model.1.submodule.1.submodule.2.1.conv.unit0.adn.N.running_var", "model.1.submodule.1.submodule.2.1.conv.unit0.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.2.1.conv.unit0.adn.A.weight", "model.1.submodule.2.0.conv.weight", "model.1.submodule.2.0.conv.bias", "model.1.submodule.2.0.adn.N.weight", "model.1.submodule.2.0.adn.N.bias", "model.1.submodule.2.0.adn.N.running_mean", "model.1.submodule.2.0.adn.N.running_var", "model.1.submodule.2.0.adn.N.num_batches_tracked", "model.1.submodule.2.0.adn.A.weight", "model.1.submodule.2.1.conv.unit0.conv.weight", "model.1.submodule.2.1.conv.unit0.conv.bias", "model.1.submodule.2.1.conv.unit0.adn.N.weight", "model.1.submodule.2.1.conv.unit0.adn.N.bias", "model.1.submodule.2.1.conv.unit0.adn.N.running_mean", "model.1.submodule.2.1.conv.unit0.adn.N.running_var", "model.1.submodule.2.1.conv.unit0.adn.N.num_batches_tracked", "model.1.submodule.2.1.conv.unit0.adn.A.weight", "model.2.0.conv.weight", "model.2.0.conv.bias", "model.2.0.adn.N.weight", "model.2.0.adn.N.bias", "model.2.0.adn.N.running_mean", "model.2.0.adn.N.running_var", "model.2.0.adn.N.num_batches_tracked", "model.2.0.adn.A.weight", "model.2.1.conv.unit0.conv.weight", "model.2.1.conv.unit0.conv.bias". 
Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/site-packages/ignite/engine/engine.py", line 737, in _internal_run
    self._fire_event(Events.STARTED)
  File "/opt/conda/lib/python3.8/site-packages/ignite/engine/engine.py", line 424, in _fire_event
    func(*first, *(event_args + others), **kwargs)
  File "/opt/monai/monai/handlers/checkpoint_loader.py", line 110, in __call__
    Checkpoint.load_objects(to_load=self.load_dict, checkpoint=checkpoint, strict=self.strict)
  File "/opt/conda/lib/python3.8/site-packages/ignite/handlers/checkpoint.py", line 566, in load_objects
    obj.load_state_dict(checkpoint[k], strict=is_state_dict_strict)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1215, in load_state_dict
    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for NewLesionsUNet:
	Missing key(s) in state_dict: "segmenter.0.down.0.conv.weight", "segmenter.0.down.0.conv.bias", "segmenter.0.down.0.res.weight", "segmenter.0.down.0.res.bias", "segmenter.0.down.0.end_seq.1.weight", "segmenter.0.down.0.end_seq.1.bias", "segmenter.0.down.1.conv.weight", "segmenter.0.down.1.conv.bias", "segmenter.0.down.1.res.weight", "segmenter.0.down.1.res.bias", "segmenter.0.down.1.end_seq.1.weight", "segmenter.0.down.1.end_seq.1.bias", "segmenter.0.down.2.conv.weight", "segmenter.0.down.2.conv.bias", "segmenter.0.down.2.res.weight", "segmenter.0.down.2.res.bias", "segmenter.0.down.2.end_seq.1.weight", "segmenter.0.down.2.end_seq.1.bias", "segmenter.0.down.3.conv.weight", "segmenter.0.down.3.conv.bias", "segmenter.0.down.3.res.weight", "segmenter.0.down.3.res.bias", "segmenter.0.down.3.end_seq.1.weight", "segmenter.0.down.3.end_seq.1.bias", "segmenter.0.u.conv.weight", "segmenter.0.u.conv.bias", "segmenter.0.u.res.weight", "segmenter.0.u.res.bias", "segmenter.0.u.end_seq.1.weight", "segmenter.0.u.end_seq.1.bias", "segmenter.0.up.0.conv.weight", "segmenter.0.up.0.conv.bias", "segmenter.0.up.0.res.weight", "segmenter.0.up.0.res.bias", "segmenter.0.up.0.end_seq.1.weight", "segmenter.0.up.0.end_seq.1.bias", "segmenter.0.up.1.conv.weight", "segmenter.0.up.1.conv.bias", "segmenter.0.up.1.res.weight", "segmenter.0.up.1.res.bias", "segmenter.0.up.1.end_seq.1.weight", "segmenter.0.up.1.end_seq.1.bias", "segmenter.0.up.2.conv.weight", "segmenter.0.up.2.conv.bias", "segmenter.0.up.2.res.weight", "segmenter.0.up.2.res.bias", "segmenter.0.up.2.end_seq.1.weight", "segmenter.0.up.2.end_seq.1.bias", "segmenter.0.up.3.conv.weight", "segmenter.0.up.3.conv.bias", "segmenter.0.up.3.res.weight", "segmenter.0.up.3.res.bias", "segmenter.0.up.3.end_seq.1.weight", "segmenter.0.up.3.end_seq.1.bias", "segmenter.1.weight", "segmenter.1.bias". 
	Unexpected key(s) in state_dict: "model.0.conv.unit0.conv.weight", "model.0.conv.unit0.conv.bias", "model.0.conv.unit0.adn.N.weight", "model.0.conv.unit0.adn.N.bias", "model.0.conv.unit0.adn.N.running_mean", "model.0.conv.unit0.adn.N.running_var", "model.0.conv.unit0.adn.N.num_batches_tracked", "model.0.conv.unit0.adn.A.weight", "model.0.conv.unit1.conv.weight", "model.0.conv.unit1.conv.bias", "model.0.conv.unit1.adn.N.weight", "model.0.conv.unit1.adn.N.bias", "model.0.conv.unit1.adn.N.running_mean", "model.0.conv.unit1.adn.N.running_var", "model.0.conv.unit1.adn.N.num_batches_tracked", "model.0.conv.unit1.adn.A.weight", "model.0.residual.weight", "model.0.residual.bias", "model.1.submodule.0.conv.unit0.conv.weight", "model.1.submodule.0.conv.unit0.conv.bias", "model.1.submodule.0.conv.unit0.adn.N.weight", "model.1.submodule.0.conv.unit0.adn.N.bias", "model.1.submodule.0.conv.unit0.adn.N.running_mean", "model.1.submodule.0.conv.unit0.adn.N.running_var", "model.1.submodule.0.conv.unit0.adn.N.num_batches_tracked", "model.1.submodule.0.conv.unit0.adn.A.weight", "model.1.submodule.0.conv.unit1.conv.weight", "model.1.submodule.0.conv.unit1.conv.bias", "model.1.submodule.0.conv.unit1.adn.N.weight", "model.1.submodule.0.conv.unit1.adn.N.bias", "model.1.submodule.0.conv.unit1.adn.N.running_mean", "model.1.submodule.0.conv.unit1.adn.N.running_var", "model.1.submodule.0.conv.unit1.adn.N.num_batches_tracked", "model.1.submodule.0.conv.unit1.adn.A.weight", "model.1.submodule.0.residual.weight", "model.1.submodule.0.residual.bias", "model.1.submodule.1.submodule.0.conv.unit0.conv.weight", "model.1.submodule.1.submodule.0.conv.unit0.conv.bias", "model.1.submodule.1.submodule.0.conv.unit0.adn.N.weight", "model.1.submodule.1.submodule.0.conv.unit0.adn.N.bias", "model.1.submodule.1.submodule.0.conv.unit0.adn.N.running_mean", "model.1.submodule.1.submodule.0.conv.unit0.adn.N.running_var", "model.1.submodule.1.submodule.0.conv.unit0.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.0.conv.unit0.adn.A.weight", "model.1.submodule.1.submodule.0.conv.unit1.conv.weight", "model.1.submodule.1.submodule.0.conv.unit1.conv.bias", "model.1.submodule.1.submodule.0.conv.unit1.adn.N.weight", "model.1.submodule.1.submodule.0.conv.unit1.adn.N.bias", "model.1.submodule.1.submodule.0.conv.unit1.adn.N.running_mean", "model.1.submodule.1.submodule.0.conv.unit1.adn.N.running_var", "model.1.submodule.1.submodule.0.conv.unit1.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.0.conv.unit1.adn.A.weight", "model.1.submodule.1.submodule.0.residual.weight", "model.1.submodule.1.submodule.0.residual.bias", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.conv.weight", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.conv.bias", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.adn.N.weight", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.adn.N.bias", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.adn.N.running_mean", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.adn.N.running_var", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.adn.A.weight", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.conv.weight", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.conv.bias", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.adn.N.weight", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.adn.N.bias", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.adn.N.running_mean", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.adn.N.running_var", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.adn.A.weight", "model.1.submodule.1.submodule.1.submodule.0.residual.weight", "model.1.submodule.1.submodule.1.submodule.0.residual.bias", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.conv.weight", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.conv.bias", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.adn.N.weight", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.adn.N.bias", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.adn.N.running_mean", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.adn.N.running_var", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.adn.A.weight", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.conv.weight", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.conv.bias", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.adn.N.weight", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.adn.N.bias", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.adn.N.running_mean", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.adn.N.running_var", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.adn.A.weight", "model.1.submodule.1.submodule.1.submodule.1.submodule.residual.weight", "model.1.submodule.1.submodule.1.submodule.1.submodule.residual.bias", "model.1.submodule.1.submodule.1.submodule.2.0.conv.weight", "model.1.submodule.1.submodule.1.submodule.2.0.conv.bias", "model.1.submodule.1.submodule.1.submodule.2.0.adn.N.weight", "model.1.submodule.1.submodule.1.submodule.2.0.adn.N.bias", "model.1.submodule.1.submodule.1.submodule.2.0.adn.N.running_mean", "model.1.submodule.1.submodule.1.submodule.2.0.adn.N.running_var", "model.1.submodule.1.submodule.1.submodule.2.0.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.1.submodule.2.0.adn.A.weight", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.conv.weight", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.conv.bias", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.adn.N.weight", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.adn.N.bias", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.adn.N.running_mean", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.adn.N.running_var", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.adn.A.weight", "model.1.submodule.1.submodule.2.0.conv.weight", "model.1.submodule.1.submodule.2.0.conv.bias", "model.1.submodule.1.submodule.2.0.adn.N.weight", "model.1.submodule.1.submodule.2.0.adn.N.bias", "model.1.submodule.1.submodule.2.0.adn.N.running_mean", "model.1.submodule.1.submodule.2.0.adn.N.running_var", "model.1.submodule.1.submodule.2.0.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.2.0.adn.A.weight", "model.1.submodule.1.submodule.2.1.conv.unit0.conv.weight", "model.1.submodule.1.submodule.2.1.conv.unit0.conv.bias", "model.1.submodule.1.submodule.2.1.conv.unit0.adn.N.weight", "model.1.submodule.1.submodule.2.1.conv.unit0.adn.N.bias", "model.1.submodule.1.submodule.2.1.conv.unit0.adn.N.running_mean", "model.1.submodule.1.submodule.2.1.conv.unit0.adn.N.running_var", "model.1.submodule.1.submodule.2.1.conv.unit0.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.2.1.conv.unit0.adn.A.weight", "model.1.submodule.2.0.conv.weight", "model.1.submodule.2.0.conv.bias", "model.1.submodule.2.0.adn.N.weight", "model.1.submodule.2.0.adn.N.bias", "model.1.submodule.2.0.adn.N.running_mean", "model.1.submodule.2.0.adn.N.running_var", "model.1.submodule.2.0.adn.N.num_batches_tracked", "model.1.submodule.2.0.adn.A.weight", "model.1.submodule.2.1.conv.unit0.conv.weight", "model.1.submodule.2.1.conv.unit0.conv.bias", "model.1.submodule.2.1.conv.unit0.adn.N.weight", "model.1.submodule.2.1.conv.unit0.adn.N.bias", "model.1.submodule.2.1.conv.unit0.adn.N.running_mean", "model.1.submodule.2.1.conv.unit0.adn.N.running_var", "model.1.submodule.2.1.conv.unit0.adn.N.num_batches_tracked", "model.1.submodule.2.1.conv.unit0.adn.A.weight", "model.2.0.conv.weight", "model.2.0.conv.bias", "model.2.0.adn.N.weight", "model.2.0.adn.N.bias", "model.2.0.adn.N.running_mean", "model.2.0.adn.N.running_var", "model.2.0.adn.N.num_batches_tracked", "model.2.0.adn.A.weight", "model.2.1.conv.unit0.conv.weight", "model.2.1.conv.unit0.conv.bias". 
Traceback (most recent call last):
  File "apps/fed_learn/validators/client_validator.py", line 68, in validate
  File "apps/mmar_conf.py", line 69, in evaluate_mmar
  File "/opt/monai/monai/engines/evaluator.py", line 120, in run
    super().run()
  File "/opt/monai/monai/engines/workflow.py", line 206, in run
    super().run(data=self.data_loader, max_epochs=self.state.max_epochs)
  File "/opt/conda/lib/python3.8/site-packages/ignite/engine/engine.py", line 702, in run
    return self._internal_run()
  File "/opt/conda/lib/python3.8/site-packages/ignite/engine/engine.py", line 775, in _internal_run
    self._handle_exception(e)
  File "/opt/conda/lib/python3.8/site-packages/ignite/engine/engine.py", line 467, in _handle_exception
    self._fire_event(Events.EXCEPTION_RAISED, e)
  File "/opt/conda/lib/python3.8/site-packages/ignite/engine/engine.py", line 424, in _fire_event
    func(*first, *(event_args + others), **kwargs)
  File "/opt/monai/monai/handlers/stats_handler.py", line 145, in exception_raised
    raise e
  File "/opt/conda/lib/python3.8/site-packages/ignite/engine/engine.py", line 737, in _internal_run
    self._fire_event(Events.STARTED)
  File "/opt/conda/lib/python3.8/site-packages/ignite/engine/engine.py", line 424, in _fire_event
    func(*first, *(event_args + others), **kwargs)
  File "/opt/monai/monai/handlers/checkpoint_loader.py", line 110, in __call__
    Checkpoint.load_objects(to_load=self.load_dict, checkpoint=checkpoint, strict=self.strict)
  File "/opt/conda/lib/python3.8/site-packages/ignite/handlers/checkpoint.py", line 566, in load_objects
    obj.load_state_dict(checkpoint[k], strict=is_state_dict_strict)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1215, in load_state_dict
    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for NewLesionsUNet:
	Missing key(s) in state_dict: "segmenter.0.down.0.conv.weight", "segmenter.0.down.0.conv.bias", "segmenter.0.down.0.res.weight", "segmenter.0.down.0.res.bias", "segmenter.0.down.0.end_seq.1.weight", "segmenter.0.down.0.end_seq.1.bias", "segmenter.0.down.1.conv.weight", "segmenter.0.down.1.conv.bias", "segmenter.0.down.1.res.weight", "segmenter.0.down.1.res.bias", "segmenter.0.down.1.end_seq.1.weight", "segmenter.0.down.1.end_seq.1.bias", "segmenter.0.down.2.conv.weight", "segmenter.0.down.2.conv.bias", "segmenter.0.down.2.res.weight", "segmenter.0.down.2.res.bias", "segmenter.0.down.2.end_seq.1.weight", "segmenter.0.down.2.end_seq.1.bias", "segmenter.0.down.3.conv.weight", "segmenter.0.down.3.conv.bias", "segmenter.0.down.3.res.weight", "segmenter.0.down.3.res.bias", "segmenter.0.down.3.end_seq.1.weight", "segmenter.0.down.3.end_seq.1.bias", "segmenter.0.u.conv.weight", "segmenter.0.u.conv.bias", "segmenter.0.u.res.weight", "segmenter.0.u.res.bias", "segmenter.0.u.end_seq.1.weight", "segmenter.0.u.end_seq.1.bias", "segmenter.0.up.0.conv.weight", "segmenter.0.up.0.conv.bias", "segmenter.0.up.0.res.weight", "segmenter.0.up.0.res.bias", "segmenter.0.up.0.end_seq.1.weight", "segmenter.0.up.0.end_seq.1.bias", "segmenter.0.up.1.conv.weight", "segmenter.0.up.1.conv.bias", "segmenter.0.up.1.res.weight", "segmenter.0.up.1.res.bias", "segmenter.0.up.1.end_seq.1.weight", "segmenter.0.up.1.end_seq.1.bias", "segmenter.0.up.2.conv.weight", "segmenter.0.up.2.conv.bias", "segmenter.0.up.2.res.weight", "segmenter.0.up.2.res.bias", "segmenter.0.up.2.end_seq.1.weight", "segmenter.0.up.2.end_seq.1.bias", "segmenter.0.up.3.conv.weight", "segmenter.0.up.3.conv.bias", "segmenter.0.up.3.res.weight", "segmenter.0.up.3.res.bias", "segmenter.0.up.3.end_seq.1.weight", "segmenter.0.up.3.end_seq.1.bias", "segmenter.1.weight", "segmenter.1.bias". 
	Unexpected key(s) in state_dict: "model.0.conv.unit0.conv.weight", "model.0.conv.unit0.conv.bias", "model.0.conv.unit0.adn.N.weight", "model.0.conv.unit0.adn.N.bias", "model.0.conv.unit0.adn.N.running_mean", "model.0.conv.unit0.adn.N.running_var", "model.0.conv.unit0.adn.N.num_batches_tracked", "model.0.conv.unit0.adn.A.weight", "model.0.conv.unit1.conv.weight", "model.0.conv.unit1.conv.bias", "model.0.conv.unit1.adn.N.weight", "model.0.conv.unit1.adn.N.bias", "model.0.conv.unit1.adn.N.running_mean", "model.0.conv.unit1.adn.N.running_var", "model.0.conv.unit1.adn.N.num_batches_tracked", "model.0.conv.unit1.adn.A.weight", "model.0.residual.weight", "model.0.residual.bias", "model.1.submodule.0.conv.unit0.conv.weight", "model.1.submodule.0.conv.unit0.conv.bias", "model.1.submodule.0.conv.unit0.adn.N.weight", "model.1.submodule.0.conv.unit0.adn.N.bias", "model.1.submodule.0.conv.unit0.adn.N.running_mean", "model.1.submodule.0.conv.unit0.adn.N.running_var", "model.1.submodule.0.conv.unit0.adn.N.num_batches_tracked", "model.1.submodule.0.conv.unit0.adn.A.weight", "model.1.submodule.0.conv.unit1.conv.weight", "model.1.submodule.0.conv.unit1.conv.bias", "model.1.submodule.0.conv.unit1.adn.N.weight", "model.1.submodule.0.conv.unit1.adn.N.bias", "model.1.submodule.0.conv.unit1.adn.N.running_mean", "model.1.submodule.0.conv.unit1.adn.N.running_var", "model.1.submodule.0.conv.unit1.adn.N.num_batches_tracked", "model.1.submodule.0.conv.unit1.adn.A.weight", "model.1.submodule.0.residual.weight", "model.1.submodule.0.residual.bias", "model.1.submodule.1.submodule.0.conv.unit0.conv.weight", "model.1.submodule.1.submodule.0.conv.unit0.conv.bias", "model.1.submodule.1.submodule.0.conv.unit0.adn.N.weight", "model.1.submodule.1.submodule.0.conv.unit0.adn.N.bias", "model.1.submodule.1.submodule.0.conv.unit0.adn.N.running_mean", "model.1.submodule.1.submodule.0.conv.unit0.adn.N.running_var", "model.1.submodule.1.submodule.0.conv.unit0.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.0.conv.unit0.adn.A.weight", "model.1.submodule.1.submodule.0.conv.unit1.conv.weight", "model.1.submodule.1.submodule.0.conv.unit1.conv.bias", "model.1.submodule.1.submodule.0.conv.unit1.adn.N.weight", "model.1.submodule.1.submodule.0.conv.unit1.adn.N.bias", "model.1.submodule.1.submodule.0.conv.unit1.adn.N.running_mean", "model.1.submodule.1.submodule.0.conv.unit1.adn.N.running_var", "model.1.submodule.1.submodule.0.conv.unit1.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.0.conv.unit1.adn.A.weight", "model.1.submodule.1.submodule.0.residual.weight", "model.1.submodule.1.submodule.0.residual.bias", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.conv.weight", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.conv.bias", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.adn.N.weight", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.adn.N.bias", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.adn.N.running_mean", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.adn.N.running_var", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.adn.A.weight", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.conv.weight", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.conv.bias", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.adn.N.weight", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.adn.N.bias", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.adn.N.running_mean", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.adn.N.running_var", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.adn.A.weight", "model.1.submodule.1.submodule.1.submodule.0.residual.weight", "model.1.submodule.1.submodule.1.submodule.0.residual.bias", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.conv.weight", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.conv.bias", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.adn.N.weight", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.adn.N.bias", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.adn.N.running_mean", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.adn.N.running_var", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.adn.A.weight", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.conv.weight", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.conv.bias", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.adn.N.weight", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.adn.N.bias", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.adn.N.running_mean", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.adn.N.running_var", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.adn.A.weight", "model.1.submodule.1.submodule.1.submodule.1.submodule.residual.weight", "model.1.submodule.1.submodule.1.submodule.1.submodule.residual.bias", "model.1.submodule.1.submodule.1.submodule.2.0.conv.weight", "model.1.submodule.1.submodule.1.submodule.2.0.conv.bias", "model.1.submodule.1.submodule.1.submodule.2.0.adn.N.weight", "model.1.submodule.1.submodule.1.submodule.2.0.adn.N.bias", "model.1.submodule.1.submodule.1.submodule.2.0.adn.N.running_mean", "model.1.submodule.1.submodule.1.submodule.2.0.adn.N.running_var", "model.1.submodule.1.submodule.1.submodule.2.0.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.1.submodule.2.0.adn.A.weight", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.conv.weight", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.conv.bias", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.adn.N.weight", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.adn.N.bias", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.adn.N.running_mean", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.adn.N.running_var", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.adn.A.weight", "model.1.submodule.1.submodule.2.0.conv.weight", "model.1.submodule.1.submodule.2.0.conv.bias", "model.1.submodule.1.submodule.2.0.adn.N.weight", "model.1.submodule.1.submodule.2.0.adn.N.bias", "model.1.submodule.1.submodule.2.0.adn.N.running_mean", "model.1.submodule.1.submodule.2.0.adn.N.running_var", "model.1.submodule.1.submodule.2.0.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.2.0.adn.A.weight", "model.1.submodule.1.submodule.2.1.conv.unit0.conv.weight", "model.1.submodule.1.submodule.2.1.conv.unit0.conv.bias", "model.1.submodule.1.submodule.2.1.conv.unit0.adn.N.weight", "model.1.submodule.1.submodule.2.1.conv.unit0.adn.N.bias", "model.1.submodule.1.submodule.2.1.conv.unit0.adn.N.running_mean", "model.1.submodule.1.submodule.2.1.conv.unit0.adn.N.running_var", "model.1.submodule.1.submodule.2.1.conv.unit0.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.2.1.conv.unit0.adn.A.weight", "model.1.submodule.2.0.conv.weight", "model.1.submodule.2.0.conv.bias", "model.1.submodule.2.0.adn.N.weight", "model.1.submodule.2.0.adn.N.bias", "model.1.submodule.2.0.adn.N.running_mean", "model.1.submodule.2.0.adn.N.running_var", "model.1.submodule.2.0.adn.N.num_batches_tracked", "model.1.submodule.2.0.adn.A.weight", "model.1.submodule.2.1.conv.unit0.conv.weight", "model.1.submodule.2.1.conv.unit0.conv.bias", "model.1.submodule.2.1.conv.unit0.adn.N.weight", "model.1.submodule.2.1.conv.unit0.adn.N.bias", "model.1.submodule.2.1.conv.unit0.adn.N.running_mean", "model.1.submodule.2.1.conv.unit0.adn.N.running_var", "model.1.submodule.2.1.conv.unit0.adn.N.num_batches_tracked", "model.1.submodule.2.1.conv.unit0.adn.A.weight", "model.2.0.conv.weight", "model.2.0.conv.bias", "model.2.0.adn.N.weight", "model.2.0.adn.N.bias", "model.2.0.adn.N.running_mean", "model.2.0.adn.N.running_var", "model.2.0.adn.N.num_batches_tracked", "model.2.0.adn.A.weight", "model.2.1.conv.unit0.conv.weight", "model.2.1.conv.unit0.conv.bias". 
Exception in executing validation: 
2021-06-21 00:44:18,060 - ClientValidator - INFO - Do validation ...
2021-06-21 00:44:18,061 - CrossSiteValManager - INFO - Exception in validating FL_global_model's model: local variable 'metrics' referenced before assignment
2021-06-21 00:44:18,065 - FederatedClient - INFO - Submitting cross validation results to server.
2021-06-21 00:44:18,085 - Communicator - INFO - Received comments:  Received Cross Validation results from org1-b.. SubmitCrossSiteValidationResults time: 0.01889824867248535 seconds
2021-06-21 00:44:18,086 - FederatedClient - INFO - More models available with server.
2021-06-21 00:44:18,088 - FederatedClient - INFO - Getting other models from server for cross validation.
2021-06-21 00:44:25,295 - Communicator - INFO - Received 1 models for validation. GetValidationModels time: 7.206388711929321 seconds
========== Validate Config Result ===========
Use GPU:  True
Multi GPU:  False
Automatic Mixed Precision:  Enabled
Determinism Evaluation:  Disabled
cuDNN BenchMark:  False
CUDA Matmul Allow TF32:  True
cuDNN Allow TF32:  True
Model:  <class 'lesion-activity-clara-fl.NewLesionsUNet.model.NewLesionsUNet'>
Dataset:  <class 'monai.data.dataset.Dataset'>
DataLoader:  <class 'monai.data.dataloader.DataLoader'>
Validate Transform #1: <class 'monai.transforms.io.dictionary.LoadImaged'>
Validate Transform #2: <class 'lesion-activity-clara-fl.NewLesionsUNet.pre_transform.GetMask'>
Validate Transform #3: <class 'monai.transforms.utility.dictionary.ToTensord'>
Validate Handler #1: <class 'monai.handlers.stats_handler.StatsHandler'>
Validate Handler #2: <class 'monai.handlers.checkpoint_loader.CheckpointLoader'>
Validate Handler #3: <class 'monai.handlers.segmentation_saver.SegmentationSaver'>
Validate Post Transforms #1: <class 'monai.transforms.post.dictionary.Activationsd'>
Validate Post Transforms #2: <class 'lesion-activity-clara-fl.NewLesionsUNet.pre_transform.RemoveSmallRegions'>
Validate Inferer:  <class 'monai.inferers.inferer.SlidingWindowInferer'>
Validate Key Metric:  <class 'monai.handlers.mean_dice.MeanDice'>
Validate Additional Metric #My own validation mean dice loss: <class 'lesion-activity-clara-fl.NewLesionsUNet.model.MyValLoss'>
========== End of Validate Config Result ===========
2021-06-21 00:44:25,776 - ignite.engine.engine.SupervisedEvaluator - INFO - Engine run resuming from iteration 0, epoch 0 until 1 epochs
2021-06-21 00:44:25,809 - ignite.engine.engine.SupervisedEvaluator - INFO - Restored all variables from /tmp/tmprbl1enet/mmar/models/local_best_model.PT_MODEL
file written: /tmp/tmprbl1enet/mmar/eval/brain_mask/brain_mask_seg.nii.gz.
2021-06-21 00:45:09,969 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files.
file written: /tmp/tmprbl1enet/mmar/eval/brain_mask/brain_mask_seg.nii.gz.
2021-06-21 00:45:24,899 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files.
file written: /tmp/tmprbl1enet/mmar/eval/brain_mask/brain_mask_seg.nii.gz.
2021-06-21 00:46:02,528 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files.
file written: /tmp/tmprbl1enet/mmar/eval/brain_mask/brain_mask_seg.nii.gz.
2021-06-21 00:46:45,839 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files.
file written: /tmp/tmprbl1enet/mmar/eval/brain_mask/brain_mask_seg.nii.gz.
2021-06-21 00:47:12,778 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files.
file written: /tmp/tmprbl1enet/mmar/eval/brain_mask/brain_mask_seg.nii.gz.
2021-06-21 00:47:16,232 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files.
file written: /tmp/tmprbl1enet/mmar/eval/brain_mask/brain_mask_seg.nii.gz.
2021-06-21 00:47:33,871 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files.
file written: /tmp/tmprbl1enet/mmar/eval/brain_mask/brain_mask_seg.nii.gz.
2021-06-21 00:48:03,250 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files.
file written: /tmp/tmprbl1enet/mmar/eval/brain_mask/brain_mask_seg.nii.gz.
2021-06-21 00:48:38,388 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files.
file written: /tmp/tmprbl1enet/mmar/eval/brain_mask/brain_mask_seg.nii.gz.
2021-06-21 00:49:20,375 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files.
file written: /tmp/tmprbl1enet/mmar/eval/brain_mask/brain_mask_seg.nii.gz.
2021-06-21 00:50:09,896 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files.
file written: /tmp/tmprbl1enet/mmar/eval/brain_mask/brain_mask_seg.nii.gz.
2021-06-21 00:50:17,931 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files.
file written: /tmp/tmprbl1enet/mmar/eval/brain_mask/brain_mask_seg.nii.gz.
2021-06-21 00:50:20,037 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files.
file written: /tmp/tmprbl1enet/mmar/eval/brain_mask/brain_mask_seg.nii.gz.
2021-06-21 00:51:04,369 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files.
file written: /tmp/tmprbl1enet/mmar/eval/brain_mask/brain_mask_seg.nii.gz.
2021-06-21 00:51:21,276 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files.
file written: /tmp/tmprbl1enet/mmar/eval/brain_mask/brain_mask_seg.nii.gz.
2021-06-21 00:51:27,700 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files.
file written: /tmp/tmprbl1enet/mmar/eval/brain_mask/brain_mask_seg.nii.gz.
2021-06-21 00:51:34,326 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files.
file written: /tmp/tmprbl1enet/mmar/eval/brain_mask/brain_mask_seg.nii.gz.
2021-06-21 00:52:10,485 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files.
file written: /tmp/tmprbl1enet/mmar/eval/brain_mask/brain_mask_seg.nii.gz.
2021-06-21 00:52:30,409 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files.
file written: /tmp/tmprbl1enet/mmar/eval/brain_mask/brain_mask_seg.nii.gz.
2021-06-21 00:52:33,670 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files.
2021-06-21 00:52:33,672 - ignite.engine.engine.SupervisedEvaluator - INFO - Got new best metric of Monai's validation mean dice loss: 0.4330536425113678
2021-06-21 00:52:33,672 - ignite.engine.engine.SupervisedEvaluator - INFO - Epoch[1] Metrics -- Monai's validation mean dice loss: 0.4331 My own validation mean dice loss: 0.4638 
2021-06-21 00:52:33,672 - ignite.engine.engine.SupervisedEvaluator - INFO - Key metric: Monai's validation mean dice loss best value: 0.4330536425113678 at epoch: 1
2021-06-21 00:52:33,672 - ignite.engine.engine.SupervisedEvaluator - INFO - Epoch[1] Complete. Time taken: 00:08:08
2021-06-21 00:52:33,672 - ignite.engine.engine.SupervisedEvaluator - INFO - Engine run complete. Time taken: 00:08:08
2021-06-21 00:52:33,773 - medl.apps.mmar_conf - INFO - Total Evaluation Time 487.9964783191681
2021-06-21 00:52:33,792 - ClientValidator - INFO - Do validation ...
2021-06-21 00:52:33,800 - FederatedClient - INFO - Submitting cross validation results to server.
2021-06-21 00:52:33,820 - Communicator - INFO - Received comments:  Received Cross Validation results from org1-b.. SubmitCrossSiteValidationResults time: 0.018047332763671875 seconds
2021-06-21 00:52:33,820 - FederatedClient - INFO - Finished cross site validation.