siddharth@FLS-1:~/packages7/org1-a/startup$ siddharth@FLS-1:~/packages7/org1-a/startup$ siddharth@FLS-1:~/packages7/org1-a/startup$ ls client_context.tenseal client.crt client.key docker.sh fed_client.json log.config readme.txt rootCA.pem signature.pkl start.sh sub_start.sh siddharth@FLS-1:~/packages7/org1-a/startup$ siddharth@FLS-1:~/packages7/org1-a/startup$ siddharth@FLS-1:~/packages7/org1-a/startup$ ./docker.sh Starting docker with nvcr.io/nvidia/clara-train-sdk:v4.0 ============= == PyTorch == ============= NVIDIA Release 21.02 (build 20138038) PyTorch Version 1.8.0a0+52ea372 Container image Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved. Copyright (c) 2014-2021 Facebook Inc. Copyright (c) 2011-2014 Idiap Research Institute (Ronan Collobert) Copyright (c) 2012-2014 Deepmind Technologies (Koray Kavukcuoglu) Copyright (c) 2011-2012 NEC Laboratories America (Koray Kavukcuoglu) Copyright (c) 2011-2013 NYU (Clement Farabet) Copyright (c) 2006-2010 NEC Laboratories America (Ronan Collobert, Leon Bottou, Iain Melvin, Jason Weston) Copyright (c) 2006 Idiap Research Institute (Samy Bengio) Copyright (c) 2001-2004 Idiap Research Institute (Ronan Collobert, Samy Bengio, Johnny Mariethoz) Copyright (c) 2015 Google Inc. Copyright (c) 2015 Yangqing Jia Copyright (c) 2013-2016 The Caffe contributors All rights reserved. NVIDIA Deep Learning Profiler (dlprof) Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved. Various files include modifications (c) NVIDIA CORPORATION. All rights reserved. This container image and its contents are governed by the NVIDIA Deep Learning Container License. By pulling and using the container, you accept the terms and conditions of this license: https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license NOTE: MOFED driver for multi-node communication was not detected. Multi-node communication performance may be reduced. siddharth@FLS-1:/workspace$ ls audit.log data run_1 run_514 run_552 run_616 run_66 run_713 run_764 run_814 run_85 run_921 run_948 run_988 challenge_data lesion-activity-clara-fl run_23 run_516 run_612 run_618 run_671 run_716 run_773 run_816 run_88 run_943 run_949 run_998 client_token.txt log.txt run_416 run_517 run_613 run_619 run_686 run_718 run_784 run_818 run_916 run_944 run_964 startup daemon_pid.fl pid.fl run_502 run_524 run_614 run_626 run_712 run_72 run_810 run_84 run_918 run_946 run_98 transfer siddharth@FLS-1:/workspace$ cd startup/ siddharth@FLS-1:/workspace/startup$ siddharth@FLS-1:/workspace/startup$ ls client.crt client.key client_context.tenseal docker.sh fed_client.json log.config readme.txt rootCA.pem signature.pkl start.sh sub_start.sh siddharth@FLS-1:/workspace/startup$ siddharth@FLS-1:/workspace/startup$ ./start.sh WORKSPACE set to /workspace/startup/.. siddharth@FLS-1:/workspace/startup$ WORKSPACE set to /workspace/startup/.. PYTHONPATH is /local/custom::/workspace:/workspace/lesion-activity-clara-fl:/opt/nvidia/medical:/workspace/lesion-activity-clara-fl/NewLesionsUNet There seems to be one instance, pid=221, running. If you are sure it's not the case, please kill process 221. siddharth@FLS-1:/workspace/startup$ ./start.sh WORKSPACE set to /workspace/startup/.. siddharth@FLS-1:/workspace/startup$ WORKSPACE set to /workspace/startup/.. PYTHONPATH is /local/custom::/workspace:/workspace/lesion-activity-clara-fl:/opt/nvidia/medical:/workspace/lesion-activity-clara-fl/NewLesionsUNet start fl because process of 239 does not exist new pid 262 2021-06-20 23:37:24,461 - FederatedClient - INFO - Successfully registered client:org1-a for lesion_activity. Got token:51876d00-e779-4b2a-a8e4-98199da1819f created /tmp/fl/org1-a/comm/training/x created /tmp/fl/org1-a/comm/training/y created /tmp/fl/org1-a/comm/training/t 2021-06-20 23:38:47,715 - ClientAdminInterface - INFO - Starting client training. rank: 0 training child process ID: 455 starting the client ..... token is: 51876d00-e779-4b2a-a8e4-98199da1819f run_number is: 711 uid: org1-a listen_port: 40593 2021-06-20 23:38:48,358 - matplotlib - WARNING - Matplotlib created a temporary config/cache directory at /tmp/matplotlib-n2belqcu because the default path (/home/siddharth/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing. 2021-06-20 23:38:48,386 - matplotlib.font_manager - INFO - Generating new fontManager, this may take some time... 2021-06-20 23:38:48,750 - ProcessExecutor - INFO - waiting for process to finish Created the listener on port: 40593 2021-06-20 23:38:50,556 - torch.distributed.nn.jit.instantiator - INFO - Created a temporary directory at /tmp/tmpprlyle2t 2021-06-20 23:38:50,556 - torch.distributed.nn.jit.instantiator - INFO - Writing /tmp/tmpprlyle2t/_remote_module_non_sriptable.py no max_epochs specified for SupervisedTrainer args, use global var 'epochs: 10'. ========== Train Config Result =========== Num Epochs: 10 Use GPU: True Multi GPU: False Automatic Mixed Precision: Disabled Determinism Training: Enabled cuDNN BenchMark: False CUDA Matmul Allow TF32: True cuDNN Allow TF32: True Model: Loss: Optimizer: LR Scheduler: Train Dataset: Train DataLoader: Train Transform #1: Validate Dataset: Validate DataLoader: Validate Transform #1: Validate Transform #2: Validate Transform #3: Validate Transform #4: Train Handler #1: Train Handler #2: Train Handler #3: Train Handler #4: Validate Handler #1: Validate Handler #2: Validate Handler #3: Validate Post Transforms #1: Validate Post Transforms #2: Validate Inferer: Validate Key Metric: Validate Additional Metric #My own validation mean dice loss: Train Inferer: ========== End of Train Config Result =========== 2021-06-20 23:40:47,590 - FederatedClient - INFO - Starting to fetch global model. 2021-06-20 23:40:50,356 - Communicator - INFO - Received lesion_activity model at round 0 (19259350 Bytes). GetModel time: 2.758026361465454 seconds Get global model for round: 0 pull_models completed. Status:True rank:0 2021-06-20 23:40:50,366 - ClientTrainer - INFO - ClientTrainer abort signal: False 2021-06-20 23:40:50,367 - AssignVariables - INFO - Vars 148 of 0 assigned. 2021-06-20 23:40:50,371 - ignite.engine.engine.SupervisedEvaluator - INFO - Engine run resuming from iteration 0, epoch 0 until 1 epochs 2021-06-20 23:49:15,057 - ignite.engine.engine.SupervisedEvaluator - INFO - Got new best metric of Monai's validation mean dice loss: 2.1376723452704027e-05 2021-06-20 23:49:15,058 - ignite.engine.engine.SupervisedEvaluator - INFO - Epoch[1] Metrics -- Monai's validation mean dice loss: 0.0000 My own validation mean dice loss: 1.0000 2021-06-20 23:49:15,058 - ignite.engine.engine.SupervisedEvaluator - INFO - Key metric: Monai's validation mean dice loss best value: 2.1376723452704027e-05 at epoch: 1 2021-06-20 23:49:15,142 - ignite.engine.engine.SupervisedEvaluator - INFO - Epoch[1] Complete. Time taken: 00:08:25 2021-06-20 23:49:15,142 - ignite.engine.engine.SupervisedEvaluator - INFO - Engine run complete. Time taken: 00:08:25 2021-06-20 23:49:15,191 - ignite.engine.engine.SupervisedTrainer - INFO - Engine run resuming from iteration 0, epoch 0 until 1 epochs 2021-06-20 23:49:16,131 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 1/313 -- train_loss: 1.7788 2021-06-20 23:49:16,403 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 2/313 -- train_loss: 1.1806 2021-06-20 23:49:16,643 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 3/313 -- train_loss: 1.1150 2021-06-20 23:49:16,927 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 4/313 -- train_loss: 1.1047 2021-06-20 23:49:17,167 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 5/313 -- train_loss: 1.0875 2021-06-20 23:49:17,411 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 6/313 -- train_loss: 1.0839 2021-06-20 23:49:17,666 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 7/313 -- train_loss: 1.0805 2021-06-20 23:49:17,945 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 8/313 -- train_loss: 1.0759 2021-06-20 23:49:18,186 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 9/313 -- train_loss: 1.0586 2021-06-20 23:49:18,457 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 10/313 -- train_loss: 1.0594 2021-06-20 23:49:18,772 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 11/313 -- train_loss: 1.0549 2021-06-20 23:49:19,031 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 12/313 -- train_loss: 1.0572 2021-06-20 23:49:19,275 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 13/313 -- train_loss: 1.0455 2021-06-20 23:49:19,515 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 14/313 -- train_loss: 1.0290 2021-06-20 23:49:19,760 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 15/313 -- train_loss: 1.0299 2021-06-20 23:49:20,001 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 16/313 -- train_loss: 1.0391 2021-06-20 23:49:20,386 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 17/313 -- train_loss: 1.0163 2021-06-20 23:49:20,626 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 18/313 -- train_loss: 0.9644 2021-06-20 23:49:20,867 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 19/313 -- train_loss: 1.0070 2021-06-20 23:49:21,108 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 20/313 -- train_loss: 1.0074 2021-06-20 23:49:21,349 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 21/313 -- train_loss: 1.0195 2021-06-20 23:49:21,589 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 22/313 -- train_loss: 1.0142 2021-06-20 23:49:21,846 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 23/313 -- train_loss: 1.0021 2021-06-20 23:49:22,087 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 24/313 -- train_loss: 1.0036 2021-06-20 23:49:22,381 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 25/313 -- train_loss: 0.9769 2021-06-20 23:49:22,620 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 26/313 -- train_loss: 0.9993 2021-06-20 23:49:22,902 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 27/313 -- train_loss: 1.0040 2021-06-20 23:49:23,161 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 28/313 -- train_loss: 1.0100 2021-06-20 23:49:23,401 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 29/313 -- train_loss: 1.0085 2021-06-20 23:49:23,643 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 30/313 -- train_loss: 0.9494 2021-06-20 23:49:23,884 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 31/313 -- train_loss: 1.0030 2021-06-20 23:49:24,124 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 32/313 -- train_loss: 0.9796 2021-06-20 23:49:24,484 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 33/313 -- train_loss: 0.9866 2021-06-20 23:49:24,724 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 34/313 -- train_loss: 0.9811 2021-06-20 23:49:24,965 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 35/313 -- train_loss: 0.9236 2021-06-20 23:49:25,205 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 36/313 -- train_loss: 0.9639 2021-06-20 23:49:25,451 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 37/313 -- train_loss: 0.9399 2021-06-20 23:49:25,692 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 38/313 -- train_loss: 0.9578 2021-06-20 23:49:25,951 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 39/313 -- train_loss: 0.9253 2021-06-20 23:49:26,190 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 40/313 -- train_loss: 0.9513 2021-06-20 23:49:26,484 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 41/313 -- train_loss: 0.9410 2021-06-20 23:49:26,727 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 42/313 -- train_loss: 0.9100 2021-06-20 23:49:27,017 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 43/313 -- train_loss: 0.9253 2021-06-20 23:49:27,275 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 44/313 -- train_loss: 0.9187 2021-06-20 23:49:27,516 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 45/313 -- train_loss: 0.9652 2021-06-20 23:49:27,757 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 46/313 -- train_loss: 0.9201 2021-06-20 23:49:27,999 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 47/313 -- train_loss: 0.9332 2021-06-20 23:49:28,243 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 48/313 -- train_loss: 0.9424 2021-06-20 23:49:28,589 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 49/313 -- train_loss: 0.9147 2021-06-20 23:49:28,830 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 50/313 -- train_loss: 0.9144 2021-06-20 23:49:29,071 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 51/313 -- train_loss: 0.8789 2021-06-20 23:49:29,310 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 52/313 -- train_loss: 0.9302 2021-06-20 23:49:29,550 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 53/313 -- train_loss: 0.8790 2021-06-20 23:49:29,790 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 54/313 -- train_loss: 0.9032 2021-06-20 23:49:30,050 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 55/313 -- train_loss: 0.9228 2021-06-20 23:49:30,291 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 56/313 -- train_loss: 0.9241 2021-06-20 23:49:30,606 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 57/313 -- train_loss: 0.9027 2021-06-20 23:49:30,845 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 58/313 -- train_loss: 0.9155 2021-06-20 23:49:31,147 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 59/313 -- train_loss: 0.8755 2021-06-20 23:49:31,407 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 60/313 -- train_loss: 0.9060 2021-06-20 23:49:31,648 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 61/313 -- train_loss: 0.8769 2021-06-20 23:49:31,889 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 62/313 -- train_loss: 0.9095 2021-06-20 23:49:32,136 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 63/313 -- train_loss: 0.8883 2021-06-20 23:49:32,376 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 64/313 -- train_loss: 0.8939 2021-06-20 23:49:32,728 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 65/313 -- train_loss: 0.8480 2021-06-20 23:49:32,969 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 66/313 -- train_loss: 0.9196 2021-06-20 23:49:33,210 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 67/313 -- train_loss: 0.8792 2021-06-20 23:49:33,451 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 68/313 -- train_loss: 0.8584 2021-06-20 23:49:33,692 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 69/313 -- train_loss: 0.8783 2021-06-20 23:49:33,932 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 70/313 -- train_loss: 0.8902 2021-06-20 23:49:34,191 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 71/313 -- train_loss: 0.8980 2021-06-20 23:49:34,432 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 72/313 -- train_loss: 0.8774 2021-06-20 23:49:34,732 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 73/313 -- train_loss: 0.8424 2021-06-20 23:49:34,973 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 74/313 -- train_loss: 0.8734 2021-06-20 23:49:35,271 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 75/313 -- train_loss: 0.8704 2021-06-20 23:49:35,530 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 76/313 -- train_loss: 0.8459 2021-06-20 23:49:35,770 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 77/313 -- train_loss: 0.8890 2021-06-20 23:49:36,011 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 78/313 -- train_loss: 0.8047 2021-06-20 23:49:36,252 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 79/313 -- train_loss: 0.8970 2021-06-20 23:49:36,493 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 80/313 -- train_loss: 0.8309 2021-06-20 23:49:36,831 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 81/313 -- train_loss: 0.7640 2021-06-20 23:49:37,070 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 82/313 -- train_loss: 0.9029 2021-06-20 23:49:37,310 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 83/313 -- train_loss: 0.8576 2021-06-20 23:49:37,551 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 84/313 -- train_loss: 0.7710 2021-06-20 23:49:37,792 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 85/313 -- train_loss: 0.8116 2021-06-20 23:49:38,034 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 86/313 -- train_loss: 0.8667 2021-06-20 23:49:38,295 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 87/313 -- train_loss: 0.8189 2021-06-20 23:49:38,535 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 88/313 -- train_loss: 0.7889 2021-06-20 23:49:38,841 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 89/313 -- train_loss: 0.7700 2021-06-20 23:49:39,087 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 90/313 -- train_loss: 0.8792 2021-06-20 23:49:39,377 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 91/313 -- train_loss: 0.8112 2021-06-20 23:49:39,630 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 92/313 -- train_loss: 0.7362 2021-06-20 23:49:39,869 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 93/313 -- train_loss: 0.7958 2021-06-20 23:49:40,109 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 94/313 -- train_loss: 0.7525 2021-06-20 23:49:40,350 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 95/313 -- train_loss: 0.8110 2021-06-20 23:49:40,609 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 96/313 -- train_loss: 0.7573 2021-06-20 23:49:40,974 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 97/313 -- train_loss: 0.8979 2021-06-20 23:49:41,215 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 98/313 -- train_loss: 0.8434 2021-06-20 23:49:41,455 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 99/313 -- train_loss: 0.8421 2021-06-20 23:49:41,695 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 100/313 -- train_loss: 0.7432 2021-06-20 23:49:41,936 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 101/313 -- train_loss: 0.7535 2021-06-20 23:49:42,176 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 102/313 -- train_loss: 0.8247 2021-06-20 23:49:42,436 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 103/313 -- train_loss: 0.7098 2021-06-20 23:49:42,677 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 104/313 -- train_loss: 0.7469 2021-06-20 23:49:42,991 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 105/313 -- train_loss: 0.7930 2021-06-20 23:49:43,232 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 106/313 -- train_loss: 0.7947 2021-06-20 23:49:43,528 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 107/313 -- train_loss: 0.7860 2021-06-20 23:49:43,794 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 108/313 -- train_loss: 0.7777 2021-06-20 23:49:44,035 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 109/313 -- train_loss: 0.8433 2021-06-20 23:49:44,281 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 110/313 -- train_loss: 0.9248 2021-06-20 23:49:44,522 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 111/313 -- train_loss: 0.8201 2021-06-20 23:49:44,762 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 112/313 -- train_loss: 0.7065 2021-06-20 23:49:45,110 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 113/313 -- train_loss: 0.7891 2021-06-20 23:49:45,353 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 114/313 -- train_loss: 0.8051 2021-06-20 23:49:45,610 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 115/313 -- train_loss: 0.8107 2021-06-20 23:49:45,850 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 116/313 -- train_loss: 0.6782 2021-06-20 23:49:46,090 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 117/313 -- train_loss: 0.7186 2021-06-20 23:49:46,332 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 118/313 -- train_loss: 0.7307 2021-06-20 23:49:46,586 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 119/313 -- train_loss: 0.8198 2021-06-20 23:49:46,828 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 120/313 -- train_loss: 0.6721 2021-06-20 23:49:47,141 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 121/313 -- train_loss: 0.8145 2021-06-20 23:49:47,382 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 122/313 -- train_loss: 0.7917 2021-06-20 23:49:47,678 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 123/313 -- train_loss: 0.7335 2021-06-20 23:49:47,941 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 124/313 -- train_loss: 0.8007 2021-06-20 23:49:48,180 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 125/313 -- train_loss: 0.7919 2021-06-20 23:49:48,425 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 126/313 -- train_loss: 0.7502 2021-06-20 23:49:48,665 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 127/313 -- train_loss: 0.7977 2021-06-20 23:49:48,906 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 128/313 -- train_loss: 0.7800 2021-06-20 23:49:49,246 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 129/313 -- train_loss: 0.7629 2021-06-20 23:49:49,486 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 130/313 -- train_loss: 0.7109 2021-06-20 23:49:49,732 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 131/313 -- train_loss: 0.8303 2021-06-20 23:49:49,971 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 132/313 -- train_loss: 0.7919 2021-06-20 23:49:50,210 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 133/313 -- train_loss: 0.8328 2021-06-20 23:49:50,451 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 134/313 -- train_loss: 0.7374 2021-06-20 23:49:50,729 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 135/313 -- train_loss: 0.7865 2021-06-20 23:49:50,970 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 136/313 -- train_loss: 0.7526 2021-06-20 23:49:51,285 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 137/313 -- train_loss: 0.7176 2021-06-20 23:49:51,525 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 138/313 -- train_loss: 0.7804 2021-06-20 23:49:51,812 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 139/313 -- train_loss: 0.8158 2021-06-20 23:49:52,074 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 140/313 -- train_loss: 0.8804 2021-06-20 23:49:52,314 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 141/313 -- train_loss: 0.8244 2021-06-20 23:49:52,558 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 142/313 -- train_loss: 0.7051 2021-06-20 23:49:52,799 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 143/313 -- train_loss: 0.8198 2021-06-20 23:49:53,041 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 144/313 -- train_loss: 0.7566 2021-06-20 23:49:53,413 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 145/313 -- train_loss: 0.7014 2021-06-20 23:49:53,653 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 146/313 -- train_loss: 0.8323 2021-06-20 23:49:53,894 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 147/313 -- train_loss: 0.8162 2021-06-20 23:49:54,139 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 148/313 -- train_loss: 0.7680 2021-06-20 23:49:54,380 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 149/313 -- train_loss: 0.7733 2021-06-20 23:49:54,621 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 150/313 -- train_loss: 0.7740 2021-06-20 23:49:54,874 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 151/313 -- train_loss: 0.7740 2021-06-20 23:49:55,114 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 152/313 -- train_loss: 0.7790 2021-06-20 23:49:55,427 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 153/313 -- train_loss: 0.7903 2021-06-20 23:49:55,667 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 154/313 -- train_loss: 0.6858 2021-06-20 23:49:55,958 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 155/313 -- train_loss: 0.7772 2021-06-20 23:49:56,219 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 156/313 -- train_loss: 0.7624 2021-06-20 23:49:56,462 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 157/313 -- train_loss: 0.8525 2021-06-20 23:49:56,703 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 158/313 -- train_loss: 0.6428 2021-06-20 23:49:56,944 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 159/313 -- train_loss: 0.8446 2021-06-20 23:49:57,191 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 160/313 -- train_loss: 0.7681 2021-06-20 23:49:57,529 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 161/313 -- train_loss: 0.7623 2021-06-20 23:49:57,769 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 162/313 -- train_loss: 0.6079 2021-06-20 23:49:58,009 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 163/313 -- train_loss: 0.6997 2021-06-20 23:49:58,250 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 164/313 -- train_loss: 0.7765 2021-06-20 23:49:58,495 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 165/313 -- train_loss: 0.7228 2021-06-20 23:49:58,736 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 166/313 -- train_loss: 0.7778 2021-06-20 23:49:58,991 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 167/313 -- train_loss: 0.8513 2021-06-20 23:49:59,231 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 168/313 -- train_loss: 0.8140 2021-06-20 23:49:59,533 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 169/313 -- train_loss: 0.7934 2021-06-20 23:49:59,773 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 170/313 -- train_loss: 0.8066 2021-06-20 23:50:00,067 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 171/313 -- train_loss: 0.8091 2021-06-20 23:50:00,327 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 172/313 -- train_loss: 0.7078 2021-06-20 23:50:00,567 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 173/313 -- train_loss: 0.7134 2021-06-20 23:50:00,825 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 174/313 -- train_loss: 0.6757 2021-06-20 23:50:01,066 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 175/313 -- train_loss: 0.7385 2021-06-20 23:50:01,307 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 176/313 -- train_loss: 0.7530 2021-06-20 23:50:01,660 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 177/313 -- train_loss: 0.8590 2021-06-20 23:50:01,900 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 178/313 -- train_loss: 0.8074 2021-06-20 23:50:02,145 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 179/313 -- train_loss: 0.6993 2021-06-20 23:50:02,387 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 180/313 -- train_loss: 0.7372 2021-06-20 23:50:02,633 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 181/313 -- train_loss: 0.7514 2021-06-20 23:50:02,873 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 182/313 -- train_loss: 0.7455 2021-06-20 23:50:03,127 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 183/313 -- train_loss: 0.6650 2021-06-20 23:50:03,367 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 184/313 -- train_loss: 0.7073 2021-06-20 23:50:03,688 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 185/313 -- train_loss: 0.7518 2021-06-20 23:50:03,927 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 186/313 -- train_loss: 0.7663 2021-06-20 23:50:04,208 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 187/313 -- train_loss: 0.7389 2021-06-20 23:50:04,461 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 188/313 -- train_loss: 0.7399 2021-06-20 23:50:04,701 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 189/313 -- train_loss: 0.7190 2021-06-20 23:50:04,941 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 190/313 -- train_loss: 0.7617 2021-06-20 23:50:05,181 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 191/313 -- train_loss: 0.7087 2021-06-20 23:50:05,422 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 192/313 -- train_loss: 0.6417 2021-06-20 23:50:05,786 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 193/313 -- train_loss: 0.7620 2021-06-20 23:50:06,030 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 194/313 -- train_loss: 0.8182 2021-06-20 23:50:06,271 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 195/313 -- train_loss: 0.7058 2021-06-20 23:50:06,512 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 196/313 -- train_loss: 0.7097 2021-06-20 23:50:06,753 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 197/313 -- train_loss: 0.6028 2021-06-20 23:50:06,999 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 198/313 -- train_loss: 0.7452 2021-06-20 23:50:07,264 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 199/313 -- train_loss: 0.8088 2021-06-20 23:50:07,505 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 200/313 -- train_loss: 0.7102 2021-06-20 23:50:07,798 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 201/313 -- train_loss: 0.7996 2021-06-20 23:50:08,039 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 202/313 -- train_loss: 0.7707 2021-06-20 23:50:08,326 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 203/313 -- train_loss: 0.7500 2021-06-20 23:50:08,587 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 204/313 -- train_loss: 0.6500 2021-06-20 23:50:08,827 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 205/313 -- train_loss: 0.7041 2021-06-20 23:50:09,068 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 206/313 -- train_loss: 0.8051 2021-06-20 23:50:09,309 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 207/313 -- train_loss: 0.7670 2021-06-20 23:50:09,550 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 208/313 -- train_loss: 0.8052 2021-06-20 23:50:09,919 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 209/313 -- train_loss: 0.6873 2021-06-20 23:50:10,160 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 210/313 -- train_loss: 0.7941 2021-06-20 23:50:10,401 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 211/313 -- train_loss: 0.6602 2021-06-20 23:50:10,642 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 212/313 -- train_loss: 0.7372 2021-06-20 23:50:10,885 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 213/313 -- train_loss: 0.7065 2021-06-20 23:50:11,142 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 214/313 -- train_loss: 0.6208 2021-06-20 23:50:11,401 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 215/313 -- train_loss: 0.7694 2021-06-20 23:50:11,642 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 216/313 -- train_loss: 0.7622 2021-06-20 23:50:11,935 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 217/313 -- train_loss: 0.8282 2021-06-20 23:50:12,175 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 218/313 -- train_loss: 0.7202 2021-06-20 23:50:12,461 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 219/313 -- train_loss: 0.7345 2021-06-20 23:50:12,710 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 220/313 -- train_loss: 0.6691 2021-06-20 23:50:12,952 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 221/313 -- train_loss: 0.7454 2021-06-20 23:50:13,191 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 222/313 -- train_loss: 0.7665 2021-06-20 23:50:13,432 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 223/313 -- train_loss: 0.6608 2021-06-20 23:50:13,673 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 224/313 -- train_loss: 0.6976 2021-06-20 23:50:14,025 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 225/313 -- train_loss: 0.7260 2021-06-20 23:50:14,265 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 226/313 -- train_loss: 0.7538 2021-06-20 23:50:14,505 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 227/313 -- train_loss: 0.6425 2021-06-20 23:50:14,746 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 228/313 -- train_loss: 0.7126 2021-06-20 23:50:14,987 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 229/313 -- train_loss: 0.6885 2021-06-20 23:50:15,228 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 230/313 -- train_loss: 0.7355 2021-06-20 23:50:15,481 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 231/313 -- train_loss: 0.6790 2021-06-20 23:50:15,721 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 232/313 -- train_loss: 0.7376 2021-06-20 23:50:16,024 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 233/313 -- train_loss: 0.7937 2021-06-20 23:50:16,283 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 234/313 -- train_loss: 0.5432 2021-06-20 23:50:16,577 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 235/313 -- train_loss: 0.7099 2021-06-20 23:50:16,836 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 236/313 -- train_loss: 0.6993 2021-06-20 23:50:17,077 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 237/313 -- train_loss: 0.7537 2021-06-20 23:50:17,318 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 238/313 -- train_loss: 0.7819 2021-06-20 23:50:17,559 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 239/313 -- train_loss: 0.6909 2021-06-20 23:50:17,800 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 240/313 -- train_loss: 0.7717 2021-06-20 23:50:18,148 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 241/313 -- train_loss: 0.6432 2021-06-20 23:50:18,389 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 242/313 -- train_loss: 0.7190 2021-06-20 23:50:18,629 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 243/313 -- train_loss: 0.8586 2021-06-20 23:50:18,869 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 244/313 -- train_loss: 0.6890 2021-06-20 23:50:19,109 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 245/313 -- train_loss: 0.7767 2021-06-20 23:50:19,349 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 246/313 -- train_loss: 0.7520 2021-06-20 23:50:19,601 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 247/313 -- train_loss: 0.6770 2021-06-20 23:50:19,841 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 248/313 -- train_loss: 0.7017 2021-06-20 23:50:20,132 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 249/313 -- train_loss: 0.6503 2021-06-20 23:50:20,371 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 250/313 -- train_loss: 0.6745 2021-06-20 23:50:20,667 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 251/313 -- train_loss: 0.8094 2021-06-20 23:50:20,920 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 252/313 -- train_loss: 0.7624 2021-06-20 23:50:21,159 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 253/313 -- train_loss: 0.7290 2021-06-20 23:50:21,417 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 254/313 -- train_loss: 0.6824 2021-06-20 23:50:21,659 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 255/313 -- train_loss: 0.6418 2021-06-20 23:50:21,904 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 256/313 -- train_loss: 0.7173 2021-06-20 23:50:22,254 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 257/313 -- train_loss: 0.6799 2021-06-20 23:50:22,494 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 258/313 -- train_loss: 0.7523 2021-06-20 23:50:22,738 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 259/313 -- train_loss: 0.7568 2021-06-20 23:50:22,979 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 260/313 -- train_loss: 0.6326 2021-06-20 23:50:23,220 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 261/313 -- train_loss: 0.6318 2021-06-20 23:50:23,461 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 262/313 -- train_loss: 0.7211 2021-06-20 23:50:23,727 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 263/313 -- train_loss: 0.7891 2021-06-20 23:50:23,968 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 264/313 -- train_loss: 0.6909 2021-06-20 23:50:24,273 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 265/313 -- train_loss: 0.6614 2021-06-20 23:50:24,519 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 266/313 -- train_loss: 0.6207 2021-06-20 23:50:24,813 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 267/313 -- train_loss: 0.8070 2021-06-20 23:50:25,072 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 268/313 -- train_loss: 0.7395 2021-06-20 23:50:25,312 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 269/313 -- train_loss: 0.7063 2021-06-20 23:50:25,553 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 270/313 -- train_loss: 0.7815 2021-06-20 23:50:25,794 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 271/313 -- train_loss: 0.7001 2021-06-20 23:50:26,035 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 272/313 -- train_loss: 0.6941 2021-06-20 23:50:26,373 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 273/313 -- train_loss: 0.6538 2021-06-20 23:50:26,627 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 274/313 -- train_loss: 0.7709 2021-06-20 23:50:26,868 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 275/313 -- train_loss: 0.7832 2021-06-20 23:50:27,108 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 276/313 -- train_loss: 0.7774 2021-06-20 23:50:27,350 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 277/313 -- train_loss: 0.6173 2021-06-20 23:50:27,590 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 278/313 -- train_loss: 0.7499 2021-06-20 23:50:27,844 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 279/313 -- train_loss: 0.6478 2021-06-20 23:50:28,085 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 280/313 -- train_loss: 0.7890 2021-06-20 23:50:28,392 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 281/313 -- train_loss: 0.6836 2021-06-20 23:50:28,633 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 282/313 -- train_loss: 0.6763 2021-06-20 23:50:28,926 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 283/313 -- train_loss: 0.5574 2021-06-20 23:50:29,189 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 284/313 -- train_loss: 0.7914 2021-06-20 23:50:29,430 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 285/313 -- train_loss: 0.7275 2021-06-20 23:50:29,673 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 286/313 -- train_loss: 0.7727 2021-06-20 23:50:29,914 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 287/313 -- train_loss: 0.7021 2021-06-20 23:50:30,155 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 288/313 -- train_loss: 0.7667 2021-06-20 23:50:30,510 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 289/313 -- train_loss: 0.6334 2021-06-20 23:50:30,750 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 290/313 -- train_loss: 0.7873 2021-06-20 23:50:30,990 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 291/313 -- train_loss: 0.6399 2021-06-20 23:50:31,231 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 292/313 -- train_loss: 0.6857 2021-06-20 23:50:31,471 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 293/313 -- train_loss: 0.6707 2021-06-20 23:50:31,729 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 294/313 -- train_loss: 0.7646 2021-06-20 23:50:31,980 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 295/313 -- train_loss: 0.7645 2021-06-20 23:50:32,221 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 296/313 -- train_loss: 0.5945 2021-06-20 23:50:32,515 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 297/313 -- train_loss: 0.7412 2021-06-20 23:50:32,759 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 298/313 -- train_loss: 0.7768 2021-06-20 23:50:33,057 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 299/313 -- train_loss: 0.7522 2021-06-20 23:50:33,309 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 300/313 -- train_loss: 0.6580 2021-06-20 23:50:33,549 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 301/313 -- train_loss: 0.8013 2021-06-20 23:50:33,789 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 302/313 -- train_loss: 0.7442 2021-06-20 23:50:34,030 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 303/313 -- train_loss: 0.6359 2021-06-20 23:50:34,271 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 304/313 -- train_loss: 0.6693 2021-06-20 23:50:34,626 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 305/313 -- train_loss: 0.7133 2021-06-20 23:50:34,866 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 306/313 -- train_loss: 0.7574 2021-06-20 23:50:35,106 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 307/313 -- train_loss: 0.7888 2021-06-20 23:50:35,347 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 308/313 -- train_loss: 0.6895 2021-06-20 23:50:35,588 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 309/313 -- train_loss: 0.7158 2021-06-20 23:50:35,827 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 310/313 -- train_loss: 0.7793 2021-06-20 23:50:36,081 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 311/313 -- train_loss: 0.7754 2021-06-20 23:50:36,321 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 312/313 -- train_loss: 0.6393 2021-06-20 23:50:36,523 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/1, Iter: 313/313 -- train_loss: 0.7116 2021-06-20 23:50:36,523 - ignite.engine.engine.SupervisedEvaluator - INFO - Engine run resuming from iteration 0, epoch 0 until 1 epochs 2021-06-20 23:58:57,602 - ignite.engine.engine.SupervisedEvaluator - INFO - Got new best metric of Monai's validation mean dice loss: 0.18583615124225616 2021-06-20 23:58:57,602 - ignite.engine.engine.SupervisedEvaluator - INFO - Epoch[1] Metrics -- Monai's validation mean dice loss: 0.1858 My own validation mean dice loss: 0.7905 2021-06-20 23:58:57,602 - ignite.engine.engine.SupervisedEvaluator - INFO - Key metric: Monai's validation mean dice loss best value: 0.18583615124225616 at epoch: 1 2021-06-20 23:58:57,665 - ignite.engine.engine.SupervisedEvaluator - INFO - Epoch[1] Complete. Time taken: 00:08:21 2021-06-20 23:58:57,665 - ignite.engine.engine.SupervisedEvaluator - INFO - Engine run complete. Time taken: 00:08:21 2021-06-20 23:58:57,718 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch[1] Complete. Time taken: 00:09:42 2021-06-20 23:58:57,869 - ignite.engine.engine.SupervisedTrainer - INFO - Train completed, saved final checkpoint: checkpoint_final_iteration=313.pt 2021-06-20 23:58:57,869 - ignite.engine.engine.SupervisedTrainer - INFO - Engine run complete. Time taken: 00:09:43 Send model to server. 2021-06-20 23:58:57,969 - FederatedClient - INFO - Starting to push model. 2021-06-20 23:58:57,970 - Communicator - INFO - Send lesion_activity at round 0 2021-06-20 23:58:58,033 - Communicator - INFO - Received comments: Received round 0 from org1-a (715 Bytes, 1223 seconds). SubmitUpdate time: 0.0630192756652832 seconds 2021-06-20 23:58:58,035 - ignite.engine.engine.SupervisedTrainer - INFO - Terminate signaled. Engine will stop after current iteration is finished. 2021-06-20 23:58:58,035 - ignite.engine.engine.SupervisedEvaluator - INFO - Terminate signaled. Engine will stop after current iteration is finished. 2021-06-20 23:59:00,907 - ProcessExecutor - INFO - process finished with return code 0 cross_validation child process ID: 3908 starting the client ..... token is: 51876d00-e779-4b2a-a8e4-98199da1819f run_number is: 711 uid: org1-a 2021-06-20 23:59:01,532 - matplotlib - WARNING - Matplotlib created a temporary config/cache directory at /tmp/matplotlib-vm91u6p1 because the default path (/home/siddharth/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing. 2021-06-20 23:59:01,559 - matplotlib.font_manager - INFO - Generating new fontManager, this may take some time... 2021-06-20 23:59:03,020 - CrossSiteValManager - INFO - Cross validation timeout is set to 10 minutes. 2021-06-20 23:59:09,317 - Communicator - INFO - Server reply to SubmitBestLocalModel: Received best model from org1-a.. SubmitBestLocalModel time: 6.236904144287109 seconds 2021-06-20 23:59:09,337 - FederatedClient - INFO - Getting other models from server for cross validation. 2021-06-20 23:59:16,251 - Communicator - INFO - Received 1 models for validation. GetValidationModels time: 6.9129273891448975 seconds 2021-06-20 23:59:16,626 - torch.distributed.nn.jit.instantiator - INFO - Created a temporary directory at /tmp/tmpkj3lcqzz 2021-06-20 23:59:16,626 - torch.distributed.nn.jit.instantiator - INFO - Writing /tmp/tmpkj3lcqzz/_remote_module_non_sriptable.py ========== Validate Config Result =========== Use GPU: True Multi GPU: False Automatic Mixed Precision: Enabled Determinism Evaluation: Disabled cuDNN BenchMark: False CUDA Matmul Allow TF32: True cuDNN Allow TF32: True Model: Dataset: DataLoader: Validate Transform #1: Validate Transform #2: Validate Transform #3: Validate Handler #1: Validate Handler #2: Validate Handler #3: Validate Post Transforms #1: Validate Post Transforms #2: Validate Inferer: Validate Key Metric: Validate Additional Metric #My own validation mean dice loss: ========== End of Validate Config Result =========== 2021-06-20 23:59:18,753 - ignite.engine.engine.SupervisedEvaluator - INFO - Engine run resuming from iteration 0, epoch 0 until 1 epochs 2021-06-20 23:59:18,775 - ignite.engine.engine.SupervisedEvaluator - INFO - Restored all variables from /tmp/tmp0c67bru3/mmar/models/local_best_model.PT_MODEL file written: /tmp/tmp0c67bru3/mmar/eval/brain_mask/brain_mask_seg.nii.gz. 2021-06-21 00:01:44,176 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files. file written: /tmp/tmp0c67bru3/mmar/eval/brain_mask/brain_mask_seg.nii.gz. 2021-06-21 00:02:31,701 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files. file written: /tmp/tmp0c67bru3/mmar/eval/brain_mask/brain_mask_seg.nii.gz. 2021-06-21 00:05:07,581 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files. file written: /tmp/tmp0c67bru3/mmar/eval/brain_mask/brain_mask_seg.nii.gz. 2021-06-21 00:07:09,883 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files. file written: /tmp/tmp0c67bru3/mmar/eval/brain_mask/brain_mask_seg.nii.gz. 2021-06-21 00:09:05,278 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files. file written: /tmp/tmp0c67bru3/mmar/eval/brain_mask/brain_mask_seg.nii.gz. 2021-06-21 00:09:09,471 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files. file written: /tmp/tmp0c67bru3/mmar/eval/brain_mask/brain_mask_seg.nii.gz. 2021-06-21 00:09:55,419 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files. file written: /tmp/tmp0c67bru3/mmar/eval/brain_mask/brain_mask_seg.nii.gz. 2021-06-21 00:12:16,633 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files. file written: /tmp/tmp0c67bru3/mmar/eval/brain_mask/brain_mask_seg.nii.gz. 2021-06-21 00:14:58,518 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files. file written: /tmp/tmp0c67bru3/mmar/eval/brain_mask/brain_mask_seg.nii.gz. 2021-06-21 00:18:26,091 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files. file written: /tmp/tmp0c67bru3/mmar/eval/brain_mask/brain_mask_seg.nii.gz. 2021-06-21 00:20:22,511 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files. file written: /tmp/tmp0c67bru3/mmar/eval/brain_mask/brain_mask_seg.nii.gz. 2021-06-21 00:20:42,535 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files. file written: /tmp/tmp0c67bru3/mmar/eval/brain_mask/brain_mask_seg.nii.gz. 2021-06-21 00:20:45,174 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files. file written: /tmp/tmp0c67bru3/mmar/eval/brain_mask/brain_mask_seg.nii.gz. 2021-06-21 00:22:35,529 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files. file written: /tmp/tmp0c67bru3/mmar/eval/brain_mask/brain_mask_seg.nii.gz. 2021-06-21 00:23:05,717 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files. file written: /tmp/tmp0c67bru3/mmar/eval/brain_mask/brain_mask_seg.nii.gz. 2021-06-21 00:23:15,087 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files. file written: /tmp/tmp0c67bru3/mmar/eval/brain_mask/brain_mask_seg.nii.gz. 2021-06-21 00:23:25,358 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files. file written: /tmp/tmp0c67bru3/mmar/eval/brain_mask/brain_mask_seg.nii.gz. 2021-06-21 00:26:20,350 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files. file written: /tmp/tmp0c67bru3/mmar/eval/brain_mask/brain_mask_seg.nii.gz. 2021-06-21 00:27:47,169 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files. file written: /tmp/tmp0c67bru3/mmar/eval/brain_mask/brain_mask_seg.nii.gz. 2021-06-21 00:27:51,366 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files. 2021-06-21 00:27:51,380 - ignite.engine.engine.SupervisedEvaluator - INFO - Got new best metric of Monai's validation mean dice loss: 0.12495744973421097 2021-06-21 00:27:51,380 - ignite.engine.engine.SupervisedEvaluator - INFO - Epoch[1] Metrics -- Monai's validation mean dice loss: 0.1250 My own validation mean dice loss: 0.8689 2021-06-21 00:27:51,380 - ignite.engine.engine.SupervisedEvaluator - INFO - Key metric: Monai's validation mean dice loss best value: 0.12495744973421097 at epoch: 1 2021-06-21 00:27:51,380 - ignite.engine.engine.SupervisedEvaluator - INFO - Epoch[1] Complete. Time taken: 00:28:33 2021-06-21 00:27:51,381 - ignite.engine.engine.SupervisedEvaluator - INFO - Engine run complete. Time taken: 00:28:33 2021-06-21 00:27:51,427 - medl.apps.mmar_conf - INFO - Total Evaluation Time 1712.6733803749084 2021-06-21 00:27:51,434 - ClientValidator - INFO - Do validation ... 2021-06-21 00:27:51,437 - FederatedClient - INFO - Submitting cross validation results to server. 2021-06-21 00:27:51,459 - Communicator - INFO - Received comments: Received Cross Validation results from org1-a.. SubmitCrossSiteValidationResults time: 0.020747900009155273 seconds 2021-06-21 00:27:51,460 - FederatedClient - INFO - More models available with server. 2021-06-21 00:27:51,461 - FederatedClient - INFO - Getting other models from server for cross validation. 2021-06-21 00:27:54,240 - Communicator - INFO - Received 1 models for validation. GetValidationModels time: 2.7776782512664795 seconds ========== Validate Config Result =========== Use GPU: True Multi GPU: False Automatic Mixed Precision: Enabled Determinism Evaluation: Disabled cuDNN BenchMark: False CUDA Matmul Allow TF32: True cuDNN Allow TF32: True Model: Dataset: DataLoader: Validate Transform #1: Validate Transform #2: Validate Transform #3: Validate Handler #1: Validate Handler #2: Validate Handler #3: Validate Post Transforms #1: Validate Post Transforms #2: Validate Inferer: Validate Key Metric: Validate Additional Metric #My own validation mean dice loss: ========== End of Validate Config Result =========== 2021-06-21 00:27:54,441 - ignite.engine.engine.SupervisedEvaluator - INFO - Engine run resuming from iteration 0, epoch 0 until 1 epochs 2021-06-21 00:27:54,448 - ignite.engine.engine.SupervisedEvaluator - ERROR - Engine run is terminating due to exception: Error(s) in loading state_dict for NewLesionsUNet: Missing key(s) in state_dict: "segmenter.0.down.0.conv.weight", "segmenter.0.down.0.conv.bias", "segmenter.0.down.0.res.weight", "segmenter.0.down.0.res.bias", "segmenter.0.down.0.end_seq.1.weight", "segmenter.0.down.0.end_seq.1.bias", "segmenter.0.down.1.conv.weight", "segmenter.0.down.1.conv.bias", "segmenter.0.down.1.res.weight", "segmenter.0.down.1.res.bias", "segmenter.0.down.1.end_seq.1.weight", "segmenter.0.down.1.end_seq.1.bias", "segmenter.0.down.2.conv.weight", "segmenter.0.down.2.conv.bias", "segmenter.0.down.2.res.weight", "segmenter.0.down.2.res.bias", "segmenter.0.down.2.end_seq.1.weight", "segmenter.0.down.2.end_seq.1.bias", "segmenter.0.down.3.conv.weight", "segmenter.0.down.3.conv.bias", "segmenter.0.down.3.res.weight", "segmenter.0.down.3.res.bias", "segmenter.0.down.3.end_seq.1.weight", "segmenter.0.down.3.end_seq.1.bias", "segmenter.0.u.conv.weight", "segmenter.0.u.conv.bias", "segmenter.0.u.res.weight", "segmenter.0.u.res.bias", "segmenter.0.u.end_seq.1.weight", "segmenter.0.u.end_seq.1.bias", "segmenter.0.up.0.conv.weight", "segmenter.0.up.0.conv.bias", "segmenter.0.up.0.res.weight", "segmenter.0.up.0.res.bias", "segmenter.0.up.0.end_seq.1.weight", "segmenter.0.up.0.end_seq.1.bias", "segmenter.0.up.1.conv.weight", "segmenter.0.up.1.conv.bias", "segmenter.0.up.1.res.weight", "segmenter.0.up.1.res.bias", "segmenter.0.up.1.end_seq.1.weight", "segmenter.0.up.1.end_seq.1.bias", "segmenter.0.up.2.conv.weight", "segmenter.0.up.2.conv.bias", "segmenter.0.up.2.res.weight", "segmenter.0.up.2.res.bias", "segmenter.0.up.2.end_seq.1.weight", "segmenter.0.up.2.end_seq.1.bias", "segmenter.0.up.3.conv.weight", "segmenter.0.up.3.conv.bias", "segmenter.0.up.3.res.weight", "segmenter.0.up.3.res.bias", "segmenter.0.up.3.end_seq.1.weight", "segmenter.0.up.3.end_seq.1.bias", "segmenter.1.weight", "segmenter.1.bias". Unexpected key(s) in state_dict: "model.0.conv.unit0.conv.weight", "model.0.conv.unit0.conv.bias", "model.0.conv.unit0.adn.N.weight", "model.0.conv.unit0.adn.N.bias", "model.0.conv.unit0.adn.N.running_mean", "model.0.conv.unit0.adn.N.running_var", "model.0.conv.unit0.adn.N.num_batches_tracked", "model.0.conv.unit0.adn.A.weight", "model.0.conv.unit1.conv.weight", "model.0.conv.unit1.conv.bias", "model.0.conv.unit1.adn.N.weight", "model.0.conv.unit1.adn.N.bias", "model.0.conv.unit1.adn.N.running_mean", "model.0.conv.unit1.adn.N.running_var", "model.0.conv.unit1.adn.N.num_batches_tracked", "model.0.conv.unit1.adn.A.weight", "model.0.residual.weight", "model.0.residual.bias", "model.1.submodule.0.conv.unit0.conv.weight", "model.1.submodule.0.conv.unit0.conv.bias", "model.1.submodule.0.conv.unit0.adn.N.weight", "model.1.submodule.0.conv.unit0.adn.N.bias", "model.1.submodule.0.conv.unit0.adn.N.running_mean", "model.1.submodule.0.conv.unit0.adn.N.running_var", "model.1.submodule.0.conv.unit0.adn.N.num_batches_tracked", "model.1.submodule.0.conv.unit0.adn.A.weight", "model.1.submodule.0.conv.unit1.conv.weight", "model.1.submodule.0.conv.unit1.conv.bias", "model.1.submodule.0.conv.unit1.adn.N.weight", "model.1.submodule.0.conv.unit1.adn.N.bias", "model.1.submodule.0.conv.unit1.adn.N.running_mean", "model.1.submodule.0.conv.unit1.adn.N.running_var", "model.1.submodule.0.conv.unit1.adn.N.num_batches_tracked", "model.1.submodule.0.conv.unit1.adn.A.weight", "model.1.submodule.0.residual.weight", "model.1.submodule.0.residual.bias", "model.1.submodule.1.submodule.0.conv.unit0.conv.weight", "model.1.submodule.1.submodule.0.conv.unit0.conv.bias", "model.1.submodule.1.submodule.0.conv.unit0.adn.N.weight", "model.1.submodule.1.submodule.0.conv.unit0.adn.N.bias", "model.1.submodule.1.submodule.0.conv.unit0.adn.N.running_mean", "model.1.submodule.1.submodule.0.conv.unit0.adn.N.running_var", "model.1.submodule.1.submodule.0.conv.unit0.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.0.conv.unit0.adn.A.weight", "model.1.submodule.1.submodule.0.conv.unit1.conv.weight", "model.1.submodule.1.submodule.0.conv.unit1.conv.bias", "model.1.submodule.1.submodule.0.conv.unit1.adn.N.weight", "model.1.submodule.1.submodule.0.conv.unit1.adn.N.bias", "model.1.submodule.1.submodule.0.conv.unit1.adn.N.running_mean", "model.1.submodule.1.submodule.0.conv.unit1.adn.N.running_var", "model.1.submodule.1.submodule.0.conv.unit1.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.0.conv.unit1.adn.A.weight", "model.1.submodule.1.submodule.0.residual.weight", "model.1.submodule.1.submodule.0.residual.bias", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.conv.weight", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.conv.bias", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.adn.N.weight", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.adn.N.bias", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.adn.N.running_mean", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.adn.N.running_var", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.adn.A.weight", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.conv.weight", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.conv.bias", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.adn.N.weight", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.adn.N.bias", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.adn.N.running_mean", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.adn.N.running_var", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.adn.A.weight", "model.1.submodule.1.submodule.1.submodule.0.residual.weight", "model.1.submodule.1.submodule.1.submodule.0.residual.bias", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.conv.weight", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.conv.bias", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.adn.N.weight", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.adn.N.bias", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.adn.N.running_mean", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.adn.N.running_var", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.adn.A.weight", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.conv.weight", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.conv.bias", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.adn.N.weight", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.adn.N.bias", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.adn.N.running_mean", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.adn.N.running_var", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.adn.A.weight", "model.1.submodule.1.submodule.1.submodule.1.submodule.residual.weight", "model.1.submodule.1.submodule.1.submodule.1.submodule.residual.bias", "model.1.submodule.1.submodule.1.submodule.2.0.conv.weight", "model.1.submodule.1.submodule.1.submodule.2.0.conv.bias", "model.1.submodule.1.submodule.1.submodule.2.0.adn.N.weight", "model.1.submodule.1.submodule.1.submodule.2.0.adn.N.bias", "model.1.submodule.1.submodule.1.submodule.2.0.adn.N.running_mean", "model.1.submodule.1.submodule.1.submodule.2.0.adn.N.running_var", "model.1.submodule.1.submodule.1.submodule.2.0.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.1.submodule.2.0.adn.A.weight", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.conv.weight", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.conv.bias", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.adn.N.weight", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.adn.N.bias", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.adn.N.running_mean", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.adn.N.running_var", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.adn.A.weight", "model.1.submodule.1.submodule.2.0.conv.weight", "model.1.submodule.1.submodule.2.0.conv.bias", "model.1.submodule.1.submodule.2.0.adn.N.weight", "model.1.submodule.1.submodule.2.0.adn.N.bias", "model.1.submodule.1.submodule.2.0.adn.N.running_mean", "model.1.submodule.1.submodule.2.0.adn.N.running_var", "model.1.submodule.1.submodule.2.0.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.2.0.adn.A.weight", "model.1.submodule.1.submodule.2.1.conv.unit0.conv.weight", "model.1.submodule.1.submodule.2.1.conv.unit0.conv.bias", "model.1.submodule.1.submodule.2.1.conv.unit0.adn.N.weight", "model.1.submodule.1.submodule.2.1.conv.unit0.adn.N.bias", "model.1.submodule.1.submodule.2.1.conv.unit0.adn.N.running_mean", "model.1.submodule.1.submodule.2.1.conv.unit0.adn.N.running_var", "model.1.submodule.1.submodule.2.1.conv.unit0.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.2.1.conv.unit0.adn.A.weight", "model.1.submodule.2.0.conv.weight", "model.1.submodule.2.0.conv.bias", "model.1.submodule.2.0.adn.N.weight", "model.1.submodule.2.0.adn.N.bias", "model.1.submodule.2.0.adn.N.running_mean", "model.1.submodule.2.0.adn.N.running_var", "model.1.submodule.2.0.adn.N.num_batches_tracked", "model.1.submodule.2.0.adn.A.weight", "model.1.submodule.2.1.conv.unit0.conv.weight", "model.1.submodule.2.1.conv.unit0.conv.bias", "model.1.submodule.2.1.conv.unit0.adn.N.weight", "model.1.submodule.2.1.conv.unit0.adn.N.bias", "model.1.submodule.2.1.conv.unit0.adn.N.running_mean", "model.1.submodule.2.1.conv.unit0.adn.N.running_var", "model.1.submodule.2.1.conv.unit0.adn.N.num_batches_tracked", "model.1.submodule.2.1.conv.unit0.adn.A.weight", "model.2.0.conv.weight", "model.2.0.conv.bias", "model.2.0.adn.N.weight", "model.2.0.adn.N.bias", "model.2.0.adn.N.running_mean", "model.2.0.adn.N.running_var", "model.2.0.adn.N.num_batches_tracked", "model.2.0.adn.A.weight", "model.2.1.conv.unit0.conv.weight", "model.2.1.conv.unit0.conv.bias". 2021-06-21 00:27:54,449 - ignite.engine.engine.SupervisedEvaluator - ERROR - Exception: Error(s) in loading state_dict for NewLesionsUNet: Missing key(s) in state_dict: "segmenter.0.down.0.conv.weight", "segmenter.0.down.0.conv.bias", "segmenter.0.down.0.res.weight", "segmenter.0.down.0.res.bias", "segmenter.0.down.0.end_seq.1.weight", "segmenter.0.down.0.end_seq.1.bias", "segmenter.0.down.1.conv.weight", "segmenter.0.down.1.conv.bias", "segmenter.0.down.1.res.weight", "segmenter.0.down.1.res.bias", "segmenter.0.down.1.end_seq.1.weight", "segmenter.0.down.1.end_seq.1.bias", "segmenter.0.down.2.conv.weight", "segmenter.0.down.2.conv.bias", "segmenter.0.down.2.res.weight", "segmenter.0.down.2.res.bias", "segmenter.0.down.2.end_seq.1.weight", "segmenter.0.down.2.end_seq.1.bias", "segmenter.0.down.3.conv.weight", "segmenter.0.down.3.conv.bias", "segmenter.0.down.3.res.weight", "segmenter.0.down.3.res.bias", "segmenter.0.down.3.end_seq.1.weight", "segmenter.0.down.3.end_seq.1.bias", "segmenter.0.u.conv.weight", "segmenter.0.u.conv.bias", "segmenter.0.u.res.weight", "segmenter.0.u.res.bias", "segmenter.0.u.end_seq.1.weight", "segmenter.0.u.end_seq.1.bias", "segmenter.0.up.0.conv.weight", "segmenter.0.up.0.conv.bias", "segmenter.0.up.0.res.weight", "segmenter.0.up.0.res.bias", "segmenter.0.up.0.end_seq.1.weight", "segmenter.0.up.0.end_seq.1.bias", "segmenter.0.up.1.conv.weight", "segmenter.0.up.1.conv.bias", "segmenter.0.up.1.res.weight", "segmenter.0.up.1.res.bias", "segmenter.0.up.1.end_seq.1.weight", "segmenter.0.up.1.end_seq.1.bias", "segmenter.0.up.2.conv.weight", "segmenter.0.up.2.conv.bias", "segmenter.0.up.2.res.weight", "segmenter.0.up.2.res.bias", "segmenter.0.up.2.end_seq.1.weight", "segmenter.0.up.2.end_seq.1.bias", "segmenter.0.up.3.conv.weight", "segmenter.0.up.3.conv.bias", "segmenter.0.up.3.res.weight", "segmenter.0.up.3.res.bias", "segmenter.0.up.3.end_seq.1.weight", "segmenter.0.up.3.end_seq.1.bias", "segmenter.1.weight", "segmenter.1.bias". Unexpected key(s) in state_dict: "model.0.conv.unit0.conv.weight", "model.0.conv.unit0.conv.bias", "model.0.conv.unit0.adn.N.weight", "model.0.conv.unit0.adn.N.bias", "model.0.conv.unit0.adn.N.running_mean", "model.0.conv.unit0.adn.N.running_var", "model.0.conv.unit0.adn.N.num_batches_tracked", "model.0.conv.unit0.adn.A.weight", "model.0.conv.unit1.conv.weight", "model.0.conv.unit1.conv.bias", "model.0.conv.unit1.adn.N.weight", "model.0.conv.unit1.adn.N.bias", "model.0.conv.unit1.adn.N.running_mean", "model.0.conv.unit1.adn.N.running_var", "model.0.conv.unit1.adn.N.num_batches_tracked", "model.0.conv.unit1.adn.A.weight", "model.0.residual.weight", "model.0.residual.bias", "model.1.submodule.0.conv.unit0.conv.weight", "model.1.submodule.0.conv.unit0.conv.bias", "model.1.submodule.0.conv.unit0.adn.N.weight", "model.1.submodule.0.conv.unit0.adn.N.bias", "model.1.submodule.0.conv.unit0.adn.N.running_mean", "model.1.submodule.0.conv.unit0.adn.N.running_var", "model.1.submodule.0.conv.unit0.adn.N.num_batches_tracked", "model.1.submodule.0.conv.unit0.adn.A.weight", "model.1.submodule.0.conv.unit1.conv.weight", "model.1.submodule.0.conv.unit1.conv.bias", "model.1.submodule.0.conv.unit1.adn.N.weight", "model.1.submodule.0.conv.unit1.adn.N.bias", "model.1.submodule.0.conv.unit1.adn.N.running_mean", "model.1.submodule.0.conv.unit1.adn.N.running_var", "model.1.submodule.0.conv.unit1.adn.N.num_batches_tracked", "model.1.submodule.0.conv.unit1.adn.A.weight", "model.1.submodule.0.residual.weight", "model.1.submodule.0.residual.bias", "model.1.submodule.1.submodule.0.conv.unit0.conv.weight", "model.1.submodule.1.submodule.0.conv.unit0.conv.bias", "model.1.submodule.1.submodule.0.conv.unit0.adn.N.weight", "model.1.submodule.1.submodule.0.conv.unit0.adn.N.bias", "model.1.submodule.1.submodule.0.conv.unit0.adn.N.running_mean", "model.1.submodule.1.submodule.0.conv.unit0.adn.N.running_var", "model.1.submodule.1.submodule.0.conv.unit0.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.0.conv.unit0.adn.A.weight", "model.1.submodule.1.submodule.0.conv.unit1.conv.weight", "model.1.submodule.1.submodule.0.conv.unit1.conv.bias", "model.1.submodule.1.submodule.0.conv.unit1.adn.N.weight", "model.1.submodule.1.submodule.0.conv.unit1.adn.N.bias", "model.1.submodule.1.submodule.0.conv.unit1.adn.N.running_mean", "model.1.submodule.1.submodule.0.conv.unit1.adn.N.running_var", "model.1.submodule.1.submodule.0.conv.unit1.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.0.conv.unit1.adn.A.weight", "model.1.submodule.1.submodule.0.residual.weight", "model.1.submodule.1.submodule.0.residual.bias", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.conv.weight", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.conv.bias", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.adn.N.weight", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.adn.N.bias", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.adn.N.running_mean", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.adn.N.running_var", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.adn.A.weight", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.conv.weight", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.conv.bias", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.adn.N.weight", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.adn.N.bias", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.adn.N.running_mean", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.adn.N.running_var", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.adn.A.weight", "model.1.submodule.1.submodule.1.submodule.0.residual.weight", "model.1.submodule.1.submodule.1.submodule.0.residual.bias", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.conv.weight", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.conv.bias", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.adn.N.weight", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.adn.N.bias", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.adn.N.running_mean", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.adn.N.running_var", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.adn.A.weight", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.conv.weight", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.conv.bias", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.adn.N.weight", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.adn.N.bias", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.adn.N.running_mean", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.adn.N.running_var", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.adn.A.weight", "model.1.submodule.1.submodule.1.submodule.1.submodule.residual.weight", "model.1.submodule.1.submodule.1.submodule.1.submodule.residual.bias", "model.1.submodule.1.submodule.1.submodule.2.0.conv.weight", "model.1.submodule.1.submodule.1.submodule.2.0.conv.bias", "model.1.submodule.1.submodule.1.submodule.2.0.adn.N.weight", "model.1.submodule.1.submodule.1.submodule.2.0.adn.N.bias", "model.1.submodule.1.submodule.1.submodule.2.0.adn.N.running_mean", "model.1.submodule.1.submodule.1.submodule.2.0.adn.N.running_var", "model.1.submodule.1.submodule.1.submodule.2.0.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.1.submodule.2.0.adn.A.weight", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.conv.weight", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.conv.bias", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.adn.N.weight", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.adn.N.bias", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.adn.N.running_mean", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.adn.N.running_var", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.adn.A.weight", "model.1.submodule.1.submodule.2.0.conv.weight", "model.1.submodule.1.submodule.2.0.conv.bias", "model.1.submodule.1.submodule.2.0.adn.N.weight", "model.1.submodule.1.submodule.2.0.adn.N.bias", "model.1.submodule.1.submodule.2.0.adn.N.running_mean", "model.1.submodule.1.submodule.2.0.adn.N.running_var", "model.1.submodule.1.submodule.2.0.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.2.0.adn.A.weight", "model.1.submodule.1.submodule.2.1.conv.unit0.conv.weight", "model.1.submodule.1.submodule.2.1.conv.unit0.conv.bias", "model.1.submodule.1.submodule.2.1.conv.unit0.adn.N.weight", "model.1.submodule.1.submodule.2.1.conv.unit0.adn.N.bias", "model.1.submodule.1.submodule.2.1.conv.unit0.adn.N.running_mean", "model.1.submodule.1.submodule.2.1.conv.unit0.adn.N.running_var", "model.1.submodule.1.submodule.2.1.conv.unit0.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.2.1.conv.unit0.adn.A.weight", "model.1.submodule.2.0.conv.weight", "model.1.submodule.2.0.conv.bias", "model.1.submodule.2.0.adn.N.weight", "model.1.submodule.2.0.adn.N.bias", "model.1.submodule.2.0.adn.N.running_mean", "model.1.submodule.2.0.adn.N.running_var", "model.1.submodule.2.0.adn.N.num_batches_tracked", "model.1.submodule.2.0.adn.A.weight", "model.1.submodule.2.1.conv.unit0.conv.weight", "model.1.submodule.2.1.conv.unit0.conv.bias", "model.1.submodule.2.1.conv.unit0.adn.N.weight", "model.1.submodule.2.1.conv.unit0.adn.N.bias", "model.1.submodule.2.1.conv.unit0.adn.N.running_mean", "model.1.submodule.2.1.conv.unit0.adn.N.running_var", "model.1.submodule.2.1.conv.unit0.adn.N.num_batches_tracked", "model.1.submodule.2.1.conv.unit0.adn.A.weight", "model.2.0.conv.weight", "model.2.0.conv.bias", "model.2.0.adn.N.weight", "model.2.0.adn.N.bias", "model.2.0.adn.N.running_mean", "model.2.0.adn.N.running_var", "model.2.0.adn.N.num_batches_tracked", "model.2.0.adn.A.weight", "model.2.1.conv.unit0.conv.weight", "model.2.1.conv.unit0.conv.bias". Traceback (most recent call last): File "/opt/conda/lib/python3.8/site-packages/ignite/engine/engine.py", line 737, in _internal_run self._fire_event(Events.STARTED) File "/opt/conda/lib/python3.8/site-packages/ignite/engine/engine.py", line 424, in _fire_event func(*first, *(event_args + others), **kwargs) File "/opt/monai/monai/handlers/checkpoint_loader.py", line 110, in __call__ Checkpoint.load_objects(to_load=self.load_dict, checkpoint=checkpoint, strict=self.strict) File "/opt/conda/lib/python3.8/site-packages/ignite/handlers/checkpoint.py", line 566, in load_objects obj.load_state_dict(checkpoint[k], strict=is_state_dict_strict) File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1215, in load_state_dict raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format( RuntimeError: Error(s) in loading state_dict for NewLesionsUNet: Missing key(s) in state_dict: "segmenter.0.down.0.conv.weight", "segmenter.0.down.0.conv.bias", "segmenter.0.down.0.res.weight", "segmenter.0.down.0.res.bias", "segmenter.0.down.0.end_seq.1.weight", "segmenter.0.down.0.end_seq.1.bias", "segmenter.0.down.1.conv.weight", "segmenter.0.down.1.conv.bias", "segmenter.0.down.1.res.weight", "segmenter.0.down.1.res.bias", "segmenter.0.down.1.end_seq.1.weight", "segmenter.0.down.1.end_seq.1.bias", "segmenter.0.down.2.conv.weight", "segmenter.0.down.2.conv.bias", "segmenter.0.down.2.res.weight", "segmenter.0.down.2.res.bias", "segmenter.0.down.2.end_seq.1.weight", "segmenter.0.down.2.end_seq.1.bias", "segmenter.0.down.3.conv.weight", "segmenter.0.down.3.conv.bias", "segmenter.0.down.3.res.weight", "segmenter.0.down.3.res.bias", "segmenter.0.down.3.end_seq.1.weight", "segmenter.0.down.3.end_seq.1.bias", "segmenter.0.u.conv.weight", "segmenter.0.u.conv.bias", "segmenter.0.u.res.weight", "segmenter.0.u.res.bias", "segmenter.0.u.end_seq.1.weight", "segmenter.0.u.end_seq.1.bias", "segmenter.0.up.0.conv.weight", "segmenter.0.up.0.conv.bias", "segmenter.0.up.0.res.weight", "segmenter.0.up.0.res.bias", "segmenter.0.up.0.end_seq.1.weight", "segmenter.0.up.0.end_seq.1.bias", "segmenter.0.up.1.conv.weight", "segmenter.0.up.1.conv.bias", "segmenter.0.up.1.res.weight", "segmenter.0.up.1.res.bias", "segmenter.0.up.1.end_seq.1.weight", "segmenter.0.up.1.end_seq.1.bias", "segmenter.0.up.2.conv.weight", "segmenter.0.up.2.conv.bias", "segmenter.0.up.2.res.weight", "segmenter.0.up.2.res.bias", "segmenter.0.up.2.end_seq.1.weight", "segmenter.0.up.2.end_seq.1.bias", "segmenter.0.up.3.conv.weight", "segmenter.0.up.3.conv.bias", "segmenter.0.up.3.res.weight", "segmenter.0.up.3.res.bias", "segmenter.0.up.3.end_seq.1.weight", "segmenter.0.up.3.end_seq.1.bias", "segmenter.1.weight", "segmenter.1.bias". Unexpected key(s) in state_dict: "model.0.conv.unit0.conv.weight", "model.0.conv.unit0.conv.bias", "model.0.conv.unit0.adn.N.weight", "model.0.conv.unit0.adn.N.bias", "model.0.conv.unit0.adn.N.running_mean", "model.0.conv.unit0.adn.N.running_var", "model.0.conv.unit0.adn.N.num_batches_tracked", "model.0.conv.unit0.adn.A.weight", "model.0.conv.unit1.conv.weight", "model.0.conv.unit1.conv.bias", "model.0.conv.unit1.adn.N.weight", "model.0.conv.unit1.adn.N.bias", "model.0.conv.unit1.adn.N.running_mean", "model.0.conv.unit1.adn.N.running_var", "model.0.conv.unit1.adn.N.num_batches_tracked", "model.0.conv.unit1.adn.A.weight", "model.0.residual.weight", "model.0.residual.bias", "model.1.submodule.0.conv.unit0.conv.weight", "model.1.submodule.0.conv.unit0.conv.bias", "model.1.submodule.0.conv.unit0.adn.N.weight", "model.1.submodule.0.conv.unit0.adn.N.bias", "model.1.submodule.0.conv.unit0.adn.N.running_mean", "model.1.submodule.0.conv.unit0.adn.N.running_var", "model.1.submodule.0.conv.unit0.adn.N.num_batches_tracked", "model.1.submodule.0.conv.unit0.adn.A.weight", "model.1.submodule.0.conv.unit1.conv.weight", "model.1.submodule.0.conv.unit1.conv.bias", "model.1.submodule.0.conv.unit1.adn.N.weight", "model.1.submodule.0.conv.unit1.adn.N.bias", "model.1.submodule.0.conv.unit1.adn.N.running_mean", "model.1.submodule.0.conv.unit1.adn.N.running_var", "model.1.submodule.0.conv.unit1.adn.N.num_batches_tracked", "model.1.submodule.0.conv.unit1.adn.A.weight", "model.1.submodule.0.residual.weight", "model.1.submodule.0.residual.bias", "model.1.submodule.1.submodule.0.conv.unit0.conv.weight", "model.1.submodule.1.submodule.0.conv.unit0.conv.bias", "model.1.submodule.1.submodule.0.conv.unit0.adn.N.weight", "model.1.submodule.1.submodule.0.conv.unit0.adn.N.bias", "model.1.submodule.1.submodule.0.conv.unit0.adn.N.running_mean", "model.1.submodule.1.submodule.0.conv.unit0.adn.N.running_var", "model.1.submodule.1.submodule.0.conv.unit0.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.0.conv.unit0.adn.A.weight", "model.1.submodule.1.submodule.0.conv.unit1.conv.weight", "model.1.submodule.1.submodule.0.conv.unit1.conv.bias", "model.1.submodule.1.submodule.0.conv.unit1.adn.N.weight", "model.1.submodule.1.submodule.0.conv.unit1.adn.N.bias", "model.1.submodule.1.submodule.0.conv.unit1.adn.N.running_mean", "model.1.submodule.1.submodule.0.conv.unit1.adn.N.running_var", "model.1.submodule.1.submodule.0.conv.unit1.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.0.conv.unit1.adn.A.weight", "model.1.submodule.1.submodule.0.residual.weight", "model.1.submodule.1.submodule.0.residual.bias", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.conv.weight", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.conv.bias", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.adn.N.weight", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.adn.N.bias", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.adn.N.running_mean", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.adn.N.running_var", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.adn.A.weight", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.conv.weight", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.conv.bias", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.adn.N.weight", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.adn.N.bias", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.adn.N.running_mean", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.adn.N.running_var", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.adn.A.weight", "model.1.submodule.1.submodule.1.submodule.0.residual.weight", "model.1.submodule.1.submodule.1.submodule.0.residual.bias", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.conv.weight", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.conv.bias", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.adn.N.weight", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.adn.N.bias", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.adn.N.running_mean", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.adn.N.running_var", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.adn.A.weight", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.conv.weight", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.conv.bias", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.adn.N.weight", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.adn.N.bias", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.adn.N.running_mean", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.adn.N.running_var", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.adn.A.weight", "model.1.submodule.1.submodule.1.submodule.1.submodule.residual.weight", "model.1.submodule.1.submodule.1.submodule.1.submodule.residual.bias", "model.1.submodule.1.submodule.1.submodule.2.0.conv.weight", "model.1.submodule.1.submodule.1.submodule.2.0.conv.bias", "model.1.submodule.1.submodule.1.submodule.2.0.adn.N.weight", "model.1.submodule.1.submodule.1.submodule.2.0.adn.N.bias", "model.1.submodule.1.submodule.1.submodule.2.0.adn.N.running_mean", "model.1.submodule.1.submodule.1.submodule.2.0.adn.N.running_var", "model.1.submodule.1.submodule.1.submodule.2.0.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.1.submodule.2.0.adn.A.weight", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.conv.weight", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.conv.bias", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.adn.N.weight", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.adn.N.bias", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.adn.N.running_mean", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.adn.N.running_var", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.adn.A.weight", "model.1.submodule.1.submodule.2.0.conv.weight", "model.1.submodule.1.submodule.2.0.conv.bias", "model.1.submodule.1.submodule.2.0.adn.N.weight", "model.1.submodule.1.submodule.2.0.adn.N.bias", "model.1.submodule.1.submodule.2.0.adn.N.running_mean", "model.1.submodule.1.submodule.2.0.adn.N.running_var", "model.1.submodule.1.submodule.2.0.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.2.0.adn.A.weight", "model.1.submodule.1.submodule.2.1.conv.unit0.conv.weight", "model.1.submodule.1.submodule.2.1.conv.unit0.conv.bias", "model.1.submodule.1.submodule.2.1.conv.unit0.adn.N.weight", "model.1.submodule.1.submodule.2.1.conv.unit0.adn.N.bias", "model.1.submodule.1.submodule.2.1.conv.unit0.adn.N.running_mean", "model.1.submodule.1.submodule.2.1.conv.unit0.adn.N.running_var", "model.1.submodule.1.submodule.2.1.conv.unit0.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.2.1.conv.unit0.adn.A.weight", "model.1.submodule.2.0.conv.weight", "model.1.submodule.2.0.conv.bias", "model.1.submodule.2.0.adn.N.weight", "model.1.submodule.2.0.adn.N.bias", "model.1.submodule.2.0.adn.N.running_mean", "model.1.submodule.2.0.adn.N.running_var", "model.1.submodule.2.0.adn.N.num_batches_tracked", "model.1.submodule.2.0.adn.A.weight", "model.1.submodule.2.1.conv.unit0.conv.weight", "model.1.submodule.2.1.conv.unit0.conv.bias", "model.1.submodule.2.1.conv.unit0.adn.N.weight", "model.1.submodule.2.1.conv.unit0.adn.N.bias", "model.1.submodule.2.1.conv.unit0.adn.N.running_mean", "model.1.submodule.2.1.conv.unit0.adn.N.running_var", "model.1.submodule.2.1.conv.unit0.adn.N.num_batches_tracked", "model.1.submodule.2.1.conv.unit0.adn.A.weight", "model.2.0.conv.weight", "model.2.0.conv.bias", "model.2.0.adn.N.weight", "model.2.0.adn.N.bias", "model.2.0.adn.N.running_mean", "model.2.0.adn.N.running_var", "model.2.0.adn.N.num_batches_tracked", "model.2.0.adn.A.weight", "model.2.1.conv.unit0.conv.weight", "model.2.1.conv.unit0.conv.bias". Traceback (most recent call last): File "apps/fed_learn/validators/client_validator.py", line 68, in validate File "apps/mmar_conf.py", line 69, in evaluate_mmar File "/opt/monai/monai/engines/evaluator.py", line 120, in run super().run() File "/opt/monai/monai/engines/workflow.py", line 206, in run super().run(data=self.data_loader, max_epochs=self.state.max_epochs) File "/opt/conda/lib/python3.8/site-packages/ignite/engine/engine.py", line 702, in run return self._internal_run() File "/opt/conda/lib/python3.8/site-packages/ignite/engine/engine.py", line 775, in _internal_run self._handle_exception(e) File "/opt/conda/lib/python3.8/site-packages/ignite/engine/engine.py", line 467, in _handle_exception self._fire_event(Events.EXCEPTION_RAISED, e) File "/opt/conda/lib/python3.8/site-packages/ignite/engine/engine.py", line 424, in _fire_event func(*first, *(event_args + others), **kwargs) File "/opt/monai/monai/handlers/stats_handler.py", line 145, in exception_raised raise e File "/opt/conda/lib/python3.8/site-packages/ignite/engine/engine.py", line 737, in _internal_run self._fire_event(Events.STARTED) File "/opt/conda/lib/python3.8/site-packages/ignite/engine/engine.py", line 424, in _fire_event func(*first, *(event_args + others), **kwargs) File "/opt/monai/monai/handlers/checkpoint_loader.py", line 110, in __call__ Checkpoint.load_objects(to_load=self.load_dict, checkpoint=checkpoint, strict=self.strict) File "/opt/conda/lib/python3.8/site-packages/ignite/handlers/checkpoint.py", line 566, in load_objects obj.load_state_dict(checkpoint[k], strict=is_state_dict_strict) File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1215, in load_state_dict raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format( RuntimeError: Error(s) in loading state_dict for NewLesionsUNet: Missing key(s) in state_dict: "segmenter.0.down.0.conv.weight", "segmenter.0.down.0.conv.bias", "segmenter.0.down.0.res.weight", "segmenter.0.down.0.res.bias", "segmenter.0.down.0.end_seq.1.weight", "segmenter.0.down.0.end_seq.1.bias", "segmenter.0.down.1.conv.weight", "segmenter.0.down.1.conv.bias", "segmenter.0.down.1.res.weight", "segmenter.0.down.1.res.bias", "segmenter.0.down.1.end_seq.1.weight", "segmenter.0.down.1.end_seq.1.bias", "segmenter.0.down.2.conv.weight", "segmenter.0.down.2.conv.bias", "segmenter.0.down.2.res.weight", "segmenter.0.down.2.res.bias", "segmenter.0.down.2.end_seq.1.weight", "segmenter.0.down.2.end_seq.1.bias", "segmenter.0.down.3.conv.weight", "segmenter.0.down.3.conv.bias", "segmenter.0.down.3.res.weight", "segmenter.0.down.3.res.bias", "segmenter.0.down.3.end_seq.1.weight", "segmenter.0.down.3.end_seq.1.bias", "segmenter.0.u.conv.weight", "segmenter.0.u.conv.bias", "segmenter.0.u.res.weight", "segmenter.0.u.res.bias", "segmenter.0.u.end_seq.1.weight", "segmenter.0.u.end_seq.1.bias", "segmenter.0.up.0.conv.weight", "segmenter.0.up.0.conv.bias", "segmenter.0.up.0.res.weight", "segmenter.0.up.0.res.bias", "segmenter.0.up.0.end_seq.1.weight", "segmenter.0.up.0.end_seq.1.bias", "segmenter.0.up.1.conv.weight", "segmenter.0.up.1.conv.bias", "segmenter.0.up.1.res.weight", "segmenter.0.up.1.res.bias", "segmenter.0.up.1.end_seq.1.weight", "segmenter.0.up.1.end_seq.1.bias", "segmenter.0.up.2.conv.weight", "segmenter.0.up.2.conv.bias", "segmenter.0.up.2.res.weight", "segmenter.0.up.2.res.bias", "segmenter.0.up.2.end_seq.1.weight", "segmenter.0.up.2.end_seq.1.bias", "segmenter.0.up.3.conv.weight", "segmenter.0.up.3.conv.bias", "segmenter.0.up.3.res.weight", "segmenter.0.up.3.res.bias", "segmenter.0.up.3.end_seq.1.weight", "segmenter.0.up.3.end_seq.1.bias", "segmenter.1.weight", "segmenter.1.bias". Unexpected key(s) in state_dict: "model.0.conv.unit0.conv.weight", "model.0.conv.unit0.conv.bias", "model.0.conv.unit0.adn.N.weight", "model.0.conv.unit0.adn.N.bias", "model.0.conv.unit0.adn.N.running_mean", "model.0.conv.unit0.adn.N.running_var", "model.0.conv.unit0.adn.N.num_batches_tracked", "model.0.conv.unit0.adn.A.weight", "model.0.conv.unit1.conv.weight", "model.0.conv.unit1.conv.bias", "model.0.conv.unit1.adn.N.weight", "model.0.conv.unit1.adn.N.bias", "model.0.conv.unit1.adn.N.running_mean", "model.0.conv.unit1.adn.N.running_var", "model.0.conv.unit1.adn.N.num_batches_tracked", "model.0.conv.unit1.adn.A.weight", "model.0.residual.weight", "model.0.residual.bias", "model.1.submodule.0.conv.unit0.conv.weight", "model.1.submodule.0.conv.unit0.conv.bias", "model.1.submodule.0.conv.unit0.adn.N.weight", "model.1.submodule.0.conv.unit0.adn.N.bias", "model.1.submodule.0.conv.unit0.adn.N.running_mean", "model.1.submodule.0.conv.unit0.adn.N.running_var", "model.1.submodule.0.conv.unit0.adn.N.num_batches_tracked", "model.1.submodule.0.conv.unit0.adn.A.weight", "model.1.submodule.0.conv.unit1.conv.weight", "model.1.submodule.0.conv.unit1.conv.bias", "model.1.submodule.0.conv.unit1.adn.N.weight", "model.1.submodule.0.conv.unit1.adn.N.bias", "model.1.submodule.0.conv.unit1.adn.N.running_mean", "model.1.submodule.0.conv.unit1.adn.N.running_var", "model.1.submodule.0.conv.unit1.adn.N.num_batches_tracked", "model.1.submodule.0.conv.unit1.adn.A.weight", "model.1.submodule.0.residual.weight", "model.1.submodule.0.residual.bias", "model.1.submodule.1.submodule.0.conv.unit0.conv.weight", "model.1.submodule.1.submodule.0.conv.unit0.conv.bias", "model.1.submodule.1.submodule.0.conv.unit0.adn.N.weight", "model.1.submodule.1.submodule.0.conv.unit0.adn.N.bias", "model.1.submodule.1.submodule.0.conv.unit0.adn.N.running_mean", "model.1.submodule.1.submodule.0.conv.unit0.adn.N.running_var", "model.1.submodule.1.submodule.0.conv.unit0.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.0.conv.unit0.adn.A.weight", "model.1.submodule.1.submodule.0.conv.unit1.conv.weight", "model.1.submodule.1.submodule.0.conv.unit1.conv.bias", "model.1.submodule.1.submodule.0.conv.unit1.adn.N.weight", "model.1.submodule.1.submodule.0.conv.unit1.adn.N.bias", "model.1.submodule.1.submodule.0.conv.unit1.adn.N.running_mean", "model.1.submodule.1.submodule.0.conv.unit1.adn.N.running_var", "model.1.submodule.1.submodule.0.conv.unit1.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.0.conv.unit1.adn.A.weight", "model.1.submodule.1.submodule.0.residual.weight", "model.1.submodule.1.submodule.0.residual.bias", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.conv.weight", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.conv.bias", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.adn.N.weight", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.adn.N.bias", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.adn.N.running_mean", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.adn.N.running_var", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.1.submodule.0.conv.unit0.adn.A.weight", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.conv.weight", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.conv.bias", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.adn.N.weight", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.adn.N.bias", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.adn.N.running_mean", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.adn.N.running_var", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.1.submodule.0.conv.unit1.adn.A.weight", "model.1.submodule.1.submodule.1.submodule.0.residual.weight", "model.1.submodule.1.submodule.1.submodule.0.residual.bias", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.conv.weight", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.conv.bias", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.adn.N.weight", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.adn.N.bias", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.adn.N.running_mean", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.adn.N.running_var", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit0.adn.A.weight", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.conv.weight", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.conv.bias", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.adn.N.weight", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.adn.N.bias", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.adn.N.running_mean", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.adn.N.running_var", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.1.submodule.1.submodule.conv.unit1.adn.A.weight", "model.1.submodule.1.submodule.1.submodule.1.submodule.residual.weight", "model.1.submodule.1.submodule.1.submodule.1.submodule.residual.bias", "model.1.submodule.1.submodule.1.submodule.2.0.conv.weight", "model.1.submodule.1.submodule.1.submodule.2.0.conv.bias", "model.1.submodule.1.submodule.1.submodule.2.0.adn.N.weight", "model.1.submodule.1.submodule.1.submodule.2.0.adn.N.bias", "model.1.submodule.1.submodule.1.submodule.2.0.adn.N.running_mean", "model.1.submodule.1.submodule.1.submodule.2.0.adn.N.running_var", "model.1.submodule.1.submodule.1.submodule.2.0.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.1.submodule.2.0.adn.A.weight", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.conv.weight", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.conv.bias", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.adn.N.weight", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.adn.N.bias", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.adn.N.running_mean", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.adn.N.running_var", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.1.submodule.2.1.conv.unit0.adn.A.weight", "model.1.submodule.1.submodule.2.0.conv.weight", "model.1.submodule.1.submodule.2.0.conv.bias", "model.1.submodule.1.submodule.2.0.adn.N.weight", "model.1.submodule.1.submodule.2.0.adn.N.bias", "model.1.submodule.1.submodule.2.0.adn.N.running_mean", "model.1.submodule.1.submodule.2.0.adn.N.running_var", "model.1.submodule.1.submodule.2.0.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.2.0.adn.A.weight", "model.1.submodule.1.submodule.2.1.conv.unit0.conv.weight", "model.1.submodule.1.submodule.2.1.conv.unit0.conv.bias", "model.1.submodule.1.submodule.2.1.conv.unit0.adn.N.weight", "model.1.submodule.1.submodule.2.1.conv.unit0.adn.N.bias", "model.1.submodule.1.submodule.2.1.conv.unit0.adn.N.running_mean", "model.1.submodule.1.submodule.2.1.conv.unit0.adn.N.running_var", "model.1.submodule.1.submodule.2.1.conv.unit0.adn.N.num_batches_tracked", "model.1.submodule.1.submodule.2.1.conv.unit0.adn.A.weight", "model.1.submodule.2.0.conv.weight", "model.1.submodule.2.0.conv.bias", "model.1.submodule.2.0.adn.N.weight", "model.1.submodule.2.0.adn.N.bias", "model.1.submodule.2.0.adn.N.running_mean", "model.1.submodule.2.0.adn.N.running_var", "model.1.submodule.2.0.adn.N.num_batches_tracked", "model.1.submodule.2.0.adn.A.weight", "model.1.submodule.2.1.conv.unit0.conv.weight", "model.1.submodule.2.1.conv.unit0.conv.bias", "model.1.submodule.2.1.conv.unit0.adn.N.weight", "model.1.submodule.2.1.conv.unit0.adn.N.bias", "model.1.submodule.2.1.conv.unit0.adn.N.running_mean", "model.1.submodule.2.1.conv.unit0.adn.N.running_var", "model.1.submodule.2.1.conv.unit0.adn.N.num_batches_tracked", "model.1.submodule.2.1.conv.unit0.adn.A.weight", "model.2.0.conv.weight", "model.2.0.conv.bias", "model.2.0.adn.N.weight", "model.2.0.adn.N.bias", "model.2.0.adn.N.running_mean", "model.2.0.adn.N.running_var", "model.2.0.adn.N.num_batches_tracked", "model.2.0.adn.A.weight", "model.2.1.conv.unit0.conv.weight", "model.2.1.conv.unit0.conv.bias". Exception in executing validation: 2021-06-21 00:27:54,456 - ClientValidator - INFO - Do validation ... 2021-06-21 00:27:54,456 - CrossSiteValManager - INFO - Exception in validating FL_global_model's model: local variable 'metrics' referenced before assignment 2021-06-21 00:27:54,458 - FederatedClient - INFO - Submitting cross validation results to server. 2021-06-21 00:27:54,474 - Communicator - INFO - Received comments: Received Cross Validation results from org1-a.. SubmitCrossSiteValidationResults time: 0.015302419662475586 seconds 2021-06-21 00:27:54,474 - FederatedClient - INFO - More models available with server. 2021-06-21 00:27:54,475 - FederatedClient - INFO - Getting other models from server for cross validation. 2021-06-21 00:28:01,429 - Communicator - INFO - Received 1 models for validation. GetValidationModels time: 6.952807426452637 seconds ========== Validate Config Result =========== Use GPU: True Multi GPU: False Automatic Mixed Precision: Enabled Determinism Evaluation: Disabled cuDNN BenchMark: False CUDA Matmul Allow TF32: True cuDNN Allow TF32: True Model: Dataset: DataLoader: Validate Transform #1: Validate Transform #2: Validate Transform #3: Validate Handler #1: Validate Handler #2: Validate Handler #3: Validate Post Transforms #1: Validate Post Transforms #2: Validate Inferer: Validate Key Metric: Validate Additional Metric #My own validation mean dice loss: ========== End of Validate Config Result =========== 2021-06-21 00:28:01,691 - ignite.engine.engine.SupervisedEvaluator - INFO - Engine run resuming from iteration 0, epoch 0 until 1 epochs 2021-06-21 00:28:01,733 - ignite.engine.engine.SupervisedEvaluator - INFO - Restored all variables from /tmp/tmp3ix_ixt0/mmar/models/local_best_model.PT_MODEL file written: /tmp/tmp3ix_ixt0/mmar/eval/brain_mask/brain_mask_seg.nii.gz. 2021-06-21 00:28:41,985 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files. file written: /tmp/tmp3ix_ixt0/mmar/eval/brain_mask/brain_mask_seg.nii.gz. 2021-06-21 00:28:56,919 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files. file written: /tmp/tmp3ix_ixt0/mmar/eval/brain_mask/brain_mask_seg.nii.gz. 2021-06-21 00:29:30,782 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files. file written: /tmp/tmp3ix_ixt0/mmar/eval/brain_mask/brain_mask_seg.nii.gz. 2021-06-21 00:30:08,061 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files. file written: /tmp/tmp3ix_ixt0/mmar/eval/brain_mask/brain_mask_seg.nii.gz. 2021-06-21 00:30:35,610 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files. file written: /tmp/tmp3ix_ixt0/mmar/eval/brain_mask/brain_mask_seg.nii.gz. 2021-06-21 00:30:39,500 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files. file written: /tmp/tmp3ix_ixt0/mmar/eval/brain_mask/brain_mask_seg.nii.gz. 2021-06-21 00:30:56,238 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files. file written: /tmp/tmp3ix_ixt0/mmar/eval/brain_mask/brain_mask_seg.nii.gz. 2021-06-21 00:31:25,239 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files. file written: /tmp/tmp3ix_ixt0/mmar/eval/brain_mask/brain_mask_seg.nii.gz. 2021-06-21 00:31:57,784 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files. file written: /tmp/tmp3ix_ixt0/mmar/eval/brain_mask/brain_mask_seg.nii.gz. 2021-06-21 00:32:34,136 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files. file written: /tmp/tmp3ix_ixt0/mmar/eval/brain_mask/brain_mask_seg.nii.gz. 2021-06-21 00:33:10,725 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files. file written: /tmp/tmp3ix_ixt0/mmar/eval/brain_mask/brain_mask_seg.nii.gz. 2021-06-21 00:33:19,675 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files. file written: /tmp/tmp3ix_ixt0/mmar/eval/brain_mask/brain_mask_seg.nii.gz. 2021-06-21 00:33:22,022 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files. file written: /tmp/tmp3ix_ixt0/mmar/eval/brain_mask/brain_mask_seg.nii.gz. 2021-06-21 00:33:58,573 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files. file written: /tmp/tmp3ix_ixt0/mmar/eval/brain_mask/brain_mask_seg.nii.gz. 2021-06-21 00:34:15,416 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files. file written: /tmp/tmp3ix_ixt0/mmar/eval/brain_mask/brain_mask_seg.nii.gz. 2021-06-21 00:34:22,916 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files. file written: /tmp/tmp3ix_ixt0/mmar/eval/brain_mask/brain_mask_seg.nii.gz. 2021-06-21 00:34:30,612 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files. siddharth@FLS-1:/workspace/startup$ file written: /tmp/tmp3ix_ixt0/mmar/eval/brain_mask/brain_mask_seg.nii.gz. 2021-06-21 00:35:03,436 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files. file written: /tmp/tmp3ix_ixt0/mmar/eval/brain_mask/brain_mask_seg.nii.gz. 2021-06-21 00:35:21,440 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files. file written: /tmp/tmp3ix_ixt0/mmar/eval/brain_mask/brain_mask_seg.nii.gz. 2021-06-21 00:35:25,323 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files. 2021-06-21 00:35:25,324 - ignite.engine.engine.SupervisedEvaluator - INFO - Got new best metric of Monai's validation mean dice loss: 0.4330388605594635 2021-06-21 00:35:25,324 - ignite.engine.engine.SupervisedEvaluator - INFO - Epoch[1] Metrics -- Monai's validation mean dice loss: 0.4330 My own validation mean dice loss: 0.4638 2021-06-21 00:35:25,324 - ignite.engine.engine.SupervisedEvaluator - INFO - Key metric: Monai's validation mean dice loss best value: 0.4330388605594635 at epoch: 1 2021-06-21 00:35:25,324 - ignite.engine.engine.SupervisedEvaluator - INFO - Epoch[1] Complete. Time taken: 00:07:24 2021-06-21 00:35:25,324 - ignite.engine.engine.SupervisedEvaluator - INFO - Engine run complete. Time taken: 00:07:24 2021-06-21 00:35:25,387 - medl.apps.mmar_conf - INFO - Total Evaluation Time 443.696031332016 2021-06-21 00:35:25,406 - ClientValidator - INFO - Do validation ... 2021-06-21 00:35:25,413 - FederatedClient - INFO - Submitting cross validation results to server. 2021-06-21 00:35:25,432 - Communicator - INFO - Received comments: Received Cross Validation results from org1-a.. SubmitCrossSiteValidationResults time: 0.01924729347229004 seconds 2021-06-21 00:35:25,433 - FederatedClient - INFO - Finished cross site validation. siddharth@FLS-1:/workspace/startup$