siddharth@FLS-1:~/packages3/org1-a/startup$ siddharth@FLS-1:~/packages3/org1-a/startup$ siddharth@FLS-1:~/packages3/org1-a/startup$ ls client.crt client.key docker.sh fed_client.json log.config readme.txt rootCA.pem signature.pkl start.sh sub_start.sh siddharth@FLS-1:~/packages3/org1-a/startup$ ./docker.sh Starting docker with nvcr.io/ea-nvidia-clara-train/clara-train-sdk:v4.0-EA2 ============= == PyTorch == ============= NVIDIA Release 20.08 (build 15516749) PyTorch Version 1.7.0a0+8deb4fe Container image Copyright (c) 2020, NVIDIA CORPORATION. All rights reserved. Copyright (c) 2014-2020 Facebook Inc. Copyright (c) 2011-2014 Idiap Research Institute (Ronan Collobert) Copyright (c) 2012-2014 Deepmind Technologies (Koray Kavukcuoglu) Copyright (c) 2011-2012 NEC Laboratories America (Koray Kavukcuoglu) Copyright (c) 2011-2013 NYU (Clement Farabet) Copyright (c) 2006-2010 NEC Laboratories America (Ronan Collobert, Leon Bottou, Iain Melvin, Jason Weston) Copyright (c) 2006 Idiap Research Institute (Samy Bengio) Copyright (c) 2001-2004 Idiap Research Institute (Ronan Collobert, Samy Bengio, Johnny Mariethoz) Copyright (c) 2015 Google Inc. Copyright (c) 2015 Yangqing Jia Copyright (c) 2013-2016 The Caffe contributors All rights reserved. Various files include modifications (c) NVIDIA CORPORATION. All rights reserved. NVIDIA modifications are covered by the license terms that apply to the underlying project or file. NOTE: MOFED driver for multi-node communication was not detected. Multi-node communication performance may be reduced. ┌─────────────────────────────────────────────────────────────┐ │ npm update check failed │ │ Try running with sudo or get access │ │ to the local update config store via │ │ sudo chown -R $USER:$(id -gn $USER) /home/siddharth/.config │ └─────────────────────────────────────────────────────────────┘ siddharth@FLS-1:/workspace$ siddharth@FLS-1:/workspace$ cd startup/ siddharth@FLS-1:/workspace/startup$ ls client.crt client.key docker.sh fed_client.json log.config readme.txt rootCA.pem signature.pkl start.sh sub_start.sh siddharth@FLS-1:/workspace/startup$ ./start.sh WORKSPACE set to /workspace/startup/.. siddharth@FLS-1:/workspace/startup$ WORKSPACE set to /workspace/startup/.. PYTHONPATH is /local/custom::/workspace/blt-monai:/opt/nvidia/medical:/workspace/blt-monai/unet3d There seems to be one instance, pid=348, running. If you are sure it's not the case, please kill process 348. siddharth@FLS-1:/workspace/startup$ ./start.sh WORKSPACE set to /workspace/startup/.. siddharth@FLS-1:/workspace/startup$ WORKSPACE set to /workspace/startup/.. PYTHONPATH is /local/custom::/workspace/blt-monai:/opt/nvidia/medical:/workspace/blt-monai/unet3d start fl because process of 365 does not exist new pid 386 Matplotlib created a temporary config/cache directory at /tmp/matplotlib-168evtnp because the default path (/home/siddharth/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing. 2021-05-14 05:53:17,852 - torch.distributed.nn.jit.instantiator - INFO - Created a temporary directory at /tmp/tmp02kllntj 2021-05-14 05:53:17,852 - torch.distributed.nn.jit.instantiator - INFO - Writing /tmp/tmp02kllntj/_remote_module_non_sriptable.py 2021-05-14 05:53:17,910 - ClientModelManager - INFO - privacy module disabled. 2021-05-14 05:53:17,946 - FederatedClient - INFO - Successfully registered client:org1-a for example_project. Got token:e0dd00b0-c2f4-4c17-8fba-d6bc52b5a408 created /tmp/fl/comm/training/x created /tmp/fl/comm/training/y created /tmp/fl/comm/training/t 2021-05-14 05:54:00,409 - ServerAdminInterface - INFO - Starting client training. rank: 0 training child process ID: 549 starting the client ..... no max_epochs specified for SupervisedTrainer args, use global var 'epochs: 15'. deployed MONAI client trainer. 2021-05-14 05:54:02,438 - FederatedClient - INFO - Starting to fetch global model. 2021-05-14 05:54:05,123 - Communicator - INFO - Received example_project model at round 0 (16334771 Bytes). GetModel time: 2.6842880249023438 seconds Get global model for round: 0 AssignVariables... n_ext 48 FED_UPDATE_OPS fedlearn_ops Updated local variables to be assigned. 2021-05-14 05:54:05,148 - AssignVariables - INFO - Vars 48 of 48 assigned. assign_ops 48 updated_local_model 48 2021-05-14 05:54:05,153 - ClientModelManager - INFO - Setting global federated model data (4081379 elements) 2021-05-14 05:54:05,155 - ClientModelManager - INFO - Round 0: local model updated pull_models completed. Status:True rank:0 2021-05-14 05:54:05,243 - ignite.engine.engine.SupervisedEvaluator - INFO - Engine run resuming from iteration 0, epoch 0 until 1 epochs 2021-05-14 05:54:47,858 - ignite.engine.engine.SupervisedEvaluator - INFO - Got new best metric of validation mean dice: 0.00422763591632247 2021-05-14 05:54:47,858 - ignite.engine.engine.SupervisedEvaluator - INFO - Epoch[1] Metrics -- validation loss: -5.9199 validation mean dice: 0.0042 2021-05-14 05:54:47,858 - ignite.engine.engine.SupervisedEvaluator - INFO - Key metric: validation mean dice best value: 0.00422763591632247 at epoch: 1 2021-05-14 05:54:47,885 - ignite.engine.engine.SupervisedEvaluator - INFO - Epoch[1] Complete. Time taken: 00:00:42 2021-05-14 05:54:47,885 - ignite.engine.engine.SupervisedEvaluator - INFO - Engine run complete. Time taken: 00:00:43 2021-05-14 05:54:47,944 - ignite.engine.engine.SupervisedTrainer - INFO - Engine run resuming from iteration 0, epoch 0 until 3 epochs 2021-05-14 05:55:02,552 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 1/125 -- train_loss: 176.5626 2021-05-14 05:55:03,274 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 2/125 -- train_loss: 70.9007 2021-05-14 05:55:03,998 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 3/125 -- train_loss: 6.1936 2021-05-14 05:55:04,722 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 4/125 -- train_loss: 10.4949 2021-05-14 05:55:05,450 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 5/125 -- train_loss: 5.4529 2021-05-14 05:55:06,175 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 6/125 -- train_loss: 4.6019 2021-05-14 05:55:06,899 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 7/125 -- train_loss: 5.0659 2021-05-14 05:55:07,625 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 8/125 -- train_loss: 4.5512 2021-05-14 05:55:13,969 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 9/125 -- train_loss: 3.9684 2021-05-14 05:55:14,693 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 10/125 -- train_loss: 3.8457 2021-05-14 05:55:15,420 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 11/125 -- train_loss: 3.9101 2021-05-14 05:55:16,145 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 12/125 -- train_loss: 3.4618 2021-05-14 05:55:16,866 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 13/125 -- train_loss: 2.7920 2021-05-14 05:55:17,594 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 14/125 -- train_loss: 3.7456 2021-05-14 05:55:18,320 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 15/125 -- train_loss: 3.9018 2021-05-14 05:55:19,047 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 16/125 -- train_loss: 2.6617 2021-05-14 05:55:26,816 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 17/125 -- train_loss: 2.4686 2021-05-14 05:55:27,537 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 18/125 -- train_loss: 5.3201 2021-05-14 05:55:28,258 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 19/125 -- train_loss: 3.7963 2021-05-14 05:55:28,987 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 20/125 -- train_loss: 2.4864 2021-05-14 05:55:29,717 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 21/125 -- train_loss: 2.8503 2021-05-14 05:55:30,447 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 22/125 -- train_loss: 3.5102 2021-05-14 05:55:31,180 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 23/125 -- train_loss: 4.1749 2021-05-14 05:55:31,906 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 24/125 -- train_loss: 2.3154 2021-05-14 05:55:37,435 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 25/125 -- train_loss: 2.8087 2021-05-14 05:55:38,161 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 26/125 -- train_loss: 2.1969 2021-05-14 05:55:38,887 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 27/125 -- train_loss: 2.1559 2021-05-14 05:55:39,960 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 28/125 -- train_loss: 2.3557 2021-05-14 05:55:40,689 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 29/125 -- train_loss: 3.0014 2021-05-14 05:55:41,419 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 30/125 -- train_loss: 2.4045 2021-05-14 05:55:42,148 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 31/125 -- train_loss: 2.3076 2021-05-14 05:55:42,879 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 32/125 -- train_loss: 1.9418 2021-05-14 05:55:48,444 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 33/125 -- train_loss: 2.3390 2021-05-14 05:55:49,174 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 34/125 -- train_loss: 1.7659 2021-05-14 05:55:49,902 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 35/125 -- train_loss: 2.7236 2021-05-14 05:55:52,383 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 36/125 -- train_loss: 2.2980 2021-05-14 05:55:53,113 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 37/125 -- train_loss: 2.1704 2021-05-14 05:55:53,845 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 38/125 -- train_loss: 2.3757 2021-05-14 05:55:54,578 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 39/125 -- train_loss: 2.2728 2021-05-14 05:55:55,312 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 40/125 -- train_loss: 1.6085 2021-05-14 05:55:58,794 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 41/125 -- train_loss: 2.1816 2021-05-14 05:55:59,526 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 42/125 -- train_loss: 1.6633 2021-05-14 05:56:00,259 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 43/125 -- train_loss: 2.2647 2021-05-14 05:56:02,427 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 44/125 -- train_loss: 2.1364 2021-05-14 05:56:03,156 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 45/125 -- train_loss: 3.5361 2021-05-14 05:56:03,888 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 46/125 -- train_loss: 2.0079 2021-05-14 05:56:04,618 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 47/125 -- train_loss: 2.0365 2021-05-14 05:56:05,350 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 48/125 -- train_loss: 1.6483 2021-05-14 05:56:08,133 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 49/125 -- train_loss: 2.1087 2021-05-14 05:56:08,863 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 50/125 -- train_loss: 1.8014 2021-05-14 05:56:09,595 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 51/125 -- train_loss: 2.3601 2021-05-14 05:56:13,359 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 52/125 -- train_loss: 1.6964 2021-05-14 05:56:14,091 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 53/125 -- train_loss: 2.7483 2021-05-14 05:56:14,824 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 54/125 -- train_loss: 1.6839 2021-05-14 05:56:15,559 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 55/125 -- train_loss: 1.9793 2021-05-14 05:56:16,290 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 56/125 -- train_loss: 3.2847 2021-05-14 05:56:18,212 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 57/125 -- train_loss: 2.5710 2021-05-14 05:56:18,944 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 58/125 -- train_loss: 1.8179 2021-05-14 05:56:19,676 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 59/125 -- train_loss: 1.5812 2021-05-14 05:56:24,353 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 60/125 -- train_loss: 3.2989 2021-05-14 05:56:25,083 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 61/125 -- train_loss: 3.8984 2021-05-14 05:56:25,813 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 62/125 -- train_loss: 1.6449 2021-05-14 05:56:26,547 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 63/125 -- train_loss: 3.6106 2021-05-14 05:56:27,279 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 64/125 -- train_loss: 1.9817 2021-05-14 05:56:29,071 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 65/125 -- train_loss: 2.1798 2021-05-14 05:56:29,797 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 66/125 -- train_loss: 2.1941 2021-05-14 05:56:30,557 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 67/125 -- train_loss: 2.3717 2021-05-14 05:56:35,145 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 68/125 -- train_loss: 1.5748 2021-05-14 05:56:35,875 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 69/125 -- train_loss: 1.9719 2021-05-14 05:56:36,610 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 70/125 -- train_loss: 2.1707 2021-05-14 05:56:37,345 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 71/125 -- train_loss: 3.0329 2021-05-14 05:56:38,081 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 72/125 -- train_loss: 3.4816 2021-05-14 05:56:41,741 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 73/125 -- train_loss: 3.2819 2021-05-14 05:56:42,473 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 74/125 -- train_loss: 1.5687 2021-05-14 05:56:43,206 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 75/125 -- train_loss: 1.8334 2021-05-14 05:56:45,680 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 76/125 -- train_loss: 2.5225 2021-05-14 05:56:46,416 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 77/125 -- train_loss: 2.0671 2021-05-14 05:56:47,152 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 78/125 -- train_loss: 1.9752 2021-05-14 05:56:47,887 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 79/125 -- train_loss: 1.8088 2021-05-14 05:56:48,619 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 80/125 -- train_loss: 1.6482 2021-05-14 05:56:53,828 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 81/125 -- train_loss: 1.9575 2021-05-14 05:56:54,555 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 82/125 -- train_loss: 1.6094 2021-05-14 05:56:55,291 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 83/125 -- train_loss: 2.5953 2021-05-14 05:56:56,582 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 84/125 -- train_loss: 1.9061 2021-05-14 05:56:57,318 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 85/125 -- train_loss: 1.4642 2021-05-14 05:56:58,055 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 86/125 -- train_loss: 1.4118 2021-05-14 05:56:58,791 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 87/125 -- train_loss: 5.0622 2021-05-14 05:56:59,530 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 88/125 -- train_loss: 3.4696 2021-05-14 05:57:03,517 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 89/125 -- train_loss: 1.2777 2021-05-14 05:57:04,253 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 90/125 -- train_loss: 1.3961 2021-05-14 05:57:05,299 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 91/125 -- train_loss: 1.4280 2021-05-14 05:57:07,120 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 92/125 -- train_loss: 1.6929 2021-05-14 05:57:07,855 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 93/125 -- train_loss: 1.7069 2021-05-14 05:57:08,589 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 94/125 -- train_loss: 1.8751 2021-05-14 05:57:09,326 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 95/125 -- train_loss: 2.2560 2021-05-14 05:57:10,059 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 96/125 -- train_loss: 1.9153 2021-05-14 05:57:13,556 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 97/125 -- train_loss: 2.4022 2021-05-14 05:57:14,290 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 98/125 -- train_loss: 1.7294 2021-05-14 05:57:15,052 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 99/125 -- train_loss: 1.7756 2021-05-14 05:57:17,465 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 100/125 -- train_loss: 1.8858 2021-05-14 05:57:18,204 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 101/125 -- train_loss: 1.7886 2021-05-14 05:57:18,942 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 102/125 -- train_loss: 1.5743 2021-05-14 05:57:19,679 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 103/125 -- train_loss: 2.5413 2021-05-14 05:57:20,417 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 104/125 -- train_loss: 2.2717 2021-05-14 05:57:25,792 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 105/125 -- train_loss: 2.1277 2021-05-14 05:57:26,530 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 106/125 -- train_loss: 2.7000 2021-05-14 05:57:27,267 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 107/125 -- train_loss: 2.5668 2021-05-14 05:57:28,026 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 108/125 -- train_loss: 1.6429 2021-05-14 05:57:28,762 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 109/125 -- train_loss: 1.7646 2021-05-14 05:57:29,498 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 110/125 -- train_loss: 2.1447 2021-05-14 05:57:30,238 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 111/125 -- train_loss: 1.9790 2021-05-14 05:57:30,974 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 112/125 -- train_loss: 3.3275 2021-05-14 05:57:36,599 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 113/125 -- train_loss: 1.7578 2021-05-14 05:57:37,335 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 114/125 -- train_loss: 1.6245 2021-05-14 05:57:38,071 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 115/125 -- train_loss: 1.5863 2021-05-14 05:57:38,805 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 116/125 -- train_loss: 1.5079 2021-05-14 05:57:39,541 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 117/125 -- train_loss: 1.6416 2021-05-14 05:57:40,277 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 118/125 -- train_loss: 1.5914 2021-05-14 05:57:41,012 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 119/125 -- train_loss: 1.3934 2021-05-14 05:57:41,748 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 120/125 -- train_loss: 1.7872 2021-05-14 05:57:46,387 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 121/125 -- train_loss: 1.3126 2021-05-14 05:57:47,123 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 122/125 -- train_loss: 1.8095 2021-05-14 05:57:47,856 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 123/125 -- train_loss: 1.4241 2021-05-14 05:57:48,595 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 124/125 -- train_loss: 4.8685 2021-05-14 05:57:49,330 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 1/3, Iter: 125/125 -- train_loss: 2.9614 2021-05-14 05:57:49,330 - ignite.engine.engine.SupervisedEvaluator - INFO - Engine run resuming from iteration 0, epoch 0 until 1 epochs 2021-05-14 05:58:32,739 - ignite.engine.engine.SupervisedEvaluator - INFO - Got new best metric of validation mean dice: 0.4593188166618347 2021-05-14 05:58:32,740 - ignite.engine.engine.SupervisedEvaluator - INFO - Epoch[1] Metrics -- validation loss: -0.0689 validation mean dice: 0.4593 2021-05-14 05:58:32,740 - ignite.engine.engine.SupervisedEvaluator - INFO - Key metric: validation mean dice best value: 0.4593188166618347 at epoch: 1 2021-05-14 05:58:32,775 - ignite.engine.engine.SupervisedEvaluator - INFO - Epoch[1] Complete. Time taken: 00:00:43 2021-05-14 05:58:32,775 - ignite.engine.engine.SupervisedEvaluator - INFO - Engine run complete. Time taken: 00:00:43 2021-05-14 05:58:32,843 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch[1] Complete. Time taken: 00:03:44 2021-05-14 05:58:44,195 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 1/125 -- train_loss: 1.6534 2021-05-14 05:58:45,622 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 2/125 -- train_loss: 1.3090 2021-05-14 05:58:46,360 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 3/125 -- train_loss: 1.7417 2021-05-14 05:58:47,101 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 4/125 -- train_loss: 2.0974 2021-05-14 05:58:47,837 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 5/125 -- train_loss: 2.0062 2021-05-14 05:58:48,576 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 6/125 -- train_loss: 1.5046 2021-05-14 05:58:49,313 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 7/125 -- train_loss: 1.7329 2021-05-14 05:58:50,052 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 8/125 -- train_loss: 1.3673 2021-05-14 05:58:54,457 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 9/125 -- train_loss: 1.6643 2021-05-14 05:58:55,656 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 10/125 -- train_loss: 2.1558 2021-05-14 05:58:56,393 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 11/125 -- train_loss: 1.2368 2021-05-14 05:58:57,131 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 12/125 -- train_loss: 1.4245 2021-05-14 05:58:57,869 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 13/125 -- train_loss: 1.5401 2021-05-14 05:58:58,607 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 14/125 -- train_loss: 1.2591 2021-05-14 05:58:59,344 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 15/125 -- train_loss: 1.7601 2021-05-14 05:59:00,082 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 16/125 -- train_loss: 5.3710 2021-05-14 05:59:04,802 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 17/125 -- train_loss: 1.9462 2021-05-14 05:59:05,638 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 18/125 -- train_loss: 3.7782 2021-05-14 05:59:06,372 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 19/125 -- train_loss: 2.5211 2021-05-14 05:59:07,114 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 20/125 -- train_loss: 1.6373 2021-05-14 05:59:07,852 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 21/125 -- train_loss: 2.1023 2021-05-14 05:59:08,591 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 22/125 -- train_loss: 2.6236 2021-05-14 05:59:09,328 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 23/125 -- train_loss: 1.8911 2021-05-14 05:59:10,064 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 24/125 -- train_loss: 2.0935 2021-05-14 05:59:14,059 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 25/125 -- train_loss: 1.6095 2021-05-14 05:59:14,896 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 26/125 -- train_loss: 2.1664 2021-05-14 05:59:15,631 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 27/125 -- train_loss: 1.9634 2021-05-14 05:59:16,369 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 28/125 -- train_loss: 1.4246 2021-05-14 05:59:17,107 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 29/125 -- train_loss: 1.7318 2021-05-14 05:59:17,847 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 30/125 -- train_loss: 1.8585 2021-05-14 05:59:18,582 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 31/125 -- train_loss: 1.9279 2021-05-14 05:59:19,318 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 32/125 -- train_loss: 1.4068 2021-05-14 05:59:23,501 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 33/125 -- train_loss: 2.3134 2021-05-14 05:59:24,506 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 34/125 -- train_loss: 3.7187 2021-05-14 05:59:25,894 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 35/125 -- train_loss: 1.6729 2021-05-14 05:59:26,631 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 36/125 -- train_loss: 1.7383 2021-05-14 05:59:27,365 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 37/125 -- train_loss: 1.7832 2021-05-14 05:59:28,100 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 38/125 -- train_loss: 1.6790 2021-05-14 05:59:28,836 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 39/125 -- train_loss: 1.8581 2021-05-14 05:59:29,575 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 40/125 -- train_loss: 1.6081 2021-05-14 05:59:33,742 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 41/125 -- train_loss: 1.9544 2021-05-14 05:59:34,500 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 42/125 -- train_loss: 1.9891 2021-05-14 05:59:35,282 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 43/125 -- train_loss: 2.0334 2021-05-14 05:59:36,020 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 44/125 -- train_loss: 1.4496 2021-05-14 05:59:36,760 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 45/125 -- train_loss: 1.7690 2021-05-14 05:59:37,527 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 46/125 -- train_loss: 1.5677 2021-05-14 05:59:38,259 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 47/125 -- train_loss: 1.6142 2021-05-14 05:59:38,996 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 48/125 -- train_loss: 1.3364 2021-05-14 05:59:43,847 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 49/125 -- train_loss: 1.3861 2021-05-14 05:59:44,669 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 50/125 -- train_loss: 2.4567 2021-05-14 05:59:45,610 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 51/125 -- train_loss: 2.3794 2021-05-14 05:59:46,351 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 52/125 -- train_loss: 2.5490 2021-05-14 05:59:47,089 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 53/125 -- train_loss: 2.3493 2021-05-14 05:59:47,826 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 54/125 -- train_loss: 1.6557 2021-05-14 05:59:48,561 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 55/125 -- train_loss: 2.4046 2021-05-14 05:59:49,299 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 56/125 -- train_loss: 2.1657 2021-05-14 05:59:53,752 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 57/125 -- train_loss: 1.8698 2021-05-14 05:59:54,488 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 58/125 -- train_loss: 1.5064 2021-05-14 05:59:55,250 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 59/125 -- train_loss: 1.3663 2021-05-14 05:59:55,987 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 60/125 -- train_loss: 1.7096 2021-05-14 05:59:56,724 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 61/125 -- train_loss: 1.6543 2021-05-14 05:59:57,461 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 62/125 -- train_loss: 2.0655 2021-05-14 05:59:58,193 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 63/125 -- train_loss: 1.4279 2021-05-14 05:59:58,931 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 64/125 -- train_loss: 3.4718 2021-05-14 06:00:03,073 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 65/125 -- train_loss: 1.3844 2021-05-14 06:00:03,810 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 66/125 -- train_loss: 1.5180 2021-05-14 06:00:05,409 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 67/125 -- train_loss: 1.4656 2021-05-14 06:00:06,167 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 68/125 -- train_loss: 1.6675 2021-05-14 06:00:06,906 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 69/125 -- train_loss: 1.5312 2021-05-14 06:00:07,642 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 70/125 -- train_loss: 1.9380 2021-05-14 06:00:08,379 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 71/125 -- train_loss: 1.5305 2021-05-14 06:00:09,115 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 72/125 -- train_loss: 1.4061 2021-05-14 06:00:12,749 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 73/125 -- train_loss: 1.9833 2021-05-14 06:00:13,485 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 74/125 -- train_loss: 2.5227 2021-05-14 06:00:14,996 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 75/125 -- train_loss: 3.3695 2021-05-14 06:00:15,735 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 76/125 -- train_loss: 2.7350 2021-05-14 06:00:16,472 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 77/125 -- train_loss: 1.8775 2021-05-14 06:00:17,210 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 78/125 -- train_loss: 1.3520 2021-05-14 06:00:17,945 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 79/125 -- train_loss: 1.9830 2021-05-14 06:00:18,682 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 80/125 -- train_loss: 1.7670 2021-05-14 06:00:22,499 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 81/125 -- train_loss: 1.9880 2021-05-14 06:00:23,234 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 82/125 -- train_loss: 1.6834 2021-05-14 06:00:24,859 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 83/125 -- train_loss: 1.7649 2021-05-14 06:00:25,594 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 84/125 -- train_loss: 2.1421 2021-05-14 06:00:26,329 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 85/125 -- train_loss: 1.3721 2021-05-14 06:00:27,064 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 86/125 -- train_loss: 1.6110 2021-05-14 06:00:27,800 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 87/125 -- train_loss: 1.3415 2021-05-14 06:00:28,540 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 88/125 -- train_loss: 2.4440 2021-05-14 06:00:32,438 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 89/125 -- train_loss: 1.6899 2021-05-14 06:00:33,178 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 90/125 -- train_loss: 1.8908 2021-05-14 06:00:34,138 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 91/125 -- train_loss: 1.4914 2021-05-14 06:00:34,873 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 92/125 -- train_loss: 3.2943 2021-05-14 06:00:35,608 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 93/125 -- train_loss: 1.6635 2021-05-14 06:00:36,345 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 94/125 -- train_loss: 1.4605 2021-05-14 06:00:37,113 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 95/125 -- train_loss: 2.0586 2021-05-14 06:00:37,850 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 96/125 -- train_loss: 2.7380 2021-05-14 06:00:42,925 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 97/125 -- train_loss: 2.6611 2021-05-14 06:00:43,661 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 98/125 -- train_loss: 1.6985 2021-05-14 06:00:44,523 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 99/125 -- train_loss: 1.4192 2021-05-14 06:00:45,261 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 100/125 -- train_loss: 1.7676 2021-05-14 06:00:46,000 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 101/125 -- train_loss: 1.3596 2021-05-14 06:00:46,736 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 102/125 -- train_loss: 1.5697 2021-05-14 06:00:47,474 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 103/125 -- train_loss: 1.7819 2021-05-14 06:00:48,211 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 104/125 -- train_loss: 1.8329 2021-05-14 06:00:52,753 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 105/125 -- train_loss: 1.7692 2021-05-14 06:00:53,489 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 106/125 -- train_loss: 2.2758 2021-05-14 06:00:54,250 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 107/125 -- train_loss: 1.6211 2021-05-14 06:00:54,988 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 108/125 -- train_loss: 1.3572 2021-05-14 06:00:55,725 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 109/125 -- train_loss: 1.9866 2021-05-14 06:00:56,463 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 110/125 -- train_loss: 2.3089 2021-05-14 06:00:57,203 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 111/125 -- train_loss: 4.0469 2021-05-14 06:00:57,941 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 112/125 -- train_loss: 2.0309 2021-05-14 06:01:02,866 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 113/125 -- train_loss: 1.7386 2021-05-14 06:01:03,602 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 114/125 -- train_loss: 2.1062 2021-05-14 06:01:04,338 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 115/125 -- train_loss: 1.3445 2021-05-14 06:01:05,075 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 116/125 -- train_loss: 1.6721 2021-05-14 06:01:05,812 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 117/125 -- train_loss: 1.5996 2021-05-14 06:01:06,548 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 118/125 -- train_loss: 1.3223 2021-05-14 06:01:07,284 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 119/125 -- train_loss: 1.6036 2021-05-14 06:01:08,022 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 120/125 -- train_loss: 1.4280 2021-05-14 06:01:11,814 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 121/125 -- train_loss: 1.4298 2021-05-14 06:01:12,554 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 122/125 -- train_loss: 1.3759 2021-05-14 06:01:13,290 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 123/125 -- train_loss: 2.0122 2021-05-14 06:01:14,026 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 124/125 -- train_loss: 2.0506 2021-05-14 06:01:14,764 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 2/3, Iter: 125/125 -- train_loss: 2.2778 2021-05-14 06:01:14,764 - ignite.engine.engine.SupervisedEvaluator - INFO - Engine run resuming from iteration 0, epoch 1 until 2 epochs 2021-05-14 06:01:57,033 - ignite.engine.engine.SupervisedEvaluator - INFO - Epoch[2] Metrics -- validation loss: -0.0706 validation mean dice: 0.4504 2021-05-14 06:01:57,033 - ignite.engine.engine.SupervisedEvaluator - INFO - Key metric: validation mean dice best value: 0.4593188166618347 at epoch: 1 2021-05-14 06:01:57,034 - ignite.engine.engine.SupervisedEvaluator - INFO - Epoch[2] Complete. Time taken: 00:00:42 2021-05-14 06:01:57,034 - ignite.engine.engine.SupervisedEvaluator - INFO - Engine run complete. Time taken: 00:00:42 2021-05-14 06:01:57,100 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch[2] Complete. Time taken: 00:03:24 2021-05-14 06:02:08,461 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 1/125 -- train_loss: 2.3619 2021-05-14 06:02:09,277 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 2/125 -- train_loss: 2.0552 2021-05-14 06:02:10,015 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 3/125 -- train_loss: 1.4231 2021-05-14 06:02:10,750 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 4/125 -- train_loss: 1.9192 2021-05-14 06:02:11,491 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 5/125 -- train_loss: 1.8903 2021-05-14 06:02:12,230 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 6/125 -- train_loss: 1.8594 2021-05-14 06:02:12,970 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 7/125 -- train_loss: 1.8384 2021-05-14 06:02:13,713 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 8/125 -- train_loss: 2.0869 2021-05-14 06:02:18,520 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 9/125 -- train_loss: 1.6155 2021-05-14 06:02:19,257 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 10/125 -- train_loss: 1.2756 2021-05-14 06:02:19,995 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 11/125 -- train_loss: 1.6997 2021-05-14 06:02:20,730 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 12/125 -- train_loss: 1.4962 2021-05-14 06:02:21,464 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 13/125 -- train_loss: 1.7678 2021-05-14 06:02:22,205 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 14/125 -- train_loss: 1.6151 2021-05-14 06:02:22,947 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 15/125 -- train_loss: 1.6122 2021-05-14 06:02:23,687 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 16/125 -- train_loss: 2.0250 2021-05-14 06:02:27,600 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 17/125 -- train_loss: 1.3924 2021-05-14 06:02:29,321 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 18/125 -- train_loss: 2.0378 2021-05-14 06:02:30,082 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 19/125 -- train_loss: 2.3881 2021-05-14 06:02:30,818 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 20/125 -- train_loss: 1.9006 2021-05-14 06:02:31,555 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 21/125 -- train_loss: 1.4302 2021-05-14 06:02:32,291 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 22/125 -- train_loss: 2.4170 2021-05-14 06:02:33,029 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 23/125 -- train_loss: 2.1552 2021-05-14 06:02:33,769 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 24/125 -- train_loss: 1.9465 2021-05-14 06:02:37,845 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 25/125 -- train_loss: 1.2194 2021-05-14 06:02:38,868 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 26/125 -- train_loss: 1.8197 2021-05-14 06:02:39,636 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 27/125 -- train_loss: 1.2737 2021-05-14 06:02:40,372 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 28/125 -- train_loss: 1.6804 2021-05-14 06:02:41,107 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 29/125 -- train_loss: 1.7348 2021-05-14 06:02:41,845 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 30/125 -- train_loss: 1.6672 2021-05-14 06:02:42,585 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 31/125 -- train_loss: 1.7813 2021-05-14 06:02:43,321 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 32/125 -- train_loss: 1.5412 2021-05-14 06:02:46,991 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 33/125 -- train_loss: 1.5696 2021-05-14 06:02:48,835 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 34/125 -- train_loss: 1.9028 2021-05-14 06:02:49,594 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 35/125 -- train_loss: 2.1389 2021-05-14 06:02:50,331 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 36/125 -- train_loss: 1.8577 2021-05-14 06:02:51,069 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 37/125 -- train_loss: 1.5547 2021-05-14 06:02:51,809 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 38/125 -- train_loss: 2.5138 2021-05-14 06:02:52,546 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 39/125 -- train_loss: 2.3235 2021-05-14 06:02:53,282 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 40/125 -- train_loss: 1.5514 2021-05-14 06:02:56,260 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 41/125 -- train_loss: 1.5059 2021-05-14 06:02:58,559 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 42/125 -- train_loss: 1.9038 2021-05-14 06:02:59,383 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 43/125 -- train_loss: 1.4410 2021-05-14 06:03:00,122 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 44/125 -- train_loss: 1.3572 2021-05-14 06:03:00,855 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 45/125 -- train_loss: 1.3351 2021-05-14 06:03:01,587 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 46/125 -- train_loss: 1.6925 2021-05-14 06:03:02,324 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 47/125 -- train_loss: 1.6597 2021-05-14 06:03:03,062 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 48/125 -- train_loss: 1.4071 2021-05-14 06:03:06,023 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 49/125 -- train_loss: 2.8220 2021-05-14 06:03:08,840 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 50/125 -- train_loss: 2.4613 2021-05-14 06:03:09,575 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 51/125 -- train_loss: 1.9809 2021-05-14 06:03:10,310 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 52/125 -- train_loss: 1.1002 2021-05-14 06:03:11,072 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 53/125 -- train_loss: 1.6960 2021-05-14 06:03:11,808 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 54/125 -- train_loss: 2.0317 2021-05-14 06:03:12,546 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 55/125 -- train_loss: 1.8118 2021-05-14 06:03:13,282 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 56/125 -- train_loss: 2.6920 2021-05-14 06:03:15,157 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 57/125 -- train_loss: 1.5392 2021-05-14 06:03:18,456 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 58/125 -- train_loss: 2.1847 2021-05-14 06:03:19,192 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 59/125 -- train_loss: 1.5794 2021-05-14 06:03:19,927 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 60/125 -- train_loss: 1.3137 2021-05-14 06:03:20,661 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 61/125 -- train_loss: 2.0184 2021-05-14 06:03:21,398 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 62/125 -- train_loss: 1.8784 2021-05-14 06:03:22,135 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 63/125 -- train_loss: 1.8533 2021-05-14 06:03:22,874 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 64/125 -- train_loss: 1.2535 2021-05-14 06:03:24,936 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 65/125 -- train_loss: 1.7201 2021-05-14 06:03:27,646 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 66/125 -- train_loss: 1.3694 2021-05-14 06:03:28,687 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 67/125 -- train_loss: 1.6546 2021-05-14 06:03:29,426 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 68/125 -- train_loss: 1.2662 2021-05-14 06:03:30,185 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 69/125 -- train_loss: 2.1061 2021-05-14 06:03:30,919 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 70/125 -- train_loss: 1.6260 2021-05-14 06:03:31,654 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 71/125 -- train_loss: 1.7688 2021-05-14 06:03:32,392 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 72/125 -- train_loss: 1.4373 2021-05-14 06:03:34,480 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 73/125 -- train_loss: 1.7864 2021-05-14 06:03:37,236 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 74/125 -- train_loss: 3.0035 2021-05-14 06:03:38,407 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 75/125 -- train_loss: 1.6987 2021-05-14 06:03:39,145 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 76/125 -- train_loss: 1.4911 2021-05-14 06:03:39,904 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 77/125 -- train_loss: 1.7893 2021-05-14 06:03:40,640 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 78/125 -- train_loss: 1.2700 2021-05-14 06:03:41,376 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 79/125 -- train_loss: 1.5624 2021-05-14 06:03:42,107 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 80/125 -- train_loss: 1.4562 2021-05-14 06:03:43,801 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 81/125 -- train_loss: 1.2890 2021-05-14 06:03:46,506 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 82/125 -- train_loss: 1.7875 2021-05-14 06:03:47,859 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 83/125 -- train_loss: 1.5361 2021-05-14 06:03:48,598 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 84/125 -- train_loss: 2.1696 2021-05-14 06:03:49,354 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 85/125 -- train_loss: 1.6369 2021-05-14 06:03:50,090 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 86/125 -- train_loss: 1.2816 2021-05-14 06:03:50,826 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 87/125 -- train_loss: 1.2853 2021-05-14 06:03:51,565 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 88/125 -- train_loss: 1.2861 2021-05-14 06:03:53,221 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 89/125 -- train_loss: 1.3720 2021-05-14 06:03:55,500 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 90/125 -- train_loss: 2.1788 2021-05-14 06:03:57,453 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 91/125 -- train_loss: 3.0968 2021-05-14 06:03:58,190 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 92/125 -- train_loss: 1.3452 2021-05-14 06:03:58,947 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 93/125 -- train_loss: 1.8347 2021-05-14 06:03:59,684 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 94/125 -- train_loss: 1.2884 2021-05-14 06:04:00,417 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 95/125 -- train_loss: 1.9319 2021-05-14 06:04:01,156 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 96/125 -- train_loss: 1.3833 2021-05-14 06:04:03,057 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 97/125 -- train_loss: 1.8257 2021-05-14 06:04:05,322 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 98/125 -- train_loss: 1.5525 2021-05-14 06:04:07,300 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 99/125 -- train_loss: 1.1036 2021-05-14 06:04:08,042 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 100/125 -- train_loss: 1.4280 2021-05-14 06:04:08,778 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 101/125 -- train_loss: 2.2413 2021-05-14 06:04:09,514 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 102/125 -- train_loss: 2.3732 2021-05-14 06:04:10,251 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 103/125 -- train_loss: 1.7797 2021-05-14 06:04:10,991 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 104/125 -- train_loss: 2.0386 2021-05-14 06:04:12,634 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 105/125 -- train_loss: 1.9115 2021-05-14 06:04:15,356 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 106/125 -- train_loss: 1.4650 2021-05-14 06:04:17,319 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 107/125 -- train_loss: 1.4453 2021-05-14 06:04:18,058 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 108/125 -- train_loss: 1.5264 2021-05-14 06:04:18,799 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 109/125 -- train_loss: 1.3838 2021-05-14 06:04:19,535 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 110/125 -- train_loss: 1.6017 2021-05-14 06:04:20,270 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 111/125 -- train_loss: 1.6270 2021-05-14 06:04:21,008 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 112/125 -- train_loss: 1.3342 2021-05-14 06:04:22,073 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 113/125 -- train_loss: 1.7163 2021-05-14 06:04:24,826 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 114/125 -- train_loss: 2.2595 2021-05-14 06:04:26,599 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 115/125 -- train_loss: 1.9895 2021-05-14 06:04:27,336 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 116/125 -- train_loss: 1.6670 2021-05-14 06:04:28,072 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 117/125 -- train_loss: 1.5146 2021-05-14 06:04:28,812 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 118/125 -- train_loss: 1.2655 2021-05-14 06:04:29,548 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 119/125 -- train_loss: 1.7381 2021-05-14 06:04:30,281 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 120/125 -- train_loss: 1.7463 2021-05-14 06:04:31,053 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 121/125 -- train_loss: 1.6594 2021-05-14 06:04:33,881 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 122/125 -- train_loss: 1.5264 2021-05-14 06:04:35,479 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 123/125 -- train_loss: 1.7733 2021-05-14 06:04:36,218 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 124/125 -- train_loss: 1.2108 2021-05-14 06:04:36,953 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 3/3, Iter: 125/125 -- train_loss: 1.2610 2021-05-14 06:04:36,954 - ignite.engine.engine.SupervisedEvaluator - INFO - Engine run resuming from iteration 0, epoch 2 until 3 epochs 2021-05-14 06:05:19,196 - ignite.engine.engine.SupervisedEvaluator - INFO - Got new best metric of validation mean dice: 0.4877413809299469 2021-05-14 06:05:19,196 - ignite.engine.engine.SupervisedEvaluator - INFO - Epoch[3] Metrics -- validation loss: -0.0617 validation mean dice: 0.4877 2021-05-14 06:05:19,196 - ignite.engine.engine.SupervisedEvaluator - INFO - Key metric: validation mean dice best value: 0.4877413809299469 at epoch: 3 2021-05-14 06:05:19,226 - ignite.engine.engine.SupervisedEvaluator - INFO - Epoch[3] Complete. Time taken: 00:00:42 2021-05-14 06:05:19,226 - ignite.engine.engine.SupervisedEvaluator - INFO - Engine run complete. Time taken: 00:00:42 2021-05-14 06:05:19,294 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch[3] Complete. Time taken: 00:03:22 2021-05-14 06:05:19,365 - ignite.engine.engine.SupervisedTrainer - INFO - Train completed, saved final checkpoint: checkpoint_final_iteration=375.pt 2021-05-14 06:05:19,365 - ignite.engine.engine.SupervisedTrainer - INFO - Engine run complete. Time taken: 00:10:31 Send model to server. 2021-05-14 06:05:19,575 - FederatedClient - INFO - Starting to push model. setup local_model_dict local_model_dict 48 2021-05-14 06:05:19,629 - Communicator - INFO - Send example_project at round 0 2021-05-14 06:05:22,088 - Communicator - INFO - Received comments: example_project Received round 0 from org1-a (16334852 Bytes, 685 seconds). SubmitUpdate time: 2.458900213241577 seconds 2021-05-14 06:05:22,104 - FederatedClient - INFO - Starting to fetch global model. No token for this client in current round. Waiting for server new round. No token for this client in current round. Waiting for server new round. No token for this client in current round. Waiting for server new round. No token for this client in current round. Waiting for server new round. No token for this client in current round. Waiting for server new round. No token for this client in current round. Waiting for server new round. pull_models completed. Status:False rank:0 2021-05-14 06:05:52,253 - FederatedClient - INFO - Starting to fetch global model. No token for this client in current round. Waiting for server new round. No token for this client in current round. Waiting for server new round. No token for this client in current round. Waiting for server new round. No token for this client in current round. Waiting for server new round. No token for this client in current round. Waiting for server new round. No token for this client in current round. Waiting for server new round. pull_models completed. Status:False rank:0 2021-05-14 06:06:22,302 - FederatedClient - INFO - Starting to fetch global model. No token for this client in current round. Waiting for server new round. No token for this client in current round. Waiting for server new round. No token for this client in current round. Waiting for server new round. No token for this client in current round. Waiting for server new round. No token for this client in current round. Waiting for server new round. No token for this client in current round. Waiting for server new round. pull_models completed. Status:False rank:0 2021-05-14 06:06:52,351 - FederatedClient - INFO - Starting to fetch global model. No token for this client in current round. Waiting for server new round. No token for this client in current round. Waiting for server new round. No token for this client in current round. Waiting for server new round. No token for this client in current round. Waiting for server new round. No token for this client in current round. Waiting for server new round. No token for this client in current round. Waiting for server new round. pull_models completed. Status:False rank:0 2021-05-14 06:07:22,500 - FederatedClient - INFO - Starting to fetch global model. No token for this client in current round. Waiting for server new round. No token for this client in current round. Waiting for server new round. No token for this client in current round. Waiting for server new round. No token for this client in current round. Waiting for server new round. No token for this client in current round. Waiting for server new round. No token for this client in current round. Waiting for server new round. pull_models completed. Status:False rank:0 2021-05-14 06:07:52,649 - FederatedClient - INFO - Starting to fetch global model. No token for this client in current round. Waiting for server new round. No token for this client in current round. Waiting for server new round. No token for this client in current round. Waiting for server new round. No token for this client in current round. Waiting for server new round. No token for this client in current round. Waiting for server new round. No token for this client in current round. Waiting for server new round. pull_models completed. Status:False rank:0 2021-05-14 06:08:22,798 - FederatedClient - INFO - Starting to fetch global model. No token for this client in current round. Waiting for server new round. No token for this client in current round. Waiting for server new round. 2021-05-14 06:08:35,945 - Communicator - INFO - Received example_project model at round 1 (16334772 Bytes). GetModel time: 3.1189322471618652 seconds Get global model for round: 1 AssignVariables... n_ext 48 FED_UPDATE_OPS fedlearn_ops Updated local variables to be assigned. 2021-05-14 06:08:35,970 - AssignVariables - INFO - Vars 48 of 48 assigned. assign_ops 48 updated_local_model 48 2021-05-14 06:08:35,990 - ClientModelManager - INFO - Setting global federated model data (4081379 elements) 2021-05-14 06:08:35,990 - ClientModelManager - INFO - Round 1: local model updated pull_models completed. Status:True rank:0 2021-05-14 06:08:36,019 - ignite.engine.engine.SupervisedEvaluator - INFO - Engine run resuming from iteration 0, epoch 3 until 4 epochs 2021-05-14 06:09:17,450 - ignite.engine.engine.SupervisedEvaluator - INFO - Got new best metric of validation mean dice: 0.49623361229896545 2021-05-14 06:09:17,450 - ignite.engine.engine.SupervisedEvaluator - INFO - Epoch[4] Metrics -- validation loss: -0.0650 validation mean dice: 0.4962 2021-05-14 06:09:17,450 - ignite.engine.engine.SupervisedEvaluator - INFO - Key metric: validation mean dice best value: 0.49623361229896545 at epoch: 4 2021-05-14 06:09:17,478 - ignite.engine.engine.SupervisedEvaluator - INFO - Epoch[4] Complete. Time taken: 00:00:41 2021-05-14 06:09:17,478 - ignite.engine.engine.SupervisedEvaluator - INFO - Engine run complete. Time taken: 00:00:41 2021-05-14 06:09:17,648 - ignite.engine.engine.SupervisedTrainer - INFO - Engine run resuming from iteration 375, epoch 3 until 6 epochs 2021-05-14 06:09:28,348 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 1/125 -- train_loss: 1.3929 2021-05-14 06:09:29,077 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 2/125 -- train_loss: 2.1981 2021-05-14 06:09:29,806 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 3/125 -- train_loss: 1.6780 2021-05-14 06:09:30,532 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 4/125 -- train_loss: 1.6284 2021-05-14 06:09:31,260 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 5/125 -- train_loss: 2.2757 2021-05-14 06:09:31,990 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 6/125 -- train_loss: 1.8597 2021-05-14 06:09:32,717 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 7/125 -- train_loss: 1.8105 2021-05-14 06:09:33,447 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 8/125 -- train_loss: 1.4315 2021-05-14 06:09:37,700 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 9/125 -- train_loss: 2.5030 2021-05-14 06:09:38,430 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 10/125 -- train_loss: 1.7371 2021-05-14 06:09:39,162 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 11/125 -- train_loss: 1.3992 2021-05-14 06:09:39,892 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 12/125 -- train_loss: 1.6398 2021-05-14 06:09:40,621 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 13/125 -- train_loss: 1.4854 2021-05-14 06:09:41,353 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 14/125 -- train_loss: 1.6382 2021-05-14 06:09:42,085 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 15/125 -- train_loss: 1.2945 2021-05-14 06:09:42,819 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 16/125 -- train_loss: 1.1339 2021-05-14 06:09:46,470 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 17/125 -- train_loss: 2.1229 2021-05-14 06:09:47,202 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 18/125 -- train_loss: 1.3856 2021-05-14 06:09:47,937 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 19/125 -- train_loss: 1.5904 2021-05-14 06:09:48,668 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 20/125 -- train_loss: 1.6790 2021-05-14 06:09:49,401 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 21/125 -- train_loss: 1.2941 2021-05-14 06:09:50,134 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 22/125 -- train_loss: 2.5755 2021-05-14 06:09:50,868 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 23/125 -- train_loss: 1.7421 2021-05-14 06:09:51,601 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 24/125 -- train_loss: 1.6079 2021-05-14 06:09:55,615 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 25/125 -- train_loss: 2.2361 2021-05-14 06:09:56,347 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 26/125 -- train_loss: 1.7467 2021-05-14 06:09:57,083 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 27/125 -- train_loss: 1.8748 2021-05-14 06:09:57,815 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 28/125 -- train_loss: 1.8224 2021-05-14 06:09:58,546 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 29/125 -- train_loss: 2.0940 2021-05-14 06:09:59,280 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 30/125 -- train_loss: 1.5468 2021-05-14 06:10:00,015 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 31/125 -- train_loss: 1.8566 2021-05-14 06:10:00,748 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 32/125 -- train_loss: 1.6820 2021-05-14 06:10:04,504 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 33/125 -- train_loss: 1.4534 2021-05-14 06:10:05,238 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 34/125 -- train_loss: 1.7661 2021-05-14 06:10:05,976 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 35/125 -- train_loss: 1.1715 2021-05-14 06:10:06,707 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 36/125 -- train_loss: 1.5509 2021-05-14 06:10:07,441 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 37/125 -- train_loss: 2.3389 2021-05-14 06:10:08,176 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 38/125 -- train_loss: 3.3859 2021-05-14 06:10:08,913 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 39/125 -- train_loss: 1.7036 2021-05-14 06:10:09,646 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 40/125 -- train_loss: 1.4904 2021-05-14 06:10:13,408 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 41/125 -- train_loss: 1.1219 2021-05-14 06:10:14,141 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 42/125 -- train_loss: 1.2471 2021-05-14 06:10:14,874 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 43/125 -- train_loss: 1.3375 2021-05-14 06:10:15,611 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 44/125 -- train_loss: 1.5424 2021-05-14 06:10:16,346 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 45/125 -- train_loss: 1.5679 2021-05-14 06:10:17,083 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 46/125 -- train_loss: 1.7923 2021-05-14 06:10:17,824 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 47/125 -- train_loss: 1.7698 2021-05-14 06:10:18,562 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 48/125 -- train_loss: 2.4612 2021-05-14 06:10:22,465 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 49/125 -- train_loss: 2.4704 2021-05-14 06:10:23,204 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 50/125 -- train_loss: 1.9853 2021-05-14 06:10:23,937 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 51/125 -- train_loss: 1.9448 2021-05-14 06:10:24,669 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 52/125 -- train_loss: 1.9385 2021-05-14 06:10:25,407 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 53/125 -- train_loss: 1.5607 2021-05-14 06:10:26,145 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 54/125 -- train_loss: 1.2941 2021-05-14 06:10:26,884 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 55/125 -- train_loss: 1.7339 2021-05-14 06:10:27,623 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 56/125 -- train_loss: 2.3123 2021-05-14 06:10:31,326 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 57/125 -- train_loss: 1.4871 2021-05-14 06:10:32,057 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 58/125 -- train_loss: 1.7451 2021-05-14 06:10:32,795 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 59/125 -- train_loss: 1.7018 2021-05-14 06:10:33,534 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 60/125 -- train_loss: 1.2792 2021-05-14 06:10:34,277 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 61/125 -- train_loss: 1.5929 2021-05-14 06:10:35,014 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 62/125 -- train_loss: 1.6241 2021-05-14 06:10:35,751 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 63/125 -- train_loss: 1.6488 2021-05-14 06:10:36,492 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 64/125 -- train_loss: 1.8395 2021-05-14 06:10:40,212 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 65/125 -- train_loss: 1.3184 2021-05-14 06:10:40,949 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 66/125 -- train_loss: 1.6899 2021-05-14 06:10:41,689 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 67/125 -- train_loss: 1.8864 2021-05-14 06:10:42,429 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 68/125 -- train_loss: 1.8980 2021-05-14 06:10:43,168 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 69/125 -- train_loss: 1.5959 2021-05-14 06:10:43,906 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 70/125 -- train_loss: 1.7683 2021-05-14 06:10:44,647 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 71/125 -- train_loss: 1.3415 2021-05-14 06:10:45,386 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 72/125 -- train_loss: 1.7755 2021-05-14 06:10:49,221 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 73/125 -- train_loss: 1.4719 2021-05-14 06:10:49,961 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 74/125 -- train_loss: 2.1772 2021-05-14 06:10:50,702 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 75/125 -- train_loss: 1.2670 2021-05-14 06:10:51,440 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 76/125 -- train_loss: 1.9639 2021-05-14 06:10:52,177 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 77/125 -- train_loss: 1.5118 2021-05-14 06:10:52,915 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 78/125 -- train_loss: 1.3761 2021-05-14 06:10:53,656 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 79/125 -- train_loss: 1.1637 2021-05-14 06:10:54,394 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 80/125 -- train_loss: 1.4339 2021-05-14 06:10:58,447 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 81/125 -- train_loss: 1.1601 2021-05-14 06:10:59,186 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 82/125 -- train_loss: 1.9042 2021-05-14 06:10:59,924 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 83/125 -- train_loss: 1.8448 2021-05-14 06:11:00,663 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 84/125 -- train_loss: 9.0427 2021-05-14 06:11:01,403 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 85/125 -- train_loss: 1.3836 2021-05-14 06:11:02,140 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 86/125 -- train_loss: 2.0346 2021-05-14 06:11:02,878 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 87/125 -- train_loss: 2.3247 2021-05-14 06:11:03,613 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 88/125 -- train_loss: 1.7707 2021-05-14 06:11:07,600 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 89/125 -- train_loss: 2.2303 2021-05-14 06:11:08,342 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 90/125 -- train_loss: 2.1605 2021-05-14 06:11:09,080 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 91/125 -- train_loss: 2.1338 2021-05-14 06:11:09,818 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 92/125 -- train_loss: 3.0227 2021-05-14 06:11:10,557 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 93/125 -- train_loss: 1.7397 2021-05-14 06:11:11,298 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 94/125 -- train_loss: 2.8848 2021-05-14 06:11:12,036 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 95/125 -- train_loss: 2.9061 2021-05-14 06:11:12,774 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 96/125 -- train_loss: 3.9762 2021-05-14 06:11:16,183 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 97/125 -- train_loss: 1.8914 2021-05-14 06:11:16,921 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 98/125 -- train_loss: 4.3929 2021-05-14 06:11:17,659 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 99/125 -- train_loss: 1.8737 2021-05-14 06:11:18,399 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 100/125 -- train_loss: 2.2314 2021-05-14 06:11:19,141 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 101/125 -- train_loss: 1.7076 2021-05-14 06:11:19,879 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 102/125 -- train_loss: 2.3260 2021-05-14 06:11:20,619 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 103/125 -- train_loss: 2.0051 2021-05-14 06:11:21,357 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 104/125 -- train_loss: 1.7787 2021-05-14 06:11:25,253 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 105/125 -- train_loss: 1.9092 2021-05-14 06:11:25,993 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 106/125 -- train_loss: 1.7287 2021-05-14 06:11:26,732 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 107/125 -- train_loss: 2.1253 2021-05-14 06:11:27,470 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 108/125 -- train_loss: 2.6205 2021-05-14 06:11:28,211 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 109/125 -- train_loss: 1.3479 2021-05-14 06:11:28,950 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 110/125 -- train_loss: 1.5618 2021-05-14 06:11:29,690 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 111/125 -- train_loss: 3.8454 2021-05-14 06:11:30,429 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 112/125 -- train_loss: 1.8290 2021-05-14 06:11:34,146 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 113/125 -- train_loss: 1.3344 2021-05-14 06:11:34,886 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 114/125 -- train_loss: 1.5483 2021-05-14 06:11:35,625 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 115/125 -- train_loss: 1.5914 2021-05-14 06:11:36,363 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 116/125 -- train_loss: 1.3855 2021-05-14 06:11:37,103 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 117/125 -- train_loss: 1.5842 2021-05-14 06:11:37,843 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 118/125 -- train_loss: 1.3380 2021-05-14 06:11:38,583 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 119/125 -- train_loss: 1.7126 2021-05-14 06:11:39,323 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 120/125 -- train_loss: 1.6177 2021-05-14 06:11:42,882 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 121/125 -- train_loss: 2.9891 2021-05-14 06:11:43,621 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 122/125 -- train_loss: 2.1715 2021-05-14 06:11:44,360 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 123/125 -- train_loss: 2.8965 2021-05-14 06:11:45,100 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 124/125 -- train_loss: 1.3546 2021-05-14 06:11:45,840 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 4/6, Iter: 125/125 -- train_loss: 1.7089 2021-05-14 06:11:45,840 - ignite.engine.engine.SupervisedEvaluator - INFO - Engine run resuming from iteration 0, epoch 3 until 4 epochs 2021-05-14 06:12:27,985 - ignite.engine.engine.SupervisedEvaluator - INFO - Epoch[4] Metrics -- validation loss: -0.0681 validation mean dice: 0.4688 2021-05-14 06:12:27,985 - ignite.engine.engine.SupervisedEvaluator - INFO - Key metric: validation mean dice best value: 0.49623361229896545 at epoch: 4 2021-05-14 06:12:27,986 - ignite.engine.engine.SupervisedEvaluator - INFO - Epoch[4] Complete. Time taken: 00:00:42 2021-05-14 06:12:27,986 - ignite.engine.engine.SupervisedEvaluator - INFO - Engine run complete. Time taken: 00:00:42 2021-05-14 06:12:28,056 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch[4] Complete. Time taken: 00:03:10 2021-05-14 06:12:39,069 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 1/125 -- train_loss: 1.4098 2021-05-14 06:12:39,806 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 2/125 -- train_loss: 1.6200 2021-05-14 06:12:40,547 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 3/125 -- train_loss: 1.2843 2021-05-14 06:12:41,288 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 4/125 -- train_loss: 1.6311 2021-05-14 06:12:42,030 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 5/125 -- train_loss: 1.6586 2021-05-14 06:12:42,769 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 6/125 -- train_loss: 2.3304 2021-05-14 06:12:43,509 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 7/125 -- train_loss: 2.0545 2021-05-14 06:12:44,251 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 8/125 -- train_loss: 1.1682 2021-05-14 06:12:47,767 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 9/125 -- train_loss: 1.8649 2021-05-14 06:12:48,507 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 10/125 -- train_loss: 2.0573 2021-05-14 06:12:49,247 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 11/125 -- train_loss: 1.7272 2021-05-14 06:12:49,988 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 12/125 -- train_loss: 2.0375 2021-05-14 06:12:50,727 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 13/125 -- train_loss: 1.3094 2021-05-14 06:12:51,467 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 14/125 -- train_loss: 2.5863 2021-05-14 06:12:52,208 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 15/125 -- train_loss: 1.6712 2021-05-14 06:12:52,953 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 16/125 -- train_loss: 1.3728 2021-05-14 06:12:56,549 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 17/125 -- train_loss: 2.3804 2021-05-14 06:12:57,287 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 18/125 -- train_loss: 1.8104 2021-05-14 06:12:58,028 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 19/125 -- train_loss: 1.7194 2021-05-14 06:12:58,767 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 20/125 -- train_loss: 2.4457 2021-05-14 06:12:59,506 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 21/125 -- train_loss: 1.6923 2021-05-14 06:13:00,247 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 22/125 -- train_loss: 1.4686 2021-05-14 06:13:00,988 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 23/125 -- train_loss: 1.5627 2021-05-14 06:13:01,727 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 24/125 -- train_loss: 2.3167 2021-05-14 06:13:05,423 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 25/125 -- train_loss: 1.6584 2021-05-14 06:13:06,186 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 26/125 -- train_loss: 1.3817 2021-05-14 06:13:06,926 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 27/125 -- train_loss: 1.4891 2021-05-14 06:13:07,666 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 28/125 -- train_loss: 1.7373 2021-05-14 06:13:08,406 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 29/125 -- train_loss: 1.7336 2021-05-14 06:13:09,147 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 30/125 -- train_loss: 1.4634 2021-05-14 06:13:09,887 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 31/125 -- train_loss: 1.6610 2021-05-14 06:13:10,626 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 32/125 -- train_loss: 3.7142 2021-05-14 06:13:14,312 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 33/125 -- train_loss: 1.8404 2021-05-14 06:13:15,074 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 34/125 -- train_loss: 1.8982 2021-05-14 06:13:15,815 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 35/125 -- train_loss: 1.6444 2021-05-14 06:13:16,555 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 36/125 -- train_loss: 1.4106 2021-05-14 06:13:17,292 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 37/125 -- train_loss: 1.5084 2021-05-14 06:13:18,033 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 38/125 -- train_loss: 1.3438 2021-05-14 06:13:18,775 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 39/125 -- train_loss: 1.6407 2021-05-14 06:13:19,514 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 40/125 -- train_loss: 1.4104 2021-05-14 06:13:23,127 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 41/125 -- train_loss: 1.7044 2021-05-14 06:13:23,887 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 42/125 -- train_loss: 1.2710 2021-05-14 06:13:24,626 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 43/125 -- train_loss: 1.1823 2021-05-14 06:13:25,366 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 44/125 -- train_loss: 1.8941 2021-05-14 06:13:26,101 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 45/125 -- train_loss: 1.6522 2021-05-14 06:13:26,843 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 46/125 -- train_loss: 1.9877 2021-05-14 06:13:27,582 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 47/125 -- train_loss: 2.2418 2021-05-14 06:13:28,324 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 48/125 -- train_loss: 2.0093 2021-05-14 06:13:31,018 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 49/125 -- train_loss: 2.1865 2021-05-14 06:13:32,755 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 50/125 -- train_loss: 2.9609 2021-05-14 06:13:33,494 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 51/125 -- train_loss: 1.9445 2021-05-14 06:13:34,235 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 52/125 -- train_loss: 1.3219 2021-05-14 06:13:34,975 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 53/125 -- train_loss: 2.2062 2021-05-14 06:13:35,717 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 54/125 -- train_loss: 1.3902 2021-05-14 06:13:36,456 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 55/125 -- train_loss: 1.9009 2021-05-14 06:13:37,197 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 56/125 -- train_loss: 1.6775 2021-05-14 06:13:39,825 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 57/125 -- train_loss: 1.4167 2021-05-14 06:13:41,963 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 58/125 -- train_loss: 1.3124 2021-05-14 06:13:42,702 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 59/125 -- train_loss: 1.3041 2021-05-14 06:13:43,441 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 60/125 -- train_loss: 1.2947 2021-05-14 06:13:44,180 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 61/125 -- train_loss: 1.2817 2021-05-14 06:13:44,919 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 62/125 -- train_loss: 1.2459 2021-05-14 06:13:45,661 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 63/125 -- train_loss: 2.1909 2021-05-14 06:13:46,400 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 64/125 -- train_loss: 1.5542 2021-05-14 06:13:48,610 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 65/125 -- train_loss: 1.6640 2021-05-14 06:13:50,407 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 66/125 -- train_loss: 1.8990 2021-05-14 06:13:51,147 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 67/125 -- train_loss: 2.1314 2021-05-14 06:13:51,887 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 68/125 -- train_loss: 1.1975 2021-05-14 06:13:52,625 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 69/125 -- train_loss: 2.3205 2021-05-14 06:13:53,364 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 70/125 -- train_loss: 1.3498 2021-05-14 06:13:54,103 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 71/125 -- train_loss: 1.1797 2021-05-14 06:13:54,842 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 72/125 -- train_loss: 1.7446 2021-05-14 06:13:57,516 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 73/125 -- train_loss: 2.0285 2021-05-14 06:13:59,367 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 74/125 -- train_loss: 1.6184 2021-05-14 06:14:00,107 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 75/125 -- train_loss: 1.0831 2021-05-14 06:14:00,846 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 76/125 -- train_loss: 1.4081 2021-05-14 06:14:01,589 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 77/125 -- train_loss: 1.5113 2021-05-14 06:14:02,329 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 78/125 -- train_loss: 1.4449 2021-05-14 06:14:03,068 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 79/125 -- train_loss: 1.3411 2021-05-14 06:14:03,806 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 80/125 -- train_loss: 1.2013 2021-05-14 06:14:05,468 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 81/125 -- train_loss: 1.1961 2021-05-14 06:14:08,595 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 82/125 -- train_loss: 2.2109 2021-05-14 06:14:09,334 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 83/125 -- train_loss: 1.9766 2021-05-14 06:14:10,073 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 84/125 -- train_loss: 1.8286 2021-05-14 06:14:10,812 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 85/125 -- train_loss: 1.5387 2021-05-14 06:14:11,550 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 86/125 -- train_loss: 2.5793 2021-05-14 06:14:12,290 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 87/125 -- train_loss: 1.3992 2021-05-14 06:14:13,029 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 88/125 -- train_loss: 1.6324 2021-05-14 06:14:13,955 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 89/125 -- train_loss: 1.2408 2021-05-14 06:14:17,056 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 90/125 -- train_loss: 1.5425 2021-05-14 06:14:17,795 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 91/125 -- train_loss: 1.4396 2021-05-14 06:14:18,533 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 92/125 -- train_loss: 1.4526 2021-05-14 06:14:19,273 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 93/125 -- train_loss: 1.2248 2021-05-14 06:14:20,010 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 94/125 -- train_loss: 1.3553 2021-05-14 06:14:20,749 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 95/125 -- train_loss: 1.6936 2021-05-14 06:14:21,488 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 96/125 -- train_loss: 1.3521 2021-05-14 06:14:22,785 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 97/125 -- train_loss: 1.1846 2021-05-14 06:14:25,861 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 98/125 -- train_loss: 1.5296 2021-05-14 06:14:26,601 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 99/125 -- train_loss: 1.4017 2021-05-14 06:14:27,338 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 100/125 -- train_loss: 1.7994 2021-05-14 06:14:28,078 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 101/125 -- train_loss: 1.4799 2021-05-14 06:14:28,815 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 102/125 -- train_loss: 1.7947 2021-05-14 06:14:29,557 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 103/125 -- train_loss: 1.6657 2021-05-14 06:14:30,295 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 104/125 -- train_loss: 1.7520 2021-05-14 06:14:31,371 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 105/125 -- train_loss: 1.9793 2021-05-14 06:14:34,907 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 106/125 -- train_loss: 1.7759 2021-05-14 06:14:35,647 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 107/125 -- train_loss: 2.3178 2021-05-14 06:14:36,388 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 108/125 -- train_loss: 1.4621 2021-05-14 06:14:37,129 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 109/125 -- train_loss: 1.5198 2021-05-14 06:14:37,868 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 110/125 -- train_loss: 1.4839 2021-05-14 06:14:38,608 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 111/125 -- train_loss: 1.5366 2021-05-14 06:14:39,348 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 112/125 -- train_loss: 1.5235 2021-05-14 06:14:40,087 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 113/125 -- train_loss: 1.4745 2021-05-14 06:14:43,465 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 114/125 -- train_loss: 1.2389 2021-05-14 06:14:44,205 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 115/125 -- train_loss: 1.4666 2021-05-14 06:14:44,944 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 116/125 -- train_loss: 1.3657 2021-05-14 06:14:45,684 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 117/125 -- train_loss: 1.2079 2021-05-14 06:14:46,423 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 118/125 -- train_loss: 1.2297 2021-05-14 06:14:47,166 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 119/125 -- train_loss: 1.1821 2021-05-14 06:14:47,903 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 120/125 -- train_loss: 1.8378 2021-05-14 06:14:48,640 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 121/125 -- train_loss: 2.4850 2021-05-14 06:14:52,266 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 122/125 -- train_loss: 1.7093 2021-05-14 06:14:53,006 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 123/125 -- train_loss: 1.5165 2021-05-14 06:14:53,746 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 124/125 -- train_loss: 1.6669 2021-05-14 06:14:54,486 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 5/6, Iter: 125/125 -- train_loss: 2.0164 2021-05-14 06:14:54,486 - ignite.engine.engine.SupervisedEvaluator - INFO - Engine run resuming from iteration 0, epoch 4 until 5 epochs 2021-05-14 06:15:36,882 - ignite.engine.engine.SupervisedEvaluator - INFO - Epoch[5] Metrics -- validation loss: -0.0602 validation mean dice: 0.4727 2021-05-14 06:15:36,882 - ignite.engine.engine.SupervisedEvaluator - INFO - Key metric: validation mean dice best value: 0.49623361229896545 at epoch: 4 2021-05-14 06:15:36,896 - ignite.engine.engine.SupervisedEvaluator - INFO - Epoch[5] Complete. Time taken: 00:00:42 2021-05-14 06:15:36,896 - ignite.engine.engine.SupervisedEvaluator - INFO - Engine run complete. Time taken: 00:00:42 2021-05-14 06:15:36,969 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch[5] Complete. Time taken: 00:03:09 2021-05-14 06:15:47,763 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 1/125 -- train_loss: 1.0804 2021-05-14 06:15:48,500 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 2/125 -- train_loss: 1.1539 2021-05-14 06:15:49,242 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 3/125 -- train_loss: 1.1053 2021-05-14 06:15:49,983 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 4/125 -- train_loss: 1.1927 2021-05-14 06:15:50,721 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 5/125 -- train_loss: 1.2048 2021-05-14 06:15:51,461 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 6/125 -- train_loss: 1.6578 2021-05-14 06:15:52,205 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 7/125 -- train_loss: 1.7086 2021-05-14 06:15:52,943 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 8/125 -- train_loss: 1.4834 2021-05-14 06:15:56,738 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 9/125 -- train_loss: 1.5646 2021-05-14 06:15:57,478 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 10/125 -- train_loss: 1.3000 2021-05-14 06:15:58,217 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 11/125 -- train_loss: 1.2955 2021-05-14 06:15:58,955 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 12/125 -- train_loss: 1.2574 2021-05-14 06:15:59,695 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 13/125 -- train_loss: 1.9957 2021-05-14 06:16:00,436 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 14/125 -- train_loss: 1.1836 2021-05-14 06:16:01,175 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 15/125 -- train_loss: 1.5322 2021-05-14 06:16:01,915 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 16/125 -- train_loss: 1.7183 2021-05-14 06:16:05,609 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 17/125 -- train_loss: 1.6510 2021-05-14 06:16:06,348 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 18/125 -- train_loss: 1.6357 2021-05-14 06:16:07,088 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 19/125 -- train_loss: 1.5727 2021-05-14 06:16:07,826 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 20/125 -- train_loss: 1.7695 2021-05-14 06:16:08,565 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 21/125 -- train_loss: 1.4694 2021-05-14 06:16:09,304 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 22/125 -- train_loss: 2.3199 2021-05-14 06:16:10,043 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 23/125 -- train_loss: 1.9738 2021-05-14 06:16:10,783 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 24/125 -- train_loss: 1.8328 2021-05-14 06:16:14,458 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 25/125 -- train_loss: 1.1521 2021-05-14 06:16:15,195 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 26/125 -- train_loss: 1.4427 2021-05-14 06:16:15,935 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 27/125 -- train_loss: 1.3211 2021-05-14 06:16:16,674 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 28/125 -- train_loss: 1.4414 2021-05-14 06:16:17,413 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 29/125 -- train_loss: 1.1693 2021-05-14 06:16:18,154 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 30/125 -- train_loss: 1.4407 2021-05-14 06:16:18,894 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 31/125 -- train_loss: 1.5859 2021-05-14 06:16:19,635 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 32/125 -- train_loss: 1.1588 2021-05-14 06:16:22,863 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 33/125 -- train_loss: 1.6452 2021-05-14 06:16:23,603 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 34/125 -- train_loss: 2.1397 2021-05-14 06:16:24,344 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 35/125 -- train_loss: 1.1448 2021-05-14 06:16:25,083 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 36/125 -- train_loss: 1.7776 2021-05-14 06:16:25,822 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 37/125 -- train_loss: 1.2800 2021-05-14 06:16:26,564 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 38/125 -- train_loss: 1.6042 2021-05-14 06:16:27,302 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 39/125 -- train_loss: 1.8300 2021-05-14 06:16:28,040 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 40/125 -- train_loss: 2.1232 2021-05-14 06:16:30,708 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 41/125 -- train_loss: 1.5609 2021-05-14 06:16:31,447 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 42/125 -- train_loss: 1.3963 2021-05-14 06:16:32,186 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 43/125 -- train_loss: 1.5288 2021-05-14 06:16:32,927 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 44/125 -- train_loss: 1.5342 2021-05-14 06:16:33,670 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 45/125 -- train_loss: 1.6242 2021-05-14 06:16:34,410 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 46/125 -- train_loss: 1.2375 2021-05-14 06:16:35,149 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 47/125 -- train_loss: 1.1416 2021-05-14 06:16:35,888 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 48/125 -- train_loss: 1.9072 2021-05-14 06:16:39,521 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 49/125 -- train_loss: 1.0241 2021-05-14 06:16:40,260 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 50/125 -- train_loss: 1.6302 2021-05-14 06:16:41,001 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 51/125 -- train_loss: 1.5746 2021-05-14 06:16:41,741 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 52/125 -- train_loss: 1.5314 2021-05-14 06:16:42,479 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 53/125 -- train_loss: 2.3764 2021-05-14 06:16:43,222 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 54/125 -- train_loss: 2.0558 2021-05-14 06:16:43,960 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 55/125 -- train_loss: 1.4855 2021-05-14 06:16:44,698 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 56/125 -- train_loss: 2.1330 2021-05-14 06:16:48,252 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 57/125 -- train_loss: 1.2162 2021-05-14 06:16:48,989 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 58/125 -- train_loss: 1.6210 2021-05-14 06:16:49,727 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 59/125 -- train_loss: 1.7108 2021-05-14 06:16:50,466 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 60/125 -- train_loss: 1.1718 2021-05-14 06:16:51,205 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 61/125 -- train_loss: 1.2307 2021-05-14 06:16:51,943 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 62/125 -- train_loss: 1.1597 2021-05-14 06:16:52,682 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 63/125 -- train_loss: 2.0282 2021-05-14 06:16:53,424 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 64/125 -- train_loss: 1.2553 2021-05-14 06:16:56,924 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 65/125 -- train_loss: 1.2752 2021-05-14 06:16:57,661 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 66/125 -- train_loss: 1.3198 2021-05-14 06:16:58,399 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 67/125 -- train_loss: 1.4593 2021-05-14 06:16:59,141 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 68/125 -- train_loss: 1.0044 2021-05-14 06:16:59,882 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 69/125 -- train_loss: 1.2523 2021-05-14 06:17:00,622 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 70/125 -- train_loss: 2.5379 2021-05-14 06:17:01,362 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 71/125 -- train_loss: 1.3041 2021-05-14 06:17:02,102 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 72/125 -- train_loss: 2.0405 2021-05-14 06:17:05,788 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 73/125 -- train_loss: 1.1233 2021-05-14 06:17:06,527 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 74/125 -- train_loss: 1.8701 2021-05-14 06:17:07,266 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 75/125 -- train_loss: 1.6390 2021-05-14 06:17:08,003 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 76/125 -- train_loss: 1.5084 2021-05-14 06:17:08,743 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 77/125 -- train_loss: 1.5258 2021-05-14 06:17:09,485 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 78/125 -- train_loss: 1.3638 2021-05-14 06:17:10,224 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 79/125 -- train_loss: 1.2837 2021-05-14 06:17:10,964 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 80/125 -- train_loss: 1.9661 2021-05-14 06:17:14,637 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 81/125 -- train_loss: 1.3095 2021-05-14 06:17:15,376 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 82/125 -- train_loss: 1.2292 2021-05-14 06:17:16,119 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 83/125 -- train_loss: 1.6361 2021-05-14 06:17:16,856 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 84/125 -- train_loss: 1.6185 2021-05-14 06:17:17,594 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 85/125 -- train_loss: 1.2591 2021-05-14 06:17:18,332 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 86/125 -- train_loss: 1.4179 2021-05-14 06:17:19,072 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 87/125 -- train_loss: 1.2252 2021-05-14 06:17:19,814 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 88/125 -- train_loss: 2.0574 2021-05-14 06:17:23,376 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 89/125 -- train_loss: 1.8313 2021-05-14 06:17:24,117 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 90/125 -- train_loss: 1.8167 2021-05-14 06:17:24,857 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 91/125 -- train_loss: 1.5140 2021-05-14 06:17:25,597 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 92/125 -- train_loss: 1.7978 2021-05-14 06:17:26,340 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 93/125 -- train_loss: 2.0896 2021-05-14 06:17:27,079 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 94/125 -- train_loss: 1.7786 2021-05-14 06:17:27,818 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 95/125 -- train_loss: 1.1433 2021-05-14 06:17:28,559 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 96/125 -- train_loss: 1.5565 2021-05-14 06:17:32,304 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 97/125 -- train_loss: 1.0033 2021-05-14 06:17:33,042 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 98/125 -- train_loss: 1.6297 2021-05-14 06:17:33,781 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 99/125 -- train_loss: 1.3318 2021-05-14 06:17:34,521 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 100/125 -- train_loss: 1.1627 2021-05-14 06:17:35,261 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 101/125 -- train_loss: 1.1982 2021-05-14 06:17:36,002 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 102/125 -- train_loss: 1.1898 2021-05-14 06:17:36,742 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 103/125 -- train_loss: 1.3103 2021-05-14 06:17:37,483 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 104/125 -- train_loss: 1.9583 2021-05-14 06:17:41,382 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 105/125 -- train_loss: 1.4391 2021-05-14 06:17:42,120 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 106/125 -- train_loss: 1.0353 2021-05-14 06:17:42,858 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 107/125 -- train_loss: 1.4308 2021-05-14 06:17:43,599 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 108/125 -- train_loss: 1.8397 2021-05-14 06:17:44,341 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 109/125 -- train_loss: 1.5701 2021-05-14 06:17:45,079 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 110/125 -- train_loss: 1.7426 2021-05-14 06:17:45,819 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 111/125 -- train_loss: 2.9114 2021-05-14 06:17:46,559 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 112/125 -- train_loss: 1.6068 2021-05-14 06:17:50,247 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 113/125 -- train_loss: 1.5562 2021-05-14 06:17:50,984 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 114/125 -- train_loss: 1.4688 2021-05-14 06:17:51,722 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 115/125 -- train_loss: 1.6377 2021-05-14 06:17:52,464 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 116/125 -- train_loss: 1.2808 2021-05-14 06:17:53,202 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 117/125 -- train_loss: 1.1531 2021-05-14 06:17:53,941 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 118/125 -- train_loss: 1.5395 2021-05-14 06:17:54,680 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 119/125 -- train_loss: 1.1439 2021-05-14 06:17:55,421 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 120/125 -- train_loss: 1.4496 2021-05-14 06:17:59,148 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 121/125 -- train_loss: 1.7318 2021-05-14 06:17:59,886 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 122/125 -- train_loss: 1.7841 2021-05-14 06:18:00,627 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 123/125 -- train_loss: 1.3911 2021-05-14 06:18:01,365 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 124/125 -- train_loss: 1.1601 2021-05-14 06:18:02,103 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 6/6, Iter: 125/125 -- train_loss: 1.7006 2021-05-14 06:18:02,103 - ignite.engine.engine.SupervisedEvaluator - INFO - Engine run resuming from iteration 0, epoch 5 until 6 epochs 2021-05-14 06:18:44,452 - ignite.engine.engine.SupervisedEvaluator - INFO - Epoch[6] Metrics -- validation loss: -0.0625 validation mean dice: 0.3240 2021-05-14 06:18:44,453 - ignite.engine.engine.SupervisedEvaluator - INFO - Key metric: validation mean dice best value: 0.49623361229896545 at epoch: 4 2021-05-14 06:18:44,454 - ignite.engine.engine.SupervisedEvaluator - INFO - Epoch[6] Complete. Time taken: 00:00:42 2021-05-14 06:18:44,454 - ignite.engine.engine.SupervisedEvaluator - INFO - Engine run complete. Time taken: 00:00:42 2021-05-14 06:18:44,521 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch[6] Complete. Time taken: 00:03:08 2021-05-14 06:18:44,526 - ignite.engine.engine.SupervisedTrainer - INFO - Deleted previous saved final checkpoint: checkpoint_final_iteration=375.pt 2021-05-14 06:18:44,600 - ignite.engine.engine.SupervisedTrainer - INFO - Train completed, saved final checkpoint: checkpoint_final_iteration=750.pt 2021-05-14 06:18:44,600 - ignite.engine.engine.SupervisedTrainer - INFO - Engine run complete. Time taken: 00:09:27 Send model to server. 2021-05-14 06:18:44,821 - FederatedClient - INFO - Starting to push model. setup local_model_dict local_model_dict 48 2021-05-14 06:18:44,877 - Communicator - INFO - Send example_project at round 1 2021-05-14 06:18:47,365 - Communicator - INFO - Received comments: example_project Received round 1 from org1-a (16334853 Bytes, 615 seconds). SubmitUpdate time: 2.4875905513763428 seconds 2021-05-14 06:18:47,416 - FederatedClient - INFO - Starting to fetch global model. No token for this client in current round. Waiting for server new round. No token for this client in current round. Waiting for server new round. No token for this client in current round. Waiting for server new round. No token for this client in current round. Waiting for server new round. No token for this client in current round. Waiting for server new round. No token for this client in current round. Waiting for server new round. pull_models completed. Status:False rank:0 2021-05-14 06:19:17,564 - FederatedClient - INFO - Starting to fetch global model. No token for this client in current round. Waiting for server new round. No token for this client in current round. Waiting for server new round. No token for this client in current round. Waiting for server new round. No token for this client in current round. Waiting for server new round. No token for this client in current round. Waiting for server new round. No token for this client in current round. Waiting for server new round. pull_models completed. Status:False rank:0 2021-05-14 06:19:47,712 - FederatedClient - INFO - Starting to fetch global model. No token for this client in current round. Waiting for server new round. No token for this client in current round. Waiting for server new round. No token for this client in current round. Waiting for server new round. No token for this client in current round. Waiting for server new round. No token for this client in current round. Waiting for server new round. No token for this client in current round. Waiting for server new round. pull_models completed. Status:False rank:0 2021-05-14 06:20:17,859 - FederatedClient - INFO - Starting to fetch global model. No token for this client in current round. Waiting for server new round. No token for this client in current round. Waiting for server new round. No token for this client in current round. Waiting for server new round. No token for this client in current round. Waiting for server new round. No token for this client in current round. Waiting for server new round. No token for this client in current round. Waiting for server new round. pull_models completed. Status:False rank:0 2021-05-14 06:20:48,005 - FederatedClient - INFO - Starting to fetch global model. No token for this client in current round. Waiting for server new round. No token for this client in current round. Waiting for server new round. No token for this client in current round. Waiting for server new round. No token for this client in current round. Waiting for server new round. No token for this client in current round. Waiting for server new round. No token for this client in current round. Waiting for server new round. pull_models completed. Status:False rank:0 2021-05-14 06:21:18,150 - FederatedClient - INFO - Starting to fetch global model. No token for this client in current round. Waiting for server new round. No token for this client in current round. Waiting for server new round. No token for this client in current round. Waiting for server new round. No token for this client in current round. Waiting for server new round. No token for this client in current round. Waiting for server new round. No token for this client in current round. Waiting for server new round. pull_models completed. Status:False rank:0 2021-05-14 06:21:48,295 - FederatedClient - INFO - Starting to fetch global model. No token for this client in current round. Waiting for server new round. No token for this client in current round. Waiting for server new round. No token for this client in current round. Waiting for server new round. No token for this client in current round. Waiting for server new round. No token for this client in current round. Waiting for server new round. No token for this client in current round. Waiting for server new round. pull_models completed. Status:False rank:0 2021-05-14 06:22:18,440 - FederatedClient - INFO - Starting to fetch global model. No token for this client in current round. Waiting for server new round. No token for this client in current round. Waiting for server new round. No token for this client in current round. Waiting for server new round. No token for this client in current round. Waiting for server new round. No token for this client in current round. Waiting for server new round. No token for this client in current round. Waiting for server new round. pull_models completed. Status:False rank:0 2021-05-14 06:22:48,586 - FederatedClient - INFO - Starting to fetch global model. No token for this client in current round. Waiting for server new round. No token for this client in current round. Waiting for server new round. No token for this client in current round. Waiting for server new round. No token for this client in current round. Waiting for server new round. 2021-05-14 06:23:11,056 - Communicator - INFO - Received example_project model at round 2 (16334773 Bytes). GetModel time: 2.3260459899902344 seconds Get global model for round: 2 AssignVariables... n_ext 48 FED_UPDATE_OPS fedlearn_ops Updated local variables to be assigned. 2021-05-14 06:23:11,073 - AssignVariables - INFO - Vars 48 of 48 assigned. assign_ops 48 updated_local_model 48 2021-05-14 06:23:11,103 - ClientModelManager - INFO - Setting global federated model data (4081379 elements) 2021-05-14 06:23:11,103 - ClientModelManager - INFO - Round 2: local model updated pull_models completed. Status:True rank:0 2021-05-14 06:23:11,119 - ignite.engine.engine.SupervisedEvaluator - INFO - Engine run resuming from iteration 0, epoch 6 until 7 epochs 2021-05-14 06:23:51,809 - ignite.engine.engine.SupervisedEvaluator - INFO - Epoch[7] Metrics -- validation loss: -0.0592 validation mean dice: 0.3624 2021-05-14 06:23:51,810 - ignite.engine.engine.SupervisedEvaluator - INFO - Key metric: validation mean dice best value: 0.49623361229896545 at epoch: 4 2021-05-14 06:23:51,811 - ignite.engine.engine.SupervisedEvaluator - INFO - Epoch[7] Complete. Time taken: 00:00:40 2021-05-14 06:23:51,811 - ignite.engine.engine.SupervisedEvaluator - INFO - Engine run complete. Time taken: 00:00:41 2021-05-14 06:23:51,970 - ignite.engine.engine.SupervisedTrainer - INFO - Engine run resuming from iteration 750, epoch 6 until 9 epochs 2021-05-14 06:24:02,401 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 1/125 -- train_loss: 1.4497 2021-05-14 06:24:03,127 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 2/125 -- train_loss: 1.5347 2021-05-14 06:24:03,853 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 3/125 -- train_loss: 1.4133 2021-05-14 06:24:04,583 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 4/125 -- train_loss: 1.5753 2021-05-14 06:24:05,314 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 5/125 -- train_loss: 1.5724 2021-05-14 06:24:06,042 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 6/125 -- train_loss: 1.3722 2021-05-14 06:24:06,769 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 7/125 -- train_loss: 1.7859 2021-05-14 06:24:07,497 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 8/125 -- train_loss: 1.6047 2021-05-14 06:24:11,272 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 9/125 -- train_loss: 2.4732 2021-05-14 06:24:12,003 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 10/125 -- train_loss: 1.8478 2021-05-14 06:24:12,732 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 11/125 -- train_loss: 1.3495 2021-05-14 06:24:13,460 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 12/125 -- train_loss: 1.5199 2021-05-14 06:24:14,191 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 13/125 -- train_loss: 2.0179 2021-05-14 06:24:14,923 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 14/125 -- train_loss: 1.0654 2021-05-14 06:24:15,652 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 15/125 -- train_loss: 1.3736 2021-05-14 06:24:16,381 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 16/125 -- train_loss: 1.3480 2021-05-14 06:24:20,155 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 17/125 -- train_loss: 1.6507 2021-05-14 06:24:20,885 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 18/125 -- train_loss: 1.0983 2021-05-14 06:24:21,616 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 19/125 -- train_loss: 1.8693 2021-05-14 06:24:22,347 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 20/125 -- train_loss: 1.5797 2021-05-14 06:24:23,076 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 21/125 -- train_loss: 1.5117 2021-05-14 06:24:23,806 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 22/125 -- train_loss: 1.4960 2021-05-14 06:24:24,540 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 23/125 -- train_loss: 2.1377 2021-05-14 06:24:25,269 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 24/125 -- train_loss: 3.1251 2021-05-14 06:24:28,953 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 25/125 -- train_loss: 1.7619 2021-05-14 06:24:29,682 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 26/125 -- train_loss: 1.2656 2021-05-14 06:24:30,412 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 27/125 -- train_loss: 1.7676 2021-05-14 06:24:31,144 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 28/125 -- train_loss: 1.4161 2021-05-14 06:24:31,871 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 29/125 -- train_loss: 1.4299 2021-05-14 06:24:32,604 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 30/125 -- train_loss: 1.8426 2021-05-14 06:24:33,337 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 31/125 -- train_loss: 1.4513 2021-05-14 06:24:34,069 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 32/125 -- train_loss: 1.4924 2021-05-14 06:24:36,639 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 33/125 -- train_loss: 1.4791 2021-05-14 06:24:37,496 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 34/125 -- train_loss: 2.1516 2021-05-14 06:24:38,225 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 35/125 -- train_loss: 1.1637 2021-05-14 06:24:38,959 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 36/125 -- train_loss: 1.3605 2021-05-14 06:24:39,696 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 37/125 -- train_loss: 1.1520 2021-05-14 06:24:40,433 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 38/125 -- train_loss: 1.1261 2021-05-14 06:24:41,168 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 39/125 -- train_loss: 1.2439 2021-05-14 06:24:41,902 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 40/125 -- train_loss: 1.5318 2021-05-14 06:24:44,768 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 41/125 -- train_loss: 2.4667 2021-05-14 06:24:46,635 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 42/125 -- train_loss: 1.8497 2021-05-14 06:24:47,367 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 43/125 -- train_loss: 1.4415 2021-05-14 06:24:48,102 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 44/125 -- train_loss: 1.5159 2021-05-14 06:24:48,838 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 45/125 -- train_loss: 1.5360 2021-05-14 06:24:49,573 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 46/125 -- train_loss: 1.5305 2021-05-14 06:24:50,308 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 47/125 -- train_loss: 1.3320 2021-05-14 06:24:51,046 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 48/125 -- train_loss: 1.1385 2021-05-14 06:24:53,857 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 49/125 -- train_loss: 1.3063 2021-05-14 06:24:54,980 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 50/125 -- train_loss: 0.9598 2021-05-14 06:24:55,716 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 51/125 -- train_loss: 2.1871 2021-05-14 06:24:56,451 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 52/125 -- train_loss: 1.4300 2021-05-14 06:24:57,187 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 53/125 -- train_loss: 1.7360 2021-05-14 06:24:57,923 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 54/125 -- train_loss: 1.1491 2021-05-14 06:24:58,663 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 55/125 -- train_loss: 1.8727 2021-05-14 06:24:59,397 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 56/125 -- train_loss: 1.5299 2021-05-14 06:25:02,509 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 57/125 -- train_loss: 1.4782 2021-05-14 06:25:03,648 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 58/125 -- train_loss: 1.1736 2021-05-14 06:25:04,383 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 59/125 -- train_loss: 2.0409 2021-05-14 06:25:05,121 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 60/125 -- train_loss: 1.5822 2021-05-14 06:25:05,857 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 61/125 -- train_loss: 1.1724 2021-05-14 06:25:06,590 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 62/125 -- train_loss: 3.5540 2021-05-14 06:25:07,325 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 63/125 -- train_loss: 1.4899 2021-05-14 06:25:08,062 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 64/125 -- train_loss: 1.5696 2021-05-14 06:25:11,176 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 65/125 -- train_loss: 1.9163 2021-05-14 06:25:12,692 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 66/125 -- train_loss: 1.1305 2021-05-14 06:25:13,430 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 67/125 -- train_loss: 1.6643 2021-05-14 06:25:14,166 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 68/125 -- train_loss: 1.2286 2021-05-14 06:25:14,903 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 69/125 -- train_loss: 1.3483 2021-05-14 06:25:15,639 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 70/125 -- train_loss: 1.2666 2021-05-14 06:25:16,376 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 71/125 -- train_loss: 1.4964 2021-05-14 06:25:17,114 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 72/125 -- train_loss: 1.4609 2021-05-14 06:25:19,899 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 73/125 -- train_loss: 1.7358 2021-05-14 06:25:21,252 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 74/125 -- train_loss: 2.3234 2021-05-14 06:25:21,986 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 75/125 -- train_loss: 1.4619 2021-05-14 06:25:22,719 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 76/125 -- train_loss: 1.3028 2021-05-14 06:25:23,458 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 77/125 -- train_loss: 1.5415 2021-05-14 06:25:24,198 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 78/125 -- train_loss: 1.1611 2021-05-14 06:25:24,938 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 79/125 -- train_loss: 1.2341 2021-05-14 06:25:25,678 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 80/125 -- train_loss: 1.2422 2021-05-14 06:25:27,707 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 81/125 -- train_loss: 1.1055 2021-05-14 06:25:29,713 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 82/125 -- train_loss: 1.7575 2021-05-14 06:25:30,454 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 83/125 -- train_loss: 1.7703 2021-05-14 06:25:31,195 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 84/125 -- train_loss: 1.8635 2021-05-14 06:25:31,931 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 85/125 -- train_loss: 1.7487 2021-05-14 06:25:32,668 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 86/125 -- train_loss: 2.1522 2021-05-14 06:25:33,406 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 87/125 -- train_loss: 1.6910 2021-05-14 06:25:34,149 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 88/125 -- train_loss: 1.1873 2021-05-14 06:25:36,665 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 89/125 -- train_loss: 1.5112 2021-05-14 06:25:38,351 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 90/125 -- train_loss: 1.7906 2021-05-14 06:25:39,090 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 91/125 -- train_loss: 1.6449 2021-05-14 06:25:39,830 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 92/125 -- train_loss: 1.9028 2021-05-14 06:25:40,567 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 93/125 -- train_loss: 1.8070 2021-05-14 06:25:41,307 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 94/125 -- train_loss: 1.6865 2021-05-14 06:25:42,047 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 95/125 -- train_loss: 1.8357 2021-05-14 06:25:42,788 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 96/125 -- train_loss: 1.1155 2021-05-14 06:25:45,532 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 97/125 -- train_loss: 1.7005 2021-05-14 06:25:46,997 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 98/125 -- train_loss: 1.8215 2021-05-14 06:25:47,736 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 99/125 -- train_loss: 1.2801 2021-05-14 06:25:48,473 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 100/125 -- train_loss: 1.5585 2021-05-14 06:25:49,214 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 101/125 -- train_loss: 1.5077 2021-05-14 06:25:49,954 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 102/125 -- train_loss: 1.3794 2021-05-14 06:25:50,692 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 103/125 -- train_loss: 1.4161 2021-05-14 06:25:51,435 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 104/125 -- train_loss: 1.7417 2021-05-14 06:25:54,011 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 105/125 -- train_loss: 1.2553 2021-05-14 06:25:55,997 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 106/125 -- train_loss: 2.2363 2021-05-14 06:25:56,739 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 107/125 -- train_loss: 1.2846 2021-05-14 06:25:57,477 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 108/125 -- train_loss: 1.4344 2021-05-14 06:25:58,216 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 109/125 -- train_loss: 1.2470 2021-05-14 06:25:58,956 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 110/125 -- train_loss: 1.1755 2021-05-14 06:25:59,700 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 111/125 -- train_loss: 1.4731 2021-05-14 06:26:00,437 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 112/125 -- train_loss: 1.3070 2021-05-14 06:26:03,031 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 113/125 -- train_loss: 1.3840 2021-05-14 06:26:04,760 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 114/125 -- train_loss: 1.7365 2021-05-14 06:26:05,496 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 115/125 -- train_loss: 1.1522 2021-05-14 06:26:06,234 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 116/125 -- train_loss: 1.9740 2021-05-14 06:26:06,975 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 117/125 -- train_loss: 1.7238 2021-05-14 06:26:07,714 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 118/125 -- train_loss: 1.4649 2021-05-14 06:26:08,453 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 119/125 -- train_loss: 1.1013 2021-05-14 06:26:09,194 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 120/125 -- train_loss: 1.5434 2021-05-14 06:26:11,062 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 121/125 -- train_loss: 1.0004 2021-05-14 06:26:13,163 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 122/125 -- train_loss: 1.5796 2021-05-14 06:26:13,904 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 123/125 -- train_loss: 1.7701 2021-05-14 06:26:14,641 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 124/125 -- train_loss: 1.6282 2021-05-14 06:26:15,399 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 7/9, Iter: 125/125 -- train_loss: 1.4684 2021-05-14 06:26:15,400 - ignite.engine.engine.SupervisedEvaluator - INFO - Engine run resuming from iteration 0, epoch 6 until 7 epochs 2021-05-14 06:26:57,263 - ignite.engine.engine.SupervisedEvaluator - INFO - Epoch[7] Metrics -- validation loss: -0.0635 validation mean dice: 0.0240 2021-05-14 06:26:57,263 - ignite.engine.engine.SupervisedEvaluator - INFO - Key metric: validation mean dice best value: 0.49623361229896545 at epoch: 4 2021-05-14 06:26:57,264 - ignite.engine.engine.SupervisedEvaluator - INFO - Epoch[7] Complete. Time taken: 00:00:41 2021-05-14 06:26:57,264 - ignite.engine.engine.SupervisedEvaluator - INFO - Engine run complete. Time taken: 00:00:42 2021-05-14 06:26:57,332 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch[7] Complete. Time taken: 00:03:05 2021-05-14 06:27:07,916 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 1/125 -- train_loss: 1.4081 2021-05-14 06:27:08,655 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 2/125 -- train_loss: 1.1447 2021-05-14 06:27:09,391 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 3/125 -- train_loss: 1.2269 2021-05-14 06:27:10,131 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 4/125 -- train_loss: 2.3954 2021-05-14 06:27:10,874 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 5/125 -- train_loss: 1.1570 2021-05-14 06:27:11,613 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 6/125 -- train_loss: 1.4895 2021-05-14 06:27:12,354 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 7/125 -- train_loss: 1.5013 2021-05-14 06:27:13,095 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 8/125 -- train_loss: 1.3507 2021-05-14 06:27:16,223 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 9/125 -- train_loss: 1.1261 2021-05-14 06:27:16,961 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 10/125 -- train_loss: 1.2367 2021-05-14 06:27:17,698 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 11/125 -- train_loss: 1.6090 2021-05-14 06:27:18,436 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 12/125 -- train_loss: 1.1775 2021-05-14 06:27:19,176 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 13/125 -- train_loss: 1.0154 2021-05-14 06:27:19,921 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 14/125 -- train_loss: 1.1384 2021-05-14 06:27:20,660 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 15/125 -- train_loss: 1.2098 2021-05-14 06:27:21,399 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 16/125 -- train_loss: 1.2492 2021-05-14 06:27:24,685 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 17/125 -- train_loss: 2.0280 2021-05-14 06:27:25,422 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 18/125 -- train_loss: 2.3418 2021-05-14 06:27:26,286 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 19/125 -- train_loss: 1.4952 2021-05-14 06:27:27,026 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 20/125 -- train_loss: 1.5176 2021-05-14 06:27:27,767 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 21/125 -- train_loss: 1.6295 2021-05-14 06:27:28,510 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 22/125 -- train_loss: 1.2591 2021-05-14 06:27:29,248 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 23/125 -- train_loss: 2.1084 2021-05-14 06:27:29,990 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 24/125 -- train_loss: 1.2524 2021-05-14 06:27:32,712 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 25/125 -- train_loss: 1.3493 2021-05-14 06:27:33,455 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 26/125 -- train_loss: 1.4546 2021-05-14 06:27:35,175 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 27/125 -- train_loss: 1.5131 2021-05-14 06:27:35,915 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 28/125 -- train_loss: 1.7056 2021-05-14 06:27:36,655 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 29/125 -- train_loss: 2.1842 2021-05-14 06:27:37,395 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 30/125 -- train_loss: 1.3304 2021-05-14 06:27:38,135 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 31/125 -- train_loss: 1.0571 2021-05-14 06:27:38,875 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 32/125 -- train_loss: 1.6293 2021-05-14 06:27:40,657 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 33/125 -- train_loss: 1.6034 2021-05-14 06:27:41,749 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 34/125 -- train_loss: 1.8180 2021-05-14 06:27:44,229 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 35/125 -- train_loss: 1.5367 2021-05-14 06:27:44,971 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 36/125 -- train_loss: 1.5506 2021-05-14 06:27:45,711 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 37/125 -- train_loss: 1.7240 2021-05-14 06:27:46,449 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 38/125 -- train_loss: 1.6048 2021-05-14 06:27:47,188 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 39/125 -- train_loss: 1.2822 2021-05-14 06:27:47,929 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 40/125 -- train_loss: 1.1563 2021-05-14 06:27:49,139 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 41/125 -- train_loss: 1.3494 2021-05-14 06:27:50,738 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 42/125 -- train_loss: 1.4608 2021-05-14 06:27:52,969 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 43/125 -- train_loss: 1.5000 2021-05-14 06:27:53,711 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 44/125 -- train_loss: 1.7649 2021-05-14 06:27:54,448 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 45/125 -- train_loss: 1.7269 2021-05-14 06:27:55,188 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 46/125 -- train_loss: 1.1771 2021-05-14 06:27:55,928 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 47/125 -- train_loss: 1.3151 2021-05-14 06:27:56,667 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 48/125 -- train_loss: 1.6057 2021-05-14 06:27:57,726 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 49/125 -- train_loss: 1.4829 2021-05-14 06:27:59,622 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 50/125 -- train_loss: 1.3374 2021-05-14 06:28:02,083 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 51/125 -- train_loss: 1.0205 2021-05-14 06:28:02,818 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 52/125 -- train_loss: 1.3522 2021-05-14 06:28:03,554 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 53/125 -- train_loss: 1.2792 2021-05-14 06:28:04,292 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 54/125 -- train_loss: 0.9755 2021-05-14 06:28:05,029 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 55/125 -- train_loss: 1.2349 2021-05-14 06:28:05,773 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 56/125 -- train_loss: 1.4513 2021-05-14 06:28:06,533 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 57/125 -- train_loss: 2.2045 2021-05-14 06:28:08,626 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 58/125 -- train_loss: 1.4850 2021-05-14 06:28:10,891 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 59/125 -- train_loss: 1.2094 2021-05-14 06:28:11,630 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 60/125 -- train_loss: 2.0653 2021-05-14 06:28:12,368 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 61/125 -- train_loss: 1.6322 2021-05-14 06:28:13,105 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 62/125 -- train_loss: 1.2182 2021-05-14 06:28:13,843 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 63/125 -- train_loss: 1.6462 2021-05-14 06:28:14,581 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 64/125 -- train_loss: 1.4659 2021-05-14 06:28:15,323 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 65/125 -- train_loss: 1.1212 2021-05-14 06:28:17,480 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 66/125 -- train_loss: 1.6095 2021-05-14 06:28:19,493 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 67/125 -- train_loss: 2.0156 2021-05-14 06:28:20,231 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 68/125 -- train_loss: 1.5925 2021-05-14 06:28:20,969 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 69/125 -- train_loss: 1.3837 2021-05-14 06:28:21,708 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 70/125 -- train_loss: 1.5592 2021-05-14 06:28:22,444 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 71/125 -- train_loss: 1.1197 2021-05-14 06:28:23,181 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 72/125 -- train_loss: 2.0411 2021-05-14 06:28:23,922 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 73/125 -- train_loss: 1.5637 2021-05-14 06:28:26,341 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 74/125 -- train_loss: 1.3964 2021-05-14 06:28:27,962 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 75/125 -- train_loss: 1.4394 2021-05-14 06:28:28,724 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 76/125 -- train_loss: 0.9626 2021-05-14 06:28:29,463 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 77/125 -- train_loss: 1.0285 2021-05-14 06:28:30,199 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 78/125 -- train_loss: 1.1244 2021-05-14 06:28:30,940 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 79/125 -- train_loss: 1.6714 2021-05-14 06:28:31,680 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 80/125 -- train_loss: 1.9357 2021-05-14 06:28:32,418 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 81/125 -- train_loss: 1.3604 2021-05-14 06:28:35,172 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 82/125 -- train_loss: 1.6717 2021-05-14 06:28:36,204 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 83/125 -- train_loss: 1.5091 2021-05-14 06:28:37,209 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 84/125 -- train_loss: 1.6322 2021-05-14 06:28:37,946 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 85/125 -- train_loss: 1.6235 2021-05-14 06:28:38,682 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 86/125 -- train_loss: 1.6317 2021-05-14 06:28:39,423 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 87/125 -- train_loss: 1.6416 2021-05-14 06:28:40,163 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 88/125 -- train_loss: 1.7532 2021-05-14 06:28:40,905 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 89/125 -- train_loss: 1.5657 2021-05-14 06:28:43,989 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 90/125 -- train_loss: 1.0549 2021-05-14 06:28:44,751 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 91/125 -- train_loss: 1.4584 2021-05-14 06:28:45,841 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 92/125 -- train_loss: 1.3196 2021-05-14 06:28:46,582 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 93/125 -- train_loss: 1.3820 2021-05-14 06:28:47,319 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 94/125 -- train_loss: 1.5778 2021-05-14 06:28:48,058 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 95/125 -- train_loss: 1.4915 2021-05-14 06:28:48,799 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 96/125 -- train_loss: 1.2905 2021-05-14 06:28:49,541 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 97/125 -- train_loss: 1.0277 2021-05-14 06:28:52,469 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 98/125 -- train_loss: 1.4745 2021-05-14 06:28:53,228 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 99/125 -- train_loss: 2.6366 2021-05-14 06:28:54,066 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 100/125 -- train_loss: 1.3655 2021-05-14 06:28:54,804 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 101/125 -- train_loss: 1.1133 2021-05-14 06:28:55,541 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 102/125 -- train_loss: 1.5729 2021-05-14 06:28:56,278 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 103/125 -- train_loss: 1.4109 2021-05-14 06:28:57,015 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 104/125 -- train_loss: 1.3155 2021-05-14 06:28:57,756 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 105/125 -- train_loss: 1.3875 2021-05-14 06:29:00,538 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 106/125 -- train_loss: 1.2151 2021-05-14 06:29:01,868 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 107/125 -- train_loss: 1.5642 2021-05-14 06:29:02,881 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 108/125 -- train_loss: 1.5375 2021-05-14 06:29:03,623 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 109/125 -- train_loss: 1.2899 2021-05-14 06:29:04,361 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 110/125 -- train_loss: 1.8167 2021-05-14 06:29:05,099 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 111/125 -- train_loss: 2.1798 2021-05-14 06:29:05,838 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 112/125 -- train_loss: 1.2258 2021-05-14 06:29:06,576 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 113/125 -- train_loss: 1.2075 2021-05-14 06:29:09,578 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 114/125 -- train_loss: 1.7710 2021-05-14 06:29:10,931 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 115/125 -- train_loss: 1.3850 2021-05-14 06:29:11,667 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 116/125 -- train_loss: 1.0000 2021-05-14 06:29:12,406 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 117/125 -- train_loss: 1.0841 2021-05-14 06:29:13,143 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 118/125 -- train_loss: 1.3670 2021-05-14 06:29:13,884 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 119/125 -- train_loss: 0.9852 2021-05-14 06:29:14,621 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 120/125 -- train_loss: 1.4840 2021-05-14 06:29:15,361 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 121/125 -- train_loss: 1.8541 2021-05-14 06:29:17,521 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 122/125 -- train_loss: 1.4559 2021-05-14 06:29:19,382 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 123/125 -- train_loss: 0.9349 2021-05-14 06:29:20,119 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 124/125 -- train_loss: 1.5214 2021-05-14 06:29:20,858 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 8/9, Iter: 125/125 -- train_loss: 1.0277 2021-05-14 06:29:20,858 - ignite.engine.engine.SupervisedEvaluator - INFO - Engine run resuming from iteration 0, epoch 7 until 8 epochs 2021-05-14 06:30:02,558 - ignite.engine.engine.SupervisedEvaluator - INFO - Epoch[8] Metrics -- validation loss: -0.0566 validation mean dice: 0.0138 2021-05-14 06:30:02,558 - ignite.engine.engine.SupervisedEvaluator - INFO - Key metric: validation mean dice best value: 0.49623361229896545 at epoch: 4 2021-05-14 06:30:02,559 - ignite.engine.engine.SupervisedEvaluator - INFO - Epoch[8] Complete. Time taken: 00:00:41 2021-05-14 06:30:02,559 - ignite.engine.engine.SupervisedEvaluator - INFO - Engine run complete. Time taken: 00:00:42 2021-05-14 06:30:02,626 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch[8] Complete. Time taken: 00:03:05 2021-05-14 06:30:13,297 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 1/125 -- train_loss: 1.2171 2021-05-14 06:30:14,117 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 2/125 -- train_loss: 0.9837 2021-05-14 06:30:14,856 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 3/125 -- train_loss: 1.4879 2021-05-14 06:30:15,595 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 4/125 -- train_loss: 1.2887 2021-05-14 06:30:16,333 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 5/125 -- train_loss: 1.6145 2021-05-14 06:30:17,072 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 6/125 -- train_loss: 1.3529 2021-05-14 06:30:17,812 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 7/125 -- train_loss: 1.2951 2021-05-14 06:30:18,554 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 8/125 -- train_loss: 1.2672 2021-05-14 06:30:21,813 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 9/125 -- train_loss: 1.0690 2021-05-14 06:30:22,555 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 10/125 -- train_loss: 1.1722 2021-05-14 06:30:23,291 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 11/125 -- train_loss: 1.0431 2021-05-14 06:30:24,029 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 12/125 -- train_loss: 1.2063 2021-05-14 06:30:24,771 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 13/125 -- train_loss: 1.1035 2021-05-14 06:30:25,512 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 14/125 -- train_loss: 1.2140 2021-05-14 06:30:26,252 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 15/125 -- train_loss: 1.5410 2021-05-14 06:30:26,994 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 16/125 -- train_loss: 1.9825 2021-05-14 06:30:30,671 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 17/125 -- train_loss: 1.0902 2021-05-14 06:30:31,409 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 18/125 -- train_loss: 1.0301 2021-05-14 06:30:32,146 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 19/125 -- train_loss: 1.5109 2021-05-14 06:30:32,886 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 20/125 -- train_loss: 1.2931 2021-05-14 06:30:33,626 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 21/125 -- train_loss: 1.5987 2021-05-14 06:30:34,364 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 22/125 -- train_loss: 1.0313 2021-05-14 06:30:35,102 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 23/125 -- train_loss: 1.7994 2021-05-14 06:30:35,843 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 24/125 -- train_loss: 1.5396 2021-05-14 06:30:39,267 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 25/125 -- train_loss: 1.2199 2021-05-14 06:30:40,006 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 26/125 -- train_loss: 1.3842 2021-05-14 06:30:40,745 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 27/125 -- train_loss: 1.0013 2021-05-14 06:30:41,484 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 28/125 -- train_loss: 1.1166 2021-05-14 06:30:42,223 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 29/125 -- train_loss: 2.0570 2021-05-14 06:30:42,963 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 30/125 -- train_loss: 1.2074 2021-05-14 06:30:43,700 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 31/125 -- train_loss: 1.2521 2021-05-14 06:30:44,438 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 32/125 -- train_loss: 1.4280 2021-05-14 06:30:47,456 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 33/125 -- train_loss: 1.1032 2021-05-14 06:30:48,270 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 34/125 -- train_loss: 2.3083 2021-05-14 06:30:49,103 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 35/125 -- train_loss: 1.4693 2021-05-14 06:30:49,840 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 36/125 -- train_loss: 1.6851 2021-05-14 06:30:50,580 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 37/125 -- train_loss: 0.9894 2021-05-14 06:30:51,320 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 38/125 -- train_loss: 1.3533 2021-05-14 06:30:52,058 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 39/125 -- train_loss: 1.1738 2021-05-14 06:30:52,797 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 40/125 -- train_loss: 1.2504 2021-05-14 06:30:56,127 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 41/125 -- train_loss: 1.4583 2021-05-14 06:30:56,868 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 42/125 -- train_loss: 1.1914 2021-05-14 06:30:57,838 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 43/125 -- train_loss: 1.5164 2021-05-14 06:30:58,576 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 44/125 -- train_loss: 1.5705 2021-05-14 06:30:59,316 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 45/125 -- train_loss: 1.0634 2021-05-14 06:31:00,056 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 46/125 -- train_loss: 1.4270 2021-05-14 06:31:00,795 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 47/125 -- train_loss: 1.0856 2021-05-14 06:31:01,533 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 48/125 -- train_loss: 1.4148 2021-05-14 06:31:04,461 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 49/125 -- train_loss: 2.5814 2021-05-14 06:31:05,220 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 50/125 -- train_loss: 1.1573 2021-05-14 06:31:06,925 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 51/125 -- train_loss: 1.8870 2021-05-14 06:31:07,664 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 52/125 -- train_loss: 1.3012 2021-05-14 06:31:08,400 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 53/125 -- train_loss: 1.3884 2021-05-14 06:31:09,141 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 54/125 -- train_loss: 1.0842 2021-05-14 06:31:09,879 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 55/125 -- train_loss: 1.4669 2021-05-14 06:31:10,617 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 56/125 -- train_loss: 1.3273 2021-05-14 06:31:12,989 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 57/125 -- train_loss: 1.2260 2021-05-14 06:31:14,034 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 58/125 -- train_loss: 1.1968 2021-05-14 06:31:15,362 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 59/125 -- train_loss: 1.1903 2021-05-14 06:31:16,101 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 60/125 -- train_loss: 1.5843 2021-05-14 06:31:16,839 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 61/125 -- train_loss: 1.8729 2021-05-14 06:31:17,577 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 62/125 -- train_loss: 1.5574 2021-05-14 06:31:18,317 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 63/125 -- train_loss: 1.6810 2021-05-14 06:31:19,055 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 64/125 -- train_loss: 1.5449 2021-05-14 06:31:20,806 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 65/125 -- train_loss: 1.4644 2021-05-14 06:31:22,876 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 66/125 -- train_loss: 1.6281 2021-05-14 06:31:24,052 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 67/125 -- train_loss: 1.3101 2021-05-14 06:31:24,789 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 68/125 -- train_loss: 1.4561 2021-05-14 06:31:25,527 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 69/125 -- train_loss: 1.8659 2021-05-14 06:31:26,267 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 70/125 -- train_loss: 1.3251 2021-05-14 06:31:27,002 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 71/125 -- train_loss: 1.6050 2021-05-14 06:31:27,739 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 72/125 -- train_loss: 1.4454 2021-05-14 06:31:29,189 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 73/125 -- train_loss: 1.4899 2021-05-14 06:31:31,088 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 74/125 -- train_loss: 1.1069 2021-05-14 06:31:32,846 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 75/125 -- train_loss: 1.3910 2021-05-14 06:31:33,585 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 76/125 -- train_loss: 1.4885 2021-05-14 06:31:34,323 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 77/125 -- train_loss: 1.9895 2021-05-14 06:31:35,060 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 78/125 -- train_loss: 1.3882 2021-05-14 06:31:35,799 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 79/125 -- train_loss: 1.8285 2021-05-14 06:31:36,540 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 80/125 -- train_loss: 1.5596 2021-05-14 06:31:37,356 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 81/125 -- train_loss: 1.6181 2021-05-14 06:31:39,762 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 82/125 -- train_loss: 2.2077 2021-05-14 06:31:41,428 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 83/125 -- train_loss: 1.1216 2021-05-14 06:31:42,167 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 84/125 -- train_loss: 1.0420 2021-05-14 06:31:42,905 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 85/125 -- train_loss: 1.5507 2021-05-14 06:31:43,646 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 86/125 -- train_loss: 1.4512 2021-05-14 06:31:44,383 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 87/125 -- train_loss: 1.7658 2021-05-14 06:31:45,122 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 88/125 -- train_loss: 1.5281 2021-05-14 06:31:45,884 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 89/125 -- train_loss: 1.3838 2021-05-14 06:31:48,570 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 90/125 -- train_loss: 1.5431 2021-05-14 06:31:50,089 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 91/125 -- train_loss: 1.3588 2021-05-14 06:31:50,826 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 92/125 -- train_loss: 1.1888 2021-05-14 06:31:51,564 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 93/125 -- train_loss: 1.2282 2021-05-14 06:31:52,303 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 94/125 -- train_loss: 1.5109 2021-05-14 06:31:53,043 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 95/125 -- train_loss: 1.3985 2021-05-14 06:31:53,782 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 96/125 -- train_loss: 1.3404 2021-05-14 06:31:54,546 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 97/125 -- train_loss: 1.6286 2021-05-14 06:31:57,652 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 98/125 -- train_loss: 0.9630 2021-05-14 06:31:59,016 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 99/125 -- train_loss: 1.0543 2021-05-14 06:31:59,755 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 100/125 -- train_loss: 1.8785 2021-05-14 06:32:00,493 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 101/125 -- train_loss: 1.6527 2021-05-14 06:32:01,234 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 102/125 -- train_loss: 1.6148 2021-05-14 06:32:01,972 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 103/125 -- train_loss: 1.3313 2021-05-14 06:32:02,709 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 104/125 -- train_loss: 1.2648 2021-05-14 06:32:03,447 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 105/125 -- train_loss: 1.3951 2021-05-14 06:32:06,411 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 106/125 -- train_loss: 1.3853 2021-05-14 06:32:07,169 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 107/125 -- train_loss: 1.4012 2021-05-14 06:32:07,910 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 108/125 -- train_loss: 1.0338 2021-05-14 06:32:08,648 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 109/125 -- train_loss: 1.4429 2021-05-14 06:32:09,386 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 110/125 -- train_loss: 1.1451 2021-05-14 06:32:10,128 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 111/125 -- train_loss: 1.4720 2021-05-14 06:32:10,867 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 112/125 -- train_loss: 1.5901 2021-05-14 06:32:11,606 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 113/125 -- train_loss: 1.2736 2021-05-14 06:32:14,204 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 114/125 -- train_loss: 1.6406 2021-05-14 06:32:16,044 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 115/125 -- train_loss: 1.3596 2021-05-14 06:32:16,783 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 116/125 -- train_loss: 1.0067 2021-05-14 06:32:17,519 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 117/125 -- train_loss: 1.0461 2021-05-14 06:32:18,255 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 118/125 -- train_loss: 1.4211 2021-05-14 06:32:18,995 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 119/125 -- train_loss: 1.3075 2021-05-14 06:32:19,735 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 120/125 -- train_loss: 1.2413 2021-05-14 06:32:20,473 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 121/125 -- train_loss: 1.3905 2021-05-14 06:32:21,865 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 122/125 -- train_loss: 1.4368 2021-05-14 06:32:24,286 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 123/125 -- train_loss: 1.9720 2021-05-14 06:32:25,024 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 124/125 -- train_loss: 1.0217 2021-05-14 06:32:25,761 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch: 9/9, Iter: 125/125 -- train_loss: 1.3025 2021-05-14 06:32:25,762 - ignite.engine.engine.SupervisedEvaluator - INFO - Engine run resuming from iteration 0, epoch 8 until 9 epochs 2021-05-14 06:33:07,660 - ignite.engine.engine.SupervisedEvaluator - INFO - Epoch[9] Metrics -- validation loss: -0.0548 validation mean dice: 0.0396 2021-05-14 06:33:07,661 - ignite.engine.engine.SupervisedEvaluator - INFO - Key metric: validation mean dice best value: 0.49623361229896545 at epoch: 4 2021-05-14 06:33:07,662 - ignite.engine.engine.SupervisedEvaluator - INFO - Epoch[9] Complete. Time taken: 00:00:41 2021-05-14 06:33:07,662 - ignite.engine.engine.SupervisedEvaluator - INFO - Engine run complete. Time taken: 00:00:42 2021-05-14 06:33:07,736 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch[9] Complete. Time taken: 00:03:05 2021-05-14 06:33:07,742 - ignite.engine.engine.SupervisedTrainer - INFO - Deleted previous saved final checkpoint: checkpoint_final_iteration=750.pt 2021-05-14 06:33:07,804 - ignite.engine.engine.SupervisedTrainer - INFO - Train completed, saved final checkpoint: checkpoint_final_iteration=1125.pt 2021-05-14 06:33:07,804 - ignite.engine.engine.SupervisedTrainer - INFO - Engine run complete. Time taken: 00:09:16 Send model to server. 2021-05-14 06:33:08,036 - FederatedClient - INFO - Starting to push model. setup local_model_dict local_model_dict 48 2021-05-14 06:33:08,108 - Communicator - INFO - Send example_project at round 2 2021-05-14 06:33:10,612 - Communicator - INFO - Received comments: example_project Received round 2 from org1-a (16334854 Bytes, 607 seconds). SubmitUpdate time: 2.5033652782440186 seconds 2021-05-14 06:33:10,646 - ignite.engine.engine.SupervisedTrainer - INFO - Terminate signaled. Engine will stop after current iteration is finished. cross_validation child process ID: 7472 no max_epochs specified for SupervisedTrainer args, use global var 'epochs: 15'. 2021-05-14 06:33:13,176 - ValidationScheduler - INFO - Starting validation scheduler. 2021-05-14 06:33:13,177 - FederatedClient - INFO - Submitting best model to server. 2021-05-14 06:33:15,340 - Communicator - INFO - Server reply to SubmitBestLocalModel: Received best model from org1-a.. SubmitBestLocalModel time: 2.1356656551361084 seconds 2021-05-14 06:33:15,397 - FederatedClient - INFO - Getting other best models from server for cross validation. 2021-05-14 06:33:15,422 - Communicator - INFO - Received 0 models for validation. GetValidationModels time: 0.02389216423034668 seconds 2021-05-14 06:33:15,498 - FederatedClient - INFO - Submitting cross validation results to server. 2021-05-14 06:33:15,523 - Communicator - INFO - Received comments: Received Cross Validation results from org1-a.. SubmitCrossSiteValidationResults time: 0.024733304977416992 seconds 2021-05-14 06:33:15,598 - FederatedClient - INFO - Server asked to wait becausing models are not available yet. Waiting 60 s. 2021-05-14 06:34:15,657 - FederatedClient - INFO - Getting other best models from server for cross validation. 2021-05-14 06:34:15,680 - Communicator - INFO - Received 0 models for validation. GetValidationModels time: 0.022314786911010742 seconds 2021-05-14 06:34:15,758 - FederatedClient - INFO - Submitting cross validation results to server. 2021-05-14 06:34:15,781 - Communicator - INFO - Received comments: Received Cross Validation results from org1-a.. SubmitCrossSiteValidationResults time: 0.022068262100219727 seconds 2021-05-14 06:34:15,858 - FederatedClient - INFO - Server asked to wait becausing models are not available yet. Waiting 60 s. 2021-05-14 06:35:15,916 - FederatedClient - INFO - Getting other best models from server for cross validation. 2021-05-14 06:35:15,940 - Communicator - INFO - Received 0 models for validation. GetValidationModels time: 0.022692441940307617 seconds 2021-05-14 06:35:16,017 - FederatedClient - INFO - Submitting cross validation results to server. 2021-05-14 06:35:16,039 - Communicator - INFO - Received comments: Received Cross Validation results from org1-a.. SubmitCrossSiteValidationResults time: 0.02118706703186035 seconds 2021-05-14 06:35:16,117 - FederatedClient - INFO - Server asked to wait becausing models are not available yet. Waiting 60 s. 2021-05-14 06:36:16,178 - FederatedClient - INFO - Getting other best models from server for cross validation. 2021-05-14 06:36:16,201 - Communicator - INFO - Received 0 models for validation. GetValidationModels time: 0.02220320701599121 seconds 2021-05-14 06:36:16,279 - FederatedClient - INFO - Submitting cross validation results to server. 2021-05-14 06:36:16,302 - Communicator - INFO - Received comments: Received Cross Validation results from org1-a.. SubmitCrossSiteValidationResults time: 0.021889686584472656 seconds 2021-05-14 06:36:16,379 - FederatedClient - INFO - Server asked to wait becausing models are not available yet. Waiting 60 s. 2021-05-14 06:37:16,438 - FederatedClient - INFO - Getting other best models from server for cross validation. 2021-05-14 06:37:16,463 - Communicator - INFO - Received 0 models for validation. GetValidationModels time: 0.02384161949157715 seconds 2021-05-14 06:37:16,539 - FederatedClient - INFO - Submitting cross validation results to server. 2021-05-14 06:37:16,567 - Communicator - INFO - Received comments: Received Cross Validation results from org1-a.. SubmitCrossSiteValidationResults time: 0.026304006576538086 seconds 2021-05-14 06:37:16,639 - FederatedClient - INFO - Server asked to wait becausing models are not available yet. Waiting 60 s. 2021-05-14 06:38:16,699 - FederatedClient - INFO - Getting other best models from server for cross validation. 2021-05-14 06:38:19,102 - Communicator - INFO - Received 1 models for validation. GetValidationModels time: 2.40198016166687 seconds 2021-05-14 06:38:19,102 - FederatedClient - INFO - Validating org1-a's model on our data. 2021-05-14 06:38:19,115 - ValidationScheduler - INFO - Added Validation task Trying to execute task. 2021-05-14 06:38:21,597 - torch.distributed.nn.jit.instantiator - INFO - Created a temporary directory at /tmp/tmpwvf3nvkx 2021-05-14 06:38:21,597 - torch.distributed.nn.jit.instantiator - INFO - Writing /tmp/tmpwvf3nvkx/_remote_module_non_sriptable.py 2021-05-14 06:38:23,210 - ignite.engine.engine.SupervisedEvaluator - INFO - Engine run resuming from iteration 0, epoch 0 until 1 epochs 2021-05-14 06:38:23,223 - ignite.engine.engine.SupervisedEvaluator - INFO - Restored all variables from /tmp/tmpb9h0_kfb/mmar/models/model.pt 2021-05-14 06:38:30,767 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files. 2021-05-14 06:38:36,253 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files. 2021-05-14 06:38:41,693 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files. 2021-05-14 06:38:47,148 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files. 2021-05-14 06:38:52,603 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files. 2021-05-14 06:38:58,068 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files. 2021-05-14 06:39:03,550 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files. 2021-05-14 06:39:09,043 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files. 2021-05-14 06:39:09,045 - ignite.engine.engine.SupervisedEvaluator - INFO - Got new best metric of validation loss: -0.0811932235956192 2021-05-14 06:39:09,045 - ignite.engine.engine.SupervisedEvaluator - INFO - Epoch[1] Metrics -- validation loss: -0.0812 validation mean dice: 0.4962 2021-05-14 06:39:09,045 - ignite.engine.engine.SupervisedEvaluator - INFO - Key metric: validation loss best value: -0.0811932235956192 at epoch: 1 2021-05-14 06:39:09,046 - ignite.engine.engine.SupervisedEvaluator - INFO - Epoch[1] Complete. Time taken: 00:00:46 2021-05-14 06:39:09,046 - ignite.engine.engine.SupervisedEvaluator - INFO - Engine run complete. Time taken: 00:00:46 2021-05-14 06:39:09,091 - medl.apps.mmar_conf - INFO - Total Evaluation Time 47.68468427658081 Job execution finished. Accepting finished job. Validation job finished. Received metrics: {'validation': {'validation loss': -0.0811932235956192, 'validation mean dice': 0.49623361229896545}} Trying to encode metrics. 2021-05-14 06:39:11,168 - FederatedClient - INFO - Submitting cross validation results to server. 2021-05-14 06:39:11,192 - Communicator - INFO - Received comments: Received Cross Validation results from org1-a.. SubmitCrossSiteValidationResults time: 0.02309250831604004 seconds 2021-05-14 06:39:11,267 - FederatedClient - INFO - More models available with server. 2021-05-14 06:39:11,268 - FederatedClient - INFO - Getting other best models from server for cross validation. 2021-05-14 06:39:13,668 - Communicator - INFO - Received 1 models for validation. GetValidationModels time: 2.3985698223114014 seconds 2021-05-14 06:39:13,672 - FederatedClient - INFO - Validating org1-b's model on our data. 2021-05-14 06:39:13,678 - ValidationScheduler - INFO - Added Validation task Trying to execute task. 2021-05-14 06:39:16,104 - torch.distributed.nn.jit.instantiator - INFO - Created a temporary directory at /tmp/tmp85s0nys_ 2021-05-14 06:39:16,104 - torch.distributed.nn.jit.instantiator - INFO - Writing /tmp/tmp85s0nys_/_remote_module_non_sriptable.py 2021-05-14 06:39:17,596 - ignite.engine.engine.SupervisedEvaluator - INFO - Engine run resuming from iteration 0, epoch 0 until 1 epochs 2021-05-14 06:39:17,609 - ignite.engine.engine.SupervisedEvaluator - INFO - Restored all variables from /tmp/tmpf3i4v97y/mmar/models/model.pt 2021-05-14 06:39:25,194 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files. 2021-05-14 06:39:30,661 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files. 2021-05-14 06:39:36,189 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files. 2021-05-14 06:39:41,670 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files. 2021-05-14 06:39:47,182 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files. 2021-05-14 06:39:52,701 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files. 2021-05-14 06:39:58,240 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files. 2021-05-14 06:40:03,791 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files. 2021-05-14 06:40:03,793 - ignite.engine.engine.SupervisedEvaluator - INFO - Got new best metric of validation loss: -0.07968351244926453 2021-05-14 06:40:03,793 - ignite.engine.engine.SupervisedEvaluator - INFO - Epoch[1] Metrics -- validation loss: -0.0797 validation mean dice: 0.4723 2021-05-14 06:40:03,793 - ignite.engine.engine.SupervisedEvaluator - INFO - Key metric: validation loss best value: -0.07968351244926453 at epoch: 1 2021-05-14 06:40:03,793 - ignite.engine.engine.SupervisedEvaluator - INFO - Epoch[1] Complete. Time taken: 00:00:46 2021-05-14 06:40:03,793 - ignite.engine.engine.SupervisedEvaluator - INFO - Engine run complete. Time taken: 00:00:46 2021-05-14 06:40:03,838 - medl.apps.mmar_conf - INFO - Total Evaluation Time 47.92181062698364 Job execution finished. Accepting finished job. Validation job finished. Received metrics: {'validation': {'validation loss': -0.07968351244926453, 'validation mean dice': 0.4722886383533478}} Trying to encode metrics. 2021-05-14 06:40:05,732 - FederatedClient - INFO - Submitting cross validation results to server. 2021-05-14 06:40:05,760 - Communicator - INFO - Received comments: Received Cross Validation results from org1-a.. SubmitCrossSiteValidationResults time: 0.027294397354125977 seconds 2021-05-14 06:40:05,832 - FederatedClient - INFO - More models available with server. 2021-05-14 06:40:05,833 - FederatedClient - INFO - Getting other best models from server for cross validation. 2021-05-14 06:40:08,229 - Communicator - INFO - Received 1 models for validation. GetValidationModels time: 2.395420551300049 seconds 2021-05-14 06:40:08,236 - FederatedClient - INFO - Validating server's model on our data. 2021-05-14 06:40:08,245 - ValidationScheduler - INFO - Added Validation task Trying to execute task. 2021-05-14 06:40:10,757 - torch.distributed.nn.jit.instantiator - INFO - Created a temporary directory at /tmp/tmp3820icnv 2021-05-14 06:40:10,757 - torch.distributed.nn.jit.instantiator - INFO - Writing /tmp/tmp3820icnv/_remote_module_non_sriptable.py 2021-05-14 06:40:12,375 - ignite.engine.engine.SupervisedEvaluator - INFO - Engine run resuming from iteration 0, epoch 0 until 1 epochs 2021-05-14 06:40:12,387 - ignite.engine.engine.SupervisedEvaluator - INFO - Restored all variables from /tmp/tmpvw9gjge4/mmar/models/model.pt 2021-05-14 06:40:19,961 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files. 2021-05-14 06:40:25,544 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files. 2021-05-14 06:40:31,090 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files. 2021-05-14 06:40:36,655 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files. 2021-05-14 06:40:42,200 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files. 2021-05-14 06:40:47,730 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files. 2021-05-14 06:40:53,274 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files. 2021-05-14 06:40:58,819 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files. 2021-05-14 06:40:58,821 - ignite.engine.engine.SupervisedEvaluator - INFO - Got new best metric of validation loss: -0.07388120889663696 2021-05-14 06:40:58,821 - ignite.engine.engine.SupervisedEvaluator - INFO - Epoch[1] Metrics -- validation loss: -0.0739 validation mean dice: 0.0325 2021-05-14 06:40:58,821 - ignite.engine.engine.SupervisedEvaluator - INFO - Key metric: validation loss best value: -0.07388120889663696 at epoch: 1 2021-05-14 06:40:58,821 - ignite.engine.engine.SupervisedEvaluator - INFO - Epoch[1] Complete. Time taken: 00:00:46 2021-05-14 06:40:58,821 - ignite.engine.engine.SupervisedEvaluator - INFO - Engine run complete. Time taken: 00:00:46 2021-05-14 06:40:58,868 - medl.apps.mmar_conf - INFO - Total Evaluation Time 48.30046725273132 Job execution finished. Accepting finished job. Validation job finished. Received metrics: {'validation': {'validation loss': -0.07388120889663696, 'validation mean dice': 0.032477498054504395}} Trying to encode metrics. 2021-05-14 06:41:00,298 - FederatedClient - INFO - Submitting cross validation results to server. 2021-05-14 06:41:00,326 - Communicator - INFO - Received comments: Received Cross Validation results from org1-a.. SubmitCrossSiteValidationResults time: 0.02622389793395996 seconds 2021-05-14 06:41:00,398 - FederatedClient - INFO - Finished cross site validation. 2021-05-14 06:41:00,398 - ValidationScheduler - INFO - Shutting down validation scheduler.