SegmentationSaver not working with FL

Hi there,

I am training a segmentation model in a federated approach on a custom dataset.
I would like to save segmented images of the validation files by using the SegmentationSaver handler. This is the definition in my config_validation.json:

    {
      "name": "SegmentationSaver",
      "args": {
        "output_dir": "{MMAR_EVAL_OUTPUT_PATH}",
        "batch_transform": "lambda x: x['image_meta_dict']",
        "output_transform": "lambda x: x['pred']"
      }
    },

The images seem to be saved by the output in the terminal of the clients:


2021-07-05 12:15:05,538 - ignite.engine.engine.SupervisedEvaluator - INFO - saved all the model outputs into files.
file written: /tmp/tmp_9v57fd2/mmar/eval/kERGMGRD4/kERGMGRD4_seg.nii.gz.

Though, I can’t find any images stored in the path under the dedicated client. Does anyone have an idea why this is the case?

1 Like

I have the same issue.

Hi @LSnyd @yuling.luo. In a federated approach, during cross site validation, each client may have to validate a large number of models (from other clients etc). We delete all artifacts after each validation to conserve memory.

In the future, we’ll add support to move these artifacts using a handler. However, currently the best solution is to run validation with the local models outside of FL using clara train sdk. We can help guide you on how to do that if you would like.

Thanks,
Adil