Extremely low server validation mean dice value

Hello,

We’ve been successfully training our Monai based models on Clara train v4.0-EA2.

Below are the logs for one of our models for server & 2 clients.

Server-lesion-seg-MeanDice.txt (17.3 KB)
ClientA-lesion-seg-MeanDice.txt (174.3 KB)
ClientB-lesion-seg-MeanDice.txt (167.8 KB)

When running the client model on client data we receive a reasonable validation mean dice of around 0.4-0.5 (as seen in the server logs). However, on running client data on server model (as seen by the final lines in the server log attached above), the validation mean dice is extremely low i.e., 0.032477498054504395 & 0.022482797503471375.

Attaching MMAR configs for your reference.
config 2.zip (15.4 KB)
Please suggest.

Thanks,
Siddharth

This seems to be an intermittent error. Resolved by rerunning training again.

Thanks for the update.
Please note that things has changed a bit between GA and the EA2 you have been using specially in FL.