Feature request: easy visualization of FL performance

Dear NVIDIA,

First of all - thank you for all the great work that has been put into CLARA.
We’ve been able to do many interesting AI experiments with it, particularly with CLARA FL.

We are wondering if there is any existing feature in CLARA that would help us easily record & visualize FL performance with respect to round number?
In other words, we would like to have a way to easily save the performance of the global model (as validated on all the client data) for each round & have something like the Tensorboard to conveniently visualize it.
If something like this exists already, could you kindly let us know what feature this is & which part of the documentation should we dig into?

(our understanding is that Tensorboard is only for “CLARA TRAIN” and can only be used to visualize the performance at the “client level” but not really the global server model level performance with respect to FL rounds)

If not, we would like to propose this as a potential feature in the next release as we can imagine many FL researchers wanting to inspect the convergence of their model with respect to rounds.
To be more specific, it would be useful to be able to

  1. Do cross site validation after each round, and record the aggregated model performance as validated on each individual client dataset + weighted averaged performance for each round

  2. Have something like Tensorboard to easily visualize the result as recorded in 1

Thank you!
Andy

Hi. Thanks for the feedback. If you are just looking for the global model performance on each client’s validation set for each round, you can use the IntimeModelSelectionHandler.

It will log the performance in the server log as
"validation metric {validation_metric} from client {client_name}"

The admin can grep this information using

grep server "validation metric" log.txt

Thank you hroth3hm8y. That’s very useful to know. We will give it a go!

In that case a “Tensorboard” like feature to investigate FL performance wouldn’t be as critical, but remains a very “nice to have” feature for future releases.

Thanks!