Issue with ComputeAverageDice Metric in SDK 3.1

config_train.json (4.0 KB)
segResNetCPU.py (6.2 KB)

Hi,

Upon trying the training model with channel last, getting following error in metric computation

Traceback (most recent call last):
File “workflows/fitters/supervised_fitter.py”, line 245, in fit
File “workflows/fitters/supervised_fitter.py”, line 672, in _do_fit
File “workflows/fitters/supervised_fitter.py”, line 962, in do_validation
File “components/metrics/metric.py”, line 71, in update
File “libs/metrics/avg_from_array.py”, line 27, in update
File “libs/metrics/avg_from_array_dice.py”, line 38, in reduction_fn
File “libs/metrics/np_ops.py”, line 105, in acc_dice_np_batch
File “libs/metrics/np_ops.py”, line 41, in acc_dice_np
File “libs/metrics/np_ops.py”, line 7, in dice_np
AssertionError: shape of labels (128, 128, 128) and prediction (128, 128, 10) are inconsistent

The issue seems to be due to that fact that ComputeAverageDice is defined for channel_first only.
class ComputeAverageDice ( name , field , label_field=‘label’ , is_2d=False , remove_bg=True , logit_thresh=0.5 , is_one_hot_targets=False , report_path=None , do_summary=True , do_print=True , is_key_metric=False )

Please let me know how to resolve the upper issue.
I have uploaded my config_train.json and custom segResNet network file here.

Thanks,
Archit

Hi
Thanks for your interest in Clara Train SDK.
There are multiple issues here.
1 we don’t support running on a CPU
2 we always recommend using channel first
3- If this is a 3d problem then your sizes should be 128,128,128,numOfClasses (10)

Hope this helps

Please note we have recently release clara train V4.0 based on MONAI which uses PyTorch. Please check out the notebooks to get you started clara-train-examples/PyTorch/NoteBooks at master · NVIDIA/clara-train-examples · GitHub

  1. Is channel first recommended or is it mandatory.
    The above worked fine for channel first, but in case of channel last it showed the above error.
    The above worked as 3D in case of channel first.

We are working on Clara 3.1, and its difficult to move to a new version.
Is clara 3.1 still supported by nvidia?

Hi,

I have a further query

  1. How to write custom dice metric computation?
    The documentation shows usage of Metric class and MetricList but
  2. Where to add custom code for computation.
  3. How to access the prediction and label array.

Please let me know, as it would help in overcoming the above issue of ComputeAverageDice not support Channel Last

Hello Nvidia Team/@aharouni ,
Unfortunately, the documentation is not helpful for us to understand and implement the custom metric. This is a pretty fundamental requirement i think.
Is it possible to help us out? In this case, because of “channel first” versus “channel last”, the metric computation breaks down.
Regards,
Krishnan

HI
Having channel first is just a recommendation that our research is following, Therefore it is more tested than channel last usecase. That could explain why you are facing that error

To get to the root cause of the error could you clarify are you doing a 2d or 3d segmentation task ? how many labels are there ?

I am trying to understand at what stage is that mistake happening 128x128x128 seems like the volume shape but it seems to be missing the channels altogether,

some other things to try in your config:

  • change SegmentationKerasImagePipelineWithCache to other segmentation pipelines
  • remove RandomRotate3D if there is a bug there it might cause the result volume to have different shape than expected
  • Please clarify why are you using SymmetricPadderDiv

Hi

Sorry about your troubles
Just to understand what you are trying to do. Do you want to rewrite the compute average dice metric as you are facing an error with channel last network ? Or Are you trying to write a completely different metric ?
Just trying to see what code example we can share to unblock you

Thanks

Hi,
Thanks for the reply. We find a lot of features in Clara to be useful but just aren’t sure about the documentation. Specifically, we need our models to run on both GPU & CPU. As i understand, TF supports “channel_last” on CPU for some operations. So, we try to implement custom segmentation network using channel_last option.

  1. BYOC metric example works well for classification example irrespective of channel first or last. This is therefore not a problem.
  2. Our problem is for segmentation example. The In built AverageDice metric fails for channel_last but works for channel_first. Can you please confirm that it a bug?
  3. Currently, we are using the “negative loss as metric” as a proxy. Looking at that code, gives us some idea on how to use the build_ctx with aux operations for generating a metric. But we arent sure it is the right way to go.
  4. Keeping the above point in mind, can you please help provide an example for writing a custom metric using the model prediction and GT?
  5. There is a problem while running the non cached pipeline. It crashes after sometime. We have raised a query for that as well but awaiting response. I do not want to cross post but it is raised by @araj
    thanks so much!
    regards