Optimizer for Image Classification

[marmikshah]

Hello
I am trying to train a model using TAO. In the documentation, I see that there are 3 optimizers that we can configure, but I do not see any info on which parameters to configure for the different optimizers.
The example document only has this.

# Only ['sgd', 'adam'] are supported for optimizer
  optimizer {
      sgd {
      lr: 0.01
      decay: 0.0
      momentum: 0.9
      nesterov: False
      }
  }

Can someone please tell which parameters need to be set to use ADAM?

NOTE: I am referring to the document here Image Classification — TAO Toolkit 3.22.05 documentation

Thanks.

[Morganh]

You can refer to https://docs.nvidia.com/tao/tao-toolkit/text/multitask_image_classification.html?highlight=adam#creating-an-experiment-spec-file-specification-file-for-multitask-classification

optimizer {
adam {
epsilon: 1e-7
beta1: 0.9
beta2: 0.999
amsgrad: false
}
}

[marmikshah]

Ah! Thank you very much.

[marmikshah]

Sorry, when I try to run the config mentioned above, I am getting an error like:
google.protobuf.text_format.ParseError: 33:7 : Message type "AdamOptimizerConfig" has no field named "beta1"

[Morganh]

After checking, the parameters for classification are:

message AdamOptimizerConfig {
float lr = 1;
float beta_1 = 2;
float beta_2 = 3;
float epsilon = 4;
float decay = 5;
}

So,

optimizer {
adam {
epsilon: 1e-7
beta_1: 0.9
beta_2: 0.999
lr: 0.01 (just an example)
decay: 0.0
}
}

[marmikshah]

Oh I see. Thanks.
One last question. Can you tell me where to check these specifications? (Like the protobuf message structure you mentioned above)

[Morganh]

We will update the document. Current user guide is missing description of adam.

[marmikshah]

That would be great! Thanks alot.