How do I configure cylcic learning rates?

Hello,

I’m currently running through the tutorials and I’m investigating ways to speed up convergence.

Page 115 in the user guides states:

The default is an exponential decay, but you can choose among exponential decay, polynomial decay, exponential decay with warm-up or cyclic schedules for the learning rates.

I’ve checked learning_rate.py but I don’t recognise any cyclic learning rate class.
My question is what is the recommended way to implement cyclic learning rate?

Hi, thank you for pointing this out. You are correct that we do not currently have the cyclical learning rate in SimNet. However, we have implemented a variant of cyclical learning rate for you and it will be included in the next SimNet release. In this variant, the maximum learning rate is reduced exponentially in each iteration. For now you can use the attached code. Simply replace the current learning_rate.py file with this one, and then run the installer:

cd /simnet
python setup.py install

image
learning_rate.py (7.6 KB)

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.