How to enable AMP

Physics Simulation PhysicsNeMo-ML Model Framework Technical Support (PhysicsNeMo Only)
1

Hello,

I would like to enable AMP to reduce the used memory and time in training simnet model.
However it looks like not to work.
Could you show me how to enable AMP?

Our Environment
HW ; AWS EC2 g4dn.xlarge
Amazon EC2 G4 Instances — Amazon Web Services (AWS)
SimNet Version 21.06(docker container)
Example : helmholtz

Result(2 cases)

Case 1. without modifying code
%python ./helmholtz.py --amp AMP
…
[0m: AdamOptimizer, /usr/local/lib/python3.8/dist-packages/simnet-21.6-py3.8.egg/simnet/optimizer.py
beta1: 0.9
beta2: 0.999
epsilon: 1e-08
amp: False
…

  • AMP is inactivate.

Case 2. with modify /simnet/build/lib/simnet/optimizer.py
fix the default value of AMP True
%vi /simnet/build/lib/simnet/optimizer.py
…
group.add_argument(‘–amp’,
help=‘use Automatic Mixed Precision (AMP)? (0/1)’,
type=str2bool,
default=True)
#default=False)
…
%python ./helmholtz.py --amp AMP
result was same as Case.1

Also in the both of case1 and 2 the training time was almost same as the case without --amp AMP option.

Thanks,
Toshiaki Yokoi

3

Thanks for reporting this. This is a bug and will be fixed in the next release. For now you can activate AMP by changing the environment variable:

export TF_ENABLE_AUTO_MIXED_PRECISION=1

Please note that there are known issues with AMP when used for second and higher-order derivatives, and with AMP your training loss may go to NaN/Inf. We are working on fixing this issue.

4

Hi,

Thank you for your reply.
I’m trying that.

Thanks a lot,
Toshiaki Yokoi