NVIDIA Apex: Tools for Easy Mixed-Precision Training in PyTorch

Originally published at: https://developer.nvidia.com/blog/apex-pytorch-easy-mixed-precision-training/

Most deep learning frameworks, including PyTorch, train using 32-bit floating point (FP32) arithmetic by default. However, using FP32 for all operations is not essential to achieve full accuracy for many state-of-the-art deep neural networks (DNNs). In 2017, NVIDIA researchers developed a methodology for mixed-precision training in which a few operations are executed in FP32 while…