Adam (adaptive moment estimation) Optimization is an algorithm specifically designed to train neural networks. It uses the square of gradients to scale the learning rate (similar to RMSprop), and it uses momentum by using averages of the gradient (instead of the gradient itself like SGD). It is useful for small gradients as it uses both the first and second moments for the average instead of solely the first.