Keskar, N. S., & Socher, R. (2017). Improving generalization performance by switching from adam to sgd. arXiv preprint arXiv:1712.07628.

University of California, Berkeley

Adam stands for adaptive moment estimation.
It combines gradient descent with momentum, and RMSProp. It brings the benefits from both sides - adaptive learning rate and faster convergence with momentum.

Adam (Deep Learning Optimization Algorithm)

Improving Generalization Performance by Switching from Adam to SGD

 $ M^{t} = \frac { \beta_{1} M^{t-1} + (1 - \beta_{1} )  \nabla J(W^{t})} { 1 - (\beta_{1})^{t}} $

$ V^{t} = \frac { \beta_{2} V^{t-1} + (1 - \beta_{2} )  \nabla J^2(W^{t})} { 1 - (\beta_{2})^{t}} $

$ W^{t} = W^{t-1} - \frac{\alpha}{\sqrt{V^{t} + \epsilon}} M^{t} $

------------------------------------------------------------
$1 - (\beta_{2})^{t}$  and $1 - (\beta_{1})^{t}$ are used in order to normalize both matrices, as the authors of the algorithm noticed that M and V go to zero very fast.

$M^{t}$ - helper matrix that is similar to what we used for the momentum but normalized.

$V^{t}$ - helper matrix that is similar to what we used for the RMSprop but normalized.

$\beta_{1}, \beta_{2}$ - the terms identical to the ones in momentum and RMSprop (usually $\beta_{1}=0.9, \beta_{2}=0.999$).

$W^{t}$ - the parameters

$\alpha$ - starting learning rate (usually something around 0.001).

$\epsilon$ - it is just to avoid division by zero (usually around 1e-8).
The same principle applies to the bias parameters

Adam (Deep Learning Optimization Algorithm) Mathematical Implementation

```python
import numpy as np

class Adam():
  
    self.beta1 = 0.9
    self.beta2 = 0.99

    # Here is the M amd V matrices for the parameters and biases
    self.m_weights = np.zeros((input_dims, nodes))
    self.v_weights = np.zeros((input_dims, nodes))
    self.m_biases = np.zeros(nodes)
    self.v_biases = np.zeros(nodes)

    # need to track thecurrent time stamp
    self.curr_iter = 0

    def get_steps(self, grad_weights, grad_biases, learning_rate):

        eps = 1e-8

        # Just follows the formula and returns the update that have to be 
        # subtracted from the parametes and biases
      
        self.m_weights = self.beta1 * self.m_weights + (1 - self.beta1) * grad_weights
        self.v_weights = self.beta2 * self.v_weights + (1 - self.beta2) * np.power(grad_weights, 2)
        self.m_biases = self.beta1 * self.m_biases + (1 - self.beta1) * grad_biases
        self.v_biases = self.beta2 * self.v_biases + (1 - self.beta2) * np.power(grad_biases, 2)

        self.curr_iter += 1
        self.m_weights = self.m_weights / (1 - np.power(self.beta1, self.curr_iter))
        self.v_weights = self.v_weights / (1 - np.power(self.beta2, self.curr_iter))
        self.m_biases = self.m_biases / (1 - np.power(self.beta1, self.curr_iter))
        self.v_biases = self.v_biases / (1 - np.power(self.beta2, self.curr_iter))

        weights_step = np.multiply((learning_rate / (np.sqrt(self.v_weights) + eps)), self.m_weights)
        biases_step = np.multiply((learning_rate / (np.sqrt(self.v_biases) + eps)), self.m_biases)

        return weights_step, biases_step
```

Adam (Deep Learning Optimization Algorithm) Python Implementation

- Adam is fast, but tends to overfit
- SGD is slow but gives great results
- RMSProp sometimes works best
- SWA can easily improve quality
- AdaTune magically improves the learning rate

Learn Before

Related