 Pros:

 - No need to care about the learning rate because it changes by itself

Cons:

 - Sometimes the sum of squared gradients can get very big and therefore the learning rate can become equal to zero and model would stop learning

University of California, Berkeley

So one of the big disadvantages of momentum and nesterov momentum algorithms is that they heavily rely on the learning rate. So AdaGrad is one of the algorithms that modifies the learning rate as we go. The intuition behind the adaptive learning rate is that it goes slower with frequent features and goes faster with features that happen rarely. 

AdaGrad (Deep Learning Optimization Algorithm)

 $G^{t} = G^{t-1} +  \nabla J^2(W^{t-1}) $

$W^{t} = W^{t-1} - \frac{ \alpha} {\sqrt{G^{t} + \epsilon}}  \nabla J(W^{t-1})$
-----------------------------------------
$G^{t}$ - helper matrix for the algorithm

$W^{t}$ - the parameters

$\alpha$ - starting learning rate(usually someting around 0.1 or 0.01)

$\epsilon$ - it is just to avoid division by zero( usually around 1e-8 ) 

The same principle applies to the bias parameters

Mathematical Implementation

Pros and Cons

"""So below the AdaGrad class that shows how a way to implement it in python. The class get one layers node number and its input dimensions. A function get_steps takes in current gradients for weights and biases and returns the step that we need to update those weights and biases"""

```python

import numpy as np

class AdaGrad():

    def __init__(self, input_dims, nodes):
  
      self.learning_rate = 0.01

      # G matrix for parameters
      self.G_weights = np.zeros((input_dims, nodes))

      # G matrix for biases
      self.G_biases = np.zeros(nodes)

    def get_steps(self, grad_weights, grad_biases):

      eps = 1e-8
      
      # updating G matrixes
      self.G_weights += np.multiply(grad_weights, grad_weights)
      self.G_biases += np.multiply(grad_biases, grad_biases)

      weights_step = np.multiply((self.learning_rate / np.sqrt(self.G_weights + eps)), grad_weights)
      biases_step = np.multiply((self.learning_rate / np.sqrt(self.G_biases + eps)), grad_biases)

      return weights_step, biases_step
```

Learn Before

Related