University of Michigan - Ann Arbor

Adadelta is an optimization algorithm that has no explicit learning rate parameter. Instead, it uses the rate of change in the parameters themselves to dynamically adapt the learning rate. To accomplish this, the algorithm utilizes two specific state variables: $$\mathbf{s}_t$$ to track a leaky average of the second moment of the gradient, and $$\Delta\mathbf{x}_t$$ to track a leaky average of the second moment of the model's parameter changes. The algorithm retains standard naming conventions for these variables to maintain consistency with similar optimization methods like momentum, AdaGrad, and RMSProp.

Adadelta

Here is a very helpful article on different types of optimizer algorithms
https://ruder.io/optimizing-gradient-descent/index.html

An overview of gradient descent optimization algorithms

The Adadelta algorithm updates parameters using a sequence of operations based on leaky averages. Given a decay parameter $$\rho$$, the state variable for the gradient's second moment is updated as $$\mathbf{s}_t = \rho \mathbf{s}_{t-1} + (1 - \rho) \mathbf{g}_t^2$$. A rescaled gradient $$\mathbf{g}_t'$$ is then computed using the ratio of the root mean square of previous parameter changes to the root mean square of the gradients: $$\mathbf{g}_t' = \frac{\sqrt{\Delta\mathbf{x}_{t-1} + \epsilon}}{\sqrt{{\mathbf{s}_t + \epsilon}}} \odot \mathbf{g}_t$$. The model parameters are updated by subtracting this rescaled gradient: $$\mathbf{x}_t = \mathbf{x}_{t-1} - \mathbf{g}_t'$$. Finally, the state variable tracking the parameter changes, initialized at $$\Delta \mathbf{x}_0 = 0$$, is updated as $$\Delta \mathbf{x}_t = \rho \Delta\mathbf{x}_{t-1} + (1 - \rho) {\mathbf{g}_t'}^2$$, where $$\epsilon$$ is a small constant (e.g., $$10^{-5}$$) added to maintain numerical stability.

Learn Before

Related