In the Adadelta optimization algorithm, the decay parameter $$ho$$ controls the leaky average of the state variables. Choosing a specific value for $$ho$$ relates to the effective half-life of the historical parameter statistics. For instance, setting $$ho = 0.9$$ effectively yields a half-life of 10 for each parameter update, meaning that the influence of past squared gradients and parameter changes diminishes by half over approximately 10 iterations. This default value is found to work quite well in practice for stabilizing the adaptive learning rate.

Claude

The Adadelta algorithm updates parameters using a sequence of operations based on leaky averages. Given a decay parameter $$\rho$$, the state variable for the gradient's second moment is updated as $$\mathbf{s}_t = \rho \mathbf{s}_{t-1} + (1 - \rho) \mathbf{g}_t^2$$. A rescaled gradient $$\mathbf{g}_t'$$ is then computed using the ratio of the root mean square of previous parameter changes to the root mean square of the gradients: $$\mathbf{g}_t' = \frac{\sqrt{\Delta\mathbf{x}_{t-1} + \epsilon}}{\sqrt{{\mathbf{s}_t + \epsilon}}} \odot \mathbf{g}_t$$. The model parameters are updated by subtracting this rescaled gradient: $$\mathbf{x}_t = \mathbf{x}_{t-1} - \mathbf{g}_t'$$. Finally, the state variable tracking the parameter changes, initialized at $$\Delta \mathbf{x}_0 = 0$$, is updated as $$\Delta \mathbf{x}_t = \rho \Delta\mathbf{x}_{t-1} + (1 - \rho) {\mathbf{g}_t'}^2$$, where $$\epsilon$$ is a small constant (e.g., $$10^{-5}$$) added to maintain numerical stability.

Learn Before

Related