Concept icon
Concept

Effective Observation Window of RMSProp

In the RMSProp optimization algorithm, the effective observation window for the exponentially weighted average of squared gradients is defined by the quantity 11γ\frac{1}{1 - \gamma}, where γ\gamma is the weighting term (or decay factor). This means the state variable aggregates information over approximately the past 11γ\frac{1}{1 - \gamma} observations. A larger γ\gamma produces a longer memory and a smoother average, while a smaller γ\gamma makes the algorithm more responsive to recent gradients. For example, setting γ=0.9\gamma = 0.9 yields an effective window of 110.9=10\frac{1}{1 - 0.9} = 10 observations.

0

1

Concept icon
Updated 2026-06-17

Tags

D2L

Dive into Deep Learning @ D2L