To address the learning rate dilemma for sparse features, one approach is to adjust the learning rate based on feature occurrence. Instead of a global time-based decay $\eta = \frac{\eta_0}{\sqrt{t + c}}$, a feature-specific rate $\eta_i = \frac{\eta_0}{\sqrt{s(i, t) + c}}$ can be used, where $s(i, t)$ counts the number of nonzeros for feature $i$ observed up to time $t$. However, this method fails for data that is not strictly sparse but instead has gradients that are mostly very small and only rarely large, as it is difficult to define a clear threshold for counting a feature as observed.