Recurrent Memory Models as a Basis for Self-Attention Alternatives

Recursive Formula for Memory as a Cumulative Average

A recurrent model with an internal state h is processing a sequence of inputs. The state is updated at each step according to the rule h_i = f(h_{i-1}, input_i), where h_{i-1} is the state from the previous step and input_i is the current input. When the model processes the third input in a sequence, what information does the term h_2 (the state after the second input) represent in the computation for the new state h_3?

Analysis of Sequential Information Processing

A neural network processes a sequence of inputs by updating a hidden state h at each step i using the formula: h_i = f(h_{i-1}, input_i). Which component in this formula is directly responsible for carrying forward a compressed summary of the entire sequence processed up to the previous step (i-1)?

Recurrent Computation of $\mu_i$ and $\nu_i$ in Linear Attention

Real-Time Applications of Recurrent Models

Resurgence of Recurrent Models in Large Language Models

Sequential Token Processing in Recurrent Models

Comparison of Efficient LLM Architectures