A straightforward method for creating summary vectors for the memory component ($$\mathrm{Mem}$$) is to compute the moving average of past keys and values. For instance, $$\mathrm{Mem}$$ can be constructed from the unweighted moving average of the last $$n_c$$ key and value vectors, providing a simple yet effective way to summarize history.

Moving Average of Keys and Values for Memory Component

A moving average is a method used to find an average of a numerical metric over a specified block of time, n. For example, if you set n = 2 then you wouldn’t have any measurements for the first time period, you would then take the average of the first and second time periods to be moving average for the second time period, then you would take the average of the second and third time period to be the moving average of the third time period, next you would take the average of the third and fourth time period to me the moving average for the fourth time period, so on and so forth for all of the time periods in your data. The moving average can be thought of as a sort of sliding window. Also, note that your first moving average measurement will be at the nth datapoint because that is the first time you will be able to average n points.

University of Michigan - Ann Arbor

Google

Smoothing is a technique applied to time series to remove the fine-grained variation between time steps.

The hope of smoothing is to remove noise and better expose the signal of the underlying causal processes. Moving averages are a simple and common type of smoothing used in time series analysis and time series forecasting.
It is a trend-following indicator because it is based on past prices. Below is the list of different moving average and smoothing methods.

- Autoregression (AR)
- Moving Average (MR)
- Autoregressive Moving Average (ARMA)
- Autoregressive Integrated Moving Average (ARIMA)
 - Seasonal Autoregressive Integrated Moving Average (SARIMA)
 - Seasonal Autoregressive integrated Moving Average with Exogenous Regressors (SARIMAX)
 - Simple Exponential Smoothing (SES)
 - Holt Winter's Exponential Smoothing (HWES)


Time series learning methods: Smoothing

Reference of Foundations of Large Language Models Course

The autoregression (AR) method models the next step in the sequence as a linear function of the observations at prior time steps.

The notation for the model involves specifying the order of the model p as a parameter to the AR function, e.g. AR(p). For example, AR(1) is a first-order autoregression model.

The method is suitable for univariate time series without trend and seasonal components.

Below is an example of AR code implementation


Autoregression (AR)

The Autoregressive Integrated Moving Average (ARIMA) method models the next step in the sequence as a linear function of the differenced observations and residual errors at prior time steps.

It combines both Autoregression (AR) and Moving Average (MA) models as well as a differencing pre-processing step of the sequence to make the sequence stationary, called integration (I).

The notation for the model involves specifying the order for the AR(p), I(d), and MA(q) models as parameters to an ARIMA function, e.g. ARIMA(p, d, q). An ARIMA model can also be used to develop AR, MA, and ARMA models.

The method is suitable for univariate time series with trend and without seasonal components.

Below is an example of ARIMA code implementation


Autoregressive Integrated Moving Average (ARIMA)

Moving Average (MR)

The Autoregressive Moving Average (ARMA) method models the next step in the sequence as a linear function of the observations and resiudal errors at prior time steps.

It combines both Autoregression (AR) and Moving Average (MA) models.

The notation for the model involves specifying the order for the AR(p) and MA(q) models as parameters to an ARMA function, e.g. ARMA(p, q). An ARIMA model can be used to develop AR or MA models.

The method is suitable for univariate time series without trend and seasonal components.

Below is an example of ARMA code implementation


Autoregressive Moving Average (ARMA)

The Seasonal Autoregressive Integrated Moving Average (SARIMA) method models the next step in the sequence as a linear function of the differenced observations, errors, differenced seasonal observations, and seasonal errors at prior time steps.

It combines the ARIMA model with the ability to perform the same autoregression, differencing, and moving average modeling at the seasonal level.

The notation for the model involves specifying the order for the AR(p), I(d), and MA(q) models as parameters to an ARIMA function and AR(P), I(D), MA(Q) and m parameters at the seasonal level, e.g. SARIMA(p, d, q)(P, D, Q)m where “m” is the number of time steps in each season (the seasonal period). A SARIMA model can be used to develop AR, MA, ARMA and ARIMA models.

The method is suitable for univariate time series with trend and/or seasonal components.

Below is an example of SARIMA code implementation

Seasonal Autoregressive Integrated Moving Average (SARIMA)

The Seasonal Autoregressive Integrated Moving-Average with Exogenous Regressors (SARIMAX) is an extension of the SARIMA model that also includes the modeling of exogenous variables.

Exogenous variables are also called covariates and can be thought of as parallel input sequences that have observations at the same time steps as the original series. The primary series may be referred to as endogenous data to contrast it from the exogenous sequence(s). The observations for exogenous variables are included in the model directly at each time step and are not modeled in the same way as the primary endogenous sequence (e.g. as an AR, MA, etc. process).

The SARIMAX method can also be used to model the subsumed models with exogenous variables, such as ARX, MAX, ARMAX, and ARIMAX.

The method is suitable for univariate time series with trend and/or seasonal components and exogenous variables.

 Below is an example of SARIMAX code implementation

 Seasonal Autoregressive integrated Moving Average with Exogenous Regressors (SARIMAX)

The Simple Exponential Smoothing (SES) method models the next time step as an exponentially weighted linear function of observations at prior time steps.

SES requires a single parameter, called alpha (a), also called the smoothing factor or smoothing coefficient.

This parameter controls the rate at which the influence of the observations at prior time steps decay exponentially. Alpha is often set to a value between 0 and 1. Large values mean that the model pays attention mainly to the most recent past observations, whereas smaller values mean more of the history is taken into account when making a prediction.

The method is suitable for univariate time series without trend and seasonal components.

Below is an example of code implementation

Simple Exponential Smoothing (SES)

The Holt Winter’s Exponential Smoothing (HWES) also called the Triple Exponential Smoothing method models the next time step as an exponentially weighted linear function of observations at prior time steps, taking trends and seasonality into account.

In addition to the alpha and beta smoothing factors, a new parameter is added called gamma (g) that controls the influence on the seasonal component.

As with the trend, the seasonality may be modeled as either an additive or multiplicative process for a linear or exponential change in the seasonality.

Additive Seasonality: a linear seasonality.
Multiplicative Seasonality: an exponential seasonality.

The method is suitable for univariate time series with trend and/or seasonal components.

Below is an example of code implementation

Learn Before

Related

Learn After