A standard linear regression model can serve as a simple baseline for sequence prediction by treating the data as if it satisfies a $$\tau^{\textrm{th}}$$-order Markov condition. The model accepts the fixed-length window of the past $$\tau$$ observations as its input feature vector and predicts the next value in the sequence. In practice, this amounts to training a linear regression with a standard optimization loop—for example, using a learning rate of $$0.01$$ and training for $$5$$ epochs. While basic, this approach establishes a baseline performance level using a simple linear combination of past data before moving on to more complex sequence architectures.

Linear Regression for Sequence Prediction

To construct training data from a sequence for autoregressive models operating under a $$\tau^{\textrm{th}}$$-order Markov assumption, one extracts input–output pairs where each label is $$y = x_t$$ and the corresponding feature vector is $$\mathbf{x}_t = [x_{t-\tau}, \ldots, x_{t-1}]$$. Because sufficient history is unavailable for the first $$\tau$$ time steps, those examples are dropped, yielding $$T - \tau$$ total examples from a sequence of length $$T$$, each with a fixed input dimensionality of $$\tau$$. Rather than padding the missing early observations with zeros, this simple truncation approach is commonly used. This construction relies on the stationarity assumption—the belief that the dynamics generating the sequence do not change over time—which ensures that patterns extracted from any historical segment remain relevant for predicting future values.

Training Data Construction for Sequence Models

To overcome the challenge of variable-length inputs in autoregressive modeling, a common strategy is to condition predictions only on a fixed-length window of recent history rather than the entire past. By using a window of length $$	au$$, the model only considers the observations $$x_{t-1}, \ldots, x_{t-	au}$$ to predict $$x_t$$. This ensures that for all time steps $$t > 	au$$, the number of input arguments remains constant, allowing the use of models that require fixed-length feature vectors.

Claude

An autoregressive model predicts the next value in a sequence by regressing a signal on its own previous values. Specifically, given a time series of observations, the model estimates the conditional probability distribution $$P(x_t \mid x_{t-1}, \ldots, x_1)$$ or a key statistic thereof, such as the conditional expectation $$\mathbb{E}[(x_t \mid x_{t-1}, \ldots, x_1)]$$. A foundational challenge for autoregressive models is that the number of historical inputs $$x_{t-1}, \ldots, x_1$$ grows with $$t$$, causing each training example to have a different number of features. Two principal strategies address this: (1) conditioning on a fixed-length window of $$	au$$ recent observations, and (2) maintaining a latent summary state $$h_t$$ that compresses the entire history.

Autoregressive Model

Dive into Deep Learning

Autoregressive Windowing Strategy

A latent autoregressive model addresses the challenge of variable-length histories by maintaining a hidden summary state, $$h_t$$, of past observations. Instead of conditioning directly on a growing sequence of past inputs, the model predicts the current observation $$\hat{x}_t$$ based on the current summary state ($$\hat{x}_t = P(x_t \mid h_t)$$). Simultaneously, it updates the summary state using the previous state and the previous observation ($$h_t = g(h_{t-1}, x_{t-1})$$). Because the state $$h_t$$ is never directly observed in the data, it is considered latent.

Latent Autoregressive Model

In autoregressive modeling, rather than estimating the entire probability distribution over continuously valued sequence data, it is often more practical to focus on key statistics such as the expected value. The autoregressive conditional expectation estimates the expected value of the next sequence element, $$x_t$$, given its entire history of past observations. This is formally denoted as $$\mathbb{E}[(x_t \mid x_{t-1}, \ldots, x_1)]$$, and can be estimated using strategies like linear regression.

Autoregressive Conditional Expectation

A major challenge in standard autoregressive modeling is that the number of historical inputs, $$x_{t-1}, \ldots, x_1$$, increases as the time step $$t$$ increases. This means that when treating historical sequence data as a training set, each example has a different number of features. This variable-length input problem necessitates the use of specialized techniques, such as fixed-length windowing or latent state models, to process the data effectively.

Variable-Length Input Problem in Autoregressive Models

For natural language data composed of discrete tokens like words, sequence models are specifically called language models. They provide the capacity to evaluate the likelihood of sentences, sample new text sequences, and optimize for the most likely outputs. By applying the chain rule of probability, language modeling can be reduced to an autoregressive prediction problem, decomposing the joint density of a sequence $$P(x_1, \ldots, x_T)$$ into the product of conditional densities in a left-to-right fashion: $$P(x_1, \ldots, x_T) = P(x_1) \prod_{t=2}^T P(x_t \mid x_{t-1}, \ldots, x_1)$$. For discrete signals, the autoregressive model acts as a probabilistic classifier, outputting a full probability distribution over the vocabulary for the next word given the leftwards context.

Learn Before

Related

Learn After