A major challenge in standard autoregressive modeling is that the number of historical inputs, $$x_{t-1}, \ldots, x_1$$, increases as the time step $$t$$ increases. This means that when treating historical sequence data as a training set, each example has a different number of features. This variable-length input problem necessitates the use of specialized techniques, such as fixed-length windowing or latent state models, to process the data effectively.

Claude

An autoregressive model predicts the next value in a sequence by regressing a signal on its own previous values. Specifically, given a time series of observations, the model estimates the conditional probability distribution $$P(x_t \mid x_{t-1}, \ldots, x_1)$$ or a key statistic thereof, such as the conditional expectation $$\mathbb{E}[(x_t \mid x_{t-1}, \ldots, x_1)]$$. A foundational challenge for autoregressive models is that the number of historical inputs $$x_{t-1}, \ldots, x_1$$ grows with $$t$$, causing each training example to have a different number of features. Two principal strategies address this: (1) conditioning on a fixed-length window of $$	au$$ recent observations, and (2) maintaining a latent summary state $$h_t$$ that compresses the entire history.

Autoregressive Model

Dive into Deep Learning

To overcome the challenge of variable-length inputs in autoregressive modeling, a common strategy is to condition predictions only on a fixed-length window of recent history rather than the entire past. By using a window of length $$	au$$, the model only considers the observations $$x_{t-1}, \ldots, x_{t-	au}$$ to predict $$x_t$$. This ensures that for all time steps $$t > 	au$$, the number of input arguments remains constant, allowing the use of models that require fixed-length feature vectors.

Autoregressive Windowing Strategy

A latent autoregressive model addresses the challenge of variable-length histories by maintaining a hidden summary state, $$h_t$$, of past observations. Instead of conditioning directly on a growing sequence of past inputs, the model predicts the current observation $$\hat{x}_t$$ based on the current summary state ($$\hat{x}_t = P(x_t \mid h_t)$$). Simultaneously, it updates the summary state using the previous state and the previous observation ($$h_t = g(h_{t-1}, x_{t-1})$$). Because the state $$h_t$$ is never directly observed in the data, it is considered latent.

Latent Autoregressive Model

In autoregressive modeling, rather than estimating the entire probability distribution over continuously valued sequence data, it is often more practical to focus on key statistics such as the expected value. The autoregressive conditional expectation estimates the expected value of the next sequence element, $$x_t$$, given its entire history of past observations. This is formally denoted as $$\mathbb{E}[(x_t \mid x_{t-1}, \ldots, x_1)]$$, and can be estimated using strategies like linear regression.

Autoregressive Conditional Expectation

Variable-Length Input Problem in Autoregressive Models

For natural language data composed of discrete tokens like words, sequence models are specifically called language models. They provide the capacity to evaluate the likelihood of sentences, sample new text sequences, and optimize for the most likely outputs. By applying the chain rule of probability, language modeling can be reduced to an autoregressive prediction problem, decomposing the joint density of a sequence $$P(x_1, \ldots, x_T)$$ into the product of conditional densities in a left-to-right fashion: $$P(x_1, \ldots, x_T) = P(x_1) \prod_{t=2}^T P(x_t \mid x_{t-1}, \ldots, x_1)$$. For discrete signals, the autoregressive model acts as a probabilistic classifier, outputting a full probability distribution over the vocabulary for the next word given the leftwards context.

Learn Before

Related