Learn Before
Autoregressive Model
An autoregressive model predicts the next value in a sequence by regressing a signal on its own previous values. Specifically, given a time series of observations, the model estimates the conditional probability distribution or a key statistic thereof, such as the conditional expectation . A foundational challenge for autoregressive models is that the number of historical inputs grows with , causing each training example to have a different number of features. Two principal strategies address this: (1) conditioning on a fixed-length window of recent observations, and (2) maintaining a latent summary state that compresses the entire history.
0
1
Tags
D2L
Dive into Deep Learning @ D2L
Related
Chain Rule for Sequence Probability
Conditional Probability of the Next Token
A model is generating a sequence of words. It has already produced the words 'The', 'quick', 'brown'. According to the principle of autoregressive conditional probability, which expression correctly represents the likelihood that the next word will be 'fox', given the preceding words?
Defining Probability for a Token in a Sequence
A model is generating a sequence of elements (x₀, x₁, x₂, x₃, ...). To calculate the probability of the fourth element (x₃), the model's calculation must be conditioned on the entire preceding subsequence (x₀, x₁, x₂). A simplified model that conditions the probability of x₃ only on the immediately preceding element (x₂) would still be correctly applying the principle of autoregressive conditional probability.
Autoregressive Model