When a sequence satisfies a Markov condition with $$	au = 1$$, the data is characterized by a first-order Markov model. In this case, the factorization of the joint probability simplifies to a product of probabilities for each element given only the immediately preceding element: $$P(x_1, \ldots, x_T) = P(x_1) \prod_{t=2}^T P(x_t \mid x_{t-1})$$. When $$	au = k$$, the data is characterized by a $$k^{	extrm{th}}$$-order Markov model, which conditions on the $$k$$ previous time steps. For discrete data like language, a true Markov model estimates $$P(x_t \mid x_{t-1})$$ by counting the relative frequency of each word occurring in each context, allowing the most likely sequence to be computed efficiently using dynamic programming.

First-Order and k-th Order Markov Models

A sequence satisfies a Markov condition if the future is conditionally independent of the past, given the recent history. This means that when predicting the next element in a sequence, we can discard the history beyond the previous $$	au$$ time steps without any loss in predictive power, conditioning only on $$x_{t-1}, \ldots, x_{t-	au}$$ rather than the entire sequence history $$x_{t-1}, \ldots, x_1$$. While this is only approximately true for real text—where leftwards context continues to provide diminishing information gains—working with models that satisfy this condition obviates computational and statistical difficulties associated with processing arbitrarily long sequences.

Claude

For natural language data composed of discrete tokens like words, sequence models are specifically called language models. They provide the capacity to evaluate the likelihood of sentences, sample new text sequences, and optimize for the most likely outputs. By applying the chain rule of probability, language modeling can be reduced to an autoregressive prediction problem, decomposing the joint density of a sequence $$P(x_1, \ldots, x_T)$$ into the product of conditional densities in a left-to-right fashion: $$P(x_1, \ldots, x_T) = P(x_1) \prod_{t=2}^T P(x_t \mid x_{t-1}, \ldots, x_1)$$. For discrete signals, the autoregressive model acts as a probabilistic classifier, outputting a full probability distribution over the vocabulary for the next word given the leftwards context.

Language Model

Dive into Deep Learning

step1: training set: large corpus of text

step2: tokenize text--> one-hot encoding --> add <EOS> token

step3: detecting unknown words

step4: start with the first word of the sentence and deed it as input to the RNN.

Language model with RNN

Language model alignment involves fine-tuning a pretrained language model to follow instructions and align with human intent. Systems like ChatGPT leverage this alignment to allow users to interact conversationally, enabling the model to serve as an assistant for solving complex problems, debugging code, and performing creative writing tasks.

Language Model Alignment

Markov Condition

While the joint probability of a sequence $$P(x_1, \ldots, x_T)$$ can be mathematically factorized in reverse (right-to-left) or any random order, left-to-right (in-order) factorization is generally preferred for language modeling. First, it aligns with the natural human intuition of anticipating upcoming words while reading. Second, factorizing in order allows the same language model to easily assign probabilities to arbitrarily long sequences by continually multiplying the current probability by the conditional probability of the next token: $$P(x_{t+1}, \ldots, x_1) = P(x_{t}, \ldots, x_1) \cdot P(x_{t+1} \mid x_{t}, \ldots, x_1)$$. Third, for causally structured data where future events cannot influence the past, predicting forward ($$P(x_{t+1} \mid x_t)$$) is usually an easier predictive modeling problem than predicting backward ($$P(x_t \mid x_{t+1})$$).

Learn Before

Related

Learn After