step1: training set: large corpus of text

step2: tokenize text--> one-hot encoding --> add <EOS> token

step3: detecting unknown words

step4: start with the first word of the sentence and deed it as input to the RNN.

University of Michigan - Ann Arbor

Google

For natural language data composed of discrete tokens like words, sequence models are specifically called language models. They provide the capacity to evaluate the likelihood of sentences, sample new text sequences, and optimize for the most likely outputs. By applying the chain rule of probability, language modeling can be reduced to an autoregressive prediction problem, decomposing the joint density of a sequence $$P(x_1, \ldots, x_T)$$ into the product of conditional densities in a left-to-right fashion: $$P(x_1, \ldots, x_T) = P(x_1) \prod_{t=2}^T P(x_t \mid x_{t-1}, \ldots, x_1)$$. For discrete signals, the autoregressive model acts as a probabilistic classifier, outputting a full probability distribution over the vocabulary for the next word given the leftwards context.

Language Model

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

Language model with RNN

Language model alignment involves fine-tuning a pretrained language model to follow instructions and align with human intent. Systems like ChatGPT leverage this alignment to allow users to interact conversationally, enabling the model to serve as an assistant for solving complex problems, debugging code, and performing creative writing tasks.

Language Model Alignment

A sequence satisfies a Markov condition if the future is conditionally independent of the past, given the recent history. This means that when predicting the next element in a sequence, we can discard the history beyond the previous $$	au$$ time steps without any loss in predictive power, conditioning only on $$x_{t-1}, \ldots, x_{t-	au}$$ rather than the entire sequence history $$x_{t-1}, \ldots, x_1$$. While this is only approximately true for real text—where leftwards context continues to provide diminishing information gains—working with models that satisfy this condition obviates computational and statistical difficulties associated with processing arbitrarily long sequences.

Markov Condition

While the joint probability of a sequence $$P(x_1, \ldots, x_T)$$ can be mathematically factorized in reverse (right-to-left) or any random order, left-to-right (in-order) factorization is generally preferred for language modeling. First, it aligns with the natural human intuition of anticipating upcoming words while reading. Second, factorizing in order allows the same language model to easily assign probabilities to arbitrarily long sequences by continually multiplying the current probability by the conditional probability of the next token: $$P(x_{t+1}, \ldots, x_1) = P(x_{t}, \ldots, x_1) \cdot P(x_{t+1} \mid x_{t}, \ldots, x_1)$$. Third, for causally structured data where future events cannot influence the past, predicting forward ($$P(x_{t+1} \mid x_t)$$) is usually an easier predictive modeling problem than predicting backward ($$P(x_t \mid x_{t+1})$$).

Learn Before

Related