The fundamental goal of a language model is to predict the probability of a given sequence of tokens occurring. For a sequence mathematically represented as $$x_0, x_1, \dots, x_m$$, where $$x_0$$ serves as the start symbol, the overall likelihood is determined by applying the chain rule of probability. This calculation decomposes the joint probability of the entire sequence into a product of conditional probabilities for each subsequent token.

Language Models (LMs)

Decoding, in models such as HMM that contains hidden variables, is the task that takes as input an HMM $\lambda = (A, B)$ and a sequence of observations $O = o_1 , ..., o_T$, and finds the most probable sequence of states $Q = q_1...q_T$. 

For part-of-speech tagging, the goal of HMM decoding is to choose the tag sequence $t_1...t_n$ that is most probable given the observation sequence of n words $w_1...w_n$:
$$\hat t_{1:n} = \argmax_{t_1, ..., t_n} P(t_1...t_n|w_1...w_n)$$



HMM Tagging as Decoding

- Language Models (LMs)
- Hidden Markov Models (HMMs)

University of Michigan - Ann Arbor

Each of the other models (state machines, formal rule systems, and logic) can be augmented with probabilities. For example, the state machine can be augmented with probabilities to become the weighted automaton, or Markov model. The key advantage of probabilistic models is their ability to solve the many kinds of ambiguity problems that can be recast as “given N choices for some ambiguous input, choose the most probable one”.

Probabilistic models

An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition
https://www.cs.colorado.edu/~martin/slp2.html

Speech and Language Processing (2nd edition)

An on-going but a helpful book resource about NLP
https://web.stanford.edu/~jurafsky/slp3/

Speech and Language Processing (3rd ed. draft) 

- Spelling correction or grammatical error correction
- Speech recognition
- Machine translation

Applications of probabilistic models

Types of probabilistic models

The deep learning approach is to figure out the minimum amount of information that is absolutely needed and find out how to get a reasonable approximation of this information in the shortest amount of time. 

The latent variables are designed differently in which the training algorithm can invent the concepts it needs to model a particular dataset freely and deal with complex problems and are reusable in more different contexts.


The Deep Learning Approach to Structured Probabilistic Models

The partition function is an integral or sum over unnormalized probabilities. This function, often denoted by Z, is used to normalize an unnormalized distribution into a valid distribution.
$$ p(\textbf{x}) =  \frac{1}{Z}  \tilde{p}(\textbf{x}) .  $$

where Z is: 

$$ Z =  \int   \tilde{p}(\textbf{x}) d\textbf{x}.  $$ 
or
$$ Z =  \sum_{x}  \tilde{p}(\textbf{x}) d\textbf{x}.  $$

Z = partition function.  
$$ \tilde{p}(\textbf{x})$$ = Unnormalized distribution. 
$$p(\textbf{x}) $$= Normalized distribution. 
In the context of deep learning, Z is usually intractable, meaning that it often needs to be approximated. 


The Partition Function (introduction).

Probabilistic models use graphs to show interactions between random variables. A graph is a set of nodes connected by vertices. A probabilistic model represents random variables with nodes and their relationships with vertices between them.

Graph Model Structure

- Dramatically reduce the cost of representing probability distributions as well as learning and inference
- In the case of a directed model, sampling is accelerated
- Allow us to explicitly separate representation of knowledge from learning of knowledge or inference given existing knowledge

Advantages of Structured Modeling

There are several methods to train and evaluate undirected graphical models with intractable partition function.

One way is to avoid using partition function and this can be done by following the techniques mentioned below.

 - Log-Likelihood Gradient
 - Stochastic Maximum Likelihood and ContrastiveDivergence
 - Pseudolikelihood
 - Score Matching and Ratio Matching
 - Denoising Score Matching
 - Noise-Contrastive Estimation

Training and Evaluation of Models with Intractable Partition Functions

$$Yˆ = argmax_{Y∈y} P(Y|X)$$ However, the CRF does not compute a probability for each tag at each time step. Instead, at each time step the CRF computes log-linear functions over a set of relevant features, and these local features are aggregated and normalized to produce a global probability for the whole sequence. In a CRF, the function $$F$$ maps an entire input sequence $$X$$ and an entire output sequence $$Y$$ to a feature vector. Let’s assume we have $$K$$ features, with a weight $$w_{k}$$ for each feature $$F_{k}$$: $$p(Y|X) = \frac{exp(\sum_{k=1}^K w_{k}F_{k}(X,Y))}{\sum_{Y'∈y} exp(\sum_{k=1}^Kw_{k}F_{k}(X,Y'))}$$ We’ll call these $$K$$ functions $$F_{k}(X,Y)$$ global features, since each one is a property of the entire input sequence $$X$$ and output sequence $$Y$$. We compute them by decomposing into a sum of local features for each position $$i$$ in $$Y$$: $$F_{k}(X,Y) =\sum_{i=1}^n f_{k}(y_{i−1}, y_{i},X,i)$$ This constraint to only depend on the current and previous output tokens $$y_{i}$$ and $$y_{i−1}$$ are what characterizes a linear chain CRF. A general CRF allows a feature to make use of any output token, and are thus necessary for tasks in which the decision depend on distant output tokens. General CRFs require more complex inference, and are less commonly used for language processing.

Learn Before

Related

Learn After