One way to interpret maximum likelihood estimation is to view it as minimizing the dissimilarity between the empirical distribution and the model distribution. We can measure the degree of the dissimilarity using KL divergence.

University of Michigan - Ann Arbor

It's an important principal in selecting the model and describing the math formula of a model's loss function. Let $p_{model}(x,\theta)$ be a parametric family of possibility distribution over the same space indexed by $\theta$. The maximum likelihood estimator for $\theta$ is then defuned as 
$\theta_{ML} = argmax_{\theta}p_{model}(X;\theta) $

Maximum Likelihood Estimation

Kullback-Leibler (KL) divergence, also known as relative entropy, measures how one probability distribution diverges from a second, reference probability distribution. For discrete probability distributions $$P$$ and $$Q$$ defined on the same probability space, the KL divergence from $$Q$$ to $$P$$, denoted $$D_{\text{KL}}(P \|\| Q)$$, is the expectation of the logarithmic difference between the probabilities given by the two distributions, where the expectation is taken using the probabilities of $$P$$. The formula is: $$ D_{\text{KL}}(P \|\| Q) = \sum_{\mathbf{x}} P(\mathbf{x}) \log\left(\frac{P(\mathbf{x})}{Q(\mathbf{x})}\right) = \mathbb{E}_{\mathbf{x} \sim P} [\log P(\mathbf{x}) - \log Q(\mathbf{x})] $$ KL divergence is non-negative ($$D_{\text{KL}}(P \|\| Q) \ge 0$$) and is zero if and only if $$P$$ and $$Q$$ are identical. It is an asymmetric measure, meaning that $$D_{\text{KL}}(P \|\| Q)$$ is generally not equal to $$D_{\text{KL}}(Q \|\| P)$$.

Kullback-Leibler Divergence

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Deep Learning

Relationship between KL Divergence and MLE

The cross-entropy loss function works very well for models that predict binary classes (aka the output is between 0 and 1). It is defined as -[y*log(y-hat) +(1-y)*log(1-(y-hat))]. If y=0 the left side of the function is dropped and the right side, -log(1-(y-hat)), is used. Otherwise if y=1 the right side of the function is dropped and it uses -log(y-hat). In both instances this loss function encourages probabilities that are close to the true probability. 

Cross-entropy loss

Under appropriate conditions, as the number of training examples approaches inﬁnity, the maximum likelihood estimate of a parameter converges to the true value of the parameter.

The property of consistency of maximum likelihood

One consistent estimator may obtain lower generalization error for a ﬁxed number of samples $m$, or equivalently, may require fewer examples to obtain a ﬁxed level of generalization error.

Statistical Eﬃciency Principal of MLE

- The true distribution $p_{\text{data}}$ must lie within the model family $p_{\text{model}}(.;\theta)$. Otherwise, no esitmator can recover $p_{\text{data}}$.
- The true distribution $p_{\text{data}}$ must correspond exactly to one value of $\theta$. Otherwise, the maximum likelihood can recover the correct $p_{\text{data}}$ but will not be able to determine which value of $\theta$ was used by the data generating process.

Maximum Likelihood Estimator Properties

The gradient of the log-likelihood which is used in Maximum Likelihood Estimation can be decomposed into the following:
$$\nabla_{\boldsymbol \theta} \log p(\mathbf{x}; \boldsymbol \theta) = \nabla_{\boldsymbol \theta} \log \tilde{p}(\mathbf{x}; \boldsymbol \theta) - \nabla_{\boldsymbol \theta } \log Z(\boldsymbol \theta)$$
Where $\tilde{p}(\mathbf{x}; \boldsymbol \theta)$ is the unnormalized probabilioty density and $Z$ is the partition function. This is well-known as the decomposition into the positive phase and negative phase of learning. Due to the reliance of the partition function on the parameters, learning models by maximum likelihood is particularly difficult. 

Log-Likelihood Gradient

The training objective under the Maximum Likelihood Estimation (MLE) framework is to find the model parameters, $$\tilde{\theta}$$, that maximize the total log-probability of all sequences in a dataset $$\mathcal{D}$$. This is achieved by summing the log-probabilities of each individual sequence, `seq`, as calculated by the model parameterized by $$\theta$$. The general objective is formally expressed as: $$ \tilde{\theta} = \arg\max_{\theta} \sum_{\text{seq} \in \mathcal{D}} \log \text{Pr}_{\theta}(\text{seq}) $$ For datasets composed of input-output pairs $$(\mathbf{x}, \mathbf{y})$$, this objective can be specified as maximizing the joint log-probability of the concatenated sequences: $$ \tilde{\theta} = \underset{\theta}{\arg\max} \sum_{(\mathbf{x},\mathbf{y})\in\mathcal{D}} \log \text{Pr}_{\theta}(\text{seq}_{\mathbf{x},\mathbf{y}}) $$ This approach is equivalent to maximizing the sum of the log-likelihoods for all data points in the training set.

Maximum Likelihood Training Objective for a Dataset of Sequences

You are evaluating two simple probabilistic models, Model A and Model B, on their ability to represent a single observed data sequence from a series of coin flips: `[Heads, Tails, Heads]`. The models assume each flip is independent, so the probability of the sequence is the product of the probabilities of the individual outcomes. Given the probabilities assigned by each model below, which model provides a better fit for the observed data according to the principle of maximum likelihood estimation? Justify your answer with a calculation.

Model Selection via Likelihood

The objective of training a model is to find the set of parameters, $$\hat{\theta}$$, that minimizes the total loss across an entire dataset of sequences, $$\mathcal{D}$$. This optimization problem is formally expressed as:
$$ \hat{\theta} = \argmin_{\theta} \sum_{\mathbf{x} \in \mathcal{D}} \mathrm{Loss}_{\theta}(\mathbf{x}) $$
This loss minimization objective is mathematically equivalent to the principle of Maximum Likelihood Estimation (MLE). Specifically, when the loss function is defined as the negative log-likelihood of the data, minimizing the loss is the same as maximizing the likelihood of the data given the model parameters.

Training Objective as Loss Minimization over a Dataset

The general maximum likelihood estimation formulation for a dataset $$\mathcal{D}$$ can be re-expressed for sequential data by applying the chain rule of probability. This adaptation decomposes the log-probability of each full sequence $$\mathbf{x}$$ into a sum of conditional log-probabilities, thereby demonstrating mathematical equivalence between the standard objective and its autoregressive sequential form:
$$ \hat{\theta} = \argmax_{\theta} \sum_{\mathbf{x} \in \mathcal{D}} \log \mathrm{Pr}_{\theta}(\mathbf{x}) = \argmax_{\theta} \sum_{\mathbf{x} \in \mathcal{D}} \sum_{i=0}^{i-1} \log \mathrm{Pr}_{\theta}(x_{i+1}|x_0,...,x_{i}) $$

Mathematical Equivalence of General and Sequential MLE Objectives

A researcher is modeling a series of coin flips. They observe the following sequence of outcomes: Heads, Tails, Heads, Heads. The researcher wants to find the best parameter for their model, where the parameter represents the probability of the coin landing on Heads. According to the principle of maximum likelihood estimation, which of the following parameter values best explains the observed data?

In the context of training a Large Language Model (LLM), the optimal parameters, denoted as $$\hat{\theta}$$, are found by maximizing the conditional log-likelihood across a dataset $$D$$. This supervised learning objective involves finding the parameters $$\theta$$ that maximize the sum of the logarithmic probabilities of the true outputs $$y$$ given the inputs $$x$$, where the probability $$\text{Pr}_{\theta}(y|x)$$ is predicted by the LLM. The formula is expressed as: $$\hat{\theta} = \underset{\theta}{\arg\max} \sum_{(x,y) \in D} \log \text{Pr}_{\theta}(y|x)$$ In some contexts, the input $$x$$ can be represented by other variables, such as a context $$c$$ and a latent variable $$z$$, leading to an equivalent formulation: $$\hat{\theta} = \underset{\theta}{\arg\max} \sum_{(x,y) \in D} \log \text{Pr}_{\theta}(y|c, z)$$

Parameter Estimation via Conditional Log-Likelihood Maximization

In machine learning, training a model often involves minimizing a 'loss' or 'cost' function. However, the principle of Maximum Likelihood Estimation (MLE) is defined as *maximizing* the likelihood of the observed data. Explain how these two seemingly opposite objectives (minimizing a loss vs. maximizing a likelihood) are fundamentally equivalent.

Equivalence of Maximizing Likelihood and Minimizing Loss

Minimizing the mean squared error is mathematically equivalent to performing maximum likelihood estimation for a linear model under the assumption of additive Gaussian noise. In the negative log-likelihood objective for linear regression, if the standard deviation $$\sigma$$ is assumed to be fixed, the term $$\frac{1}{2} \log(2 \pi \sigma^2)$$ becomes a constant that can be ignored during optimization. The remaining term is identical to the squared error loss, except for the multiplicative constant $$\frac{1}{\sigma^2}$$, which does not alter the location of the minimum.

Equivalence of Squared Loss and Maximum Likelihood Estimation

To optimize a classification model using maximum likelihood estimation, we compare our predicted conditional probabilities with the actual labels. Assuming the dataset's labels $$\mathbf{Y}$$ are independent given the features $$\mathbf{X}$$, the probability of observing the correct labels is the product of individual probabilities:

$$P(\mathbf{Y} \mid \mathbf{X}) = \prod_{i=1}^n P(\mathbf{y}^{(i)} \mid \mathbf{x}^{(i)})$$

Because maximizing a product of many small probabilities is numerically unstable and computationally awkward, we take the negative logarithm. This transforms the problem into minimizing the negative log-likelihood, turning the product into a manageable sum of individual losses:

$$-\log P(\mathbf{Y} \mid \mathbf{X}) = \sum_{i=1}^n -\log P(\mathbf{y}^{(i)} \mid \mathbf{x}^{(i)}) = \sum_{i=1}^n l(\mathbf{y}^{(i)}, \hat{\mathbf{y}}^{(i)})$$

Negative Log-Likelihood Objective for Softmax Regression

The maximum likelihood cost function for learning conditional distributions is defined as the expected negative log-likelihood: $$J(\theta) = -\mathbb{E}_{x, y \sim \hat{p}_{data}} \log p_{model}(y \mid x)$$. The specific form of this cost function changes depending on the choice of model, specifically the form of $$\log p_{model}$$. An advantage of this approach is that it removes the burden of manually designing cost functions for each specific model, as the objective is automatically determined once the conditional probability distribution is specified.

Maximum Likelihood Cost Function for Conditional Distributions

An alternative approach to optimizing soft prompts involves minimizing the Kullback-Leibler (KL) divergence between the output probability distribution from the full context, $$\text{Pr}(\cdot|\mathbf{c}, \mathbf{z})$$, and the distribution from the soft prompt, $$\text{Pr}(\cdot|\sigma, \mathbf{z})$$. The goal is to find the soft prompt $$\hat{\sigma}$$ that makes these two distributions as similar as possible. The optimization is expressed by the formula: $$\hat{\sigma} = \underset{\sigma}{\arg\min}\, \text{KL}(\text{Pr}(\cdot|\mathbf{c}, \mathbf{z}) \|\| \text{Pr}(\cdot|\sigma, \mathbf{z}))$$

Formula for Soft Prompt Optimization by Minimizing KL Divergence

The RLHF policy optimization objective can be reformulated by substituting the definition of the optimal reward-weighted policy, $$\pi^{*}(\mathbf{y}|\mathbf{x})$$. This transforms the objective into one that minimizes the Kullback-Leibler (KL) divergence between the learned policy, $$\pi_{\theta}(\cdot|\mathbf{x})$$, and the optimal policy. The substitution yields: $$ \tilde{\theta} = \arg \min_{\theta} \mathbb{E}_{\mathbf{x} \sim \mathcal{D}} \mathbb{E}_{\mathbf{y} \sim \pi_{\theta}(\cdot|\mathbf{x})} \left[ \log \pi_{\theta}(\mathbf{y}|\mathbf{x}) - \log \pi^{*}(\mathbf{y}|\mathbf{x}) - \log Z(\mathbf{x}) \right] $$ Because the expectation of the difference in log-probabilities over $$\pi_{\theta}$$ corresponds to the definition of KL divergence, this simplifies to: $$ \tilde{\theta} = \arg \min_{\theta} \mathbb{E}_{\mathbf{x} \sim \mathcal{D}} \left[ \text{KL}(\pi_{\theta}(\cdot|\mathbf{x}) || \pi^{*}(\cdot|\mathbf{x})) - \log Z(\mathbf{x}) \right] $$ This derivation shows that the objective is equivalent to minimizing the KL divergence from the optimal policy, offset by the normalization term $$\log Z(\mathbf{x})$$.

Derivation of the KL Divergence Objective for Policy Optimization

A machine learning model produces a probability distribution Q over a set of outcomes, aiming to approximate a true data distribution P. During evaluation, you observe that the divergence measure $D(P \|\| Q) = \sum_{x} P(x) \log\left(\frac{P(x)}{Q(x)}\right)$ is low, while the reverse measure $D(Q \|\| P) = \sum_{x} Q(x) \log\left(\frac{Q(x)}{P(x)}\right)$ is high. Based on these results, what is the most likely characteristic of the model's distribution Q?

Consider a true probability distribution $P$ and a model's approximation $Q$ over three possible outcomes {A, B, C}. The distributions are defined as follows:

*   $P = \{A: 0.5, B: 0.3, C: 0.2\}$
*   $Q = \{A: 0.4, B: 0.4, C: 0.2\}$

Calculate the divergence from $Q$ to $P$, denoted $D_{\text{KL}}(P \|\| Q)$, using the natural logarithm (ln). Provide your answer rounded to three decimal places.

Calculating Divergence Between Distributions

You are tasked with training a small, efficient language model (the 'student' model) to mimic the behavior of a much larger, more powerful model (the 'teacher' model). The goal is to make the student's output probability distribution, $Q(y|x)$, as close as possible to the teacher's distribution, $P(y|x)$, for any given input $x$. You decide to use the measure of divergence between these two distributions as your loss function. Which of the two possible formulations of this measure should you choose to minimize, and why? Justify your choice by explaining the practical implications of your selection versus the alternative.

Choosing a Loss Function for Model Distillation

The Kullback-Leibler (KL) divergence acts as a regularization term in the loss function of a Variational Autoencoder (VAE). It improves the model in two main ways: first, it ensures that the learned latent space follows a well-defined probability distribution; second, it reduces the probability of gap formation between point clusters in the latent space, encouraging continuous and meaningful representations. The total loss is calculated by adding the reconstruction loss and the KL divergence loss. A weight factor is often applied to the reconstruction loss to balance the two components, as poorly choosing this weight term can lead to unsatisfactory results.

Learn Before

Related