In the Expectation-Maximization (EM) algorithm, directly maximizing the log-likelihood is difficult because the natural logarithm, $$\ln(x)$$, is applied to a summation over latent variables. Jensen's inequality resolves this issue by allowing the logarithm to be moved inside the expectation. This establishes a tractable lower bound on the log-likelihood. The EM algorithm can then iteratively maximize this lower bound, which significantly simplifies the calculation of gradients for parameter estimation.

University of Michigan - Ann Arbor

Google

Jensen's inequality, named after the Danish mathematician Johan Jensen, relates the value of a convex function of an integral to the integral of the convex function. It was proved by Jensen in 1906. Jensen's inequality generalizes the statement that the secant line of a convex function lies above the graph of the function, which is Jensen's inequality for two points: the secant line consists of weighted means of the convex function. For any $$t \in [0, 1]$$ and any $$x_1, x_2$$ in the domain of a convex function $$f$$, the following holds: $$f(t x_1 + (1-t) x_2) \leq t f(x_1) + (1-t) f(x_2)$$.

Jensen's Inequality

If we have complete data $$\left\{X, Z\right\}$$, it is easy to maximize the likelihood $$p(X, Z | \theta)$$. Unfortunately, with incomplete data ($$X$$ only), we must marginalize over the latent variables $$Z$$, so:

$$\ln(p(X|\theta)) = \ln\left(\sum_Z p(X,Z|\theta)\right)$$

The parameters $$\theta$$ inside the logarithm make it difficult to compute the maximum likelihood directly.

Difficulty of Maximum Likelihood Estimation with Latent Variables

Maximum likelihood from incomplete data via the EM algorithm which is proposed by Dempster, Laird and Rubin.
The link is : https://www.ece.iastate.edu/~namrata/EE527_Spring08/Dempster77.pdf

The paper proposing EM algorithm

A common application of Jensen's inequality is to bound a more complicated expression by a simpler one, such as the log-likelihood of partially observed random variables. This is utilized in variational methods using the inequality $$E_{Y \sim P(Y)}[-\log P(X \mid Y)] \geq -\log P(X)$$, since $$\int P(Y) P(X \mid Y) dY = P(X)$$. In this context, $$Y$$ typically represents an unobserved random variable (such as cluster labels in clustering), $$P(Y)$$ is its estimated distribution, and $$P(X \mid Y)$$ is the generative model, with $$P(X)$$ being the distribution with $$Y$$ integrated out.

Learn Before

Related