In linear regression condition, if we compare the simplified math format of MLE of the conditional log likehood with the Mean Squared Error, it's the same estimate aiming to minimize the mean squared error.

University of Michigan - Ann Arbor

It's an important principal in selecting the model and describing the math formula of a model's loss function. Let $p_{model}(x,\theta)$ be a parametric family of possibility distribution over the same space indexed by $\theta$. The maximum likelihood estimator for $\theta$ is then defuned as 
$\theta_{ML} = argmax_{\theta}p_{model}(X;\theta) $

Maximum Likelihood Estimation

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Deep Learning

KL divergence is used to estimate difference between probabilities. We want to measure the difference between our normal distribution with mu and log_var parameters and standard normal distribution.  Here we have following formula for the KL_Loss (it has closed form):

kl_loss = -0.5 * sum(1 + log_var - mu ^ 2 - exp(log_var))

Here sum is taken from every dimension from the latent space. In ideal situation kl_loss will be equal to zero and this is achievable  **iff** mu and log_var are equal to 0. 

Kullback-Leibler Divergence

One way to interpret maximum likelihood estimation is to view it as minimizing the dissimilarity between the empirical distribution and the model distribution. We can measure the degree of the dissimilarity using KL divergence.

Relationship between KL Divergence and MLE

The cross-entropy loss function works very well for models that predict binary classes (aka the output is between 0 and 1). It is defined as -[y*log(y-hat) +(1-y)*log(1-(y-hat))]. If y=0 the left side of the function is dropped and the right side, -log(1-(y-hat)), is used. Otherwise if y=1 the right side of the function is dropped and it uses -log(y-hat). In both instances this loss function encourages probabilities that are close to the true probability. 

Cross-entropy loss

A conditional probability estimate can be built from a generalized maximum likelihood estimator, $P(y|x;\theta)$. For all inputs, $X$, and  all observed targets, $Y$, that is 
$$\theta_{ML}=argmax_{\theta} P(Y|X;\theta)$$
That formula can, if examples are i.i.d., be decomposed into
$$\theta_{ML}=argmax_{\theta} \sum_{i=1}^{m} log P(y^{(i)}|x^{(i)};\theta)$$

Conditional Log-Likelihood

Mean Squared Error is simply the variance of a potentially biased estimator $\hat{\theta}_m$ of the Expected Value or Mean $\theta$
$$MSE(\hat{\theta}_m) = \mathbb{E}((\hat{\theta}_m - \theta)^2)$$
$$ = Bias^2(\hat{\theta}_m) + Var(\hat{\theta}_m)$$

Mean Squared Error

Relationship between MSE and MLE

Under appropriate conditions, as the number of training examples approaches inﬁnity, the maximum likelihood estimate of a parameter converges to the true value of the parameter.

The property of consistency of maximum likelihood

One consistent estimator may obtain lower generalization error for a ﬁxed number of samples $m$, or equivalently, may require fewer examples to obtain a ﬁxed level of generalization error.

Statistical Eﬃciency Principal of MLE

- The true distribution $p_{\text{data}}$ must lie within the model family $p_{\text{model}}(.;\theta)$. Otherwise, no esitmator can recover $p_{\text{data}}$.
- The true distribution $p_{\text{data}}$ must correspond exactly to one value of $\theta$. Otherwise, the maximum likelihood can recover the correct $p_{\text{data}}$ but will not be able to determine which value of $\theta$ was used by the data generating process.

Maximum Likelihood Estimator Properties

The gradient of the log-likelihood which is used in Maximum Likelihood Estimation can be decomposed into the following:
$$\nabla_{\boldsymbol \theta} \log p(\mathbf{x}; \boldsymbol \theta) = \nabla_{\boldsymbol \theta} \log \tilde{p}(\mathbf{x}; \boldsymbol \theta) - \nabla_{\boldsymbol \theta } \log Z(\boldsymbol \theta)$$
Where $\tilde{p}(\mathbf{x}; \boldsymbol \theta)$ is the unnormalized probabilioty density and $Z$ is the partition function. This is well-known as the decomposition into the positive phase and negative phase of learning. Due to the reliance of the partition function on the parameters, learning models by maximum likelihood is particularly difficult. 

Learn Before

Related