It's an important principal in selecting the model and describing the math formula of a model's loss function. Let $p_{model}(x,\theta)$ be a parametric family of possibility distribution over the same space indexed by $\theta$. The maximum likelihood estimator for $\theta$ is then defuned as 
$\theta_{ML} = argmax_{\theta}p_{model}(X;\theta) $

Maximum Likelihood Estimation

When modeling categorical data, the maximum likelihood estimate for the probability that a feature $$k$$ takes a specific value $$l$$, denoted as $$\theta_{kl} = p(x_k = l)$$, is given by its empirical frequency in the dataset:

$$\theta_{kl} = \frac{n_{kl}}{N}$$

where $$n_{kl}$$ is the number of times feature $$k$$ takes the value $$l$$, and $$N$$ is the total number of observations.

Maximum Likelihood Estimate for a Categorical Feature

Maximum likelihood is a fundamental principle in statistics and machine learning used to estimate the parameters of an assumed probability distribution. The core idea is to find the parameter values that make the observed data most probable under the assumed model. This concept forms the theoretical foundation for maximum likelihood estimation (MLE).

San Diego State University

Google

1. _Sample Space_ is set of all values that observation x can take.
2. _PDF_ (Probability Density Function) is the function mapping point from sample space to the number between 0 and 1. Area under PDF is equal to 1.
3. _Parametric Models_ are density functions having finite number of parameters.
4. _Likelihood _ is defined as $L(\theta | x) = p_{\theta}(x)$
5. _Maximum Likelihood Estimation_ is defined as $\theta = argmax L(\theta | X)$.
It is used to measure the set of parameters that are more apt to explain observed data X.

Quick Recap For Some Probability Concepts

Maximum Likelihood

The formula $\mathcal{L}(\mathbf{p}_{i+1}^{\theta}, \mathbf{p}_{i+1}^{\text{gold}})$ represents a loss function that quantifies the difference between a model's predicted probability distribution, $\mathbf{p}_{i+1}^{\theta}$, and the ground truth or "gold" probability distribution, $\mathbf{p}_{i+1}^{\text{gold}}$. The predicted distribution is parameterized by $\theta$, which are the model's parameters that are updated during training. The goal of training is typically to minimize this loss, thereby making the predicted distribution as close as possible to the true distribution. The subscript $i+1$ suggests this is often used in sequential contexts, like predicting the next element in a sequence.

Loss Function for Predicted vs. Gold Probability Distributions

Two features $$x_i$$ and $$x_j$$ are considered statistically independent if the occurrence or value of one does not affect the probability of the other. Mathematically, this is expressed as $$p(x_i|x_j) = p(x_i)$$.

Learn Before

Related

Learn After