Probability of a True Token in MLM

Predicted Probability Distribution in MLM

Example of MLM Training Objective with Multiple Masks

MLM Loss Function as Negative Log-Likelihood

A language model is being trained to fill in a masked word. For the input 'The cat sat on the [MASK]', the correct word is 'mat'. The training objective is to adjust the model to minimize the cross-entropy loss for its predictions. Below are four different potential outputs from the model, showing the probability it assigns to the word 'mat'. Which of these outputs would result in the LOWEST loss for this specific training example?

Evaluating Model Performance via Cross-Entropy Loss

According to the standard Masked Language Modeling (MLM) training objective, a model's parameters are adjusted based on the cross-entropy loss calculated for a single, strategically chosen masked token within a training batch, aiming to optimize performance on that specific prediction.

MLM Loss Function as Negative Log-Likelihood

A neural network is trained on a 4-class classification task. For a single training example where the true class is the second class, the model outputs the probability vector [0.1, 0.7, 0.1, 0.1]. The loss for this example is calculated as -log(0.7). This loss function can be interpreted as a measure of divergence between two probability distributions. What are these two distributions?

Interpreting Negative Log-Likelihood as Cross-Entropy

A neural network is being trained for a 3-class classification task (Classes A, B, C). For a single training example, the true label is 'Class B'. The model outputs the probability distribution P(A)=0.2, P(B)=0.5, P(C)=0.3. The loss for this example is calculated using the negative log-likelihood of the correct class, resulting in a loss of -log(0.5). This calculation is a direct application of the cross-entropy formula between the model's predicted distribution and the empirical distribution from the training data. What is the specific empirical probability distribution for this single training example?