A language model is being trained using a masked language modeling objective. The original input sentence is 'A quick brown fox jumps over the lazy dog'. During a training step, the tokens 'quick' (at position 2) and 'lazy' (at position 8) are masked. The model receives the corrupted input, denoted as $\bar{\mathbf{x}}$: '[CLS] A [MASK] brown fox jumps over the [MASK] dog'. Which of the following mathematical expressions correctly represents the training objective for this specific step, which the model aims to maximize?

A language model is being trained on a sentence where two words have been replaced with a special [MASK] token. The training objective is to maximize the sum of the log-probabilities of the original words at these two masked positions. Why is the objective formulated as a sum of log-probabilities rather than, for example, a product of the probabilities?

Evaluating Model Performance in MLM Training