You are preparing a sentence for a language model training task where the model must predict missing words. Given the original sentence 'The chef prepared a delicious meal for the guests.', transform it into a training input where the words 'delicious' and 'guests' are hidden. Remember to include the special token that marks the beginning of the sequence.

Google

To illustrate a BERT-style input and target output format for Masked Language Modeling, consider the sentence:

"The puppies are frolicking outside the house."

By masking two tokens, such as "frolicking" and "the", the model's input becomes:

```
[CLS] The puppies are [MASK] outside [MASK] house.
```

The corresponding target output starts with a sequence start token `<s>` and contains the predicted original words only at the masked positions, while leaving blanks for the unmasked tokens:

```
<s> ___ ___ ___ frolicking ___ the ___ ___
```

Example of BERT-style Input for Masked Language Modeling

Formatting a Sentence for Model Training

A researcher is preparing the sentence 'The quick brown fox jumps over the lazy dog.' for a language model training task where the model must predict the words 'jumps' and 'dog'. They generate the following input string for the model: `The quick brown fox [MASK] over the lazy [MASK] . [CLS]` 

Analyze this input string. Is it formatted correctly according to the standard procedure for this task? Justify your answer by identifying any specific errors in the formatting.

Learn Before

Related