Google

An example of token masking can be seen with a two-sentence input prepared for BERT. The original sequence, `[CLS] It is raining . [SEP] I need an umbrella . [SEP]`, is modified by replacing selected tokens with the `[MASK]` symbol. This results in the masked sequence: `[CLS] It is [MASK] . [SEP] I need [MASK] umbrella . [SEP]`, where the model's task is to predict the original words 'raining' and 'an'.

Example of Token Masking in a BERT Input Sequence

A language model is presented with the following modified input: `[CLS] The sky is [MASK] . [SEP] The grass is [MASK] . [SEP]`. The model's task is to predict the original words, which were 'blue' and 'green', respectively. What was the complete, original input sequence before the words were replaced?

Reconstructing an Original Input Sequence

A researcher is preparing the input sequence `[CLS] The cat sat on the mat . [SEP]` for a masked language modeling task. To train the model to predict the word 'sat', the researcher replaces 'sat' with the special token ____.

A language model is being prepared to predict missing words in a two-sentence input. The original input is: `[CLS] The sun is shining brightly . [SEP] Birds are singing loudly . [SEP]`. If the words 'shining' and 'singing' are selected for masking, which of the following represents the correctly modified input sequence?

Learn Before

Related