Google

This example illustrates a masked prediction task where a model is provided with the input `[C] The kitten [M] is [M] .`. In this structure, `[C]` represents the context, and `[M]` signifies the masked tokens. The model's objective is to predict the missing words, for instance, 'chasing' and 'ball', to form a coherent sentence. This task is distinct because the model must fill in the blanks around a known word ('is'), testing its understanding of grammatical structure.

Example of Masked Prediction with a Known Verb

A language model is given the input `The scientist [M] is [M] a new theory.` and generates two potential pairs of predictions for the masked tokens `[M]`:

Prediction A: `('developing', 'testing')`
Prediction B: `('developed', 'tested')`

Identify which prediction pair (A or B) is grammatically superior and explain *why* the presence of the word 'is' in the input makes that pair a better choice.

Analyzing Grammatical Constraints in Masked Prediction

Based on the researcher's goal and proposed training method described in the case study, analyze the primary advantage of using an input structure with a known, unmasked verb. How does this specific structure provide a more targeted test of grammatical understanding compared to a structure where all words following the subject are masked?

Analyzing a Training Method for Language Models

A language model is given the task of predicting the words for the masked tokens `[M]` in the following input: `The car [M] was [M] by the mechanic.` Which of the following pairs of predictions for the two `[M]` tokens demonstrates the strongest understanding of how the known words ('was', 'by') constrain the grammatical structure of the sentence?

Learn Before

Related