Google

Masked prediction is a language modeling task where a model must predict masked or hidden words within a sentence, based on a given context `[C]`. The placeholders for these masked words can be represented in various ways, such as `[M]`, `[X]`, or `[Y]`. For instance, with the context `[C]` being 'The kitten', the model's objective could be to fill in the blanks in phrases like '[M] chasing the [M] .', '[X] the [Y]', or within a partial sentence like `The kitten [M] is [M] .` to reconstruct a complete sentence.

Examples of Masked Prediction Tasks

This example illustrates the target sequence for a masked prediction task where the prediction order is specified. For an input like `[C] The kitten [X] the [Y]`, the model's goal is to generate the missing words in a set order. The target `[X]1 is2 chasing3 [Y]4 ball5 .` indicates that the model should predict 'is', then 'chasing' for the `[X]` placeholder, and subsequently 'ball' for the `[Y]` placeholder. The numbers define the explicit prediction sequence for the masked tokens.

Example of an Ordered Target Sequence for Masked Prediction

A language model is presented with the following input text where a portion is hidden: 'The sun shines brightly in the [X] sky.' Which of the following tasks best represents the model's primary objective if it is performing a masked prediction task?

This example demonstrates a masked prediction task where a model is provided with the input `[C] The kitten [M] chasing the [M] .`. The `[C]` token represents the context, while `[M]` signifies masked words. The model's objective is to predict the missing words, such as 'is' and 'ball', to form a coherent sentence.

Example of Masked Prediction: Kitten Chasing

This example illustrates a masked language modeling task. The model is given an input sentence with missing words, structured as `[C] The kitten [M] playing the [M] .`, where `[C]` denotes the context and `[M]` represents the masked tokens. The objective is for the model to predict the masked words to complete the sentence, for example, 'The kitten is playing with the toy'.

Example of Masked Language Modeling: Kitten Playing

This example presents a simplified target sequence for a language modeling task. The sequence `kitten is chasing ball` represents the desired output from a model. This format is notable for its lack of punctuation, capitalization, and function words like articles, providing a basic, unformatted ground truth for model training or evaluation.

Example of a Simple Target Sequence

This example illustrates a masked prediction task where a model is provided with the input `[C] The kitten [M] is [M] .`. In this structure, `[C]` represents the context, and `[M]` signifies the masked tokens. The model's objective is to predict the missing words, for instance, 'chasing' and 'ball', to form a coherent sentence. This task is distinct because the model must fill in the blanks around a known word ('is'), testing its understanding of grammatical structure.

Example of Masked Prediction with a Known Verb

This example illustrates a masked prediction task using distinct placeholders, `[X]` and `[Y]`, to mark different missing parts of a sentence. Given the context `[C] The kitten`, the model's objective is to predict the words that replace `[X]` and `[Y]` in the structure `[X] the [Y]`.

Example of Masked Prediction with Distinct Placeholders

A research team wants to build a language model that excels at proofreading by understanding the grammatical and contextual relationships between all words in a sentence. They are considering two training methods:

Method 1: The model is given the first part of a sentence (e.g., 'The cat sat on the') and must predict the next word (e.g., 'mat').

Method 2: The model is given a full sentence with some words removed (e.g., 'The [BLANK] sat on the [BLANK]') and must predict the missing words (e.g., 'cat', 'mat').

Evaluate which method is more suitable for the team's goal. Justify your choice by explaining how the information available to the model during training differs between the two methods and why that difference is important for the proofreading task.

Choosing a Model Training Strategy

Given the following scenario, which training approach should the research team choose to best achieve their specific goal? Justify your answer by explaining how the chosen approach helps the model learn the intended skills.

Learn Before

Related