Learn Before
Example of Masked Prediction with a Known Verb
This example illustrates a masked prediction task where a model is provided with the input [C] The kitten [M] is [M] .. In this structure, [C] represents the context, and [M] signifies the masked tokens. The model's objective is to predict the missing words, for instance, 'chasing' and 'ball', to form a coherent sentence. This task is distinct because the model must fill in the blanks around a known word ('is'), testing its understanding of grammatical structure.
0
1
Tags
Ch.1 Pre-training - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
Example of an Ordered Target Sequence for Masked Prediction
A language model is presented with the following input text where a portion is hidden: 'The sun shines brightly in the [X] sky.' Which of the following tasks best represents the model's primary objective if it is performing a masked prediction task?
Example of Masked Prediction: Kitten Chasing
Example of Masked Language Modeling: Kitten Playing
Example of a Simple Target Sequence
Example of Masked Prediction with a Known Verb
Example of Masked Prediction with Distinct Placeholders
Choosing a Model Training Strategy
Selecting a Training Objective for a Grammar-Focused Model
Learn After
A language model is given the task of predicting the words for the masked tokens
[M]in the following input:The car [M] was [M] by the mechanic.Which of the following pairs of predictions for the two[M]tokens demonstrates the strongest understanding of how the known words ('was', 'by') constrain the grammatical structure of the sentence?Analyzing Grammatical Constraints in Masked Prediction
Analyzing a Training Method for Language Models