Learn Before
Example of Masked Language Modeling: Kitten Chasing Ball
This example illustrates a masked language modeling task using the sentence [C] The kitten is chasing the ball . as the original text. The model is provided with an input where some words are replaced by a mask token, such as [C] The kitten [M] [M] [M] ball .. In this structure, [C] denotes the context and [M] represents the masked tokens. The model's objective is to predict the original words that were masked, which are 'is', 'chasing', and 'the'.
0
1
Tags
Ch.1 Pre-training - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
Example of Predicting Masked Words: Kitten Playing
Example of Masked Language Modeling: Kitten Chasing Ball
Example of Context-Based Prediction: Kitten Chasing Ball
In a sequence-to-sequence model, an attention mechanism calculates a score for three input vectors (A, B, and C) relative to a single output vector (D). The scoring function is the simple dot product between the output vector and each input vector. You are given the following geometric relationships:
- Vector A points in a very similar direction to Vector D.
- Vector B is orthogonal (at a 90-degree angle) to Vector D.
- Vector C points in the opposite direction of Vector D.
Which input vector
Evaluating Attention Mechanisms in Machine Translation
Calculating a Dot Attention Score
Learn After
Evaluating Masking Strategies for Model Training
A language model is being trained using a masked language modeling objective. If the original input sentence is 'The innovative chef prepares a delicious meal for the guests.', which of the following masked versions would be the most difficult for the model to reconstruct accurately?
In a masked language modeling task with the input
The student opened their [M] to study for the exam., the model's prediction for the masked word is solely determined by the words that come before the mask (The student opened their).