Learn Before
Example of Masked Language Modeling: Kitten Chasing Ball
This example illustrates a masked language modeling task using the sentence [C] The kitten is chasing the ball . as the original text. The model is provided with an input where some words are replaced by a mask token, such as [C] The kitten [M] [M] [M] ball .. In this structure, [C] denotes the context and [M] represents the masked tokens. The model's objective is to predict the original words that were masked, which are 'is', 'chasing', and 'the'.
0
1
Tags
Ch.1 Pre-training - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
Example of Predicting Masked Words: Kitten Playing
Example of Masked Language Modeling: Kitten Chasing Ball
Example of Context-Based Prediction: Kitten Chasing Ball
In a sequence-to-sequence model, an attention mechanism calculates a score for three input vectors (A, B, and C) relative to a single output vector (D). The scoring function is the simple dot product between the output vector and each input vector. You are given the following geometric relationships:
- Vector A points in a very similar direction to Vector D.
- Vector B is orthogonal (at a 90-degree angle) to Vector D.
- Vector C points in the opposite direction of Vector D.
Which input vector will receive the highest attention score, and what is the underlying reason for this?
Evaluating Attention Mechanisms in Machine Translation
Calculating a Dot Attention Score
Learn After
Evaluating Masking Strategies for Model Training
A language model is being trained using a masked language modeling objective. If the original input sentence is 'The innovative chef prepares a delicious meal for the guests.', which of the following masked versions would be the most difficult for the model to reconstruct accurately?
In a masked language modeling task with the input
The student opened their [M] to study for the exam., the model's prediction for the masked word is solely determined by the words that come before the mask (The student opened their).