Learn Before
Example of Predicting Masked Words: Kitten Playing
This example presents a language modeling task where the goal is to predict masked tokens, denoted by [M], based on the surrounding context, [C]. The specific structure provided is: [C] The kitten [M] playing the [M] .
0
1
Tags
Ch.1 Pre-training - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
Example of Predicting Masked Words: Kitten Playing
Example of Masked Language Modeling: Kitten Chasing Ball
Example of Context-Based Prediction: Kitten Chasing Ball
In a sequence-to-sequence model, an attention mechanism calculates a score for three input vectors (A, B, and C) relative to a single output vector (D). The scoring function is the simple dot product between the output vector and each input vector. You are given the following geometric relationships:
- Vector A points in a very similar direction to Vector D.
- Vector B is orthogonal (at a 90-degree angle) to Vector D.
- Vector C points in the opposite direction of Vector D.
Which input vector will receive the highest attention score, and what is the underlying reason for this?
Evaluating Attention Mechanisms in Machine Translation
Calculating a Dot Attention Score
Learn After
A language model is tasked with predicting the masked word, represented by
[M], in the sentence: 'The librarian placed the overdue [M] back on the shelf.' Based on an analysis of the surrounding words, which of the following options is the most probable prediction?A language model is tasked with predicting the masked word
[M]in the two sentences below. Analyze the surrounding words in each sentence.Sentence 1:
The skilled mechanic quickly repaired the [M] with a new part.Sentence 2:He saw the [M] over there.Which sentence provides a more constrained and helpful context for the model to make an accurate prediction, and why?
Analyzing Context for Word Prediction