Learn Before
Calculating a Dot Attention Score
In a sequence-to-sequence model using an attention mechanism, the current decoder hidden state is represented by the vector h'_t = [0.5, 1.0, -0.2]. The model is evaluating an encoder hidden state represented by the vector h = [0.6, 0.8, 0.1]. Using the dot product scoring function, calculate the attention score for this pair of vectors. Show your calculation.
0
1
Tags
Data Science
Foundations of Large Language Models Course
Computing Sciences
Ch.1 Pre-training - Foundations of Large Language Models
Foundations of Large Language Models
Application in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Example of Predicting Masked Words: Kitten Playing
Example of Masked Language Modeling: Kitten Chasing Ball
Example of Context-Based Prediction: Kitten Chasing Ball
In a sequence-to-sequence model, an attention mechanism calculates a score for three input vectors (A, B, and C) relative to a single output vector (D). The scoring function is the simple dot product between the output vector and each input vector. You are given the following geometric relationships:
- Vector A points in a very similar direction to Vector D.
- Vector B is orthogonal (at a 90-degree angle) to Vector D.
- Vector C points in the opposite direction of Vector D.
Which input vector
Evaluating Attention Mechanisms in Machine Translation
Calculating a Dot Attention Score