Learn Before
Attention Mechanism Output Analysis
Given the scenario below, which of the 'value' vectors will the final output vector most closely resemble? Justify your answer by describing how the intermediate steps of the calculation (specifically, the function that normalizes scores into weights and the final weighted sum) lead to this result.
0
1
Tags
Ch.5 Inference - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Application in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
In a mechanism that calculates attention, scores are computed by taking the dot product of a query vector with a set of key vectors. These scores are then scaled by dividing by the square root of the dimension of the vectors (e.g., ) before being passed to a Softmax function. What is the most likely adverse consequence of removing this scaling step, particularly when the vector dimension is large?
A computational mechanism is used to determine the relevance of different parts of an input sequence to a specific element. This involves several steps to produce a final output vector. Arrange the following computational steps in the correct chronological order.
Attention Mechanism Output Analysis