Learn Before
An engineer is optimizing a language model to handle very long text sequences, such as entire books. They decide to replace the standard attention mechanism with one that projects query and key vectors into a different feature space. This change allows them to substitute the original, complex normalization function with a much simpler scaling operation. What is the fundamental trade-off associated with this specific modification?
0
1
Tags
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Evaluation in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
In a modified attention mechanism designed for computational efficiency, the query and key vectors are transformed using a feature map projection. What is the primary reason for this transformation in the context of calculating the final attention output?
Role of Feature Projection in Attention Normalization
An engineer is optimizing a language model to handle very long text sequences, such as entire books. They decide to replace the standard attention mechanism with one that projects query and key vectors into a different feature space. This change allows them to substitute the original, complex normalization function with a much simpler scaling operation. What is the fundamental trade-off associated with this specific modification?