Learn Before
Analysis of Positional Penalty Growth
A positional bias mechanism uses the formula Bias = -βlog(1 + β) to penalize attention between tokens, where β is a positive value that increases with the distance between the tokens. Describe how the rate of increase of this penalty changes as the distance between tokens grows larger. In other words, is the additional penalty for one extra unit of distance greater for nearby tokens or for very distant tokens? Justify your answer based on the properties of the function.
0
1
Tags
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models
Computing Sciences
Foundations of Large Language Models Course
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
A model designer implements a positional bias using the formula
Bias = -βlog(1 + β), whereβis a positive value that increases with token distance. The goal is to penalize attention to more distant tokens. By mistake, the designer forgets the leading negative sign, implementing the formula asBias = βlog(1 + β). What is the most likely effect of this error on the model's behavior?Analysis of Positional Penalty Growth
Selecting a Positional Bias Strategy