Diagnosing Generalization Failure in a Transformer Model
Based on the provided case study, analyze the likely cause of the model's performance degradation on longer documents and explain how implementing a parameter-sharing strategy for relative positions could address the issue.
0
1
Tags
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
A language model was trained exclusively on text segments with a maximum length of 512 tokens. During inference, it must process a 1000-token document, encountering a query-key offset of 700 for the first time. Why is a model architecture that groups offsets into 'buckets' and shares a single learnable parameter per bucket better equipped to handle this novel offset than a hypothetical model that learns a unique, separate parameter for every individual offset?
Generalization Through Parameter Sharing
Diagnosing Generalization Failure in a Transformer Model