Learn Before
Analyzing the Mechanism of Prefix Tuning
A parameter-efficient fine-tuning technique involves keeping a large language model's parameters frozen while introducing a small set of new, trainable vectors. These vectors are prepended to the sequence of hidden states at each layer of the model. Analyze the primary reason why these vectors are added at every layer, rather than only at the initial input layer, to steer the model's behavior for a new task.
0
1
Tags
Ch.3 Prompting - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Architecture of Prefix Tuning
A team is tasked with adapting a very large, pre-trained language model for a specialized legal document analysis task. To conserve computational resources and avoid altering the base model, they freeze all of the original model's parameters. They then introduce a small set of new, trainable parameters that are prepended to the sequence of hidden states within each transformer layer. During training for the new task, only these new parameters are updated. Which statement best analyzes the main consequence of this specific training strategy?
Choosing a Fine-Tuning Strategy
Analyzing the Mechanism of Prefix Tuning