Learn Before
Layer-wise Influence of Prefixes
In a model adapted using prefix tuning, a sequence of trainable vectors is prepended to the hidden states at every layer, while the main model's parameters remain frozen. Analyze the functional role of these prefix vectors. Specifically, how does the influence of a prefix at an early layer likely differ from the influence of a prefix at a later, deeper layer in the network?
0
1
Tags
Ch.3 Prompting - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
A large language model is being adapted to a new task using the prefix tuning method. During the backpropagation phase of training, which components of the model architecture receive gradient updates?
A researcher is comparing two different methods for adapting a pre-trained transformer model, keeping the original model weights frozen. Method A prepends a sequence of trainable vectors to the input sequence before it enters the first layer. Method B prepends a sequence of trainable vectors to the sequence of hidden states at each layer of the model. Which statement best analyzes the architectural difference in how these methods influence the model's processing?
Layer-wise Influence of Prefixes