Google

The architecture of prefix tuning involves augmenting a standard transformer model at each layer. As illustrated in the diagram, a sequence of trainable prefix vectors (e.g., $\mathbf{p}_0^l, \mathbf{p}_1^l$) is prepended to the sequence of hidden states from the user input (e.g., $\mathbf{h}_0^l, \mathbf{h}_1^l, \dots$) at every layer $l$. While the main language model's weights are frozen, only these prefix vectors are updated during training. The final hidden states are used to generate predictions, and the loss is backpropagated to optimize the prefixes for the specific task.

Architecture of Prefix Tuning

A large language model is being adapted to a new task using the prefix tuning method. During the backpropagation phase of training, which components of the model architecture receive gradient updates?

In a model adapted using prefix tuning, a sequence of trainable vectors is prepended to the hidden states at *every* layer, while the main model's parameters remain frozen. Analyze the functional role of these prefix vectors. Specifically, how does the influence of a prefix at an early layer likely differ from the influence of a prefix at a later, deeper layer in the network?

Layer-wise Influence of Prefixes

A researcher is comparing two different methods for adapting a pre-trained transformer model, keeping the original model weights frozen. Method A prepends a sequence of trainable vectors to the input sequence *before* it enters the first layer. Method B prepends a sequence of trainable vectors to the sequence of hidden states *at each layer* of the model. Which statement best analyzes the architectural difference in how these methods influence the model's processing?

Learn Before

Related