Choosing a Fine-Tuning Strategy for Sequence Summarization
A data scientist has a pre-trained language model and wants to adapt it for a new task: classifying legal documents into one of several categories (e.g., 'Contract', 'Pleading', 'Motion'). They consider two different fine-tuning approaches:
- Approach A: They add a classification layer that takes the model's final hidden state (corresponding to the last piece of the input text) and train the system to predict the document category.
- Approach B: They add a mechanism that averages all the model's output hidden states for the entire document, feeding this single averaged representation into a classification layer to predict the document category.
Which of these two approaches is more fundamentally aligned with the goal of creating a single, comprehensive representation of the entire document? Justify your reasoning by explaining the potential limitation of the approach you did not choose.
0
1
Tags
Ch.4 Alignment - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
A development team is building a system to classify customer support emails as 'Urgent' or 'Not Urgent'. They start with a general-purpose, pre-trained language model. Their initial strategy involves feeding an email into the model and using the numerical representation of the final word as input for a classifier. This approach yields poor results, often misclassifying long emails where the concluding words are not indicative of the overall sentiment.
To improve performance, the team modifies their approach. They add a new classification component and retrain the entire system on their dataset of labeled emails. The specific goal of this retraining is to adjust the model's parameters so that it produces a single, fixed-size numerical summary that captures the meaning of the entire email. This new summary vector is then used by the classifier, leading to a significant increase in accuracy.
Which of the following statements provides the most accurate evaluation of the team's successful adaptation process?
Choosing a Fine-Tuning Strategy for Sequence Summarization
Critiquing a Document Similarity System