Evaluating Pre-training Strategies for Specialized AI
Imagine two AI development teams are building a language model intended to assist with legal document review.
- Team A advocates for pre-training the model on an extremely large and diverse dataset, including books, news articles, websites, and social media, in addition to a collection of legal texts.
- Team B argues it is more efficient to pre-train the model exclusively on a massive corpus of legal documents, court proceedings, and law journals.
Evaluate the two pre-training strategies. Which team's approach is more likely to produce a model that can robustly handle novel or unusual legal scenarios and documents it has not seen before? Justify your evaluation based on the principles of model generalization.
0
1
Tags
Ch.4 Alignment - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Evaluation in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Reduced Necessity of Fine-Tuning for Generalization with Extensive Pre-training
AI Model Training Strategy for Generalization
A research lab is developing a new language model. Their primary goal is to create a model that can reliably handle tasks and data types it was not explicitly trained on, such as analyzing niche scientific papers and summarizing newly emerging slang on social media. They are considering two main training strategies:
Strategy A: Curate a massive, diverse dataset from a wide range of sources (books, web pages, code, academic articles, social media) and use the majority of their computational budget for an extensive pre-training phase.
Strategy B: Use a smaller, more generic dataset for a quick pre-training phase, and then dedicate the majority of their computational budget to meticulously fine-tuning the model on hundreds of specific, narrow tasks.
Based on empirical findings about model generalization, which strategy is more likely to achieve the lab's primary goal and why?
Evaluating Pre-training Strategies for Specialized AI