A research lab is developing a new foundation model with a limited computational budget. They are considering two primary approaches for the initial training phase:
- Approach 1: Train the model on an extremely large and diverse dataset, incorporating text from the web, academic articles, books, and code, using a general-purpose learning objective.
- Approach 2: Train the model on a smaller, but very high-quality, curated dataset focused on a few key domains (e.g., customer service and technical support dialogues) and then immediately test its performance on tasks within those domains.
Which statement best analyzes the fundamental trade-off between these two approaches in the context of building a foundation model?
0
1
Tags
Ch.1 Pre-training - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Analyzing a Model Development Lifecycle
A research lab is developing a new foundation model with a limited computational budget. They are considering two primary approaches for the initial training phase:
- Approach 1: Train the model on an extremely large and diverse dataset, incorporating text from the web, academic articles, books, and code, using a general-purpose learning objective.
- Approach 2: Train the model on a smaller, but very high-quality, curated dataset focused on a few key domains (e.g., customer service and technical support dialogues) and then immediately test its performance on tasks within those domains.
Which statement best analyzes the fundamental trade-off between these two approaches in the context of building a foundation model?
Balancing Generalization and Specialization