Strategic Decision for a New Language Model
Based on the primary challenge of creating large models from scratch, which of the two approaches described in the case study is more feasible for the lab, and why?
0
1
Tags
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Evaluation in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Strategic Decision for a New Language Model
A small, budget-conscious startup aims to create a novel instruction-following language model. Their strategy involves integrating specialized instruction-response pairs directly into the pre-training phase. What is the most significant practical challenge this startup will likely encounter when attempting to train their new model entirely from scratch?
A well-funded academic research lab proposes to create a new, state-of-the-art, instruction-following language model. Their plan is to train the model entirely from scratch on a massive dataset of general text combined with specialized instruction-response pairs. This approach is considered a practical and cost-effective strategy for such an organization.